Submitted to USAID on March 21, 2012 This publication was produced for review by the United States Agency for International Development. EVALUATION Final Evaluation of the Advancing Armenia's Elections Management and Democratic Culture (AAEM-DC) Project ARMENIA Anzhela Aslanyan, IFES IFES EVALUATION REPORT Submitted to USAID on March 21, 2012 This publication was produced for review by the United States Agency for International Development. IFES EVALUATION REPORT MARCH 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016-00, Task Order Number AID-111-TO-11-00002 Evaluation of ABA-ROLI, CEPPS, SATR AND SSIP Projects DISCLAIMER The authors’ views expressed in this publication do not necessar ily ref lect the views of the United States Agency for International Development or the United States Government. Authors: Robert Brandstetter and Hrachya Zakoyan IFES Evaluation Report International Business & Technical Consultants, Inc. TABLE OF CONTENTS Executive Summary ......................................................................................................................... i I. Introduction ......................................................................................................................... 1 II. Developm ent Hypothesis .................................................................................................... 2 III. Evaluation Scope of Work and Methodology ..................................................................... 3 IV. Responses to the Evaluation Questions presented in the Scope of Work ........................... 5 A. General Questions................................................................................................................... 5 B. Project-Specific Questions.................................................................................................... 9 V. Findings.................................................................................................................... ......... 11 VI. Gender Considerations ...................................................................................................... 23 VII. Conclusions ....................................................................................................................... 23 VIII. Recommendations ............................................................................................................. 24 Annex I: Evaluation Scope of Work ............................................................................................. 27 Annex II: Evaluation Work Plan ................................................................................................... 35 Annex III: List of Informants for Field Interviews ....................................................................... 38 Annex IV: Interview Guides ......................................................................................................... 40 Annex V: Key Reference Documents Reviewed .......................................................................... 44 Annex VI: Schedule of Meetings .................................................................................................. 48 Annex VII: List of People Consulted ............................................................................................ 52 Annex VIII: Conflict of Interest Statements……………………………………………………..54 Annex IX: Statement of Difference……………………………………………………………...57 IFES Evaluation Report International Business & Technical Consultants, Inc. LIST OF ACRONYMS AAEM-DC “Advancing Armenia’s Election Management and Democratic Culture” Project ACEEEO Association of Central and East European Election Officials ANC Armenian National Congress CASP Civic Advocacy Support Program CEC Central Election Commission CEPPS Consortium for Elections and Political Process Strengthening CRRC Caucasus Research Resource Center EID European Institute for Democracy EMB Election Management Body EMAP Electo ral Management Advisory Panel EU European Union FOICA Freedom of Information Center of Armenia GOA Governm ent of Armenia IYC Its Your Choice LSG Local Self-government MAAC Mobilizing Action Against Corruption MP Mem ber of Parliament MAP Money and Politics NA National Assembly of the Republic of Armenia NDI National Democratic Institute NED National Endowment for Democracy NGO Non-governm ental Organization NPC National Press Club PPA People’s Party ODIHR Office for Democratic Institutions and Human Rights IFES Evaluation Report International Business & Technical Consultants, Inc. OSCE Organization for Security and Cooperation in Europe OVIR Otdel viz i registratsii [Department of Passports and Visas] PEC Precinct Election Commission PFA Policy Forum Armenia PSA Public Service Announcement RA Republic of Armenia SEPA Strengthening Electoral Processes and Administration in Armenia Project SAC Student Action Committees SOW Scope of Work TEC Territorial Election Commission USAID United States Agency for International Development VL Voters List VLAC Voters List Advisory Committee VR Voter Registry YCAP Youth Community Action Program IFES Evaluation Report International Business & Technical Consultants, Inc. Page i EXECUTIVE SUMMARY Introduction This is the final evaluation of the Advancing Armenia’s Elections Management and Democratic Culture (AAEM-DC) Project implemented by the International Foundation for Election Systems (IFES).The project began in April 2009 and is scheduled to conclude in January 2012, pending a possible six month extension. The total budget for the project was $2,000,000. The purpose of the project was to strengthen the administration of electoral processes and assist the electoral administration to meet international standards for free and fair elections through its support of the Central Election Commission (CEC) and the Passport and Visa Department of the Police of the Republic of Armenia (OVIR) which is responsible for maintaining the national voter list. The purpose of the evaluation is to assess the success of the project and to evaluate its effectiveness in achieving its programmatic goals and USAID/Armenia’s strategic objectives. Development Hypothesis The AAEM-DC project was intended to support two key electoral institutions: CEC and OVIR. The development hypothesis was that by promoting the independent and sustainable capacities of Armenia’s electoral and civic registration institutions, and deepening a commitment to international election standards and practices, the project would help revitalize public confidence and broaden participation in the electoral process. Methodology The evaluation was carried out by Mr. Hrachya Zakoyan from Armenia and Mr. Robert Brandstetter, from Washington, DC, under a contract to International Business & Technical Consultants, Inc. (IBTCI). The team reviewed available documents provided by USAID/Armenia and other written materials prior to spending 14 days of field work in Armenia. In Armenia, the team interviewed approximately 45 individual stakeholders from all levels of Armenian society. Most of the meetings were in Yerevan, although the team also visited four marzes. Meetings were primarily one-on-one informational interviews, for which the questions in the Scope of Work (SOW) served as general guidelines. The project may be active for another six months, a period which includes the parliamentary election in May 2012. Activities which are implemented during this period are not included in this evaluation. Evaluation Questions The SOW included two sets of questions for the evaluation. One set included nine general questions, and the other included five questions specifically related to the AAEM-DC project. These questions are addressed in the text of the report. Findings The IFES AAEM-DC project had four objectives: IFES Evaluation Report International Business & Technical Consultants, Inc. Page ii Project Objective 1: To improve the voter registration system and insulate it from abuse. According to OVIR and the chairman of the CEC, the voter registration list is the ‘cleanest’ it has ever been. The list is based on the State Registry of Population, which is updated every 15 days. The head of OVIR, a branch of the state police, was not amenable to most of IFES’s offers of assistance. He was very proud of his work in developing the State Registry system and did not think that he needed any additional help. OVIR did ask for some assistance to improve the process of updating the voter registry and IFES will produce a policies and procedures manual and provide training in response. But IFES had less input into working with OVIR than was originally anticipated. Another activity for this objective was to establish Voter List Advisory Committees (VLAC). These committees were intended to field verify the voter list for OVIR and do voter education and awareness activities. These activities were not implemented. The OVIR head did not welcome the VLACs and did not think they were helpful to improving the voter list. Since there were no elections as of the time of the evaluation, the outreach aspects of the VLACs were not put into practice. Project Objective 2: To assist in election administration reform. IFES helped host, organize and contribute to two major election-related international conferences. In 2010 a symposium on Electoral Legislation Reform included all of the national and international stakeholders involved with passage of a new electoral code for Armenia. These conferences, supported with funding from USAID and the international community, gave a venue for the major players involved with writing the legislation for a new code. IFES was the lead organization for the international community in the National Assembly and the Standing Committee on State and Legal Affairs, contributing a variety of technical issue papers requested by these bodies. A new electoral code was adopted by the National Assembly on 26 May 2011, and signed into law by the President on 14 June 2011. The electoral code is the authoritative law for the CEC and all elections in the Republic. IFES also made a Workflow Analysis of election administration for the CEC. The implementation of recommendations from this study is pending. IFES’s assistance to the CEC also includes the production of 40,000 accreditation preparation manuals for Precinct Election Commission (PEC) candidates and local observers, as well as training these people in accordance with the new election code. Project Objective 3: To strengthen the political will to change the election culture. This objective was intended to help overcome the massive negativity with the political system and the electoral process that captured the country after the 2008 elections. IFES proposed to conduct a series of Politician-Citizen Discussion Groups over the life of the project, with the intention of generating civilized debate and providing transparency to the political system and the electoral process. This activity was replaced by the November 2010 Electoral Legislative Reform Symposium, described above, and the Election Management Advisory Panels (EMAP). An achievement of the project’s effort to effect change in the electoral culture by working with youth and schools was the publishing of a booklet designed for use in secondary schools. IFES plans to distribute this booklet to the nation’s high schools and other educational institutions, with the approval of the Ministry of Education, Project Objective 4: To assist in campaign finance reform through a money and politics program. Aspects of the campaign finance legal reform were successful because they became an integral part of the general election administration reform and electoral code legislation. IFES IFES Evaluation Report International Business & Technical Consultants, Inc. Page iii was able to influence at least four articles in the new code concerning campaign finance. Also, an Audit and Oversight Service was established in the CEC, following a recommendation by IFES. The remaining activities- campaign finance awareness and campaign finance media training were removed from the IFES work plan at the request of USAID in the fall of 2011. Conclusions 1. The absence of a monitoring and evaluation (M&E) system with baselines and SMART (Specific, Measurable, Achievable, Realistic, Time-bound) indicators and specified targets as well as a lack of collected program data made conducting an external evaluation more difficult. Particularly, based on a review of the IFES performance monitoring plan indicator table, it is clear that a majority (around 70%) of the data to be collected (targets and actual measurement numbers) are not available (suspended activities excluded). Surveys are mentioned as a source of information for some indicators. For example, a survey was envisioned to measure the percent increase of citizens who have confidence in the work of election officials, but the evaluation team did not receive any survey details or reports. 2. The current voter list is generated from the State Registry of Population, a national database of all Armenian citizens, and maintained and regularly updated by OVIR. According to most informants, the voter list is a clean and updated list. One of IFES’s major assumptions in implementing the project was that it receives the full cooperation of the Government of Armenia and other government entities such as OVIR. While IFES has had cordial relations with the head of OVIR, the head of OVIR was less welcoming of IFES offers of technical assistance. This resistance to assistance was an important reason why IFES decreased its planned activities for the related objective. Nevertheless, some assistance was requested to improve the voter registration process and for a policies and procedures manual for OVIR officers. The Voter List Advisory Committees (VLACs) were originally a component of the assistance that IFES intended to provide OVIR to help verify the accuracy of the voter list. However, OVIR noted that the VLACs were largely superfluous to the maintenance of the voter list, and they have not been established. 3. The acceptance and passage of the new electoral code can be attributed to the leading role IFES played in the international community on this topic. It is also where IFES made the greatest impact vis-à-vis the AAEM-DC project. IFES filled this role because the major donors, OSCE and the EU, did not have the capacity to have representatives in Armenia with the technical skills needed to work regularly with the National Assembly. IFES’s ongoing assistance to the CEC to implement the administrative reforms of the code reinforces and contributes greatly to the sustainability of the reforms of the new code. Anticipated new computer hardware and software facilitated by IFES will also reinforce these reforms. 4. The three year project life was too short a timeframe to have an impact on revitalizing public confidence and broadening participation in the electoral process as it requires a significant change in the political culture, which takes time. The improvement of the IFES Evaluation Report International Business & Technical Consultants, Inc. Page iv administration of the CEC and the voters list are necessary elements in this change, but this process will take more than three years. 5. The IFES strategy of focusing on youth and the educational system as sectors for change is correct. Given the general dissatisfaction and lack of interest in politics, and the belief that anything can change for the better, working with education and youth to raise civic awareness is an effective way to inspire long-term, political cultural change. Recommendations 1. The CEC needs to develop its own standing training department. IFES could help persuade the CEC to establish such a unit, and help the CEC facilitate it. 2. The Workflow Analysis done for the CEC is a valuable management tool to increase the effectiveness and efficiency of the CEC. IFES should continue to promote its adoption by the CEC. 3. The valuable research work that IFES did during the electoral code legislative process, points to the possibility of helping to establish a Research Service in the National Assembly. If such a service is needed, IFES could provide the necessary expertise to help establish such a service. 4. IFES needs to establish a more rigorous monitoring and evaluation program, beginning with clearly defined baselines, objectives and indicators of progress. 5. The CEC has shown an interest in adopting electronic voting. IFES should rigorously explore all aspects of this innovation, as well as the adaptation of other electronic technology such as electronic testing for TEC and PECs candidates, and SMS reporting. 6. In the coming election cycles, IFES should try to increase voter awareness and voter education, particularly among youth, by developing programs focusing on schools. Social media targeting youth should also be exploited to the maximum, while not forgetting the continued importance of radio and local TV. In an era of the Arab Spring, the youth and social media may be one of the most effective means of involving Armenians in the political process. 7. Using the recently completed polling by NDI as a baseline, IFES should conduct another poll after the election to determine what attitudinal changes that might have taken place after the interventions of IFES. This information would provide IFES with real indicators about the impact of its activities. Presuming that IFES will still be assisting with election management, it could also plan to conduct polling after the 2013 elections and compare this information with the previous baseline data as a means to measure the impact of its electoral management support. 8. Campaign finance reform remains an important activity for any election system. It is an extremely complicated exercise which is often more political than technical. Given IFES’s technical expertise in this area, USAID and IFES should consider continuing to work with media and CSOs to improve campaign finance transparency IFES Evaluation Report International Business & Technical Consultants, Inc. Page v 9. USAID and IFES have been deeply involved with elections and the CEC for at least a decade. After the 2013 elections the strengths and weaknesses of the electoral system should be more obvious and easier to assess. Based on this assessment, an exit strategy could be considered. Such an exit strategy should be developed for the next two to five years focusing on the two main electoral institutions, the CEC and OVIR. The strategy should consist of IFES helping to train an experienced and responsible cadre of civil servants who are capable of managing all levels of elections in Armenia. This cadre includes CEC members, TEC and PEC members and poll workers and election observers. IFES should also support and help develop an effective training unit in the CEC that is able to provide training of election officials for all levels of the election process, as well as producing updated guidelines and manuals as needed for each election. The CEC should also have all of the necessary equipment that it needs to conduct its election work in the most effective and efficient manner. OVIR, for its part, should also continue to maintain the voter list in an effective, accurate and current manner, and make it fully transparent by putting it on line. The international community may offer to provide some additional funding for future elections, but USAID should be able to inform the Armenian Government that it will phase out its direct assistance to the CEC after the 2013 elections. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 1 I. Introduction This is the final evaluation of the Advancing Armenia’s Elections Management and Democratic Culture (AAEM-DC) Project implemented by the International Foundation for Election Systems (IFES).The project began in April 2009 and is scheduled to conclude in January 2012, pending a possible six month extension. The total budget for the project was $2,000,000. The purpose of the project was to strengthen the administration of electoral processes and assist the electoral administration to meet international standards for free and fair elections through its support of the Central Election Commission (CEC) and the Passport and Visa Department of the Police of the Republic of Armenia (OVIR), which is responsible for maintaining the national voter list. The purpose of the evaluation is to assess the success of the project and to evaluate its effectiveness in achieving its programmatic goals and USAID/Armenia’s strategic objectives. The original budget for the Cooperative Agreement that IFES had with USAID was $1,790,820. The project was amended in 2011, making the end date to January 31, 2012 and adding $200,000 in funding. In December 2011, an action memo was approved extending the project through July 31, 2012, but the funding was still under negotiation at the time of this evaluation. The project is also seeking a grant of some € 220,000 from the OSCE, pending the project extension. IFES has been the implementing partner of the Consortium for Election and Political Process Strengthening (CEPPS) program for this project. The National Democratic Institute (NDI) is an associated partner with IFES in the CEPPS program. Background Armenia became independent from the former Soviet Union in 1991. Since its origins, the country has never had a tradition or any experience with free elections, democratic practices or democratic governance. Consequently, it is experiencing a difficult transition from a Soviet Republic to an independent, democratic society. The country suffered a major economic decline during the first years of independence. As a result, the living conditions of the population decreased, the poverty level increased and a large emigration process was activated. Since 1994 some positive growth rates occurred, and during the period from 2002 to 2007 the GDP grew by double figures. However, the 2009 global economic crisis has created a sharp decline in growth, and the republic has been severely impacted by this faltering economy, forcing many more Armenians into the Diaspora. In addition to economic difficulties, since 1995 the results of every national election have been disputed and considered unacceptable by the opposition and international observers.1 Armenian “election culture” is embryonic, with the public knowing little about the election law or their rights. This has led to widespread abuse by an entrenched oligarchy that has used all of the classic mechanisms of fraud and abuse including vote buying, ballot stuffing, multiple voting, carousel voting, counting manipulation and physical intimidation. Moreover, such practices as workplace intimidation, official manipulation of government institutions and side stepping legal and procedural requirements have been common. Government officials have operated with 1 See OSCE/ODIHR Election Reports for 1996, 1998, 1999, 2003, and 2008. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 2 impunity in controlling the election results, even in the face of threatened international sanctions. They are often supported by the Armenian Diaspora, and military, economic and political assistance from Russia. The 2008 presidential elections proved to be a watershed in Armenian history, when mass demonstrations protesting the election results were put down harshly by the authorities resulting in 10 deaths and more than 200 injured. These events displayed the deep flaws in the body politic of Armenian society, resulting in the population becoming extremely apathetic and cynical about the electoral process and the possibility of democratic change in general. The society has been trying to deal with the residue ever since. Moreover, it has become imperative that Armenia makes progress in resolving its political difficulties though legitimate, credible, acceptable and inclusive electoral processes if it hopes to benefit from a closer relationship with Europe. The international community has given the government an ultimatum that if there is little progress towards valid elections in the 2012 election cycle, Europe will reconsider further assistance to Armenia and any future relationship with Europe. To remedy this situation, the government has pledged its support to implement legislative reform in the critical areas of electoral processes, media and civil society while trying to involve a wider segment of society in the political process. The current IFES project was intended to help the government address some of these critical issues, particularly by providing support to the CEC through bolstering the electoral processes administration and helping the commission meet international standards for free and fair elections. The project also intended to provide support to the OVIR and the national voter list. Lastly, the project intended to support and strengthen the public’s confidence and political will to change and bolster a more robust election culture. II. Development Hypothesis The development hypothesis for the project is that if the independent and sustainable capacities of Armenia’s electoral and civic registration institutions are improved, and there is a deepened commitment to international election standards and practices, then wider participation and an increased public confidence in the electoral process will be achieved. The following summarizes the approach behind USAID/Armenia’s Strategic Objective 2.1 “Improved Democratic Governance,” and more precisely, Intermediate Result 2: Targeted Governance Institutions Strengthened, USAID will assist targeted government institutions to improve their internal management systems and their ability to implement transition reforms, to strengthen their oversight and fiscal responsibilities (the latter especially relevant to local governments), and to help them increase their public accessibility and accountability.2 2 2 USAID/Armenia. “Strategy for 2004-2008.” 2004. [http://www.usaid.gov/locations/europe_eurasia/countries/am/docs/strategy_2004-2008.pdf] IFES Evaluation Report International Business & Technical Consultants, Inc. Page 3 III. Evaluation Scope of Work and Methodology As noted above, this is the final evaluation of the AAEM-DC project. Launched in April 2009, pending an extension the program will run through July 2012, which is an additional six-month period, This extension period includes the May 2012 parliamentary elections, major training and publication activities, as well as voter education and awareness activities. The findings, conclusions and recommendations in this report do not cover these activities. The Scope of Work (SOW) (Annex I) for the project defines four tasks for the evaluation team to review vis-à-vis the program: 1. Implementation methodology, 2. Verify the results achieved, 3. Verify the relevance of the project in addressing USAID priorities, and 4. To what extent USAID can be accountable for achieving these objectives. Additionally, the SOW asks the team two sets of questions. One set includes nine general questions related to this program evaluation as well as four others being evaluated simultaneously by other IBTCI teams.3 The second set includes five specific questions related directly to the IFES program. The general questions are: 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? 2. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? 3. What were the main achievements of the program? 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? 5. How did the implementer perform in terms of project management and how effective was the project leadership? 6. What lessons learned can be provided for future USAID programming in this area? 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? Analyze effectiveness of alternative strategies for future programming. 8. Did the agreement provide clear and achievable results against which progress and impact could be measure? 9. Analyze attribution of project successes to USAID involvement? The specific IFES-related questions from are: 1. How can the project be better designed in the future to measure impact, given the political situation faced in Armenia? 3 This evaluation was completed under a contract between USAID/Armenia and International Business &Technical Consultants, Inc. The entire contract involves the evaluation of five USAID-funded programs in Armenia. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 4 2. Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code? 3. Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? 4. Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to the electoral administration reforms? 5. Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? Methodology International Business & Technical Consultants (IBTCI) was awarded the contract for this evaluation, and formed a team of two consultants, Mr. Hrachya Zakoyan from Armenia and Mr. Robert Brandstetter from Washington, DC, to do the evaluation. The level of effort for the team was 41 days and 43 respectively. A work plan (Annex II) was drafted and interviews were conducted with IFES/Washington staff (full list of informants for field interviews can be found in Annex III). The team was in daily contact by email and Skype prior to the fieldwork, and collaborated closely in the development of interview guides (Annex IV). Prior to departure, the team reviewed available documents provided by USAID/Armenia as well as other additional relevant documents collected by the evaluators. (Annex V) Fieldwork in Armenia was conducted over a 14-day period of time, from November 21 through December 6, 2011. During this time, the team interviewed approximately 45 individuals, primarily in Yerevan but also in four marzes or districts outside of the capital. Those interviewees included high government officials (the Vice President of the National Assembly, the Chairman of the Standing Committee on State and Legal Affairs and the Head of Department of the Passport and Visa Department of RA Police (OVIR), members of the Central Election Commission, USAID and US Department of State officials, European Union and OSCE officials, the leaders of several Armenian non-governmental organizations (NGO), Armenian journalists, political party representatives, and IFES field staff. As illustrated by this list, the team was able to meet a wide spectrum of stakeholders with an extensive knowledge of the electoral process in particular and the political environment in general. The schedule of interviews (Annex VI) was initially aided through IFES/Armenia’s contacts, after which the team was able to develop a wider network of informants. Most of the meetings were one-on-one, informational interviews, for which the questions in the SOW served as general guidelines. The evaluators asked additional, specific question pertaining to the informant’s particular area of knowledge, when appropriate. Limitations on Methodology IFES Evaluation Report International Business & Technical Consultants, Inc. Page 5 One of the biggest methodological constraints the evaluators faced was the timing of the evaluation itself, which was not optimal. While the project was planned to end on 31 January 2012, many of the project activities directly related to the elections in May 2012 had not yet been initiated and thus fell out of the timeframe of the evaluation. Because of scheduling problems, the team faced some constraints in meeting everyone it had intended to meet. It was unable to meet key officials such as the CEC chairman, representatives of the Presidential Secretariat and the Ministry of Education. The team was also unable to meet with representatives from several important political parties, such as the Legal State Party and Prosperous Armenia. Originally, the team anticipated observing training sessions and having group interviews/discussions, particularly with those who had participated in Voter List Advisory Committees (VLAC) and the Election Management Advisory Panels (EMAP). Questionnaires for these groups were prepared. However, after arriving in Yerevan, the team learned that no trainings were scheduled at this time and there were no VLACs or EMAP groups that were meeting. Nevertheless, the team was able to meet with a number of individuals who had participated in EMAP meetings and VLAC activities in the past. The team was also unable to attend training sessions for the new CEC members, PEC candidates, Territorial Election Commission (TEC) commissioners, and local observers because they were scheduled to take place after the evaluation period. Meeting with these people would have provided the team the opportunity to learn about their impressions and experiences with the new CEC regulations and perhaps learn what impact these changes have done to help gain the confidence of the public. It would have also provided the evaluation team with the opportunity to assess the effectiveness of the trainings. Lastly, there was an absence of a rigorous M&E system with definite baselines, SMART indicators and targets in the PMP. In order to overcome this deficit the team tried to fill the gap of information by interviewing appropriate specialists and partners of IFES. But the results were often not satisfactory. The availability of a rigorous M&E system with definite baselines and SMART indicators would have allowed the evaluation team the ability to compare baseline data with planned and achieved results (outputs and outcomes) as well as analyze the differences. While the team was able to gather data from various informants, without quantitative baselines it is difficult to know what the impacts of the activities were. IV. Responses to the Evaluation Questions presented in the Scope of Work The following sections are responses to the specific questions as presented in the evaluation Scope of Work. The general questions in the SOW deal with a broad scope of issues relevant to USAID’s overall mission and strategic objectives. The specific questions, on the other hand, relate to distinct AAEM-DC project details and activities as part of one of the four objectives. A. General Questions 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 6 The project’s expected results were twofold: “to promote self-reliant and sustainable capacities of Armenia’s electoral and civic registration institutions,” and “to advance the entrenchment of international standards that warrant a regeneration of public confidence and expanded inclusiveness.” As these are longer-term results, it is still much too early to know if the project will be successful in achieving them. The passage of the 2011 electoral code by the National Assembly and its initial implementation by the CEC was an important step in promoting a structure for self-reliant and sustainable capacities in the electoral process. Stakeholders generally accept the code as a valid basis for the electoral process. IFES via AAEM-DC project played an important role as the lead agency for the international community in the preparation of this legislation and is working closely with the CEC to implement specific administrative parts of the legislation. An improved voter list and a more efficient and effective voter registration institution is one of the major objectives of the IFES project. The current voter list is arguably the cleanest it has ever been. Voter registration is the responsibility of the Department of Passports and Visas (OVIR), a branch of the national police, which maintains the comprehensive State Population Registry. The registry is a centrally controlled, electronic database of all citizens that is updated biweekly. The voter lists for elections are generated from this database. The sustainability of the voter list is a function of the responsibility and dedication of the OVIR officials. Like the new code and CEC reforms, the quality and validity of voter lists will be tested in the next two election cycles. Project activities to assist OVIR were reduced, although the department did ask IFES to perform tasks including updating the list, producing a policies and procedure guidebook/manual and conducting training. IFES has also been able to work on voter registration issues through EMAP activities and in other venues. The sense of pessimism and distrust in the population for the electoral system and the political system in general is deep. Whether or not these interventions by IFES will help generate an increased degree of confidence and a greater desire to participate in the electoral process remains to be seen. However, caution must be exercised in predicting even modest successes in winning the confidence and trust of the population as a result of the project interventions, and certainly within the short time frame of the project. 2. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? The electoral code is part of the national laws of Armenia and, as such, will be maintained, until it is changed. Similarly, the major administrative reforms mandated by the code for the CEC will also be maintained. As this report argues, IFES played an important role in helping with the writing of the code, and had direct input into at least 11 articles of the legislation. The reforms introduced in the code will become more entrenched by the CEC officials as become more used to carrying out their duties in accordance with the regulations. Lastly, while there may be legitimate questions about the wisdom of the voter list being generated from a national population registry that is maintained by a state police department (OVIR), the system will also be sustained. The long term sustainability of any institutional reforms (such as those implemented in Armenia) depend not only on the laws, rules and regulations, but, also on the will, dedication, and sense of IFES Evaluation Report International Business & Technical Consultants, Inc. Page 7 responsibility by public servants whose duty it is to enforce them. The general public is severely disillusioned with the political culture and in particular the electoral process. Given the high level of disappointment about the general political process, much rests on the sense of commitment among civil servants to ensure the sustainability of the interventions that IFES has helped implement. At this time, there is no way to predict how sustainable the reforms will be until they are implemented and tested in the coming elections, and IFES and the international community can evaluate whether international standards have been met. 3. What were the main achievements of the program? The program has had three main achievements: (1) providing technical assistance to the Standing Committee on State and Legal Affairs and the National Assembly to help pass into law the new electoral code; (2) providing the logistical and program support for two major international, election-related conferences; and, (3) assisting the CEC to conform to the mandate of the electoral code and preparing for the May 2012 elections. IFES was the lead organization representing the international community (the OSCE, the Venice Commission and USAID) in the initiation, debate and passage into law of the electoral code. Without this input, it is questionable if the code would meet international standards. IFES acted as a parliamentary research service by providing the Standing Committee on State and Legal Affairs (the parliamentary committee responsible for writing the electoral code) and the National Assembly with technical papers on a variety of issues such as models of Central European Electoral Commissions, and a comparative report on voter registration systems. Project specialists also provided other consultation and advice on various issues as they arose. The project played an important role in organizing, managing and participating in two successful international conferences. In 2009, the CEC hosted the annual ACEEEO Conference (Association of Central and Eastern European Election Officials) in which 32 countries were represented. In 2010, IFES planned and co-sponsored an Electoral Legislation Reform Symposium that discussed and debated specific issues being discussed by the parliament as it was formulating new electoral legislation. These two conferences played a consequential role in educating and informing legislators about international electoral processes as well as electoral practices in former Soviet countries which are also in the process of democratic reform. The conferences helped gain local support for the electoral code. The third important achievement of the project has been the ongoing assistance it has provided to the CEC, especially as it prepares for the upcoming parliamentary elections. Project specialists and staff have assisted the seven new members of the CEC with experienced advice about implementing the new administrative policies and regulations mandated by the new code. The Territorial Electoral Officers (TEC) and Precinct Electoral Officers (PEC), for example, are now required to pass examinations before they are selected to serve. IFES helped develop a computerized system to test and score thousands of candidates fairly, rapidly and free of fraud. IFES has also been heavily involved in writing, editing and publishing thousands of manuals and guidebooks for poll workers, observers, and TEC and PEC candidates. This assistance has been essential to the CEC commissioners and staff as they adapt to the new rules and regulations. 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 8 The project was very relevant. Armenian political culture is at a crisis point. After the tragedy and shock of the 2008 elections, and the government’s handling of the post-election atmosphere, the population began to believe that change was impossible and that change specifically through elections was unobtainable. The IFES project was an attempt to begin the reversal of this attitude by improving the performance, function and credibility of two basic election institutions: the CEC and the voter registration system. The hypothesis was that by developing an increased awareness and commitment to international election standards and practices in these institutions, over time the population would gain more confidence and trust in the electoral system, which would serve as motivation to participate in the political process. As noted in this report, strides forward have been made in reforming these institutions. The hypothesis that public confidence in the electoral process will be increased by improving the management, capability and credibility of the two key election institutions has yet to be tested. In fact, one oversight in the development hypothesis is that, given the level of despair in the electorate, changing the electoral culture, even slightly, will necessarily take much longer than the three-year life of the project. 5. How did the implementer perform in terms of project management and how effective was the project leadership? IFES has had two Chiefs of Party (COP) during the life of the project. Government officials who have worked with both of these people have given them high marks for their willingness to work with Armenian colleagues, their technical knowledge and sensitivity to national political issues. Members of the international community that have worked with IFES also have said that the project leadership is technically competent, reliable and knowledgeable about political issues. The Deputy Head of Office of the OSCE in Yerevan, the Long Term Policy Advisor for Strengthening Democratic Structures for the European Advisory Group to the RA and the First Counselor for the Delegation of the European Union to Armenia have noted that IFES leadership was instrumental in obtaining approval of the election code. An exceptionally well-qualified and experienced Armenian staff has supported the COPs. The Senior Elections Expert has worked for IFES for 10 years and the Deputy Head of Office has been with IFES for nine years. In addition to a long institutional memory, they bring a wealth of knowledge about the Armenian election process, as well as a large portfolio of personal relations with party leaders, members of parliament, NGOs and other stakeholders. 6. What “lessons learned” can be provided for future USAID programming in this area? The most important lesson learned of the AAEM-DC project is that an election-related project should be synchronized with the election cycle. This project was strategically designed to fall between two election cycles, but in fact, many activities, such as training activities for the election staff, could only be done during an election cycle. The termination date for the original project was April 30 2012, only a month before the National Assembly elections scheduled for May. The end date, however, was revised to end earlier, to January 31, 2012, because of budget problems. Thus there was a three month hiatus between the end of the project and the May election. IFES, the CEC, and the international partners had to cope with the insecurity of not knowing if the project and its funding was going to continue or not. This disjuncture in support and funding jeopardized the timely implementation of election related project activities. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 9 Moreover, it is only over an election cycle that the impact of the project can be adequately assessed. 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? Analyze strategies of alternative activities for future programming. Overcoming the lack of political will in Armenia is a difficult and long-term activity. If there is to be an increase in public confidence and an expanded inclusiveness in the electoral process, the focus must be on the youth of the country. The youth need to become aware of the importance of being involved in how their country is being governed and the importance of this to their daily lives and their future. Programs focusing on schools, education and youth in general, would appear to be one of the most promising ways of developing a civic sense and generating a greater feeling of confidence in the political process. The civic education programs that IFES would like to promote in secondary schools is a step in this direction. An increased use of the new social media is also an important aspect of this strategy. The wide use of smart phones and other technology connected to a growing desire for social and political awareness can be channeled by social media to develop a greater understanding and confidence in the political process. 8. Did the agreement provide clear and achievable results against which progress and impact could be measured? The agreement did not provide clear and achievable results against which progress and impact can be measured. As noted in the methodology section above, the project lacked a rigorous M&E system with definite baselines and SMART indicators. The project lacked a rigorous Performance Management Plan that made it difficult for the evaluation team to assess whether targets had been reached and what impacts had been achieved. 9. Analyze attribution of project successes to USAID involvement. As noted above, one of the major successes of the program was the critical support role that IFES played in the adoption of the electoral code. An aspect of this support was the program’s involvement in the planning, logistics and participation in the two international conferences that were held in Yerevan in 2009 (ACEEEO) and in 2010 (Legislative Reform Symposium). These conferences played an important part in heightening the awareness and educating Armenian public officials about international election standards and practices, as well as comparative practices among Central and Eastern European countries. The conferences were also sources of national pride for the participants and the nation. Informants told the evaluation team that these conferences had a beneficial effect on parliamentary members as they considered the electoral legislation. USAID provided funding for both of these conferences, and in this way contributed indirectly to the shaping and passage of the new legislation. USAID has also been a key member in the international group that has been monitoring Armenia’s democratic transition. B. Project-Specific Questions 1. How can the project be better designed in the future to measure impact, given the political situation faced in Armenia? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 10 The project is intended to help build public confidence and broaden participation in the electoral process, yet there is little baseline data available to measure an increase in confidence and a broadened participation. The project could be designed to take into consideration existing baseline data and exact, measurable targets. Where there is the lack of baseline data, this could be collected. Such an approach would allow an assessment of the situation before and after the program intervention in order to measure the impact of the intervention. 2. Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code? The IFES Election Legislation Reform Symposium, which took place in November 2010, was successful in bringing about meaningful compromise and encouraging public debate in the process of amending the electoral code. The success of the Symposium in November 2010 is reinforced by the passage of the new code in May 2011, and that it is considered an acceptable piece of legislation by most stakeholders. Its success is also demonstrated by the wide range of stakeholders who were willing to come together in an open forum to have energetic discussions and debate. Participants included representatives of political parties both in and out of the parliament, MPs and parliamentary staffs, NGOs, and OSCE, Venice Commission, and EU representatives. IFES provided a number of election experts to give papers. The MPs and representatives of parties said that this forum was valuable in exposing ideas for debating the reformed code. 3. Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? The Workflow Analysis identified a new management structure and suggested other improvements that would increase the efficiency and effectiveness of the CEC. The finished plan was given to the former CEC chairman for consideration, but unfortunately he passed away before there was a chance to seriously discuss the plan with IFES. The plan apparently has been circulated among the new members of the CEC, but the evaluation team was told that the analysis did not have anything useful for them. 4. Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to the electoral administration reforms? IFES has successfully leveraged US material assistance related to CEC equipment requests with other donors. According to the CEC, the OSCE has promised to help provide new hardware and software to the commission. This includes 41 new computers for the TECs to do their own work and to communicate with headquarters. The commission was also hopeful that the government would provide funds for new servers. IFES was already helping the CEC implement administrative reforms by assisting with the development of a computerized system for testing TEC and PEC applications for committee membership. However, this activity was being done without donor leveraged funding. At the IFES Evaluation Report International Business & Technical Consultants, Inc. Page 11 time of the evaluation, funds from the OSCE had not yet been made available. When the OSCE funds become available, IFES will be able to expand its assistance in applying electronic technology to other aspects of administrative reform. 5. Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? While IFES has a cordial relationship with OVIR, there is not a working relationship between OVIR and any VLACs. In the past, the VLACs have been most active prior to elections. Since there have not been any national elections since 2008, the VLACs have not been active. In any case, the head of OVIR claims that he already has a nearly flawless database which is updated regularly. The voter list is drawn from this list. He has said that past VLACs did little to improve his database, and he did not find them useful. V. Findings The AAEM-DC project has four principal objectives, each with corresponding activities designed to achieve these objectives. The findings are presented by activity under each objective. The conclusions and recommendations that follow are based on these findings. The four program objectives are: 1. To improve the voter registration system and insulate it from abuse 2. To assist in election administration reform 3. To strengthen the political will to change the election culture, and 4. To assist in campaign finance reform through a money and politics program. There were a number of events and unforeseen circumstances which occurred during the project that affected the implementation and impact of the objectives. These included: 1. Two incorrect assumptions were made which drove IFES’s program and impacted its work: a. The first assumption was that the project would receive the full cooperation of the Government of Armenia (GOA) and other government entities. However, the lack of full cooperation of the Head of OVIR reduced the involvement of IFES in the civil registration process. b. The second assumption was that stakeholders would maintain the political will to see the process through. Since the reform process had only just begun, it was too early to rely on the political will of stakeholders because it was not known how many of the stakeholders would support the reforms and whether increases in public confidence and participation in the electoral process would occur. 2. The passage of the new electoral code took much longer to work its way through the National Assembly than anticipated and became law in June 2011, late in the project. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 12 The code made substantial changes in the electoral process that IFES has been to responding to. 3. In September 2011, the Chairman of the CEC died suddenly. He had served as Chairman for nearly eight years, and had recently been reappointed to chair the new CEC. His long tenure gave him broad electoral experience and understanding, as well as a keen sense of the Armenian political environment. While the chairman was not always cooperative with the advice and suggestions provided to him by project representatives, IFES had created a functional, working relationship with him. His death was a great loss to the IFES program and the CEC. This loss made it more difficult to proceed with several of the technical assistance activities IFES was providing the CEC including the Workflow Analysis and recommendations. While the previous chairman would likely have guided it through the CEC, the current chairman has not yet paid the issue any attention, so no action has been taken. 4. The new chairman of the CEC was previously an administrative judge, who had served as a CEC member almost two decades earlier. He has much to learn and IFES will have to gain his trust and confidence in order to have a maximum impact. IFES has only just begun to work with him, especially as the CEC gears up for the parliamentary and presidential elections. Objective 1: To improve the voter registration system and isolate it from abuse. The Program Description listed five project activities under this objective: 1. Establish Voter List Advisory Committees (VLAC) in all 10 marzes and Yerevan. These committees consist of local NGOs and other civil society organizations (CSO) supported by IFES with the aim of working with OVIR officials and police to verify the accuracy of the voter list and clean it up as necessary.4 Another function of VLACs was to conduct voter education, voter awareness and other outreach activities. FINDING: This activity had not yet been implemented at the time of the evaluation because there had been no national elections. VLACs were presented as an activity of the project Work Plan for year 2. The work plan noted that “It is expected that the activities associated with the VLAC project will begin in the second half of 2011.” IFES informed the evaluation team that there had not been any implementation of this activity prior to the team’s arrival. The quarterly report for July 1, 2011-September 30, 2011, confirmed this. Moreover, there was nothing planned for the first quarter of FY2012 (October-December, 2011). Nevertheless, expected result (2) A of the year 2 Work Plan states that “VLAC activities [would be] scheduled in the months prior to the 2012 parliamentary vote.” Thus, there was an opportunity to establish and implement work with the VLACs for only 4 months maximum. 2. Create a better voter registration management system. IFES proposed to conduct a Workflow Analysis of the voter registration process with OVIR. FINDING: This activity never took place because of a lack of interest on the part of OVIR. 4 The voter list for the entire country is generated from a master list, the State Registry of Population, of all Armenian citizens. This list is maintained by the Passport and Visa Department of the Armenian Police, known in Armenian as OVIR. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 13 3. Draft policy and procedures manuals for OVIR voter registration activities. FINDING: This activity was to be initiated after the new election code was adopted in June 2011. The head of OVIR indicated to the evaluation team that his department did not need such assistance. 4. Create a sustainable training program for OVIR officers involved in voter registration activities. FINDING: The head of OVIR indicated that his department was not interested in such a training program. 5. Provide research papers and advice on voter registry legal reform to the parliamentary working group for the electoral code. FINDING: IFES delivered multiple analyses of the voter registry legal reform to key stakeholders in the National Assembly, and distributed its recommendations to others outside parliament. These recommendations informed relevant articles in the new electoral code. ANALYSIS OF OBJECTIVE 1 The major outcome of this objective is an improved voter list and better functioning from the administration of OVIR to maintain and sustain the list. This has been achieved, however, with little input from IFES because OVIR already has improved systems in place. Prior to 2005, the voter list was maintained by local officials, and contained massive errors. The OVIR head mentioned that as many as 30,000 people regularly applied to the court to recover their voting rights. When the AAEM-DC project was designed, this objective was clearly aimed at helping to solve this great flaw in the registration system. In addition to providing management assistance to the local registration officials, IFES also helped implement a system of Voter List Advisory Committees (VLAC) which was also intended to help correct this problem by going from house to house to verify names and residences. After the 2008 elections, the responsibility for maintaining the voter list was moved into OVIR which also maintains the computerized State Registry of Population. The voter list is now generated from the State Registry and all eligible voters are automatically on the list. OVIR has refined its database during recent years and the list is probably the ‘cleanest’ it has ever been. OVIR updates the State Registry every 15 days, receiving data on births, deaths, prisoners, and people who have received acceptance or been refused migrant status from the Ministry of Justice, the Ministry of Defense and the State Migration Service of the Ministry of Territorial Administration. At the same time, the data needed to update and maintain the voter list is also being amended. Additionally, the voters list is now available online and is open for public scrutiny. The chairman of the CEC has said that the list is “ideal” and the head of OVIR claims that it is nearly 100% accurate. He noted that during the 2009 Yerevan local election, there were 2,545 complaints regarding the list, but after verification, none of them were valid. A former member of the CEC and representative of the opposition Dashnaktsutyn Party has also asserted that the voter list was almost perfect, and that international observers were coming to Armenia to learn how this was done. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 14 The major problem that still exists with the voter list is what to do about the 450,000 to 800,000 out-of-country registrants on the list. This is a potential source of fraud, but it is a more of a highly charged political issue than a major technical problem. If a political decision were made about how to treat the out-of-country registrants, a computer program could be easily written to include or exclude voters. The Parliamentary Working Group which helped write the electoral code considered the public posting of the voter list with signatures after the counting, to verify the number of actual voters. This solution, however, is not accepted international practice, and was not approved by the working group. OVIR requested assistance in improving the process of updating the voter list and IFES will produce a guidebook/manual of policies and procedures and provide appropriate training. However, OVIR has not been particularly amenable to other assistance from IFES. IFES has reduced its intended assistance to OVIR as a result. The head of OVIR has been refining the State Registry of Population for six years and he is proud of the quality of the database. While not accepting all of IFES’s technical assistance, he did indicate that IFES could provide his system with improved hardware and software for 60 registration centers. As noted, VLACs were going to be supported in the collection of OVIR registration data. This activity was suspended because the head of the department did not think that the voter list could be improved by any VLAC activities. Despite the difficulties trying to assist OVIR, IFES continued to work at lower levels to keep the discussion of the list relevant. Voter registration issues were topics of discussion for several EMAPs, and conclusions and findings of these panels were passed on to the head of OVIR. IFES also wrote a report for general circulation on Armenian voters living abroad and voter registration issues. Objective 2: Election Administration Reform Prior to 2010, the fundamentals of election administration were essentially in place. The basics of the legal and procedural structure were capable of administering an orderly election, as long as the political will and abuses are held in check. However, as a consequence of the events of 2008, Armenia began to consider changing and updating its electoral code to be in closer accordance with European standard election law. IFES was integrally involved in the development of a new election code, as well as helping to implement the regulations of this code, particularly as they apply to the CEC and all of its administrative activities. Ten major activities were implemented in support of this objective: 1. IFES convened Election Management Advisory Panels (EMAP). FINDINGS: In early 2009, to attempt to diffuse the public anger that followed the tragic events of the 2008 elections, IFES proposed to establish ‘town hall’ type meetings to help deal with some of the discontent. Fifteen panels were held over the three years for a total of more than 200 participants.5 The agendas for the panels were set by IFES and included such topics as: civic education, public affairs journalism, the voter registration list, the use of new media, and several 5 USAID/Armenia. Case Study: Discussing and Debating Electoral Legislation. USAID/Washington, “Telling Our Story”. (http://stories.usaid.gov) IFES Evaluation Report International Business & Technical Consultants, Inc. Page 15 topics related to the electoral code. A broad range of stakeholders was invited to participate, including representatives of political parties, CEC members, media representatives, and a variety of NGOs and CSOs. The opinions, questions and problems raised by local stakeholders on these various issues were gathered by IFES and channeled to the parliamentary standing committee responsible for writing the electoral code. This information helped inform the committee of the concerns of these stakeholders. 2. In September, 2009, the CEC of Armenia hosted the annual Conference of the Association of European Election Officials (ACEEEO). This was a major international event for Armenia and the country took great pride in hosting it. This international recognition was especially important, coming only one year after the 2008 turmoil. The main theme of the conference was: “Judicial Protection of Electoral Rights”. IFES assisted the CEC with the logistics of the conference, helped provide the translation facilities, provided support for the programming and participated in the panels and sessions of the conference. FINDING: Approximately 150 participants attended the conference from 32 countries and several international organizations. It provided a rich venue for exchange between Armenian election officials in the CEC and other government representatives with election colleagues from Western and Eastern Europe. Armenian participants told the evaluation team that the conference was useful in helping them understand and compare similarities and differences among international election systems. IFES, with USAID funding, was instrumental in arranging translation facilities and the logistics of the conference, as well as participating in the conference activities. The conference also afforded the project the opportunity to exhibit before international participants the kind of USAID-funded assistance that has been critical to the work of the CEC in Armenia. 3. IFES planned and co-sponsored the two-day Electoral Legislation Reform Symposium that took place in November 2010. The conference received financial assistance from USAID, the US Embassy, OSCE, the EU and the Council of Europe, and logistical support from the National Assembly and the CEC. The subject of the symposium was the electoral code legislation currently being considered by lawmakers in the National Assembly. FINDING: More than 150 representatives of Armenian civil society, public affairs groups and political parties attended, along with international election experts from the OSCE’s Office for Democratic Institutions and Human Rights (ODIHR), the Council of Europe’s Venice Commission and IFES. The international experts provided a regional and international context for the symposium. There were four primary foci of the symposium: election administration, campaign finance, election dispute and resolution, and electoral fraud prevention. The symposium provided an important venue for the discussion and debate of these and other issues that were key questions and problems before the National Assembly. The open discussion of these points with a broad range of stakeholders helped the legislation receive wider acceptance by the public, subsequently contributing to IFES’s overall efforts in helping to guide the electoral code. 4. IFES provided technical assistance to the development and ultimate passage of the electoral code. After widely held and thorough public discussions of the draft electoral IFES Evaluation Report International Business & Technical Consultants, Inc. Page 16 code, including the two public and one closed parliamentary readings for which IFES provided technical assistance and recommendations, the final version was approved by the parliament on May 26, and signed into law by the President on 14 June 2011. FINDING: As noted, the code has met with broad acceptance by most Armenian stakeholders and the international community. IFES staff provided technical advice throughout much of the legislative process, acting as a parliamentary research service, and writing papers on aspects of the code as requested by the Standing Committee for State and Legal Affairs and a Parliamentary Working Group. IFES contributed three substantial reports at the request of the Chairman of the Standing Committee including a report on models of Central European Electoral Commissions; a comparative report on voter registration systems; and, a technical report for the National Assembly Majoritarian Constituency By-Election. In February 2011, IFES also researched a paper for the ruling coalition in parliament on four key issues in the code: voter registration; membership and leadership of election commissions; the complaint and adjudication process; and the training and regulation of local observers and monitors. Lastly, IFES submitted a paper to identify and analyze four priority issues in the draft code: election commission membership and leadership; Armenian voters living abroad and voter registration; election dispute resolution; and the training and registration of local observers and monitors. This paper was widely circulated among the political party leadership, NGOs, think tanks, the legal community and legislators. The knowledge of these issues and possible options, helped encourage public debate and discussion, which were important for the legislation to receive broad acceptance from stakeholders. 5. The management structure of the CEC required significant changes in order to conform to the new code. One of the major contributions to improve the electoral administration of the CEC was the attempt by IFES to conduct a critical path study or Workflow Analysis of the required management tasks of the CEC during an election cycle.6 The Commission has also requested new equipment in order to fulfill its new responsibilities. FINDINGS: This Workflow Analysis produced a number of recommendations to improve the management, efficiency and credibility of the CEC. They include the following: a. The Workflow Analysis noted that the CEC has a strict legal definition of its work, which is done according to a legal timeline. It was recommended that this static approach be changed to one involving strategic planning, with a mission statement, goals and objectives, scopes of work, job descriptions and other qualities of a contemporary management administration. b. Another recommendation was to change the current organizational structure from one in which there is a lack of specialization and lines of responsibility, a lack of accountability for the quality of service, and limited incentive to foster long-term relationships with political stakeholders to a management structure that divides 6 IFES. “Workflow Analysis Report Central Election Commission of Armenia.” IFES, Yerevan, 2011. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 17 the organization into functional departments. Such departments would delineate responsibility, so that: i. Clear lines of responsibility and accountability are created; ii. A system of written institutional memory is developed; iii. Key personnel can be designated for specialized training to improve overall capacity of the organization; and, iv. Each department can promote quality in delivery of services. c. The analysis recommends that a stand-alone training unit be established in the CEC, which is responsible for quality training of all election officials. d. Other recommendations included modifying requirements of election officials to participate in training before they can be nominated to their posts; tailor training for officials at different levels who have different responsibilities; establish clear procedures for registration, accreditation and training of observers; and requiring photo IDs for observers. These recommendations were given to the former CEC chairman for his review and consideration. As noted elsewhere, the chairman died before any of the advice or recommendations could be acted on or implemented. The evaluation team was told that the new CEC members are aware of the report, but they not think that the recommendations are useful to them. The Commission requested additional assistance from IFES for upgraded hardware and software in order to facilitate the Commission’s computer capacity. IFES is hoping to leverage this assistance from the EU and OSCE. The CEC also requested that USAID/IFES help to provide a new server, a large, flat screen TV and a new media room for E-day results. The CEC argues that this equipment would help improve the management capacity of the Commission, particularly on Election Day. 6. IFES has assisted the CEC with organizing an examination process for Territorial Election Commissioner (TEC) candidates, Precinct Election Commissioner (PEC) candidates and election observers that includes developing and testing a computer￾generated system that will rapidly and accurately allow all candidates to be tested and scored. FINDING: This process of electronic testing is being adopted by the CEC. Given the new requirement in the electoral coded that all new members of TECs and PECs must pass an examination before they are selected, the introduction of an electronic testing system will be an important addition to the CEC management capacity. The alternative to electronic testing is to test 40,000 candidates by using paper and pen: a long and cumbersome job. Electronic testing is also not as susceptible to fraud. It is hoped that this system will be approved and operational before the 2012 elections. 7. IFES has also played a key role in assisting the CEC in preparation for the May 2012 elections by planning and managing the production and distribution of materials in accordance with the regulations of new code. IFES is also training the national election staff. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 18 FINDING: Beginning in early 2012, IFES will undertake the training of the newly selected TEC and PEC commissioners, and election observers, as well as the preparation, printing and distribution of the more than 14,000 new manuals and guidelines for policies and procedures for the commissioners, in accordance with the new electoral code. IFES has done this work for past elections, and has received praise from the CEC as well as TEC and PEC members who have used the manuals and guidelines. 8. In May 2009, Yerevan City had municipal elections. At the CEC’s request, IFES took responsibility for training PEC staff members. It also developed and distributed manuals and guidelines for these poll workers. These elections, coming closely after the 2008 events and taking place in the capital city, were seen as an indicator of possible change in the electoral culture. FINDING: IFES trained 1,685 PEC staff members from 439 polling stations, and distributed over 4,000 manuals and guidelines for poll workers. Unfortunately, observers of the Yerevan elections noted that there was little change in the political and electoral culture since the 2008 elections. An IFES observation report commented that while the poll workers/commissioners were aware of the rules, regulations and procedures, many willfully violated the law.7 In fact, the election culture had not changed and was rift with fraud, voter intimidation and control by the same powerful people who control national elections. An independent think tank analysis also concluded that little positive change had occurred in this election.8 9. IFES, with the support of the CEC developed software and technology options for the use of SMS preliminary election reporting for PECs. The project piloted a demonstration of the use of this technology during the Yerevan elections. FINDING: Despite a successful pilot demonstration, IFES was not able to convince the former CEC chairman and higher government officials to approve the use of SMS technology. The system would have expedited reporting results and permitted greater transparency in the reporting process, an acknowledged weakness in former elections. The potential for SMS use in future national elections is undecided. 10. IFES organized trips to Poland and Washington, DC for Armenians to study political financing and to observe elections. In November 2009, IFES facilitated a study visit to Warsaw for representatives of the government, the opposition, and local NGOs to learn about the Polish experience in reforming political finance regulations. Recommendations from this group were used by IFES in developing political finance regulations in the new code. In November 2010, the CEC chairman and the CEC press secretary made a study trip to Washington, DC under the auspices of IFES, and financed by the OSCE and USAID, to observe voting in the U.S. and talk with local officials about election activities. FINDING: The participants of these study trips returned to Armenia with an increased awareness and knowledge of how campaign financing is being dealt with in a former Soviet area, 7 IFES. “Technical Observation Report on Yerevan Council Elections, May 30, 2009.” (Manuscript) 8 Policy Forum Armenia. Armenia’s 2008 Presidential Election: Select Issues and Analysis. July 2008. http://www.pf￾armenia.org/fileadmin/pfa_ uploads/PFA_ Election_Report--FINAL.pdf] IFES Evaluation Report International Business & Technical Consultants, Inc. Page 19 and how these methods could be integrated into the Armenian context. The visitors the U.S. were able to observe voting in the U.S., talk with local officials about election activities, and meet people at the IFES headquarters in Washington. ANALYSIS OF OBJECTIVE 2 The electoral code was approved by the Parliament on 26 May, 2011, and adopted into law by the President in June 2011. This approval was a major focus of the activities of this objective, and, as noted, was the highlight of the AAEM-DC project. This code is the legal basis and sets the parameters for all future local and national elections in the Republic. IFES, along with the OSCE and the EU Venice Commission, has been integrally involved in working with the parliament to husband this code through the legislative process into law. While the code is considered to be well crafted and widely accepted, it was reiterated to the team several times that a satisfactory law does not ensure the outcome of a legitimate, credible, acceptable and inclusive election. It is the political will of government officials to administer the code appropriately and the public to accept and respect the election code and regulations that contributes to whether or not the election is considered valid. Through the series of EMAPs and the two major international symposiums, IFES incrementally helped to build a broad consensus for the election reform code. The EMAPs that IFES conducted through the life of the project brought together a wide array of stakeholders and provided them with a platform to air their concerns about a variety of electoral related issues. It also provided IFES with pertinent questions and answers for use in discussions with decisions makers. While IFES's role in the ACEEEO conference in 2009 assisted with the facilitation of the meeting, it also participated in substantive discussions with participants. Finally, by helping to host the Electoral Legislation Reform Symposium in November 2010 along with the National Assembly and international donors, IFES was able to convene stakeholders across the political and social spectrum in a major conference focused entirely on electoral reform. Both Armenian and international informants said that this conference was an important influence on members of parliament who were responsible for approving the legislative reform. The finished document contained eleven articles or sections of articles that had been proposed or recommended by IFES.9 In contributing to this legislative process, IFES functioned as the ‘floor leader’ in the National Assembly for the OSCE, the EU, the Council of Europe, the US State Department, and USAID, all of whom have made significant commitments to electoral reform and internationally acceptable elections. Additionally, IFES has acted as a parliamentary research service, producing a number of research documents for the National Assembly. Informants from OSCE, the EU and the National Assembly have acknowledged IFES technical expertise, and affirmed that the role that IFES played was important for the success of the code. 9 IFES. “Comparison of the New Electoral Code—First Reading and Second and Final Readings. June 2, 2011”. (Manuscript) IFES Evaluation Report International Business & Technical Consultants, Inc. Page 20 Indeed, the international partners have expressed enough confidence in the technical election expertise of IFES, that the EU has given IFES, via OSCE, some €220.000 for its work with the CEC during the parliamentary and presidential elections. The receipt of this money is forthcoming. IFES made a key contribution to the reform and modernization of the management structure and function of the CEC with the Workflow Analysis study. As mentioned above, this study made numerous recommendations to improve the efficiency and effectiveness of the CEC. The report was being studied by the former chairman before he died. Unfortunately, no decisions were made to accept or to implement any of the findings. The evaluation team learned that the new commissioners are aware of the study, but they did not think that at this time any of the recommendations would be applicable. Given that the commission is now working tirelessly to prepare for the May elections, it seems unlikely that it will have time to reconsider the recommendations, let alone implement any of them. Ideally, the analysis will be revived for future consideration. The writing, production and distribution of manuals and guidebooks for all levels of the electoral structure – PECs, TECs, CEC, observers, and police – was one of the most frequently mentioned contributions IFES made to the electoral process. These manuals are saved by PEC and TEC members almost like awards. One TEC chairman showed the evaluation team a series of manuals and guidebooks produced by IFES going back to 1996. Equally respected by many poll workers was the training that IFES provided along with the manuals and guidebooks. Several people commented about how useful this training was to them on Election Day. Lastly, it must be noted that the success of the overall objective of electoral reform was highly dependent on the staff that IFES has been able to recruit and retain. While there was one change in the Chief of Party during the project, the Armenian senior staff has been there for the duration. The Senior Elections Expert has worked for IFES for 10 years, while the Deputy Head of Office has been with the organization for nine years. The experience, technical knowledge, language capability, and personal relationships that this highly competent, professional staff brings to IFES are immeasurable. The electoral reform legislation as well as other aspects of the IFES program owes much of its success to this local staff. Objective 3: Strengthening Political Will to Change the Election Culture This objective was intended to help overcome the massive negativity with the political system and the electoral process that captured the country after the 2008 election. According to the “Program Year Three Timeline (April 2011—December 31, 2011)”, there were six activities envisioned under this objective: Politician-Citizen Discussion Groups; Development of Student’s Guides; Voter Outreach and Motivation for Young Persons using New Media; Development of Civic Education Modules; Student Action Committees; and Teacher’s Guide for Civics Textbook. IFES implemented the first three. The remaining three activities that were part of this objective--Development of Civic Education Modules, Student Action Committees, and Teacher's Guide for Civics Textbook--were suspended, by mutual agreement between USAID and IFES. 1. Initially, IFES proposed to conduct a series of Politician-Citizen Discussion Groups over the life of the project, with the intention of generating civilized debate and providing transparency to the political system and the electoral process. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 21 FINDING: The discussion groups were never convened. They were replaced by the November 2010 Electoral Legislative Reform Symposium, described above, and became the major activity of this objective. The EMAPs also served as substitutes for these discussion groups. 2. Development of Student Guides and a comparative study of textbooks used to teach civic education in Armenian schools. FINDING: The project published a booklet designed for use in secondary schools called Young Person’s Guide to National Assembly. 10 IFES published 3,000 copies of this booklet in Armenian and 1,500 copies in English. The booklet has been given to young visitors to the National Assembly. In 2010 IFES produced a comparative study of textbooks. The study notes deficits in the social studies/civics curricula. The Ministry of Education was reluctant to give up control of the content of the suggested new material. After consideration by IFES and USAID, this project activity was suspended. 3a. Prior to the Yerevan mayoral elections, IFES worked with local partners to develop an election information website and blog for voters, as part of the voter outreach component of this objective. FINDING: The website provided information on the schedule of elections, voter registration information, information on candidates and parties, an open forum for discussions, the addresses of PECs in Yerevan, a listing of NGOs observing the election, and candidate party reports on campaign finance. The site received about 930 hits, with the voter registration information section and the section explaining the importance of the election receiving the most hits. 3b. Also as part of voter outreach, IFES conducted a voter education and voter awareness campaign connected to the municipal elections. FINDING: IFES printed and distributed 20,000 voter information stickers. These were designed to inform and motivate voters to participate in the elections. IFES also helped distribute thousands of other leaflets and newspaper supplements to raise public awareness about the elections. The project ran TV spots to raise voter awareness. As there were no independent studies made, it cannot be determined how effective this material was in motivating people to vote. ANALYSIS OF OBJECTIVE 3 As nearly all observers have noted, Armenian political culture is deeply damaged and people are very skeptical about any improvement of the situation. The principal focus of this objective was to begin working in secondary schools with young people to help change their attitudes and expectations about electoral politics. This is perhaps one of the best approaches to trying to change the political culture, but certainly a long-term activity. Rewriting secondary school textbooks and distributing them in secondary schools has a longer-term prospective. Developing voter information websites to serve the Yerevan elections had a considerably shorter prospective, as did the voter education and voter awareness campaigns during the mayoral elections. It is 10IFES/USAID. Young Person’s Guide to National Assembly. Yerevan: IFES/USAID, 2011. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 22 difficult to know what the impact of these activities or how they contributed to the strengthening of political will to change the election culture. Objective 4: Campaign Finance Reform through a Money and Politics Program This objective was intended to take on the thorny and complex issue of campaign finance. Three activities comprise this objective: 1. Campaign Finance Legal Reform 2. Campaign Finance Awareness Campaign 3. Campaign Finance Media Training FINDING: IFES was able to influence four articles in the new code concerning campaign financing: businesses are not allowed to make campaign contributions; there is an increased frequency of interim reports about campaign contributions; there are penalties for inaccurate reporting, and, finally, there is regulation of per minute rate for campaign airtime.11 An Audit and Oversight Service (AOS) existed in the CEC under the previous Election Code but its accounting practices never met the necessary rigors of an election campaign audit. The new code, with input from IFES, made substantial improvements in this service. The AOS is now staffed by a Head and two assistants who are permanent civil servants, and are appointed by the CEC by consensus. Additionally, each parliamentary party appoints one auditor to the AOS, providing more transparency to the system. The new service has also been charged with developing forms and instructions for reporting by candidates, thus providing uniform reporting and improved, accurate public reporting While the newly constituted AOS has been operating since August 2011, the evaluation team was unable to meet with any members during the evaluation. As a part of campaign finance legal reform, IFES was also instrumental in arranging for an Armenian study trip to Warsaw to review and analyze recent Polish efforts to reform campaign finance. Members of the government and members of the opposition, NGO representatives and IFES staff members were participants on this trip. Other activities planned for this objective were removed from the IFES work plan at the request of USAID in the fall of 2011. ANALYSIS OF OBJECTIVE 4: IFES has helped make an important contribution to campaign financing with its input into shaping finance-related articles in the new electoral code. As in most countries, campaign finance is a very contentious issue, and the successful implementation of these reforms, like all other aspects of this project, will depend heavily on the political will of the government, political parties and the electorate to follow the letter and the spirit of the laws. Campaign finance awareness and campaign finance media workshops were deleted from the IFES work plan at the request of USAID. 11 IFES. Comparison of the New Electoral Code-First Reading and Second and Final Readings. June 2, 2011. (Manuscript) IFES Evaluation Report International Business & Technical Consultants, Inc. Page 23 VI. Gender Considerations A recent USAID assessment on gender in Armenia 12 reported that women are well educated and participate freely in politics as voters, but they have very low representation in policy-making roles and policy development, which likely has an impact on policy. The new election code mandates that at least two of the seven members of the CEC must be women. Also, there must be at least two women in each Territorial Election Commission. Since the CEC and TECs are the highest levels of the election administration pyramid, this is a notable improvement in the administrative gender balance. This electoral reform occurred with the support of IFES and the international community as part of its general participation in the legislative reform process. With a number of local NGOs, the Venice Commission/ODHIR and IFES advocated improving the election code by ensuring that women would receive at least 20% representation on the proportional electoral lists of the National Assembly candidates. This is up from a previous 15% representation. The adopted code states that beginning from the second number on the electoral lists, in each subsequent group of five candidates, the same gender candidates should not exceed 80%. Women comprised 43% of the participants in the EMAPs conducted during the life of the project. Out of a total of 199 participants, 86 were women. At another level, IFES has supported and cooperated on this project with a number of NGOs led by women, such as the Armenian Young Women’s Association and the Armavir Development Center. These organizations have participated in VLAC and EMAP activities. VII. Conclusions 1. The absence of a monitoring and evaluation (M&E) system with baselines and SMART (Specific, Measurable, Achievable, Realistic, Time-bound) indicators and specified targets as well as a lack of collected program data made conducting an external evaluation more difficult. Particularly, based on a review of the IFES performance monitoring plan indicator table, it is clear that a majority (around 70%) of the data to be collected (targets and actual measurement numbers) are not available (suspended activities excluded). Surveys are mentioned as a source of information for some indicators. For example, a survey was envisioned to measure the percent increase of citizens who have confidence in the work of election officials, but the evaluation team did not receive any survey details or reports. 2. The current voter list is generated from the State Registry of Population, a national database of all Armenian citizens, and maintained and regularly updated by OVIR. According to most informants, the voter list is a clean and updated list. 12Elisabeth Duban and Hasmik Gevorgyan. Gender Assessment: USAID/Armenia. 2010. [http://www.usaid.gov/our_work/crosscutting_programs/wid/pubs/Armenia_Gender_Assessment_2010] IFES Evaluation Report International Business & Technical Consultants, Inc. Page 24 One of IFES’s major assumptions in implementing the project was that it receives the full cooperation of the Government of Armenia and other government entities such as OVIR. While IFES has had cordial relations with the head of OVIR, the head of OVIR was less welcoming of IFES offers of technical assistance. This resistance to assistance was an important reason why IFES decreased its planned activities for the related objective. Nevertheless, some assistance was requested to improve the voter registration process and for a policies and procedures manual for OVIR officers. The Voter List Advisory Committees (VLACs) were originally a component of the assistance that IFES intended to provide OVIR to help verify the accuracy of the voter list. However, OVIR noted that the VLACs were largely superfluous to the maintenance of the voter list, and they have not been established. 3. The acceptance and passage of the new electoral code can be attributed to the leading role IFES played in the international community on this topic. It is also where IFES made the greatest impact vis-à-vis the AAEM-DC project. IFES filled this role because the major donors, OSCE and the EU, did not have the capacity to have representatives in Armenia with the technical skills needed to work regularly with the National Assembly. IFES’s ongoing assistance to the CEC to implement the administrative reforms of the code reinforces and contributes greatly to the sustainability of the reforms of the new code. Anticipated new computer hardware and software facilitated by IFES will also reinforce these reforms. 4. The three year project life was too short a timeframe to have an impact on revitalizing public confidence and broadening participation in the electoral process as it requires a significant change in the political culture, which takes time. The improvement of the administration of the CEC and the voters list are necessary elements in this change, but this process will take more than three years. 5. The IFES strategy of focusing on youth and the educational system as sectors for change is correct. Given the general dissatisfaction and lack of interest in politics, and the belief that anything can change for the better, working with education and youth to raise civic awareness is an effective way to inspire long-term, political cultural change. VIII. Recommendations 1. The CEC needs to develop its own standing training department. IFES could help persuade the CEC to establish such a unit, and help the CEC facilitate it. 2. The Workflow Analysis done for the CEC is a valuable management tool to increase the effectiveness and efficiency of the CEC. IFES should continue to promote its adoption by the CEC. 3. The valuable research work that IFES did during the electoral code legislative process, points to the possibility of helping to establish a Research Service in the National Assembly. If such a service is needed, IFES could provide the necessary expertise to help establish such a service. 4. IFES needs to establish a more rigorous monitoring and evaluation program, beginning IFES Evaluation Report International Business & Technical Consultants, Inc. Page 25 with clearly defined baselines, objectives and indicators of progress. 5. The CEC has shown an interest in adopting electronic voting. IFES should rigorously explore all aspects of this innovation, as well as the adaptation of other electronic technology such as electronic testing for TEC and PECs candidates, and SMS reporting. [IFES was examining these new possibilities just prior to the departure of the evaluation team.] 6. In the coming election cycles, IFES should try to increase voter awareness and voter education, particularly among youth, by developing programs focusing on schools. Social media targeting youth should also be exploited to the maximum, while not forgetting the continued importance of radio and local TV. In an era of the Arab Spring, the youth and social media may be one of the most effective means of involving Armenians in the political process. 7. Using the recently completed polling by NDI as a baseline, IFES should conduct another poll after the election to determine what attitudinal changes that might have taken place after the interventions of IFES. This information would provide IFES with real indicators about the impact of its activities. Presuming that IFES will still be assisting with election management, it could also plan to conduct polling after the 2013 elections and compare this information with the previous baseline data as a means to measure the impact of its electoral management support. 8. Campaign finance reform remains an important activity for any election system. It is an extremely complicated exercise which is often more political than technical. Given IFES’s technical expertise in this area, USAID and IFES should consider continuing to work with media and CSOs to improve campaign finance transparency 9. USAID and IFES have been deeply involved with elections and the CEC for at least a decade. After the 2013 elections the strengths and weaknesses of the electoral system should be more obvious and easier to assess. Based on this assessment, an exit strategy could be considered. Such an exit strategy should be developed for the next two to five years focusing on the two main electoral institutions, the CEC and OVIR. The strategy should consist of IFES helping to train an experienced and responsible cadre of civil servants who are capable of managing all levels of elections in Armenia. This cadre includes CEC members, TEC and PEC members and poll workers and election observers. IFES should also support and help develop an effective training unit in the CEC that is able to provide training of election officials for all levels of the election process, as well as producing updated guidelines and manuals as needed for each election. The CEC should also have all of the necessary equipment that it needs to conduct its election work in the most effective and efficient manner. OVIR, for its part, should also continue to maintain the voter list in an effective, accurate and current manner, and make it fully transparent by putting it on line. The international community may offer to provide some additional funding for future elections, but USAID should be able to inform the Armenian Government that it will phase out its direct assistance to the CEC after the 2013 elections. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 26 IFES Evaluation Report International Business & Technical Consultants, Inc. Page 27 ANNEX I: EVALUATION SCOPE OF WORK Evaluation of USAID/Armenia Armenia-Turkey Rapprochement (SATR), Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA￾ROLI), and the Small Scale Infrastructure Program (SSIP) Summary: USAID/Armenia requires evaluations of the following activities: Armenia-Turkey Rapprochement (SATR), IFES and NDI electoral and political process Associate Awards under the Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA-RO LI), and the Small Scale Infrastructure Program (SSIP). The purposes of this Task Order are to evaluate the success of these projects in their relevant areas and assess effectiveness of these in achieving set programmatic goals and the USAID/Armenia’s strategic objectives. Two of the five planned evaluations are designed as midterm evaluations (Armenia￾Turkey Rapprochement, and ABA-ROLI), while the other three are designed as end-of-project evaluations (IFES, NDI and SSIP). In the case of mid-term evaluations the findings will be used to inform USAID’s determination whether they are on track achieving their set programmatic goals and targets and whether the initial design of the projects still leads them to the set objectives. They will feed information into the future work plans. In the case of end-of-project evaluations, the findings will be used to inform design and development of future projects. Therefore, the evaluations will identify “lessons learned”; assess strengths and weaknesses of strategies and activities performed under these projects; and provide recommendations to USAID for project planning purposes for the next three to five years. The Contractor will seek to capture effective approaches; analyze the utility of performance monitoring efforts; consider respective outcomes and results; and assess the influence of internal and external changes on the achievement of results. Contractor Responsibilities and Projects: The evaluation should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, including an “effectiveness and efficiency assessment” that looks at how successful the programs have been in achieving their set targets, and how effectively USG resources have been used. Additionally, USAID/Armenia would like to measure the sustainability of the project results on respective beneficiaries where applicable and possible. The Contractor shall review each project’s implementation methodology and to the degree possible, verify the results achieved, the relevance of the project in addressing USAID priorities, and to what extent USAID can be accountable for achieving those objectives. Final recommendations to USAID will help improve program outcomes, weigh sustainability factors, and address program relevance as well as cost efficiency and effectiveness. The evaluations will serve to guide how similar projects, approaches and/or work plans can be improved. The evaluations will also validate (or not) the feasibility of the initial designs of the projects and of their respective development hypotheses. SATR IFES Evaluation Report International Business & Technical Consultants, Inc. Page 28 This two-year activity is in its first year of implementation. The main objective is to promote improved Armenia-Turkey relations by engaging civil society in the reconciliation processes; establishing and developing business partnerships and regional professional networks; and facilitating government-to government dialogue. The activity is being implemented in a rapidly changing environment of Armenia-Turkey relations. Therefore the appropriateness of the design of the project needs to be explored in addition to the actual project implementation issues. Thus, this evaluation should contribute to the Mission’s understanding of whether or not adjustments are necessary in the approach and in the future planning of this project. The evaluation should aim at obtaining feedback from stakeholders and project partners both from Armenia and Turkey. CEPPS These are two separate Associate Awards under CEPPS III, implemented by IFES and NDI, both of which are in their third year of implementation and are expected to end in FY2012. Both activities are primarily aimed at improving political processes. The IFES activity supports the strengthening the administration of electoral processes and assists the electoral administration to meet international standards for free and fair elections through its support to the Central Election Commission and the Passport and Visa Department of the Police of the Republic of Armenia (OVIR). The NDI activity aims to improve the ability of citizens to effectively participate in political processes as members of political parties and helps political parties develop their ability to take part in parliamentary and presidential elections scheduled for 2012 and in 2013 respectively. The findings and recommendations from this evaluation will be reviewed for results achieved, and effectiveness of program approaches. Best practices and lessons learned will be identified which will contribute to the Mission’s decisions on future election-related activities. ABA-ROLI This is in its third year of implementation and will end in FY2012. The main objectives are to support curriculum reform in the Judicial School; cooperate with the Judicial Department of the Republic of Armenia for enhancing the judicial reforms; and provide greater access to justice through Law School legal clinics with a particular focus on protection of human rights. The purpose of evaluation is to assess the results and outcomes of this rule of law initiative and the sustainability of the achievements of the project to inform future USAID/Armenia decision￾making with regard to similar undertakings. SSIP This is a 27-month project ending in FY2012. The goal of the project is to mitigate the consequences of Global Economic Crisis through the creation of temporary employment opportunities in vulnerable rural communities by means of implementation of small scale infrastructure projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre-schools, community centers, sport halls and drinking water supply systems. This project deals with multiple communities and partners in jointly carrying out construction and work with target communities. Executing water projects has been challenging due to different factors, including multiple construction partners, delayed contribution from other donors, weather conditions, etc. However, civil construction has been progressing timely and with very good quality. Project has been modified twice to increase the number of projects (from IFES Evaluation Report International Business & Technical Consultants, Inc. Page 29 48 to 58), labor days and expand the geographic coverage. The final evaluation will look at few aspects of the program: how the targets have been met vis-à-vis original targets and benchmarks; how the partnership between all parties (IFAD, local NGO Shen, local construction companies, village Mayors, Local Supervisors, etc.) worked; what are mechanisms that would help communities sustain project gains after its completion; did the project leverage contributions or matching funds from other donors or stakeholders as anticipated originally. Evaluation Questions: The Contractor shall review and summarize the implementation and results achieved by all five activities to answer the following evaluation questions and additional questions that may be developed by the Contractor after reviewing the provided materials. For ALL projects: • To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? • Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? • What were the main achievements of the program? • How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? • How did the implementer perform in terms of project management and how effective was the project leadership? • What lessons learned‟ can be provided for future USAID programming in this area? • What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? • Did the agreement provide clear and achievable results against which progress and impact could be measured? • Analyze attribution of project successes to USAID involvement. • Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. Project-specific questions/Tasks: For SATR: • Are the project implementation approaches relevant and feasible in the current state of affairs in the Armenia-Turkey relations? • Which specific aspects of the project are or are not working in the given political situation? [This question should be explored in the Armenian and Turkish contexts separately, because some approaches/activities may be still feasible in Armenia but not in Turkey and vice versa.] • How flexible is the project in terms of adjusting activities to the changing political context? • How has this activity been able to build on the advances made under the previous “Days 2 and 3”project? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 30 • What signs exist that the project is having impact, anecdotally? Why? What concrete examples of impact (or lack of it) are given? • What is the external perception of the project’s role and impact both in Armenia and Turkey, according to the key stakeholders not involved in the project? • What recommendations can be provided to improve impact? What are the key obstacles and what recommendations can be made to minimize their effect? • How appropriate is the decentralized project implementation approach (four local partners with their Turkish counterpart organizations) to the project objectives? • How effective is the implementing partner as a consortium of local organizations: responsiveness to the donor, responsiveness to project stakeholders, information flow. What recommendations can be made to improve consortium management and operations? For CEPPS: • How can the projects be better designed in the future to measure impact, given the political situation faced in Armenia? NDI • Have NDI youth and women’s Leadership Training Academies, the NDI Women’s Candidate Schools, and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party memberships and activities, political engagements, political public gatherings, etc.) among youth and women? • Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political and policy activities. • To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations? • Have NDI/Yerevan Press Club public debates had value for participating political parties? • What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? How successfully? IFES • Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code? • Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 31 • Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to electoral administration reforms? • Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? For ABA-ROLI: • What are the most notable accomplishments of the project in the ROL area since the inception of the current agreement in 2009? • What are the factors hindering judicial independence in Armenia and what has the project done to strengthen the judiciary? • Has the project been able to increase the capacity of the Armenian judiciary to play a role in making the government more accountable? • What did the project do to ensure proper implementation and enforcement of new or existing laws? • What are the activities aimed to establish mechanisms for oversight of court proceedings o How do these mechanisms affect judicial performance? o Are the existing oversight mechanisms sufficient to activity build on the existing efforts to promote accountability and transparency in the justice sector? o If yes, please name, if not, please mention gaps. • Is there capacity or interest within the legal professionals to promote reform in ROL? If yes, what does the project do to support that interest? • How does the project support the country’s only Bar Association in helping to play a balancing role within the justice sector? • What has the project accomplished in regard to defending human rights through legal protection? • How did the project affect legal education in law schools? • Where is the most viable stakeholder support for rule of law reform likely to be found? • What targeted activities could be proposed to address Armenia specific deficiencies in the justice sector? • How successfully does the project coordinate with the international community to promote human rights? • What recommendations can be made for a more effective, integrated project design? For SSIP: • What is the ratio of “planned” or projected and “actual” jobs generated throughout the project? How do you explain the gaps (if any)? • What are social, economic and other impacts of the project on target communities and beneficiaries? • How do you think the completed projects will be maintained by the communities? What are the grounds for their sustainability? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 32 • How did the partnership work within the project between different partners (IFAD, Shen, local contractors, communities, etc.)? What worked, what did not, why? • How did the partnerships impact the project (timeliness, quality, and cost-benefit)? • In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met? • How does success in two main areas that the project tackled - water and civil construction compare in terms of progress made, achievements, implementation challenges, etc.? • Has/will CHF meet its cost-sharing commitment? Are cost-sharing valuations reasonable, consistent and adequately documented? • What are the lessons learned? What are the strengths and weaknesses of this project, its approaches or strategies? • If a similar activity is considered in the future, what should be changed in the design and technical approach? USAID’S Role in the Evaluation The USAID Mission in Armenia will: • organize a small USAID advisory group to support the Contractor in the implementation of this scope of work; • provide relevant programmatic and budgetary information to the Contractor (some relevant portions of contracts and assistance agreements are attached); • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor will: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Develop additional research and evaluation questions as needed based on the development hypothesis and on the above-mentioned evaluation questions; identify informants and stakeholders, samples and/or other relevant data sources. 3. Develop data collection tools based on the best possible methodology in accordance with the evaluation questions and feasibility considerations and provide to USAID prior to commencing field work. 4. Prepare a field work plan. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 33 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports as requested by USAID and submit final reports to USAID/Armenia for acceptance. The proposed methodology should address the need for data collection from qualitative and quantitative sources; and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. To the extent possible, data should come from facts, rather than be based on anecdotal evidence, and conclusions should be based on findings received from multiple sources. Clear, standardized data collection methodology should be described in detail to ensure reliability and consistency of the evaluation findings. Deliverables The Contractor’s deliverables shall include: 1. A written methodology plan (research design and operational work plan). 2. Provide a verbal debriefing at the end of the field work to Mission management and technical teams. 3. Prepare draft evaluation reports (electronic and hard copy) which will analyze data and summarize key findings, conclusions and recommendations. The Evaluation Report shall at a minimum contain1) an Executive Summary; 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work, all evaluation tools, all sources of information. Submit these to USAID/Armenia within three weeks after completing the fieldwork. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. The Contractor shall submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The final reports will meet the following quality standards: a) The reports will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports shall address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings will assess outcomes and impact on males and females; f)Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of people’s opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) IFES Evaluation Report International Business & Technical Consultants, Inc. Page 34 Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. The Contractor will submit Evaluation data to USAID/Armenia along with the final reports for warehousing and future use by the Mission. The data will be in easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Evaluations Timeline 1. Evaluations should be initiated around September-October 2011. The preliminary findings of the evaluations should be submitted to USAID/Armenia immediately after the completion of the field work. 2. The first drafts of the five evaluation reports shall be submitted to USAID/Armenia no later than three weeks following the completion of the field work. (For CEPPS USAID expects to have two separate reports – one on the IFES activity and one on the NDI activity). 3. The final Evaluation Reports shall be submitted no later than two weeks after final comments on the draft evaluation reports are submitted by USAID/Armenia. 4. Once finalized, the contractor is responsible for ensuring that the final approved reports are also submitted to USAID’s Development Experience Clearinghouse within three months of the completion of the reports. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 35 ANNEX II: EVALUATION WORK PLAN Introduction This is the final evaluation of the current three year IFES project, ‘Advancing Armenia’s Election Management and Democratic Culture’ (AAEM-DC). The program began in April 2009 and ends on March 31, 2012. IFES has been the implementing partner of the CEEPS (Consortium for Elections and Political Process Strengthening) program for this project. NDI has been an associated partner with IFES in the CEPPS program. In Armenia’s contentious political environment, the development hypothesis of the IFES project is that by focusing on promoting the independent and sustainable capacities of Armenia’s electoral and civic registration institutions, and deepening the commitment to international election standards and practices, the project will help to revitalize public confidence and broaden participation in the electoral process. In order to achieve these goals, the IFES project has had 4 objectives: 1. Improve the voter registration system and isolate it from abuse 2. Election Administration Reform 3. Strengthening political will to change the election culture 4. Campaign finance reform through a Money and Politics program These four objectives and their activities constitute the core of the IFES program in Armenia. The success of the program relies on two major assumptions: 1. That the GOA provides full and open cooperation during the entirety of the project, and 2. Political will is maintained by all stakeholders to see the process through to a successful conclusion. Purpose The general purposes of the evaluation as stated in the Scope of Work (SOW) are: 1. To measure and analyze the accomplishments toward achievement of the results of the activities, including an ‘effectiveness and efficiency assessment’ that examines how successful the program have been in achieving their set targets; 2. To assess how effectively USG resources have been used; and 3. To measure the sustainability of the project results on respective beneficiaries. The findings of the evaluation will be used to inform design and development of future projects. The evaluation will identify ‘lessons learned’; assess strengths and weaknesses of strategies and activities performed; and provide recommendations to USAID for project planning purposes for the next three to five years. Methodology IFES Evaluation Report International Business & Technical Consultants, Inc. Page 36 The evaluation team consists of three people: Marilyn Evans (US), Hrachya Zakoyan (Armenian) and Robert Brandstetter (US). An Armenian interpreter will also be part of the team. The team will spend 14 days in Armenia, from 20 November to 6 December 2011. Because of the structure of the CEPPS project and the different primary activities of IFES and NDI, the team has divided the work, with Evans focusing on NDI activities, and Brandstetter and Zakoyan focusing on IFES. The team will discuss their work continually and note where there are synergies and overlaps between the IFES program and the NDI program. The field work will be guided by a number of specific questions noted in the SOW: • Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code? • Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? • Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to electoral administration reforms? • Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? • How can the CEPPS projects be better designed in the future to measure impact given the political situation faced in Armenia? The research methodology that will be used in the evaluation will consist of the classic social science techniques of: desk review of the literature, one-on-one interviews with key informants (Annex I), focus groups, and, if possible, observation of training sessions. 1. The desk review prior to deployment to the field will consist of the review of all available IFES documents, including the project paper, PMPs, work plans and quarterly reports. Other relevant documents such as OSCE election reports and Freedom House reports will also be consulted. 2. Meetings with the headquarters staff at IFES and scholars of Armenia and the Caucasus in Washington will be interviewed prior to deployment. 3. The field research will consist primarily of meetings, focus groups and informational interviews with key individuals and groups, as noted in Annex I. These interviews and meetings will be both structured and unstructured. A set of questionnaires will guide the structured interviews. (Annex II) IFES Evaluation Report International Business & Technical Consultants, Inc. Page 37 4. While the majority of the interviews will take place in Yerevan, the team will also travel outside of the capital in order to visit a representative number of VLACs and provincial election committees. 5. A draft work plan can be found in Annex III. Deliverables 1. This work plan and evaluation design is the first deliverable and is sent to the USAID/AM for approval, in advance of the first briefing with USAID mission. 2. An oral debriefing at the end of the field work will be made to the mission management and technical teams. 3. Draft reports analyzing the data and summarizing the key findings, conclusions and recommendations will be provided to the mission by December 30. The format of the evaluation will follow the standard format for USAID evaluation reports,13 with the IFES/CEPPS text not exceeding 25 pages, excluding annexes. 4. Mission comments will be incorporated into the final report and submitted by February 3, 2012. 13 IBTCI evaluation teams refer to USAID’s Evaluation Learning from Experience, Bureau for Policy Planning and Learning, and Checklist for Assessing USAID Evaluation Reports for guidance. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 38 ANNEX III: LIST OF INFORMANTS FOR FIELD INTERVIEWS Key Informants USAID/Armenia: Director Jatinder Cheema COTR Mariam Gevorgyan COTR/CEPPS Nick Stokes Project Implementers NDI COP Gegham Sargsyan IFES COP Michael Getto AAEM-DC Program Partners and Involved Parties CEC (Central Election Commission) Chairman Tigran Mukuchyan & other representatives OVIR Passports and Visas Department, the Police of Armenia officials VLAC (Voters List Advisory Committee) National Assembly State & Law Committee Chairman Davit Harutyunyan National Assembly Standing Committee on State and Law, members Government representative (RA Ministry of Education) MPs (Members of Parliament) Presidential Secretariat, Mr. Garnisk Isagulyan OSEC Representative Council of Europe Representative Counterpart International representative Participants and Beneficiaries Election Management Advisory Panel (EMAP) sessions participants People trained by the EMB (Election Management Body) for Yerevan local council election Election Management Advisory Panel Chairman of the Paros Disability NGO Chairwoman of the Lusastgh NGO Representative of “It’s Your Choice” NGO IFES Evaluation Report International Business & Technical Consultants, Inc. Page 39 “Mobilizing Action Against Corruption” representatives Observation of training for the testing and piloting new computerized accreditation examination system for prospective PEC candidates and local observers Observation of the training sessions for TEC Commissioners Citizens trained in voter registration procedures Independent Experts Caucasus Institute, Director Alexander Iskandaryan Additional Informants Participants in Political-Citizen Discussion Groups Political Parties, members and candidates • National Democratic Union • National Democratic Party • Legal State Party • Armenian National Congress • Republican Party • National Party of Armenia • Heritage Party • Prosperous Armenia • Media people who have been involved with IFES IFES Evaluation Report International Business & Technical Consultants, Inc. Page 40 ANNEX IV: INTERVIEW GUIDES IFES QUESTIONNAIRE FOR CEC Final Evaluation of CEPPS/IFES/Armenia IBTCI November-December 2011 1. What are the greatest problems facing the CEC in the administration of elections today? 2. Has the IFES project helped you solve any of these problems? 3. What are the major legal problems with the administration of elections? How has IFES helped to solve them? Was the legal reform analysis that IFES implemented useful? Have there been any legislative changes as a result of this analysis? 4. What kind of support does the commission receive from the public? Is it seen as a credible institution? 5. How successful and useful was the workflow analysis that IFES did? Has it resulted in any important administrative reforms? 6. What other donors have been working with the CEC? What kind of assistance have they provided? Material support? Training support? Has this assistance been in cooperation with IFES? 7. How effective have the EMAPs been in facilitating a smoother electoral process? Have the EMAPs become institutionalized? Will the EMAPs be sustainable after the IFES project ends? 8. Have the various manuals and publications that IFES has helped produce and distribute been used by their intended targets? Young people? Poll workers, election administrators? 9. If the IFES project was extended, what kind of help would you ask for? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 41 IFES QUESTIONNAIRE FOR OVIR Final Evaluation of CEPPS/IFES/Armenia IBTCI November-December 2011 1. How complete and accurate is the current voter registration list? 2. What problems existed with the list during the most recent mayoral elections? 3. What needs to be done to make the national list perfect? 4. What has been the relationship between OVIR and IFES? 5. If IFES has been useful and helpful to you, how has the program assisted you in making your work better? 6. Are the VLACS functioning as they should in order to help you create a more accurate voters registry? 7. Has IFES helped develop a sustainable collaboration between OVIR and the VLACS? 8. Will the work that you have been doing with IFES continue after the project is over? Will the work be sustainable? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 42 IFES QUESTIONNAIRE FOR EMAP MEMBERS Final Evaluation of CEPPS/IFES/Armenia IBTCI November-December 2011 IFES helped establish an Election Management Advisory Panel which has met frequently throughout the country during the course of the project tenure. The EMAP has brought together various stakeholders involved in the voter registration process, and discussed key issues and reforms for future implementation. 1. How were you chosen to be a member of the EMAP? 2. How long have you been a member of the EMAP? 3. Did you receive any training to be on EMAP? Who gave the training? What did it consist of? 4. What do you understand is the role of the EMAP? What is the primary objective of the panels? 5. What are the major issues that the panel has had to deal with while you have been a member? Which of these issues has the panel been able to resolve? 6. Has the registration system and other aspects of the election process improved since you have been on the panel? Gotten worse? How so? Why? 7. Do you think that the panel has real influence and power to bring about change in the system? 8. Do you think that the new electoral code will make the voting in Armenia more transparent, open, fair and legitimate? 9. The CEPPS project is scheduled to end in January 2012, what are the chances that the EMAP will continue? Are the panels sustainable? 10. Did the International Symposium on Election Codes influence public debate on the process of amending the electoral code? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 43 IFES QUESTIONNAIRE FOR VLACS Final Evaluation of CEPPS/IFES/Armenia IBTCI November-December 2011 1. When you were working as a VLAC volunteer of which NGO were you a member? Where was your work? Which marz and city/town did you work in? 2. Was the training provided by IFES sufficient and adequate for you to do the job that you needed to do? 3. As a result of your work verifying the VL, how accurate do you think the VL is, now that it has been checked? What are the biggest problems remaining with the VL in the marz where you worked? 4. Will your NGO continue to verify the voter’s list before the next election even if IFES is unable to continue its support? 5. Did you report the findings of your work to the OVIR office? How cooperative was OVIR in helping you with the work? 6. What were your greatest problems as a VLAC volunteer? 7. Did you find many irregularities? What were they? IFES Evaluation Report International Business & Technical Consultants, Inc. Page 44 ANNEX V: KEY REFERENCE DOCUMENTS REVIEWED Attachment B, “Advancing Armenia’s Election Management and Democratic Culture.” Program Description (AID-111-A-09-00002), nd. CEPPS/IFES. Quarterly Report: October 1, 2009-December 31, 2009 CEPPS/IFES. Quarterly Report: April 1, 2009-June 30, 2009 CEPPS/IFES. Quarterly Report: July 1, 2009-September 30, 2009 CEPPS/IFES. Quarterly Report: October 1, 2010-December 31, 2010 CEPPS/IFES. Quarterly Report: January 1, 2010-March 31, 2010 CEPPS/IFES. Quarterly Report: July 1, 2010-September 30, 2010 CEPPS/IFES. Quarterly Report: January 1, 2010-March 31, 2011 CEPPS/IFES. Quarterly Report: April 1, 2011-June 30, 2011 CEPPS/IFES. Quarterly Report: July 1, 2011-September 30, 2011 CEPPS (IFES).Three Year Project Work plan (Updated). April 2011-December 31, 2011 CEPPS (IFES), AAEM-DC. Program Year Three Timeline: April 1, 2011-December 31, 2011. Civicus, Civil Society Index. Armenian Civil Society: from Transition to Consolidation/Policy Action Brief. Yerevan, Armenia: 2010. [www.counterpart.am] Civicus, Civil Society Index. Armenian Civil Society; from Transition to Consolidation/Analytical Country Report. Yerevan Armenia: 2010. [www.counterpart.am] Duban, Elizabeth and Hasmik Gevorgyan. Gender Assessment: USAID/Armenia. 2010.[http://www.usaid.gov/our_work/crosscutting_programs/wid/pubs/Armenia_Gender_Asses sment_2010] IFES. Training Workbook for Precinct Election Commission Members. Yerevan: IFES, 2007. IFES/Armenia. “Voter Registration in Armenia: Problems and Possible Solutions.” 10 November 2009. IFES. “Yerevan City Council Observation Notes prepared by IFES/Armenia.” [2009], (Manuscript) IFES. “Technical Observation Report on Yerevan Council Elections, May 30, 2009.” (Manuscript) IFES Evaluation Report International Business & Technical Consultants, Inc. Page 45 IFES.“PMP Indicator Table - Advancing Armenia’s Election Management and Democratic Culture (AAEM-DC), Process Indicators”.Nd. (Manuscript) IFES. “IFES Technical Report: National Assembly Majoritarian Constituency By-Election. 10 January 2010.” (Manuscript) IFES. “Civic Education Textbook Comparison Matrix.” 4 February 2010 (Manuscript) IFES. “Incident Matrix—NA Constituency 10 By-Election, 10 January 2010.” (Manuscript) IFES/Armenia. “Election Observation Summary Report” Head of Community Elections in Armavir city, Armenia, March 20, 2011. (Manuscript) IFES. “Overview of the Electoral Reform Process in Armenia and the New Electoral Code, June 8, 2011”. (Manuscript) IFES. “Comparison of the New Electoral Code—First Reading and Second and Final Readings. June 2, 2011”. (Manuscript) IFES. Guideline to Support the Voters who Can’t Fill Ballot by Themselves because of Vision Problems. (Sample ballot in Braille) [translated from Armenian] IFES/UNDF. Global Standards of Political Financing. International Fund of Electoral Systems. [translated from Armenian] IFES/USAID. Young Person’s Guide to National Assembly. Yerevan: IFES/USAID, 2011. IFES/USAID. Guideline for the Members of the PEC. Yerevan Council Elections, 31 May 2009. [translated from Armenian] IFES/USAID. Guideline for Vocational Trainings about Conducting the Elections, 2011. [translated from Armenian] IFES/USAID. Training Materials for the PEC Members. Republic of Armenia, National Assembly Elections (12 May 2007) [translated from Armenian] IFES/USAID. Guideline for the Members and Membership Candidates of Electoral Commissions. 2009. [translated from Armenian] IFES/USAID. Guideline for the Members of Precinct Electoral Commissions. Republic of Armenia, Presidential Elections, 19 February 2008. [translated from Armenian] IFES/OSCE. The Responsibilities of Policemen on the Day of Elections. [translated from Armenian] IFES. “Workflow Analysis Report Central Election Commission of Armenia.” IFES: Yerevan, 2011. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 46 International Republican Institute (IRI), The Gallup Organization, Armenian Sociological Association. Armenian National Voter Study, July 2007. Published February 2, 2008. (PowerPoint) Investigative Journalists NGO. Violence, Intimidation and legal cases against journalists and the media in Armenia 2008-2009: Cases and Analysis. OSCE: Yerevan, 2009. Iskardaryan, Alexander. “Armenia,” chapter in Nations in Transition 2011. Freedom House, 2011. [http://freedomhouse.org/template.cfm?page=678] Human Rights Watch. Democracy on Rocky Ground: Armenia’s Disputed 2008 Presidential Election, Post-election Violence, and the One-sided Pursuit of Accountability. HRW: New York, 2009. [http://www.hrw.org/sites/default/ files/reports/armenia0209webwcover.pdf] It’s Your Choice. On results of ‘It’s Your Choice’ NGO’s observation mission of the February 19,2008 Presidential elections in Armenia. Final Report. Yerevan: April 2008. Yo’av Karny. Highlanders: A Journey to the Caucausin Quest of Memory .Farr Straus & Giroux: New York, 2000. Martin-Rozumilowicz, Dr. Beata. Strengthening Electoral Processes and Administration in Armenia (SEPA).IFES/USAID: Yerevan, 2008. Mauro, Maria Teresa, Kare Vollan Jessie Pilgrim. Draft Joint Opinion on the Electoral Code of Armenia, Adopted on 26 May 2011.European Commission for Democracy Through Law (Venice Commission) and OSCE/ODIHR. Strasbourg, 29 September 2011 Opinion No. 611 / 2011 (Or. Engl.) [http://www.venice.coe.int/ docs/2011/CDL(2011)020-e.pdf] Ohman, Magnus and Hani Zainulbhai, (eds.) Political Finance Regulation: The Global Experience. IFES: Washington, DC, 2011. OSCE/ODIHR. Armenian Presidential Elections September 24, 1996. Final Report. (Simon Osborn OSCE/ODIHR International Observer Coordinator.) OSCE/ODIHR Election Observation Mission – Armenia Extraordinary Presidential Election, 16 March 1998. Preliminary Statement by the Election Observation Mission to the Armenian Extraordinary Presidential Election 16 March 1998. OSCE/ODIHR. Republic of Armenia Presidential Election. March 16 and 30 1998 Final Report. 9 April 1998. OSCE/ODIHR. Final Report: Republic of Armenia, Parliamentary Election, 30 May 1999. Warsaw 30 July 1999. [http://www.osce.org/odihr/elections/armenia/14203] OSCE/ODIHR. Preliminary Statement: OSCE Observation Mission, Republic of Armenia. 31 IFES Evaluation Report International Business & Technical Consultants, Inc. Page 47 May 1999. (14196) OSCE/ODIHR. Final Report: Parliamentary Election, 25 May 2003, Republic of Armenia. Warsaw 31 July 2003. [http://www.osce.org/odihr/ elections/ armenia/42371] OSCE/ODIHR. Republic of Armenia Parliamentary Elections. 12 May 2007. OSCE/ ODIHR Election Observation Mission Report. Warsaw ,10 September 2007. OSCE/ODIHR. OSCE/ODIHR Needs Assessment Mission Report: Republic of Armenia: Presidential Election, 19 February 2008. OSCE/ODIHR: Warsaw, 13 December 2007. [http://www.osce.org/odihr/elections/armenia/29792] OSCE/ODIHR. Republic of Armenia Presidential Elections, 19 February 2008. OSCE /ODIHR Election Observation Mission Report. Warsaw 30 May 2008. OSCE/ODIHR. Final Report Trial Monitoring Project in Armenia (April 2008 – July 2009). Warsaw 8 March 2010. [http://www.osce.org/odhir/41695] Policy Forum Armenia. Yerevan’s May 2009 Municipal Election: Statistical Analysis. September 2009. [http://www.pf-armenia.org/fileadmin/pfa_uploads/Yerevan_Municipal_Election_Report-- FINAL.pdf] Policy Forum Armenia. Armenia’s 2008 Presidential Election: Select Issues and Analysis. July 2008. http://www.pf-armenia.org/fileadmin/pfa/uploads/PFA_Election_Report--FINAL.pdf] USAID/Armenia. Case Study: Election Law Increases Transparency. USAID/Washington, “Telling Our Story”. (http://stories.usaid.gov) USAID/Armenia. Case Study: Discussing and Debating Electoral Legislation. USAID/Washington, “Telling Our Story”. (http://stories.usaid.gov) IFES Evaluation Report International Business & Technical Consultants, Inc. Page 48 ANNEX VI: SCHEDULE OF MEETINGS Date Weekday Time Meetings / Activities Place Phone Nov-21 Monday 11:00 USAID Armenian Office 1American Avenue, Yerevan 13:30 NDI Armenian Office #24, 1 Baghramyan str., Yerevan 16:00 IFES Armenian Office 10 Proshyan Str., Yerevan Nov-22 Tuesday 09:45 IYC’s Central Office Staff 48/1 Charents str., Yerevan (091) 406810 11:30 Observing a session of Youth Leadership School Erebuni Hotel, Yerevan 14:30 Independent Expert Avetik Ishkhanyan, Helsinky Committee of Armenia, Erebuni Hotel, Yerevan (091) 412043 15:00 Hovsep Khurshudyan, Spokesperson of Heritage "Political party" # 76a., 31 Moskovyan str., Yerevan (010) 536913 17:00 CEC Secretary Armen Smbatyan and member of CEC Tatev Ohanyan 21a G. Kochar Street (010) 54 35 23 Nov-23 Wednesday 10:30 Davit Harutyunyan, Chairman of Standing Committee on State and legal Affairs. Admission only with ID 19 Marshal Baghramyan. Parliament Building Yerevan (010) 588429 11:30 Samvel Nikoyan, Vice Speaker/Acting Speaker of National Assembly (010) 529540 15:00 Carel Hofstra, Acting Head of the OSCE Office in Yerevan 64/1 Sundukyan Str. Yerevan (010) 229610 17:00 Varuzhan Hoktanyan, Transparency International 6 Aygestan 9th Str. Yerevan (093) 539937 IFES Evaluation Report International Business & Technical Consultants, Inc. Page 49 Nov-24 Thursday 10:00 Discussion and updating of the schedule Marriott Hotel 11:30 Norayr Muradkhanyan, Head of OVIR 13a Mashtotci Street, Yerevan (010) 53 69 32 15:00 Gegham Hairapetyan, Chairman of TEC#1 Babajanyan 18/3, (Cultural Center Avan), Yerevan (094) 99 77 16 19:00 Hamlet Abrahamyan, from "Dashnaktsutyun" Party, ex￾member of CEC Marriott Hotel (091) 47 69 63 Nov-25 Friday 10:00 Activity planning Marriott Hotel 14:00 Caucasus Institute Alexandr Iskandaryan 39 Yeznik Koghbatsi Str., Yerevan (091) 20 80 36 15:30 Naira Arakelyan, Chairwoman of "Armavir Development Center: (IFES' VLAC partner) Armavir Marz, Armavir town, Shahumyan str.68a building (094)502039 17:00 After the meeting attending the discussion with political parties, CSOs, activities on election related topic. Event is organized by "Armavir Development Center" in cooperation with Yerevan Press Club. Saturday 10:00 Reviewing Notes Marriott Hotel 12:00 Reading additional documents provided by IFES and reports of Armenian think tanks Marriott Hotel 16:00 Planning for next week’s activities Marriott Hotel Date Weekday Time Meetings / Activities Place Phone IFES Evaluation Report International Business & Technical Consultants, Inc. Page 50 Nov-28 Monday 10:30 Lilit Asatryan, President of "Armenia Young Women's Association" NGO (IFES' VLAC partner) 37 Abovyan Street, apt. 9. (010) 580787 14:00 Arman Suleymanyan, Radio Van 13a Khanjyan 16:00 Avet Demuryan, Associated Press, Deputy of Yerevan Press Club 39/12 Mashtots Ave. (010) 56 3771 Nov-29 Tuesday 9:00 Meeting with IFES representatives 10 Proshyan Str. Yerevan 11:30 Suren Ohanyan, Chairman of "Paros" NGO and Nune Pepanyan, Chairwoman of "Lusastgh" NGO. Both NGOs are members of Disability Advocacy Coalition and partnered IFES in various voter awareness initiatives 2 Baghayn Street, 10th kindergarten building, Nor Nork 3 (099)009538, or (010) 632958 14:00 Arman Danielyan, President of Civil Society Institute 43 Aygestan 11 Str. (010) 57 43 17 17:00 Edik Baghdasaryan, Head of the Investigative Journalists NGO, Editor-in-chief Fl.8th, 1/3 Buzand Str. (094) 40 46 24 Nov-30 Wednesday 10:00 Arsen Stepanyan, Programs Director, Civil Society, Counterpart International 62 Demirchyan Street (010) 519 027 14:00 Vahan Tumasyan, Shirak Centre NGO Shirakmarz, Gyumri town 301 Varpetac Str. (0312) 66 - 992 (091) 451-337 Dec-1 Thursday 10:00 Vahan Hovsepyan, Management Adviser of Union of Information Technology Enterprises (IFES' EMAP partner) 29 Nalbandyan Street apt. 36 (091) 248881 11:00 CPI Launch, Transparency International Congress Hotel 15:00 Summarizing of collected information Marriott Hotel Dec-2 Friday 10:00 Preparation for meeting Marriott Hotel IFES Evaluation Report International Business & Technical Consultants, Inc. Page 51 13:00 Christopher Anderson - Political Officer, Lilit, Embassy of USA, Yerevan and Nick Stockes, USAID Armenian Office 1American Avenue, Yerevan (010) 494628 17:00 Onno Simons, First Counsellor, Head of the Political, Economic, Press and Information Section, Delegation of the EU to Armenia 21 Frik Str., Yerevan (010) 546495 Dec-3 Saturday 10:00 Michael Getto, IFES COP Marriott Hotel 11:30 Mayis Vanoyan Marriott Hotel 15:30 Tigran Zakharyan, Gavar town municipality Gegharkunik Marz, Gavar town Dec-5 Monday 9:30 Eduardo Lorenzo Ochoa, Long term Policy Advisor for Strengthening Democratic Structures EU Advisory Group to the RA Marriott Hotel (091)115 769 11:00 Summarizing the results of the study Marriott Hotel Dec-6 Tuesday 9:30 Preparation for debriefing Marriott Hotel 17:00 Debriefing in USAID Armenian Office I American Avenue, Yerevan IFES Evaluation Report International Business & Technical Consultants, Inc. Page 52 ANNEX VII: LIST OF PEOPLE CONSULTED Hamlet Abrahamyan Former member, CEC; member of “Dashnaktsutyun” Party, Yerevan Oksanna Abrahamyan Deputy Head of Office/Operations Manager, IFES/Armenia Christopher Anderson Political Officer, US Embassy, Yerevan Naira Arakelyan Executive Director, Armavir Development Center, Armavir Lilit Asatryan President, Armenian Young Women’s Association/Yerevan Edik Baghdasaryan Head of the NGO Editor-in-Chief, Investigative Journalists/HETQ Online/Yerevan Anthony Bowyer Program Manager, Caucasus & Central Asia, IFES/Washington Stephen M. Brager Director, DG Office, USAID/Armenia Jatinder Cheema Mission Director, USAID/Armenia Arman Danielyan President, Civil Society Institute, Yerevan Hamazasp Danielyan Monitoring and Evaluation Officer, IFES/Armenia Avet Demuryan Vice President, Yerevan Press Club, Yerevan Michael Getto COP/IFES/Armenia Mariam Gevorgyan M&E Specialist, USAID/Armenia Gegham Hairapetyan Chairman, Territorial Election Committee #1, Yerevan Harutyun Hambardzumyan President “Its Your Choice” NGO, Yerevan Davit Harutyunyan Chairman of Standing Committee on State and Legal Affairs, National Assembly, Republic of Armenia Alexandra Hiatt Senior Program Assistant, Eurasia, NDI/Washington Carel Hofstra Deputy Head of Office, OSCE/Yerevan Varuzhan Hoktanyan Executive Director, Transparency International, Yerevan Vahan Hovepyan Management Advisor of Union of Information Technology Enterprises IFES Evaluation Report International Business & Technical Consultants, Inc. Page 53 Tamara Hovnanyan Human Rights Department Coordinator, Civil Society Institute, Yerevan Avetik Ishkhanyan Chairman, Armenian Helsinki Committee, Yerevan Alexander Iskandaryan Director, Caucasus Institute, Yerevan Hovsep Kurshudyan Spokesperson, Heritage Party, Yerevan Samvel Nikoyan Vice President of the National Assembly of the Republic of Armenia Muradkhanyan Norayr Head of Department, Passport & Visa Department (OVIR), Police of RA, Yerevan Eduardo de Lorenzo Ochoa Long Term Policy Advisor for Strengthening Democratic Structures, European Union Advisory Group to the RA, Yerevan Ms. Tatevik-Ohanyan Member of Central Election Commission, Yerevan Suren Ohanyan President Paros Disabled Centre of Development of Education and Culture, NGO, Yerevan Nune Pepanyan Chairwoman, Lusastgh (NGO), Disability Advocacy, Yerevan Nick Stokes COTR, CEPPS Project, USAID/Armenia Argam Sanasaryan Head of IT Subdivision, Passport and Visas Department of Police (OVIR), Yerevan Gegham Sargsyan COP/NDI/Armenia Onno Simons First Counselor, Head of the Political, Economic, Press and Information Section, European Union, Yerevan Armen Smbatyan Secretary, CEC, Yerevan Arsen Stepanyan Programs Director, Civil Society, Counterpart International, Yerevan Arman Suleymanyan Radio Van Vahan Tumasyan President, Shirak Centre NGO, Gyumri Edmund R. Rhoads Senior Program Officer, Central & Eastern Europe, NDI/Washington IFES Evaluation Report International Business & Technical Consultants, Inc. Page 54 Mayis Vanoyan Local Infrastructure Consultant, Yerevan Aghassi Yesayan Senior Elections Expert, IFES, Armenia Tigran Zakharyan, Specialist, Gavar town municipality Irina S. Zaslavskaya Program Officer, Europe & Asia, IFES, Washington IFES Evaluation Report International Business & Technical Consultants, Inc. Page 55 ANNEX VIII: CONFLICT OF INTEREST STATEMENTS IFES Evaluation Report International Business & Technical Consultants, Inc. Page 56 Disclosure of Conflict of Interest Name Hrachya Zakoyan Title Monitoring and Evaluation Expert Organization International Business & Technical Consultants, Inc. Evaluation Position? … Team Leader ⌧ Team member Evaluation Award Number(contract or other instrument, if applicable) Contracted under RAN-I-00-09-00016, Task Order Number AID-111-TO-11-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Evaluation of ABA-ROLI, CEPPS, SATR AND SSIP Projects - Associate Awards under Consortium for Elections and Political Process Strengthening, implemented by IFES and NDI I have real or potential conflicts of interest to disclose. … Yes ⌧ No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Signature: Date: 06.03.2012 IFES Evaluation Report International Business & Technical Consultants, Inc. Page 57 ANNEX IX: STATEMENT OF DIFFERENCE IFES Response to USAID Project Evaluation Report – March 20, 2012 IFES appreciates the opportunit y to have its progra m "Advancing Armenia's Election Management and Democratic Culture" assessed by USAID and is gra teful for the time and detail put into the effort by Mr. Robert Brandstetter and Mr. Hrachya Zakoyan. IFES would like to provide the following response for the official record: • IFES acknowledges challenges in the monitoring an d evaluation component of the project. There may have been a misunderstanding in the reading of the approved IFES PMP c hart; the mark “N/A” meant “not applicable” an d not “not available.” Th is is true for the non-gray ed out seg ments of the PMP chart as well as the grayed-out sections, in wh ich of t he former there was still an expectatio n that activities would be conducted before the end of the three-year agreement. However, it could have been better represented that the activities in question were delayed and had not yet produced data, not simply that there was no data available. • In addition, while quantitative indicators were provided and reported on annually for activities conducted, more qualitative indicator s could have been included along wit h a m ore ro bust data collection process to convey the full degree of i mpact of project initiatives, parti cularly as there were many changes in project activities over the last three years. To remedy this, IFES has designed a more comprehensive performance monitoring plan for the project ext ension that will better capture bot h quantitative and qualitative indicators. Further, the m onitoring and evaluation officer in Yerevan will receive additional training on m easurement methodologies and approaches as per the Project Monitoring Plan. Lastly, IFES is refining interi m results and setting achievable goals that can be measured against benchmarks per quarter. • IFES has had and continues to have good, produc tive relationships with all of its governmental partners. Ho wever, in the last six months there have been chan ges at the to p of the OV IR. The previous head of OVIR, Norayr Muradkhanyan, was supportive of IFES, though resisted some of th e proposed joint initiatives on voter registration reform and update that had been proposed at the outset of the project. His successor, Hovhannes Kocharyan, is eager to move forward with development and distribution of voter registration procedural manuals to OVIR officers throughout Armenia during the National Assem bly election period i n t he spring of 2012. These manuals wil l help OVIR officers assist Ar menian voters to confirm the ir place in the voter registry in tim e to vote in the May parliamentary elections. Regarding t he political will of host countr y partners, gi ven the focus of the elections component of the project and the fact that there is one major local partner with who m IFES is mandated to work in a position to im plement election reforms, IFES must assu me t he CEC i s committed to holdi ng open, com petitive elections and the pr ocesses that underpin them . I FES is aware, however, of the difficulties faced by officials in the CEC, which is less an independent body in practice than in name, and of the difficulties in working with this body as well as the OVIR/Police. In its initial proposal to USAID IFES stated: IFES acknowledges that continued work with its key governmental partners in Armenia, the CEC and the Police, has proven difficult at times. But these bodies are permanent, and one must strategically render assistance in a way that does not legitimize a negative process, but rather encourages behavioral change and stated commitment to improving participatory democracy. -CEPPS-IFES Proposal Advancing Armenia’s Election Management and Democratic Culture (AAEM-DC), submitted to USAID, March 2009, p. 22. IFES Evaluation Report International Business & Technical Consultants, Inc. Page 58 In addition, I FES holds as one of its cr itical assumptions in Armenia that the CEC and other local, governmental partners are co mmitted to cooperation in support of democratic elections. Whil e IFES can adjust assistance approaches, as it has in Armenia during this project to account for key developments and “local realities,” the underl ying belief is that the government of Armenia is fundamentally in support of IFES assistance in the areas outlined in the project proposal.