Submitted to USAID on March 20, 2012 This publication was produced for review by the United States Agency for International Development. EVALUATION Evaluation of the National Democratic Institute's Last Three Years of Programming in Armenia ARMENIA NDI/Armenia NDI EVALUATION REPORT Submitted to USAID on March 20, 2012 This publication was produced for review by the United States Agency for International Development. NDI EVALUATION REPORT MARCH 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016-00, Task Order Number AID-111-TO-11-00002 Evaluation of ABA-ROLI, CEPPS, SATR AND SSIP Projects DISCLAIMER The authors’ views expressed in this publication do not necessar ily ref lect the views of the United States Agency for International Development or the United States Government. Author: Marilyn Evans NDI Evaluation Report International Business & Technical Consultants, Inc. TABLE OF CONTENTS Executive Summary ......................................................................................................................... i I. Introduction ......................................................................................................................... 1 II. Evaluation Scope and Methodology ................................................................................... 2 III. Findings ............................................................................................................................... 3 A. USAID General Evaluation Questions ......................................................................... 4 B. USAID Evaluation Questions Specific to NDI’s Work .............................................. 11 IV. Conclusions ....................................................................................................................... 17 A. USAID General Evaluation Questions: ...................................................................... 17 B. USAID Evaluation Questions Specific to NDI’s Programs ....................................... 21 V. Successes................................................................................................................... ........ 23 VI. Lessons Learned ................................................................................................................ 23 VII. Outcomes and Impacts on Males and Females ................................................................. 24 VIII. Recommendations ............................................................................................................. 25 Annex I: Evaluation Scope of Work ............................................................................................. 27 Annex II: Evaluation Work Plan ................................................................................................... 35 Annex III: List of Informants for Field Interviews ....................................................................... 39 Annex IV: Interview Guides ......................................................................................................... 40 Annex V: Key Reference Documents Reviewed .......................................................................... 48 Annex VI: Conflict of Interest Statement………………………………………………………..49 Annex VII: Statement of Difference……………………………………………………………..50 NDI Evaluation Report International Business & Technical Consultants, Inc. LIST OF ACRONYMS ANC – Armenian National Congress, a coalition of Armenian political parties CEC – Central Election Committee CEPPS – Consortium for Elections and Political Process Strengthening EID – European Institute for Democracy IFES – International Foundation for Electoral Systems IYC – It’s Your Choice, an Armenian NGO NED – National Endowment for Democracy NOW – youth group associated with ANC NDI – National Democratic Institute NDU – National Democratic Union, an Armenian political party NGO – non-government organization OSCE – Organization for Security and Co-operation in Europe PMT – performance measurement tool PPA – People’s Party of Armenia, an Armenian political party SMS – short messaging service TI – Transparency International TOT – training of trainers NDI Evaluation Report International Business & Technical Consultants, Inc. Page i EXECUTIVE SUMMARY USAID determined to evaluate the National Democratic Institute’s (NDI) last three years of programming in Armenia, which ended on January 9, 2012. During this timeframe, NDI has served as an implementing partner of the CEPPS (Consortium for Elections and Political Process Strengthening) program, an associate partner of IFES (International Foundation for Electoral Systems). International Business & Technical Consultants, Inc., selected by USAID, evaluated both NDI and IFES separately through desk and field study. The purpose of USAID’s evaluation of NDI is to assess the following: 1) to measure and analyze the accomplishments toward achievement of the results of the activities, including an ‘effectiveness and efficiency’ assessment that examines how successful the program has been in achieving set targets; 2) to assess how effectively USG resources have been used; and 3) to measure the sustainability of the project results on respective beneficiaries. Evidence to support this report’s conclusions and recommendations was acquired through both desk review of relevant documents, largely but not exclusively NDI reports, and extensive field interviews. Thirty-six interviews were conducted of program recipients, NDI’s Country Director and executives of international organizations also dedicated to democratic development. Because of NDI’s stated objectives, nearly half the interviews (individual and group) were conducted outside of the capital, covering seven of ten marzes (regions), including both major cities and villages. NDI has addressed obstacles of Armenian political and civil society since 1995. Faced with an increasingly intractable central government that dominates political life in Armenia, during the last three years, NDI chose a pragmatic approach to encourage increased citizen participation. They engaged citizens across the nation in “bottom-up” tactics to loosen the governing coalition’s hold on political life. During the term of this agreement, while continuing to provide support to political parties, NDI primarily focused on population segments that have only marginal influence but demonstrate high motivation to become powerful “agents of change,” namely women and youth. Additionally, programs were implemented in regions, not just citizens within Yerevan, Armenia’s capital. NDI’s change of emphasis these last three years, from a highly focused political party program, to a more robust political plus civil society program has produced results. Although continuing to serve political parties, they chose to focus on youth members who demonstrate high motivation to become centrally engaged in their parties’ decisions, with the strong potential of demanding a voice and responsible roles. More than 200 young men and women of NGOs and political parties have participated in Youth Leadership Academies, a training program that includes representatives of both governing and opposition parties. All those interviewed highlighted an important result: the novel and important experience of connecting with youth representatives of other parties and listening to their differing perspectives. The hope is that this experience will create collaboration across parties in the near and distant future. However, NDI’s leadership training needs to expand qualitatively to ensure that pragmatic skills are internalized to provide short-term benefits to their parties or NGOs, as well as assure program sustainability. NDI Evaluation Report International Business & Technical Consultants, Inc. Page ii In addition to the Youth Leadership Academies, NDI’s Youth Forum members of Vanadzor were trained and coached to implement community advocacy projects. The committee members and their volunteers have a successful record of lobbying local governments to gain resources and opportunities to solve local problems. Similar projects were reported as having just been initiated in Vayots, Dzor and Dilijan, groups which are expected in the future to be equally successful. However, the number of effective advocacy projects is much fewer than NDI promised. Women, both young and established, received substantial benefit from NDI’s programs. Of eighty-two (82) village women who participated in NDI’s Women Candidate Schools and also ran for office in the 2011 local elections, sixty-one (61) were elected to their councils or as mayors. Although a fine success, village councils are political entities without budgets, and therefore, have limited power. Hopefully, these successful women will encourage similar participation by additional women of their village but NDI should focus on assisting the election of women at influential levels, e.g. city councils and parliamentary seats. The Women’s Forum is another successful program which strives to increase women’s participation in political and civic life. The Forum consists of a broad range of women representatives, led by 13 women executives of the private, public and NGO sectors. Their national conference gained the involvement of approximately 300 women, representing a broad range geographically and professionally, including civil society leaders. Among their achievements, thus far, is their recommendation to NDI to train women candidates and support successful lobbying of parliament to amend the election code to improve women’s positions on party lists for parliamentary elections. During their national conference, women identified four key concerns for women: healthcare, political participation, economic empowerment and domestic violence. Currently, the committees are finalizing platforms per each issue, a first step before development and implementation of advocacy plans. The women board members are highly motivated and appreciative of NDI but also very dependent. A sustainability plan for this activity needs to be adopted and executed. NDI’s project also included programs that would facilitate more government outreach to opposition groups and citizens. The Presidential Secretariat requested a training program to enable more effective outreach to citizens. After two years of communication between NDI and the Secretariat, NDI was spurned by the Secretariat’s office, indicating interest in training solely at a facility in Europe. NDI changed tactics as an attempt to address this objective by working with respected media NGOs to organize programs with political party representatives of the governing coalition and opposition parties, presentations and discussions broadcast for public consumption. Although not directly addressing the objective of facilitating government outreach to opposition groups and citizens, political party representatives agree that the programs benefited their organizations and the public generally. In advance of the 2012 Parliamentary Elections, political parties will be highly motivated to gain NDI’s counsel and training. Although the needs of all parties will need to be addressed, NDI must assure support is offered to the opposition coalition and parties to make certain that citizens have choices for political party membership and as electors. Overall, we recommend that NDI continue a “bottom-up” approach to engage citizens who will demand government responsiveness, a rational assumption. NDI should continue their successful work with youth and women, within both political parties and civil society. However, NDI Evaluation Report International Business & Technical Consultants, Inc. Page iii some programs should be upgraded. For instance, a focus on women candidates should also include positions of high influence, rather than solely village councils; youth leadership training should offer pragmatic utilization of developed skills; and it is important to develop a sustainability plan for the Women’s Forum. Although one youth advocacy committee has achieved success, NDI did not accomplish nearly as much within this programming arena as projected. It is a field of service requiring more committed future focus. The stakes remain high for Armenia. Based on increased poverty, higher unemployment, greater immigration and a government perceived domestically and internationally as highly corrupt and unresponsive, it is possible that citizens will become more apathetic and disillusioned. It is important to continue investment in programs such as those led by NDI to provide citizens a sense of hope that their efforts can contribute to a more robust democratic state. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 1 I. INTRODUCTION NDI’s Development Challenge and USAID’s Approach The three-year project implemented by the National Democratic Institute for International Affairs (NDI), “Political Process Development in Armenia,” began January 9, 2009, and ended on January 9, 2012. NDI was an implementing partner, along with IFES1 , of the CEPPS (Consortium for Elections and Political Process Strengthening) program. Given the context, USAID proposed a program in which NDI would both improve the ability of citizens to effectively participate in political processes as members of political parties and help political parties develop their ability to take part in parliamentary and presidential elections scheduled for 2012 and 2013 respectively.2 International Business & Technical Consultants, Inc. (IBTCI) was selected by USAID to evaluate both NDI and IFES programs through desk and field study through separate assessments. The text of this report serves as an evaluation of NDI’s recent three-year project. As stated in NDI’s program description, the turmoil following the February 2008 presidential election resulted in an increased contentious political environment based on a highly centralized government and acrimony among the political parties. In the aftermath, citizens became more dissatisfied with their leaders and had fewer options to express political views. The concern was growing apathy and potentially political extremism. In view of the citizen disconnect with the nation’s decision-makers, NDI proposed a new approach in their CEPPS/NDI agreement. They planned a more robust bottom-up approach to political party development and civic advocacy, based on the assumption that citizens would demand government and political party responsiveness. A program goal was to develop and support a new group of “agents for change” at the local level, assuming that local issues would generate greater interest than those on the national stage. Additionally, NDI focused on improving government outreach to opposition groups and citizens, plus, increasing the effectiveness of election and government monitoring groups. In order to achieve these goals, the NDI three-year work plan had three core objectives: 1. Prom ote greater responsiveness among political and civic groups to citizens’ interests at the local and national levels by increasing the capacity of citizens to organize and engage proactively in local community issues and political processes. The two anticipated results of this objective were the following: Political parties and nongovernmental organizations (NGOs) identify and address issues of concern to citizens and develop policy and advocacy agendas. 2. Facilitate more government outreach to opposition groups and citizens. The anticipated result was the following: National government policymakers demonstrate more inclusive approaches to problem solving and policymaking. 1 The Evaluation of IFES was carried out simultaneously and reported separately under this same, IBTCI executed contract. 2 Evaluation SOW; page 2 NDI Evaluation Report International Business & Technical Consultants, Inc. Page 2 3. Assess and, if conditions permit, contribute to the integrity of elections through more effective domestic election monitoring. The anticipated result was the following: parties or civic groups practice new election monitoring methods. These three objectives and their activities constitute the core of the NDI program implemented throughout Armenia during their recent three-year program. Gegham Sarkisyan, Resident Country Director, was the staff member providing overall management responsibility for program implementation. The purpose of the evaluation for the close-out of the three-year project is to measure and analyze accomplishments, assess how effective USG resources were used and measure the sustainability of projects for beneficiaries. II. EVALUATION SCOPE AND METHODOLOGY The specific purposes of the evaluation are the following (See Annex I for Evaluation Scope of Work; Annex II for the Evaluation Work Plan, Annex III for the List of Interviewees; and Annex IV for the Interview Guides): 1. To measure and analyze the accomplishments toward achievement of the results of the activities, including an “effectiveness and efficiency assessment” that examines how successful the program has been in achieving their set targets; 2. To assess how effectively USG resources have been used; and 3. To measure the sustainability of the project results on respective beneficiaries. The findings of the evaluation will be used to inform the design and development of future projects. The evaluation identifies lessons learned; assesses the strengths and weaknesses of the program’s strategies and activities performed; and provides recommendations to USAID for project planning purposes for the next three to five years. NDI’s implementation methodology in relation to USAID’s priorities was evaluated, including notation of results achieved and vis-á-vis NDI's program objectives mentioned above, which were to improve the ability of citizens to effectively participate in political processes as members of political parties and help political parties develop their ability to take part in parliamentary and presidential elections scheduled for 2012 and 2013 respectively. The findings presented in this evaluation report are based on the following evaluation methodology: a desk review of NDI and It’s Your Choice (IYC) documents (see Annex V for a list of key reference documents) and other relevant documents (e.g. OSCE election and Freedom House reports); and interviews with individuals and groups, mainly women and youth, totaling 36 sessions and including approximately 90 people. Due to NDI’s program focus of political party decentralization, sessions were conducted in Yerevan and five additional regions (marzes), plus with representatives of a seventh marze. Key informants were members of nine political parties, seven international organizations, three domestic NGOs, members of the Women’s NDI Evaluation Report International Business & Technical Consultants, Inc. Page 3 Forum Board, a member of the government, members of the Vanadzor Youth Advocacy Project, nine groups of youth party members and three groups of women village council members. Limitations of Evaluation Methodology Evaluation limitations were minimal. For instance, some political party members were unavailable due to their travel schedules and others may have been unwilling to meet, evidenced by lack of returned phone calls. Oftentimes we were limited to connections provided by NDI, potentially skewing the interview sample. Due to winter weather and road conditions, only seven of the 10 regions were visited and few villages, reducing the number of interviews conducted outside of the capital. To maintain objectivity, with the exception of one meeting, all interviews were conducted at locations external to NDI operations and all without the presence of NDI staff members. Two meetings occurred with NDI’s Resident Director during the first two days of field interviews, but based on his unreasonable demands (e.g. obtaining a list of scheduled contacts in advance of interviews, a copy of the draft evaluation report prior to USAID’s briefing and an exit interview), the evaluation team limited communication with the local NDI office to requesting contact information. Overall, we feel strongly that those interviewed provided factual information and candid perspectives. Findings addressed in the next section are organized according to USAID’s general and specific evaluation questions per the SOW, addressed through the desk review of documents and field interviews. III. FINDINGS Background The context within which NDI has implemented its program during the last three years is relevant to their challenges and successes. Contributing to political paralysis, political parties overall remain in the early stages of development with few engaging their memberships meaningfully. They remain highly centralized with little interest to involve members in decision-making. Most are directed by one or in some instances, two to three male members. NDI has conducted programs for political parties for more than a decade to enable de-centralized organizations that utilize internal democratic practices. Sometimes leadership has refused or given lip-service to such training and coaching with the exception of the timeframe a few months in advance of an election for the purposes of GOTV (get-out-the-vote) campaigns and training members as election monitoring proxies. Opposition parties are described as relatively weak, often uninspiring and splintered. They either are in relatively early stages of development (Heritage and Free Democrats) or suffer from lack of cohesion and/or influential leadership (Armenian National Congress, including People’s Party of Armenia, Republic Party and Armenian National Movement). Some of those interviewed believe that if the Armenian National Congress (ANC) became the governing coalition, they would rule similarly to the current coalition, not advancing democracy but rather their own interests. However, the youth of these parties are active and perceived as motivated to create NDI Evaluation Report International Business & Technical Consultants, Inc. Page 4 change, and often described as courageous. We understand this ray of hope through our interviews of youth party members and the existence of NOW, the opposition coalition’s youth group. Also, citizens express frustration with lack of charismatic opposition leadership and may not understand that such leaders will surface through bottom-up political party and civil society development. The ruling coalition (Republican Party, Legal State Party and Prosperous Armenia Party) have managed to limit the space of opposition parties through exceptionally strong control by the president and parliament, including much of the country’s economic resources and businesses. In addition, they control government budgets and the media. As an example, in 2011 they used government resources for their own interests, providing special services (e.g. services of healthcare, education, transportation, etc.), perceived as pre-election manipulation. Furthermore, within Armenia’s conservative culture, women have limited influence in political and economic spheres, as well as civil society. This under-developed talent pool remains largely untapped but with the potential of becoming a major source of influence to benefit Armenia’s future. NDI’s objectives and programs address the political paralysis by providing some options for citizens to express their views and contribute to decisions that affect their lives. NDI has developed a three-year program for “party building and civic advocacy from an issue-based, bottom-up approach that is focused on developing the ‘demand’ for political responsiveness,” as stated in the CEPPS/NDI agreement, Attachment B – Program Description. A. Findings Associated with USAID General Evaluation Questions 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? NDI had three objectives: promote greater responsiveness among political and civic groups to citizens’ interests at the local and national levels by increasing the capacity of citizens to organize and engage in local community issues and political processes; facilitate more government outreach to opposition groups and citizens; and assess and, if conditions permit, contribute to the integrity of elections through more effective domestic monitoring. The following describes results per each objective. First Objective: Promote greater responsiveness among political and civic groups to citizens’ interest at the local and national levels by increasing the capacity of citizens to organize and engage in local community issues and political processes. In regard to the first objective, NDI demonstrated fine success in several arenas but with limited success in others, as revealed through their performance measurement tools (PMTs). First, their choice of population segments, women and youth, was good, as both are under-represented populations that were highly motivated to participate in training and advocacy programs. However, much of NDI’s efforts were on training which lessened focus on the practical NDI Evaluation Report International Business & Technical Consultants, Inc. Page 5 application of the training. There were only a few advocacy projects organized during the three￾year cycle that help reinforce the training. The activities that NDI implemented in support of this objective were Youth Leadership Academies, Youth Forum, Women’s Forum and Women Candidate Schools, described in more detail below. Youth Leadership Academies were implemented for approximately 200 youth representing political parties and NGOs and provided leadership training through interactive programming. Youth participants were highly supportive of their experience, stating that they acquired more pragmatic skills compared with other leadership training programs. Plus, the experience was deemed as valuable because it was an opportunity to meet and develop relationships with youth of other political parties. NDI reported generally that some NGO participants utilized skills acquired for the benefit of their NGO’s, but there is no concrete data to confirm this assertion. A Youth Forum board of 14 members representing political parties and NGOs was formed to engage youth nationally and regionally. Youth Forum members of Vanadzor demonstrate NDI’s fine success for training and coaching youth to organize and implement advocacy projects. Members acquired skills for local community development, including lobbying local officials to provide necessary resources, gaining community input for issues of concern, planning and executing development projects, recruiting volunteers, utilizing a variety of options for communication and organizing town hall meeting. Projects include clean-up of City Park, recreational events and classes for youth and citizen survey of community priorities. Another project was implemented in Vayots-Dzor and a third was recently initiated in Dilijan but the status of these activities is unknown as they were not represented in quarterly reports available through mid-2011 nor described by the Resident Director. However, the first year PMT stated that such projects would occur in seven localities. Another fine national success was the Women’s Forum that implemented a national conference involving more than 300 women to gain their insights about crucial issues faced by Armenian women. Additionally, they were a significant force to successfully lobby Parliament for Central Election Commission (CEC) legislation to improve the status of women on party lists in advance of Parliamentary Elections. Currently, they are planning additional efforts to advocate for reform in regards to four selected issues of keen interest to women: health care, gender violence, economic opportunities and political engagement. Launching an advocacy campaign was a first year expectation according to the PMT; however, this did not occur until the final year. Two executive committee members stated that the Women’s Forum could not continue without NDI’s support. During year three, NDI stated that Youth and Women Forums would conduct public education around political party positions on their issues, but forum members interviewed did not describe such programming. NDI assisted political parties to develop skills and engage in community organizing and outreach based on issues of importance to Yerevan residents. During the first year prior to the Yerevan city election, 16 parties of both the government coalition and opposition parties were trained, addressing the following topics: community organizing, volunteer mobilization, door-to-door canvasing, alternative means of communicating parties’ messages to voters, and conducting campaigns based on issues of importance to citizens. Additionally, they organized a training experience of similar topics for eight political party representatives in Warsaw at the European NDI Evaluation Report International Business & Technical Consultants, Inc. Page 6 Institute for Democracy, funded by the National Endowment for Democracy. NDI contracted with the Caucasus Research Resource Center (CRRC) to conduct focus groups about issues of interest to women and youth citizens, research results that were provided to political parties for platform development. NDI’s successful Women Candidate Schools supported their first objective and focused on local engagement. Of eighty-two (82) village women who participated in NDI’s Women Candidate Schools and also ran for office in the 2011 local elections, sixty-one (61) were elected to their councils or as mayors. Women interviewed reported acquiring skills from NDI for campaigning, including public outreach, a new experience for their villages. The three elements for training were reported as the following: creating and developing campaign themes based on their platforms, planning and conducting door-to-door canvasing and developing campaign literature. NDI continues training and coaching women elected officials to provide skills for constituent services, another important effort. Post training topics include the role of elected officials on councils, human rights violations pertaining to women, and constituent services. To further their education, NDI arranged for 10 newly elected women to participate in a study trip to Lithuania, enabling further training and an opportunity to meet their Lithuanian women counterparts. Participants were pleased with their experience, another opportunity to bolster skills. To provide political party and NGO representatives with information about issues important for Armenian youth and women citizens, NDI engaged the Caucasus Research Resource Center to conduct focus groups in Yerevan and two regions. A survey of citizens’ attitudes also was implemented during the final quarter of the three year program to support work of the political parties, as well as youth and women participants of NDI programs. During year two, NDI expected to create “democracy networks” but democracy networks were never described as a program or activity during the interviews with youth, women, political party member nor NDI representatives. However, NDI’s programs for youth and women informally enabled citizens to make connections and networks across political parties and sectors as described by participants. Overall, NDI fulfilled much of this objective as described above but a portion was unfilled, e.g. two youth advocacy projects instead of seven, delay but success for advocacy by the Women’s Forum, a lack of formalized democracy networks and public education campaigns by Youth and Women Forums. Second Objective: Facilitate more government outreach to opposition groups and citizens. NDI’s success for this objective is limited. They pursued developing strategies with the Presidential Secretariat during the first two years without success, having had only several meetings and phone calls in that timeframe. It was only at the two-year point of attempting engagement with the Presidential Secretariat that they told NDI they wanted training in another country, which NDI was not going to provide. There is no evidence that NDI attempted to work with another government entity to facilitate direct outreach to opposition groups and citizens. Because of their inability to engage with the Presidential Secretariat, they altered their strategy to create new programs, described by NDI as debates. Political party members of both governing and opposition parties made presentations about and discussed policy issues. According to NDI NDI Evaluation Report International Business & Technical Consultants, Inc. Page 7 reports, these programs received media coverage. Additionally, these programs were implemented in partnership with the National Press Club and the Armenian Center for Political International Studies NGO and were positively characterized by interviewed political party members because they enabled their parties to educate citizens about their perspectives on issues. However, the programs did not directly involve government outreach to citizens and opposition groups, as originally planned but was limited to both government coalition and opposition parties making presentations about political issues. Third Objective: Assess and, if conditions permit, contribute to the integrity of elections through more effective domestic election monitoring. NDI’s anticipated result for this objective was that parties or civic groups practice new election monitoring methods. Much of their work focused on It’s Your Choice (IYC), a national election and government monitoring group. NDI assisted IYC to use new methods of internal and external communication and for volunteer recruitment. Results recorded by NDI were the following: IYC created and disseminated a newsletter for members and interested stakeholders but the newsletter was discontinued due to lack of funds; IYC updated their website; IYC solicited monthly reports from each marze (region) organization; IYC recruited new members through their school mock election program and website; IYC improved questionnaires for monitors; and IYC installed software to improve the efficiency of gaining voting results from each voting center on election day through SMS communication. Based on reporting and interviews, IYC developed improved strategies for internal communication with their volunteer force of several thousand participants and through their Mock Elections civic education program in schools as they recruited additional volunteers. Additionally, IYC reported that the improvement of their questionnaires for monitors and utilization of software was to increase reporting efficiency and timing through use of SMS communication. However, IYC’s performance as a national monitoring group continues to be controversial, characterized by those interviewed as minimally effective as election monitors. Also in support of this objective, NDI trained members of three political parties as Election Day proxies, totaling 28 trainings with 35 to 40 participants per session. 2. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? Sustainability is limited for most NDI programs and institutions. The Youth Forum in Vanadzor may be sustainable within their community, because they continue to advocate on various issues with local officials and organize successful community projects. The other two referenced Youth Forum groups are in the early stages of development and have not yet demonstrated project success or sustainability. Expansion of these programs could possibly lead to sustainability if the youth leaders of Vanadzor became trainers of trainers (TOTs) for their peers in other communities. Women trained and elected through NDI’s Women Candidate Schools can probably independently continue serving constituents and conduct future campaigns successfully. However, program sustainability would require additional effort, e.g. the current cadre of women becoming TOTs, recruiting and training other women candidates for future elections. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 8 There is little evidence of sustainability for the Youth and Women Leadership Academies. Participants interviewed described an informal process of communicating ideas acquired during training to other youth political party members but not a formalized training process. However, some youth participants reported training members of their NGOs. Training the graduates as trainers would boost the possibility of sustainability. The Women’s Forum is highly dependent on NDI resources and probably could not continue without their support, as noted by members. Similarly, the political “debates” through NDI’s partnership with the National Press Club and Armenian Center for Political International Studies probably are not sustainable without resources from NDI or another donor. 3. What are the main achievements of the program? • NDI created a cadre of Vanadzor youth leaders who successfully planned and implemented advocacy and community development projects. Evidence suggests that as a local committee, their work is sustainable. • Of eighty-two (82) village women who participated in NDI’s Women Candidate Schools and also ran for office in the 2011 local elections, sixty-one (61) were elected to their councils or as mayors. • Youth across political parties developed relationships through the Youth Leadership Academies and gained pragmatic leadership skills. • The Women’s Forum that successfully advocated for election code changes and is posed to advocate in other key areas. • Representatives of governing and opposition parties publicly presented views about issues for public consumption through media coverage. 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? Generally, Armenian citizens are discouraged about their futures with poverty increasing and substantial migration to other countries for employment. Also, their powerful central government is perceived as intractable and uninterested in the welfare of the majority of its citizens. Based on these factors, NDI wisely chose a bottom-up intervention approach as a programming strategy, working to gain citizen involvement to pursue influence nationally and regionally. With a focus on women and youth, sectors both under-represented in political processes and highly motivated to create change, NDI’s programs successfully recruited, motivated and trained youth and women citizens with skills to advocate for change. However, true advocacy that addresses citizens’ priorities was limited to a few projects, one locally and nationally, an area for further exploration. Additionally, NDI was unable to engage the central government as a consultant to develop outreach strategies for opposition groups. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 9 5. How did the implementer perform in terms of project management and how effective was the project leadership? The project was well managed in terms of providing training, evidenced in part by the high number of women, youth, and political party and NGO representatives who received the training on leadership and campaigning skills, as well as their overall characterization that their NDI training experiences were highly valuable. In regards to implementing successful advocacy projects as planned, the program was much less successful. This may be attributed to how the overall training program was managed. In other words, it appears that program management focused more on providing the training to the largest number of participants possible rather than giving adequate attention to ensuring that the training was being internalized by the participants so that they could apply what they learned after the training. Also, NDI was unable to facilitate direct central government outreach to opposition groups and citizens (their second program objective) because they could not gain traction through either the Presidential Secretariat or an alternative government entity. Another challenge which was inadequately addressed relates to their third objective: “assess, and if conditions permit, contribute to the integrity of elections through more effective election monitoring.” NDI chose to address this objective largely through consulting services for IYC, the national election monitoring NGO. However, according to the records and interviews, IYC did not change dynamically and continues to be either unknown or garner limited respect among political and civil society leaders interviewed. 6. What lessons learned can be provided for future USAID programming in this area? One positive lesson learned was that in a country with a strong central government such as Armenia, it is necessary to engage a wide range of participants directly, rather than seeking to influence participants via the central government itself. This was illustrated by NDI’s original attempts to encourage the central government to develop policies and laws to benefit a majority of citizens. When it became apparent that this method was unsuccessful, NDI transitioned towards broad-based programming by generating engagement of less represented population segments such as women and youth, as these groups have proven to be highly motivated and open to training and coaching as citizen advocates. Training for skills development needs to be followed by practical options to utilize the skills and internalize learning for Youth Leadership Academy participants. Advocacy and community development projects could provide such experience and fulfill NDI’s objective for advocacy programs. Enabling youth to represent political parties in their interactions through programming and thereby develop relationships may have long lasting benefits for Armenia. This is a lesson that was applied by NDI should be replicated. Specific plans for sustainability should be built into work plans. NDI has created new organizations, e.g. Women’s Forum, Youth Forum, Youth Leadership Academies, Women Candidate Schools, that cannot continue without resources such as those provided by NDI. Additionally, to increase the scale of those served, work plans with training programs should include TOT programs to assure maximum reach to many citizens and embolden those selected as TOT participants with advanced skills. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 10 Because NDI’s PMTs were not well defined after the first year of program implementation, it is difficult to discern with certainty what exactly their program successes are. If NDI’s PMTs were better defined and monitored, identifying these successes would have been easier. Additionally, NDI’s quarterly program reports did not directly and consistently address the indicators in the PMTs to measure success. For instance, during the first year, NDI planned seven advocacy projects but only one was implemented. The lesson learned is that if USAID is concerned with success, it may need to clearly state expectations for detailed plans and monitor progress more closely. 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? NDI’s focus on youth and women sectors has proven to be a valuable strategy and should be continued, while also supporting political party development. For the sake of efficiency and cost-effectiveness, instead of increasing the number of trained youth and women in their Youth Leadership Academies and Women Candidate Training Schools, NDI should focus on TOT programs. This can substantially increase participant numbers while using fewer central office staff members and other resources. Since training for the sake of training yields minimal long-term results, the graduates should practice skills acquired through advocacy efforts and implementation of community projects. This could increase substantially the number of advocacy projects that influence government and create a sense of success, while assuring skill internalization. The Women Candidate Training Schools should continue because it is a very successful program. However, rather than focus on village councils, in the future NDI should focus on assisting women run for offices (e.g. city councils and parliament) that can influence government meaningfully, unlike most village councils. Village councils have limited power since they do not have budgets or their own resources. The Women’s Forum should be continued for a set timeframe and should require the executive committee to create a sustainability plan, including acquisition of additional donors or other means for self-sufficiency, and lessen NDI’s burden and the Forum’s dependency on NDI. NDI should have attempted services with another governmental office or agency after being rebuffed by the Presidential Secretariat. Their political party representative “debate” programs substituted for consultation with the Presidential Secretariat did not fulfill this objective. Investment in focus group research and opinion surveys should be re-evaluated. Such projects may be too costly compared to their benefit. When interviewing political party members and other citizen activists, none were aware of the 2010 focus group research results. This lack of acknowledgement of focus group results makes their investment seem questionable. NDI has trained IYC for years but their organization remains highly centralized and perceived as marginally successful as a force that demands free and fair elections. NDI’s role with IYC should be re-evaluated. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 11 8. Did the agreement provide clear and achievable results against which progress and impact could be measured? The agreement generally provided clear results for the first year, albeit some PMTs were not easily measured. This was limited for subsequent years. When questioned about lack of PMTs in the work plan and limited mention of PMTs in quarterly reports, the Resident Director stated that he understood that they were not required by USAID. 9. Analyze attribution of project successes to USAID involvement. USAID’s investment in NDI programming has resulted in several concrete benefits: 61 women candidates were elected of 82 who were trained and also ran for office; 200 youth were trained on leadership skills and they received the unique opportunity to develop cross-party relationships; political party representatives presented their views on issues that were covered by the media for political party and the public’s benefit; successful advocacy projects were implemented by the Vanadzor committee of the Youth Forum and Women’s Forum and will likely continue; and political party members were trained as election monitoring proxies and on citizen outreach. 10. Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. As mentioned previously, NDI needs to focus more on TOT programs to broaden the number of trained and involved citizens since successful advocacy requires a critical mass of citizens to influence a powerful centralized government. Also, to assure that skills taught are internalized, NDI needs to coach their training participants to use their skills implement actual programs, such as advocacy or community development projects. Promotion of cross borders programs could be beneficial for women and youth leaders to acquire insights about challenges and successful strategies to influence similar powerful centralized governments. Development of an organization of women political party leaders could be beneficial, providing a forum to exchange experiences and develop strategies to increase influence within their respective political parties. B. Findings in Response to USAID Evaluation Questions Specific to NDI’s Work USAID Evaluation Question #1: Have NDI youth and women’s Leadership Training Academies, NDI’s Women’s Candidate Schools and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party membership and activities, political engagements, political public gatherings, etc.) among youth and women? The following describes results specific to the three programs referenced in the question: Youth Leadership Academies, Women’s Candidate Schools and Women’s Forum. Youth Leadership Academies Participants of NDI’s three Youth Leadership Training Academies demonstrated substantial enthusiasm about their NDI experience, as a venue for interaction among political party members NDI Evaluation Report International Business & Technical Consultants, Inc. Page 12 and acquire practical leadership skills. Approximately 200 young men and women have participated in the training academies since program inception. The leadership training program involves a mix of young men and women political party members and NGOs, as described by NDI’s Resident Director, a 60-40 split of party and NGO members. Launched during the first year, the eight-month program consists of three groups, one of young women and two of mixed gender youth, with approximately 40 members per group. Topics covered include: Civil Society & Democracy: Civic Activism, The Political System and Elections; Human Rights in Armenia; Leadership; Public Speaking; Armenia’s Integration into International Institutions; Conflict & Negotiation; Political Management; Communication Strategies; Information Security; Campaign Organizing; Election Code and Voter Education; Risk Assessment and Management; Coalition Building; and Time Management and Project Planning. Presentations were made by experts and training included interactive elements. Among the 30 youth members of political parties interviewed, all were highly appreciative of NDI’s leadership program. The concrete value of training was reported as the following: tools for interfacing with mass media, making presentations, organizing and leading, tools for assessing human rights violations, learning to respect opinions of others and negotiating. Four women were interviewed who participated in the women-only group. All indicated that they were unaware of the reasons they were chosen for the group and if given a choice, prefer training experience with both genders. We observed one hour of a four-hour seminar for 40 youth members who generally consist of equal numbers of men and women whom, apart from only two members, are members of political parties. Most participants seemed very well engaged and participated in the topic’s discussion about human rights issues, specifically debating about Armenia’s election code versus implementation, double standards posed by European monitoring organizations, etc. In regard to increased social and political involvement, the results were mixed. Very positively, participants of all nine youth groups interviewed indicated that they gained connections to youth members across parties, relationships that they believe will be valuable in the future. According to one youth member, due to experiences during the leadership seminars youth members of various parties were meeting on their own which is judged as a fine outcome. NDI was well regarded as trainers, providing practical skills that other international youth leadership seminars were wont to deliver. Although the above described increased social engagement is laudable, there was no evidence that involvement resulted in increased political involvement, such as increased party membership, political engagement and public gatherings. Women Candidate Schools Women Candidate Schools were exceptionally successful for village candidates. Of eighty-two (82) village women who participated in NDI’s Women Candidate Schools and also ran for office in the 2011 local elections, sixty-one (61) were elected to their councils or as mayors. All interviewed expressed keen appreciation of NDI’s training program. Of the three groups of women interviewed who represented three marzes and the candidates listed in quarterly reports, all were elected to village councils as council members or mayors. Although this is overall remarkable within the context of Armenia’s conservative culture, unfortunately these councils NDI Evaluation Report International Business & Technical Consultants, Inc. Page 13 have little power due to their lack of budgets, and the control exerted over the councils by marze (regional) governments led by governors who are appointed by the country’s president. Therefore, village councils are considerably less competitive than other councils and, of course, parliamentary positions. The majority of the women appreciated the skills they developed for canvassing, developing campaign platforms and writing materials for public outreach (e.g. campaign brochures). One candidate stated that in her village male candidates traditionally conduct campaign events on the village square, which is not considered an acceptable campaign option for women candidates and the reason canvassing was an essential and successful means for her campaign. It was interesting to note that most women trained by NDI were members of political parties, but during our interviews, all stated that they did not receive campaign support from their parties but only from NDI. This is an additional example of the lack of political party outreach regionally and to women. NDI has conducted post-election follow-on coaching sessions for those elected to assure success as public servants. These services are greatly appreciated, e.g. methods and options for constituent services. Also, in October 2011, some participants attended training in Lithuania, a chance to bolster their skills for constituent services and become acquainted with Lithuanian women elected officials. Of those interviewed, all indicated the trip was very worthwhile. While NDI’s candidate schools are an impressive beginning for village women to gain political traction, in order to gain meaningful power they must be elected and appointed to positions on city councils and other political bodies. Women’s Forum NDI created a Women’s Forum organization led by a 13-member board of directors consisting of highly successful women executives representing NGOs, political parties and government. The purpose is to increase women’s effectiveness and opportunities for influence in Armenia’s political affairs. The chairwoman stated that similar efforts to organize women leaders have occurred previously, as she recalled, a similar group organized about five years or so ago with funding from European sources. Support was lost and the group disbanded. It was quite evident during the meeting with the board and individual meetings with five board members that NDI’s support is well appreciated but also crucial for continuation. NDI and the Women’s Forum executive committee organized in May 2011 the National Conference of Women in Politics, as planned, the first of an annual event. It was perceived as highly successful, engaging approximately 300 women across Armenian sectors and regions. Speakers represented other countries, providing insights about experiences and advice for becoming more influential in Armenian affairs. Based on conference deliberations, they organized four discussion groups representing top priorities: healthcare, economic empowerment, political participation and gender-based violence. As of December 2011, each group was finalizing their issue platforms that will be made public. The next step will be development of advocacy efforts. NDI is providing coaching for each stage of the Forum’s work. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 14 Also noted, the Forum members assisted NGOs and other interested groups to successfully lobby parliament to gain support for women on political party lists for parliamentary elections. The new election code improves their positions on party lists to every sixth position rather than tenth. NDI’s support includes administrative assistance, meeting space, communication support and advice for advancing their development and advocacy efforts. The Forum’s immediate future does not appear sustainable without this substantial assistance from NDI or a similar organization. There is no evidence of a self-funding plan. USAID Evaluation Question #2: Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political policy activities. None of the youth political party or NGO participants of the Youth Leadership Academies interviewed described efforts as of yet to launch community advocacy projects or other activities to achieve political or policy goals. Beyond acquiring leadership skills, they have benefited from opportunities to meet and listen to their peers of other parties and NGOs. All of those interviewed placed great value on the experience to acquire skills and develop relationships with members of other parties which potentially has long-term value. NDI reported that a few NGO participants have used skills acquired for the benefit of their organizations but specifics are limited. Although graduates of the youth leadership program have not developed community advocacy projects, a Youth Forum group in Vanadzor has benefited keenly from NDI’s advocacy training and coaching. Initiated about one year ago, they have successfully completed several community projects that included gaining assistance from local government representatives, as well as recruiting and organizing as many as 170 youth volunteers. The six representatives interviewed were highly motivated describing ongoing projects: increasing healthcare services for needy community members, City Park cleaning, canvassing citizens to gauge concerns and ideas for projects, recreational events and classes for youth. The Vanadzor youth advocacy group efforts are on-going and they continue to benefit from NDI’s coaching efforts. This successful youth advocacy project partially fulfills NDI’s first program objective of increasing the capacity of citizens to organize and engage proactively. NDI reported that two other Youth Forum committees are being organized in Vayots Dzor and Dilijan but as of December 2011, they had not yet organized advocacy or community development projects. USAID Evaluation Question #3: To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations? Nearly all youth leadership participants who were interviewed indicated that lessons learned from leadership training were discussed with fellow youth political party members during youth committee meetings and during informal sessions, but none of them described nor was there evidence of formalized TOT programs. USAID Evaluation Question #4: Have NDI/Yerevan Press Club debates had value for participating political parties? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 15 We met with both the directors of the National Press Club of Yerevan and Armenian Center for Political International Studies, an NGO. The latter director has dual roles, directing his NGO and Radio Free Europe, although stating firmly that Radio Free Europe has not had and cannot have direct involvement with NDI for political reasons. The program consists of presentations about political issues made by political party representatives. Both government coalition and opposition parties have participated. NDI reported coverage by H1 Public TV, AR TV, Yerevan TV, ALM, Yerkir Media, Kentron TV and Public Radio. National Press Club’s formerly monthly and more recently semi-monthly “debates,” a term coined by NDI, focused on general political issues while the NGO’s four radio “debates” focus on election topics. We were told by the National Press Club director that due to the run-up for next year’s Parliamentary Elections, topics for discussion may change so that potentially all programs will be devoted to election issues. Both directors perceive value in bringing representatives of political parties together to make presentations and discuss issues for public audiences, an exceptional forum in Armenia, and appreciate NDI’s support. NDI confers with the directors about topics and assists to recruit political party representatives. NDI directly supports the National Press Club with funding to cover costs for each program (95,000 drams). Political parties that participated were enthusiastic about the programming, opportunities to publicly contrast policy stands. One party member also stated that the programs were one of the few opportunities for the public to listen to differences in party viewpoints since debates in parliament are not broadcast. We are uncertain about the value to the public, potentially an area of study for NDI. USAID Question #5: What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? The following are four key obstacles for Armenia’s democratic development that were described by the interviewees during the field study and from the documents reviewed. NDI’s programs address some of the obstacles directly and others indirectly, acknowledging such challenges in their work plan. NDI’s project activities that address the obstacles are described below. 1. Armenia’s opposition coalition and parties are perceived as weak and, thus far, unable to truly challenge the governing parties’ coalition. This is an obstacle to full participation by citizens because successful democracies require citizens to truly have options as voters and political party members or serious apathy may result. NDI provided direct training and coaching to a range of political parties, including those of the opposition. Unfortunately, often the leadership of parties has very limited interest in the development of democratic organizations that truly involve members in decision-making. NDI addressed this problem through a “bottom-up” approach by offering, for example, training and coaching to youth political party members through their Youth Leadership Academies. As a result, the project aims to increase the influence of youth within their parties, encouraging skills development for more effective party organizing. It is recommended that the Youth Leadership Academies continue and involve more youth NDI Evaluation Report International Business & Technical Consultants, Inc. Page 16 leaders, with additional skills for projects that can serve a board base of party members, e.g. development of a formal TOT program for other youth party members based on leadership skills currently taught and an election monitoring TOT program for party members who will serve as election proxies. As mentioned by NDI’s Resident Director, youth committee members could serve as the influential committee to promote agreement by all parties for an election code of conduct. During an interview with the Resident Director about the Armenian National Congress (ANC), a coalition of opposition political parties, he questioned/expressed doubt about the ANC’s current motivation to remain an opposition force. His stated uncertainty was based on rumors that they “were making deals” with the government in advance of the 2012 Parliamentary Elections and that currently, they were not communicating with other opposition parties. However, others interviewed reported that ANC was a strong and legitimate opposition group. 2. Women have limited influence in their culturally conservative environment. In spite of a heritage of more meaningful involvement during their Communist past, the present cultural environment limits participation of woman and women leaders. This obstacle prevents women from being perceived as serious actors within their society, addressing with influence issues of concern and creating positive change within political and civil society spheres. NDI’s programs successfully address women’s lack of influence within their society by increasing options for participation. Both the Women’s Candidate Schools and Women’s Forum are projects that aim to mitigate women’s relatively limited power. The Women’s Candidate Schools trained women village council and mayoral candidates of whom 82 also ran for office. Of the group of trained women who ran for office, 61 were elected, a 74% success rate. Although highly successful as far as election success, the village councils have very limited power; therefore, NDI’s future training needs to focus on women candidates for city councils and parliamentary seats. The Women’s Forum is addressing concerns of women, currently preparing to develop advocacy campaigns for their four chosen issues: healthcare, domestic violence, economic empowerment and political participation. They organized a highly regarded national conference, attended by approximately 300 women from across Armenia, resulting in agreement about four issues. Another conference is planned for next year. In collaboration with a broad range of citizens, including representatives of NGOs and political parties, the Women’s Forum successfully lobbied parliament for an improved election code, increasing women’s position on political party lists, a major achievement. The Women’s Forum Board of 13 members, interviews conducted with the entire group and individually with five members, expressed great appreciation to NDI for resources and consulting support. A major challenge is sustainability of the working group if NDI’s sponsorship is minimized. In addition to the two programs described above, women are equally represented in NDI’s Youth Leadership Academies and the Vanadzor Youth Forum group. 3. Domination by the governing coalition is another obstacle, because it is perceived as highly and increasingly corrupt. Unemployment and poverty are driving migration to Russia and other countries. Citizens are demoralized and have become apathetic, choosing to disengage in their country’s and communities’ futures, negatively impacting their democracy. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 17 This obstacle is substantial and multi-faceted. NDI addresses this concern overall with their “bottom-up” approach, supporting efforts by citizens at local levels where positive change is more possible. One example is the Vanadzor Youth Forum group that is successfully working with their local government and volunteers to enable community development. Through NDI training and coaching, they have acquired skills for public outreach to ascertain issues of importance, create advocacy plans, communicate and negotiate with elected officials, and recruit and organize volunteers to successfully implement community development projects. It appears that the group requires little support from NDI after completing several successful projects. This local engagement may provide a worthwhile model for citizens, demonstrating their ability to impact local government and improve their communities. NDI’s work plan indicated an intention to develop similar advocacy projects to be implemented by Youth Leadership Academy members but as of yet, not realized. Coaching women village council members about their roles and options as public servants is another NDI program that encourages effective citizen involvement at the local level. NDI sponsors programs with two media NGOs, described as debates. Political party members present and discuss perspectives on issues. The programs, covered by the media, serve to inform the electorate while providing increased transparency about the views of political parties. Such informative experiences may mitigate a sense of apathy. 4. Generally citizens do not believe that previous elections were conducted in a free and fair manner nor do they express optimism for significant improvements in future elections. This creates apathy, reducing citizen involvement in elections that feel rigged. For more than a decade, NDI and USAID have supported It’s Your Choice (IYC), a national election monitoring NGO, with resources and consulting services. During field interviews, we found that either the organization was unknown or generated limited respect as an effective election monitoring organization. IV. CONCLUSIONS NDI’s overall strategy to create programs that encourage citizen involvement in democratic processes, a “bottom-up” approach, succeeded well overall. Due to an extremely centralized and powerful government that is perceived as self-serving, corrupt, disconnected from citizens and uninterested in encouraging participatory democracy, NDI’s approach to create “citizen demand” was the right approach. They motivated and trained young men and women as well as established women as “agents of change.” However, in some cases, success was limited or has yet to be met. A. Conclusions Related to USAID’s General Evaluation Questions: NDI Evaluation Report International Business & Technical Consultants, Inc. Page 18 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? NDI had three objectives: Promote greater responsiveness among political and civic groups to citizens’ interests at the local and national levels by increasing the capacity of citizens to organize and engage in local community issues and political processes; facilitate more government outreach to opposition groups and citizens; and assess and, if conditions permit, contribute to the integrity of elections through more effective domestic monitoring. First Objective: Promote greater responsiveness among political and civic groups to citizens’ interest at the local and national levels by increasing the capacity of citizens to organize and engage in local community issues and political processes Overall, NDI was largely successful in fulfilling this objective through the creation of the Women’s Forum, Youth Forum, Vanadzor youth project, Youth Leadership Academies and Women’s Candidate Schools. There were a few exceptions that demonstrate less success compared to the original activities intended to achieve this objective: one youth advocacy project was implemented instead of seven; despite the two year delay, the Women’s Forum was an advocacy but success; no formalized democracy networks nor public education campaigns were conducted by Youth and Women Forums. Second Objective: Facilitate more government outreach to opposition groups and citizens. NDI was not able to fulfill this objective. They initiated communication with the Presidential Secretariat who apparently during the first year of programming expressed interest in receiving technical assistance on citizen outreach. After approximately two unsuccessful years of attempting to establish more concretely the relationship, NDI realized it was being rebuffed. The resident director stated that the Presidential Secretariat finally indicated only an interest in training within an international setting, an experience that NDI did not offer. There is no evidence that NDI attempted to engage another government entity that would be interested in conducting outreach to opposition groups, which should have been explored to achieve this objective. As an alternative, NDI developed “debates” in partnership with two NGOs, involving political party representatives of both government coalition and opposition parties, although valuable, they do not satisfy this objective. Third Objective: Assess and, if conditions permit, contribute to the integrity of elections through more effective domestic election monitoring NDI’s anticipated result for this objective was that parties or civic groups practice new election monitoring methods. NDI continued to provide consulting services to IYC, the national election monitoring group. Topics focused on improved election monitoring and organization effectiveness. Through many interviews, IYC was not evaluated as contributing significantly to the integrity of the elections. Many political party members were trained also to monitor elections but it was not possible to evaluate if their effectiveness was increased since no measurement system was in place. This objective was fulfilled minimally. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 19 2. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? Generally, the programs and institutions in Armenia will not be sustainable without continued significant support from NDI, e.g. Women’s Forum, Youth Leadership Academies, national press programs that involve political party representatives and Women Candidate Schools. However, the Youth Forum members of Vanadzor appear to be functioning well and generally independently. Skill development probably has been internalized for Youth Forum members of Vanadzor, as well as participants of the Women Candidate Schools, those who were elected and continue to provide constituent services. Linkages among youth across political parties may be lasting, those who participated in Youth Leadership Academies. 3. What are the main achievements of the program? NDI had many valuable achievements: Youth Forum of Vanadzor successfully implementing community projects that required local government cooperation; 61 women were elected to village councils after receiving training; 200 youth political party and NGO members trained with leadership skills and developing cross-party relationships as Youth Leadership Academy participants; organizing Women’s Forum that successfully advocated for an improved election code and implemented a national women’s conference that addresses issues of concerns; and enabling a public forum for political party representatives to present their views on political issues. 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? NDI’s theory of development was sound, creating a bottom-up approach for citizen engagement due to the powerful intractable central government. With a focus on women and youth, sectors both under-represented in political processes and highly motivated to create change, NDI’s programs successfully recruited, motivated and trained youth and women citizens with skills to advocate for change. However, true advocacy was limited to a few projects, one locally and nationally. 5. How did the implementer perform in terms of project management and how effective was the project leadership? NDI had many fine achievements which are a testament to the resident director’s leadership and management, especially as a training institution. However, some objectives were not met which could be due to his lack of foresight in maximizing on opportunities as they surfaced during the three-year program. For instance, many youth citizens were trained in groups but none created TOT programs for their political parties or NGOs. This was a missed opportunity for NDI in terms of the value that the participants would take away from the training, i.e. maximizing what they learned by applying the new skills to projects or advocacy campaigns. This would have also broadened the reach of the program because these activities would have included more participants than those who attended the initial trainings. Another missed opportunity was training women candidates for only village councils as opposed to higher level offices such as city councils, elected offices that have much greater influence than those of villages but are more competitive and therefore, more difficult for women candidate success. In sum, program NDI Evaluation Report International Business & Technical Consultants, Inc. Page 20 leadership could have planned and implemented programs with a longer term impact, such as TOT programs for youth and women, more advocacy projects and a focus on city councils for women candidates rather than village councils. 6. What lessons learned can be provided for future USAID programming in this area? Choosing youth and women as a focus for programming was a key positive factor for NDI’s success and potentially Armenia’s future. Good training programs were developed for 200 youth participants through the Youth Leadership Academies. However, the program needs to be bolstered with practical opportunities to internalize learning, such as advocacy and community development projects. TOT programs need to be implemented to reach more youth, as well as women, for all NDI training programs. Sustainability needs to be a focus of all planning and included in work plans. To objectively and effectively evaluate progress, PMTs need further development and monitoring. 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? It is difficult to evaluate cost-effectiveness since program budgets were not made available, a USAID decision. However, efficiency was addressed. The focus on youth and women population segments should continue, while supporting political party development. Training needs to be a first step for the Youth Leadership Academy participants, followed by practical projects that can assure internalization of skills. Women candidate training and coaching should continue but with a focus on offices that have clear influence, e.g. city councils and parliament. The sustainability of Women’s Forum, utilization of public opinion surveys and focus groups, and NDI’s consulting relationship with IYC should be evaluated by NDI and USAID, programs which may need to be scaled back or abandoned. 8. Did the agreement provide clear and achievable results against which progress and impact could be measured? The agreement generally provided clear results for the first year (although some PMTs were not easily measured) but was limited for subsequent years. 9. Analyze attribution of project successes to USAID involvement. NDI had many valuable achievements that can be attributed to USAID’s support, including the following. Youth Forum of Vanadzor successfully implemented community projects that required local government cooperation. Of eighty-two (82) women candidates trained and who ran for office, sixty-one (61) subsequently were elected to village councils. Two hundred (200) youth political party and NGO members were trained with leadership skills and developed cross￾party relationships as Youth Leadership Academy participants. The Women’s Forum was organized, successfully advocating for an improved election code and implementing a national women’s conference that addresses issues of concerns. A public forum was instituted to enable political party representatives to present their views on political issues. 10. Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 21 NDI needs to develop TOT programs to broaden the reach of their training programs; leadership training needs to be supplemented with projects that allow skills to be internalized; cross￾political party program of women leaders could be developed; and regional programming with youth and women population segments should be considered for skill development and regional political-social advancement. B. Conclusions Related to USAID Evaluation Questions Specific to NDI’s Programs: USAID Evaluation Question #1: Have NDI youth and women’s Leadership Training Academies, NDI’s Women’s Candidate Schools and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party membership and activities, political engagements, political public gatherings, etc.) among youth and women? The following describes conclusions specific to the three programs referenced in the question above. Youth Leadership Academies The NDI Leadership Training Academies successfully resulted in increased social and political involvement in at least one significant way, linking young party members across party lines. Approximately 200 young men and women have participated in the academy training, the majority representing political parties. Through interviews, participants stated the key benefit is the interaction among political party members, a means of developing relationships and understanding policy perspectives of other parties. One Free Democrats party member stated that due to their NDI experience, youth members of their party are meeting socially with their peers of other parties, a very positive benefit. However, we did not surface facts about increased party memberships, political engagements, public gatherings, etc. Women Candidate Schools Women Candidate Schools also resulted in increased social and political involvement at the village level and among villages. During their campaigns and subsequent constituent services, the elected village council members and mayors were engaged in public gatherings and door-to￾door canvassing. Several interviewed claimed that more women will be interested in running for council seats due to their successful campaigns and elections. Women’s Forum The Women’s Forum, the organizing body for the National Conference of Women, is an effective advocacy organization. Their work has and is poised to increase social and political involvement among women representing a broad range both geographically and across social sectors. With other civic activists, they successfully lobbied Parliament for a new election code that improves the standing of women candidates on political party lists. Currently, they are developing advocacy campaigns based on the four areas of concern that surfaced through working groups during this year’s National Conference of Women. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 22 USAID Evaluation Question #2: Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political policy activities. Although highly popular among participants, there is no evidence yet that the Youth Leadership Academies participants have developed social and political projects or achieved policy goals. However, a youth group in Vanadzor, a committee of the Youth Forum, has planned and implemented successful community projects that have engaged local governments. Two other Youth Forum committees were formed, one each in Vayots Dzor and Dilijan to organize similar local community advocacy. USAID Evaluation Question #3: To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations? Most of the Youth Leadership Academy participants interviewed indicated that they informally discussed lessons learned with fellow youth political party members. However, none described a formalized training program which suggests this was not an NDI program goal. USAID Evaluation Question #4: Have NDI/Yerevan Press Club public debates had value for participating political parties? The National Press Club and Armenian Center for Political International Studies conducted programs, described as “debates” by NDI, that provide value for participating political parties, opportunities for open discussion that are made available to the public through media. This NDI program brings together members of governing and opposition parties for presentations and discussion about political and social issues, a chance to contrast their views among their parties in public and for public consumption. Political party members support continuation of this program. USAID Evaluation Question #5: What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? How successfully? NDI created programming that addressed key obstacles endemic to their society, including an intractable government coalition that prevents significant influence by the opposition and citizens at large. NDI’s decision to develop programming through a “bottom-up” approach, with emphasis on women and youth, is an effective overall strategy. Both women and youth participants of NDI programs expressed keen motivation to acquire skills that will lead to successful advocacy of local and national governments. However, two major obstacles need to be addressed more effectively. First, NDI’s second objective was generally unfilled, facilitating government outreach to opposition groups. They communicated with the Presidential Secretariat for approximately two years as an attempt to engage their office by providing consulting services for citizen and opposition groups outreach but the office never committed participation. NDI could have pursued another government entity but instead conducted multi-party programs, presentations NDI Evaluation Report International Business & Technical Consultants, Inc. Page 23 and discussions by both pro-government and opposition political parties for public consumption. Although a worthy program, it does not fulfill the objective. Secondly, related to their third objective, the public overall does not perceive that elections are free and fair but as fraudulent and corrupt. With NDI’s counsel and support, IYC has led efforts to monitor elections across the nation for more than a decade. Yet the results of their monitoring based on interviews are either unknown or not well respected. Additionally, NDI trained political party members as election monitoring proxies in advance of the Yerevan municipal elections but the result of this effort related to the election was not communicated. V. SUCCESSES • NDI created a cadre of Vanadzor youth leaders to independently continue local advocacy and organize community good-works projects. • Of eighty-two (82) women candidates who were trained and ran for office, 61 were elected as local village council members and mayors. • Two hundred youth had the opportunity to acquire pragmatic leadership skills through participation in Youth Leadership Academies. Participants also benefited though development of relationships across political parties. • The Women’s Forum successfully advocated for election code changes and is posed to advocate in other key areas. • Representatives of governing and opposition parties publicly presented views about issues for public consumption through media coverage. VI. LESSONS LEARNED NDI very wisely chose creation of a dynamic bottom-up citizen participation program based on the ongoing challenges posed by their highly centralized powerful government that has not demonstrated motivation to serve the greater population. Other institutions within Armenia and other countries in early stages of democratic development with similar challenges can model their objectives based on NDI’s efforts focused on youth and women citizens. A concern and potential lesson for NDI Armenia relates to training programs. The Youth Leadership Academies trained 200 participants with skills that they want and need as leaders. However, in addition to seeking expansion of the popular program for more participants, NDI needs to take this program a step further. Options should be created for graduates to enable internalization of skills through practical experiences, e.g. development of advocacy projects, community development projects. NDI Armenia could increase the value and reach of their successful training programs for youth and women across the country through the development of TOT programs, a formalized program NDI Evaluation Report International Business & Technical Consultants, Inc. Page 24 of training and coaching peer trainers. Those participating also would advance their skills as trainers. Another potential lesson for NDI Armenia relates to their second objective, “facilitate more government outreach to opposition groups and citizens.” NDI tried for approximately two years to engage the Presidential Secretariat that initially indicated interest for development of outreach strategies to engage citizens but finally declined consulting services. After a few months, when it was clear that nothing would come of the attempted engagement, NDI could have offered services to other government organizations or entities that would have helped achieve this objective. Their “debate” program in partnership with National Press Club and Armenian Center for Political International Studies, although of value, does not equate with direct central government involvement to fulfill this objective. Another lesson learned is that sustainability is an issue across all NDI Armenia projects and should be addressed at the beginning of program implementation. It is expected that some trained citizens, e.g. women elected officials and youth in Vanadzor, will continue their work but the institutions NDI is creating, (e.g. Women’s Forum, Youth Leadership Academies and Youth Forums) currently are not sustainable without NDI resources. NDI Armenia implemented a successful project for women political party leaders, creating a coalition that investigated options to increase women’s power as decision-makers within their parties. Although representing highly competitive political parties, the women immediately realized that they shared challenges within their parties, essentially a lack of true influence with the male leaders. Together they sought and practiced strategies to gain more power within their parties while developing relationships based on cross-party cooperation. Such a program might be successful in Armenia due to the keen motivation of women leaders to impact political institutions and their current limited influence. PMTs were outlined with fair clarity for the first year but much less so for subsequent years. Plus, NDI’s quarterly reports did not directly and consistently address the indicators. For instance, during the first year, NDI planned seven advocacy projects but only one was implemented. USAID may need to clearly state expectations for detailed plans and monitor progress closely. VII. OUTCOMES AND IMPACTS ON MALES AND FEMALES NDI’s general program focus was on youth and women. Nearly all youth programs equally invited and served male and females, with one exception, a female only Youth Leadership Academy group. As noted in the report women participants in this group would have preferred to attend trainings with both genders, a recommendation recorded in this evaluation. Armenia is male-dominated across institutions, as well as culturally. Therefore, NDI’s focus on programming for women to increase their influence politically was most appropriate and yielded success. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 25 VIII. RECOMMENDATIONS The development challenge for Armenia is encouraging meaningful citizen participation as effective political actors, including members of political parties, civil society organizations and movements and citizens at large. Due to the strong central government that is perceived as self￾serving and corrupt, NDI chose a fine tactic: a bottom-up approach for citizen involvement. Next year’s Parliamentary Elections loom large, and it is an important opportunity for citizens to feel they have electoral choices and can trust election results. Based on NDI’s fine success of training youth of political parties and NGOs, as well as supporting women leaders, the recommendations support and expand their current work. The following are specific recommendations: 1. Continue women candidate training for the upcoming parliamentary election and post￾election coaching. In advance of the next local elections, focus on city council candidates as prime training targets due to the power of those political entities, as opposed to village councils. However, village candidates should continue as a secondary focus, a good training ground for women leaders and an opportunity to provide models locally to encourage additional women to pursue public service. 2. Develop a multi-party training and coaching program for women political party leaders to enable greater influence and representation within their parties. 3. Continue coaching and supporting the Women’s Forum Board and its members, especially since the next step is development of advocacy campaigns for their four sectors, an opportunity to create valuable policy and legislative changes within sectors of greatest concern for Armenian women. 4. Create a sustainability plan for the Women’s Forum for their national conferences and all programs, coaching them on fund raising skills, potential ways to earn income, acquiring collaborative relationships, etc. 5. Continue youth leadership training that includes all political parties and NGOs. Re￾evaluation gender segmentation since there is strong evidence that the women would prefer integration with men as academy participants. Take leadership training to the next level, developing pragmatic programs to practice skill development. For instance, youth members should be trained and practice as TOTs for their parties and NGOs, an opportunity for them to continue skill development while training additional youth members and others. Make them responsible and accountable for formal training of their fellow youth members. Also, coach training graduates to plan and implement advocacy and community development projects, as originally planned. 6. Train youth and women party members as TOTs to enable party members to serve as election monitors (proxies) for the upcoming Parliamentary Election, providing youth members with a central role within their parties while acquiring additional practical skills. 7. Present the idea of an Election Code of Conduct to youth political members, potentially gaining their support to adopt the code as youth committee members and subsequently, encourage their party leader to adopt the code for the entire party. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 26 8. Continue multi-party public programs with the National Press Club and Armenian Center for International Political Studies. Add youth and women presenters, gaining their perspectives on issues of concern. Work with the two NGOs to develop a program sustainability plan. 9. Re-assess NDI’s consulting relationship to It’s Your Choice, the national election monitoring NGO. 10. Assure political party training and consulting programs are offered to all opposition coalitions and parties, including their youth members. 11. Define PMTs and indicators more fully, including definitions of data for collection, and monitor results in more detail through quarterly reports. 12. Enable better coordination among international organizations so that programs for women and youth are not duplicated during this period of fewer resources available for Armenian citizens. Other organizations that address leadership training and project development for youth and women, including candidate training, are World Learning, OSCE, UNDP, Eurasia Foundation and the Ministry of Culture. It is recommended that USAID or another international body with equivalent status coordinate such efforts. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 27 ANNEX I: EVALUATION SCOPE OF WORK Evaluation of USAID/Armenia Armenia-Turkey Rapprochement (SATR), Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA￾ROLI), and the Small Scale Infrastructure Program (SSIP) Summary: USAID/Armenia requires evaluations of the following activities: Armenia-Turkey Rapprochement (SATR), IFES and NDI electoral and political process Associate Awards under the Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA-RO LI), and the Small Scale Infrastructure Program (SSIP). The purposes of this Task Order are to evaluate the success of these projects in their relevant areas and assess effectiveness of these in achieving set programmatic goals and the USAID/Armenia’s strategic objectives. Two of the five planned evaluations are designed as midterm evaluations (Armenia￾Turkey Rapprochement, and ABA-ROLI), while the other three are designed as end-of-project evaluations (IFES, NDI and SSIP). In the case of mid-term evaluations the findings will be used to inform USAID’s determination whether they are on track achieving their set programmatic goals and targets and whether the initial design of the projects still leads them to the set objectives. They will feed information into the future work plans. In the case of end-of-project evaluations, the findings will be used to inform design and development of future projects. Therefore, the evaluations will identify “lessons learned”; assess strengths and weaknesses of strategies and activities performed under these projects; and provide recommendations to USAID for project planning purposes for the next three to five years. The Contractor will seek to capture effective approaches; analyze the utility of performance monitoring efforts; consider respective outcomes and results; and assess the influence of internal and external changes on the achievement of results. Contractor Responsibilities and Projects: The evaluation should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, including an “effectiveness and efficiency assessment” that looks at how successful the programs have been in achieving their set targets, and how effectively USG resources have been used. Additionally, USAID/Armenia would like to measure the sustainability of the project results on respective beneficiaries where applicable and possible. The Contractor shall review each project’s implementation methodology and to the degree possible, verify the results achieved, the relevance of the project in addressing USAID priorities, and to what extent USAID can be accountable for achieving those objectives. Final recommendations to USAID will help improve program outcomes, weigh sustainability factors, and address program relevance as well as cost efficiency and effectiveness. The evaluations will serve to guide how similar projects, approaches and/or work plans can be improved. The evaluations will also validate (or not) the feasibility of the initial designs of the projects and of their respective development hypotheses. SATR NDI Evaluation Report International Business & Technical Consultants, Inc. Page 28 This two-year activity is in its first year of implementation. The main objective is to promote improved Armenia-Turkey relations by engaging civil society in the reconciliation processes; establishing and developing business partnerships and regional professional networks; and facilitating government-to government dialogue. The activity is being implemented in a rapidly changing environment of Armenia-Turkey relations. Therefore the appropriateness of the design of the project needs to be explored in addition to the actual project implementation issues. Thus, this evaluation should contribute to the Mission’s understanding of whether or not adjustments are necessary in the approach and in the future planning of this project. The evaluation should aim at obtaining feedback from stakeholders and project partners both from Armenia and Turkey. CEPPS These are two separate Associate Awards under CEPPS III, implemented by IFES and NDI, both of which are in their third year of implementation and are expected to end in FY2012. Both activities are primarily aimed at improving political processes. The IFES activity supports the strengthening the administration of electoral processes and assists the electoral administration to meet international standards for free and fair elections through its support to the Central Election Commission and the Passport and Visa Department of the Police of the Republic of Armenia (OVIR). The NDI activity aims to improve the ability of citizens to effectively participate in political processes as members of political parties and helps political parties develop their ability to take part in parliamentary and presidential elections scheduled for 2012 and in 2013 respectively. The findings and recommendations from this evaluation will be reviewed for results achieved, and effectiveness of program approaches. Best practices and lessons learned will be identified which will contribute to the Mission’s decisions on future election-related activities. ABA-ROLI This is in its third year of implementation and will end in FY2012. The main objectives are to support curriculum reform in the Judicial School; cooperate with the Judicial Department of the Republic of Armenia for enhancing the judicial reforms; and provide greater access to justice through Law School legal clinics with a particular focus on protection of human rights. The purpose of evaluation is to assess the results and outcomes of this rule of law initiative and the sustainability of the achievements of the project to inform future USAID/Armenia decision￾making with regard to similar undertakings. SSIP This is a 27-month project ending in FY2012. The goal of the project is to mitigate the consequences of Global Economic Crisis through the creation of temporary employment opportunities in vulnerable rural communities by means of implementation of small scale infrastructure projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre-schools, community centers, sport halls and drinking water supply systems. This project deals with multiple communities and partners in jointly carrying out construction and work with target communities. Executing water projects has been challenging due to different factors, including multiple construction partners, delayed contribution from other donors, weather conditions, etc. However, civil construction has been progressing timely and with very good quality. Project has been modified twice to increase the number of projects (from NDI Evaluation Report International Business & Technical Consultants, Inc. Page 29 48 to 58), labor days and expand the geographic coverage. The final evaluation will look at few aspects of the program: how the targets have been met vis-à-vis original targets and benchmarks; how the partnership between all parties (IFAD, local NGO Shen, local construction companies, village Mayors, Local Supervisors, etc.) worked; what are mechanisms that would help communities sustain project gains after its completion; did the project leverage contributions or matching funds from other donors or stakeholders as anticipated originally. Evaluation Questions: The Contractor shall review and summarize the implementation and results achieved by all five activities to answer the following evaluation questions and additional questions that may be developed by the Contractor after reviewing the provided materials. For ALL projects: • To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? • Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? • What were the main achievements of the program? • How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? • How did the implementer perform in terms of project management and how effective was the project leadership? • What lessons learned‟ can be provided for future USAID programming in this area? • What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? • Did the agreement provide clear and achievable results against which progress and impact could be measured? • Analyze attribution of project successes to USAID involvement. • Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. Project-specific questions/Tasks: For SATR: • Are the project implementation approaches relevant and feasible in the current state of affairs in the Armenia-Turkey relations? • Which specific aspects of the project are or are not working in the given political situation? [This question should be explored in the Armenian and Turkish contexts separately, because some approaches/activities may be still feasible in Armenia but not in Turkey and vice versa.] • How flexible is the project in terms of adjusting activities to the changing political context? • How has this activity been able to build on the advances made under the previous “Days 2 and 3”project? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 30 • What signs exist that the project is having impact, anecdotally? Why? What concrete examples of impact (or lack of it) are given? • What is the external perception of the project’s role and impact both in Armenia and Turkey, according to the key stakeholders not involved in the project? • What recommendations can be provided to improve impact? What are the key obstacles and what recommendations can be made to minimize their effect? • How appropriate is the decentralized project implementation approach (four local partners with their Turkish counterpart organizations) to the project objectives? • How effective is the implementing partner as a consortium of local organizations: responsiveness to the donor, responsiveness to project stakeholders, information flow. What recommendations can be made to improve consortium management and operations? For CEPPS: • How can the projects be better designed in the future to measure impact, given the political situation faced in Armenia? NDI • Have NDI youth and women’s Leadership Training Academies, the NDI Women’s Candidate Schools, and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party memberships and activities, political engagements, political public gatherings, etc.) among youth and women? • Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political and policy activities. • To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations? • Have NDI/Yerevan Press Club public debates had value for participating political parties? • What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? How successfully? IFES • Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code? • Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 31 • Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to electoral administration reforms? • Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? For ABA-ROLI: • What are the most notable accomplishments of the project in the ROL area since the inception of the current agreement in 2009? • What are the factors hindering judicial independence in Armenia and what has the project done to strengthen the judiciary? • Has the project been able to increase the capacity of the Armenian judiciary to play a role in making the government more accountable? • What did the project do to ensure proper implementation and enforcement of new or existing laws? • What are the activities aimed to establish mechanisms for oversight of court proceedings o How do these mechanisms affect judicial performance? o Are the existing oversight mechanisms sufficient to activity build on the existing efforts to promote accountability and transparency in the justice sector? o If yes, please name, if not, please mention gaps. • Is there capacity or interest within the legal professionals to promote reform in ROL? If yes, what does the project do to support that interest? • How does the project support the country’s only Bar Association in helping to play a balancing role within the justice sector? • What has the project accomplished in regard to defending human rights through legal protection? • How did the project affect legal education in law schools? • Where is the most viable stakeholder support for rule of law reform likely to be found? • What targeted activities could be proposed to address Armenia specific deficiencies in the justice sector? • How successfully does the project coordinate with the international community to promote human rights? • What recommendations can be made for a more effective, integrated project design? For SSIP: • What is the ratio of “planned” or projected and “actual” jobs generated throughout the project? How do you explain the gaps (if any)? • What are social, economic and other impacts of the project on target communities and beneficiaries? • How do you think the completed projects will be maintained by the communities? What are the grounds for their sustainability? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 32 • How did the partnership work within the project between different partners (IFAD, Shen, local contractors, communities, etc.)? What worked, what did not, why? • How did the partnerships impact the project (timeliness, quality, and cost-benefit)? • In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met? • How does success in two main areas that the project tackled - water and civil construction compare in terms of progress made, achievements, implementation challenges, etc.? • Has/will CHF meet its cost-sharing commitment? Are cost-sharing valuations reasonable, consistent and adequately documented? • What are the lessons learned? What are the strengths and weaknesses of this project, its approaches or strategies? • If a similar activity is considered in the future, what should be changed in the design and technical approach? USAID’S Role in the Evaluation The USAID Mission in Armenia will: • organize a small USAID advisory group to support the Contractor in the implementation of this scope of work; • provide relevant programmatic and budgetary information to the Contractor (some relevant portions of contracts and assistance agreements are attached); • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor will: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Develop additional research and evaluation questions as needed based on the development hypothesis and on the above-mentioned evaluation questions; identify informants and stakeholders, samples and/or other relevant data sources. 3. Develop data collection tools based on the best possible methodology in accordance with the evaluation questions and feasibility considerations and provide to USAID prior to commencing field work. 4. Prepare a field work plan. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 33 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports as requested by USAID and submit final reports to USAID/Armenia for acceptance. The proposed methodology should address the need for data collection from qualitative and quantitative sources; and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. To the extent possible, data should come from facts, rather than be based on anecdotal evidence, and conclusions should be based on findings received from multiple sources. Clear, standardized data collection methodology should be described in detail to ensure reliability and consistency of the evaluation findings. Deliverables The Contractor’s deliverables shall include: 1. A written methodology plan (research design and operational work plan). 2. Provide a verbal debriefing at the end of the field work to Mission management and technical teams. 3. Prepare draft evaluation reports (electronic and hard copy) which will analyze data and summarize key findings, conclusions and recommendations. The Evaluation Report shall at a minimum contain 1) an Executive Summary; 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work, all evaluation tools, all sources of information. Submit these to USAID/Armenia within three weeks after completing the fieldwork. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. The Contractor shall submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The final reports will meet the following quality standards: a) The reports will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports shall address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings will assess outcomes and impact on males and females; f)Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of people’s opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) NDI Evaluation Report International Business & Technical Consultants, Inc. Page 34 Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. The Contractor will submit Evaluation data to USAID/Armenia along with the final reports for warehousing and future use by the Mission. The data will be in easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Evaluations Timeline 1. Evaluations should be initiated around September-October 2011. The preliminary findings of the evaluations should be submitted to USAID/Armenia immediately after the completion of the field work. 2. The first drafts of the five evaluation reports shall be submitted to USAID/Armenia no later than three weeks following the completion of the field work. (For CEPPS USAID expects to have two separate reports – one on the IFES activity and one on the NDI activity). 3. The final Evaluation Reports shall be submitted no later than two weeks after final comments on the draft evaluation reports are submitted by USAID/Armenia. 4. Once finalized, the contractor is responsible for ensuring that the final approved reports are also submitted to USAID’s Development Experience Clearinghouse within three months of the completion of the reports. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 35 ANNEX II: EVALUATION WORK PLAN Introduction This is the final evaluation of the current three-year NDI project, ‘Political Process Development in Armenia.’ The program began January 9, 2009 and ends on January 9, 2012. NDI has been the implementing partner of the CEPPS (Consortium for Elections and Political Process Strengthening) program for this project. IFES has been an associated partner with NDI in the CEPPS program. The turmoil following the February 2008 presidential election resulted in an increasingly contentious political environment based on a highly centralized and extremely powerful government, and acrimony among the political parties. This led to policy paralysis and few options for citizens to express their views and contribute to decisions that affect their lives. In view of citizen disconnect with the nation’s decision-makers, NDI focused on a more robust bottom-up approach to political party development and civic advocacy, dedicated to developing citizen “demand” for government and political party responsiveness. The program aims to generate and support a new group of “agents for change” at the local level and related to local issues that can generate greater interest than those on the national stage. Additionally, NDI focused on expanding government outreach to opposition groups and citizens, as well as increasing the effectiveness of election and government monitoring groups. To achieve these goals, NDI has three objectives: 1. Promote greater responsiveness among political and civic groups to citizens’ interests at the local and national levels by increasing the capacity of citizens to organize and engage proactively in local community issues and political processes; 2. Facilitate more government outreach to opposition groups and citizens; and 3. Assess and, if conditions permit, contribute to the integrity of elections through more effective domestic election monitoring. These three objectives and their activities constitute the core of the NDI program in Armenia. Purpose The general purposes of the evaluation as stated in the Scope of Work (SOW) are: 1. To measure and analyze the accomplishments toward achievement of the results of the activities, including an ‘effectiveness and efficiency assessment’ that examines how successful the program has been in achieving their set targets; 2. To assess how effectively USG resources have been used; and 3. To measure the sustainability of the project results on respective beneficiaries. The findings of the evaluation will be used to inform the design and development of future projects. The evaluation will identify lessons learned; assess strengths and weaknesses of strategies and activities performed; and provide recommendations to USAID for project planning purposes for the next three to five years. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 36 Methodology The evaluation team consists of the following three people: Marilyn Evans (US), Hrachya Zakoyan (Armenia) and Robert Brandsetter (US). The team will spend 14 days in Armenia, from November 20 to December 6, 2011. Because of the structure of the CEPPS project and the different primary activities, Brandsetter and Zakoyan will focus on IFES and Evans on NDI with interpreter assistance of Ruzanna Ghurbanyan. The team will discuss their work continually and where there are synergies and overlaps between NDI and IFES programs, these will be noted. The field work will be guided by a number of specific questions noted in the SOW: • Have NDI youth and women’s Leadership Training Academies, NDI’s Women’s Candidate Schools, and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party membership and activities, political engagements, political public gatherings, etc.) among youth and women? • Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political and policy activities. • To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations? • Have NDI/Yerevan Press Club public debates had value for participating political parties? • What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? How successfully? The research methodology that will be used in the evaluation will consist of the classic social science techniques including desk review of the literature, one-on-one interviews with key informants (Annex III), focus groups and, if possible, observation of training or consulting sessions. 1. The desk review prior to deployment to the field will consist of the review of all available NDI documents, including the project paper, PMPs, work plans and quarterly reports. Other relevant documents such as OSCE election and Freedom House reports will be consulted. 2. Headquarters staff at NDI and scholars of Armenia and the Caucasus in Washington will be interviewed prior to deployment. 3. The field research will consist primarily of meetings, focus groups and informational interviews with key individuals and groups, noted in the Annex III. These interviews and meetings will be both structured and unstructured. A set of questions will guide the structured interviews (Annex IV). NDI Evaluation Report International Business & Technical Consultants, Inc. Page 37 4. Most of the interviews will occur in Yerevan, however, the team also will conduct some interviews in the regions (marzes), including with members of political parties, It’s Your Choice (IYC), Women and Youth Forums, women elected officials and former candidates. Interviews in regions are required based on NDI’s goals of generating local involvement of citizens within their national organizations and as members of advocacy campaigns. 5. A draft work plan is in Annex II. Deliverables 1. This work plan and evaluation design is the first deliverable and is sent to USAID/AM in advance of the first briefing with USAID, during which time approval for will be sought. 2. An oral debriefing at the end of the field work will be made to the mission management and technical teams. 3. Draft reports analyzing the data and summarizing the key findings, conclusions and recommendations that will be provided to the mission by December 30. The format of the evaluation will follow the standard format for USAID evaluation report3 , with the NDI/CEPPS text report not exceeding 25 pages, excluding annexes. 4. Mission comments will be incorporated into final report and submitted by Feb. 3, 2012. Work Plan Schedule Week Day/date Planned activities Deliverable Team Leader Team Member Team Member Week 1 Mon, Nov 21 Initial Meeting with USAID COTR, etc to discuss this work plan Detailed Evaluation Design and proposed Work Plan Robert Marily n Hrachya Tues, Nov 22 Meetings with IFES & NDI Staff Robert Marilyn Hrachya Wed, Nov 23 – Sat, Nov 26 Meetings and scheduling meetings Marilyn Week 2 Mon, Nov 28 – Sat, Dec 3 Interviews TBA Marilyn Week 3 Mon, Dec 4 Interviews TBA Marilyn Tues, Dec 5 Debrief Preparation Robert Hrachya Wed, Dec Debriefing USAID Out Briefing Robert Marilyn 3 IBTCI evaluation teams refer to USAID’s “EVALUATION Learning from Experience,” Bureau for Policy, Planning, and Learning and “Checklist for Assessing USAID Evaluation Reports” for guidance. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 38 Week Day/date Planned activities Deliverable Team Leader Team Member Team Member 6 Post￾Field Work Mon, Dec 12-27 Analyze data/make key findings and recommendations Robert Marilyn Tues, Dec 27 Draft report submit to IBTCI Robert Marilyn Fri, Dec 30, 2011 Draft report submit to USAID IBTCI Jan 1 to 19 USAID Comment Period Fri, Jan 20 - Feb 2 Receive comments from USAID and make relevant revisions for the final report Robert Marilyn Thurs, Feb 3, 2012 Final copy submitted to USAID IBTCI NDI Evaluation Report International Business & Technical Consultants, Inc. Page 39 ANNEX III: LIST OF INFORMANTS FOR FIELD INTERVIEWS Key Informants USAID/Armenia: Director, Jatinder Cheema COTR, Mariam Gevorgyan COTR/CEPPS, Nick Stokes Project Implementers NDI COP, Gegham Sargsyan IFES COP, Michael Getto NDI Program Partners, Involved Parties and Experts • Women Forum Members – in Yerevan and regions • Youth Forum Members – in Yerevan and regions • It’s Your Choice – Harut Hambaryan, his central office and regional members • Political parties, members of Yerevan and regional offices: o National Armenian National Coalition l Democratic Union (NDU) o National Democratic Party o Legal State Party o Armenian National Congress (ANC) o Armenian Revolutionary Federation (Dashnaks) o Republican Party o Republic Party o National Party of Armenia o Solidarity Movement o People’s Party of Armenia (PPA) o Prosperous Armenia Party o Heritage Party • Participants of NDI Women’s Candidate Schools • Participants of National Conference of Women in Politics • Presidential Secretariat, Mr. Garnisk Isagulyan and/or his staff NDI Evaluation Report International Business & Technical Consultants, Inc. Page 40 ANNEX IV: INTERVIEW GUIDES IT’S YOUR CHOICE (IYC) QUESTIONNAIRE GUIDE Final Evaluation of CEPPS/NDI Armenia IBTCI November – December 2011 Questions posed to central office staff and members of regional offices to ascertain success for decentralization for decision-making and two-way communications: 1. Describe communications processes within the organization. Are the processes based on recent changes or continuation of successful ones from? If available, please provide example pieces. 2. What are volunteer recruitment strategies? Who and how are volunteers recruited? Have recruiting practices changed or based on previous successful practices? 3. How did IYC monitor elections in 2008? And in 2009? Do you plan to continue the same practices in 2012? (Gain insights: are they now monitoring beginning earlier, e.g. 3-months in advance of elections, do they plan to use mobile phone based messaging for quick reporting at polling sites, etc.?) 4. Describe the post-election reports. How were results presented? What was the feedback from civil society groups, political parties, NGOs, CEC, the international community, etc.? Did IYC receive negative feedback about the reports? Do you have samples for our review? 5. Describe the leadership of IYC. How long have these leaders served in their positions? Are there plans for any changes? 6. What are the sources of IYC funding? Do you expect to be similarly funded for the next few years? What are your strategies for fund raising and the sustainability plan? 7. What kind of assistance do you receive from NDI? Is the assistance useful and if so, how? What assistance was most helpful? Can there be improved cooperation between NDI and IYC? If so, how? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 41 POLITICAL PARTY MEMBER QUESTIONNAIRE GUIDE Final Evaluation of CEPPS/NDI Armenia IBTCI November – December 2011 The main thrust of NDI’s work with political parties is to facilitate decentralization for decision￾making and to assure two-way communications. Another goal was to facilitate collaboration between the ruling party (Republican Party) and opposition parties. Interviews can include the following questions or subjects asked of both central and regional political party leaders. Name: Political party: Regional, local or central office: 1. How does your political party make decisions? Who is involved? What is the process? What are the means and processes of communication within your party? Gain insights: Is there evidence of de-centralization, e.g. policy and platform development through the bottom-up approach of meaningful involvement of regional offices, two-way communications to regional party activists unrelated to election activities (get-out-the-vote and poll watching), etc. 2. Did the parties utilize NDI focus group results and information? If so, how was the information utilized? Did party members acquire knowledge and skills for the process for developing platforms? Gain insights: Were results utilized for development of platforms, policies and messaging in advance of the 2009 elections or in preparation for upcoming parliamentary elections? Or as background for developing and implementing successful local projects? 3. Did the parties implement successful local projects based on focus group results or other means of gauging citizen concerns and needs? 4. How did your party develop platforms, messaging and conduct get-out-the-vote campaigns? Gain insights: Were the political parties effective for the 2009 election for retail level, person-to-person and community-based organizing, utilizing such tactics as door-to-door canvassing and other direct voter contact activities? 5. Had your party or regional organization of your party work with NDI previously? Did you work with them on grass-roots organizing of current and potentially future members? If so, what was the outcome? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 42 Gain insight: Did NDI work with new parties or branches of parties that were committed to grass-roots organizing? Were they successful? 6. Did your party conduct policy forums with regional members about issues of concern? What was the result of the forums? Gain insights: If the party conducted regional or local policy forums, did the information derived from discussion help shape policies and platforms in advance of the 2009 and upcoming parliamentary election? 7. Is your party on-track for development of platforms for the upcoming parliamentary elections? What is the status of platform development? 8. Has your branch of the party received training for voter outreach, targeting and message development in preparation for the upcoming parliamentary elections? When? Was the training helpful? Describe. Did your party train others as trainers to reach more members? What was the result? 9. Did NDI training specifically target youth and women, successfully engaging them in party decision-making? Did your party train others as trainers to reach more members? What was the result? 10. Was your party branch trained and coached for election monitoring? What were the topics or what did your members learn? Gain insights: Advanced training to include such topics as ballot accuracy, ability to fully campaign, fair treatment by mass media, voter list accuracy, gaining remedies from courts for violation of rights 11. Did representatives of your party participate in National Press Club meetings during the last three years? Were they useful for your party? If so, please describe. Did they allow and/or encourage discussion and potential collaboration among parties? If so, please describe. Did such discussion and collaboration include members of the Republican Party? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 43 REPUBLIC OF ARMENIA PRESIDENTIAL SECRETARIAT QUESTIONNAIRE GUIDE Final Evaluation of CEPPS/NDI Armenia IBTCI November – December 2011 NDI has offered services to the Secretariat, including per the office’s request, consultation for outreach communications strategies. Additionally, NDI intends to enable collaboration between the Republican Party (ruling party) and opposition parties. 1. Did the Secretariat confer with NDI? If so, when and for what purposes? 2. Did the Secretariat request consulting from NDI to develop effective communication strategies for public outreach and to gauge public reaction to proposed policies? If so, was support acquired? What was the outcome? Was it of value? 3. Were members of the Secretariat involved in National Press Club debates and discussions? If so, please describe. 4. In regards to the Secretariat’s outreach to opposition parties, did NDI provide assistance? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 44 NATIONAL PRESS CLUB QUESTIONNAIRE GUIDE Final Evaluation of CEPPS/NDI Armenia IBTCI November – December 2011 NDI partnered with the Yerevan- based National Press Club f or debates and discussions, for the purpose of gaining co llaboration among political pa rties and activ ists and generally broadenin g the quantity and quality of public debate. 1. What was the involvement of NDI for your m eetings to discuss and debate political and social and political topics? (Inquire: Did NDI contribute to program development, recruit speakers, etc.?) 2. Did NDI contribute to your success during the last 3 years? If so, how? 3. How many debate sessions occurred during the last three years? 4. Did the d ebate programs contribute to m embers of the ruling and opp osition parties discussing and collaborating a bout significant issues in a congenial “space? ” W ere there outcomes that affected public policy? If so, please describe. 5. Could NDI assist in other wa ys that would provide suppo rt in general and for your programs? 6. Have other international organizations provided support? Is so, whom and how? 7. Does the National Press Club plan to continue their work for the next several years? If so, will the objec tives and progr amming remain sim ilar to those o f the las t f ew years? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 45 WOMEN CANDIDATES AND ELECTED OFFICIALS QUESTIONNAIRE GUIDE Final Evaluation of CEPPS/NDI Armenia IBTCI November – December 2011 NDI conducted campaign training for women candidates for local and national elections, noting that many were successfully elected. NDI also formed a women’s group to support the elected officials for their related work. Candidate name: Office: Region or district: 1. Did you attend a candidate school training conducted by NDI? If so, when and where? 2. Did you receive training as a candidate by any other NGO? If so, by whom and when? 3. Were you a candidate for public office? If so, what race and when? Were you successful? If not, what were the obstacles for your campaign? Other questions for candidates: 4. What skills did you acquire or were honed from NDI training? How did the skills contribute to your campaign? 5. Did you have a campaign committee? If so, who were they and how were they involved? 6. Did you develop a campaign platform? How did you decide issues for inclusion? Did the platform include solutions? 7. Did you develop a communication plan? What were your main messages for campaigning? How did you communicate with your constituents? Were you able to utilize the media? If so, how? 8. Did your campaign plan include grassroots campaigning? If so, how? Was this effective? 9. What tactics did your campaign use for get-out-the-vote for Election Day? 10. If elected, has NDI provided additional training or support to assist as you serve your constituents? If so, please describe. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 46 Questions for candidates and non-candidates: 11. Did you use the skills acquired through NDI training for other activities or projects for your political party or in general? If so, please describe. (Inquire: Did they train other party members with skills acquired or implement political or social projects utilizing the skills, potentially including government monitoring?) 12. Is there additional training that you need currently as a candidate, elected official or an activist? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 47 YOUTH AND WOMEN FORUM INTERVIEW OR FOCUS GROUP QUESTIONNAIRE GUIDE Final Evaluation of CEPPS/NDI Armenia IBTCI November – December 2011 The two sets of focus groups involving members of Youth and Women Forums will be similar since the objectives and programming for both Forums generally are dedicated to development of skills for advocacy campaigns. The focus groups will involve solely women or youth members of the two Forums. As possible, focus groups will be conducted in regions, not exclusively Yerevan. General questions: 1. When and where did you participate in NDI training seminars and/or workshops? 2. What was the purpose of the training? 3. Did you acquire or better hone skills for organizing? If so, what were they? 4. Are you involved currently in an advocacy campaign or plan to become involved? If so, please describe. Is your involvement related to NDI training and coaching? If so, how? (Advocacy campaign topics that might be discussed, both whom involved and how: choosing an issue, conceptualizing solutions, designing the plan, involving others of the organizations and collaborative partners or stakeholders, training “members,” communications planning and implementation both internally and externally, use of media and coverage, building coalitions, etc.) 5. If involved in community organizing or advocacy campaigns, what communication channels do you use? Is your communication plan and processes related to NDI training? 6. Are you involved in government monitoring? If so, how? Is this effort related to NDI? 7. What additional training or knowledge do you need to conduct successful advocacy efforts? NDI Evaluation Report International Business & Technical Consultants, Inc. Page 48 ANNEX V: KEY REFERENCE DOCUMENTS REVIEWED Caucasus Research Resource Center. “The Caucasus & External Powers,” September 2010 Caucasus Research Resource Center. “Proposed Survey of Political Attitudes,” Armenia, A Program of Eurasia Partnership Foundation November 2011. CEPPS NDI PMP 2009 CEPPS NDI. Quarterly Reports from January 2009 through September 2011 CEPPS NDI. First Annual Work Plan January 2009 CEPPS NDI. Cooperative Agreement, No. AID-111-A-09-00001 CEPPS NDI. “It’s Your Choice,” Final Report On Monitoring of City Council Elections May 31 NDI. “Results of NDI Focus Groups in Yerevan, Shirak Region, Vayots-Dzor Region.” Power Point Presentation. May 2010 NDI. “Results of NDI Focus Groups on Political Barriers and Opportunities,” May 2010 NDI Evaluation Report International Business & Technical Consultants, Inc. Page 49 ANNEX VI: CONFLICT OF INTEREST STATEMENT NDI Evaluation Report International Business & Technical Consultants, Inc. Page 50 ANNEX VII: STATEMENT OF DIFFERENCE From: National Democratic Institute To: USAID Re: Statement of Differences regarding USAID’s Evaluation Report of NDI’s Programs in Armenia, March 2012 The National De mocratic Institu te (NDI or the Institu te) would like to respond to the “NDI Evaluation Report”, conducted by International Business & Techni cal Consultants, Inc, (IBTCI) and submitted to USAID in March 2012, and o ffer this statement of differences for the record. NDI appreciates the opportunity to do so and appreciates USAID’s continued support for NDI’s programs in Armenia. NDI would like to note that it received the evaluation report on March 13, 2012, the day after the cost-extension of its current agreement with USAID was finalized. Therefore, although the report states that its findings will be used to inform the design and development of future projects (p.2), the Institute was unable to use the report as a resource for its m ost recent proposal due to the aforementioned tim eline. NDI will, however, continue to use the rep ort, and par ticularly th e recommendations section, in the development of its workplan, performance monitoring plan, and in the implementation of our program activities overall. Overview It is worth noting at the onset th at the evaluation unfortunately did not seem to focus on the “big picture” political dynamics of Armenia which is integral for understanding the design of the NDI program and the corresponding cha llenges and opportunities for the future. There is alm ost no mention of the history of problematic elections that has contributed to governing institutions that are neither represen tative of nor accountable to the citizens of Arm enia. A driving focus of NDI’s program is to e mpower Armenian citizens to participate in politics toward reclaiming the political process and, eventuall y, the political institutions that govern their lives. This is the rationale behind NDI’s “bottom -up” approach cited in the assessm ent report and also influences the design and i mplementation of NDI program activities. In addition, an inhospitable political climate has required some degree of programmatic adaptability in order to seize opportunities as they arise and minimize potential pitfalls. NDI has identified several potential “agents of change”—women, youth, civil society—that offer the m ost prom ise, m otivation, and innovation to bring about change. NDI’s engagem ent wit h them could generally be described as a “voice, space, accountability” approach. As evidenced by NDI focus groups, the political, social and ec onomic clim ate has proven m ost difficult for traditionally marginalized groups, such as wom en and youth, requiring greater initial focus than anticipated on helping women and youth to reclaim some political space in which to operate and focus to help am plify their voices on issues of need and interest. The com ing years offer NDI Evaluation Report International Business & Technical Consultants, Inc. Page 51 opportunity to build on these gains while adding more focus on holding political and government leaders more accountable for reforms. NDI believes this approach has greater chances of success for prom oting internal party reform, for instance, rathe r than engag ing political p arty lead ers directly on internal refo rm with the hope th at it m ight positiv ely impact wom en and youth. Another aim of NDI’s approach is toward fo stering greater dialogue and interaction am ong political party wom en and youth with the hope of reducing political party acrimony and increasing focus on issues of need. NDI recognizes the time constraints of the assessm ent and page constraints of the assessm ent report, though the absence of infor mation at this level m akes it difficult to see if the overall purpose and f unction of the NDI program is truly understood in order to effectively evaluate if it is meeting its goals and objectives. Despite very little adv ance information prov ided about the scope of the evaluation or the expected time commitment of NDI staff (the Institute was only informed on November 3 that an evaluation team would be heading out to Arm enia the week of Nove mber 21), NDI has tried to be as helpful as possible. Our W ashington, DC st aff made time available to m eet with Robert Brandstetter and speak with Marilyn Evans by phone prior to departure and provide several written documents to them. Our staff in Yerevan made themselves and our programs available to the evalu ation team . At the direct and written request of the evalu ation team , NDI/Arm enia scheduled meetings wi th specific program partners and p olitical leaders. These m eetings were inexplicably and unfortunately canceled, however, which might have helped to fill in some of the information gaps noted in the report. In addition, NDI asked US AID/Yerevan about the appropriateness of requesting an ex it inte rview of the evalu ators in ord er to addr ess any f inal questions or information gaps. Based on USAID’s favorable response, NDI m ade the request of the IBTCI assessment team. On page 3 of the report, the assessment team refers to this request in the context of “unreasonable dem ands” made by th e NDI Country Director which is inaccurate and inappropriate considering that the request was neither unreasonable nor a dem and. This characterization, we feel, is entire ly not appropriate for this repor t and undercuts its credibility. Respectfully, NDI requests that th is statement be removed from the report. W e believe NDI has gone out of its way to be maximally accommodating and responsive to the assessm ent team in Washington, DC and in Yerevan and on short noti ce. Our intent h as been to be helpf ul and cooperative, but not at all controlling or influencing of the agenda. It is unfortunate that the IBTCI team responded to the N DI Country Directo r’s request with “limited communication with the local NDI office” (pg. 3) which we believ e has contributed to numerous factual errors, adm itted gaps in inform ation, and m isunderstandings and misrepresentations tha t an exit in terview wa s intended to address. IBTCI’s cancelation of meetings th ey spec ifically reques ted f or NDI to schedu le also d id n ot cr eate f or the m ost favorable impression of NDI, USAID and IBTCI among key contacts in the democracy arena. As noted in the report, IBTCI cold calls to set up appointm ents independently of NDI and USAID were not very successful which seem ingly li mited the available information on which to base conclusions. We were also curious about the description of the assessm ent m ethodology that described “NDI’s program f ocus on politica l party dec entralization” that led to the selection of key informants from nine political parties (page 2). This is not an accurate description of the focus of NDI’s program which lends som e cause for fu rther concern for the methodology em ployed. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 52 Empowering women and youth as agents of change is a prim ary focus of NDI’s program with political parties and civil society organizations as the secondary focus. This being said, having worked in Ar menia since 1995, NDI has a deep appreciation for the enormous challenges and frustrations involved in undertaking such a comprehensive assessm ent in approximately two weeks on the ground. With this in mind, aside from some general concerns regarding the assessm ents overall methodology a nd some s pecific concerns noted below, NDI agrees with many findings of the report and its overall recommendations. Some of these deserve particular mention: o NDI appreciates the recognition by IBTCI and USAID of its “bottom -up” approach and focus on women and youth as agents of cha nge. NDI public opinion research has further illustrated b oth the eno rmous challenges th at these cons tituencies face, but also the opportunities they bring and their potential fo r change. W e have argued our “theory of change” for future efforts in our extension proposal to USAID and are seeing increased momentum and results from our programs in these areas. o We agree that future NDI program s with wom en and youth will need to loo k at opportunities to take these constituencies to the next level. While initial programs have focused on building political sp ace and a cadre of experien ce and em powered activists, we hope continued progress to lead to continue opportunities such as building support for women candidates within politic al parties, leveraging women and youth to foster cross￾party dialogue and cooperation, and leverage women and youth leaders to spread training and lead advocacy and community development projects. o We share the im portance of perform ance m onitoring and are striving to im prove the accuracy of our m easurements and systematizing our data collection. In term s of its performance m onitoring tools, the Inst itute was of the und erstanding that USAID/Armenia requested that NDI not submit annual workplans or Perform ance Monitoring Plans after the first year. Under its current cost extension, the Institute intends to submit an updated workplan and PMP through the end of the grant on March 31, 2013. o NDI firmly shares the f ocus of the assessm ent team and USAID on sustainability. F rom the initial program design phase, we’ve sought to maximize participation of host-country staff and local partners toward empoweri ng local ownership over NDI program s with sustainability in m ind. NDI public opinion re search has been extrem ely useful as a programmatic tool to ensure that programs and respective efforts are effectively meeting the needs o f the citizen s they are designed to address. N DI progr ams therefore have focused on e mpowering activities and helping them to ref ine their stra tegies and techniques, while th e focus of ad vocacy campaigns and policy agen das have come directly from Ar menian citizen s. NDI em ploys a g reat deal of mentoring and guided practice in its app roach, enabling more experi enced program participants to do the s ame for less experienced ones. The result is that, ov er time, NDI is ab le to play le ss of a role and allow more experienced program participants to direct activities while NDI cultivates more entry- level pa rticipants. NDI’ s capacity is growing steady as is the quantity and quality of women and youth activists. None of NDI’s programs are so dependent on NDI or costly as to preclude future sustainability. NDI Evaluation Report International Business & Technical Consultants, Inc. Page 53 o The upcoming election cycle offer the opportunity to take It’s Your Choice to the next level as a dom estic election m onitoring group. The organization has shown increasing willingness for intern al reform , devel oping new tech niques and m ethods, and a commitment to f ield qu ality, trained m onitors. The upcom ing elections will tes t this commitment and their ability to demonstrate quality performance in what are expected to be contentious elections. o NDI agrees with the importance of assistance to parties across the spectrum and has done so in Armenia since 1995. NDI and USAID policy reflects the principle of working with democratic parties acro ss the spectrum re gardless of political orientation. W hile officially there are dozens of parties in existence in Armenia, NDI focuses on num erous opposition and government-affiliated parties that are the most active in politics. NDI also noticed numerous factual errors, oversights, misrepresentations, and misunderstandings in the report. W e have chosen not to detail each , but sim ply to focus on the m ost important of them. Below, please find a summary: 1. Women candidates: The IBTCI report notes th at Armenian village councils “hav e limited power” and that NDI should expand its training of wom en candi dates to those at the city council and national level (p ages ii, 12-13, etc.). It also mentions that one wom an candidate for the Yerevan City C ouncil interviewed wa s not offered NDI’s tr aining (page 13). The Institute would like to point out that Yerevan municipal elections occurred in May 2009, prior to the May 2010 launch of NDI’s women candidate training program . The May 2012 parliamentary elections represent the first opportunity for women candidates for higher office since NDI launched th is program and will feature th e new electoral quota for wom en candidates on national party lists (the quota w as passed as the successful result of NDI￾supported advocacy efforts). NDI has, in fact, incorporated support for women candidates for national office into its n ew USAID agreem ent. The Institute still f irmly believes, however, that the re is imm ense value and op portunity in tra ining wom en to ru n at the loc al level. Village councils are the first line of government, the closest to the people, and have proven to be effective incubators for em erging politic ians. Local elections repres ent v aluable opportunities for women to gain th e political experience necessary for competing for higher office. At the local level, NDI is empowering women to take on l eadership roles, make local councils m ore representative of wom en, and provide a foundation fo r networking wom en leaders who face substantial so cial and cu ltural barriers to political participation. Demonstrating that women can overcome these ba rriers to successfully win local election is helping to encourage more wom en to run for office and build a larger pool of qualified women candidates for parties to draw on in m eeting the new quota for party lists for national office. These efforts are cons istent with NDI’s “bottom -up” approach touted in the assessment report. Starting too high and too fast coul d be counterproductive, if wom en candidates are elected to h igher office wit hout the n ecessary ex perience to govern effectively. Furthermore, past NDI experien ce in Armenia has shown that levels of electoral fraud are typically greater for positions of hi gher influence (Arm enia has yet to hold a national election since independence that has met even basic international standards or even upheld Arm enia’s own electoral legislation). N DI’s approach therefore has targeted local elections where the opportunities for gains ar e the greatest and while building a broad constituency of politica lly ac tive wom en who can help wom en candidates to protect their NDI Evaluation Report International Business & Technical Consultants, Inc. Page 54 votes at the ballot box at the national level and effectively fight for their rights in the event of electoral manipulation. 2. Training-of trainers (TOT) through leadership schools: IBTCI reports that “none of [the NDI youth leadership participants] described nor was there evidence of for malized TOT programs” (page 14) an d that “ NDI reported generally that som e NGO participan ts utilized skills acquired for the benefit of their NGO’s, but there is no concrete data to confirm this assertion” (page 5). It recomm ends throughout the report that NDI consider incorporating TOT methodologies. In fact, the In stitute respectfully notes that the sch ools already include practical follow-up sessions designed to help participants develop an d refine their public speaking a nd train ing skills. NDI incorpo rated th ese sessions s pecifically to enable participants to effectively spread the information within their respective organizations. While the suggestion that NDI participants are not prepared or provided with the skills necessary to spread their knowledge to peers and colleagues is inaccurate, the Institute agrees that m ore could be done to form alize follow-through to ensure that participants conduct training sessions f or their respe ctive organ izations in a tim ely and m ore syste matic m anner. NDI regrets that the evaluators declined to attend one of these TOT sessions and declined written information and verbal interviews with NDI staff to provide more evidence of results of NDI TOT sessions. A brief summary of som e of th e concrete data NDI aim ed to provide the IBTCI team includes the following: o At an NDI evaluation session, 16 youth lead ership participants detailed training events, forums, debates, workshops, and disc ussions they have conducted or plan to conduct for NGOs and political parties to shar e the skills and inform ation they have acquired through NDI’s leadership schools. Th us far, 10 workshops with a total of 175 participants have been organized. Som e examples include a m ember of the Free Democrats Party organizing a sem inar for 30 to 40 trade union representatives on negotiation and m anagement skills; a gra duate conducting one sem inar on conflict management for 30 students and another workshop on youth in politics for her civic organizations; a Republican Party mem ber conducting a seminar on leadership for 30 people during a party m eeting in Tsa khkadzor; one NGO m ember presenting on leadership s kills and th eory f or his organiza tion’s board m embers; one graduate organizing a workshop on public speaking fo r 20 NGO m embers; a m ember of the ANC distributing m aterials from the leader ship school to his fellow party m embers and organizing a discussion and ice-breaker se ssion with approximately 15 of them; a member of the Prosperous Arm enia party organizing a training on leadership for 12 other members of his party; and a member of the Youth Party organizing seminars on working with the media for 30 people in Araksavan. 3. Youth Leadership Acad emies: The IBTCI report asserts that “although the above described social engagement is laudable, there was no evid ence that involvement resulted in in creased political involvement, such as increased part y membership, political engagement and public gatherings” (page 12). While im pact on this leve l is not en tirely in th e control of youth and not likely to be im pacted in the short-term as an indirect result of N DI youth leadership programs. The Institute sim ilarly offered to provide th e IBTCI tea m with evidence to demonstrate some of the results of youth leadership programs based on NDI’s regular follow￾up interviews with leadership school participants and graduates. Several have joined, or even NDI Evaluation Report International Business & Technical Consultants, Inc. Page 55 created, new groups; others have been promoted within thei r existing organizations; and many are conducting trainings for their peers and colleagues based on the leadership schools’ curricula. The evaluators choose not in terview the NDI staff in charge of collecting and monitoring performance data. Some examples include the following: o One NDI participant said that the leader ship workshops gave him the courage to become director of the Young Leaders Club NGO. o One wom an participant has been assigned to be a Republican Party organizer of cultural events. o Two graduates have been nam ed a ssistants to Legal State m embers of parliam ent because of the skills they learned through the schools. o Several graduates have volunteered to be tr ainers for future academ ies, and h ave expressed interest in joining the NDI women’s and/or youth boards. o One participant, who is a m ember of the Republican Party, said that he has been selected to work at party headquarters during the upcoming parliamentary election. o Another participant is runni ng as a single-m andate candidate in Hrazdan and will b e on the Legal State’s party list. o Another par ticipant us ed the skills im parted by NDI’s school to become head of a Yeghegnadzor university student council. o One graduate, after the conclusion of the leadership school, became head of the youth council and chief of the Legal State headquarters in Ajapnyak. o One wom an said that, before attending th e school, she w as just a m ember of her organization. After graduating, she becam e a project manager and was put in charg e of an entire office. o Another woman beca me head of P rosperous Armenia’s Young W omen’s Center in Ararat after graduating. o Before attending the school, one participant was an assistant of an organization. After graduating from the school, she becam e responsible for its leadership program s and was tasked with preparing press releases for its website. o After gradu ating f rom the leade rship sc hool, a m ember of Prosperous Arm enia became a trainer for the regular workshops his party conducts for young members. 4. Advocacy projects: The IBTCI reports is cr itical of the fact that only “one youth advocacy project was implemented instead of seven” (p age 18, etc.). We are unc lear of the impression that NDI in tended to co nduct seven youth advo cacy projects, but believ e it perhaps to be a mistaken impression of NDI’s indicator that “parties or NGOs in at least seven localities have engaged in community im provement projects.” The Institute notes that it supported partners conducting exactly seven cam paigns or comm unity projects under this grant and in all 10 regional districts of Ar menia. The report corr ectly recognizes the NDI women’s conference participants and the wom en’s board condu cting advocacy around the wom en’s policy platform and the NDI youth participants in Vanadzor who i mplemented communit y advocacy projects. The report states that a sim ilar project in Vayots Dzor is exp ected to b e successful in the future, but the youth in th is community have already conducted a very successful clean-up and beautification cam paign of a local park. The report seem ingly overlooked NDI’s sm all grants program in Ar menia. NDI supported four civil society organizations conducting four sepa rate projects aimed at increasing citizen participation and representation in th e Arm enian political p rocess, particu larly am ong m arginalized group s NDI Evaluation Report International Business & Technical Consultants, Inc. Page 56 before the upcom ing pa rliamentary elections. Specifically, NDI assist ed the Civil Consent organization to equip youth with the tools necessar y to protect their vote in every d istrict in the country; the Civic Academ ic Center educ ated disabled voters in Gegharkunik, Kotayk, and Yerevan on their election rights, trained 39 disabled persons to be election m onitors, and conducted a letter-writing cam paign advocating for polling stations to be m ade m ore handicap-accessible; the Armenian Center for P olitical and International Studies encouraged pensioners and the elderly to fi ght against election abuse in ev ery district; and the U nion of Civil Society conducted m ock elections in high schools in every region throughout the country. This adds up to seven different groups that conducted regional advocacy campaigns and/or community projects. In looking at the co rrect wording of the NDI indicator, even if only considering NDI’s s mall grants program alone, Institute supported ca mpaigns in all 10 regions of Armenia. Therefore, NDI believes it sufficiently met this indicator. 5. Government outreach: The report refers to NDI’ s “inability to engag e with the Pr esidential Secretariat” and that there is “no evidence that NDI attem pted to work with another government entity to facilitate direct outreach to oppositio n groups and citizens” (page 6, etc.). While the legitimacy of gove rnment institutions has been severely dim inished by the conduct of elections in Armenia, NDI’s particular aim under this objective was to engage the staff of the executive branch in the hopes of at least facilitating comm unication and policy debate about local need s and issues . Despite pe rsistent NDI efforts, it becam e clear that the executive staff were interested in cooperation w ith NDI only if it included travel to Europe. Given the apparen t la ck of political will, NDI deem ed that th is was not a worthwhile investment of resources. NDI did, however, c onsider working with other government entities toward better engagement with citizens. NDI ruled out cooperation with the National Assembly for a variety of reasons including the controversial nature of the disputed 2007 elections, questionable political will, and pot ential f or duplica tion with other U. S. and European assistance program s. Similar rationale applied to local governm ent. NDI instead focused efforts on fostering constructive polic y debate on issues between governm ent and opposition. These pro grams have helped to facilitate more governm ent outreach to opposition groups and citizens and encouraged issue-based political dialogue that is otherwise not taking p lace without NDI assistance. . In addition, in th e strategic approach section of its proposal, the Inst itute wrote that “NDI’s appr oach would be to reinforce existing but tentativ e c onciliatory initiative s within th e executive b ranch, such as th e willingness to engage in National Press Club d iscussions with the oppos ition and rhetorical commitments to public dialogue. T he aim woul d be to build over the three years toward more systematic procedures for public consultation and feedback.” Therefore, these activities were approved by USAID as m eeting the sec ond objective and were conducted as planned. They have further provided valuable information to citizens through the media with which to use in holding political official s accountable and in inform ing the electoral decisions. As indicated by NDI’s recent poll, this type of information has been absent in Armenian political discourse to date. The IBTCI assess ment report notes that “accord ing to NDI reports, thes e programs received media coverage” (page 6-7). NDI also has kept copies of the actual media coverage for documentation. 6. Public Opin ion Resea rch: IBTCI questioned the cost-effect iveness of NDI’s focus groups. The actual costs associated with the focus groups were very m inimal, especially in working through a local vendor and com pared to the be nefits NDI has seen in comm unicating with NDI Evaluation Report International Business & Technical Consultants, Inc. Page 57 and motivating partne rs. The resear ch allowed the Institute to share unf iltered inf ormation with the government and parties, and has largel y confirmed that the issues NDI is addressing are p riorities for citizen s. For exam ple, th e fi ndings of the research were in strumental for directing the content of the Vanadzor youth’ s community initiative and the nationwide women’s policy platform. In a ddition, the nationwide public opi nion poll that NDI fielded in December 2011 serves as one of th e few source s of unbiased inform ation reflectin g what citizens think on a wide range of political, social, and econom ic subjects. The poll has already been useful for political parties tailor ing their cam paign platforms in anticipation of the May 2012 parliam entary elections, and will help citizens and civil so ciety organizations, including the women’s and youth boards, refine their advocacy campaigns going forward. 7. Election monitoring : T he Institute would like to point o ut that ther e were no nationa l elections to monitor under this grant. IYC did monitor the 2009 Yerevan municipal elections, but which were lim ited in geographic scope. Th is might account for why som e people were unaware of the effort and is also a very lim ited effort off of which to base IYC’s reputation. Furthermore, as IYC entered in to an agreem ent with US AID direc tly, and NDI does not provide any funding to IYC, some of IYC’s projects and monitoring decisions are outside the control of the Institute. It should be furthe r noted that because of the highly controversial nature of Arm enian elections and highly polari zed relations between politica l parties, IYC plays a difficult role and is often criticized by political parties. Some political parties strive to discredit the organizati on and its findings against those re sponsible for fraud while others criticizes IYC for not being able to do more to alter th e process and/or the outcom e. Therefore, interviews along these lines shoul d not be depended on as the only source for information in the determination that NDI fulfilled its programmatic objective minimally. Once again, NDI appreciates th e work of the IBTCI evaluation team and particularly of USAID/Armenia in our continued cooperation. W e hope this fee dback is useful and can be incorporated into the report. Please let us know if you have any questions whatsoever regarding the inform ation provided. W e look forward to our continued cooper ation and continued successes to come.