January 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Andrew Green, Barak Hoffman, George Welton, and Tamara Pataraia through Social Impact. Insert high-quality photograph representing the project being evaluated to replace this instruction and fill this text box. EVALUATION Mid-Term Performance Evaluation of Political Party Strengthening in Georgia (PPSG) and Parliamentary Strengthening Project (PSP) ii | P a g e Cover Photo: Youth Political Leaders School (YPLS) event in Georgia, courtesy of the International Republican Institute (IRI) iii | P a g e MID-TERM PERFORMANCE EVALUATION OF POLITICAL PARTY STRENGTHENING IN GEORGIA (PPSG) AND PARLIAMENTARY STRENGTHENING PROJECT (PSP) March 18, 2013 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iv | P a g e CONTENTS Executive Summary................................................................................................................................................................................. 1 Evaluation Purpose & Evaluation Questions .................................................................................................................................... 9 Project Background ..............................................................................................................................................................................11 NDI’s Parliamentary Strengthening Project (PSP) ........................................................................................................................12 IRI’s Political Party Strengthening in Georgia (PPSG)..................................................................................................................13 Evaluation Methods & Limitations ....................................................................................................................................................16 Findings, Conclusions & Recommendations ...................................................................................................................................20 Findings: NDI-PSP .................................................................................................................................................................................20 Evaluation Question 1: Transparency As An Intended Result .............................................................................................20 Evaluation Question 2: Capacity As An Intended Result.......................................................................................................23 Evaluation Question 3: Commitment to Sustain Results .......................................................................................................25 Evaluation Question 4: Most and Least Successful ..................................................................................................................28 Conclusions: NDI-PSP .........................................................................................................................................................................29 Findings: IRI-PPSG .................................................................................................................................................................................30 Evaluation Question 2: Transparency As An Intended Result .............................................................................................30 Evaluation Question 2: Capacity As An Intended Result.......................................................................................................31 Evaluation Question 3: Commitment to Sustain Results .......................................................................................................34 Evaluation Question 4: Most and Least Successful ..................................................................................................................35 Conclusions: IRI-PPSG .........................................................................................................................................................................35 Recommendations ................................................................................................................................................................................36 Annexes ...................................................................................................................................................................................................39 Annex I: Evaluation Scope of Work ................................................................................................................................................. 42 Annex II: Evaluation Methods and Limitations ..............................................................................................................................42 Annex III: Data Collection Instruments ..........................................................................................................................................48 Annex IV: Sources of Information ....................................................................................................................................................60 Annex V: Disclosure of any Conflicts of Interest .........................................................................................................................63 Annex VII: Evidence Matrix ................................................................................................................................................................70 v | P a g e ACRONYMS CDM Christian Democratic Movement CEPPS Consortium for Elections and Political Process Strengthening DG Democracy and Governance EU European Union FWL Future Women Leaders GD Georgian Dream GFSIS Georgian Foundation for Strategic and International Studies GYLA Georgian Young Lawyers’ Association IRI International Republican Institute KI Key Informant M&E Monitoring and Evaluation MP Member of Parliament NDI National Democratic Institute nGnI New Generation-New Initiative NGO Non-governmental organization NIMD Netherlands Institute for Multi-Party Democracy OSCE Organisation for Security and Co-operation in Europe PCC Parliamentary Communications Center PPSG Political Party Strengthening in Georgia PSP Parliamentary Strengthening Project SI Social Impact, Inc. SIDA Swedish International Development Cooperation Agency SSI Semi-Structured Interview TI Transparency International UK United Kingdom UNDP United Nations Development Programme UNM United National Movement USAID U.S. Agency for International Development USG U.S. Government YFE Youth for Free Elections YPLS Youth Political Leaders School 1 | P a g e EXECUTIVE SUMMARY USAID/Georgia contracted Social Impact, Inc. (SI) to conduct a mid-term performance evaluation of Political Party Strengthening in Georgia (PPSG) and the Parliamentary Strengthening Project (PSP), both funded through the USAID/Washington CEPPS mechanism. Implemented by the International Republican Institute (IRI), PPSG has a budget of $6.0 million and a timeline of July 2010 – July 2014. Implemented by the National Democratic Institute (NDI), PSP has a budget of $6.7 million and a timeline of September 2009 – August 2013. Specifically, this evaluation was conducted to: a) determine how effective each project has been in achieving its intended results; b) assess the opportunities for key successes to be replicated; and c) provide recommendations for PSP regarding the focus of any future programming and for PPSG regarding any necessary course corrections in the remaining years of implementation. The timeframe covered by the evaluation was the start of project implementation (PSP in 2009, PPSG in 2010) through June 2012. The evaluation was conducted over a three-week period from late October to mid￾November 2012.1 The results of the evaluation will be used by USAID/Georgia for improving ongoing interventions in the area of political processes in light of the 2012 parliamentary and upcoming 2013 presidential elections, in order to focus on the activities that are most meaningful and critical for improving the environment for political competition in Georgia and to identify directions of further assistance to Parliament. The audience of the evaluation will be USAID, particularly its Democracy and Governance (DG) Office, as well as other USG agencies conducting political processes programming in Georgia. In addition, the results of the evaluation will be useful for USAID’s current implementing partners to improve their interventions. With similar purpose, the results of the study will be shared with local stakeholders: Parliament, political parties, other donors working in this area, and interested NGOs. Finally, evaluation results will also be used for reporting purposes to Washington-based stakeholders. BACKGROUND Political change in Georgia since the Rose Revolution of 2003-04 produced a mixture of positive and negative results. Under President Saakashvili and his United National Movement (UNM) party’s dominance in Parliament, extensive political and economic reforms successfully changed the business climate for foreign investors, reduced corruption significantly, improved administrative efficiency, and brought Georgia’s military functions into line with Euro-Atlantic norms. The overwhelming dominance of the executive branch and the closed decision-making practices that marked major policy reform processes, however, kept citizens, opposition parties, and civil society on the sidelines. Protests in 2007 against the autocratic practices of UNM leaders were violently suppressed, and the snap Parliamentary elections in 2008 were widely perceived to be unfair and fraudulent by opposition parties. The disastrous 2008 war with Russia only cemented the autocratic, insular nature of the regime, the rejectionist posture of opposition parties and advocacy NGOs, and the broader apathy of citizens disgusted with both sides. Most observers would agree that the concentration of power in the executive branch exacerbated long￾standing problems of political competition and accountability. The structure of parliament was altered multiple times, but it still seemed to be more a creature of the dominant political power than an independent branch of the state. The executive branch is quite powerful, with high levels of discretion 1 The evaluation took place less than one month after the October 2012 parliamentary elections, which saw a turnover of legislative power. Note that the evaluation was originally expected to be conducted in July 2012. 2 | P a g e and a lack of meaningful internal accountability agencies. The political party system has been dominated by Saakashvili’s UNM. Two opposition parties (Christian Democratic Movement, New Rights) moved toward constructive engagement instead of public protest in the 2010-11 process of amending the Constitution and revising the Electoral Code, earning the enmity of the six parties remaining in boycott of Parliament. Otherwise, though, opposition parties have been weak and poorly-funded, with little outreach beyond Tbilisi. Indeed, Georgian political parties have historically been centered in Tbilisi, with few bothering to campaign out in the regions. The entry of billionaire businessman Bidzina Ivanishvili to Georgian politics in late 2011 shook up the party system. With financial resources that exceeded UNM’s, Ivanishvili’s Georgian Dream (GD) party grew into a coalition encompassing five additional parties: Free Democrats, Republicans, Conservatives, National Forum, and Industrialists. The coalition’s surprising victory in the 1 October Parliamentary elections changed the distribution of power at a particularly delicate constitutional moment. The NDI and IRI programs were designed to build upon opportunities for political development and address the deficiencies of Parliament and parties over the last three years. Both party institutes have many years of experience working with parties and Parliament in Georgia. PSP was originally designed to deal with the key deficiencies of Parliament, subject to political opportunities, for three years with funding of $6.7 million. The implicit hypothesis common to such projects about democratic development was that Parliament as an institution and a branch of government can be strengthened by: promoting a closer connection between representatives and citizens that could lead to more responsive elected officials; building the kind of policy knowledge and information paths needed to improve legislation; institutionalizing best practices of parliamentary procedure and openness to increase public confidence in Parliament; and enabling Parliamentary factions to conduct better oversight of executive branch activities in order to increase political accountability. PPSG, funded for three years at $6.0 million ($6.25 million with added funding), was designed to deal with some glaring deficiencies in Georgian political parties. The implicit hypothesis common to such projects about democratic development was that political competition would be enhanced by opposition parties that draw support from not just outside Tbilisi, but also in multiple regions—a stepping stone to becoming truly national parties. Political systems become more responsive to societal needs when there is a connection between parties and citizens. In addition, policy-based party platforms present citizens with much more meaningful choices than do personality-based parties, and policy coalitions can arise when parties realize they have common policy approaches. Political parties that cannot replenish or expand their leadership by recruiting and retaining youth activists run the risk of slow death through activist attrition and withering voter support. EVALUATION QUESTIONS The evaluation was based on a series of evaluation questions to elicit the results of PSP and PPSG projects to date. As requested in the scope of work, the evaluation report answers the following questions: 1. To what extent have the intended results of the PSP project been achieved in relation to Objectives 1,2, and 3 of the project? 2. To what extent have the intended results of the PPSG project been achieved in relation to Objectives 1, 2, and 3 of the project? 3 | P a g e 3. What has been the commitment of target organizations and groups (Parliament, political parties, majoritarian MPs, youth, women constituencies) to sustain the results of these projects? Specific focus on PCC activities should be made for PSP. a. Did the participants obtain needed skills and knowledge as a result of these projects? b. Were the skills and knowledge considered useful? c. Were the skills and knowledge used in the behaviors of individuals and/or organizations? 4. What are the most and the least successful interventions as perceived by main stakeholders (Parliament (with specific focus on PCC activities), political parties, relevant NGOs, media)? a. How do political parties perceive IRI polling and how effectively do they use information obtained from the polls? The evaluation also provides recommendations on how to improve the PPSG project, whether program revisions are needed for the remainder of that project, and what focus a new Parliamentary program should have. SI implemented a technical approach of standard rapid appraisal methods to answer the evaluation questions and provide recommendations. This approach was selected because of the absence of rigorous impact evaluation approaches in the programs’ designs and uncertainty over the existence and nature of baseline data. Rapid appraisal methods include:  Key informant semi-structured interviews, which are useful for gaining qualitative information on a variety of topics from knowledgeable people. The team interviewed 81 people in Tbilisi, Kutaisi, Batumi, and Telavi. Program activity participants were identified by the implementers and then selected by the team based on availability, not random selection.  Review of materials and secondary sources of data, which yield important confirmation of activities, potential baseline data, and external quantitative or qualitative evidence of activities. The team examined website visit, YouTube viewing, voting record database usage, and other data, in addition to implementer materials.  Focus groups, which can provide in-depth information on a small set of narrowly-defined topics. The team conducted four focus groups, two in Batumi and two in Telavi, in each city one focus goup of citizens and and another of regional NGO/media leaders.  Mini-surveys, which can generate information from otherwise unavailable target population segments on a small set of specific, defined topics. The team conducted mini-surveys of 25 former MPs, 10 regional journalists, and 17 youth leaders. For the evaluation of the two programs, the mixture of these methods was different for each activity, as would be expected due to differences in the nature and implementation of the activities. We did not rely heavily on quantitative data, however, due in large part to the poor data context of these projects, i.e., limited project monitoring and evaluation (M&E) plans, inapplicability or unavailability of relevant survey data, and so on. The evaluation was conducted 29 October – 15 November, 2012, with data gathering conducted in Tbilisi, Kutaisi, Batumi, and Telavi. Parallel analysis was conducted on the data gathered through these rapid appraisal methods. In this analytical approach, types of data on activities are analyzed in parallel for potential findings about that activity, then analyzed across data types to identify findings for the activity relevant to the evaluation question, and ultimately to conclusions at the activity and evaluation question levels. For example, the team analyzed semi-structured interview responses about youth debates from the debaters themselves 4 | P a g e and then from relevant party leaders; then information from IRI documents and statistics on viewership of youth debates; then narratives from focus groups with citizens, NGOs, and media; then data from a mini-survey from youth leaders; and then analyzed potential findings across these data sources to generate synthesis findings for that activity. The essence of evaluation is comparison—typically across time or geography, better still across both; an impact evaluation further attempts to compare across units that have or have not received some sort of programming treatment. In the context of the programs subject to this mid-term evaluation, however, there were significant barriers to comparing along any dimension, and the poor data context complicated the task of drawing valid conclusions. As with all evaluations, this effort faced problems with recall bias, response bias, multiple donor programs that complicated attribution, and so on; triangulation through application of multiple methods and parallel analysis of data sources helps reduce, but not eliminate these problems. By far the biggest limitation on this evaluation was the data-poor context in which these programs operated, due largely to the wholly inadequate M&E efforts by both implementers. Both programs had multiple distinct, ongoing activities under each of their three objectives. Yet, according to the documents supplied by IRI and NDI, neither plan had more than seven or eight indicators, including outputs, for the targeted three objectives. Of particular concern was that NDI did not seem to have an approved M&E plan, nor was it gathering data on its proposed indicators; USAID/Georgia did provide the team with a report showing four numerical indicators from NDI, but these did not match either of the M&E plans provided to the team by NDI’s DC and Tbilisi offices. For its part, IRI was indeed tracking performance on its very limited set of indicators, with considerable detail and timeliness. Both implementers, NDI in particular, would not infrequently list activities under different objectives. For the evaluation team, then, it was never completely clear whether we had identified all of the activities under each program’s objectives, nor could we rely on program documents to identify the intended results of the programs. The evaluation was further burdened by three types of constraints on the team’s ability to gather useful data. First, the timing of the evaluation was problematic: the evaluation timeframe ended in June 2012, but the team was in the field more than four months later. Even worse, the turnover in power as a result of the October parliamentary elections meant that many opposition party leaders were in government or Parliamentary leadership, and many former ruling party MPs lost their seats. Combined with the physical move of Parliament from Tbilisi to Kutaisi, the team’s ability to schedule time or even locate desired interviewees was severely diminished. Second, the demand-driven, highly-responsive nature of party and parliamentary assistance results in programs characterized by activities tailored specifically for a certain target group or by activities that are short-lived because of the need to be opportunistic. The end result is that these sorts of programs often have a large number of distinct activities and even variations of the same activities, which complicates analysis of their effects. Finally, the common practice of leaders or activists attending multiple trainings on different topics from the same implementer, or even trainings on similar topics from different implementers, muddies attribution. FINDINGS The evaluation team identified the following key findings for the PSP:  Faction-Journalist Interaction: Journalists enjoyed improved access to MPs and were able to gather policy information. Journalists credited NDI with being able to get MPs to attend the events. 5 | P a g e  Majoritarian MP Constituency Outreach: Only a small number of MPs were actively engaged in outreach activities like MP websites, communication and media training, and constituency responsiveness, but that small number valued the experience positively.  Parliamentary Communications Center (PCC): A small number of MPs participated in the “Your Parliament” video debates, but that small number gave very positive assessments to the training and the experience; only four of the 29 participating MPs returned to the current Parliament. Viewership data is lacking, but Georgian TV viewing hours, the audiences for participating stations, positive responses in focus groups, and Internet views are all low.  Future Women Leaders (FWL): Participants valued the training and applied the lessons and skills to their professional benefit. Committee heads claimed that the committees experienced improved efficiency and effectiveness.  Parliamentary Internships: Participants valued the training and applied the lessons and skills to their professional benefit. MPs and committee staff valued the work of the interns and rated the program higher than other internship programs.  Public Meetings: NDI sponsored 38 meetings attended by an average of 130 people; an average of 17 NGOs were invited, and 7 media outlets were present at the one-third of the meetings where any media were present. Consensus among interviewed MPs, parliamentary staff, NGOs, regional media, and citizens was that the meetings opened legislative topics to public scrutiny.  NGO Liaison Office: The Office increased publicly-available information on draft bills and other documents on Parliament’s activities, worked with approximately 100 NGOs, and hosted 80 meetings. MPs and NGO leaders agreed that the Office provided access between the two groups that was, in the past or in the absence of the Office, a rare occurrence.  Voting Records Database: The database is the only original source of MP voting information and bill text available to the public.  Key Committees for Institutional Reform: Committee staff and NGOs valued trainings on institutional reform issues, but the varying set of issues, the changing set of people engaged on any issues, and the one-off nature of the trainings did not result in committee staff capacity.  Commitment to Sustain Results: The full scope of Parliament’s commitment was in the early stages of discussion at the time of the evaluation. The new Parliament leadership is committed to continuing cooperation with NDI, although the new parliament building had no space set aside for the PCC.  Most and Least Successful: In relative terms of more successful, the FWL and Parliamentary Internship activities were universally valued, did not generate recommendations for serious changes, and provided benefits beyond the participants. The NGO Liaison Office was valued by users yet seemed to benefit only the NGOs, while the Voting Records Database is unique yet most useful as source information for a secondary website by Transparency International (TI)- Georgia. Relatively less successful activities include the majoritarian MP constituency outreach and PCC, in both cases due to the small number of involved MPs and lack of evidence of engaged citizens. 6 | P a g e The evaluation team identified the following key findings for the PPSG:  Regional Party-Building: Regional party-building activities did help parties expand their party structure into regions, engage in more public activities, and train party activists. The Christian Democratic Movement (CDM), Republican Party, and New Rights Party developed and implemented regional party-building strategies appropriate for their support base and resources, while others were too small or poorly-financed to apply training knowledge. The Republicans and Free Democrats were more active outside Tbilisi than in the past, as were the generally more active UNM and to a lesser extent CDM.  Polling and Focus Groups: Polling and focus group results from IRI (and NDI) are the only such information sources available to parties. Party representatives reported using the research results on policy issues to shape party positions and/or messaging.  GFSIS Political Academy: Participants benefitted in terms of policy knowledge and enhanced career prospects, and practiced ‘peaceful coexistence’ with their political rivals. The parties themselves benefitted when participants informally duplicated seminars for party leaders and activists.  Youth Political Development: Youth participants may have attended both youth party-building activities and YPLS (and YFE), and so could not necessarily distinguish between the two modes. Participants valued training on media, party communication, organizational development, as well as interaction with fellow party activists in the region and even rival youth activists. YPLS participants have moved or will be moving up into regular party structures or even government positions. Regular party structures coordinate more with youth wings where they exist, particularly the Free Democrats, Republicans, and the National Forum, and youth wings have also become more integrated with regular party committees.  Youth Debates: Some debate teams constructed their argumentation and evidence with regard to existing party positions or ideology, e.g., the Republicans and UNM, and so the televised debates had the effect of presenting party positions to viewers, while the CDM and Labour debate teams’ strategies informed the party’s messaging approach. For the debaters, the training on debating skills, argumentation, camera presence, and audience engagement were highly valued and put to use in party activities; although some participants had relevant skills before going into the youth debates, the camera presence and audience engagement skills were new and tremendously valuable to their professional careers. Parties quickly saw the potential of debates as a recruiting tool, and party leaders from CDM, Free Democrats, Republicans, and National Forum were actively engaged in selecting the teams via internal competition and subsequently working with their teams to hone skills and develop policy positions.  Commitment to Sustain Results: Junior coalition parties the Republicans and National Forum intend to continue building the regional bases of their parties. Among the junior coalition parties and extra-Parliamentary parties like CDM, the youth wings are viewed as critical for the ongoing functioning of the parties. Parties like the Republicans and National Forum recognize the need to develop a fuller party platform based on policy positions, but they do not have the internal capacity currently. No party is committed to sustaining any of these activities on its own, due almost entirely to the lack of financial resources for all parties but UNM.  Most and Least Successful: GFSIS Political Party Academy participants unanimously and overwhelmingly placed a high value on the knowledge and experience they had, as did the parties. Every one of the debaters the team interviewed thought it was a positive experience 7 | P a g e that s/he would recommend to others in the party. Parties valued the skills and experience of the debaters, too, including tasking past debaters with training of subsequent teams. Paradoxically, one of the most successful activities – regional party-building – turned into the least successful, because of political events completely beyond the control of IRI. Some of the more substantial parties leading into the campaign season had been eager practitioners of the skills learned in training, and they were having organization success as a result. For the Republicans and Free Democrats, though, the rise of the GD Coalition meant that the merging of regional offices in summer undermined party capacity and limited each party’s influence over the offices. Looking past the timeframe of this evaluation to the October election, we see that although CDM and New Rights adapted and implemented the lessons from training to suit their needs and resources, both parties were wiped out in the election results. CONCLUSIONS The evaluation team concluded that PSP’s record in enhancing transparency in Parliament is decidedly mixed. While activities like encouraging faction-journalist interaction and operating the NGO Liaison Office were useful and successful, other activities like public meetings and the voting records database were less so; of much more limited success were the majoritarian MP constituency outreach and the Parliamentary Communications Center. There is no question that the PSP has built useful capacity in a range of different areas. NDI’s program has been particularly effective in its longer-term training programs for women leaders and parliamentary interns. Both programs not only have attracted highly qualified participants, but have also helped create more effective committees in parliament. By contrast, short-term trainings and workshops for majoritarian MPs and committees appear to be having a less visible impact, due in part to the severely truncated role that Parliament plays in governing Georgia. It is clear that although we can talk about enhanced transparency, strengthened capacity, and the relative success of various PSP activities, the reality is that none of these activities could be self-sustaining at this time. Participants and partners are certainly more than willing to continue cooperation with NDI, though. The evaluation team concluded that the IRI program increased transparency between parties and citizens in terms of direct engagement in the regions and to a lesser extent indirect engagement via the youth debates. Transparency was not a major intended result of the program, however. With respect to capacity, the team concluded that PPSG did strengthen party, youth wing, and individual capacity in identifiable ways, but a lack of data prevents us from confirming some hoped-for outcomes like increased youth wing membership. We concluded that regional party-building activities did help parties expand their party structure into regions, engage in more public activities, and train party activists, but the rise of merged GD Coalition offices undermined this capacity. On the policy side, GFSIS participants clearly gained policy knowledge, benefitting themselves and their parties. The team concluded that the polling and focus group results have been useful to parties as a source of information for shaping messaging, particularly for specific policy issues. Finally, the team concluded that youth party￾building did help youth wings expand their activities and provide important skills for youth leaders that are useful for the regular party organization, but there is no reliable evidence that youth wing memberships increased. The youth debates helped youth leaders gain skills in debating, argumentation, and working on camera. Debaters subsequently used these skills in media interactions for their parties and enjoyed promotion into the regular party. 8 | P a g e As with PSP, though, the evaluation team concluded that parties are committed to sustaining the results of IRI activities, but they can only do so in continued cooperation with IRI. RECOMMENDATIONS The evaluation team unanimously and strongly felt that neither program developed meaningful M&E plans. Future projects to work with political parties or Parliament should have much more robust results-based management systems. Another main recommendation is that a new Parliamentary program needs to focus more of its attention on the sustainability of its interventions. Operationally, we suggest that over the next few years, NDI transition from running programs directly and focus support more on trainings, i.e., the NGO Liaison office, the voting records database, the PCC, and MP websites could all be run by stakeholder organizations over time. The FWL and Parliamentary internship programs are strong performers that could be continued as is. The majoritarian MP constituency outreach was not a successful activity, but the changed political environment may be an opportunity for this activity to have much more effect. Finally, working with individual committees on short-term, demand-driven public discussions was similarly less useful but may be of more use in the new climate. The team’s main recommendation for the current PPSG project is that it should review and re-focus some activities but could maintain most of the existing activities to better effect in the current political climate. While regional party-building efforts were paradoxically successful and not, due to the dynamics of electoral coalitions, the change in power means that the former opposition parties need assistance as much as they did before. The issues for PPSG are whether to set up a two-tiered approach to assisting parties, as well as how to grapple with the future of the junior parties in the GD Coalition. A major legislative issue will be campaign and party finance reform, which should be addressed by IRI under an objective not subject to this evaluation. The GFSIS Political Party Academy and the youth debates are strong activities that could be continued with minor revisions. Similarly, IRI’s dissemination of polling and focus group results is valuable to party partner and should continue. In addition, the team recommends that IRI use its polling efforts to gather data for M&E purposes. Finally, the youth party-building needs to be re-focused to eliminate redundancy and over-tailoring of activities. 9 | P a g e EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE USAID/Georgia contracted Social Impact, Inc. (SI) to conduct a mid-term performance evaluation of Political Party Strengthening in Georgia (PPSG) and the Parliamentary Strengthening Project (PSP), both funded through the USAID/Washington CEPPS mechanism. Implemented by the International Republican Institute (IRI), PPSG has a budget of $6.0 million and a timeline of July 2010 – July 2014. Implemented by the National Democratic Institute (NDI), PSP has a budget of $6.7 million and a timeline of September 2009 – August 2013. Specifically, this evaluation was conducted to: a) determine how effective each project has been in achieving its intended results; b) assess the opportunities for key successes to be replicated; and c) provide recommendations for PSP regarding the focus of any future programming and for PPSG regarding any necessary course corrections in the remaining years of implementation. The timeframe covered by the evaluation was the start of project implementation (PSP in 2009, PPSG in 2010) through June 2012. The evaluation was conducted over a three-week period from late October to mid￾November 2012.2 The results of the evaluation will be used by USAID/Georgia for improving ongoing interventions in the area of political processes in light of the 2012 parliamentary and upcoming 2013 presidential elections, in order to focus on the activities that are most meaningful and critical for improving the environment for political competition in Georgia and to identify directions of further assistance to Parliament. The audience of the evaluation will be USAID, particularly its Democracy and Governance (DG) Office, as well as other USG agencies conducting political processes programming in Georgia. In addition, the results of the evaluation will be useful for USAID’s current implementing partners to improve their interventions. With similar purpose, the results of the study will be shared with local stakeholders: Parliament, political parties, other donors working in this area, and interested NGOs. Finally, evaluation results will also be used for reporting purposes to Washington-based stakeholders. EVALUATION QUESTIONS The evaluation is based on the series of evaluation questions to elicit the results of PSP and PPSG projects to date. As stated in the scope of work, the evaluation report must provide answers to the following questions: 1. To what extent have the intended results of the PSP project been achieved in relation to Objectives 1,2, and 3 of the project? 2. To what extent have the intended results of the PPSG project been achieved in relation to Objectives 1, 2, and 3 of the project? 3. What has been the commitment of target organizations and groups (Parliament, political parties, majoritarian MPs, youth, women constituencies) to sustain the results of these projects? Specific focus on PCC activities should be made for PSP. 2 The evaluation took place less than one month after the October 2012 parliamentary elections, which saw a turnover of legislative power. Note that the evaluation was originally expected to be conducted in July 2012. 10 | P a g e a. Did the participants obtain needed skills and knowledge as a result of these projects? b. Were the skills and knowledge considered useful? c. Were the skills and knowledge used in the behaviors of individuals and/or organizations? 4. What are the most and the least successful interventions as perceived by main stakeholders (Parliament (with specific focus on PCC activities), political parties, relevant NGOs, media)? a. How do political parties perceive IRI polling and how effectively do they use information obtained from the polls? The evaluation also provides recommendations on how to improve the PPSG project, whether program revisions are needed for the remainder of that project, and what focus a new Parliamentary program should have. 11 | P a g e PROJECT BACKGROUND Political change in Georgia since the Rose Revolution of 2003-04 produced a mixture of positive and negative results. Under President Saakashvili and his United National Movement (UNM) party’s dominance in Parliament, extensive political and economic reforms successfully changed the business climate for foreign investors, reduced corruption significantly, improved administrative efficiency, and brought Georgia’s military functions into line with Euro-Atlantic norms. The overwhelming dominance of the executive branch and the closed decision-making practices that marked major policy reform processes, however, kept citizens, opposition parties, and civil society on the sidelines. By the second half of 2007, growing frustration with the President and the ruling party generated large and peaceful public protests, which were violently broken up by security forces. As the crisis continued, Saakashvili called snap Parliamentary elections in early 2008 that produced a landslide victory for UNM, but were widely considered by opposition parties to have been fraudulent. Leaders from eight opposition parties subsequently refused to accept their mandates in Parliament or simply boycotted Parliament entirely. Ongoing protests and the disastrous summer 2008 war with Russia prevented Parliament from meeting until early 2009. Most observers would agree that the concentration of power in the executive branch exacerbated long￾standing problems of political competition and accountability. The structure of parliament was altered multiple times, but still seemed to be more a creature of the dominant political power than an independent branch of the state. The executive branch is quite powerful, with high levels of discretion and a lack of meaningful internal accountability agencies. UNM’s majoritarian MPs elected in 2008 were largely neophytes with limited connection to their putative districts, selected by UNM for loyalty or prestige. Parliamentary committees were neither effective watchdogs of executive agencies nor sources of legislative initiatives. In the aftermath of the events of 2007-08, the ruling party’s closed processes seemed to close even further: UNM limited the number of party officials empowered to make decisions, while opposition parties and civil society engaged politics through protest or a limited number of media outlets. Despite this, the Parliamentary leadership has supported efforts to professionalize staff, open up Parliamentary processes to the public, and encourage MP outreach to citizens. The planned move of Parliament to a new building in Kutaisi, however, was viewed by many as the physical exile of the legislature, both as an institution and a focal point for protest. The political party system has been dominated by Saakashvili’s UNM. Business fears of selective punishment resulted in UNM dominance of campaign financing, spending almost 90 percent of total campaign funding for all parties in past elections. Two opposition parties (Christian Democratic Movement, New Rights) moved toward constructive engagement instead of public protest in the 2010- 11 process of amending the Constitution and revising the Electoral Code, earning the enmity of the six parties remaining in boycott of Parliament. Otherwise, though, opposition parties have been weak and poorly-funded, with little outreach beyond Tbilisi. Indeed, Georgian political parties have historically been centered in Tbilisi, with few bothering to campaign out in the regions. This is due in part to limited resources available to the opposition parties and the personality-based nature of some parties, as well as a preoccupation with capital-based politics, but the result was that UNM was able to dominant regions almost unchallenged; this dominance translated into limitations on and disincentives for citizens in the regions to pursue their political preferences. The entry of billionaire businessman Bidzina Ivanishvili to Georgian politics in late 2011 shook up the party system. With financial resources that exceeded UNM’s, Ivanishvili’s ‘Georgian Dream’ (GD) party grew into a coalition encompassing five additional parties: Free Democrats, Republicans, Conservatives, National Forum, and Industrialists. The coalition’s surprising victory in the 1 October Parliamentary elections changed the distribution of power at a particularly delicate constitutional moment; note that 12 | P a g e CDM, New Rights, or any other party did not meet the threshold for representation in Parliament. The 2010 Constitutional amendments shifted considerable authority from the President to the Prime Minister, but the shift does not take place until the Presidential election slated for October 2013, which means an uncertain period of co habitation for the two power blocs. The NDI and IRI programs were designed to build upon opportunities for political development and address the deficiencies of Parliament and parties over the last three years. Both party institutes have many years of experience working with parties and Parliament in Georgia. NDI’S PARLIAMENTARY STRENGTHENING PROJECT (PSP) PSP was originally designed to deal with the key deficiencies of Parliament, subject to political opportunities, for three years with funding of $6.7 million; note that NDI also implemented Parliamentary strengthening programs prior to 2009. The implicit hypothesis common to such programs about democratic development is that: Parliament as an institution and a branch of government could be strengthened by promoting a closer connection between representatives and citizens that could lead to more responsive elected officials; building the kind of policy knowledge and information paths needed to improve legislation; institutionalizing best practices of parliamentary procedure and openness to increase public confidence in Parliament; and enabling Parliamentary factions to conduct better oversight of executive branch activities in order to increase political accountability. This implicit development hypothesis is captured in the original four objectives of the project, but the later addition of election-based activities added two more objectives: 1. Improve Parliamentary outreach and constituency relations; 2. Improve the quality of policy debate in Parliament; 3. Enhance Parliamentary transparency and institutional reform; 4. Strengthen Parliamentary oversight; 5. Strengthen the capacity of local organizations to monitor electoral activities throughout the election cycle; and 6. Contribute to safeguarding electoral integrity through a thorough international pre-election assessment. The Georgia-specific assumptions underlying the hypothesis are that MPs would be willing to engage in these activities, that the norms of behavior promoted by these activities would be adopted and continued by the MPs, and that external actors would be able to engage more freely and regularly with Parliament. As requested in the scope of work, this evaluation was only concerned with the first three objectives:  Activities under Objective 1 include working with majoritarian MPs, committees, and Parliamentary factions. After a period of extended discussion with UNM leadership in Parliament, NDI worked with willing majoritarian MPs to create member websites, develop constituency outreach strategies, and provide training on communication skills. A related add-on activity is the PCC, run by NDI partner Internews, which is a fully-functional studio capable of producing content for free distribution to media outlets; that activity began after extended discussions with Parliament leadership. The primary activity of the PCC was to produce debates between MPs on issues of district interest identified by media and NGOs from the relevant region; the segments were generally embedded by regional media outlets in their normal news broadcast. Another activity was to facilitate interaction between MPs and journalists as a way to increase media access to Parliament and help elected officials develop relationships with 13 | P a g e journalists. A fourth major activity was facilitation of public meetings by Parliamentary committees, including communication training for staff. Although the meeting venue was often Parliament itself, the point of the activity was to encourage norms of openness and access. Finally, the NGO Liaison Office facilitated the sharing of draft legislation and other documents with NGOs, as well as set up meetings between NGOs and key MPs or committee members; note that this activity could be, and was in various NDI reports, listed under other objectives.  Activities under Objective 2 to strengthen the policy capacity of Parliament include the FWL program and the Parliamentary Internship program. The FWL program identifies an annual cohort of 20-30 female staffers, who undergo a year-long training program in communication, public speaking, strategic planning, and project management. The internship program is highly￾competitive, assigning interns to committees or factions for research and analytical tasks.  Objective 3 includes such disparate activities as cooperating with key committees on institutional reform and establishing an online voting records database. The former activity overlaps to some extent with Objective 1’s public meetings activity, but it focuses on a small number of committees engaged in Constitutional amendments, Electoral Code revisions, or Parliament’s own rules of procedure. The project would facilitate the public meeting, share draft materials with NGOs and media, and encourage public engagement. The database was the first of its kind in Georgia, a record of voting on the third reading of proposed legislation, searchable by bill number. Other donor programs are complementary or tangentially related to PSP. UNDP has been implementing a program to professionalize the Parliament staff, including standard position descriptions, performance reviews, and training on parliamentary procedure. The Westminster Foundation for Democracy has implemented activities to work with key committees dealing with the adoption of program budgeting, help journalists understand budget documents, and deal with the logistics of moving Parliament to Kutaisi. The EU and OSCE have in years preceding this program run programs to help committees with institutional reform. Finally, ongoing EU and USAID programs deal with local government, ministries, and Parliamentary committees on program budgeting. IRI’S POLITICAL PARTY STRENGTHENING IN GEORGIA (PPSG) PPSG, funded for three years at $6.0 million, was designed to deal with some glaring deficiencies in Georgian political parties. The implicit hypothesis common to such projects about democratic development was that political competition would be enhanced by opposition parties that draw support from not just outside Tbilisi, but also in multiple regions—a stepping stone to becoming truly national parties. Political systems become more responsive to societal needs when there is a connection between parties and citizens. In addition, policy-based party platforms present citizens with much more meaningful choices than do personality-based parties, and policy coalitions can arise when parties realize they have common policy approaches. Political parties that cannot replenish or expand their leadership by recruiting and retaining youth activists run the risk of slow death through activist attrition and withering voter support. This implicit development hypothesis is built into the project objectives, along with two additional objectives addressing election capacity and NGO dialogue: 1. Build organizational capacity of parties regionally and nationally, and build parties’ connections to constituents; 2. Strengthen parties’ policy orientation and ability to build coalitions; 3. Encourage development of young political leaders in and outside party structures; 14 | P a g e 4. Increase election proficiency of political parties; and 5. Address the overall enabling environment through dialogue among parties, government leaders, and civil society stakeholders. The Georgia-specific assumptions underlying the hypothesis are that parties have both the will and the resources to undertake regional party-building efforts, that party leadership will not feel threatened by the emergence of regional or youth leaders, and that parties accept the need to develop policy-based platforms to compete for votes. As requested in the scope of work, this evaluation was only concerned with the first three objectives:  Activities under Objective 1 include training tailored to the needs of the target party: designing an internal structure, developing a strategic plan, instituting internal democracy, improving center-region communication, and grassroots activities. On occasion, training topics would cover election or campaigning issues, which would more properly fit under objectives not evaluated here. IRI also held events specifically for CDM’s women’s organization. IRI also conducted public opinion surveys and focus groups on political topics, and it shares results with party leaders so that parties have better information for development of policy positions.  Activities under Objective 2 include the GFSIS Political Party Academy and facilitation of discussions on cooperation by party partners. The GFSIS academy is a 10-month program open to up to 30 activists nominated by 16 parties, with no party allowed more than two participants; note that the program was shortened to seven months in 2012 because of the election campaign. Originally begun as a UK Embassy-funded activity, GFSIS has structured a program based on weekly meetings that takes participants through political, economic, and social topics in depth. Participants are encouraged to ask questions, debate, and interact generally. The seminars are taught by topic experts, with some topics covered over a period of 3-4 weeks. IRI worked closely with party partners among the opposition to facilitate discussions on common policy issues, most importantly electoral code reform. Objective 2 also addresses the capacity of parties to build coalitions. The main effort by IRI was to support the activities of the opposition parties in the Election Code Working Group, which is more properly listed under Objectives 4 or 5 (as IRI did in its quarterly report narratives), neither of which were included in this evaluation. IRI did have an indicator defined for ‘consensus-building forums,’ but did not report against it.  Activities in Objective 3 fall under the general heading of “youth political development,” but the nature of the activities is quite varied. The two main activities are the YPLS and the youth debates; the Youth for Free Elections (YFE) multi-party activity also falls in this heading, but youth leader interviewees who mentioned similar activities did not see them as separate from other IRI-supported activities, thus there are serious attribution problems. Moreover, the YFE activities should more appropriately have been listed under Objectives 4 or 5. The YPLS events, implemented by New Generation-New Initiative (nGnI), are tailored to the needs of the youth organizations and participating youth activists, covering such topics as leadership, project management, civic education, advocacy, social networking, event planning, and so on. Since 2009, YPLS has run six-week programs in Gori, Kutaisi, Zugdidi, Telavi, and Rustavi for 25 participants, 16 from partner parties and 9 open seats for others; there are now roughly 120 alumni. IRI also held a small grant competition among YPLS participants for community development project funding. The youth debates were a distinctly different activity and yet drew on many of the same youth leaders. Designed as a political debate competition between teams from party youth 15 | P a g e organizations, participants were selected by their regular and youth organization leaders for training in argumentation, debate tactics, on-camera and audience engagement skills, and policy issues. Topics were recommended by national and regional NGOs and media, and televised on Kavkazia TV (with distribution to regional media outlets). The scoring process was adjusted with successive rounds, moving from a three-person panel of experts to inclusion of in-studio audience voting, to tie-breaking by means of call-ins. Other youth training events covered democratic development, election and campaign strategies, ideology, fundraising, and the role of youth wings in regular party structures. Note again that some training topics are more appropriately found under an objective not evaluated here; this includes the Youth for Free Elections (YFE) cooperative events that for many participants was amalgamated with other youth political development activities. Note that IRI has been working with political parties in Georgia since before the Rose Revolution. NDI currently works with political parties with SIDA funding, but only with those parties without regular IRI relationships. Finally, NIMD offers assistance to parties on strategic planning, leadership recruitment, and multi-week Democracy Schools. 16 | P a g e EVALUATION METHODS & LIMITATIONS USAID/Georgia requested a mid-term performance evaluation of both PSP and PPSG to determine respective achievements to date, implementation progress, perceived value, potential sustainability, and beneficiary engagement. As noted in TIPS #11, Introduction to Evaluation at USAID, a mid-term evaluation is primarily a formative evaluation intended to furnish information that will guide program improvement; in this case, there are also strong elements of process and summative evaluations too. EVALUATION TEAM The SI Team was composed of Andrew Green, Barak Hoffman, George Welton, and Tamara Pataraia. Team Leader Dr. Green is SI’s DG Practice Leader, with extensive applied research experience in the Europe & Eurasia region. Technical Expert Dr. Hoffman is the Executive Director of Georgetown’s Center for Democracy & Civil Society and has deep experience in applied research on political institutions and parties. Dr. Welton was the team’s Evaluation Expert, and he has many years of experience conducting research in Georgia. Dr. Pataraia was the Local Evaluation Expert, with 12 years of experience conducting applied research in Georgia on a variety of democracy and governance issues. The SI Team utilized GeoWel Research for implementation of the focus group protocols, mini-surveys, summaries of discussions, and English translation of documents. In addition, GeoWel Research handled logistical functions in Kutaisi, Batumi, and Telavi. EVALUATION DESIGN SI implemented a technical approach of standard rapid appraisal methods to answer the evaluation questions and provide recommendations. This evaluation relied primarily on key informant interviews, focus groups, review of materials and secondary sources of data, and mini-surveys. The mixture of these methods was different for each activity, as would be expected due to differences in the nature and implementation of the activities. We did not rely heavily on quantitative data, however, due in large part to the poor data context of these projects, i.e., limited project M&E plans, inapplicability or unavailability of relevant survey data, and so on. • Materials and other information were obtained from implementers, direct participants, and secondary sources. Materials reviewed for PSP included, for example, project documents, parliamentary documents on faction activities, and news stories. Materials reviewed for PPSG policy trainings included project documents, parliamentary documents on policy discussions, and party documents on policy platform development. The review yielded important information about some baseline values and progress related to the programs. Data in this category were most useful for evaluation questions 1, 2, and 3a. The team gathered some quantitative data from sources outside the implementing partners’ M&E plan, e.g., call in votes and texts for youth debate shows, growth in youth groups, hits on MP websites, use of MP website discussion forum, use of the voting records database, and questions asked in parliamentary question hour. • The details of semi-structured interviews of key informants differed depending on the activity, the specific person, and ‘causal distance’ from the activity; some key informants were involved in multiple activities within a program or even across the two programs, so interview protocols for some types of key informants consisted of multiple modules of questions. The interview protocols have specific questions designed to elicit concrete examples of knowledge utility and utilization. The desired interviewees were not selected randomly, but were identified through 17 | P a g e the implementers, reputation, or existing contacts of evaluation team members. The list of desired interviewees included people from organizations that did not benefit directly from the evaluated activities, in an attempt to obtain unbiased information. The team interviewed 81 people in Tbilisi, Kutaisi, Batumi, and Telavi. Data in this category were useful for all evaluation questions. • Focus groups of citizens and of NGOs and media covered legislator-citizen interactions (majoritarian MP constituency outreach, public meetings, regional party building) and party￾citizen idea communication (youth debates, Your Parliament) provided in-depth information on how those individuals and organizations view the relationship of political actors and society. Two focus groups were held in each Batumi and Telavi. Data in this category were most useful in questions 1, 2, and 3c. • Mini-surveys of target population segments unreachable through other means provided some information not otherwise available. The team conducted mini-surveys of 25 former majoritarian MPs, 10 regional media, and 17 regional youth leaders. Data in this category were most useful in questions 1, 2, and 3abc. As noted in TIPS #5, Using Rapid Appraisal Methods, triangulation through the use of multiple methods reduces the effects of bias and can increase the validity of findings. Parallel analysis was conducted on the data gathered through these rapid appraisal methods. In this analytical approach, types of data on activities are analyzed in parallel for potential findings about that activity, then analyzed across data types to identify findings for the activity relevant to the evaluation question, and ultimately to conclusions at the activity and evaluation question levels. For example, the team analyzed semi-structured interview responses about youth debates from the debaters themselves and then from relevant party leaders; then information from IRI documents and statistics on viewership of youth debates; then narratives from focus groups with citizens, NGOs, and media; then data from a mini-survey from youth leaders; and then analyzed potential findings across these data sources to generate synthesis findings for that activity. For evaluation questions 1 and 2 on intended results of the two projects, the evaluation team utilized the concepts of transparency (the provision of and access to information) and capacity (imparting knowledge that enables action) to evaluate the various activities under each project. SI adopted this approach in part because of the inadequacy of the implementers’ M&E plans, as well as the the need to employ a common analytical narrative. WORK PLAN This mid-term evaluation focused on two complex projects with multiple standalone activities. In conjunction with uncertainty over the availability of party leaders and new MPs for interviews, as well as the availability and willingness of former MPs for interviews, the evaluation’s logistics were quite complicated and changed on a last-minute basis. The evaluation took place from 29 October – 15 November. The team worked in Tbilisi during the first week and travelled to Kutaisi at the start of the second week. Given that the new Parliament was in the midst of its first session, the team experienced difficulties scheduling a sufficient number of interviews to be productive in Kutaisi for more than three days. One team member travelled on to Batumi to conduct a small number of interviews and monitor the focus groups, while the remaining team members returned to Tbilisi. At the start of the third week, one team member travelled to Telavi to again conduct a small number of interviews and monitor the focus groups; it should be noted that in both Batumi and Telavi, a number of our desired interviewees were targeted to be focus group participants, which reduced the number of semi-structured interviews to be held. The full timeline of evaluation activities can be seen in the charts in Annex B. 18 | P a g e LIMITATIONS AND CONSTRAINTS The essence of evaluation is comparison—typically across time or geography, better still across both; an impact evaluation further attempts to compare across units that have or have not received some sort of programming treatment. In the context of the programs subject to this mid-term evaluation, however, there are significant barriers to comparing along any dimension, and the poor data context complicates the task of drawing valid conclusions. There are some noteworthy limitations on comparison, and hence attribution.  First, recall bias was a problem noted by all team members. Political party or parliamentary programs often include activities that were conducted in years before the current program periods, or that were conducted by other implementers in addition to IRI or NDI, e.g., NIMD Democracy Schools. A similar problem is that participants in multiple training activities may be blending their experiences into a composite memory, e.g., youth leaders may have received training on youth party-building, participated in YPLS, participated in youth debates, and even attended training for regional party-building.  Second, response bias is a common problem for program evaluations. For example, a former participant may give the interviewer positive remarks about a strategic planning retreat at a nice resort because she wants it to continue in order that her staff can enjoy a nice weekend vacation. The team was acutely aware that party representatives and Parliamentary leaders were fully cognizant that a negative evaluation could mean the end of a project that provided them with needed training.  Third, selection bias in the form of contacts provided by the implementers can mean an evaluation team only hears from people with positive experiences. The most effective approach to combating bias is to use multiple sources of data to triangulate on an evaluation issue, as is often accomplished through qualitative reliability matrices. By combining information found in documents or interviews from multiple sources, any one piece of biased data would not skew the analysis. Another approach that pertains specifically to interviews is the inclusion of key informants from organizations that do not directly benefit from the evaluated program and the use of questions about specific examples of knowledge use. By far the biggest limitation on this evaluation was the data-poor context in which these programs operated, due largely to the wholly inadequate M&E efforts by both implementers. Both programs had multiple distinct, ongoing activities under each of their three objectives subject to this evaluation, and yet, according to the documents supplied by IRI and NDI, neither had indicators for each of their activities: NDI has nine distinct activities and seven indicators, including outputs, while IRI has six distinct activities and eight indicators, including outputs. Of particular concern was that NDI did not seem to have an approved M&E plan at all, nor was it gathering data on its proposed indicators. USAID/Georgia did provide the team with a report showing four numerical indicators from NDI, but these did not match either of the M&E plans provided to the team by NDI’s DC and Tbilisi offices. For its part, IRI was indeed tracking performance on its very limited set of indicators, with considerable detail and timeliness. Both implementers, NDI in particular, would not infrequently list activities under different objectives. For the evaluation team, then, it was never completely clear whether we had identified all of the activities under each program’s objectives, nor could we rely on program documents to identify the intended results of the programs. The evaluation was burdened by three types of constraints on the team’s ability to gather useful data. First, the timing of the evaluation was problematic: the evaluation timeframe ended in June 2012, but the 19 | P a g e team was in the field more than four months later. Even worse, the turnover in power as a result of the October parliamentary elections meant that many opposition party leaders were in government or Parliamentary leadership, and many former ruling party MPs lost their seats. Combined with the physical move of Parliament from Tbilisi to Kutaisi, the team’s ability to schedule time with or even locate desired interviewees was severely diminished. Second, the demand-driven, highly-responsive nature of party and parliamentary assistance results in programs characterized by activities tailored specifically for a certain target group or activities that are short-lived because of the need to be opportunistic. The end result is that these sorts of programs often have a large number of distinct activities, which complicates analysis of their effects. Third, the common practice of leaders or activists attending multiple trainings on different topics from the same implementer, or even trainings on similar topics from different implementers, muddies attribution. 20 | P a g e FINDINGS, CONCLUSIONS & RECOMMENDATIONS FINDINGS: NDI-PSP Because NDI’s M&E plan is in essence non-existent, and because individual activities under the PSP do not fall neatly under objectives, it makes more sense to discuss the results of all program components that fall under the three objectives. We categorize the activities according to their impact on transparency and capacity, the two broad outcomes the PSP is trying to impact. Evaluation Question 1: Transparency As An Intended Result Enhancing transparency, at the most basic level, is achieved by improving the provision of and access to information by citizens, media, and NGOs. NDI undertook seven activities under Objectives 1-3 of the PSP to achieve this general result. NDI’s activities clearly improved access to information for the media, NGOs, and the general public. Faction-Journalist Interaction Our interviews and focus groups confirmed that NDI’s activities to facilitate interaction between MPs and journalists improved the flow of information from the media about Parliament’s activities. First, journalists to whom we spoke believed that most MPs, especially those in the UNM leadership, would not have attended the meetings if NDI had not hosted them. Second, these journalists also felt the meetings improved their access to MPs and provided them with policy information they hitherto were unable to collect. For example, a journalist from Telavi said that, “The program was very effective. I did not have any chance to communicate with MPs other than activities organized by NDI, as the parties refused to communicate with [my media outlet] at all.” Similarly, another journalist from the Kakheti region claimed the meetings offered a rare chance for him to ask questions to the representatives of political parties. Majoritarian MP Constituency Outreach Majoritarian MP outreach efforts provided useful training for the small number of majoritarian MPs that actively participated. Majoritarian MPs in Georgia, in general, need a significant amount of assistance with conducting effective outreach, and NDI provided useful training in this area. At the same time, only a small number of majoritarian MPs took these activities seriously. This small group of majoritarian MPs evaluated NDI’s activities positively and demonstrated that they produced publicly-available information. The clearest evidence that most majoritarian MPs did not take NDI’s outreach trainings seriously is their weak use of MP websites. NDI designed and built websites for all 75 majoritarian MPs. The websites for most majoritarian MPs were in place for about two years during the PSP. Over that time, the average majoritarian MP had 3,200 website visits, and the median was approximately 2,600.3 Given the poor content on most websites, the small amount of traffic is not surprising. Close to one￾third of websites had no information at all, while only just over one-fifth had anything more substantive than biographical information, such as the activities in which the majoritarian MP was engaged. 3 Forthcoming research funded by the Heinrich Böll Foundation found that Georgian blogs received at least 3000 visits per month, despite the small community and public awareness of bloggers. http://www.livestream.com/heinrichboellfoundationsouthcaucasus/video?clipId=pla_dcd6737d-1573-4e5a-9390- 9b0e55148b38&utm_source=lslibrary&utm_medium=ui-thumb 21 | P a g e Majoritarian MPs did not maintain their websites. Close to 60 percent had never updated their sites, while just over 10 percent had updated them within the last six months as of October 2012 (See Tables C and D in Annex VI). Finally, in a survey we conducted of 29 majoritarian MPs who participated in NDI training, only 11 said they had used the website facility at all, and only four said they had updated the sites regularly. Parliamentary Communications Center The debates hosted by the PCC generated new and useful public information about policy issues. Yet, the PCC’s impact on enhancing the quality of debate inside Parliament and increasing the visibility of Parliament’s work has been weak. On the surface, it appears that the PCC was a successful program. The PCC produced 173 debates between MPs, and 20 regional TV channels carried a total of 1,554 broadcasts of them (see Table D in Annex VI for a list of the stations). The MPs who participated and with whom we spoke had a very positive assessment of the debates, as did approximately 15 members of the media familiar with the program with whom we spoke. These individuals were pleased that the debates focused on regional and local issues, which receive relatively little coverage, and that the PCC reached out to local NGOs and news outlets to solicit questions for the debates. Although we lack firm data on the number of people who watched the debates, closer examination of the program suggests (but does not prove) that the PCC’s impact was fairly limited. First, while NDI trained 49 MPs (35 UNM and 14 opposition) to debate on the PCC, only 29 participated in a debate (18 UNM and 11 opposition). Just under 25 percent of UNM’s majoritarian MPs debated even once. As a result, each UNM MP who participated took part in an average of nine debates, while each opposition MP conducted 16 segments. Second, while we lack firm data on viewership, the evidence we do possess suggests that it was low. According to surveys conducted by the Caucasus Research Resource Center, outside of Tbilisi around half of the population has local TV reception, but only about one-quarter watch local TV every day. We have no way of knowing how many watch the nightly news on which the PCC program was aired. People with whom we spoke who worked for stations carrying the debates (Channel 9, Mega TV, and Rioni TV) generally claimed to have small audiences. Moreover, in two focus groups in Batumi, one with 8 local NGO leaders and media representatives, and a second with citizens, only one person had seen and remembered the Your Parliament segment.4 The most systematic data we obtained on the number of people watching PCC debates is from the PCC website and YouTube. According to data supplied by NDI, there have been 35,380 viewings of the PCC debates online. While the total number sounds impressive, the average is just about 200 per debate. Public Meetings Data obtained through interviews and focus groups made clear that regional media, local NGOs, Parliamentary staff, MPs, and citizens valued the public meetings that NDI held. In particular, consensus among those we interviewed was that the meetings opened legislative topics to effective public scrutiny. According to data supplied by NDI, the organization has sponsored 38 public meetings over the course of the PSP (See Table F in Annex VI). The largest number of public meetings was about the government budget, which accounts for just less than one-third of the total number of meetings. It is not obvious to the team why NDI coded PCC training (mainly on communications and camera work) as public meetings since these were trainings that targeted MPs and their bureau staff. The public meetings have produced at least one clear impact, Parliament’s decision to drop a proposed amendment to increase the number of MPs. According to NDI’s Fiscal Year 2012 Quarter 1 report (8): 4 This only includes focus groups from Batumi because the PCC debates had very low coverage in Telavi. 22 | P a g e NDI…and the Parliament of Georgia launched public consultations on a constitutional amendment that would increase the number of MPs from the current 150 to 190 following the 2012 parliamentary elections… Immediately following the public meetings, parliament decided not to proceed with the proposed increase of legislators from 150 to 190, citing negative public feedback received at the public meetings as one of the major reasons. NDI cannot claim full credit for this outcome. Nevertheless, UNM MP and former chair of the Budget and Finance Committee Zurab Melikishvili agreed that NDI’s meetings helped contribute to it. While other public meetings did not produce the same clear policy impacts as those on the proposed Constitutional changes, they generated substantial local interest. An average of 130 people attended the public meetings according to NDI’s data. An average of 17 NGOs was invited to each of the meetings as well. The media covered about one-third of the meetings, with approximately seven media outlets covering those events. NGO Liaison Office Individually and in focus groups, we spoke with approximately 15 NGOs that were aware of the NGO Liaison Office. Our findings suggest that the office has undoubtedly increased publicly available information on bills and other documents about Parliament’s activities. MPs, Parliamentary interns, and committee staff with whom we spoke corroborated these findings. For example, interns told us that the NGO Liaison Office routinely contacted them for information about their committees. Crucially, participating MPs and NGOs agree that NDI’s convening power provided access for the latter to the former, which they otherwise would have found difficult to obtain during the last Parliament. NDI has been very inclusive, working with approximately 100 NGOs. NGOs have participated in about 80 meetings hosted by NDI during the course of the PSP, according to the data NDI provided. Every one of the 15 NGOs with which we spoke said that NDI’s newsletters (which approximately 400 NGOs and individuals receive) are the only public source of information on Parliamentary activities, especially committee meetings. The NGO Liaison Office website receives about 100 visitors per day. The most visible impact we were able to identify was on campaign finance. According to Transparency International, the NGO Liaison Office played a central role in pressuring UNM to revise a campaign finance law that participating NGOs felt would restrict their political activities. Voting Records Database The voting records database is one of the most successful parts of the PSP. The website is the only source of MP voting information and bill text available to the public. It is also the area in which the PSP has the largest external impact. On the negative side, many people we interviewed said the website is difficult to use and therefore does not receive much traffic. However, Transparency International￾Georgia, in a completely separate effort, has scraped the information from the website and combined it with other publicly available information on MPs to create a very useful website, ShenMartav. The website has been averaging 1,100 visits per month over the past three months. We view this as a positive external impact because, on its own initiative, a very influential NGO that has a wide reach is employing data that NDI is gathering to broadcast to a wider audience. As a result, the database is providing useful information beyond the audience NDI is specifically targeting. 23 | P a g e Evaluation Question 2: Capacity As An Intended Result We define capacity as “imparting knowledge to enable action.” PSP attempted to strengthen the capacity of Parliament with five activities. Majoritarian MP Constituency Outreach NDI held nearly 40 trainings for majoritarian MPs and their staff. The trainings included communication and media skills, discussed how to respond to constituency requests, and provided direct outreach efforts such as websites that NDI largely designed. While those who participated in the activities found them useful, especially training and outreach skills, the demand for NDI’s program was low. NDI provided a list of 49 majoritarian MPs from the previous Parliament5 who had been involved in the outreach programs. Of the 25 we were able to reach, 6 only four regularly participated in NDI majoritarian MP outreach meetings, such as those with NGOs or journalists. One partial exception to the aforementioned trend was NDI’s work with Unity for Justice, a coalition of MPs from small parties that served as an opposition faction in the previous Parliament. Unity for Justice reached out to NDI for assistance, and the organization was very responsive to the coalition’s requests. According to coalition members with whom we spoke, NDI assisted them in setting up weekly strategy meetings, facilitated contacts with NGOs, and helped them revise the Election Code. Overall, the MPs in the coalition and their staff claimed that the faction was more effective as an opposition block to UNM in Parliament than it would otherwise have been if it did not receive support from NDI. Parliamentary Communications Center The PCC encountered many of the same problems as the majoritarian MP outreach programs. Most of the MPs who participated thought the trainings were useful, especially in developing communication and media skills. Yet, as noted above in our discussion of the PCC’s activities, there was a low level of interest in the debates. In addition, the program has lost nearly all of its capacity. Of the 29 MPs who participated in the program in the previous Parliament, only four won re-election in October 2012. Future Women Leaders The FWL program has an excellent reputation in Parliament. Approximately 40 women participated in the program during the evaluation period, and NDI has held about 20 trainings. The program is attracting women with high-ranking staff positions in Parliament, such as office heads, lawyers, and chief committee specialists. MPs and committee staff, including David Janiashvili, Chief of Staff to the former Speaker of Parliament, also praised FWL for increasing professionalism among committees. MPs and staff claimed that participants in the program routinely gain skills that improve the efficiency and effectiveness of committee work. Participants in the FWL program with whom we spoke praised it enormously:  A committee Chief of Staff said that because of the program she became a more confident public speaker, improved her conflict management skills, and learned how to give positive feedback, constructive criticism, and positive reinforcement. She is confident that she runs her office far more effectively than she did before the program.  Another committee Chief of Staff stated that because of the FWL program, she learned critical thinking and analytical skills, how to motivate her staff, and became a much better communicator. She claimed that NDI’s trainings were more effective than most because they seek out well-qualified trainers and they are conducted over a fairly long time period of time. 5 All but four of them lost their seats in the October 2012 election. 6 The vast majority of telephone numbers for the other 24 MPs were out of service. 24 | P a g e  A committee Chief Specialist claimed that improved public speaking skills was the major benefit of the program for her. She believes that she is a far more interesting, interactive, and engaging speaker because of what she learned in the program and that her skills are beneficial for career advancement.  A committee Leading Specialist said that the main benefits of the program for her were skills in management, leadership, communications, conflict resolution, and public speaking. She provided a clear example of how the program benefitted her professionally as well. This FWL participant was a trainer of trainers for the 2012 elections, and said she ran the activity much more effectively because of what she learned in FWL. She participated in the program because of the positive experience another woman in her office had in the program. The keys to the success of the FWL program appear to be its length, NDI’s strict attendance policy, and the quality of the trainers. Participants said the program typically contains nine trainings over a seven￾month period and that participants do not receive a certificate of completion if they miss more than two trainings. Participants said the length of the training permitted the program to cover a range of activities in a structured manner, and many thought the trainings should be longer. They compared the FWL very favorably to shorter workshops and one-off activities that rarely provided useful skills. That NDI hired very effective trainers was also a common sentiment among participants. Parliamentary Internships The internship program is also among the strongest elements of the PSP. Highly motivated individuals apply to the program, interns perform substantive work for key committees, and they benefit substantially from the program, especially in terms of professional development and job placement. The program is very competitive and it has an excellent reputation within Parliament, government, NGOs, and in social science departments at Ilia and Tblisi State Universities. We were able to speak with six of the 26 of the interns who NDI has selected over the course of the program. The overwhelming perceptions of the program among interns with whom we spoke were:  Committees are grateful to have NDI interns  Program has an excellent reputation  Program has a very competitive application process  NDI interns are the only ones in Parliament who do substantive work  Trainings were very useful, especially for communication skills Views of interns among MPs and committee staff with whom we spoke were equally positive. MPs and staff on the Self-Government and Mountainous Regions, Human Rights and Civil Integration, Legal Affairs, and Foreign Relations Committees valued the work of the NDI interns and they are eager to have them in their offices. They also viewed the program as competitive and agreed that NDI interns are much more capable and useful than those who Parliament hires itself. The benefits of the program for the interns are clear as well. Out of 26 interns, eight are currently working for NGOs, six are working for the Georgian government or Parliament, and four are pursuing graduate degrees in European universities. Key Committees for Institutional Reform According to data supplied by NDI, the organization held approximately 20 trainings for committees and NGOs, mainly for the Legal Issues Committee. These meetings and trainings covered a range of issues, such as budget, laws on political unions, and hate speech. According to the staff who participated in 25 | P a g e these activities, the trainings on Rules of Procedure in Parliament and changes to the Electoral Code were the most useful. The impact of NDI’s work with key committees in Parliament is mixed based on the results of our interviews with committee staff and our review of NDI’s activities in this area. On the one hand, according to our interviews with staff members, NDI is responsive to their needs and interests. Moreover, the staff appreciates trainings and NDI’s efforts to reach out to journalists and NGOs. At the same time, the impact on increasing overall professionalism in committees is less clear, especially in comparison to the FWL and intern programs. A number of factors contribute to the relatively less than successful outcome of NDI’s work with key committees for institutional reform as compared to other aspects of the PSP. Some of the factors are within NDI’s control, while others are outside it. As we mentioned above, since the last Parliament often did not engage in the policymaking process, many committees lack the opportunity to influence policy outcomes and debates in Parliament. We do not consider it reasonable to hold NDI accountable for this deficiency. However, NDI did not focus its activities in as structured a way as it did with the FWL and intern programs. The downside of NDI’s responsiveness to the interests and needs of committees is that NDI often responded to short-term committee requests, such as debate around a specific bill, with one-off events. The result was a diverse set of activities that failed to aggregate as a clear objective. Evaluation Question 3: Commitment to Sustain Results The commitment of target groups and organizations to use the skills and knowledge they learned through participating in the PSP varies by activity and correlates strongly (but not identically) with the success of each component of the program. We summarize our findings below by level of commitment. The strongest commitment to continue participating in NDI’s activities that we observed was among participants in the FWL and Parliamentary intern programs. As described in the previous section, we were able to find numerous examples of how individuals who participated in these programs are using what they learned in their jobs, such as skills in public speaking, leadership, and conflict management. We also noted that participants in these programs, especially the Parliamentary interns, saw clear career benefits as a result of their participation in NDI programs. In addition, we found that the programs produced external benefits as the staff in Parliament who did not participate in the FWL or intern programs value the skills they provide to others. Committee staff with whom we spoke made clear to us they value these programs enormously and very much would like for them to continue. The next strongest commitment to using skills and knowledge the PSP provided are among the NGOs that worked with the NGO Liaison Office and the journalists with whom NDI worked. It is clear to the team that the journalists and NGOs who participated in the PSP thought NDI provided very valuable opportunities for them to meet MPs and politicians they otherwise would have difficulty contacting. We heard numerous examples of how their involvement in the PSP allowed them to work more effectively: journalists from 24 Hours, Gurjaani TV, and Lagodheki TV, for example, told us explicitly that NDI provided them with a very rare opportunity to directly discuss policy issues with MPs. Commitment from these two groups to using the skills and knowledge they learned is perhaps somewhat less than among the FWL and interns because the latter two benefitted more from the PSP than former two. The commitment of staff on Key Committees on Institutional Reform7 to using the skills and knowledge they learned through the PSP appears to be somewhat weak. This is more because they lack the 7 Budget and Finance, Legal Affairs, and Self-Government and Mountainous Regions Committees 26 | P a g e opportunity to do so than because they feel the trainings had little value. To the contrary, they appreciate the training opportunities greatly. The problem, as explained above, is that the committee system in the Georgian Parliament remains underdeveloped due to the subordinate status of the legislative branch compared to the executive branch in Georgia. Were this to change (for example, if the current Parliament seeks to perform greater oversight over the executive branch), greater opportunities to use the skills and knowledge they gained would arise. Under this scenario, we believe the trainings NDI provided for these committees would have a much stronger impact on the effectiveness of their work than it does at the moment. Commitment to majoritarian MP outreach and the PCC is nearly non-existent in the current Parliament, although discussions between the Parliament’s leadership and NDI were taking place while the evaluation team was in the field. This is not because participants failed to appreciate these activities. Rather, it is because almost all MPs who were involved in these programs in substantive ways lost their seats. For example, of the 32 MPs who taped a PCC debate, only four won seats in the current Parliament. As a result, there is little institutional memory of these programs in the current Parliament and hence few existing skills to use. Commitment to Self-Sustaining Activities We found very weak commitment to sustaining any activities in the PSP without the continued support of NDI or another external donor. In our interviews, we routinely asked if interviewees believed PSP activities would continue if NDI stopped supporting them. As we detail below, we were unable to find many people who believed this would happen. In part, the weak commitment of program beneficiaries to continue the activities without NDI’s assistance results from NDI failing to incorporate an evolution to self-sustaining activities into the program. For example, NDI could have been working with Parliament to maintain the voting records database and/or MP websites. Instead, NDI performed these activities itself. NDI could have been working with local NGOs to transition from running the NGO Liaison Office itself to having a local organization, such as the Georgian Young Lawyers’ Association GYLA, undertake these responsibilities. Of course, the practicality of achieving sustainability depends on the ability of local partners to take over. In the case of the NGO Liaison Office, its effectiveness was definitely due in large part to the influence that NDI holds more generally, and its resultant ability to harass MPs into engaging in situations where they lack inherent interest. Therefore, it is difficult to envision how all of the functions of the NGO Liaison Office could be easily taken over by a local organization. At the same time, this challenge is not true of the voting records database, the maintenance of which is a largely technical exercise. Our investigations did seem to suggest that NDI was not always making strenuous efforts to hand off its activities to other organizations. More focused activity in this area could greatly enhance the sustainability of the program. At the current time, Parliament or other stakeholders are unlikely to revive the programs that NDI was running on its own. For example, NDI is not maintaining the voting records database and it remains unclear whether the information technology (IT) department in Parliament will do so. As a result, at the moment, there is no voting records database for the current Parliament. Similarly, NDI is not running the NGO Liaison Office. We were unable to find any individuals in Parliament or outside organizations that are planning to do so. Along the same lines, the existence of NDI interns who are better qualified than those hired directly by Parliament does not appear to be influencing the Parliament to adopt a more systematic approach to recruiting and managing its interns. Similarly, while committee staff with whom we spoke generally 27 | P a g e valued the skills NDI interns brought to their committees, we found no evidence to suggest they would seek out individuals with these skills were NDI to end the internship program. In addition, our discussions with MPs and committee staff provided no evidence that Parliament will organize for itself programs such as the FWL or trainings for committee staff. Further, interviewees gave no indications that MPs will organize public meetings similar to the ones NDI arranged, although it is possible this may occur given that parliamentary elections have become far more competitive. Finally, due to lack of institutional memory, majoritarian MP outreach activities and programs like the PCC are unlikely to revive in the absence of NDI’s assistance (although parties may undertake such activities for their own MPs). Commitment to Cooperate with NDI While the weak commitment of PSP beneficiaries to self-sustaining activities may be disappointing (yet unsurprising), on the positive side, the team believes that the interest in cooperating with NDI is as strong in the current Parliament, if not stronger, than it was in the last Parliament. As a result, the evidence we collected suggests that the PSP is likely to have a greater impact in the current Parliament than the previous one. The results of the October 2012 election suggest that Parliament could become more influential in the policymaking process, and that Georgian Dream and UNM will need to reach out to the media and NGOs to explain their policy preferences. This is an auspicious environment for the NGO Liaison Office and outreach to journalists. Faction-journalist interactions during the last Parliament would not have had occurred without NDI facilitation. Similarly, the NGO Liaison Office has provided services that past Parliaments did not, and NGOs valued the information and access to MPs the office facilitated. UNM was not committed to either program, and its MPs attended the meetings only because NDI organized them. However, former opposition parties were strongly committed to these activities. FWL and Parliamentary internship participants benefitted enormously from these programs, as explained in detail in this evaluation. In addition, we have made clear that participants in the programs improve the work on the committees on which they serve and that both programs have an excellent reputation, especially the internship program. If NDI were to end the programs, it would have a negative impact on the effectiveness of committees. At the same time, it is clear that if NDI were to maintain the programs in their current forms, they would continue to attract highly qualified applicants. The voting records database is an extremely useful product. Parliament would welcome NDI’s continued assistance in running the website and, as explained above, MPs and staff are uncertain whether Parliament would manage the database itself if NDI stopped running the program. It is also possible that domestic NGOs, such as GYLA and TI-Georgia, would step in to fill this void, although neither had any plans to do so at the time of the evaluation. Majoritarian MP constituency outreach and PCC had limited active engagement from UNM MPs, but participating opposition MPs welcomed the opportunities. Our analysis and findings suggest that majoritarian MPs in the current Parliament would appreciate these opportunities more than their counterparts in the previous Parliament. Most important, parliamentary elections have become far more competitive. All of the new GD majoritarian MPs beat incumbents from UNM. As a result, they understand they could lose their seats if they fail to live up to the expectations their constituents have for them. In addition, many new majoritarian MPs lack outreach and media skills, have little political experience, and are not well known in their constituents. For these reasons, current majoritarian MPs are likely to place greater value on the outreach and media skills offered through these NDI programs 28 | P a g e than their counterparts in the last Parliament. NDI was examining this issue while we were conducting the evaluation. Public meetings would not, to a large extent, have been held without NDI facilitation. NGOs and media at both the national and regional levels valued the events, while UNM majoritarian MPs participated only because NDI organized the activities. For the same reasons that current majoritarian MPs may be more interested in constituency outreach than those in the previous Parliament, NDI is likely to find a greater commitment to public meetings among majoritarian MPs in the existing Parliament. In addition, NGOs and the media appreciate the opportunities NDI provides for them to attend committee meetings. We have no evidence that majoritarian MPs would organize these meetings on their own, although it is too early to state this definitively as the new Parliament had only begun its term at the time of the evaluation. Key committees on institutional reform valued the capacity training that NDI provided to their staff. The committee staff with whom we spoke made it clear to us that not only would they would like these opportunities to continue, but also that they would like NDI to provide longer training courses and more of them. If NDI were to provide trainings to committee staff similar to those it provides to FWL participants and interns, demand to participate in them would be high. Evaluation Question 4: Most and Least Successful The terms ‘most’ and ‘least’ are here defined in relative, not absolute, terms. The evaluation team found value in all of the main activities in PSP, but relative differences can be distinguished in light of the recent elections. More Successful For the reasons detailed below, the more successful activities of the PSP are the FWL and internship program. First, each person with whom we spoke about these programs had only positive things to say about them. Second, no one recommended serious changes to the programs to make them more effective. Rather, the modal suggestion we received was not to alter either program at all. Third, that the benefits of the program extend to those who did not participate was clear as Parliamentary staff provided numerous examples of how participants of both programs improved the effectiveness of their committee’s work. Fourth, participants in both programs, and especially the interns, could clearly demonstrate how their careers improved because of the program. Less Successful The two less successful elements of the PSP were majoritarian MP outreach and the PCC. Most majoritarian MPs were indifferent to NDI’s outreach efforts, and we found no visible external benefits of NDI’s efforts. One area that was particularly unsuccessful was the majoritarian MP websites. Close to two-thirds of MPs never updated their websites after NDI designed them and only about 20 percent posted anything more than biographical information on the websites. The PCC suffered many of the same problems as the majoritarian MP outreach. While those who participated found it useful, demand to take part among MPs was low. Second, while we are unable to verify the number of people who had seen the debate broadcasts, the only metric we do possess that would enable us to determine whether people are watching the debates are views of them on the PCC website and YouTube. These data suggest that very few Georgians saw the debates on the Internet. Ultimately, though, we lack adequate data to allow us to make firm claims about the impact of the program on the public. 29 | P a g e CONCLUSIONS: NDI-PSP PSP’s record of enhancing transparency in Parliament is decidedly mixed. The faction-journalist interaction was viewed positively by both sides, particularly by regional media. Similarly, the NGO Liaison Office has quite successfully brought NGOs into legislative processes on a frequent basis, with working relationships between MPs and NGO leaders developing as a result. Of more moderate success for transparency would be the public meetings, which brought together MPs, citizens, NGOs, and media in the regions for discussions about political topics. Beyond public interest, though, it is not clear that the information was being used to further policy positions. The voting records database is also an invaluable resource, but the limited search feature meant that the information was most valuable as source data for a separate effort by an outside party. Of much more limited success for transparency would be the majoritarian MP constituency outreach, which was clearly an activity primarily for UNM backbenchers. With the exception of one or two clear outliers, few MPs availed themselves of NDI’s assistance in a meaningful way. The PCC was similar, whereby a small number of enthusiastic participants obscured limited interest by MPs, and the actual effect on citizens cannot be ascertained. There is no question that the PSP has built useful capacity in a range of different areas. NDI’s program has been particularly effective in its longer-term training programs, including the FWL and Parliamentary internship programs. Both programs not only have attracted highly qualified participants, but have also helped create more effective committees in Parliament. By contrast, short-term trainings and workshops for majoritarian MPs and committees appear to be having a less visible impact. Overall, while the individual activities in the PSP have been successful in building capacity, they have failed to reach the broader overall objective of having committees play a more central role in Parliament. However, this is an outcome that is beyond NDI’s control. Throughout the period under evaluation, Parliament remained far less influential in Georgian politics than the executive branch. As a result, MPs rely less on committees than in countries where the legislative branch has a more active policymaking role. This is not an outcome that NDI can (or perhaps even should try to) influence. At the same time, NDI might have been able to achieve more substantive results if it had designed its activities in this area similar to its work with FWL and internship program participants. A key strength of the FWL and internship programs, was longer-term engagement around a structured set of activities by a defined group of people. The result was that participants in these programs feel they gained a set of very useful skills. By contrast, the wide range of committee trainings involved a changing set of people engaging on a varying set of issues. This demand-driven approach led to a set of useful individual activities that failed to aggregate to a clear impact, other than a vague increase in the capacity of committee staff. The FWL program is without question one of the strongest components of the PSP. The program has demonstrably strengthened the skills of participants, increased their professional opportunities, and had a positive impact in the offices where participants are working. NDI deserves an enormous amount of credit for the success of the internship program. First, NDI established a very rigorous set of selection criteria consisting of an application, a test, and an interview. Interns viewed the interview as particularly challenging because they were interviewed serially by 10-15 committee staff. Second, NDI works very hard to ensure that committees provide interns with useful work and that the committees treat the interns well. NDI will not send interns to committees that lack 30 | P a g e these commitments. Third, NDI works very closely with the interns and provides them with very useful trainings and job placement. The trainings for key committees for institutional reform have broadly increased the capacity of committee staff, but the trainings have not led to overall more effective committees. It is clear that although we can talk about enhanced transparency, strengthened capacity, and the relative success of various PSP activities, the reality is that none of these activities could be self-sustaining at this time. Participants and partners are certainly more than willing to continue cooperation with NDI, though. FINDINGS: IRI-PPSG The Political Party Strengthening in Georgia program by IRI was on the whole an effective program, but as with the Parliamentary Strengthening Program by NDI there are key activities that were vulnerable to the dynamics of Georgian political development. While PPSG did act to promote the flow of information about parties to citizens and to enhance the capacity of both parties and party activists, the mid-term effectiveness of some activities was undercut by the rise of the Georgian Dream coalition. As mentioned previously, here we are defining ‘intended results’ in terms of the broader concepts of ‘transparency’ and ‘capacity,’ rather than performance against the monitoring and evaluation plan or other more abstract concepts. Note again that IRI’s monitoring and evaluation plan did not adequately cover the range of activities or the range of effects, but the program did keep track of its performance for the few indicators it used. The evaluation team also calls attention to activities under Objective 2 that address the ‘ability to build coalitions’ portion of that objective. The main effort by IRI was to support the activities of the opposition parties in the Election Code Working Group, which is more properly listed under Objectives 4 or 5 (as IRI did in its quarterly report narratives), neither of which were included in this evaluation. IRI did have an indicator defined for ‘consensus-building forums’ that was reported in the very first quarter of the project, but at the same time the activity reporting shifted to Objective 4. Evaluation Question 2: Transparency As An Intended Result We examined the performance of the IRI program with respect to how well it enhanced transparency in terms of the provision of and ease of access to information about parties and their policy positions. The two activities that are most relevant here are regional party-building (Objective 1) and the youth debates (Objective 3). Regional Party-Building Regional party-building activities did indeed act to increase the flow of information from parties to citizens. IRI provided training and other informal assistance to parties to assist them in developing party structures that were more broadly based across various regions in Georgia; IRI largely met its targets for these indicators:  Constituent outreach activities (outcome) – 14 actual activities versus 15 planned in FY11, with FY12 performance distorted by the campaign period  Party structures strengthened (outcome) – 16 actual versus 16 planned for two-year period We found that among the parties participating in the training, the CDM, the Republican Party, and New Rights Party developed and implemented regional party-building strategies appropriate for their support base and resources, while others were too small or poorly-financed to apply training knowledge; note 31 | P a g e that we also came across some evidence in interviews with leaders from the Free Democrat’s youth wing that the party had implemented a regional strategy, too, but we were unable to meet with any of that party’s leadership with knowledge of this activity due to the change in political power. Examples cited in interviews and implementer documents include the Republicans’ regional offices, CDM’s ‘zone coordinators,’ and National Forum’s regional issue ‘passports’ for party leaders. These developments had an effect on increasing party interaction with citizens and communicating information about party positions. Party representatives reported more activities in the regions and better center-region communication, for example, New Rights would coordinate with regional activists to hold rallies in towns where they had no office, and regional offices of the Republicans would organize public activities to demonstrate the party’s commitment to an area; again, IRI performance monitoring showed this as well. Our mini-survey of 10 regional media outlets demonstrated quite clearly that some of the participating parties had been much more publicly-engaged in the regions than in previous election campaigns8, while external experts and NGO leaders made similar observations. In particular, we found that the Republicans and Free Democrats were more active outside Tbilisi than in the past, as was to a lesser extent CDM, all of which published or updated outreach strategies. UNM was also mentioned by experts and NGO leaders, but it did not participate in IRI’s trainings on this topic. Contradictory evidence was found in the Telavi focus group, where participants cited NIMD’s partiebi.ge website as improved outreach by parties. Both UNM and CDM had generally been the most active parties in the regions. Youth Debates To a lesser extent, the youth debates also acted to increase the flow of information about party positions and to create a new flow between youth and party leaders. We found that some debate teams would construct their argumentation and evidence with regard to existing party positions or ideology, e.g., the Republicans and UNM, and so the televised debates had the effect of presenting party positions to viewers; it is also possible that viewers assumed that any policy position taken by any debate team represented that party’s official position, but we have no evidence of that. For topics that had not been addressed by a youth wing’s party, we found that the CDM and Labour debate teams’ strategies informed the party’s messaging approach. Broadcast viewership of the youth debates, as with the ‘Your Parliament’ videos, is difficult to determine. Data on YouTube views indicates that eleven debates were viewed a total of 2802 times, or an average of 255 time each (see Table G in Annex VI). Evaluation Question 2: Capacity As An Intended Result We also examined the performance of the IRI program with respect to how well it built the capacity of political parties and their youth wings in terms of imparting knowledge to enable action. The most relevant activities here are regional party-building, polling and focus groups (Objective 1); GFSIS Political Party Academy (Objective 2); and youth party-building and youth debates (Objective 3). Regional Party-Building Regional party-building activities did indeed act to strengthen the capacity of parties. IRI provided training and other informal assistance to parties to assist them in developing party structures that were more broadly based across various regions in Georgia; IRI’s reporting showed that they exceeded the output indicator for individuals receiving training (3268 versus 2200 across the last two years; 30% were women). We found that among the parties participating in the training, the CDM, the Republican Party, and New Rights Party developed and implemented regional party-building strategies appropriate for their support base and resources, while others were too small or poorly-financed to apply training 8 Mini-survey respondents were asked to rate party outreach in 2012 compared to 2008 on a scale of 1 (a lot worse) to 5 (a lot better), resulting in an average score of 4.3 32 | P a g e knowledge; note again our lack of evidence directly from Free Democrats leadership. CDM took advantage of the multiple leaders within the party and then adopted a ‘zone coordinator’ approach to development. New Rights established a network of party activists that served as advance teams for travelling party rallies. The Republicans opened field offices in regions outside Tbilisi and Batumi, their traditional bases, and put in place communication structures to integrate the offices with headquarters; the Free Democrats did similarly according to interviews with the leaders from the party’s youth wing. As noted in the above ‘transparency’ discussion, party representatives, youth leaders, regional media outlets, NGOs, and external experts all reported that the Republicans and Free Democrats were more active outside Tbilisi than in the past, as was UNM and to a lesser extent CDM. Polling and Focus Groups IRI’s dissemination of polling and focus groups results fills an important gap in parties’ knowledge – much as NDI’s polling does, although with different donor funding – because these are the only such information sources available to parties, with the exception of UNM and Georgian Dream. Party leaders from the Republicans, CDM, National Forum, and the Social Democrats claimed that the political party ratings were not useful, but the evaluation team believes this to be a reaction to the NDI poll problem.9 IRI missed its output indicator target for assistance on development of programmatic platforms and poicy agendas (actual 11 organizations versus 16 planned), but exceeded its outcome indicator target for parties publishing or updating outreach strategies based on polling data (11 actual versus 7 planned). Party representatives reported using the research results on policy issues to shape party positions and/or messaging, although GD Coalition parties were more limited in this due to coalition dynamics. Both party representatives and external experts noted that IRI and NDI have different approaches to dissemination, with IRI taking a more behind-the-scenes approach. GFSIS Political Party Academy IRI had no indicators for this activity, but the GFSIS Political Party Academy is highly valued by both participants and party leaders for the policy capacity it imparted. We found that participants benefitted in terms of policy knowledge and enhanced career prospects. Participants were particularly appreciative of the training on policy analysis, economic development, and foreign policy, often singling out the seminars led by Papava and Rondeli specifically. All six of the participants we interviewed reported promotions within the regular party and/or policy-relevant positions in government following the October elections. In addition to the new skills and knowledge that participants brought to their party-based activities, we found that the parties themselves benefitted when participants would informally duplicate seminars for party leaders and activists, and larger parties like UNM, the Free Democrats, and the Republicans invited some policy lecturers to party-hosted seminars. An unintended consequence of the Academy is one that was mentioned by every participant interviewed: they learned peaceful coexistence with their political rivals. In fact, each Academy cohort has set up a Facebook group page that is used extensively by alumni participants. 9 In July and early September 2012, NDI released the results of surveys on political attitudes that were strongly criticized by opposition party leaders, who believed that the low level of support that the results showed for the GD Coalition severely understated their actual support. The results of the August survey released in September showed UNM with 37% support, the Georgian Dream coalition with 12% (down from 18% previously), ‘don’t know’ 22%, ‘refuse to answer’ 21% (up from 16% previously), ‘no party’ 3%, and other parties totalled 5%. Every party leader interviewee responded to the question about the utility of IRI polling and focus groups with an extended complaint about NDI’s polls. In fairness, although there are survey techniques for gathering better quality data on political preferences, NDI’s surveys were of generally high quality, certainly could have been an accurate reflection of attitudes, and were conducted before the prison abuse scandal erupted; the problem may have been one of messaging. 33 | P a g e Given that participants were already high-achieving youth leaders or prominent activists within their parties, the evaluation team asked participants the implicit counterfactual about what value-added the program had for them: the common answer was that although they would have had similar professional opportunities, they were exposed to a range and depth of topics new to them, as well as the positive experience of interacting with political rivals; the responses echoed complaints the team heard from external experts about the quality of university education in Georgia. Youth Political Development As discussed in the project background section, youth political development is a general heading for a variety of activities, including the youth debates that will be discussed separately below; IRI’s output indicator, participants in youth leadership programs (labelled in some quarterly reports as ‘civic education’ programs), counted debate training when that topic was part of a larger training on multiple topics. Youth political development was mainly in the form of the YPLS, but also other youth training events run by IRI’s local trainers. The Youth for Free Elections (YFE) multi-party cooperative activity also falls in this heading, but youth leader interviewees who mentioned similar activities did not see them as separate from other IRI-supported activities, thus there are serious attribution problems. Moreover, the YFE activities should more appropriately have been listed under Objective 4 or Objective 5, neither of which were covered in this evaluation. We found that youth participants may have been involved in multiple types of political development activities, and so could not necessarily distinguish between them; moreover, some participants also received regular party-building training and were involved in youth debates. Although IRI missed its target for participants in youth leadership programs (1337 actual versus 1800 planned during the last two years; 35% women), due largely to the advent of the campaign period, participants generally valued training on media, party communication, organizational development, as well as interaction with fellow party activists in the region and even rival youth activists. Regular party and youth wing representatives reported increased activities by the youth wings, e.g., community events, rallies, social activities, that attracted attention from local youth; neither the parties nor the youth wings were able to provide any membership numbers, however. The trainings, YPLS, and even YFE activities encouraged outreach events, with YPLS running a small grants competition for its participants; these events were captured in an outcome indicator for ‘initiatives implemented by parties,’ with IRI exceeding its targets significantly (25 actual versus 13 planned). YPLS participants from the youth wings of the Free Democrats, the Republicans, and NDP reported expectations of moving up into regular party structures or even government positions due in part to their leadership training in the wake of the October elections. Implementer staff, party representatives, and youth activists noted that party central offices began to coordinate more with youth wings where they exist, particularly the Free Democrats, Republicans, and the National Forum. Youth wings have also become more integrated with regular party committees, as seen in the Social Democrats, the Republicans, the Free Democrats, National Forum, and to a more limited extent UNM. Youth Debates No activity seems to have caught the attention of regular and youth party leaders as much as the youth debates did, based on the enthusiastic interview responses on this activity. For the eleven debaters the team interviewed, the training on debating skills, argumentation, camera presence, and audience engagement were highly valued and put to use in party activities. Debaters from CDM, New Rights, and National Forum reported benefitting professionally from their experiences. As with the GFSIS Political Party Academy, some debaters already had some debate or other relevant experience before IRI’s debate training (especially teams in Tbilisi), so the evaluation team asked debaters the counterfactual 34 | P a g e question about value-added: the common response was that, yes, they had some skills before going into the youth debates, but the camera presence and audience engagement skills specifically were new and tremendously valuable to their professional careers. Parties quickly saw the potential of debates as a recruiting tool, and party leaders from CDM, the Free Democrats, the Republicans, and National Forum were actively engaged in selecting the teams via internal competition and subsequently working with their teams to hone skills and develop policy positions. Party representatives and debaters alike complained that the experts were partisan – which obviously could not be generally true – and were unhappy with Kavkazia’s limited audience, but liked the addition of call-in voting. Evaluation Question 3: Commitment to Sustain Results Target organizations and groups participating in IRI programs are indeed committed to sustain the results of the programs, but not in a self-sustaining way. Rather, as a result of low levels of human and financial resources, as well as the turnover in power following the October elections, there is a commitment to continued cooperation with IRI. Commitment to Using Skills and Knowledge The strongest commitments would be in regional party-building, GFSIS Political Party Academy, and youth party-building. As the evaluation team concluded, the parties benefitted from the regional party￾building training, although the rise of merged GD Coalition offices during the campaign severely undermined the capacity of the junior coalition parties’ regional capacity. Still, the junior coalition parties intend to remain as independent entities, and so intend to continue building the regional bases of their parties. Along the same lines, youth party-building was effective to some extent, but among the junior coalition parties and extra-Parliamentary parties like CDM, the youth wings are viewed as critical for the ongoing functioning of the parties. This is particularly true for the Republicans and the Free Democrats, which have seen the overwhelming majority of their leadership move into Parliament leadership or government, leaving few party leaders left to run the party itself. Parties also highly value the GFSIS Political Party Academy for its role in building policy capacity among party leaders, particularly younger party leaders. Parties like the Republicans and National Forum recognize the need to develop a fuller party platform based on policy positions, but do not have the internal capacity currently. A more moderate level of commitment would be for the youth debates and IRI’s research results. While the youth debates captured the attention of parties – particularly the former opposition parties – the changed political environment has diverted attention to pressing matters of exercising power, and youth leaders are moving up into the regular party or government posts. As noted above, parties value IRI’s research results on policy issues, but mostly for messaging purposes; the policy capacity of parties is still limited, so the full information value of the research cannot be exploited. Commitment to Self-Sustaining Activities No party was prepared to commit to sustaining any of these activities on its own. This is due almost entirely to the lack of financial resources, for all parties but UNM. The former opposition parties operated on very low budgets prior to the campaign period, such that IRI’s funding of training and other events was critical. Even for parties that now have a small number of MPs, the funding they receive from the state budget is still quite low. Moreover, many of the party activists or leaders that provided internal training or applied skills learned in trainings are no longer able to play as substantial a role in the functioning of their parties as before. As noted above, even structural changes were undermined by the merger of coalition members’ regional offices and electoral fortunes: the regional party offices of the Republicans, Free Democrats, and National Forum were merged into the Georgian Dream coalition 35 | P a g e regional offices, for example, while CDM can no longer financially support its ‘zone coordinators’ and New Rights has largely shut down even its headquarters. Evaluation Question 4: Most and Least Successful The terms ‘most’ and ‘least’ are here defined in relative, not absolute, terms. The evaluation team found value in all of the main activities in PPSG, but relative differences can be distinguished in light of the recent elections. Most Successful Two programs clearly fall into this category: the GFSIS Political Party Academy and the youth debates. Academy participants unanimously and overwhelmingly placed a high value on the knowledge and experience they had – not just more knowledge of policy issues and analysis, but the interaction between peers from different parties. Within parties, Academy participants improve their policy capacity, are viewed as future leaders, and share their knowledge with other party activists. Both implementer staff and participants suggested minor changes, a sign that the program is well-conceived and –implemented as it is. The youth debates were also quite successful. Every one of the debaters the team interviewed thought it was a positive experience that s/he would recommend to others in the party. Debaters recounted ways in which the skills they learned in training helped not just in the debates themselves, but in later party work such as media relations, political rallies, and the like. Parties valued the skills and experience of the debaters, too, including tasking past debaters with training of subsequent teams. Least Successful Paradoxically, one of the most successful activities turned into the least successful, because of political events completely beyond the control of IRI: regional party-building. This was unquestionably a successful activity until the rise of the GD Coalition and the day of voting. Some of the more substantial parties leading into the campaign season had been eager practitioners of the skills learned in training, and were having organization success as a result. Parties like CDM, New Rights, the Republicans, and the Free Democrats all put into place regional party-building strategies in accordance with their own resources and bases of support. For the Republicans and Free Democrats, though, the rise of the GD Coalition meant that their regional offices were merged in summer into a single coalition office for the region, and generally the dominant party in the region became the dominant party in the coalition office. Looking past the timeframe of this evaluation to the October election, we see that although CDM and New Rights adapted and implemented the lessons from training to suit their needs and resources, both parties were wiped out in the election results. Four parties were successfully employing training knowledge, but electoral politics severely undermined all of the progress that was made. CONCLUSIONS: IRI-PPSG The evaluation team concluded that the IRI program increased transparency between parties and citizens in terms of direct engagement in the regions and to a lesser extent indirect engagement via the youth debates. Transparency was not a major intended result of the program, however. With respect to capacity, the team concluded that PPSG did strengthen party, youth wing, and individual capacity in identifiable ways, but a lack of data prevents us from confirming some hoped-for outcomes like increased youth wing membership or more general youth engagement. We concluded that regional party-building activities did help parties expand their party structure into regions, engage in more public 36 | P a g e activities, and train party activists, but as will be discussed in later sections, the rise of merged GD Coalition offices undermined this capacity. On the policy side, GFSIS participants clearly gained policy knowledge, benefitting themselves and their parties as discussed above. The team concluded that the polling and focus group results have been useful to parties as a source of information to use for shaping messaging, particularly for specific policy issues. The evaluation team concluded that youth political development, through whatever mode, did help youth wings expand their activities, and provide important skills for youth leaders that are useful for the regular party organization, but there is no reliable evidence that youth wing memberships or general youth engagement increased. The youth debates helped youth leaders gain skills in debating, argumentation, and working on camera. As cited in interviews, debaters subsequently used these skills in media interactions for their parties, and enjoyed promotion into the regular party. As with PSP, though, the evaluation team concluded that parties are committed to sustaining the results of IRI activities, but they can only do so in continued cooperation with IRI. RECOMMENDATIONS This section briefly outlines our recommendations for the focus a new Parliamentary program and how to improve the PPSG project in its remaining years of implementation. In short, the evaluation team believes that the existing programs could to a large extent be continued as is, but they could be more effective in the changed political context than before. However, both projects contain activities that should be carefully reviewed for interest, sustainability, or feasibility. CEPPS Activities in Georgia Both PSP and PPSG did not develop and maintain M&E plans adequate for capturing not just the range of their activities, but the full range of effects for each activity. It should be noted that IRI’s efforts to maintain performance data was strong and credible, but NDI’s efforts were minimal at best. Future projects to work with political parties or Parliament should require much more robust M&E practices. New Parliamentary Program Our main recommendation is that a new Parliamentary program needs to focus more of its attention on the sustainability of its activities. Operationally, this means we suggest that over the next few years, a new program ought to transition from running programs directly and focus support more on trainings. There are many elements of the current project that Parliament, NGOs, and/or the media could run themselves and should take over on a sustainable basis. Alternatively, given the capacity needs in Parliament, a new program should continue to play a central role on training. Below are our concrete recommendations:  Future Women Leaders and Parliamentary Interns: These two programs are working very well. We urge that these two programs continue and do not recommend changing any aspects of them.  NGO Liaison Office, voting records database, PCC, and MP websites: These are programs that Parliament, the media, or NGOs can run eventually. We believe that any new program should stop implementing these programs directly in the medium term, and begin the process of turning them over to domestic organizations. The new program would benefit from an explicit and firm Parliament posture on meeting and information transparency, perhaps with rhetorical assistance from the international community. A new program should examine more closely MPs’ interest in maintaining the PCC before deciding whether to continue with it. 37 | P a g e  Majoritarian MP Outreach: The current Parliament offers an environment far more conducive to majoritarian MP outreach than the previous one, as discussed in this report. We believe that a new program should reach out to the newly-elected majoritarian MPs, especially those with little experience in politics, to gauge their interest.  Committees on Key Institutional Reform: As discussed in the report, the team believes more structured types of trainings, such as NDI conducts with FWL and Parliamentary internship participants, offer a better model for engaging constructively with these committees than the short-term demand-driven approach NDI hitherto appears to have followed. We urge a new program to continue to work with committees and perhaps even expand work with them, as committees are likely to have a stronger policy role in the current Parliament as different parties now control the executive and legislative branches. Revising PPSG The team’s main recommendation is that the current PPSG project should review and re-focus some activities, but could maintain most of the existing activities to better effect in the current political climate. Specifically:  Regional Party-Building: This remains a critical need of the Georgian political party system. Despite the setbacks to the noteworthy progress that had been made prior to the campaign period, some of the capacity that had been built remains. The former opposition parties are in perhaps greater need for organizational development than in the past, as a new generation of party organization leaders moves in to replace those who have left for Parliament or government. The issue for IRI going forward is whether to make its activities available to all parties as before, or just to those parties that have proven capable of electing officials at some level and/or have sufficient financing. Narrowing the target field runs counter to the spirit of USAID guidance for party assistance, though, and Georgian political parties have a habit of rising from the dead to live once more. A related issue is how IRI should prepare to deal with the uncertain future of the junior coalition parties as independent organizations. Will the Free Democrats and the Republicans maintain their existence as parties separate from GD, or will they merge into the latter? If a party wishes to remain independent, how should IRI help re-build the capacity that once existed? Perhaps the regional party-building activities need to be split into two tracks, one dealing with party organization as an outgrowth of party-caucus relations, and a second dealing with more basic organization development skills. Junior coalition parties and UNM would be included in the former, and all parties in the latter. A major issue for the former opposition parties, except GD, is party and campaign financing. Although this might be more properly covered under Objective 5, finances are the fundamental problem of party-building in Georgia.  Polls and Focus Groups: These activities should also continue, but with revisions to the methodology of both in order to capture data needed for M&E purposes. Specifically, survey instruments should be revised to cover inter alia regional party-building, outreach, and youth recruitment.  GFSIS Political Party Academy: This program should continue as is. Both GFSIS and former participants suggested relatively minor changes such as returning to a 10-month period, scheduling more ‘Hot Coffee, Hot Issues’ events, and adding a practicum work product. 38 | P a g e  Youth Political Development: This general activity should be re-focused. It was not clear to youth activists that there was any meaningful distinction between YPLS and other youth training events, other than the duration and mix of participants. In addition, YPLS overlaps in some topic areas and some regions with NIMD’s Democracy Schools, so better coordination between the two projects could avoid duplication of efforts.  Youth Debates: The youth debates should be expanded and revised. The regional rounds were a boon to party activists outside Tbilisi, so more should be planned in the future. Former debaters and party leaders made suggestions to improve the voting, the expert panel, the audience, and so on, but the only item that all agreed on was the need to televise the debates on a channel with better coverage than Kavkazia TV. 39 | P a g e ANNEXES 40 | P a g e ANNEX I: EVALUATION STATEMENT OF WORK MID-TERM PERFORMANCE EVALUATION OF THE POLITICAL PARTY STRENGTHENING IN GEORGIA (PPSG) AND PARLIAMENTARY STRENGTHENING PROJECT (PSP) FOR USAID/GEORGIA Background: Georgia’s political environment today is characterized by an unbalanced party system, with relatively low and decreasing levels of political competition, high and increasing levels of political polarization and distrust, the eclipse of policy discussion by debate over the rules of the game, and the over￾personalization of politics and of parties. Because of the distrust in the fairness of the system, political parties have refused to constructively engage in existing political institutions and the resulting increased cynicism in the population has discouraged broader political participation. Polarization between the ruling party, which currently holds a significant advantage, and the numerous opposition groups, has intensified and spilled over into civil society and media organizations as well, and has negatively affected the broader political and public discourse in Georgia. Multiple factors have contributed to the increased political polarization. The government has used its large parliamentary majority to enact its reforms with only limited consultation with government backbenchers, its grassroots party membership, opposition or civil society. In 2010 Constitution of Georgia was hastily adopted without waiting for the Venice commission recommendations. Additionally the 2011 negotiations over the election code dissolved due to failure of ruling party and opposition to reach compromise. This lack of consensus-building on government reforms further increased political polarization in country. The overall enabling environment for the operation and development of societally based political parties and political pluralism has become significantly worse and increasingly problematic and restrictive, particularly since the November 2007 events and the August 2008 war with Russia. Although one can point to some positive elements or conditions in most of the areas affecting political party development, significant problems and challenges are present with regard to virtually all of the factors and conditions that enable societally based parties to function and develop, including even the most fundamental. These problems and inhibiting factors in the enabling environment have adversely shaped political party structures, behavior, and choices, especially those of opposition parties. The enabling environment for political pluralism, competition, and political party development is significantly worse in the regions than in the capital. While political membership and affiliation is generally not restricted in the capital, freedom of association at the regional and local levels appears to be significantly limited and discouraged through political and economic pressures exerted by regional and local authorities. The weak party system directly impacts parliamentary development, and contributes to parliament’s limited ability to moderate political polarization. Most directly, the weak party system contributed to the high parliamentary turn-over—64% in the 2008 parliamentary elections. It has also contributed to a shallow pool of Members of Parliament (MPs) who are empowered within the parliamentary factions to take decisions or to communicate with the media. The limited number of decision-makers in the parliamentary factions has made it difficult for parliament to move beyond crisis management and to address lower-tier political issues, including such issues as parliamentary reform. Although moderate voices exist in civil society, their impact on parliament has also been limited – often choosing to advocate directly with the government or through the media, rather than through engagement in parliament. 41 | P a g e In October 2010 Parliament adopted amendments to the constitution, which will significantly reduce powers of next President in favor of Prime Minister and the government. The new constitution will go into force upon the inauguration of the next president, who is scheduled to be elected in October 2013. As soon as the new constitution goes into force in 2013, the government existing at the time will resign to give way to the Parliament, elected in 2012, to compose the new government. SI Responsibilities and Projects: The purpose of this evaluation is to: a) determine how effective (or otherwise) the projects have been in achieving their intended results; b) assess the opportunities for key successes to be replicated and c) to provide recommendations regarding focus of any future programming (PSP) and whether course corrections are necessary in remaining years of implementation (PPSG). The timeframe to be covered by the evaluation is from the start of the projects (PSP -2009 and PPSG – 2010) through the initiation of this evaluation on/about November 2012. The results of the evaluation will be used by USAID/Georgia for improving ongoing interventions in the area of political processes in light of the upcoming 2012 and 2013 elections, in order to focus on the activities that are most meaningful and critical for improving the environment for political competition in Georgia and to identify directions of further assistance of the parliament of Georgia. The audience of the evaluation will be USAID and in particular its Democracy and Governance (DG) office, as well as other USG agencies political processes programming in Georgia. In addition, the results of the evaluation will be useful for USAID’s current implementing partners to improve their interventions. With similar purpose, the results of the study will be shared with other stakeholders locally – Parliament, Political Parties, other donors working in this area, and interested NGOs. Finally, evaluation results will also be used for reporting purposes to Washington￾based stakeholders. The evaluation must be based on the series of evaluation questions to elicit the results of PSP and PPSG projects to date. The evaluation report must provide answers to the following questions: 1. To what extent have the intended results of the PSP project been achieved in relation to the objectives 1, 2, and 3 of the project (Statement of Work, p.3)? 2. To what extent have the intended results of the PPSG project been achieved in relation to the objectives 1, 2, and 3 of the project (Statement of Work p.4)? 3. What has been the commitment of target organizations and groups (Parliament of Georgia, political parties, majoritarian MPs, youth, women constituencies) to sustain the results of these projects (specific focus on PCC activities should be made for PSP)? a. Did the participants obtain needed skills and knowledge as a result of these projects? b. Were the skills and knowledge considered useful? c. Were the skills and knowledge used in the behaviors of individuals and/or organizations? 4. What are the most and the least successful interventions as perceived by main stakeholders (Parliament of Georgia (specific focus on PCC activities should be made), political parties, relevant NGOs, media)? a. How do political parties perceive IRI polling and how effectively do they use information obtained from the polls? 42 | P a g e ANNEX II: EVALUATION METHODS AND LIMITATIONS USAID/Georgia requested a mid-term performance evaluation of both PSP and PPSG to determine respective achievements to date, implementation progress, perceived value, potential sustainability, and beneficiary engagement. As noted in TIPS #11, Introduction to Evaluation at USAID, a mid-term evaluation is primarily a formative evaluation intended to furnish information that will guide program improvement; in this case, there are also strong elements of process and summative evaluations too. Evaluation Team The SI Team was composed of Andrew Green, Barak Hoffman, George Welton, and Tamara Pataraia. Team Leader Dr. Green is SI’s DG Practice Leader, with extensive applied research experience in the Europe & Eurasia region. Technical Expert Dr. Hoffman is the Executive Director of Georgetown’s Center for Democracy & Civil Society and has deep experience in applied research on political institutions and parties. Dr. Welton was the team’s Evaluation Expert, and he has many years of experience conducting research in Georgia. Dr. Pataraia was the Local Evaluation Expert, with 12 years of experience conducting applied research in Georgia on a variety of democracy and governance issues. The SI Team utilized GeoWel Research for implementation of the focus group protocols, mini-surveys, summaries of discussions, and English translation of documents. In addition, GeoWel Research handled logistical functions in Kutaisi, Batumi, and Telavi. Evaluation Design SI implemented a technical approach of standard rapid appraisal methods to answer the evaluation questions and provide recommendations. This evaluation relied primarily on key informant interviews, focus groups, review of materials and secondary sources of data, and mini-surveys. The mixture of these methods was different for each activity, as would be expected due to differences in the nature and implementation of the activities. We did not rely heavily on quantitative data, however, due in large part to the poor data context of these projects, i.e., limited project M&E plans, inapplicability or unavailability of relevant survey data, and so on. • Materials and other information were obtained from implementers, direct participants, and secondary sources. Materials reviewed for PSP included, for example, project documents, parliamentary documents on faction activities, and news stories. Materials reviewed for PPSG policy trainings included project documents, parliamentary documents on policy discussions, and party documents on policy platform development. The review yielded important information about some baseline values and progress related to the programs. Data in this category were most useful for evaluation questions 1, 2, and 3a. The team gathered some quantitative data from sources outside the implementing partners’ M&E plan, e.g., call in votes and texts for youth debate shows, growth in youth groups, hits on MP websites, use of MP website discussion forum, use of the voting records database, and questions asked in parliamentary question hour. • The details of semi-structured interviews of key informants differed depending on the activity, the specific person, and ‘causal distance’ from the activity; some key informants were involved in multiple activities within a program or even across the two programs, so interview protocols for some types of key informants consisted of multiple modules of questions. The interview protocols have specific questions designed to elicit concrete examples of knowledge utility and utilization. The desired interviewees were not selected randomly, but were identified through the implementers, reputation, or existing contacts of evaluation team members. The list of 43 | P a g e desired interviewees included people from organizations that did not benefit directly from the evaluated activities, in an attempt to obtain unbiased information. The team interviewed 81 people in Tbilisi, Kutaisi, Batumi, and Telavi. Data in this category were useful for all evaluation questions. • Focus groups of citizens and of NGOs and media covered legislator-citizen interactions (majoritarian MP constituency outreach, public meetings, regional party building) and party￾citizen idea communication (youth debates, Your Parliament) provided in-depth information on how those individuals and organizations view the relationship of political actors and society. Two focus groups were held in each Batumi and Telavi. Data in this category were most useful in questions 1, 2, and 3c. • Mini-surveys of target population segments unreachable through other means provided some information not otherwise available. The team conducted mini-surveys of 25 former majoritarian MPs, 10 regional media, and 17 regional youth leaders. Data in this category were most useful in questions 1, 2, and 3abc. As noted in TIPS #5, Using Rapid Appraisal Methods, triangulation through the use of multiple methods reduces the effects of bias and can increase the validity of findings. Parallel analysis was conducted on the data gathered through these rapid appraisal methods. In this analytical approach, types of data on activities are analyzed in parallel for potential findings about that activity, then analyzed across data types to identify findings for the activity relevant to the evaluation question, and ultimately to conclusions at the activity and evaluation question levels. For example, the team analyzed semi-structured interview responses about youth debates from the debaters themselves and then from relevant party leaders; then information from IRI documents and statistics on viewership of youth debates; then narratives from focus groups with citizens, NGOs, and media; then data from a mini-survey from youth leaders; and then analyzed potential findings across these data sources to generate synthesis findings for that activity. For evaluation questions 1 and 2 on intended results of the two projects, the evaluation team utilized the concepts of transparency (the provision of and access to information) and capacity (imparting knowledge that enables action) to evaluate the various activities under each project. SI adopted this approach in part because of the inadequacy of the implementers’ M&E plans, as well as the the need to employ a common analytical narrative. Work Plan This mid-term evaluation focused on two complex projects with multiple standalone activities. In conjunction with uncertainty over the availability of party leaders and new MPs for interviews, as well as the availability and willingness of former MPs for interviews, the evaluation’s logistics were quite complicated and changed on a last-minute basis. The evaluation took place from 29 October – 15 November. The team worked in Tbilisi during the first week and travelled to Kutaisi at the start of the second week. Given that the new Parliament was in the midst of its first session, the team experienced difficulties scheduling a sufficient number of interviews to be productive in Kutaisi for more than three days. One team member travelled on to Batumi to conduct a small number of interviews and monitor the focus groups, while the remaining team members returned to Tbilisi. At the start of the third week, one team member travelled to Telavi to again conduct a small number of interviews and monitor the focus groups; it should be noted that in both Batumi and Telavi, a number of our desired interviewees were targeted to be focus group participants, which reduced the number of semi-structured interviews to be held. The full timeline of evaluation activities can be seen in the charts in Tables A and B in this Annex. 44 | P a g e Limitations and Constraints The essence of evaluation is comparison—typically across time or geography, better still across both; an impact evaluation further attempts to compare across units that have or have not received some sort of programming treatment. In the context of the programs subject to this mid-term evaluation, however, there are significant barriers to comparing along any dimension, and the poor data context complicates the task of drawing valid conclusions. There are some noteworthy limitations on comparison, and hence attribution.  First, recall bias was a problem noted by all team members. Political party or parliamentary programs often include activities that were conducted in years before the current program periods, or that were conducted by other implementers in addition to IRI or NDI, e.g., NIMD Democracy Schools. A similar problem is that participants in multiple training activities may be blending their experiences into a composite memory, e.g., youth leaders may have received training on youth party-building, participated in YPLS, participated in youth debates, and even attended training for regional party-building.  Second, response bias is a common problem for program evaluations. For example, a former participant may give the interviewer positive remarks about a strategic planning retreat at a nice resort because she wants it to continue in order that her staff can enjoy a nice weekend vacation. The team was acutely aware that party representatives and Parliamentary leaders were fully cognizant that a negative evaluation could mean the end of a project that provided them with needed training.  Third, selection bias in the form of contacts provided by the implementers can mean an evaluation team only hears from people with positive experiences. The most effective approach to combating bias is to use multiple sources of data to triangulate on an evaluation issue, as is often accomplished through qualitative reliability matrices. By combining information found in documents or interviews from multiple sources, any one piece of biased data would not skew the analysis. Another approach that pertains specifically to interviews is the inclusion of key informants from organizations that do not directly benefit from the evaluated program and the use of questions about specific examples of knowledge use. By far the biggest limitation on this evaluation was the data-poor context in which these programs operated, due largely to the wholly inadequate M&E efforts by both implementers. Both programs had multiple distinct, ongoing activities under each of their three objectives subject to this evaluation, and yet, according to the documents supplied by IRI and NDI, neither had indicators for each of their activities: NDI has nine distinct activities and seven indicators, including outputs, while IRI has six distinct activities and eight indicators, including outputs. Of particular concern was that NDI did not seem to have an approved M&E plan at all, nor was it gathering data on its proposed indicators. USAID/Georgia did provide the team with a report showing four numerical indicators from NDI, but these did not match either of the M&E plans provided to the team by NDI’s DC and Tbilisi offices. For its part, IRI was indeed tracking performance on its very limited set of indicators, with considerable detail and timeliness. Both implementers, NDI in particular, would not infrequently list activities under different objectives. For the evaluation team, then, it was never completely clear whether we had identified all of the activities under each program’s objectives, nor could we rely on program documents to identify the intended results of the programs. The evaluation was burdened by three types of constraints on the team’s ability to gather useful data. First, the timing of the evaluation was problematic: the evaluation timeframe ended in June 2012, but the team was in the field more than four months later. Even worse, the turnover in power as a result of the 45 | P a g e October parliamentary elections meant that many opposition party leaders were in government or Parliamentary leadership, and many former ruling party MPs lost their seats. Combined with the physical move of Parliament from Tbilisi to Kutaisi, the team’s ability to schedule time with or even locate desired interviewees was severely diminished. Second, the demand-driven, highly-responsive nature of party and parliamentary assistance results in programs characterized by activities tailored specifically for a certain target group or activities that are short-lived because of the need to be opportunistic. The end result is that these sorts of programs often have a large number of distinct activities, which complicates analysis of their effects. Third, the common practice of leaders or activists attending multiple trainings on different topics from the same implementer, or even trainings on similar topics from different implementers, muddies attribution. 46 | P a g e Table A: Gantt Chart of Activities = Activity = Deliverable ACTIVITY October 2012 November 2012 December 2012 January 2013 Weeks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Team Mobilization; Preparation Document Review Team Planning Meeting Analysis of available data; Site selection for field data collection Develop data collection instruments Draft evaluation design and work plan to USAID/Georgia Expatriate team members travel to Georgia In-Brief at USAID/Georgia to discuss evaluation methodology/data collection instruments Key Informant Interviews in Tbilisi Key Informant Interviews in Kutaisi Focus Groups/Key Informant Interviews in Batumi and Telavi Process data from fieldwork; Prepare debrief Debrief Presentation at USAID/Georgia Analyze data; Prepare draft evaluation report; Internal quality assurance review Submit draft evaluation report to USAID Revisions to draft evaluation report Submit revised draft evaluation report Incorporate comments from USAID Submit final evaluation report to USAID 47 | P a g e Table B: Timeline of Deliverables Deliverable Due Date Contract Awarded September 14, 2012 Document Review October 1-10, 2012 Draft Evaluation Design and Work Plan to USAID October 25, 2012 In-Country Work October 29- November 15, 2012 USAID/Georgia In-Brief October 31, 2012 Final Evaluation Design and Work Plan to USAID November 2, 2012 USAID/Georgia Out-Brief November 15, 2012 Outline of Final Evaluation Report to USAID November 15, 2012 Draft Final Evaluation Report to USAID November 30,, 2012 USAID Comments to SI December 10, 2012 Revised Evaluation Report to USAID December 31, 2012 USAID Comments to SI January 15, 2013 Final Evaluation Report to USAID January 20, 2013 Contract Completion January 25, 2013 48 | P a g e ANNEX III: DATA COLLECTION INSTRUMENTS Key Informant (KI) Semi-Structured Interview (SSI) Question Matrix Implementer Direct Participant Indirect/External IRI Obj1: Regional Party-Building Q2 Which parties participated? Why? What activities did you use to increase parties’ internal capacity? What activities did you use to parties’ ability to develop policies? What activities did you use to identify and recruit youth leaders? How responsive were parties to the activities? Which were more and less interested in participating? Why did your party participate? In which party strengthening activities were you a participant? How did the trainings help you with the ability to develop polices? Do you have any examples? Were offices or resources dedicated? Are the offices/resources still there? How responsive were parties to the activity? Which were more and less interested in participating? Do regional offices or resources exist? Which parties are capable of building themselves at the regional or district levels? What role do regional party leaders play in the national organization and policy￾setting? Q3 How were regions/districts prioritized by parties? How much did parties implement their strategic plan? What parties have trainers or a training unit? How do you think parties will continue to use the skills you gave them beyond the life of the program? Were offices or resources dedicated? Are the offices/resources still there? Have parties adapted training materials on their own? How would you characterize your working relationship with IRI? What trainers or training unit exist with your party? Do you think you think the benefits of the training IRI provided will exist after the program ends? Which parts? Why? Which parts of the trainings do you think people in your party will tend not to use? Have training materials been adapted? What are some examples of regional party leaders influencing policy positions by the party? What regions or districts were prioritized by which parties? Q4 What parts of the program were most successful and least successful? Why? What parts of the program were most successful and least successful? Why? IRI Obj2: GFSIS Policy Trainings Q2 How did the trainings support political party strengthening? What did you learn in the trains that helped you make How would MPs/staff develop policy depth and experience? 49 | P a g e How did the trainings support parties’ capacity to develop policies? How were participants chosen? more informed policy analysis? Q3 Do you think participants are continuing to use what they have learned after they have finished the trainings? What are some examples of application after the course? How are you applying what you have learned since the trainings ended? Which MPs/offices are strong on policy issues? Q4 What were the more effective and less effective parts of the trainings? What were the more effective and less effective parts of the trainings? IRI Obj2: Polls Q2 Which parties participated? Why? Which parties used poll info for policy development or campaign strategy? Why did your party participate? How has your party used poll info? Are there other poll sources you rely on? Does the party have regular methods for learning about citizen interests? Have you seen the results of public opinion polls in the media? Q3 Will the influence of polls persist beyond electoral campaigns? What parties could support or conduct their own polls? Does the party intend to support or conduct its own polls in the future? Did IRI’s polling activities change your opinion on the importance of polls? Q4 What aspects of the polling were most and least effective? What aspects of the polling were most and least effective? IRI Obj3: Youth Party-Building Q2 Which parties participated? Why? What activities did you organize to recruit youths? Why did your party participate? Do offices or resources to train youth leaders exist? 50 | P a g e Q3 How were regions/districts prioritized by parties? How much did parties implement their youth strategic plan? Were offices or resources dedicated to youth wings? Are the offices/resources still there? Have parties adapted youth training materials on their own? How responsive were parties to the activity? Which were more and less interested in participating? How/why were regions/districts prioritized? How much of the youth strategic plan was implemented? Have youth training materials been adapted? What are some examples of youth leaders influencing policy positions by the party? How responsive were parties to the activity? Which were more and less interested in participating? What regions or districts were prioritized by which parties? Do regional offices or resources exist? Which parties are capable of building themselves among youth? What role do youth leaders play in the national organization and policy￾setting? Q4 What were the most and least successful activities in this area? What were the most and least successful activities in this area? IRI Obj3: Youth Debates Q2 Which parties participated? Why? What activities did you undertake to encourage youth political debates? Why did your party participate? In which activities did you participate? Q3 How were participants chosen? What training was provided? To what extent did any of the policy debates influence party platforms? How were participants chosen? Was sufficient training provided? How has this affected your political career? What are some examples of how debated policy issues influenced your party’s platform? Were the participants from parties chosen well? Did the debates influence party platforms at all, or perhaps public opinion on specific issues? Q4 Which youth wings could reproduce these activities on their own? Could your youth wing train future debate participants? NDI Obj1: Faction-Journalists Interaction Q1 What benefit did MPs get? Journalists? How would you describe your working relationship with NDI? What benefit did you get from this activity? Is this benefit of value to MPs and/or journalists? Has it stimulated any follow￾on interactions? 51 | P a g e Q3 What are some examples of follow-up interactions outside this activity? How do you see journalists you trained benefitting from the program since the trainings ended? How did the trainings make you a better journalist? Did you engage in any follow￾up interactions outside this activity? Do you engage in any new or repeated interactions outside this activity? Has it stimulated any new or repeated interactions outside of the activity? Q4 What were the more and less successful parts of the program? What were the more and less successful parts of the program? NDI Obj1: Majoritarian MPs & Constituency Outreach Q1 What types of activities did you conduct under this aspect of the program? How were majoritarian MPs selected for this activity? How responsive was parliament to the activity? For which activities were you a participant? What was the benefit to you or your MP of this activity? How responsive was parliament to the activity? Is majoritarian MP outreach an activity that would have meaning to citizens? Q3 How were outreach strategies developed? To what extent did MPs or bureau staff follow the strategies? In what ways did MPs/staff adapt or extend the strategies? To what extent could surviving MPs/staff create their own outreach strategies? What does the change in bureau staff mean for the outreach capacity developed in this activity? Did the outreach strategy make sense for your district? What activities did or did not fit the political culture of the district? Did you revise or add to the outreach strategy? Could you create your own outreach strategy now, or would additional assistance be needed? What happens to outreach capacity when bureau staff change? Which majoritarian MPs made efforts to reach out to their district constituencies? Did the nature of party organization affect this at all? Q4 What were the most and least successful aspects of the program? What were the most and least successful aspects of the program? NDI Obj1: PCC 52 | P a g e Q1 How receptive was Parliament to this activity? What types of activities did you conduct under this aspect of the program? How receptive was Parliament to this activity? How receptive were MPs? Did this depend on party at all? To what extent did MPs benefit from the PCC? How receptive was Parliament to this activity? For which activities were you a participant? What benefit did you get from the PCC? Could you give an example of citizen awareness of the media you generated? How might you use the PCC now? Have you participated in a your parliament debate? Why or why not? What is the purpose or benefit of Your Parliament videos and other media generated by MPs? Are citizens aware of these MP-generated media? What do they think of it? Is it making the more politically engaged or aware? Q3 How will the PCC be established in Kutaisi, and might its use change because of the needs of MPs to overcome geography? Does the PCC allow for activities that in the future could be done by consumer electronics and social media? Do you think the your parliament debates will continue to exist after the NDI program ends? How else could or should MPs generate media about Parliament activities? Q4 What were the most and least successful aspects of the program? What were the most and least successful aspects of the program? NDI Obj2: Future Women Leaders Q1 How receptive was Parliament to this activity? How were participants chosen? What training did you provide? How receptive was Parliament to this activity? How were participants chosen? Was sufficient training provided? What trainings did you attend? How has this affected your professional career? Has the program helped participants play a stronger and more professional role in their areas? Q3 In what foreseeable ways might this produce more women leaders in civil service, parties, or NGOs? In what foreseeable ways might this produce professional opportunities for you in civil service, parties, or NGOs? In what foreseeable ways might this produce more women leaders in civil service, parties, or NGOs? Q4 What were the most and least successful aspects of the program? What were the most and least successful aspects of the program? NDI Obj2: Internship Program 53 | P a g e Q1 How receptive was Parliament to this activity? How were participants chosen? What training was provided? How receptive was Parliament to this activity? How were participants chosen? Was sufficient training provided? How has the trainings affected your professional career? Has the program helped participants play a stronger and more professional role in their areas? Q3 How are participants using what they learned in the trainings to do their job more effectively? In what foreseeable ways might the program activities produce leaders in civil service, parties, or NGOs? How are you using what you learned in the trainings to do your job more effectively? In what foreseeable ways might this produce professional opportunities for you in civil service, parties, or NGOs? How will you use what you have learned after program activities end? In what foreseeable ways might this produce more leaders in civil service, parties, or NGOs? Q4 What were the most and least successful aspects of the program? What were the most and least successful aspects of the program? NDI Obj2: NGO Liaison Office Q1 How receptive was Parliament to this activity? How receptive were NGOs to this activity? How receptive were MPs? Did this depend on party at all? To what extent did MPs benefit from this? To what extent did NGOs benefit from this? What benefit did you get from the NGO Liaison Office? Could you give an example of a policy issue that was affected by MP-NGO interaction? Did you engage in any follow￾up interactions outside this activity? What is the purpose or benefit of the NGO Liaison Office? Could you give an example of a policy issue that was affected by MP-NGO interaction? Has it stimulated any follow￾on interactions? Q3 How are NGOs using what they learned to be more effective in their work? How will the NGO Liaison Office be established in Kutaisi, and might its use change because of geography? How are you using what you learned to be more effective in their work? Do you engage in any new or repeated interactions outside this activity? How might Parliament’s move to Kutaisi affect this? Has it stimulated any new or repeated interactions outside of the activity? How might Parliament’s move to Kutaisi affect this? Q4 What were the most and least successful aspects of the program? What were the most and least successful aspects of the program? NDI Obj1 and 3: Key Committees on Institutional Reform, Public Meetings 54 | P a g e Q1 How were policy issues selected for this activity? How were outreach strategies developed? To what extent did committees/MPs follow the strategies? In what ways did committees/MPs adapt or extend the strategies? How receptive was Parliament to this activity? What was the benefit to the committee or MPs of this activity? Did the outreach strategy make sense for the policy issues? Did your committee/MP revise or add to the outreach strategy? Could you give an example of policy substance that changed as a result of this activity? How receptive was Parliament to this activity? Which committees/MPs made efforts to reach out to citizens on important policy issues? Did citizens value the public discussion of these policy issues? Did any substance change as a result? Q3 To what extent could these committees under new leadership create their own outreach strategies? What does the change in committee staff mean for the outreach capacity developed in this activity? Could these committees/MPs create their own outreach strategy now, or would additional assistance be needed? What happens to outreach capacity when committee staff change? Is public discussion of important policy issues an activity that would have meaning to citizens? Q4 What were the most and least successful aspects of the program? What were the most and least successful aspects of the program? NDI Obj3: Voting Records Database Q1 How receptive was Parliament to this activity? How receptive were end users to this activity? How receptive were MPs? Did this depend on party at all? To what extent did MPs benefit from this? To what extent did end users benefit? What benefit did you get from access to voting records? Could you give an example of how you have used this information? What is the purpose or benefit of the voting records database? Could you give an example of how you have used this information? Q3 Is Parliament capable of maintaining the system and data entry? How are you maintaining the system? Will you have the resources to do so after the program ends? Q4 What were the most and least successful aspects of the program? What were the most and least successful aspects of the program? Is this a resource that has meaning for political or policy analysis? 55 | P a g e Mini-Survey of Youth Organization Members The evaluation team conducted a mini-survey of 17 youth leaders from 10 different political party youth wings. The survey was conducted in Georgian. “IRI has given us your contact information. As you know, IRI has been engaging in work to develop the youth organization of many of the Georgian political parties. We are currently conducting a review of that program for USAID and would appreciate 10 minutes of your time to collect your impressions of the program. Your answers will be kept confidential. Would you like to continue?” Name; Organization; Position in Organization; Location of Organization  Were you involved in the youth-organization development training? a) How would you assess it (very good, good, moderately bad or very bad)? b) How many trainings did you do, and when? c) What was the nature of the training that you undertook? d) What was the best thing about them? e) What was the worst thing about them? f) Has the training and other IRI work had a major effect on the efficiency and effectiveness of the youth organization (very positive, somewhat positive not positive at all)?  Engagement in the youth debates a) Have you been engaged in this program as a debator or as a supporter/organiser? Yes/no b) Were you a debater? c) How would you assess it (very good, good, moderately bad or very bad)? d) What was the best thing about them? e) What was the worst thing about them?  Has the membership of your youth organisation increased, decreased or stayed the same since January 2011 (increased a lot, increased a little, stayed the same, decreased a little, decreased a lot) a) Do you think that the training and the debates increased your membership (yes a lot, yes a little, no) b) Do you think the trainings and debates improved the communication skills of those involved (yes a lot, yes a little, no not at all) c) Would it be possible to get exact numbers on this? Who should I call/email? Mini-Survey for Majoritarian MPs We were provided with a list of 49 MPs that had been provided with training in communication, in anticipation that they would be involved in the ‘Our Parliament’ videos. Of those 49, 29 of them ultimately were involved in the production of the video. The other 20 chose not to, as they did not feel comfortable on-camera, or simply lost interest. We attempted to contact all 49, and were able to contact 25; 19 of those we contacted had produced a ‘Your Parliament’ video. “NDI has provided us with your contact information because we are currently undertaking monitoring and evaluation of their Parliamentary Strengthening Program. As Majoritarian MPs, in the previous parliament, were one of the main recipients of the project we would like to ask you a few questions about your experience and attitude towards the project. The interview will take approximately 15 mins. Your answers will be kept confidential. Would you like to continue?” Name of interviewer; Name of respondent; Party; Region  Were you involved in trainings and outreach development with NDI?  Can you describe that nature of your engagement?  So far this year, how many separate trainings/events did you or your staff attend organised by NDI? 56 | P a g e a. How useful did you find these trainings (very useful, moderately useful, not very useful, entirely useless - code 4-1) b. Did you find them useful in helping you conduct more effective outreach to your constituents (yes, very….(4-1) c. What did you find useful about it? d. Is there any particular information or skill that you developed during the training that you found valuable?  NDI also provided connection with NGOs and journalists for MPs through the Parliamentary Communication Center, a weekly parliamentary newsletter and an NGO forum a. Were you ever connected with NGOs/journalists by NDI? b. How often were you connected to NGOs/journalists by NDI? (weekly/monthly or a couple of times a year) c. What was the subject and purpose of these meetings/discussions?  Did you produce an 'Your Parliament' video this year? a. If yes, did you consider it a useful exercise (very useful, moderately useful, moderately useless, entirely useless - (code:4-1)) i. Did you get very much feed-back on it from normal citizens (yes a lot, yes a little, no - none at all) - code 3-1 ii. Are there any major ways in which it could have been improved? b. (If they are continuing) Would you like to see this repeated in the next parliament (yes definitely, yes probably, no_ - (3-1)  NDI provided web-space for majoritarian MPs. a. Did you use this? b. What did you use it for? c. Did you get feed-back from constituents to suggest that they had used it (yes a lot, yes a little, no) Mini-Survey of Media Organisations We spoke to 10 journalists from regional media outlets who were regular attendees of NDI’s faction/journalist interaction. “NDI has given us your contact information. As you know, NDI has been engaging in work to develop the way in which the parliament connects to citizens of Georgia. We are currently conducting a review of that program for USAID and would appreciate 10 mins of your time to collect your impressions of the program. Your answers will be kept confidential. Would you like to continue?” Name of interviewer; Name; Organisation; Position in Organization; Location of Organisation  Do you feel that the outreach/communication on the part of political parties has been better in this election than in previous elections? - a lot better, somewhat better, the same, somewhat worse, a lot worse  What about outreach outside of Tbilisi? a lot better, somewhat better, the same, somewhat worse, a lot worse a) Which parties do you think were effective? b) Why? c) Can you give examples?  Did you make use of the parliamentary bulletins provided with the assistance of NDI? a) How did you use them?  Have you had any engagement with parliamentary factions, facilitated by NDI? a) Describe the nature of engagement? b) How regular has the engagement been (at least once a month, at least once in every 3 months, at least once a year, less than once a year) c) Can you highlight a couple of ways in which NDI's engagement in helping connect journalists to factions helped you to achieve objectives you would not have achieved otherwise? 57 | P a g e d) Do you think it was effective in helping develop better understanding of parliament in the media? (very effective, somewhat effective, somewhat ineffective, very ineffective)  To regional journalists - are you aware of the 'Our Parliament' program? Yes/No a) Do you think it helped people develop a better understanding of their majoritarian MPs? (a lot, a little, not at all)  Overall, can you think of any ways in which the parliament could communicate better with the media? 58 | P a g e Focus Groups The logic for meeting with NGO/the media is that these groups are likely to be the primary conduit through which the different programs will have connected to the political parties and parliament/parliamentarians. Clearly, two focus groups of a small number of people will not give a particularly robust set of results, but the general level of contact with the groups and issues that the different programs have sought to disseminate, should be illustrative, particularly when combined with other metrics of interaction/outreach that we are collecting. Focus groups were held at USAID-financed Civic Engagement Centers. The focus groups were recorded and an annotated summary with illustrative quotes created for analytical purposes. Focus groups will last for 90 minutes. NGO/media Participants were identified and recruited through a mix of NDI recommendations and evaluation team members’ contacts. In Batumi, we had eight participants, while in Telavi we had six. In terms of citizen focus groups, we recruited ‘informed citizens’, rather than ‘average citizens’. Therefore, we will not simply be selecting people off the street, but we will identify local people recommended by NGOs with activities related to civic education, as well as through our own networks of NGO contacts. In Batumi, we had nine participants, while in Telavi we had 14 participants. Media/NGO Focus Groups Party Outreach (IRI Obj 1), Majoritarian MPs and Constituency Outreach (NDI Obj 1)  Which of the political parties have made themselves visible in the region this year? o How did that compare to the 2008 parliamentary elections?  How about Majoritarian MPs? o Have they engaged in outreach? o Have any of you used their websites at all? o How did that compare to the 2008 parliamentary elections? Youth Debates (IRI Obj 3)  To what extent have NGOs/media interacted with citizens on the substance of the debates? o Can you recall any televised policy debates in the last two years? o What was the substance of the youth policy debate you watched? o How did your thinking on that policy issue change? Did it change your view of that party? Faction-Journalist Interaction, PCC (NDI Obj 1)  Have the media or NGOs made use of the Parliamentary Communication Center in any way?  Has NDI worked to facilitate any of the groups present in attempts to connect with the parliament?  MY Parliament videos (in Batumi where they had them)- To what extent have NGOs/media interacted with citizens on the substance of the videos? o Can you recall any videos produced by MPs in the last year? o What did you learn from the video?  MY Parliament videos (in Batumi where they had them) o Do you think people would have watched them if they had been shown here? General Forward Looking  NDI’s main project was to try and support engagement between parliament and the citizens, directly and through NGOs and the media. o Are there any particular issues that you would like to be informed about that are currently hard to find-out relating to the operation of the parliament or the MPs?  IRI’s main project was intended to try and strengthen the outreach of political parties and their engagement with citizens, NGOs and the media. o Are there any particular issues that you would like to be informed about that are currently hard to find-out relating to the operation of parties?  (if there is time) Are there any particular issues that you think are particularly issues AND where the parliament/political parties fail to communicate clearly? 59 | P a g e Citizen Focus Groups Party Outreach (IRI Obj 1), Majoritarian MPs and Constituency Outreach (NDI Obj 1)  Which of the political parties have made themselves visible in the region this year? o Have you met anyone running for parliament? o Have you attended a public meeting, rally or campaign event? o How did that compare to the 2008 parliamentary elections?  How about Majoritarian MPs? o Did you have any contact with your majoritarian MP or his/her opponent before the campaign started o Have you ever met your majoritarian MP? o Would you say that you know much about your MP? o How did that compare to the 2008 parliamentary elections? Youth Debates (IRI Obj 3)  Did you watch any of the youth debates (debates between youth in differing parties)? o (for those who did) What was the substance of the youth policy debate you watched? o (for those who did not) why not? If you had heard about them – would you have watched? Faction-Journalist Interaction, PCC (NDI Obj 1)  MY Parliament videos (in Batumi where they had them)- To what extent have NGOs/media interacted with citizens on the substance of the videos? o Can you recall any videos produced by MPs in the last year? o What did you learn from the video?  MY Parliament videos (in Batumi where they had them) o Do you think people would have watched them if they had been shown here? General Forward Looking  NDI’s main project was to try and support engagement between parliament and the citizens, directly and through NGOs and the media. o Are there any particular issues that you would like to be informed about that are currently hard to find-out relating to the operation of the parliament or the MPs?  IRI’s main project was intended to try and strengthen the outreach of political parties and their engagement with citizens, NGOs and the media. o Are there any particular issues that you would like to be informed about that are currently hard to find-out relating to the operation of parties ?  (if there is time) Are there any particular issues that you think are particularly issues AND where the parliament/political parties fail to communicate clearly? 60 | P a g e ANNEX IV: SOURCES OF INFORMATION PERSONS CONTACTED Name Affiliation Alexandre Sakhelashvili Youth leader, GFSIS Participant: Free Democrats Alexander Sokolowski USAID/E&E Ana Vachadze former intern: IRI (2011-2012, Unity for Justice Faction) Andrea Keerbs IRI Country Director Andro Loladze former MP staff Archil Tsertsvadze Youth Leader: UNM Arnold Stepanian PMMG Beka Lakveheliani People’s Party- Youth Wing Besik Danelia GFSIS Participant: Democratic Movement-United Georgia Bakhtiyor Nishanov IRI Deputy Director Chiora Taktakishvili MP: Outgoing First Deputy Chair of Legal Issues Committee Cory Welt Professor, George Washington University Dato Gamisonia 24 Hours David Khijakadze nGnI trainer David Usupashvili Party Leader: Republicans Davit Dvali CDM Kutaisi City Council Davit Janiashvili MP staff: Chief of Staff of the Parliament Davit Kvetenadze GFSIS Participant: National Forum Derek Dohler Transparency International Dodo Shonava GBP. Channel2 Eka Azarashvili CEC Spokesman, FWL Eka Kemularia MP Staff: Chief Specialist of Human Rights and Civil Integration Cmte (FWL) Eka Rostomashvili Transparency International Eka Saatashvili Fortuna Radio Fron Nahzi GPAC COP Genadi Uchumbegashvili Internews Ghia Nodia Researcher: Professor, Ilia State University, CIPDD Gia Tsagareishvili MP: Member of “Unity for Justice” Faction Gia Zhorzholiani MP: incoming Soc. Dem. Georgian Dream Giorgi Gogoladze Youth leader, GFSIS Participant: National Democratic Party Giorgi Taktakishvili GFSIS Participant: Christian Democratic Movement Guram Chakhvadze MP: Outgoing Deputy Chair of Budget and Finance Committee Irakli Absandze Channel 9 Irakli Lekvinadze Expert for youth debates Irakli Tskhvediani New Rights, NGNI Irina Khasaia Chief of Staff of Healthcare and Social Issues Committee – FWL 2010-2011 Jondi Baghaturia MP: Member of “Unity for Justice” Faction Ketevan Mcedlidze Assistant to outgoing Vice-Speaker Mikheil Machavariani Keti Chavchava Executive Director, nGnI Keti Emukhvari GFSIS Alexander Rondeli GFSIS Koba Turmanidze Researcher: Country Director, CRRC Lali Khubulava Lagodekhi TV Lasha Aivazashvili Youth Leader: CDM Lasha Meskhi Youth Leader: Republican party Laura Jewett NDI Regional Director Levan Nishnianidze Intern: Procedural Issues and Rules Committee Levan Natroshvili Transparency International 61 | P a g e Levan Tsutskiridze NIMD Levan Vepkhvadze MP: Outgoing Vice-Speaker of the Parliament, Christian-Democrats Faction Luis Navarro NDI Country Director Magda Anikashvili Former MP, Christian-Democrats Faction Maka Gigauri MP Staff: Press Officer of the Parliament Mariam Pirtskhalaishvili former intern: LTA Assistant (2011-2012, Foreign Relations Committee) Mariam Robitashvili Intern: Gender Equity Council Mark Mullen Independent analyst Merab Kikabidze Mega TV Mikheil Jgenti Former MP chief of staff Misha Khetsuriani Youth Leader: Conservatives Nana Kalandarishvili Intern: Human Rights and Civil Integration Committee Natia Kuprashvili Regional Broadcaster Association Natia Pavliashvili GFSIS Participant: Georgia’s Way Nino Khutsidze Civil.ge Nino Lomjaria NGO: ISFED Nino Maisuradze MP Staff: Chief of Staff of the Education, Science and Culture Cmte (FWL) Nodar Jikia GYLA Ruso Machaidze Rezonansi Salome Svanadze former intern: Swedish Embassy (2010- 2011, European Integration Cmte) Shalva Kiknavelidze Youth Leader: National Forum (Youth Forum) Sopo Guruli UNDP Tamar Chikovani Expert for youth debates Tamar Chugoshvili NGO: Georgian Young Lawyers’ Association Tamar Ghvinianidze Rioni TV Tamar Kavkasidze GFSIS Participant: UNM Tamta Svanidze Youth Leader: Republican party Teona Gogishivili ISFED Thomas de Waal Carnegie Endowment for International Peace Tornike Zurabashvili former intern: IFES, (2010-2011, Foreign Relations Committee) Vladimer Jishiashvili MP Staff: Chief of Staff of Regional Policy, Self-Government and Mountainous Regions Committee Vladimir Papava Researcher: Senior Fellow, GFSIS Zurab Melikishvili MP: Outgoing Chair of Budget and Finance Committee Batumi NGO/Media focus group Nino Tavlalashvili GYLA Ilia Verdzadze BTUC Merab Tsulukidze Channel 1 Tariel Tsetskladze Journalists for Freedom /Guria News/Channel 9 Maia Katamadze Batumi agency for Development, Education and Employment Madona Beridze ISFED Maia Shavadze Channel 25 Vazha Megrelidze Democracy institute Telavi NGO/Media focus group Marekh Mgaloblishvili GYLA Enri Kobakhidze TV Tanamgzavri Zeinab Kobiashvili Union “Step Forward Valeri Gremelashvili Regional Development Agency Nana Kibishauri Chveni Radio/Chveni Gazeti Christine Telavi Agency for Education, Development and Employment Batumi Citizens focus group Nine citizens 62 | P a g e Telavi Citizens focus group Fourteen citizens Mini-survey of youth organization members 17 youth leaders from 10 different party youth wings Mini-survey of majoritarian MPs 25 former majoritarian MPs Mini-survey of journalists from regional media outlets 10 journalists 63 | P a g e ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST July 12, 2012 Ms. Eka Gamezardashvili and Mr. James C. Athanas USAID/Caucasus Mission 11 George Balanchine Street 0131 Tbilisi, Georgia Subject: Request for Task Order Proposal (RFTOP) #SOL-114-12-000008 Mid-Term Performance Evaluation of the Political Party Strengthening Project (PPSG) and Parliamentary Strengthening Project Dear Ms. Gamezardashvili: With regards to the subject task order proposal, I would like to confirm Social Impact has played no previous role in either the PPSG or PSP projects and we attest to the company’s lack of conflict of interest. I would furthermore like to assure USAID that Social Impact has received no information from the implementing organizations beyond that which is already publically available. Sincerely, Rolf Sartorius President 64 | P a g e 65 | P a g e 66 | P a g e 67 | P a g e 68 | P a g e ANNEX VI: TABLES TABLE C: MAJORITARIAN MP WEBSITE CONTENT MP Website Information Party No Content Basic biography Full biography Full biography and some information about activities Full biography and extensive information about activities Total CDM 1 1 NDP 1 1 RPG 2 2 UNM 19 17 17 10 5 68 Blank 3 3 Total 24 17 18 10 6 75 TABLE D: MAJORITARIAN MP WEBSITE UPDATE FREQUENCY Frequency of MP Website Updates Party Six months or less Six months to one year One to two years Never Total CDM 1 1 NDP 1 1 RPG 2 2 UNM 8 13 9 38 68 Blank pages 3 3 Grand Total 8 14 9 44 75 TABLE E: PCC MEDIA DEBATES BY CHANNEL Station #Debates Station #Debates Argo 5 Jikha TV 1 Bolneli TV 1 K. K. TV 9 Borjomi TV 2 Kolkheti TV 4 Channel 25 9 Lagodekhi TV 21 Channel 9 14 Marneuli TV 2 Dia 3 Mega TV 5 Guria TV 21 Ninth Wave 15 Gurjaani TV 10 Odishi TV 3 Imervizia 17 Rioni TV 14 Jikha TV 1 Trialeti 8 69 | P a g e TABLE F: PUBLIC MEETINGS BY TYPE Meeting Type Number Budget 12 PCC training 8 Public outreach 5 Constitutional amendments 4 Reports on trainings 3 Administrative Infringement and Procedural Code Working Group 2 Law on Payment Systems 1 Legislative drafting 1 Speech and news release writing 1 Training of trainers 1 TABLE G: YOUTH DEBATE VIEWS ON YOUTUBE Date Views 22-11-2011 162 24-12-2011 247 21-1-2012 444 28-1-2012 319 4-2-2012 268 18-2-2012 311 10-3-2012 207 24-3-2012 136 7-4-2012 274 7-4-2012 235 5-5-2012 199 TOTAL 2802 AVERAGE 255 70 | P a g e ANNEX VII: EVIDENCE MATRIX Obj1Training on regional party-building GFSIS policy trainings Polls Training on youth party-building Youth debates Interaction between factions and journalists Majoritarian MPs & constituency outreach PMC/PCC (also Obj3) Public meetings and field visits in regions on budgetary & other legislative matters Future Women Leaders Internship Program NGO Liaison Office Cooperate with key committees on institutional reform Voting Records Database PMP, project documents X X X X X X X X X X X X X X Media stories, relevant survey results, etc. X X X X Other documents, media stories, etc. X X X X X X Other documents, relevant survey results, etc. X Parliament documents, other materials X X X X X SSI with FWL participants X SSI with Internship participants X SSI with youth participants X X SSI with journalists X SSI with Parliament staff X SSI with Maj MPs X X X SSI with MPs X X X X X SSI with national party leaders X X SSI with regional party leaders X X SSI with NGOs X SSI with analysts, donors X SSI with journalists X X SSI with Parliament staff X X X SSI with MPs X X SSI with national party leaders X X X X SSI with regional party leaders X X X SSI with NGOs X X X Focus Groups with Citizens X X X X X Focus Groups with NGOs+Media X X X X X Mini-survey of MMPs X X X Mini-survey of Media X X X X Mini-Survey of Youth Leaders X X SSI with analysts, donors X X X X X X X X X X SSI with journalists X SSI with NGOs X SSI with regional party leaders X Implementer Materials External Views Focus Groups Indirect Beneficiary Direct Beneficiary Direct Beneficiary Materials Secondary Materials Mini-Surveys Parliamentary Strengthening Program (NDI) Obj2 Obj3 Obj1 Obj2 Obj3 Political Party Strengthening in Georgia (IRI) 71 | P a g e U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523