February 2013
This publication was produced at the request of the United States Agency for International Development. It was pre￾pared independently by Democracy International, Inc.
EVALUATION
Golos Impact Evaluation – Final Report
GOLOS IMPACT 
EVALUATION 
FINAL REPORT 
February 2013 
Contract No.: AID- DFD-1-00-05-00198-00 Order No.: AID-OAA-TO-11-00062 
PHOTO CREDIT 
The photograph on the front cover was taken by Kris Thorpe of Democracy International, Inc. on 
March 4, 2012. The photograph depicts a press conference given by Golos on election day. 
DISCLAIMER 
The author’s views expressed in this publication do not necessarily reflect the views of the United States 
Agency for International Development or the United States Government. 
CONTENTS 
Executive Summary ............................................................................................................................................ 1
Introduction ......................................................................................................................................................... 2
Political Context ................................................................................................................................................. 3
Activities Description ........................................................................................................................................ 4
Evaluation Questions ......................................................................................................................................... 6
Methodology ........................................................................................................................................................ 9
Results ................................................................................................................................................................. 13
Conclusion .......................................................................................................................................................... 25
Annex A: Key Tables and Graphs .............................................................................................................. A-1
Annex B: Golos, Election Monitoring and the Russian Internet .......................................................... B-1
Annex C: Election Forensics ....................................................................................................................... C-1
Annex D: Experiments on Attitudes to Observers and Foreign Funding ........................................ D-1
Annex E: Evaluation Statement of Work .................................................................................................. E-1
Annex F: Disclosure of any Conflicts of Interest .................................................................................... F-1
GOLOS IMPACT EVALUATION—FINAL REPORT 1 
EXECUTIVE SUMMARY 
Golos is a leading Russian nongovernmental organization (NGO) that aims to defend the rights 
of voters by observing elections in a range of regions throughout Russia. Golos’s main objective 
is to provide trustworthy information about Russian elections to voters, political parties, elec￾tion commissions, other NGOs, media outlets, and the domestic and international community. 
Using a combination of qualitative and quantitative techniques, including elite interviews, analysis 
of internet and social media networks, election forensics, and online surveys, Democracy Inter￾national, Inc. (DI) evaluated Golos’s role in influencing the quality of elections at the systemic 
and polling station levels and communicating its message to key constituencies. 
Overall, our analysis suggests that Golos has had a significant effect on elections and electoral 
politics in Russia. Our findings can be summarized as follows: 
 At the systemic level, Golos played an important educational role in the response to the 
fraudulent Duma elections of December 2011. Golos’s expertise and network made it a 
dominant player in the evaluation of elections and a key source of information on elec￾tion quality for independent Russian media. 
 Though Golos itself did not play a major role in directly mobilizing protests or new ob￾servers, there is evidence that the organization facilitated the operations of other elec￾tion observation groups in managing the upsurge in civic activism after the December 
Duma elections. 
 At the polling station level, there is not much evidence that Golos observers were able 
to prevent ballot stuffing in either the Duma or presidential elections. However, there 
are intriguing differences between polling stations observed by Golos and those not ob￾served by Golos. Most notably, there is significant evidence that polling stations Golos 
observed recorded lower vote shares for United Russia in the Duma elections com￾pared to other polling stations in the same neighborhood. We also found that observed 
polling stations were less likely to record 100 percent turnout than polling stations 
without Golos observers. 
 Golos is widely respected as an honest and reliable source of information and is highly 
valued by key stakeholder groups, including educated, prosperous, urban internet users. 
Golos has a significant and positive profile among these sections of the Russian public, 
despite serious government efforts to disparage Golos. 
 Golos’s impact among the broader public is much more limited. 
 Going forward, Golos’s primary challenge will be to continue to develop the strategy 
and resources needed to improve its public relations and political communications. 
GOLOS IMPACT EVALUATION—FINAL REPORT 2 
INTRODUCTION 
Golos is a Russian nongovernmental organization (NGO) that aims to defend the rights of vot￾ers by observing elections in several regions throughout the Russian Federation. Golos is regis￾tered in Russia as an NGO and strives to maintain a strict policy of independence from political 
parties. Golos’s observation activities include long-term election monitoring, election day obser￾vation, a postelection parallel count of polling station reports, and advocacy work aimed at influ￾encing election laws and procedures. Golos’s main objective is to provide trustworthy infor￾mation about Russian elections to key stakeholders, including voters, political parties, election 
commissions, other NGOs, media outlets, and the domestic and international community. 
Golos was funded, in part, by the United Stated Agency for International Development (USAID), 
indirectly via the National Democratic Institute (NDI). USAID has commissioned Democracy 
International, Inc. (DI) to evaluate the impact of Golos’s work. For the purposes of this study, 
“impact” should be defined as Golos’s intended effect on the openness and fairness of the elec￾toral process. 
DI conducted an impact evaluation comprising surveys, qualitative research, quasi-experiments, 
and experiments in order to gauge the effectiveness of USAID support for Golos in Russia. 
There are numerous challenges to assessing the impact of an organization like Golos in a coun￾try such as Russia. Some challenges, such as isolating the effects of Golos’s own activities from 
broader trends in the country, are inherent in any evaluation of this kind. Other challenges are 
more specific to a context in which a difficult political environment puts limits on the types of 
feasible evaluation methods. Overcoming these challenges requires a sound theoretical frame￾work underpinning the evaluation and a corresponding and clearly defined set of measures. In 
this context, the evaluation employs multiple research methods and multiple indicators to assess 
Golos’s impact. This report summarizes the findings of the different aspects of the evaluation. 
GOLOS IMPACT EVALUATION—FINAL REPORT 3 
POLITICAL CONTEXT 
In winter 2011–2012, Russia faced elections for its two most important political institutions—
the parliament (or Duma) and the presidency. Before the election cycle began, incumbent Presi￾dent Dmitri Medvedev announced his intention to stand down and support the return of then￾Prime Minister and former two-term president Vladimir Putin. Putin, in turn, stated that he 
would appoint Medvedev as prime minister. This decision was supported by United Russia, the 
largest party in the Duma. 
Hopes for a smooth ratification of these decisions by the Russian electorate, however, were 
dashed in the Duma elections of December 4, 2011. United Russia in the official results won 49 
percent of the votes, but there were widespread allegations of electoral fraud both from domes￾tic monitoring organizations such as Golos and from international observers. Protests against 
fraud quickly followed, and by the end of December they had escalated into the largest protests 
seen in the Russian capital since the fall of communism. 
Beyond the implications for Russia, these dramatic events had significant implications for both 
Golos and the impact evaluation. For Golos, the protests were at the same time a vindication of 
their efforts to publicize election fraud and a cause of increased pressure from the Russian gov￾ernment. On the positive side, the mass protests intensified public interest in election monitor￾ing, leading to tens of thousands of people signing up to volunteer as election monitors for the 
March presidential elections. Under pressure to show that the presidential elections would be 
clean, the Russian government installed webcams in all polling stations for the presidential con￾test. The protests also led the Kremlin administration to introduce new legislation that would, 
among other things, ease the registration of political parties to compete in future elections and 
reinstate elections for regional governors, which had been abolished in 2005. 
However, the heightened political tension also meant an increase in pressure on Golos. Golos 
has been the subject of an intense campaign in parliament and the media designed to portray 
them as foreign agents and increased scrutiny from tax and other authorities both in the capital 
and in the regions. Golos’s leadership in Moscow has been directly targeted for harassment and 
the organization was forced to find new headquarters in the days leading up to the presidential 
election. In addition, on the day of the Duma elections, Golos’s communications systems came 
under intense cyber-attack. 
From the perspective of the impact evaluation, however, the election cycle and the postelection 
reaction presented a perfect opportunity to view the role played by Golos. With most observ￾ers surprised, even stunned, at the extent of the protests, we were able to assess what role, if 
any, Golos played in perhaps the biggest shake-up in Russian politics in a decade. 
GOLOS IMPACT EVALUATION—FINAL REPORT 4 
ACTIVITIES DESCRIPTION
In undertaking the evaluation, we attempted to assess the impact of a range of long-term and 
short-term activities undertaken by Golos with support from USAID. These activities can be 
broken into four categories: (1) long-term elections-related observation; (2) short-term obser￾vation of the December 2011 Duma elections; (3) short-term observation of the March 2012 
presidential election; and (4) postelection reporting and advocacy. These activities are summa￾rized below. 
LONG-TERM ELECTIONS-RELATED OBSERVATION 
 Golos reports on the registration of parties and candidates and campaigns in 48 locali￾ties, highlighting where violations have occurred at the local elections committee level. 
 Golos produces an online Map of Violations1 that illustrates the incidence of election vi￾olations reported by Russian citizens and activists. Reported violations are mapped for 
federal and regional elections and continuously updated. Though Golos does not investi￾gate or independently verify the individual reports, the organization produces written 
statements and press-releases about these violations on an ongoing basis. Before the 
2011 Duma elections, there were 5,000 reports of violations in the pre-election period, 
a substantial increase from the 2007 Duma elections. Nevertheless, it is impossible to 
know whether this increase is due to more elections violations or better reporting. 
SHORT-TERM OBSERVATION – DECEMBER 2011 DUMA ELECTIONS 
 Golos trains its nationwide network of activists in electoral law and monitoring tech￾niques. Due to Russian election law that restricts access to polling stations to political 
party representatives and the media, Golos-trained monitors are all registered as corre￾spondents for its newspaper, Civic Voice, which is published five times per year with a 
circulation of 2000 to 4000 copies. Golos monitors were all paid in the Duma elections 
and all volunteers in the presidential elections. 
 Golos conducts election day observation through a mix of fixed and mobile monitors 
(approximately 2,000 monitors covering approximately 4,000 polling stations in 40 re￾gions). Monitors complete a checklist with more than 60 questions designed to check 
the implementation of the range of administrative requirements in Russia’s election laws. 
The checklists do not contain an overall assessment question. The results of checklists 
are submitted by telephone or email to Golos’s regional headquarters as they are com￾pleted, resulting in a full dataset soon after the polls close on election day. The file al￾lows analysis across different regions, and aspects of the process. 
 Golos also operates an election day hotline with 40 operators for the general public to 
report violations. In the 2011 Duma elections, the hotline received 3,000 calls (roughly 
half of which were thought to by Golos to be genuine, with the rest probably an at-
 
1 Available at http://www.kartanarusheniy.ru/. 
GOLOS IMPACT EVALUATION—FINAL REPORT 5 
tempt to hamper Golos communications). Citizens also submit reports of violations to 
the website. 
 Golos election day activities include press conferences, live blogging, and tweeting to 
keep journalists and members of the public up-to-date on reports of violations, and real￾time feedback from the observers. 
SHORT-TERM OBSERVATION – MARCH 2012 PRESIDENTIAL ELEC￾TION 
 Golos conducted a large-scale observation of the presidential election. Through its net￾works of correspondents, Golos monitored violations during the voting and counting 
processes. There were 1,218 volunteer observer groups, who visited approximately 
6,400 polling stations in 45 cities. 
 Members of the public were able to report violations of election law and procedures 
through the Map of Electoral Violations hotline, cosponsored by Golos and the online 
newspaper Gazeta.ru. 
 Golos also implemented a new SMS-CEC (Central Election Commission) project where 
observers sent copies of official polling station protocols to Golos via SMS. 
POSTELECTION REPORTING AND ADVOCACY 
 Golos held several well attended postelection press conferences to summarize the re￾sults of the election observation and make a statement on the legitimacy of the election. 
 Golos produced a full report containing all of the published analyses of the elections and 
the map of election violations. Full details of the reports and related activities for both 
elections are available online at http://www.golos.org/elections/2011-12-04 and 
http://www.golos.org/elections/2012-03-04. 
 A key element of Golos’s postelection activity has been advocacy work within the Duma 
and elsewhere to promote reform of the electoral code and to influence election laws 
and procedures. 
 Golos does extensive work on local and municipal elections and campaigns throughout 
the year—not just according the federal election cycle. Notable recent work includes 
participation in the campaign for a rerun of fraudulent elections in the city of Astrakhan, 
which resulted in a hunger strike and a substantial campaign involving people from 
across Russia. While this report cannot address in detail all the work Golos does in dif￾ferent contexts, Golos’s work in situations that often have limited exposure in the in￾ternational media, the organization’s work is nonetheless important in the construction 
of a law-bound democratic state in Russia. 
GOLOS IMPACT EVALUATION—FINAL REPORT 6 
EVALUATION QUESTIONS 
In recent years, social scientists have increasingly seen elections in authoritarian contexts like 
Russia not simply as an irrelevant sham, but rather as crucial to maintaining nondemocratic re￾gimes. One key function of elections in authoritarian elections is to illustrate the capacity of the 
ruling group or party to dominate the polls and demonstrate real underlying popularity. To do 
this, incumbents need to actively maintain the fiction that elections are free and fair, or to show 
that they can administratively deliver victory with ease. In this context, independent observers 
who can provide reliable and credible evidence of electoral violations to key stakeholders and 
politically active groups both within the country and abroad can play a crucial role in shaping 
both the narrative and the politics of the postelection period. Evaluating Golos’s ability to play 
this role requires assessing both the credibility and quality of Golos’s reporting and the use and 
effectiveness of their communication strategy. 
In addition to postelection reporting, we might expect that Golos would have a direct effect on 
the quality of elections. Some scholars argue that election observers, through the activity of ob￾servation itself, can contribute to cleaner and more honestly conducted elections. In addition, 
election day checklists can help diagnose systematic administrative issues with the conduct of 
elections and promote reform. The effect of observers on election quality is most likely to be 
seen where observers have the cooperation of local and national political authorities. In the 
Russian context, this kind of support is clearly absent, and Golos is very circumscribed in the 
extent to which it can act as a deterrent to electoral abuses. Nevertheless, as part of the evalua￾tion, we attempt to identify what effect (if any) Golos has had on the conduct of elections. 
With this theoretical background in mind, we reviewed strategic documentation from Golos, 
conducted in-depth interviews with Golos Director Lilya Shibanova and Deputy Director Grigo￾ry Melkonyants, and observed Golos’s activities around the 2011 Duma and 2012 presidential 
elections. Based on this understanding, we defined two distinct evaluation domains: (1) Golos’s 
effect on the quality of the electoral process at the overall system level and at the polling station 
level; and (2) the effectiveness and credibility of Golos communications. 
SYSTEMIC AND POLLING STATION LEVEL EFFECTS 
A central issue for the research is to evaluate what effect, if any, Golos’s activities have had on 
the quality of the electoral process in Russia. There are two levels at which Golos might have an 
effect on the quality of Russian elections: at the level of the national political system, and at the 
level of the conduct of elections and vote counting at the polling places themselves. For each of 
these levels, we ask the question: Is Golos having an effect on making elections in Russia fairer 
and more open? 
Breaking this general question down, we collected evidence to test the following hypotheses: 
 Hypothesis 1: Golos’s activities put pressure on the Russian government to improve 
the rules governing elections and to improve the quality and impartiality of en￾forcement of those rules. 
We examined this general hypothesis by testing three specific sub-hypotheses relating to 
the direct and indirect effects Golos had in the politics of the December 2011–May 
2012 election cycle: 
GOLOS IMPACT EVALUATION—FINAL REPORT 7 
o Hypothesis 1a: Through its core activities, Golos had a direct effect on stake￾holder and media perceptions of whether the elections were free and fair. 
o Hypothesis 1b: Golos influenced the politics of the postelection period because 
other election-oriented groups and initiatives relied to a critical extent on the 
informational and communications infrastructure developed by Golos over the 
years since its activities began. 
o Hypothesis 1c: Golos’s network of activists played an important informal role in 
in growing and coordinating the upsurge in interest and participation in civic ac￾tivism around elections in Russia. 
 Hypothesis 2: Polling stations actually observed and assessed by Golos show less ev￾idence of electoral fraud than polling stations in the same regions not observed by 
Golos. 
COMMUNICATIONS AND RELIABILITY
In addition to the overall effects on election quality, we also set out to investigate whether 
Golos effectively communicates its message and whether Golos is seen as a credible source of 
information and analysis of elections by key constituencies. We conceived of Golos key constit￾uencies in two ways. First, we defined a group of stakeholders which included leading members 
of the opposition, civil society, media (both independent and pro-government), political consult￾ants, and high profile business groups. In interviewing stakeholders we focused on the following 
hypotheses: 
 Hypothesis 3: Stakeholders use and value the information gathered by Golos. 
 Hypothesis 4: Stakeholders find the information gathered by Golos to be a reliable guide 
to actual underlying patterns of fraud. 
 Hypothesis 5: Stakeholders find Golos less credible because they are supported by for￾eign sources. 
Given the sharp division in Russian politics between the voters at large and the so-called “middle 
classes,” who have made up much of the opposition to the current regime, we defined the sec￾ond key constituency for Golos as middle- to upper-income Russians with at least some higher 
education who use the internet and who live in one of Russia’s 10 cities with a population of 
over one million people. While we might not expect the public at large to have much useful 
knowledge about electoral observation organizations, Golos’s visibility and influence among ur￾ban middle classes is a key test of its communication and public relations strategy. Focusing on 
these groups, we tested the following hypotheses: 
 Hypothesis 6: Middle-class urbanites are aware of Golos and trust Golos as a major 
source of information on elections in Russia. 
 Hypothesis 7: Middle-class urbanites who are exposed to Golos materials are likely to 
be more skeptical about the quality of Russian elections. 
GOLOS IMPACT EVALUATION—FINAL REPORT 8 
 Hypothesis 8: Middle-class urbanites who are exposed to Golos materials and also to 
government counter-claims that Golos is supported by foreigners are less likely to be 
influenced by Golos materials. 
GOLOS IMPACT EVALUATION—FINAL REPORT 9 
METHODOLOGY 
Conducting an impact evaluation of an organization like Golos is complex. There is no simple 
experimental design or natural experiment that can be used to isolate whether or not the elec￾toral context would have been different without Golos. As such, a multimethod approach is re￾quired to allow us to triangulate and come up with an overall assessment. In this section, we 
discuss the various methods used. 
QUALITATIVE RESEARCH 
As noted above, the evaluation sought to assess Golos’s importance in pushing for improve￾ments in electoral processes in Russia and to gauge the degree to which Golos is a credible and 
influential player in Russian politics. One extremely important way to do this is simply to inter￾view key participants and observers in the Russian election process and ask their opinion. Did 
Golos feature in the politics around the elections? What role did observer reports and related 
materials play in the response to the elections on the part of civil society, opposition groups and 
the government? What changes, if any, were introduced into the way elections were conducted 
between the Duma elections and the presidential elections and to what extent might we consid￾er Golos’s activities to have played a role in these? What are the main strengths and weakness 
of Golos’s operations? 
To address these issues, we discussed with Golos the groups the organization felt were its key 
target audience, or stakeholders. Recognizing that not all citizens are equally important in terms 
of political influence, and aware of its limited ability to access government controlled television, 
Golos has sought in large part to work through its influence on key elites the government, op￾position, media and business. Consequently, we used a local consulting firm to help identify lead￾ing Russian citizens from a range of different fields, representing a spectrum of pro-government, 
opposition, and independent expert opinion. Respondents were selected in six categories: gov￾ernment influentials, business influentials, independent election observers, social scienc￾es/political consultancy, media/publicity, and protest movement/opposition. We sought two re￾spondents in each group, though the categories are necessarily rather fluid given the nature of 
Russian politics, business and society. Our goal was to interview people who were closely in￾volved in politics and, though well known in political circles, might be expected to give a rela￾tively frank account of their opinions. We were able to recruit a sample of 11 very prominent 
people, including four individuals with clear pro-government links, two respondents who are 
publicly known as opposition figures, and five people who had worked on both sides at different 
times. Each respondent sat for an hour-long face-to-face interview that aimed to cover both 
general issues of the current moment in Russian politics and specific questions relating to the 
role played by Golos within the general population of groups and actors involved in the electoral 
process in Russia. This exceptional collection of interviews helps us to see “beyond the num￾bers” and analyze the role played by Golos in shaping views of elections in Russia. 
QUANTITATIVE RESEARCH 
In addition to the qualitative analysis, we also developed a range of quantitative metrics that 
could be used to verify qualitative data. Four distinct quantitative techniques were used to eval￾uate different aspects of Golos’s impact. 
In order to get an additional perspective beyond the qualitative research on the degree to which 
GOLOS IMPACT EVALUATION—FINAL REPORT 10
Golos featured in the politics around the elections and the role played by observer reports and 
related materials in the emergence of the protest movement, we analyzed Golos’s position on 
the internet and social media. Given that the internet and social media are now clearly the prin￾cipal context within which open discussion and political organization take place in Russian poli￾tics in general, and in the opposition in particular, a central piece of the research was to look at 
how often and in what context Golos and election monitoring were discussed on the internet 
during and after the December 4, 2011 Russian parliamentary elections, the March 4, 2012 Rus￾sian presidential elections, and the ensuing protest mobilizations. 
In addition to looking at the quantity of communications, we used a combination of computer￾assisted and human analysis to examine the content of relevant communication and the relation￾ship of these communications to events and third-party sources of information (e.g., videos and 
other media, links to Golos from other websites, links to others from Golos, etc.). This piece of 
the project seeks to identify what audiences Golos and other independent election monitors 
reach most effectively and what relevant audiences they fail to reach. In addition, we gain insight 
into how Golos is connected to its allies and its audiences, including the social pathways by 
which ideas are spread. 
Data covering the period from November 1, 2011 through March 31, 2012 were drawn from a 
full-text corpus of more than 2,000 Russian-language blogs (primarily from the Live Journal plat￾form) and approximately 50 online news media sites, as well as social-networking systems Face￾book and Twitter. Methodologies employed included meme detection and tracking, network 
and cluster analysis. The specific data and methods used are described in detail in Annex B. This 
innovative piece of research allows an assessment of the relationship between Golos’s activities 
(and those of its allies) and public perceptions of the election results, as well as the subsequent 
emergence of independent election monitoring at the core of the opposition movement. 
Beyond the issue of Golos’s role in creating pressure for fairer elections at the system level, we 
also used quantitative analysis of election returns by polling station (i.e., election forensics) to 
assess the direct impact of election observers on voter fraud. The analysis took advantage of the 
fact that observation of polling stations was randomized within neighborhoods (raiony). Hence, 
while we cannot compare observed polling stations to all unobserved polling stations, we can 
compare election returns from observed and unobserved polling stations within the same neigh￾borhoods to see what differences there were. Specifically, we look to see if there is more evi￾dence of ballot stuffing at unobserved stations, whether there are differences between different 
kinds of region, and whether alleged cheating by using mobile ballot boxes or absentee ballots 
was more likely in stations without election observers. 
In order to evaluate the credibility and reach of Golos’s communications strategy, we undertook 
two types of public opinion research. Taking advantage of nationally representative public opin￾ion surveys conducted in four monthly omnibus polling and polls conducted before and after 
both parliamentary and presidential elections. The surveys were all fielded by the Levada Analyt￾ic Center. In the omnibus surveys, the sample size was approximately 1,600 respondents, drawn 
from urban and rural areas of 45 Russian regions. The Duma election surveys were conducted in 
two waves, each involving approximately 1,200 respondents in 46 regions, before and immedi￾ately after the Duma elections. 
While broad national surveys are interesting, Golos rightly felt that a combination of govern￾ment restrictions on their access to television and the relatively esoteric nature of their work 
meant that a high profile among the population at large was unlikely to be achievable. Neverthe-
GOLOS IMPACT EVALUATION—FINAL REPORT 11
less, given Golos’s freedom to spread its message on the internet, we defined an intermediate 
group of citizens that Golos might have a realistic chance of reaching. This group we refer to as 
Socially and Politically Active Russian Citizens (SPARCs) and are defined as frequent internet 
users, living in major cities (i.e., cities with a population of more than 1 million), with higher edu￾cation and who are relatively prosperous. Drawing on internet survey panels used in market 
research, we conducted both pre- and postpresidential election surveys. Approximately 1,200 
respondents in each round participated in 20–25 minute long surveys probing their attitudes and 
responses to materials by and about Golos and other topical political issues. In the context of 
the postelection protests in December–March, this group is one in which many political analysts 
and actors were extremely interested. Our two wave survey of SPARCs is, we believe, the first 
to use the internet to focus on key political demographics in Russia. 
The questions focused on a number of key issues, including awareness of Golos and other 
NGOs and whether Golos and other groups are perceived to be honest, accurate, and inde￾pendent. In particular, we used embedded survey experiments in which different prompts are 
randomly assigned to different respondents to analyze whether perceptions of Golos and its 
work on elections are shaped either by the fact that it receives significant funding from foreign 
sources, or by propaganda efforts to claim that Golos works on behalf of foreign governments. 
Furthermore, since Golos circulates a lot of its material on electoral violations through other 
means, such as newspapers and YouTube, we also assessed voters’ perceptions of the general 
fairness of elections and whether attitudes about fairness affect attitudes about the legitimacy of 
the government. 
LIMITATIONS 
We are confident that the evaluation’s multimethod approach allows us to make a fair and well￾founded evaluation of Golos’s work. Nevertheless, there are a number of caveats that must be 
kept in mind when considering the results presented below. In this section, we discuss the eval￾uation’s main limitations, focusing on methodological constraints. 
We used a mixture of qualitative and quantitative methods to conduct the evaluation. Qualita￾tive interviews with participants and close observers are essential in an evaluation of this nature, 
but interviews suffer from well-known flaws. Respondents are often self-interested and have 
biases in both perception and in what they choose to report to interviewers. Their assessments 
are inherently subjective. In order to account as much as possible for the inherently subjective 
nature of respondents’ assessment, we interviewed across different fields of specialization as 
well as across political perspectives. Consequently, the broad consensus on the importance of 
Golos across the political spectrum, even among those who see their role as purely negative, is 
quite strong evidence of the key role they have played. 
Quantitative methods also have their limitations. The internet and social media analysis tech￾niques we have used to evaluate Golos’s role in postelection politics are relatively new and 
there is an absence of comparative data and established standards of inference. For example, we 
are able to analyze the nature of the networks between different election observer groups in 
Russia, but we know neither how these networks compare to those between election observer 
groups in other countries, nor how they compare to networks between other kinds of group in 
Russia. Moreover, while network analysis itself is not new, there may be particularities of analyz￾ing internet activity that are not currently well understood. 
Similarly, there are several limitations that apply to analysis of polling station data (also known as 
GOLOS IMPACT EVALUATION—FINAL REPORT 12
“election forensics”) to compare relative levels of fraud. Although it has a highly precise￾sounding name, election forensics is a rather inexact science, and only large differences between 
levels of fraud in different sets of polling stations can be identified. Even in these instances, we 
do not have definitive evidence that can prove the existence of fraud, but rather knowledge of 
the statistical likelihood of certain patterns. Unlikely patterns are cause for suspicion and differ￾ences in patterns suggest differences between observed and unobserved polling stations, but are 
not definitive proof of electoral violations. 
A related critique of election forensics is that we cannot be certain that a correlation between 
turnout and the percentage of the vote for a particular party is due to fraud or simply due to an 
effective “get out the vote” effort. Generally speaking, this critique is quite powerful. In the par￾ticular design we have used, however, where we look for differences between randomly select￾ed polling stations within the same neighborhoods, this problem should not arise unless there is 
some correlation between the selection of observed polling stations and the effectiveness of 
voter mobilization efforts. Although Golos did not select cities or neighborhoods randomly, the 
polling stations selected for observation within neighborhoods (raiony) were randomized using 
standard techniques based on Kish tables. As a result, there should be no systematic differences 
between observed and unobserved polling stations within neighborhoods. In this regard, there￾fore, the experimental design is quite robust. 
Nevertheless, there are possible issues with the treatment that was actually applied to polling 
stations and the difference between treatment and control groups. We only have data from 
Golos observers, and not from election observers from other groups that were active in both 
elections. Thus, we cannot be sure that the “unobserved” polling stations were completely un￾observed (though we can be sure they were not observed by Golos). For the Duma elections, 
where Golos was the most significant independent election monitoring group, the problem is 
smaller, though the issue of political party observers remains. For the presidential election, the 
problem is potentially larger since election observers from other groups were quite active. 
However, even in this case, the main efforts of other election monitoring groups were focused 
in Moscow and Moscow oblast. Outside of these regions, therefore, the extent of the treatment 
problem should be relatively small. 
Compared to election forensics and analysis of Golos’s web presence and networks, the proper￾ties and limitations of survey data are well understood. Most importantly, given the nature of 
the data, it is very difficult to distinguish causation from mere correlation. While we can show, 
as we do below, that people who are more familiar with Golos’s work are also more skeptical 
of the quality of Russian elections, it is hard to demonstrate that Golos’s materials in fact caused 
this increased skepticism. One way we have mitigated this problem is by separating respondents 
and randomly subjecting them to differently framed questions. This technique helps us to exam￾ine the effect of, for example, specifically mentioning Golos when asking about election observ￾ers. However, even with this kind of experimental design, separating out the effect of the 
prompt from information previously available to respondents is inherently difficult. 
It is clear, therefore that this evaluation, like any other, has its limitations. Each of the specific 
methods we use has its strengths and weaknesses. Nevertheless, by adopting a multimethod 
approach, including rigorous experimental and quantitative techniques, and triangulating between 
different kinds of evidence, we believe that we are in a very strong position to make a well￾founded assessment of the project. 
GOLOS IMPACT EVALUATION—FINAL REPORT 13
RESULTS 
So far, we have identified the key questions addressed by the evaluation and the techniques used 
to answer them. In this section, we summarize the results, focusing on the two domains de￾scribed above: (1) systemic and polling station effects, and (2) communications and reliability. 
IS GOLOS HAVING AN EFFECT ON MAKING ELECTIONS IN RUSSIA 
FAIRER AND MORE OPEN? 
In this section, we use qualitative interviews and internet and social media analysis to examine 
the role Golos played in generating pressure on Russian authorities for change in the conduct of 
elections in Russia. We break down the possible effects into three different mechanisms: (1) 
direct effects, (2) indirect effects through other organizations, and (3) indirect effects through 
individual activists. We find substantial evidence that Golos is an important opinion former, both 
through its effect on key Russian media outlets and through its strong internet presence. We 
also observe Golos and its activists playing an important role in the postelection mobilization, 
even as Golos sought to preserve its political neutrality. 
We also look at the narrower question of Golos’s impact on the conduct of elections at the 
observed polling stations. We compare the results at observed polling stations with unobserved 
polling stations in the same neighborhoods. Particularly for the Duma elections, potential evi￾dence of electoral irregularities is pervasive, and there is clear evidence of different levels of 
fraud in observed and unobserved polling stations, indicating that the presence of Golos election 
observers had an effect. While it is impossible to tell on the basis of the evidence here what the 
source of the differences was, the data do provide prima facie evidence that the presence of 
Golos observers had an effect on reducing fraud at least in the Duma elections. 
SYSTEMIC LEVEL 
Golos’s effect on making Russian elections fairer and more open is perhaps the central issue in 
our evaluation and, at the same time, represents an extremely high bar for an organization with 
a small budget and limited access to television, compared to the tremendous resources of the 
Russian state. Nevertheless, our findings suggest that Golos is indeed a key player in the process 
of monitoring, regulating, and advocating for change in the electoral process. As high as the bar 
is, it seems clear that Golos has also had a positive effect in pushing for fairer, more open elec￾tions in Russia. In this section, we consider two principal sources of evidence to support such a 
claim: qualitative interviews with stakeholders and analysis of the role of Golos with respect to 
upsurge in civic activism after the December Duma elections that had such a big impact on con￾temporary Russian politics. Broader survey evidence is discussed later. 
In assessing Golos’s role during the 2011–2012 election period, we identified three discrete 
ways in which Golos may have had an impact. First, through its core activities, Golos had a di￾rect impact on key media and opinion formers about the quality of the elections. Second, other 
election-oriented groups and initiatives relied to a critical extent on the informational and com￾munications infrastructure developed by Golos over the course of its existence. Third, Golos’s 
networks of activists played an important informal role in growing and facilitating the upsurge in 
interest and participation in election monitoring activities between the December Duma elec￾tions and the March presidential contest. Each of these aspects were examined in detail through 
qualitative interviews with stakeholders and by analyzing Golos’s online and social media pres-
GOLOS IMPACT EVALUATION—FINAL REPORT 14
ence to discover how its network related to and interacted with other groups in the aftermath 
of the Duma elections. 
Hypothesis 1a: Golos, through its core activities, had a direct impact on stakeholder and media percep￾tions of whether the elections were free and fair. 
In qualitative interviews, respondents stressed Golos’s importance in shaping the information 
environment of Russian elections. According to our interviews, Golos has been seen for many 
years as the most knowledgeable, reliable, and sophisticated organization conducting election 
monitoring in Russia. In addition, many see Golos as a morally uplifting, even inspirational, organ￾ization. Respondents emphasized that Golos is seen as an organization that “is not afraid” and is 
“effective and, therefore, persecuted.” As one well-known businessman said, “Even those who 
are critical in public say to themselves, ‘thank God such people exist.’”
Quantitative analysis of Golos’s internet presence confirmed the qualitative interview findings. 
Golos clearly has a substantial and prominent online presence that extends well beyond its pro￾prietary websites and social media pages. While attention to Golos often peaks during external￾ly-created scandals—mostly due to actions taken against Golos by the government—there is a 
broad and consistent flow of attention, particularly by online media outlets, to Golos’s reporting 
and analytic activities (see Annex B). Moreover, Golos’s voice is heard and retransmitted by 
online media outlets that served as key informational hubs during the December 2011 and 
March 2012 protests. As Table 1 in Annex A illustrates, key online newspapers and radio sta￾tions that played a major role in disseminating information about the December elections and 
subsequent protests, such as Kommersant.ru, Gazeta.ru, and Echo Moskvy, relied heavily on 
information from Golos for stories on election monitoring, violations, and legislation. Golos was 
the key group to which these media outlets turned, and, as such, Golos played a crucial role in 
communicating and distributing news of electoral fraud. Consequently, there is good qualitative 
and quantitative evidence that Golos has a direct impact on stakeholder and independent media 
perceptions of the fairness of elections. 
Hypothesis 1b: Golos influenced the politics of the postelection period because other election-oriented 
groups and initiatives relied to a critical extent on the informational and communications infrastructure 
developed by Golos over the course of implementation of its core activities. 
In contrast to its role in shaping perceptions of the quality of elections, Golos was not seen as a 
major force in mobilizing protesters after the elections in any of the qualitative interviews, and, 
further, Golos’s leadership did not see this as its primary task. Golos has always maintained a 
position of political neutrality and has never been directly involved in political mobilization. Nev￾ertheless, given its expertise and network, it is not unreasonable to examine Golos’s role in the 
broader mobilization and, in particular in the upsurge in election monitoring activity. Golos has 
by far the largest collection of election-related materials, including reports, handbooks, and legal 
briefs, of any organization, owing to its history and professional focus and so might be expected 
to play an important role in this area. 
Qualitative interviews with Golos officials and leaders of other mobilizing groups, such as the 
League of Voters (the largest association campaigning for clean elections founded after the Du￾ma elections of December 2011), testify to Golos’s role in providing start-up organizations with 
key materials and trainings for volunteer observers. Golos’s willingness to cooperate with and 
assist newcomer organizations was critical in helping these groups build the organizational infra￾structure and technical capacity they needed to put thousands of new volunteer observers in 
GOLOS IMPACT EVALUATION—FINAL REPORT 15
the field for the presidential elections. It is also notable that Golos did not interpret the emer￾gence of these new organizations as a challenge to its preeminent role in the field of election 
observation, but rather worked hard to contribute to the new sense of activism and tried to 
make sure new volunteers were as well trained as possible. 
The qualitative interviews are corroborated by the quantitative analysis of Golos’s online pres￾ence, which both dwarfs that of its peers and is closely connected to the work of others. To 
illustrate, Figure 1 in Annex A shows a network map of 5,514 hyperlinks into (“inlinks) and out 
of (“outlinks”) websites—including proprietary sites and social media pages—involving Golos 
and other election monitoring organizations. The map illustrates Golos’s dominance on webpag￾es focused on elections and election observation in Russia, and its integral role in the election 
movement as a whole. 
A couple of caveats should be kept in mind. First, due to the nature of the data, the links includ￾ed are static, and many of them may be several years old or more. As such, Golos’s dominance 
in this network map—primarily on the blogging platform Live Journal, but also on its homepage 
and Twitter—is to some extent an artifact of Golos’s relatively longer history. Nonetheless, be￾cause the internet itself is cumulative, consisting of pages and resources that, unless actively re￾moved, do not disappear, this is more or less an accurate representation of the election moni￾toring-related landscape on the Russian-language World Wide Web. Second, it is noteworthy 
that most of the links to Golos are to its home page rather than to specific documents. Thus, it 
is difficult to identify which key pieces of Golos’s work played a major role. 
Nevertheless, these caveats notwithstanding, there is good reason to credit Golos with having 
played a major role in the postelection period through the reliance of other groups on the ex￾pertise and technical capacity that Golos has built over the years. Golos played a major role in 
helping to train the wave of citizens newly interested in election monitoring and was clearly in￾tegral to the process of information exchange about elections and election monitoring in Russia. 
Hypothesis 1c: Golos’s networks of activists played an important informal role in the upsurge in interest 
and participation in civic activism around elections in Russia. 
A third way in which Golos might have contributed to pressure for improvements in Russian 
elections in the recent cycle were its activists (as opposed to the organization) playing an infor￾mal role in broadening and strengthening the protest movement. Once again, Golos’s policy of 
political neutrality limited the degree to which Golos’s core staff and activists were active in 
other organizations’ social media presence. Nevertheless, there is significant evidence that 
Golos’s on-line community played an important role in the network as a whole, bringing with 
them the habits of discussion and distribution of Golos’s analytic materials. 
We analyzed interactions by users of the Facebook communities of Golos, and the two other 
most significant election observer groups that emerged following the protests against fraud in 
the Duma elections—Citizen Observer (Grazhdanin Nabliudatel’) and the League of Voters (Li￾ga Izbiratelei). We looked at Facebook on the assumption that the membership of Golos’s Face￾book community is likely to reflect members of its core network of activists. Figure 2 in Annex 
A presents a network map illustrating relationships between people who are part of one or 
more of the three observer groups’ Facebook communities posting and commenting on each 
other’s pages during a five-day period leading up to the presidential election. This period was 
selected as one likely to witness peak contention and thus capture the largest number of partici￾pants. The dataset, which includes only publicly visible interactions and was collected using 
open-source software, comprises 13,238 interactions by 780 users. A map of the resulting amal-
GOLOS IMPACT EVALUATION—FINAL REPORT 16
gamated network is presented in Figure 2. The map illustrates two key points. First, the map 
demonstrates the importance of Golos’s community (in green) in sheer size relative to the oth￾er networks. In this period, as volunteer observers were gearing up for the presidential elec￾tions, participants in Golos’s Facebook community were very actively involved in discussions and 
exchanging information on election observation. Second, it is interesting to note that while each 
of the communities display extensive internal communication, there is relatively little intermin￾gling among organizations, despite broadly shared thematic interests. Connections between the 
three organizations are dominated by a small group of “brokers” (for detailed analysis of these 
brokers, please see Annex B). This helps to illustrate a division of labor within the movement as 
a whole, within which Golos provided information and analysis, Citizen Observer provided logis￾tics and organization, and the League of Voters focused on political rhetoric and mobilization. 
Hence, the evidence suggests Golos’s influence on the increase in civic activism after the De￾cember Duma elections was not limited to providing training or technical know-how. Over the 
years Golos has built up a substantial network of people interested in elections and election ob￾servations. Network analysis of social media suggests that these activists played a major role in 
the upsurge in interest and participation in civic activism around elections before the March 
2012 presidential elections. 
POLLING STATION LEVEL 
Having looked at Golos’s overall role in postelection opinion shaping and civic activism, in this 
section we examine the effect, if any, Golos observers had on outcomes at the polling station 
level in both the Duma and presidential elections. We also explain why, under the circumstanc￾es, the impact of Golos observers at the polling station level is likely to be relatively limited, and 
detail evidence of some small but interesting effects. 
There are a number of reasons to suspect that Golos’s ability to deter fraud at polling stations is 
likely to be limited. First, legal constraints limit the presence of observers to members of politi￾cal parties and the media. Consequently, Golos observers are registered not as monitors but as 
reporters from their newspaper, Civic Voice. Observers make themselves known to the chair of 
each polling station and, when not impeded, are able to observe the opening of the polling sta￾tions, the voting, and the count. However, conditions make it difficult for Golos to accurately 
supervise the use of various techniques, such as the use of absentee ballots and mobile ballot 
boxes that increase opportunities for ballot stuffing, vote repression, and other forms of fraud. 
Second, in addition to Golos, election observers from the so-called “system” political parties 
(i.e., parties allowed to register and compete in elections) have been present in most polling sta￾tions for many years, including the recent Duma and presidential elections. Given widespread 
evidence of fraud, these observers have little or no effect on the integrity of the voting. There￾fore, for Golos observers to have an effect, they would need to be doing something different 
from the party observers. Evidence of differences between Golos and non-Golos observed poll￾ing stations would thus raise additional issues of what makes Golos observers different from 
party observers. 
Third, Golos activities tend to be limited in areas where the most obvious fraud occurs. For ex￾ample, Golos observation in the North Caucasus region, where fraud, ballot-stuffing, and falsifi￾cation are rampant, is minimal. Notwithstanding these caveats, in this section we focus on the 
following hypothesis: 
GOLOS IMPACT EVALUATION—FINAL REPORT 17
Hypothesis 2: Polling stations actually observed and assessed by Golos show less evidence of electoral 
fraud than polling stations in the same regions not observed by Golos. 
In Annex C, we report upon analyses of election data that allow us to assess whether the pres￾ence or absence of Golos observers made a difference to the level of fraud at specific polling 
stations either in the Duma elections of December 2011 or the presidential elections of March 
2012. Although Golos observers were not deployed randomly in terms of the regions or even 
cities selected for observation, the actual polling stations selected within specific urban neigh￾borhoods (raiony) were selected at random. This allows us to examine whether there were dif￾ferences in the extent of the evidence of fraud in polling stations that had Golos observers 
(treatment group) and the remaining polling stations in the same neighborhoods that did not 
have Golos observers (control group). 
Despite the difficulties Golos encountered in running its election observation operations, there 
is some evidence that the very presence of Golos’s observers at polling stations had an effect in 
both reducing electoral fraud and in changing its nature, particularly in the Duma elections, 
though there is little direct evidence on the mechanisms at work. 
Most importantly, we found that unobserved polling stations had a higher average vote share for 
United Russia in the Duma elections than polling stations where election observers were pre￾sent. The differences were quite substantial and statistically significant – the mean United Russia 
vote in unobserved polling stations was 42.5 percent, while in observed polling stations in the 
same districts the mean vote share was only 36.8 percent. There were similar, but much smaller 
and statistically insignificant, differences for the presidential election, when observed polling sta￾tions had a mean vote for Vladimir Putin of 56.8 percent and unobserved polling stations 57.7 
percent. This is clear evidence of an observer effect in the Duma elections (and confirms findings 
of other researchers looking at the effect of observers within Moscow alone). 
As Annex C shows, however, it is not clear what explains the observer effect in the Duma elec￾tions. In both elections and in both kinds of polling stations, United Russia/Putin benefited to 
roughly the same extent from higher turnout – so there is little evidence there of differential 
levels of ballot stuffing. Part of the difference between polling stations, however, seems to be 
due to the fact that a higher proportion of unobserved polling stations than observed polling 
stations had a turnout of 100 percent. Although some observed polling stations did feature 100 
percent turnout, this result seems harder to “achieve” with observers around. 
Nevertheless, two potentially important sources of fraud identified by Golos and others – mo￾bile ballot boxes and absentee ballots – do not seem to be responsible for the differences be￾tween observed and unobserved polling stations, though they do seem to be used in a dubious 
manner. Much of the discussion of fraud in Russian elections has focused on mobile ballot boxes 
and absentee ballots. Mobile boxes are taken to hospitals, nursing homes and to others who 
cannot physically make it to the polling station to vote. While they ostensibly increase participa￾tion in elections, these boxes are often very difficult for observers to follow and so have fre￾quently been identified as representing a major opportunity for fraud. Similarly, absentee voting 
has been seen a major source of fraud. Absentee voting means voting at a polling station other 
than at one’s place of residence. While intended to allow voters who happen to away from 
home on the day of the election to participate in elections, absentee voting has been repeated 
identified as a source of election fraud, either through additional pressure on vote choice when 
voting in the workplace, or through so-called “carousel” voting, where organized groups cast 
absentee ballots in more than one polling station. 
GOLOS IMPACT EVALUATION—FINAL REPORT 18
Our findings suggest some very interesting points about mobile ballots and absentee voting. 
First, our findings suggest that mobile ballot boxes were indeed a major source of fraud. Every 
additional vote in a mobile ballot box translated into 0.96 additional votes for United Russia. 
However, the difference between observed and unobserved polling stations cannot be account￾ed for by mobile ballot boxes and absentee ballots. In fact, observed polling stations show evi￾dence of higher degrees of fraud from these sources than from unobserved polling stations, in￾dicating that while observers overall had an effect of reducing fraud, they also had the effect of 
changing the character of cheating, making those responsible for polling stations resort to tech￾niques that were harder to observe on the ground (but that ironically are easier to see in foren￾sics). 
The findings presented here and in Annex C need to be treated with some caution. Although it 
has a highly precise sounding name, election forensics is really a rather inexact science, and only 
large differences between levels of fraud in different sets of polling stations can be identified. 
Moreover, while randomization within neighborhoods helps us create appropriate treatment 
and control groups with respect to Golos observers, other observers were operating too, so 
the treatment/control distinction is less clean than we would like it. Nevertheless, the presence 
of other observers is mainly an issue for the presidential elections and, even then, mainly an is￾sue in Moscow and St. Petersburg. Consequently, though the results of the election forensic 
analysis cannot be seen as definitive, we believe that the results are both informative and quite 
surprising given the constraints under which Golos has been working. 
COMMUNICATION AND RELIABILITY
Does Golos do an effective job of communicating its message to its intended audience? In order 
to address this question, we looked at two different parts of this audience: a narrowly defined 
group of stakeholders and a more broadly defined group of highly educated, middle class, urban 
internet users. We analyze each of these groups separately. 
Both of these approaches involve studying perceptions, rather than attempting to evaluate Golos 
against a parallel observation mission on a random sample of polling stations. In the current po￾litical context in Russia, organizing such a parallel evaluation program was impossible. Part of the 
difficulty was due to problems in gaining accreditation for observers and obtaining access to poll￾ing stations under existing Russian legislation. Even more serious, however, was fear on the part 
of companies capable of undertaking such a project. Companies large and experienced enough 
to do this kind of work were reluctant to become involved due to fear of economic reprisals 
from the state. 
In the absence of such a rigorous test, we focused on perceptions of Golos’s reliability and 
communications. As the qualitative interviews and quantitative surveys both suggest, Golos is 
held in high regard for the quality of its network, its expertise, the accuracy of its reports and its 
ability to train observers. Nonetheless, the interviews highlight some room for further im￾provement, especially with regard to developing a public relations strategy and improving the 
quality of web and press communication. 
STAKEHOLDERS 
Qualitative interviews with members of key stakeholder groups identified by Golos—including 
government officials, opposition forces, media, business people, and public commentators—
GOLOS IMPACT EVALUATION—FINAL REPORT 19
were used to assess perceptions of Golos’s communication strategy and reliability. In order to 
ensure the validity of the responses, stakeholders were informed that we were interested in 
election observers of all kinds, but were not told of our specific interest in Golos. Thus, re￾spondents volunteered information on which organizations they worked with, whose data they 
valued, and the strengths and weaknesses of the various organizations. Our discussions focused 
on the following three hypotheses: 
Hypothesis 3: Stakeholders use and value the information gathered by Golos. 
Virtually all respondents were well aware of Golos as an organization and its primary activities. 
There was close to unanimity that Golos has been the most important independent election 
observer group—the longest standing, the best known, and most authoritative. A key element 
of the Golos model that observers praised was the strength it draws from its horizontal, net￾worked form of organization, which gives it a reach across the country that no other organiza￾tion can match. 
Many respondents referred to Golos’s primary contribution as being less in directly making elec￾tions fairer, than in performing an educative, even inspirational, function for Russian citizens. 
Golos helps to strengthen the idea of the rule of law and gives citizens somewhere to turn when 
they are under pressure from authority figures. This view was common not just to opposition 
respondents but also among influential figures in business and the press. Respondents in largely 
pro-government official media also had some praise for Golos. One believed that Golos plays a 
crucial role because they are Russian nationals who understand the situation in Russia in ways 
that foreign observers from organizations like the Organization for Security and Co-operation in 
Europe (OSCE) cannot. 
However, qualitative interviews with stakeholders also identified problems. Several well-placed 
respondents suggested that, while there was generally a lot of respect for the legal and technical 
expertise of Golos and its network, its message was often lost due to the absence of a strong 
public relations strategy and effective communications. Respondents both in the opposition 
movement and within the public affairs and political communications industry in Russia felt that 
Golos needed a stronger public relations strategy to counteract allegations that they act as for￾eign agents. It was also noted that Golos’s website, press conferences, and materials did not al￾ways present information in the most user-friendly, comprehensible way. As a result, several 
respondents felt that some of Golos’s impact was lost due to weak communications. 
While interviewees were limited in the extent to which they could comment on internal ques￾tions within Golos, Golos leadership argued that part of the reason for Golos’s weak communi￾cation strategy was financial. Given its budgetary resources, they argued, Golos had to make 
hard choices, and chose to focus on its primary task of providing legal advice, election monitor￾ing, and educational information about elections in Russia. While Golos has consistently worked 
to improve its communications over time, developing a proper public relations strategy, and hir￾ing a dedicated public relations team, has been deemed impossible given historic and current 
budgetary resources. In discussing this matter with Golos representatives, it was clear that the 
organization is open to more aggressive approaches in this regard, but that with current financial 
resources, it will be very difficult to achieve. Nevertheless, even with these limitations it was 
clear that Golos is both used and highly valued in the stakeholder community. 
Hypothesis 4: Stakeholders find the information gathered by Golos to be a reliable guide to actual un￾derlying patterns of fraud. 
GOLOS IMPACT EVALUATION—FINAL REPORT 20
As noted above, most of the stakeholders interviewed held Golos’s professionalism and the 
quality of their reports in high regard. They view the legal and election specialists who work 
with Golos as the best in the country with a high degree of integrity. It is absolutely clear that 
most of our interviewees felt that Golos provides the most rigorous, detailed, and serious eval￾uation of Russian elections of any of the groups involved, domestic or foreign. Golos was re￾peatedly lauded for its attempt to bring scientific standards of evaluation into the field of elec￾tion observation. 
There were criticisms of Golos, however, especially from pro-government respondents. Some 
questioned the accuracy of Golos’s claims, saying, for example, that the “map of violations,” for 
example, was never verified or followed up by Golos with legal claims. These criticisms of the 
map of violations were also voiced by some opposition activists.
Another set of criticisms referred to the quality of polling station monitoring activity. In some 
cases, these criticisms were ill-founded—several respondents argued that sampling a few thou￾sand polling stations rather than covering them all was an invalid approach. This opinion is, of 
course, erroneous—statistical sampling has been used in many countries to generate reliable 
estimates for vote tallies and the properties are quite well understood. Others, on the other 
hand, made more telling criticisms. One respondent argued that the quality of the election moni￾toring would be improved with a fully deployed parallel vote count methodology. The system 
introduced by Golos for observers to take photographs of polling station protocols and submit 
them by SMS was also criticized as being unreliable. More credible would be a system in which 
observers submitted copies of the original protocols accompanied by a copy of the respondent’s 
passport. This would make the whole process more responsible and credible. 
However, Golos is aware of these criticisms. Running a full parallel vote count operation had 
been one of Golos’s goals for this election, but a combination of government obstruction, nota￾bly the unpredictable and probably nonrandom exclusion of observers from polling stations, and 
cost considerations made this something that Golos could not achieve within its budget. The 
SMS system for gathering copies of protocols was also a second-best solution adopted in order 
to be able to respond quickly to demands for Golos to make assessments of the election. Un￾fortunately, because SMS copies of protocols were not able to be gathered systematically (for 
the same reasons as a parallel vote count was not conducted), the results of this process were 
useless as a basis for making overall claims about fraud and elections results. This point was not 
well understood by some journalists—and perhaps not well explained by Golos representa￾tives—leading to incorrect reporting of Golos position on the results of the presidential elec￾tion. 
Under present political conditions, it is not possible to carry out the kind of research that would 
provide independent evidence of whether Golos’s information does actually provide a reliable 
guide to underlying patterns of fraud. This would require duplicating at least some of what 
Golos does, a task no other organization is presently both willing and able to undertake. Never￾theless, it is also clear that, partisan criticism notwithstanding, Golos is viewed by stakeholders 
as providing the most reliable information available. 
Hypothesis 5: Stakeholders find Golos less credible because they are supported by foreign sources. 
Golos’s reliance on foreign funding was discussed with all stakeholder respondents. For the 
most part, respondents sympathetic to Golos did not see reliance on foreign funding as a real 
problem from a technical or reliability standpoint. However, even sympathetic respondents rec-
GOLOS IMPACT EVALUATION—FINAL REPORT 21
ognized that in the Russian political context, the association with foreign funds creates a political 
vulnerability that the Russian government is both willing and able to exploit. 
A minority of pro-government respondents were outright hostile to Golos and doubted their 
independence. “Golos are foreign agents [whose] job is subversion—all the rest is a cover,” 
complained one famous television journalist. Hence, while most do not see foreign support as a 
problem in itself, it does create a sense of political vulnerability that Golos’s opponents are keen 
to try to exploit. 
QUANTITATIVE FINDINGS 
To assess public awareness of Golos, we inserted questions into nationally representative sur￾veys conducted before and after the Duma elections as well as before and after the presidential 
elections. We focused more closely on the section of the electorate most likely to be the mar￾ket for Golos—namely, educated internet users from Russia’s biggest cities. We find that Golos 
has, unsurprisingly, a low, but growing, profile among the population as a whole. However, 
among educated, urban internet users, Golos has a higher and more accurate profile and 
knowledge of Golos is correlated with skepticism toward the quality of elections in Russia. In 
addition, we find that while people do not seem to distrust Golos, the issue of foreign funding is 
a potential source of political vulnerability within the urban middle classes. 
Hypothesis 6: Middle-class urbanites are aware of Golos and trust Golos as a major source of infor￾mation on elections in Russia. 
Nationally representative surveys suggest that only a relatively small proportion of Russians are 
aware of Golos. In our surveys conducted during the course of the Duma elections, only nine 
percent of respondents said they had heard of Golos. Moreover, only 16 percent of these re￾spondents correctly identified what Golos does from a list, with 27 percent selecting incorrect 
answers. After the election, accurate identification of Golos rose to 35 percent. From this 
group, most people (38 percent) had a middling opinion of Golos, 27 percent a high opinion, and 
19 percent a low opinion. In other words, among the public at large, Golos does not have a 
strong, positive profile. 
Beyond the raw numbers lies the question of how to interpret the meaning of the data. Is nine 
percent of the population a lot or a little? To answer this, we need to put the proportion of 
people who have heard of Golos into the context of other organizations. For many years, sur￾veys have shown the best known and most popular politically-oriented NGO in Russia to be the 
Committee of Soldiers’ Mothers. In our survey, more than half of respondents have heard of it, 
and of them, 62 percent rate the organization highly. Groups focused on more divisive topics 
than the rights of servicemen and conscripts, not surprisingly, are the subject of more divided 
views. The pro-Putin youth organization Nashi and the human rights/collective memory organi￾zation Memorial each have approximately 30 percent name recognition. Memorial is quite popu￾lar, while one in four people who have heard of Nashi view the organization negatively. Golos is 
also outdone in name recognition by Blue Buckets, a motorists’ group protesting elite privileges 
on Russian highways. Though the organization is only a few years old, almost one in five Russians 
have heard of it and, of those, 37 percent approve of its activities. Compared to these groups, 
the profile of and support for Golos among the general public is rather limited. However, given 
the rather technical nature of Golos’s activities (relative to these other groups), perhaps this is 
not surprising. 
GOLOS IMPACT EVALUATION—FINAL REPORT 22
When we look, however, at our internet polls of educated, urban Russians with relatively high 
disposable income, the pattern is rather different. Given the chance to express various degrees 
of familiarity with Golos, 17 percent before the presidential elections and 22 percent after the 
elections expressed at least partial familiarity with Golos (Table 2). Moreover, of those express￾ing some familiarity with Golos, more than half (57 percent) correctly identified Golos’s field of 
activity and only 12 percent made an incorrect identification. This confirms the findings that 
Golos enjoys a strong web presence among educated, urban Russians. Among those who had 
heard of Golos, 48 percent had heard of Golos through the internet, 16 percent of respondents 
reported visiting Golos’s website by the time the elections were over, and an impressive 41 
percent reported were familiar with Golos’s trademark Map of Violations. 
These results suggest that while the population at large does not know much about Golos, ur￾ban elites exhibit a significant degree of awareness of Golos and trust Golos as a major source 
of information on elections in Russia. 
Hypothesis 7: Middle-class urbanites who are exposed to Golos materials are likely to be more skeptical 
about Russian elections.
 
There is also significant evidence that those who are familiar with Golos are also more likely to 
think that both the Duma and the presidential elections were not free or fair. Table 3 in Annex 
A presents the results of regression analysis from our survey of educated, internet-using, mid￾dle-class, urban Russians. The table presents results separately for the Duma (models 1-3) and 
presidential elections (models 4-6). Models 1 and 4 present the simplest models that quite accu￾rately predict respondents’ belief that the elections were not free and fair. Knowledge of Golos 
is measured on five-point scale ranging from “completely unfamiliar” to “very familiar.” In the 
Duma elections, according to Model 1, a one-level increase in knowledge was associated with a 
21 percent increase in the probability that respondents thought the elections were not free and 
fair. This result is statistically significant with 99 percent confidence. For the presidential elec￾tions, the corresponding increase was 17 percent, significant at the 95 percent level. 
Given the nature of the data, we cannot show that knowledge of Golos causes skepticism about 
the elections—it is certainly possible that those who did not believe in the fairness of the elec￾tions sought out more information about Golos. However, as the different models illustrate, the 
relationship between knowledge of Golos is quite robust to the inclusion of other predictors. 
Models 1 and 4 show that even when we control for other important factors, such as sex, age, 
views on the general direction of the country, levels of education, frequency of watching state 
television, and whether the respondent voted for Putin or not, there is still a robust relationship 
between knowledge of Golos and a critical attitude to the elections. Models 2 and 5 show that 
this holds when we include different kinds of internet and social media use, and models 3 and 6 
hold when we control for more presidential vote choices. In other words, knowledge of Golos, 
while somewhat limited, is generally associated with a critical perspective on the freedom and 
fairness of elections in the recent election cycle. Hence, hypothesis 7, that middle class urbanites 
who are exposed to Golos materials are likely to be more skeptical about Russian elections 
cannot be rejected.
Hypothesis 8: Middle-class urbanites who are exposed to Golos materials and also to government coun￾ter-claims that Golos is supported by foreigners are less likely to be influenced by Golos materials. 
In the online surveys, we also embedded survey experiments to probe the extent to which re￾spondents trusted election observers, of various kinds, and to assess how vulnerable Golos’s 
GOLOS IMPACT EVALUATION—FINAL REPORT 23
credibility is to attacks from the media alleging foreign control over the organization. The results 
demonstrate a considerable degree of support for and trust in domestic observers. However, 
the data suggest that, though attitudes to foreign involvement are complicated, there is some 
skepticism about foreign election observers and considerable opposition to foreign funding of 
Russian observers, even among highly educated, upper income, internet-using urbanites. Moreo￾ver, state-sponsored efforts to shape such attitudes through media campaigns do seem to have 
some effect. Overall, the results show that there is real support for both domestic and foreign 
election monitoring, though there does seem to be evidence that it is important that observers 
are seen as an organically grown from domestic sources rather than imported from abroad. The 
following paragraphs summarize the main results of the experiments. (Details and full results of 
the experiments are presented in Annex D.) 
First, it is clear that, at least by the time of the presidential election in March 2012, educated, 
urban Russians had strongly positive attitudes towards election monitors. There is a relatively 
high degree of trust in election monitors. Both before and after the presidential elections, ap￾proximately half of the respondents (51 percent before and 48 percent after) said they trusted 
election observer reports either completely or somewhat. On the other hand, only 11 percent 
in each wave expressed suspicion . More than four in 10 respondents remain to be convinced 
either way. 
Positive attitudes extended to respondents wanting observers to have largely unfettered access 
to polling stations. In both waves, more than 80 percent of respondents supported either free 
or only lightly regulated access to polling stations. There is also some evidence that attitudes to 
election observers grew even more favorable between the two waves of the survey. 
We find that this support also extends to observers from the OSCE, though foreign observers 
as a whole are quite a heterogeneous group and are not all thought of as being analogous. Sup￾port for OSCE observers specifically is about the same as for observers in general, while “for￾eign observers,” who might include observers invited from neighbors like Belarus or states such 
as China and Kazakhstan, are viewed less positively. 
Respondents also believe that monitors have a direct effect in improving the fairness of elec￾tions. In the first wave, 60 percent of respondents agreed with this position, while only 12 per￾cent disagreed. In the second wave, 56 percent agreed and nine percent disagreed. 
However, urban, educated Russians are quite skeptical of foreign funding of election observers, 
though attitudes may be softening over time. In the first wave, 44 percent of respondents 
thought that foreign financing of domestic election monitoring organizations should be banned 
completely and 26 percent thought that foreign financing should be tightly regulated. Only 22 
percent of respondents felt that such assistance should be able to be given either freely or only 
subject to light regulation. There is evidence, however, that attitudes softened somewhat be￾tween the two waves of the survey. After the presidential elections, the proportion of respond￾ents thinking that foreign support should be banned fell to 36 percent, and 29 percent felt that 
such assistance could be freely given or only lightly regulated. 
Within the various observer groups, the effects of Golos as a “brand” are difficult to identify. 
Responses did not change significantly when Golos was mentioned specifically, in either a posi￾tive or a negative direction. 
Finally, it is important to bear in mind that Golos’s activities have been taking place in an increas-
GOLOS IMPACT EVALUATION—FINAL REPORT 24
ingly hostile political context as state-sponsored and allied media have stepped up attacks on 
Golos. These attacks have primarily focused around accusations of political bias and being agents 
of a foreign power, rather than charges of incompetence. Charges that Golos’s evaluations are 
preconceived ideas ordered up by foreigners, particularly the United States, are common. There 
is some evidence that the Russian government’s attacks on Golos can have an effect on attitudes 
towards observers, especially attacks on television. Respondents exposed to anti-Golos propa￾ganda videos exhibit significantly more negative attitudes than those not exposed. 
Probing attitudes to observers further, we asked respondents in the second wave who they 
thought had funded the unprecedentedly large number of observers who turned out to super￾vise the March presidential elections. This question was asked of respondents, some of whom 
were also shown video footage from the Russian NTV channel making allegations against elec￾tion observers. Again, differences between the two groups were marked. In the control group, 
half of all respondents thought that private Russian citizens paid most of the expenses of ob￾servers, while only 35 percent of those who had watched the NTV footage thought private Rus￾sian citizens were the main funders. 
The other big difference was in the proportion who felt the protesters were paid for by ill￾intentioned foreigners. While in both groups the proportion who felt the observers were paid 
for by foreign governments who want to support free and fair elections was essentially the same 
(10 and 11 percent respectively), in the NTV group 23 percent believed that the observers were 
mostly funded by foreign governments who wanted to destabilize Russia—this was nearly dou￾ble the 13 percent in the control group who thought the same. 
Overall, the results provide qualified support for the hypothesis that urban elites who are ex￾posed to Golos materials and also to government counter-claims that Golos is supported by 
foreigners are less likely to be influenced by Golos materials. While respondents have generally 
positive attitudes to election observers, those exposed to the harshest anti-Golos propaganda 
are substantially more suspicious of election observers and their claims. 
GOLOS IMPACT EVALUATION—FINAL REPORT 25
CONCLUSION 
This report presents an evaluation of Golos’s effectiveness in its efforts to contribute to open￾ness and fairness in Russian elections. Given the complexities of Golos’s task and a difficult polit￾ical context, the evaluation used multiple methods to attempt to assess the organization’s im￾pact. 
Although Golos often works in difficult circumstances and under substantial pressure from Rus￾sian authorities, qualitative and quantitative data indicate that Golos is an influential force. Quali￾tative interviews with key stakeholders testify to Golos’s importance in pressing for respect for 
the law and for improvements in election-related legislation. In addition, it is clear that Golos is 
the most experienced, most qualified, and most respected election monitoring organization in 
Russia, a fact emphasized by interviewees from across the political spectrum. With its wide net￾work and extensive experience, Golos also played a key institutional role in facilitating the mas￾sive increase in popular participation in election monitoring that characterized the March 2012 
presidential election. 
Quantitative data demonstrate that while Golos’s profile among the Russian public as a whole is 
quite limited, the organization is trusted and influential among the urban middle classes. This fact 
is even more notable given the efforts of the government to characterize Golos as foreign 
stooges. Moreover, analysis of internet data showed that Golos and its activists played an im￾portant role in the growth of election-related civic activism that took place following the fraudu￾lent December 2011 Duma elections. 
We have also identified areas where Golos’s work could be improved and strengthened. The 
system for monitoring and observing elections is evolving and it would be desirable, given ap￾propriate resources, to seek to develop a truly representative parallel vote tabulation in future 
federal elections. 
Most significantly, much could be done to improve Golos’s public relations and communications 
strategy. While this report did not look at the degree to which Golos’s communications have 
improved over time, our evaluation suggests that there is significant room for improvement. 
While this would require significant new financial resources, improved communications would 
increase Golos’s ability to reach and influence the broader Russian public. Clearly, this will not 
be easy. Golos exists in a very difficult media environment in which television is heavily pro￾government and has been actively used to paint Golos as foreign agents. Our results show that 
foreign financial support, while absolutely essential to Golos’s previous activities, remains a po￾tential source of political vulnerability. Given the highly uncertain prospects of shifting to purely 
domestic support, developing an explicit public relations strategy would help Golos to counter 
the vigorous efforts of the Russian government to paint Golos as unreliable foreign agents. In￾deed, the increasingly hostile political environment in which Golos operates makes it more, not 
less, important that Golos have the best communications operation as possible. 
 GOLOS IMPACT EVALUATION—FINAL REPORT A-1 
ANNEX A: KEY TABLES AND 
GRAPHS 
Figure 1: Golos and Other Election Organizations: Hyperlink Network Map
 GOLOS IMPACT EVALUATION—FINAL REPORT A-2 
Figure 2: Golos (green), Grazhdanin Nabliudatel’ (blue) and Liga Izbiratelei (red) on 
Facebook, March 2012 
 GOLOS IMPACT EVALUATION—FINAL REPORT A-3 
Table 1: Golos Thematic Clusters and Online Media Sources 
Media / Cluster
Elect￾ions 
Moni￾toring 
and 
Viola￾tions 
Reports
Elect￾oral 
Legisla￾tion
Pres￾sure on 
Golos 
observ￾ers in 
regions
ODIHR
Ob￾servers
Fine for 
illegal 
agita￾tion
NTV 
and 
"Surkov
's Prop￾aganda"
Shibano
bano￾va's 
note￾book
Roskom
kom￾nadzor
warning 
for 
gazeta.r
u
DDOS 
attacks 
on Dec 
4th
TOTAL
kommersant.ru 51 19 -- 9 2 -- -- -- -- 81 
golos-org LJ 29 24 7 3 -- -- -- -- -- 63
gazeta.ru 22 2 -- 1 9 3 6 1 2 46
echo.msk.ru/blog 13 18 1 -- 1 -- -- -- 1 34 
newsru.com 11 -- -- -- 3 3 1 2 2 22
svpressa.ru 9 2 -- -- 1 -- 2 -- 3 17
lenta.ru/ 2 -- -- -- 3 1 3 2 4 15
www.kavkaz￾uzel.ru 12 1 -- -- 1 -- -- -- 1 15 
aif.ru 11 3 -- -- -- -- -- -- -- 14
anticompromat 
LJ 5 2 -- -- 1 3 2 -- 1 14
vz.ru 3 -- -- -- 4 6 1 -- -- 14 
bfm.ru 2 -- -- -- 4 1 -- 1 4 12
www.news2.ru 3 2 -- -- 3 1 1 1 1 12
fontanka.ru 1 1 -- -- 3 2 2 1 1 11
pravda.ru 6 -- -- -- 1 2 -- -- -- 9 
vedomosti.ru 8 -- -- -- -- -- -- -- -- 8
rian.ru 2 1 1 -- -- -- 1 -- 2 7
inosmi.ru 4 -- -- -- 1 -- -- 1 -- 6 
regnum.ru 4 1 -- 1 -- -- -- -- -- 6
expert.ru 3 -- -- -- 2 -- -- -- -- 5
rg.ru 4 1 -- -- -- -- -- -- -- 5
pro-kurator.ru 2 -- -- 1 -- 1 1 -- -- 5 
Total 207 77 9 15 39 23 20 9 22 421
 GOLOS IMPACT EVALUATION—FINAL REPORT A-4 
Table 2: Familiarity with Golos before and after the presidential election (internet 
survey) 
Familiarity with Golos Before the Elections After the Elections
Very Familiar 36
(3%)
52
(4%)
Fairly Familiar 64
(5%)
116
(9%)
Partially Familiar 109
(9%)
101
(8%)
Fairly Unfamiliar 238
(19%)
232
(19%)
Completely Unfamiliar 698
(57%)
668
(55%)
Don’t Know 68
(6%)
50
(4%)
Total Observations (N) 1213 1219
 GOLOS IMPACT EVALUATION—FINAL REPORT A-5 
Table 3: Regression Analysis of Knowledge of Golos and Perceptions of Fraud 
(1) (2) (3) (4) (5) (6)
Duma Duma Duma President President President
Gender 0.333* 0.345** 0.380** 0.312* 0.298* 0.368**
(0.173) (0.171) (0.171) (0.173) (0.173) (0.173)
Age -0.0200*** -0.0184** -0.0200*** -0.0210*** -0.0164** -0.0206***
(0.00707) (0.00754) (0.00709) (0.00717) (0.00771) (0.00722)
Direction -1.845*** -1.809*** -1.710*** -1.702*** -1.687*** -1.564***
(0.172) (0.171) (0.176) (0.181) (0.181) (0.185)
Education 0.466** 0.451** 0.441** -0.0299 -0.0240 -0.0291
(0.208) (0.207) (0.208) (0.204) (0.204) (0.205)
State TV -0.259*** -0.251*** -0.243*** -0.225*** -0.238*** -0.219***
(0.0818) (0.0822) (0.0805) (0.0751) (0.0766) (0.0744)
Internet 0.0967 0.223
(0.124) (0.138)
V Kontakte 0.0605 0.403*
(0.206) (0.208)
Odnoclass. -0.195 -0.168
(0.184) (0.185)
Facebook -0.0344 0.126
(0.176) (0.176)
Know Golos 0.190*** 0.196*** 0.179** 0.157** 0.151** 0.139**
(0.0721) (0.0691) (0.0697) (0.0714) (0.0690) (0.0690)
Putin Voter -1.331*** -1.372*** -1.397*** -2.607*** -2.636*** -2.741***
(0.218) (0.218) (0.237) (0.349) (0.348) (0.358)
Fin. Status -0.0907 0.0176
(0.119) (0.122)
Follow pol. 0.0967 0.0331
(0.132) (0.132)
Ziuganov 0.345 0.266
(0.290) (0.263)
Prokhorov 0.249 0.104
(0.212) (0.199)
Splitter -0.417* -0.653***
(0.216) (0.214)
Constant 0.106 -0.395 -0.137 1.563 0.500 1.639
(1.149) (1.138) (1.038) (1.127) (1.130) (1.028)
Obs. 862 866 866 895 899 899
Log
likelihood -445.8 -447.5 -445.4 -434.6 -432.0 -430.7
Chi2 300.7 302.8 307.1 335.5 346.5 348.9
GOLOS IMPACT EVALUATION—FINAL REPORT B-1 
ANNEX B: GOLOS, ELECTION 
MONITORING AND THE RUS￾SIAN INTERNET 
BACKGROUND 
Working with the Center for the Study of New Media & Society at the New Economic School, 
Democracy International undertook a research project to evaluate the role played by Golos and 
the independent election monitoring movement before, during, and after the December 4, 2011 
Russian parliamentary elections, the March 4, 2012 Russian presidential elections, and the ensu￾ing protest mobilization. Through a combination of computer-assisted and human analysis, the 
project directly addressed the following questions: 
1. How often and in what context Golos and election monitoring were discussed on the 
internet during the relevant period; 
2. The content of relevant communication and relationship with events and third-party 
sources of information (videos and other media, in-links, out-links, etc.); 
3. What audiences Golos and other independent election monitors reach most effectively, 
and what relevant audiences they fail to reach; and 
4. The network characteristics of Golos, its allies, and its audiences, including social path￾ways of meme and information distribution, bottlenecks, gatekeepers, and the attendant 
effects on communication and mobilization. 
With these questions in mind, DI conducted an assessment of the mechanisms that could serve 
to underpin a relationship between Golos’s activities (and those of its allies) and public percep￾tions of the election results, as well as the subsequent emergence of independent election moni￾toring at the core of the opposition movement. 
RESEARCH DESIGN 
The project sought to test three empirical hypotheses regarding the role and impact of Golos 
during the 2011–2012 election period. These hypotheses identify three discrete ways in which 
Golos may have been argued to have had an impact: 
1. First, through its core activities, Golos had a direct impact, including on Russian citizens’ 
perceptions of whether the elections were free and fair; 
2. Second, other election-oriented groups and initiatives relied to a critical extent on the 
informational and communications infrastructure developed by Golos over the course 
of implementation of its core activities; and 
3. Third, Golos’s networks of activists and other interested parties played an important in￾formal role in growing and coordinating election-oriented activism. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-2 
In each case, the study will set out to support or reject these hypotheses by applying a combina￾tion of methods to a range of empirically verifiable predictions (see Table 1 below).
Table 1: Hypotheses, Predictions, and Sources 
Empirical Hypothesis Verifiable Prediction Relevant Data Sources
Through its core activities, Golos had 
a direct impact, including on Russian 
citizens’ perceptions of whether the 
elections were free and fair.
1) Online discussions of elections 
actively reference Golos and/or 
Golos-backed projects, Golos 
leaders, including monitoring, 
etc.
2) Golos materials are among the 
most frequently distributed with 
regard to elections, particularly 
with regard to perceptions of 
freedom/fairness of elections
1) Meme and theme detection and 
analysis2 of OSM3
2) Link analysis4
Other election-oriented groups and 
initiatives relied to a critical extent 
on the informational and communica￾tions infrastructure developed by 
Golos over the course of implemen￾tation of its core activities.
1) Golos handbooks, methodolo￾gies, etc., are actively used by 
other election-oriented activists
2) Formal and informal communi￾cation spaces generated by 
Golos activities are central in 
dispersion of election-oriented 
activism
1) Link analysis
2) Social-network analysis5 of 
OSM
Golos’s networks of activists and 
other interested parties played an 
important informal role in growing 
and coordinating election-oriented 
activism.
1) Golos leaders, activists, and pre￾existing partners are central in 
broader election-oriented activ￾ism networks
1) Social-network analysis of OSM
Data covering the period from November 1, 2011 through March 31, 2012 were drawn from a 
full-text corpus of more than 2,000 Russian-language blogs (primarily from the Live Journal plat￾form) and approximately 50 online news media sites, as well as social-networking systems, in￾cluding Facebook and Twitter. Methodologies employed included meme detection and tracking, 
network and cluster analysis, and other applied qualitative sociological and political science 
methods. 
HYPOTHESIS 1: IMPACT ON PERCEPTIONS OF ELECTION FAIRNESS 
Did Golos impact Russian citizens’ perceptions of whether the elections were free and fair? A 
direct answer to that question would require complex survey research that is beyond the scope 
of this report, which focuses exclusively on online data. Nevertheless, our data allows us to in￾vestigate the prominence and valence of Golos’s online presence and activities within an audi￾ence that can broadly be assumed to perceive the parliamentary and presidential elections as 
less than free and fair. 
 
2 Identifying textual topic and object ‘markers’, analysis of their distribution, including cluster analysis
3 Online Social Media: blogs (Live Journal), Twitter, Facebook, VKontakte 
4 Analysis of the use of hyperlinks in online communication 
5 Analysis of the connections between social actors (individuals or institutions), organized into networks, including 
analysis of the underlying structures of those networks, and how the networks grow and shift structurally over time 
GOLOS IMPACT EVALUATION—FINAL REPORT B-3 
Existing survey data suggest that fewer than half of Russians saw the elections as a real contest, 
rather than a staged affair. According to a poll conducted by the Levada Center, 37 percent of 
respondents believed that the Central Election Commission would engage in at least some 
cheating, compared to 34 percent who believed it would not.6 Whatever role Golos—and other 
election monitoring initiatives that emerged during and after the Duma election, including Liga 
Izbirateley and Grazhdanin Nablyudatel’—may have played in creating those perceptions, it was 
only one of a multitude of factors, and the perceptions themselves far exceeded any awareness 
of Golos or its peers. In fact, three-quarters of respondents reported no awareness of Golos or 
Grazhdanin Nablyudatel’, while only 16 and 20 percent of respondents were aware of Golos’s 
and Grazhdanin Nablyudatel’s activities, respectively.7
Significant numbers of Russians were also engaged in online discussions and mobilization related 
to the elections, although the actual size of this community should not be overstated. A census 
of online social media communities in Facebook and its Russian equivalent, VKontakte, found 
44,807 Facebook users and 20,044 VKontakte users actively engaged in six ‘community pages;’ a 
larger number—164,092 on Facebook and 107,883 on VKontakte—identified themselves as par￾ticipants in 11 major protest events.8 Even if we take a “best case” estimate and assume that all 
of these individuals were unique users and there was no overlap, which is almost certainly not 
the case, this mobilized online community would amount to no more than 2.5 percent of the 
population of Moscow and slightly more than 3 percent of Facebook and VKontakte users in 
Russia. Blogs, particularly the popular Russian blogging platform Live Journal, are also an im￾portant locus of debate and discussion, and available data point to a significant spike in discus￾sion of Golos and its peers during the election period. However, even at its peak, no more than 
0.05 percent of Russian blog posts published in any given week dealt with the election monitor￾ing initiatives, and no more than 0.02 percent of posts mentioned Golos specifically (see Figure 
1). Thus, if we are looking for evidence of an effect that Golos may nor may not have had on 
Russian online public opinion, we are dealing with a remarkably small subset of the population, 
whose identity and, indeed, demographic profile are poorly known. Thus, even well-structured 
polling research would struggle to reach robust conclusions. 
The two peaks in blog discussion of Golos correspond to peaks in broader online media cover￾age of Golos (see Figure 2) and represent two discrete and distinct story lines. In the first in￾stance, in November and December 2011, coverage and discussion had less to do with the elec￾tions and more to do with Golos itself, including a number of high-profile encounters between 
Golos and various representatives (or presumed representatives) of the state: the release of ‘A 
Voice from Nowhere’,
9 a libelous ‘documentary’ on Golos’s activities aired on NTV; the deten￾tion of Golos director Lilya Shibanova at Moscow’s Sheremet’evo airport and subsequent con￾fiscation of her laptop computer; and a series of DDoS attacks on Golos resources. The second 
peak, in February–March 2012, is election-focused, driven by plans for monitoring the March 4 
presidential ballot and subsequent reaction. It is noteworthy that the attention pattern for Golos 
 
6 “Vybory 2012 v otsenkakh rossiian in perspektivy sleduiushchikh 12 let.” Levada Center, 6 March 2012. 
http://www.levada.ru/06-03-2012/vybory-2012-v-otsenkakh-rossiyan-i-perspektivy-sleduyushchikh-12-let; accessed 30 
July 2012. 
7 Ibid. 
8 Panchenko, Egor. 2012. “Mitingi ‘Za chestnye vybory’: Protestnaia aktivnost’ v sotsial’nykh setiakh” in Digital Icons: 
Studies in Russian, Eurasian and Central European New Media, 7: 149-154.
9 Golos neotkuda 
GOLOS IMPACT EVALUATION—FINAL REPORT B-4 
(and, for that matter, Grazhdanin Nablyudatel’) differs markedly from that of Liga Izbirateley; the 
latter emerges much more in relation to the postelection protests, and attention to Liga Izbi￾rateley peaks after the March 4 ballot, while attention to Golos and Grazhdanin Nablyudatel’ 
peaks before the vote. Discussed in more detail later in the report, this pattern reflects different 
roles taken on by the respective movement organizations. 
Figure 1: Russian Blog Mentions of Golos (blue), Grazhdanin Nablyudatel’ (yellow) 
and Liga Izbirateley (green) 
GOLOS IMPACT EVALUATION—FINAL REPORT B-5 
Figure 2: Timeline of Golos Online Mentions 
Somewhat greater leverage on the potential impact of Golos on online perceptions is gained by 
switching from a temporal to a thematic perspective. To this end, we compiled a one-year da￾taset, based on the Media Cloud project of the Berkman Center at Harvard University.10 This 
data contains daily crawls from top Russia media and TV sites, official web portals of govern￾ment bodies, the 1,000 most popular Russian blogs, and a random sample of 1,000 other blogs. 
After cleaning the data from noise and non-Russian sources, we obtained 40 million news stories 
and blog posts covering period from 1st Jan through 4th Dec 2011, with total size of 200 giga￾bytes. 
We then developed a clustering algorithm, which processes tens of thousands of potentially rel￾evant news stories and groups them into a set of topically centered clusters related to Golos, 
which were manually filtered by a human. The results of this semi-automated analysis of promi￾nent events and corresponding media coverage are presented in Table 2, in the form of a distri￾bution of stories from media sources and (a few) blogs across the nine robust thematic clusters 
derived from the data. Only information sources that have at least five stories mentioning Golos 
are included in the table. Key sources – which consistently respond to Golos-related events and 
appear in at least five thematic clusters – are italicized. An alternative view of these clusters, 
viewed as a network map, is presented in Figure 3. 
 
10 “Media Cloud: A tool and automated methods for identifying agendas in blogs and mainstream media,” Berkman 
Center for Internet and Society, Harvard University Law School. http://www.mediacloud.org; accessed 20 July 2012. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-6 
Table 2: Golos Thematic Clusters and Online Media Sources 
Media / Cluster
Elections 
Monitoring 
and Viola￾tions Re￾ports
Elect￾oral 
Legisla￾tion
Pressure 
on Golos 
observers
in regions
ODIH
R
ob￾servers
Fine for 
illegal 
agitation
NTV and 
"Surkov's 
Propa￾ganda"
Shibano
bano￾va's 
note￾book
Roskom￾nadzor
warning 
for 
gazeta.ru
DDOS 
attacks 
on Dec 
4th
TO￾TAL
kommersant.ru 51 19 -- 9 2 -- -- -- -- 81
golos-org LJ 29 24 7 3 -- -- -- -- -- 63
gazeta.ru 22 2 -- 1 9 3 6 1 2 46
echo.msk.ru/blog 13 18 1 -- 1 -- -- -- 1 34
newsru.com 11 -- -- -- 3 3 1 2 2 22
svpressa.ru 9 2 -- -- 1 -- 2 -- 3 17
lenta.ru/ 2 -- -- -- 3 1 3 2 4 15
www.kavkaz-uzel.ru 12 1 -- -- 1 -- -- -- 1 15
aif.ru 11 3 -- -- -- -- -- -- -- 14
anticompromat LJ 5 2 -- -- 1 3 2 -- 1 14
vz.ru 3 -- -- -- 4 6 1 -- -- 14
bfm.ru 2 -- -- -- 4 1 -- 1 4 12
www.news2.ru 3 2 -- -- 3 1 1 1 1 12
fontanka.ru 1 1 -- -- 3 2 2 1 1 11
pravda.ru 6 -- -- -- 1 2 -- -- -- 9
vedomosti.ru 8 -- -- -- -- -- -- -- -- 8
rian.ru 2 1 1 -- -- -- 1 -- 2 7
inosmi.ru 4 -- -- -- 1 -- -- 1 -- 6
regnum.ru 4 1 -- 1 -- -- -- -- -- 6 
expert.ru 3 -- -- -- 2 -- -- -- -- 5
rg.ru 4 1 -- -- -- -- -- -- -- 5
pro-kurator.ru 2 -- -- 1 -- 1 1 -- -- 5
Total 207 77 9 15 39 23 20 9 22 421
GOLOS IMPACT EVALUATION—FINAL REPORT B-7 
Figure 3: Mapping Golos Thematic Clusters and Media Sources 
Two observations emerge from a review of these clusters. First, while the ‘stories about Golos,’
such as the NTV broadcast, Shibanova’s laptop and the DDoS attacks, are concentrated in time 
and thus produce significant spikes in attention, they pale in comparison – both in terms of scale 
and range of sources – to the ‘stories by Golos’, particularly the long-running work that Golos 
does on election monitoring, violation reporting and electoral legislation. Second, while relative￾ly few Russians may be aware of Golos, the organization receives a great deal of attention from 
major online news outlets, including several – such as Ekho Moskvy, newsru.com, gazeta.ru and 
lenta.ru – that played a prominent role as meme distributors during the December and March 
protests.11
 
11 Greene, Samuel A. 2012. “Twitter and the Russian Street: Memes, Networks and Mobilization.” CNMS Working 
Paper 2012/1, May 2012. Moscow: Center for the Study of New Media & Society. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-8 
What we have, then, are the makings of a causal mechanism that can lend partial support to the 
hypothesis that Golos has, indeed, had an impact on online public opinion vis-à-vis the perceived 
legitimacy of Russia’s recent parliamentary and presidential elections. While the causal link be￾tween awareness of electoral fraud, including awareness gained online, has yet to be demon￾strated, and thus it would be premature to argue that Golos had a direct role in fostering the 
shift from small-scale to large-scale election-related protest activity, the following seem clear: 
 Golos – through its own activities and those of others – was an important online pres￾ence; 
 Messages both “about” and “by” Golos were regularly distributed through channels that 
were critically important to the election-related mobilization; and 
 Golos’s online presence is underpinned by the informational and reportorial aspects of 
its mission, as distinct from the more directly mobilizational approaches of its peers. 
HYPOTHESIS 2: CENTRALITY IN THE BROADER ELECTION MONITOR￾ING MOVEMENT 
Does online evidence speak to whether or not other election-oriented groups and initiatives 
relied to a critical extent on the informational and communications infrastructure developed by 
Golos over the course of implementation of its core activities? There is certainly reason to be￾lieve that this sort of reliance existed. First and foremost, Golos has a long history of involve￾ment in Russian election monitoring, while its two closest peers – Liga Izbiratelei and Grazhda￾nin Nabliudatel’ – emerged only in this electoral season. Thus, Golos has at its disposal both 
greater experience and a greater reserve of informational infrastructure with which to support 
its activities. Second, Golos did generate and manage online resources of particular value and 
interest to the election monitoring community, and, indeed, of broader interest; these include 
the ‘violations map’ produced together with Gazeta.ru, an SMS portal for submitting information 
on potential violations, a map of election results, and others (see Figure 4). As with the previous 
hypothesis, any conclusion based on online evidence should be triangulated with offline research, 
including interviews with decision-makers and participants in the various movement organiza￾tions. However, online data can point us in important directions. 
To address this hypothesis directly, we compiled a database of hyperlinks into and out of (‘in￾links’ and ‘outlinks’) websites – including proprietary sites and social media pages – involving 
Golos and other election monitoring organizations. The software involved begins with a seeded 
core of websites – in this case, the proprietary home pages and related social media pages of 
Golos, Grazhdanin Nablyudatel’ and Liga Izbiratelei – and then ‘crawls’ three levels ‘deep’ from 
each hyperlink found on the page.12 The resulting dataset includes 5,514 links, a network map of 
which is presented in Figure 5. 
Because of the nature of the data, the fact that the links included are static, and many of them 
may be several years old or more, the dominance of Golos in this network map – on the blog￾ging platform Live Journal first and foremost, but also on its homepage and on Twitter – is to 
 
12 Ackland, Robert, Rachel Gibson et al. 2005. “Virtual Observatory for the Study of Online Networks (VOSON).” 
Australian National University. http://voson.anu.edu.au; accessed 30 July 2012. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-9 
some extent an artifact of Golos’s longer history compared to its peers. Nonetheless, because 
the internet itself is cumulative, consisting of pages and resources that, unless actively removed, 
do not disappear, this is more or less an accurate representation of the election monitoring￾related landscape that would face a user of the Russian-language World Wide Web. 
Figure 4: Popularity of Golos Projects in Internet Searches 
GOLOS IMPACT EVALUATION—FINAL REPORT B-10
Figure 5: Golos et. al, Hyperlink Network Map 
A census of these links, however, when reduced down to those nodes (i.e., individual Web pag￾es) that have at least two inlinks or outlinks, reveals a somewhat different picture. Thus, Table 3 
presents links to Golos resources of various kinds from throughout the election monitoring 
community’s Web presence, including the networks of Golos itself, its peers, the sites of other 
activists, and relevant online social media pages. The result, first of all, is a drastic reduction in 
the number of linked pages, from more than 5,500 down to 84, reflecting a network that is 
broad but not dense. Moreover, just over half of all of the links come from within Golos’s own 
network. Within this relatively small universe of links, there are a reasonable number of links 
coming in from other movement organizations – particularly Liga Izbirateley – and an even larger 
number coming from OSM (predominantly from blogs). However, it is noteworthy that the vast 
majority of these links are to root pages – i.e., to Golos’s home page, Facebook page, Twitter 
page, SMS portal and so on – rather than to reports, handbooks and other, more narrowly fo￾cused resources. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-11
Table 3: Inlinks to Golos Resources from Elsewhere in the Election Monitoring 
Community 
Link Source
URL Golos 
Net￾work
Liga 
Izb. 
Net￾work
Gra￾zhd. 
Nab. 
Net￾work
Other 
Activist 
Net￾works
Onlin
e 
So￾cial 
Me￾dia
Oth￾er
To￾tal
Total 
non￾Golo
s 
In￾links
http://www.golos.org 8 2 0 3 9 7 29 21
http://www.facebook.com/golos.o
rg
5 3 1 0 0 1 10 5
http://twitter.com/golos_org 6 2 0 0 1 0 9 3
http://sms.golos.org 1 1 0 0 5 0 7 6
http://golos-org.livejournal.com 1 0 0 0 2 1 4 3
http://youtube.com/user/videogol
os
2 1 1 0 0 0 4 2
http://obuchenie.golos.org/nabluda
tel
2 0 0 0 2 0 4 2
http://kodeks.golos.org 4 0 0 0 0 0 4 0
http://stat.golos.org 1 0 0 0 1 1 3 2
http://www.flickr.com/photos/golo
sorg
2 0 0 0 0 0 2 0
http://videogolos.rutube.ru 2 0 0 0 0 0 2 0
http://www.kartanarusheniy.ru 2 0 0 0 0 0 2 0
http://golos_org.livejournal.com 2 0 0 0 0 0 2 0
http://files.golos.org/IMG/pdf/dokl
ad_11-03-13_11.pdf
2 0 0 0 0 0 2 0
TOTAL 40 9 2 3 20 10 84 44
This available evidence, we argue, reflects the generic importance of Golos to the broader ‘net￾work’ of election monitoring initiatives: the volume of material that Golos presents on the Web 
is such that its peers may defer to Golos rather than try to compete. However, the evidence 
neither supports nor rejects the hypothesis that specific Golos materials, including reports, 
methodologies and handbooks, are of critical importance to the other election monitoring 
movement organizations. If evidence to or against this effect is to be uncovered, it will have to 
be through offline research. 
HYPOTHESIS 3: BROADENING THE MOVEMENT 
Does Golos as a movement organization bring something to the broader movement network 
that emerged in December 2011, something that would serve, informally, to strengthen and bal￾ance that network and thus the movement as a whole? There are, we believe, three ways in 
which this might be expected to be the case, each of which we will explore below. First, Golos 
might bring to the movement its established network of professionals and activists, whose expe￾rience, expertise and connections would make them natural leaders and coordinators, even as 
organizational dividing lines blurred in the heat of contestation. Second, Golos may have brought 
along its informal network of ‘friends’, its audience of habitual or committed readers and follow￾ers, who might have then been central in linking the amalgamated movement network together. 
And lastly, Golos’s culture and habits of communication may generally have served as a unique 
and important component of the mobilization, perhaps contributing to an efficient division of 
labor among the three main movement organizations. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-12
All of the data used to test this hypothesis come from an analysis of interactions by users of the 
Facebook communities of Golos, Grazhdanin Nabliudatel’ and Liga Izbiratelei—specifically, peo￾ple posting and commenting—during a five-day period leading up to the 4 March 2012 presiden￾tial election. The dataset, which includes only publicly visible interactions and was collected using 
open-source software, comprises 13,238 interactions by 780 users.13 A sample drawn from ‘lik￾ers’ rather than ‘commenters’ would have been more commensurate in size with the numbers 
found by Egor Panchenko’s online mobilization census quoted earlier in this report but would 
have reflected a lower level of engagement. A map of the resulting amalgamated network is pre￾sented in Figure 6. 
Figure 6: Golos (green), Grazhdanin Nabliudatel’ (blue) and Liga Izbiratelei (red) on 
Facebook, March 2012 
We test the first sub-hypothesis—namely, that Golos staff and/or long-term activists were cen￾tral to the movement as a whole—by searching for and identifying ‘brokers’ within the amalga-
 
13 Smith, M., N. Milic-Frayling et al. 2010. “NodeXL: a free and open network overview, discovery and exploration 
add-in for Excel 2007/2010,” Social Media Research Foundation. http://www.smrfoundation.org; accessed 30 July 
2012; Ceni, A., B. Hogan and M. Smith. 2011. “Facebook Spigot for NodeXL,” Social Media Research Foundation. 
http://www.smrfoundation.org; accessed 30 July 2012; Bastian, M., S. Heymann and M. Jacomy. 2009. “Gephi: an open 
source software for exploring and manipulating networks.” http://gephi.org; accessed 30 July 2012; Batagelj, V. and A. 
Mrvar. 2011. “Pajek – Program for Large Network Analysis,” http://pajek.imfm.si; accessed 30 July 2012. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-13
mated network. Table 4 presents those individuals, nine in total, who have at least 10 interac￾tions in at least two of the sub-communities within the amalgamated network. They are charac￾terized in the table by their rank on various measures of network centrality: 
 Betweenness centrality, reflecting the degree to which the actor is a unique link be￾tween disparate large network components; 
 Closeness centrality, reflecting the degree to which the actor is close to large numbers 
of other actors; 
 Eigenvector centrality, reflecting the degree to which the actor is close to other actors 
with high degrees of centrality; and 
 Degree, simply the number of links the actor has to other actors. 
(Ranks within the top 20 are reported in the table. Rank was chosen as an indicator, because 
measures of centrality are relative, rather than absolute, and thus one actor’s centrality is mean￾ingful only in comparison to another’s.)
In addition, actors in the table are characterized by their degree within the composite network 
as a whole and within each sub-network, and by ‘degree overlap’, i.e., the number of connec￾tions that are simultaneously in more than one network. In many ways, this last indicator is the 
crucial one, because it reflects direct—rather than mediated—cross-network interaction. 
GOLOS IMPACT EVALUATION—FINAL REPORT B-14
Table 4: Election Monitoring Community ‘Brokers’ (Facebook)
Rank Network Degree De￾mographics
Actor Be￾tween￾ness
Close
-ness
Eig￾en￾vec￾tor
De￾gree
Compo￾site
Golo
s
Gra￾zhd.Nab.
Li￾ga 
Izb.
De￾gree 
Over￾lap
City Work
-place
Oleg 
Vladi￾mirov
1 12 10 2 297 145 0 16
1
9 Mos￾cow
--
Maria 
Mosko￾vskaia
5 -- -- 7 159 140 20 0 1 Mos￾cow
Free￾lance 
jour￾nal-ist
Denis 
Stepa￾nov
6 15 13 11 139 48 8 85 2 Mos￾cow
Pixonic
Mikhail 
Turnov
-skiy
4 17 -- 12 138 80 0 58 0 Mos￾cow
Peter￾Service
Olga 
Ruben
20 -- -- -- 93 62 0 32 1
Mos￾cow
Free￾lance 
design￾er
Svet￾lana 
Boch￾arova 12 -- -- -- 93 76 19 0 2
Mos￾cow
Gazeta
.ru
Alex￾ander 
Rad￾zievskiy 15 -- -- -- 80 29 0 51 0
Mos￾cow Yandex
Svet￾lana 
As￾trakh￾antseva 16 -- -- -- 55 43 12 0 0
Mos￾cow --
Daria 
Agapov
a 18 -- -- -- 33 0 14 19 0
St. 
Peters￾burg --
Two things are immediately apparent. First, while some individuals do take active part in various 
sub-networks, there is not a lot of direct brokerage in the amalgamated network as a whole: 
given the numbers of interactions involved, very few of them simultaneously involve different 
sub-components of the network. Second, and perhaps even more strikingly, none of the brokers 
are Golos staff or close affiliates. While some prominent Golos activists, such as Grigorii 
Melkoniants, are found within the Facebook sample, they are not visible in the non-Golos parts 
of the network. Of the nine brokers identified, three are computer programmers, two are jour￾nalists, one is an interior designer and three could not be determined; eight of the nine live in 
Moscow, with the remaining one in St. Petersburg. They are, thus, enthusiasts, whose interests 
happen to span the network as a whole, and who spread ideas and messages from one part of 
the network to another more by an accident of presence than by purposeful or directed action. 
While not supporting the first sub-hypothesis, it does suggest a closer look at the second sub￾hypothesis, to repeat, that Golos’s informal ‘following’ or ‘audience’ plays an important role in 
GOLOS IMPACT EVALUATION—FINAL REPORT B-15
gluing the network together. To test this hypothesis, we conducted a ‘triad census,’ counting the 
numbers of various isomorphic triads within the network, i.e., the relationships between any 
given set of three actors. We did this for the network as a whole and for each of the three sub￾networks, in every case also comparing the results to a random distribution of triads in a net￾work of the same size. The results are presented in Table 5. In the final column, the table also 
reports ‘new connections’, defined simply as the difference between the number of triads found 
in the composite network and the sum of the networks individually—in other words, the degree 
to which bringing the three sub-networks together creates something greater than the sum of 
the parts. 
Table 5: Triadic Census of Golos, Grazhdanin Nabliudatel’ and Liga Izbiratelei Fa￾cebook Co-Commenter Networks 
Triad Type Golos Grazhdanin 
Nabliudatel’
Liga 
Izbiratelei
Composite New Connec￾tions*
Empty Triad
(1-003)
1 548 863
(0.45)
151 492
(0.28)
13 850 724
(0.32)
69 161 719
(0.15)
53 610 640
Dyad
(3-102)
537 766
(-0.37)
38 405
(-0.39)
3 769 913
(-0.39)
9 137 968
(-0.46)
4 791 884
Incomplete 
Triad
(11-201)
49 723
(-0.78)
2 029
(-0.82)
207 777
(-0.83)
314 037
(-0.80)
54 508
Complete 
Triad
(16-300)
26 588
(0.35)
995
(0.50)
146 536
(0.84)
174 336
(2.58)
217
Variance versus expected random triad distribution in parentheses 
* Difference between number of triads found in the Composite Network and the sum of the networks individually 
Reviewing the variance versus expected random triad distributions (reported in parentheses), 
we note that all four networks—the three individual sub-groups and the composite—follow 
broadly the same pattern. Both have more than expected empty triads (sets of three individuals 
with no connections between them) and complete triads (sets in which all three are connected), 
and fewer than expected dyads (sets in which only two are connected) and incomplete triads 
(sets that include two dyads). Following Granovetter’s theory of the strength of weak ties, what 
we see here is a mobilizational network that we might not expect to be productive and resilient, 
comprising numerous disconnected individuals and numerous closely bonded sets, but relatively 
few of the ‘weaker’ ties that give rise to the long pathways that move information and ideas 
from one part of the network to another. This is true of the individual sub-networks, as well as 
of the composite network as a whole. However, the combination of the networks into one 
whole does lead to a significant increase in the weaker ties: 52 percent of the dyads and 17% of 
the incomplete triads in the composite network are ‘new’. To an extent, this exponential in￾crease in weaker connections is to be expected any time a network grows in size, but there is 
nothing inevitable about it. Indeed, the fact of the creation of these new, weaker links within the 
network serves to emphasize the conclusion of the previous sub-section that the network is 
supported not so much by a cohort of centrally located ‘brokers’, but by a large and disparate 
set of ‘foraging’ activists.
If we are to gauge the extent to which Golos’s following of ‘foraging activists’ contributes some￾thing in particular to the strength and/or sustainability of the amalgamated movement net-
GOLOS IMPACT EVALUATION—FINAL REPORT B-16
work—i.e., to test the third sub-hypothesis—we need to understand a bit more about what 
these individuals are doing. To do this, we turn to a very simple textual analysis. Because our 
dataset includes the texts of posts to the Golos, Liga Izbiratelei and Grazhdanin Nabliudatel’ Fa￾cebook community pages, but because these texts themselves are too numerous to read effec￾tively, we can undertake a very basic machine-driven content analysis, presented below in the 
form of three word clouds (see Figures 7, 8 and 9). 
It is here that the division of labor within the movement as a whole comes into particularly 
sharp focus. The Golos texts, comparatively, are more focused on news and events; the names 
of institutions, political parties and politicians figure prominently, alongside the sort of language 
mostly commonly found in formal statements. Notably, there is relatively little variation in size 
within the word clouds, indicating a fairly even informational spread, rather than the presence of 
dominant themes or rhetoric. The word cloud for Grazhdanin Nabliudatel’, by contrast, is dom￾inated by the word ‘observers’ (in fact, in the genitive plural, indicating that the observers are 
being done something to, most likely called into action). 
Figure 7: Golos Word Cloud 
GOLOS IMPACT EVALUATION—FINAL REPORT B-17
Figure 8: Grazhdanin Nabliudatel’ Word Cloud
Figure 9: Liga Izbiratelei Word Cloud 
In fact, the texts from the Grazhdanin Nabliudatel’ community are much more logistically ori￾ented, providing activist election observers with instructions and encouragement ahead of and 
during election day. Lastly, the Liga Izbiratelei word cloud is dominated by mobilizational rheto￾ric. Most prominent is a reference and link to a Live Journal blog post by popular comedian, tel-
GOLOS IMPACT EVALUATION—FINAL REPORT B-18
evision host and philanthropist Tatiana Lazareva—the title of which is “Not One Vote for 
Putin.” Other prominent words include threatening portrayals of the state (“forcing,” “pres￾sure”) and encouraging words for activists (“friends,” “mittens”). While it would be an over￾statement to argue that there is a strict division of labor, this basic textual analysis strongly sug￾gests that, purposefully or not, each of the three sub-components of the network fell into its 
own specific role: Golos providing information and reporting; Grazhdanin Nabliudatel’ providing 
organization and logistics; Liga Izbiratelei providing the rhetoric of mobilization and the ‘heat’ of 
contention. 
Further evidence for this conclusion emerges when we repeat the Facebook data collection 
around the time of the May 6 protests and the rise of the Russian Occupy Abai movement. As 
Figure 10 makes abundantly clear, with neither the need for election-related information nor for 
the logistics of launching a major monitoring effort, Golos and Grazhdanin Nabliudatel’ fall away, 
and Liga Izbiratelei is left almost to itself. 
Figure 10: Golos (green), Grazhdanin Nabliudatel’ (blue) and Liga Izbiratelei (red) 
on Facebook, May 2012 
GOLOS IMPACT EVALUATION—FINAL REPORT B-19
Thus, while research conducted exclusively with reference to online data can, at best, only pro￾vide hints at answers, and offline research is imperative if those hints are to be transformed into 
robust conclusions, the available evidence suggests that Golos’s network does make a critical 
contribution to the election monitoring community online. It is not, however, the stalwarts of 
the movement that make this contribution; rather, it is the organization’s ‘intangibles,’ the peo￾ple who follow it to the movement, the relationships they bring and form, what they talk about 
and how. Indeed, to sum up this research as a whole, while it is difficult to measure, the evi￾dence suggests that alongside its resources, experience, people and activity, or perhaps part and 
parcel with all of that, Golos brings an ethos to the movement that is a constituent part of its 
success. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-1 
ANNEX C: ELECTION FORENSICS 
In this Annex we report upon analyses of official Central Election Commission (CEC) election data that 
allow us to assess whether the presence or absence of Golos observers made a difference to the level 
of fraud at specific polling stations either in the Duma elections of December 2011 or the presidential 
elections of May 2012. 
The report begins with an outline of the methodology used to identify evidence of election fraud and a 
consideration of the limitations of these methods. Although Golos observers were not deployed ran￾domly in terms of the regions or even cities selected for observation, the actual polling stations selected 
within specific urban neighborhoods (raiony) were selected at random. This creates the opportunity for 
us to examine whether there were differences in the extent of the evidence of fraud in polling stations 
that had Golos observers (treatment group) and the remaining polling stations in the same neighbor￾hoods that did not have Golos observers (control group). 
Although it has a highly precise sounding name, election forensics is really a rather inexact science, and 
only large differences between levels of fraud in different sets of polling stations can be identified. More￾over, while randomization within neighborhoods helps us create appropriate treatment and control 
groups, as we note below, there are other elements of the context that cannot be completely con￾trolled that may introduce experimentally undesirable variation. Consequently, the results of the elec￾tion forensic analysis cannot be seen as definitive. 
Despite these limitations, we believe that the results of the analysis are highly informative. In sum, while 
we find plenty of evidence of fraud (especially in the Duma elections) we found little evidence that the 
simple presence of Golos observers made a major difference in the extent of fraud across polling sta￾tions. Interestingly, however, while there is little evidence of more or less ballot stuffing in observed 
versus unobserved polling stations, there is some evidence of an observer effect. 
 Unobserved polling stations have higher average vote share for United Russia. 
 Observed polling stations are less likely than unobserved polling stations to record 100 percent 
turnout. 
 Votes cast in mobile ballot boxes and using absentee ballots systematically favored United Russia 
and Vladimir Putin more in observed polling stations than in unobserved polling stations. 
Interpreting these effects on the basis of current data is complicated as there are a number of plausible 
explanations, but the data do provide at least some prima facie evidence, especially in the Duma elec￾tions that the sheer presence of election observers does affect important outcomes in polling stations in 
at least a limited way. 
METHODOLOGY 
The science of election forensics is still in its infancy, but there are already some widely used and re￾spected ways of extracting evidence of fraud or ballot stuffing from official returns at the polling station 
level. Among the many different techniques used in this field, we use two that stand out as being particu￾larly useful and reliable for looking at electoral fraud in Russia. The first is to use regression analysis to 
look at the relationship between voter turnout and the vote share for particular political parties or can-
GOLOS IMPACT EVALUATION—FINAL REPORT C-2 
didates. As turnout out goes up, in general, the vote share for each candidate or party should remain 
reasonably consistent. Where this is not the case—where for example increases in turnout benefit only 
the ruling party—there is potential evidence of electoral malfeasance at work. Moreover, the extent of 
deviation in the vote for one candidate or party from the experience of others can be used as one esti￾mate of the overall extent of ballot stuffing in a group of polling stations. 
Identifying standards of evidence in election forensics is not easy. Myagkov, Ordeshook and Shakin 
(2009) advocate treating as worthy of further attention any coefficient of correlation between turnout 
and a given candidate’s vote share that is greater than the same candidate’s or party’s overall vote share. 
For instance, if United Russia’s vote share were 0.6 in a given region and the correlation between turn￾out and United Russia’s vote share in that region were 0.7, this would suggest that something unusual 
may be happening in that region. In some cases, to corroborate evidence of positive correlation be￾tween turnout and vote share for the dominant candidate or party, we looked for negative correlation 
between turnout and the vote shares of opposition candidates or parties. 
In addition, we looked for correlations between United Russia’s or Putin’s vote share and the propor￾tion of ballots cast in mobile ballot boxes. In Russia, mobile ballot boxes are used to allow voting in 
homes and hospitals, ostensibly to help the elderly, sick, disabled, or others without the means to get to 
the polls. However, they have also been used in offices and universities by supervisors and administra￾tors claiming to need them for employees, students, and residents, providing a way for employers, uni￾versity directors, and other institutional authorities to violate the confidentiality of employees’ or stu￾dents’ votes or influence them through intimidation. There were instances of applications for mobile 
voting being received from people who had not asked for the ballot box to be brought to them and later 
claimed to know nothing about the applications in their names, or who could not have filled out an ap￾plication because they were blind or recently deceased14. The purpose of examining this relationship 
separately is to determine whether fraud or falsification was perpetrated in ways that observers, who 
were only present at polling stations, would have had more difficulty witnessing. Given frequent com￾plaints about the use of absentee ballots we also looked at the relationship between absentee ballots 
and parties’/candidates vote share.
The second technique we use to estimate variation in the extent of fraud in the groups of regions and 
polling stations is the distribution of voter turnout across polling stations. Generally, voter turnout 
should be distributed “normally” (i.e., bell-shaped) with most polling stations having a turnout near the 
mean and fewer at the extremes. Once we take into account factors such as age or income that might 
explain natural variability, significant departures from this distribution, especially ones that vary between 
political parties or candidate can be treated as prima facie evidence of fraud and can be used to generate 
estimates of the degree of ballot-stuffing or vote stealing that has taken place. 
The large number of polling stations in Russia (approximately 95,000 in 83 regions), and the large num￾ber actually observed by Golos (approximately 4,000 in 40 regions for the Duma elections) allow us 
enough observations to use these techniques to draw some reliable conclusions about relative patterns 
of fraud in Golos observed and unobserved regions, as well as observed and unobserved polling stations. 
To determine whether the presence of observers influenced results, we analyzed results from what es￾sentially amounted to a control set and a treatment set. The treatment set consisted of all polling sta￾tions (uchastki) where Golos observers were present before and during elections, while the control set 
 
14 White, S. 2011. “Elections Russian-Style.” Europe-Asia Studies 63(4), 531-56. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-3 
consisted of all remaining polling stations in the same neighborhoods (raiony). Selecting by raion enabled 
us to place polling stations from the same neighborhoods into both the control and treatment sets, 
which should help to reduce the effect of confounding variables such as voters’ income, ethnicity, or 
other demographic factors likely to differ from neighborhood to neighborhood. 
During the elections, Golos had observers working in polling stations in 40 of Russia’s 83 regions for the 
State Duma elections, and in 48 regions for the presidential elections. Russia’s regions are divided into 
three different types that are relevant for the analysis—two cities of Federal significance (Moscow and 
St. Petersburg), 55 regions (oblasti or krai) and 21 Republics (respubliki). For the Duma elections, 
Golos’s staff recorded observations in Moscow and St. Petersburg, six republics, and twenty-nine oblasts 
or krais. Before and during the presidential elections, Golos’s observers kept records of their work in 
both federal cities, seven republics, and thirty-three oblasts or krais. Most of Golos’s observers worked 
in urban locations, usually in a regional capital but occasionally in another large city (as in Novokuznetsk 
in Kemerovo Oblast’) or more than one city per region (as in Sochi, Novorossijsk, and Krasnodar in 
Krasnodar Oblast’). However, in several regions, the observers worked in neighborhoods outside the 
city limits and even in rural areas. The number of neighborhoods (raiony) per region to which Golos 
sent observers ranged from one (as in the Kostroma region) to more than a hundred (as in Moscow). 
The total number of polling stations in each neighborhood (either with or without observers) varied 
greatly, but most had between 20 and 100. Even where Golos’s observers were present in only one 
polling station in a given neighborhood, we included that polling station in the analysis, placing the ob￾served polling station in the treatment set and unobserved polling stations in the same raion in the con￾trol set. 
METHODOLOGICAL LIMITATIONS 
There are a number of limitations to the conclusions that can be drawn from the analysis. First, as noted 
above, there are no clear standards that separate conclusive evidence of fraud from suggestive evidence 
in the field of election forensics. We have no DNA evidence that can rule fraud out. Instead, what we 
have is knowledge of the statistical likelihood of certain patterns. Unlikely patterns are cause for suspi￾cion and differences in patterns are suggestive of differences between observed and unobserved polling 
stations, but they cannot be definitive proof. 
A related critique of election forensics is that we cannot be certain that, for example, a correlation be￾tween turnout and the vote for a particular party is due to fraud or simply due to an effective “get out 
the vote” effort. In general this critique is quite powerful. In the particular design we have used, howev￾er, where we look for differences between randomly selected polling stations within the same neighbor￾hoods, this problem should not arise unless there is some correlation between the selection of polling 
stations to observe and the effectiveness of voter mobilization efforts. We are, after all, not looking for 
evidence of fraud, but for differences in the evidence of fraud between observed and unobserved polling 
stations. 
Nevertheless, the design puts a lot of weight on the selection of polling stations within neighborhoods as 
being actually randomly selected. We have no reason to believe that this is not the case. Observers used 
Kish tables to select the polling stations to observe. Nevertheless, we could not possibly observe the 
process by which all observers actually chose polling stations. 
More important, at least potentially, are possible issues with the treatment that was actually applied to 
polling stations and the difference between treatment and control groups. We only have data from 
Golos observers, though election observers from other groups were active in both elections. Hence we 
cannot be sure that the “unobserved” polling stations were completely unobserved (though we can be 
GOLOS IMPACT EVALUATION—FINAL REPORT C-4 
sure they were not observed by Golos). For the Duma elections, where Golos was the most significant 
independent election monitoring group, this problem is small. For the presidential election, the problem 
is potentially larger since election observers from other groups were quite active. However, even in this 
case, the main efforts of other election monitoring groups were focused in Moscow and Moscow Ob￾last—outside of these regions the extent of the problem should be relatively small. 
Finally, in terms of generalizing from the Russian experience to other cases, there is an additional limita￾tion that needs to be borne in mind that relates to the specific context of election observation in Russia. 
Neither Golos nor other domestic observer groups have a right under Russian election law to observe 
elections. However, journalists do and so the data we have are reports from journalists accredited with 
a Golos newspaper. In terms of the data and the training of observers this is mostly a distinction without 
a difference. However, it does have one potentially important implication. This is that Golos observers 
act a little differently from typical election observers. While observers typically wear hats or uniforms 
that are highly visible and specifically identify them as observers, Golos observers do not. They do iden￾tify themselves to the head of the polling station election commission and interact with commission offi￾cials, but they are not likely to be visible to voters and others in the polling station. Whether this makes 
a difference to their effectiveness is unknown. 
RESULTS 
We present the results as follows. We separate the analysis by techniques, presenting the results of 
plotting turnout and vote share first, then looking at what the distribution of turnout can tell us about 
observed and unobserved polling stations. We also separate the analysis of the December Duma elec￾tions and the presidential elections of May 2012. For the Duma elections, we begin with the results for 
all neighborhoods in Russia with at least some polling stations observed. We then break the picture 
down more carefully between each of the politically relevant kinds of regions—oblasts and krais, Mos￾cow and St. Petersburg, and, then, those Republics that had some election observation effort by Golos. 
We look in some detail at individual regions that seem to present different results than the groups as a 
whole. For the presidential elections, we look at patterns in the vote for Vladimir Putin in all of Russia’s 
regions, then present data on voting patterns for his two principal challengers—Gennady Zyuganov and 
Mikhail Prokhorov. In the last section, we compare data patterns on mobile ballot boxes and absentee 
ballots between observed and unobserved polling stations. 
TURNOUT AND VOTE SHARE ANALYSIS
DUMA ELECTIONS 
We begin by comparing patterns between observed and unobserved polling stations for all neighbor￾hoods in the Russian Federation where Golos provided at least some observers. Figure 1 shows the re￾lationship between turnout and United Russia’s vote share for all observed polling stations throughout 
the country, and Figure 2 shows the correlation between turnout and United Russia’s vote share for 
unobserved polling stations in the same neighborhoods. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-5 
Figure 1: Russia Duma Elections – Sta￾tions with Observers
Figure 2: Russia Duma Elections – Sta￾tions with Observers
Figure 1: Intercept: 0.048***, 
Correlation coefficient = 0.563***, p-value = 0.000
Mean United Russia vote share = 0.368
Number of observations = 3,754
R
2= 0.26
Figure 2: Intercept: 0.083***, 
Correlation coefficient = 0.570***, p-value = 0.000 
Mean United Russia vote share = 0.425
Number of observations = 14,667
R
2=0.26
Looking at the effect of turnout on United Russia’s vote share, there is some prima facie evidence of 
fraud—the slope coefficients in each group are greater than the corresponding mean vote shares for 
United Russia. This means that a higher voter turnout means a higher United Russia vote share. Howev￾er, as the correlation coefficients are well below 1, this does not mean that there is strong evidence of 
direct ballot stuffing—increased turnout does mean more votes for other parties too, at least in these 
regions.15 More importantly for our purposes, the correlation coefficients in both observed and unob￾served polling stations are virtually identical. This means that the charts provide no evidence that the 
presence of observers at polling stations deterred fraud or falsification. 
Nevertheless, the results are intriguing. The presence of observers does, in any case, seem to have af￾fected United Russia’s overall vote share, since United Russia’s mean vote share in unobserved polling 
stations was 42.5 percent, while its mean vote share in observed polling stations was 36.8 percent. It is 
difficult to know how to interpret this difference. Given the large number of both observed and unob￾served polling stations, it is unlikely that this difference in means emerged by chance. However, since 
our criterion is to interpret correlation coefficients greater than United Russia’s mean vote share as po￾tential evidence of fraud, we should be cautious. In our data, the unobserved mean vote share for Unit￾ed Russia is 42.5 percent and the unobserved correlation coefficient or slope is 0.57. This is a smaller 
difference than that between the observed vote share of 36.8 percent and the observed correlation co￾efficient of 0.56. In a set of polling stations where ballot stuffing had occurred, we would expect the dif￾ference between this coefficient and the favored party’s vote share to be greater than in a set of polling 
stations where no ballot stuffing had occurred. Therefore, the difference between observed and unob￾served stations in mean vote shares is not likely due to simple ballot stuffing in the unobserved stations. 
Put differently, instead of a different slope, which would indicate blocks of all (or almost all) United Rus￾sia’s votes being added to ballot boxes in unobserved polling stations, the correlation line is shifted up—
the intercept is higher. This would suggest that for any given level of turnout, unobserved polling sta￾tions receive a larger proportion of United Russia votes. There are at least two possible explanations. 
 
15 We should remember that Golos did not deploy observers in the Caucasus region where election fraud was most obvious. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-6 
One possibility is that the randomization of polling stations did not eliminate systematic differences be￾tween observed and unobserved polling stations. Without access to additional data at the polling station 
level, it is impossible to rule this in or out. Alternatively, if the selection really was random with respect 
to variables that affect voting behavior, then some the differences in the means come not from different 
levels of ballot stuffing (as we will see there is evidence of ballot stuffing in both sets) but from other 
forms of fraud that are perpetrated in the absence of election observers. It is relatively easy to think of 
possibilities—e.g., perhaps voters in unobserved polling stations are systematically more likely to be 
pressured or encouraged to vote for the ruling party thus raising UR’s proportion of votes for any given 
level of turnout. However, on the basis of the data available here, it is impossible to say for sure. 
The next stage in the analysis is to look for any potential differences in the observer effect in certain 
regions or groups of regions that might be hidden by the overall national data. We look in turn at data 
from oblasts and krais, the two Federal cities of Moscow and St. Petersburg and the ethnic Republics. 
Like aggregated results for all raiony in the country, the results taken from just the oblasts/krais (Figs. 3 
and 4) and federal cities (Figs. 5 and 6) show no clear differences between observed and unobserved 
stations in evidence of fraud. All of these sets—cities or oblasts, observed or unobserved—show coeffi￾cients for correlation between turnout and United Russia’s vote share greater than the mean of United 
Russia’s vote share, indicating probable fraud or falsification. In the oblasts turnout is somewhat more 
highly correlated with United Russia vote share in unobserved polling stations (0.58) compared to ob￾served polling stations (0.53), though the differences are relatively small. For Moscow and St. Petersburg, 
the reverse is true, but here the number of observed polling stations is too small to support any conclu￾sions. It is worth noting, however, that once again the observed stations have a lower mean United Rus￾sia vote than the unobserved stations. 
Figure 3: Oblasts and Krais, Observed Figure 5: Oblasts and Krais, Unobserved
Figure 3:Intercept: 0.066***
Correlation coefficient = 0.529***, p-value = 0.000
Mean United Russia vote share = 0.36
Number of observations = 3,030
R
2= 0.25
Figure 4: Intercept: 0.084***
Correlation coefficient = 0.576***, p-value = 0.000
Mean United Russia vote share = 0.42
Number of observations = 8,667
R
2= 0.30
GOLOS IMPACT EVALUATION—FINAL REPORT C-7 
Figure 5: Federal Cities Observed Figure 6: Federal Cities Unobserved
Figure. 5: Intercept: 0.029
Correlation coefficient = 0.618***, p-value = 0.000
Mean United Russia vote share = 0.39
Number of observations = 178
R
2= 0.19
Fig. 6: Intercept: 0.104***
Correlation coefficient = 0.505***, p-value = 0.000
Mean United Russia vote share = 0.42
Number of observations = 4,241
R
2= 0.18
By contrast, the results for the republics show a more surprising result: by the criterion of correlated 
turnout and vote share, observed stations show more evidence of fraud than unobserved stations in the 
republics at first glance. Figures 7 and 8 show the correlation between turnout and United Russia’s vote 
share in the republics. 
Figure 7: Republics Observed Figure 8: Republics Unobserved
Figure 7: Intercept: -0.045, 
Correlation coefficient = 0.738***, p-value = 0.000
Mean United Russia vote share = 0.421
Number of observations = 417
R
2= 0.33
Figure 8 : Intercept: 0.326***, 
Correlation coefficient = 0.404***, p-value = 0.000
Mean United Russia vote share = 0.603
Number of observations = 869
R
2= 0.17
However, a closer look shows that this conclusion would be wrong. The high correlation coefficient 
seen in observed stations is in fact the result of aggregation error. In the Republic of Mari-El and the Re￾public of Karelia, the mean turnout was 59 percent and United Russia won between 20 percent and 40 
percent at most observed polling stations. In Adygeja, the Altai Republic, the Komi Republic, and Ta￾tarstan, the mean turnout was 69 percent and United Russia won between 60 percent and 80 percent in 
most observed polling stations. Lumping these two groups together creates the appearance of a rela￾tionship between turnout and United Russia’s vote share that is eliminated when Mari-El and Karelia are 
treated separately, as shown in Figures 9-14. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-8 
Figure 9: Adygeja, Altai, Komi, and 
Tatarstan, Observed
Figure 9: Adygeja, Altai, Komi, and 
Tatarstan, Unobserved
Figure 9: Intercept: 0.304***
Correlation coefficient = 0.404***, p-value = 0.000
Mean United Russia vote share = 0.583
Number of observations = 159
R
2= 0.17
Figure 10: Intercept: 0.329***
Correlation coefficient = 0.400***, p-value = 0.000
Mean United Russia vote share = 0.601
Number of observations = 830\
R
2= 0.17
Figure 11: Republic of Mari-El Observed Figure 12: Republic of Mari-El Unob￾served
Figure 11: Intercept: -0.098
Correlation coefficient = 0.681***, p-value = 0.000
Mean United Russia vote share = 0.36
Number of observations = 139
R
2= 0.31
Figure 12: Intercept: 0.362
Correlation coefficient = 0.437, p-value = 0.278
Mean United Russia vote share = 0.74
Number of observations = 31
R
2= 0.04
GOLOS IMPACT EVALUATION—FINAL REPORT C-9 
Figure 13: Repubic of Karelia Observed Figure 14: Repubic of Karelia Observed
Figure 13: Intercept: 0.319***
Correlation coefficient = -0.080, p-value = 0.296
Mean United Russia vote share = 0.28
Number of observations = 119
R
2= 0.01
Figure 14: Intercept: 0.234
Correlation coefficient = 0.124, p-value = 0.297
Mean United Russia vote share = 0.33
Number of observations = 8
R
2= 0.18 (Adj. R2= 0.04)
Analyzed in this way, all the republics except Karelia still show positive, statistically significant correla￾tions between turnout and United Russia’s vote share. The coefficient of correlation between turnout 
and United Russia’s vote share is still greater in the observed stations than in the unobserved stations in 
the Republic of Mari-El, but this appears to be an accident of distribution resulting from the small num￾bers of polling stations. 
PRESIDENTIAL ELECTIONS, MARCH 2012 
The fraud detection results found for the presidential elections are comparable to those from the Duma 
elections: Putin’s vote share is positively correlated with turnout in both observed and unobserved sta￾tions, with neither showing clearer evidence of falsification than the other (Figures 15 and 16). These 
results differ from those found for the Duma elections in that neither of the slope coefficients is larger 
than Putin’s mean vote share. This suggests that the evidence for ballot stuffing in regions that had at 
least some Golos observers is weak. However, while there is little evidence of ballot stuffing in favor of 
Vladimir Putin, there is evidence that the second- and third-place candidates did not enjoy similar bene￾fits from high turnout (Figs. 17-20). This suggests that either the get out the vote operation for the 
Prime Minister was more effective than that of his competitors, or that some other phenomena was at 
play that meant that Ziuganov and Prohkorov suffered from higher turnout. Whatever the evidence of 
fraud, there is little evidence of difference between observed and unobserved polling stations. Both sets 
of stations had similar correlations between turnout and vote share and similar mean levels of support 
for all three candidates examined here. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-10
Figure 16: Observed Stations – All 
Observed Rajony
Figure 16: Unobserved Stations – All 
Observed Rajony
Figure 16: Intercept: 0.390***, 
Correlation coefficient = 0.285***, p-value = 0.000, 
Number of observations = 9,017 
Mean vote share for Putin = 0.576
R2= 0.11
Figure 16: Intercept: 0.390***, 
Correlation coefficient = 0.285***, p-value = 0.000, 
Number of observations = 9,017 
Mean vote share for Putin = 0.576
R2= 0.11
Figure 17: Observed Polling Stations -
Zyuganov’s Share
Figure 18: Unobserved Polling Stations -
Zyuganov’s Share
Figure 17: Intercept: 0.258***
Correlation coefficient = -0.101***, p-value = 0.000
Mean vote share = 0.19
Number of observations = 6999
R2= 0.03
Figure 18: Intercept: 0.270***
Correlation coefficient = -0.135***, p-value = 0.000
Mean vote share = 0.18
Number of observations = 9017
R2= 0.08
GOLOS IMPACT EVALUATION—FINAL REPORT C-11
Figure 19: Observed Polling Stations -
Prokhorov’s Vote Share
Figure 20: Unobserved Polling Stations -
Prokhorov’s Vote Share
Figure 19: Intercept: 0.175***
Correlation coefficient = -0.099***, p-value = 0.000
Mean United Russia vote share = 0.11
Number of observations = 6999
R2= 0.03
Figure 20: Intercept: 0.210***
Correlation coefficient = -0.146***, p-value = 0.000
Mean vote share = 0.12
Number of observations = 9017
R2= 0.08
DISTRIBUTION OF TURNOUT 
DUMA ELECTION 
Figure 21 shows the distribution of turnout in observed and unobserved polling stations for the Duma 
elections. There are a number of elements to observe. First, the dotted lines, which mark the means of 
the distributions, show that both distributions have right skew, indicating more polling stations with high 
turnout than would normally be expected. However, since both have similar skew, there is little evi￾dence here of an observer effect. Second, as noted above, turnout distributions expected with ballot￾box-stuffing would tend toward bimodality—that is we would expect two (or more) peaks where there 
is ballot stuffing and only one where there is not. The distribution for observed stations is clearly multi￾modal, with a number of small peaks often near round numbers (this effect was widely remarked upon 
in the Russian blogosphere).16 Moreover, there is a marked second peak at 100 percent turnout. The 
extent to which these non-Gaussian distributions reflect malfeasance is not uncontroversial, but for our 
purposes what matters are not the distributions individually, but differences between them. As Figure 21 
shows, both functions show very similar non-Gaussian characteristics and so show no clear difference 
between observed and unobserved polling stations in terms of the probability that fraud or falsification 
occurred. 
 
16 http://www.newsland.ru/news/detail/id/838730/ The position that votes should exhibit a normal distribution is not uncontro￾versial. See e.g., http://www.significancemagazine.org/details/webexclusive/1424089/Mathematical-proof-of-fraud-in-Russian￾elections-unsound.html 
GOLOS IMPACT EVALUATION—FINAL REPORT C-12
Figure 21: Distribution of Turnout for Duma Elections – Unobserved and Observed Polling 
Stations 
The one marked difference between the two sets of polling stations is the larger peak at 100 percent 
turnout evident in unobserved polling stations. There is a peak there too for observed polling stations, 
but it is proportionately smaller. This is a potentially important observer effect, which suggests that arti￾ficially inflating turnout is easier in unobserved polling stations, but not impossible in observed stations 
too. 
In Figure 22 we look at the distribution of vote share for United Russia. After the Duma elections, the 
saw-tooth shape and pronounced rightward skew in the distribution of United Russia’s vote share con￾trasted with the normal distributions of other parties’ vote shares, and was one of the first and clearest 
signs that fraud had taken place. Here again both distributions indicate possible fraud, but the pattern of 
fraud are not particularly different, with the important exception of the large difference in the mean be￾tween observed and unobserved stations that we noted before. 
Figure 22: Vote share for the dominant party in the State Duma elections, at observed sta￾tions and unobserved stations in the same neighborhoods 
GOLOS IMPACT EVALUATION—FINAL REPORT C-13
PRESIDENTIAL ELECTION 
As in the Duma election, the turnout distribution for this election is skewed to the right for both ob￾served and unobserved stations, but, based on the position of the means, seems to deviate less from the 
normal for both (see Figure 23). Once again we have a larger peak at 100 percent turnout in unobserved 
stations, which in part accounts for the larger mean vote share for Vladimir Putin in unobserved sta￾tions. 
Figure 23: Distribution of Turnout in the Presidential Election at Observed and Unob￾served Polling Stations in the Same Neighborhoods 
Observed Mean Turnout: 62.6%, Unobserved Mean Turnout: 65.4% 
The distribution of Putin’s vote share in the presidential elections is also not perfectly normal (Figure 
24), but again shows less evidence of artificially inflated numbers and little difference between observed 
and unobserved polling stations. 
Figure 24: Vote share for Putin at Observed and Unobserved Polling stations in the Same 
Neighborhoods 
Observed Mean Vote Share: 56.8%, Unobserved Mean Vote Share: 57.6% 
GOLOS IMPACT EVALUATION—FINAL REPORT C-14
Mobile Ballot Boxes and Absentee Ballots 
There is another correlation worth comparing between observed and unobserved polling stations as 
well as between elections. If United Russia’s or Putin’s vote shares correlate positively with the percent￾age of votes cast in mobile ballot boxes, it would suggest that their supporters were attempting to influ￾ence the results using a method less likely to be detected by polling-station-based observers. 
Figures 25-28 represent the correlation of the proportion of ballots cast in mobile ballot boxes for each 
election and vote shares for United Russia and Putin.17 As these figures clearly show, the correlation 
between the proportion of votes cast in mobile ballot boxes and the vote for either United Russia or 
Vladimir Putin is positive and statistically significant in both elections. Mobile ballot box users were more 
likely to support United Russia or Vladimir Putin. Given that the population using mobile ballot boxes is 
probably not representative of the population as a whole, this should not be surprising. However, im￾portantly for the comparison of observed and unobserved polling stations, the coefficient is nearly twice 
as large in observed polling stations in the Duma elections and half as large again in the presidential elec￾tions as in unobserved polling stations. What might account for this? 
Figure 25: Duma Elections – Observed 
Polling Stations
Figure 26: Duma Elections – Unobserved 
Polling Stations
Figure 25: Intercept: 0.350***
Correlation coefficient = 0.382***, p-value = 0.000
Mean United Russia vote share = 0.37
Number of observations = 3,754
R
2= 0.03
Figure 26: Intercept: 0.409***
Correlation coefficient = 0.200***, p-value = 0.278
Mean United Russia vote share = 0.42
Number of observations = 14,667
R
2= 0.02
 
17 Since the selection of polling stations is randomized, the results are very similar if we look at the absolute num￾ber of votes cast in mobile ballot boxes instead of the proportion of votes. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-15
Figure 27: Presidential Elections –
Observed Polling Stations
Figure 28: Presidential Elections –
Unobserved Polling Stations
Figure 27: Intercept: 0.543***
Correlation coefficient = 0.434***, p-value = 0.000
Putin’s mean vote share = 0.57
Number of observations = 6,999
R
2= 0.11
Figure 28: Intercept: 0.555***
Correlation coefficient = 0.301***, p-value = 0.278
Putin’s mean vote share = 0.58
Number of observations = 9,017
R
2= 0.08
One possible, benign explanation for the first correlation is that some demographic sector that uses 
these mobile ballot boxes particularly often, such as the elderly, is more likely to support United Russia. 
This is certainly possible, though we do not currently have the data to test this possibility. However, 
even if it were the case, differences in who votes in mobile ballot boxes would not account for the dif￾ference between observed and unobserved polling stations. It might also be possible to interpret this as 
a disconfirmation that the mobile ballot boxes are more likely to be stuffed. After all, if the correlation 
was especially evident at observed stations, one could argue that this proves that correlation between 
mobile ballot box voting and vote share is simply a poor indicator of fraud. 
However, the fact that this correlation occurred precisely where observers should have been monitor￾ing the ballot boxes is telling, especially if the use of mobile ballot boxes made the ballot boxes particu￾larly difficult to monitor. There are nominal provisions in place allowing observers to check all ballot 
boxes, but it was frequently the case that observers were not able to monitor mobile boxes while they 
were away from the polling stations. It is plausible that where observers were present at the polling sta￾tion, perpetrators of fraud relied more on this particular nonstandard voting procedure to alter the 
numbers. 
In addition to votes from mobile ballot boxes, another important form of non-standard voting, absentee 
certification, might also have been used to alter the count. In Russia, absentee voting means voting at a 
polling station other than at your place of residence. While intended to allow voters who happen to 
away from home on the day of the election to participate in elections, absentee voting has been repeat￾ed identified by Golos and others as a major source of election fraud, either through additional pressure 
on vote choice when voting in the workplace, or through so-called “carousel” voting, where organized 
groups cast absentee ballots in more than one polling station. 
Our findings suggest that there are indeed good reasons to be wary of absentee ballots in Russia. To 
investigate the effect of absentee voting, we tested the hypothesis of a positive correlation between the 
dominant party’s or candidate’s vote share and the proportion of absentee votes to total votes. To 
check for this correlation we use the ratio of accepted, valid absentee certifications to the total number 
of valid ballots as our independent variable, and use Putin’s or United Russia’s vote share as our de￾pendent variable. The results are presented in Figures 29-32, which show the same pattern with absen￾tee voting that we found with mobile ballot box voting: the correlation between Putin’s or United Rus-
GOLOS IMPACT EVALUATION—FINAL REPORT C-16
sia’s vote share and the proportion of absentee votes is positive and larger for observed stations, than 
for unobserved stations. In fact, the correlation between absentee votes and vote share for United Rus￾sia or Putin is actually negative for unobserved stations, suggesting that in unobserved polling stations, 
absentee ballots were not a significant source of fraud. Again, this might be due to demographics: expat￾riates, frequent travelers, and students educated away from home, for instance, might have more liberal, 
cosmopolitan, and pro-Western views. Once again, however, demographics ought not to be able to ex￾plain the differences between observed and unobserved polling stations. 
Figure 29: Duma Elections, Observed 
Stations
Figure 30: Duma Elections, Unobserved 
Stations
Figure 29: Intercept: 0.345***
Correlation coefficient = 0.737***, p-value = 0.000
United Russia’s mean vote share = 0.37
Number of observations = 3,755
R
2= 0.01
Figure 30: Intercept: 0.436***
Correlation coefficient: -0.394***, p-value = 0.000
United Russia’s mean vote share = 0.42
Number of observations = 14,667
R
2= 0.003
Figure 31: Presidential Election – Ob￾served Stations
Figure 32: Presidential Election –
Unobserved Stations
Figure 31: Intercept: 0.565***
Correlation coefficient = 0.085, p-value = 0.085
Putin’s mean vote share = 0.57
Number of observations = 6,999
R
2= 0.0004
Figure 32: Intercept: 0.580***
Correlation coefficient = -0.129**, p-value = 0.003
Putin’s mean vote share = 0.58
Number of observations = 9,017
R
2= 0.0009
Whatever the explanation, the clear, visible difference between observed and unobserved polling sta￾tions in the degree to which both kinds of nonstandard voting favor Putin and United Russia is intriguing. 
GOLOS IMPACT EVALUATION—FINAL REPORT C-17
It seems that high vote shares for Putin and United Russia depended in part on these nonstandard votes 
wherever Golos’s observers were present. Our best evidence seems to show that Golos’s observation 
at polling stations did little to prevent fraud, but can we conclude that it compelled the perpetrators to 
resort to other means of falsifying results? 
Unfortunately, the forensics techniques we have used are too inexact to prove or disconfirm this con￾clusion. We have shown an effect of nonstandard voting that differs consistently between observed and 
unobserved polling stations, but on the basis of these data alone the appropriate interpretation is un￾clear. In short, our findings raise questions worthy of further and deeper investigation. 
GOLOS IMPACT EVALUATION—FINAL REPORT D-1 
ANNEX D: EXPERIMENTS ON 
ATTITUDES TO OBSERVERS 
AND FOREIGN FUNDING 
In this annex, we review the main results of the SPARCs survey of attitudes of election observ￾ers. We argue that the data demonstrate a considerable degree of support for and trust in do￾mestic observers. However, the data suggest that although attitudes toward foreign involvement 
are complicated, even among highly educated, upper income, internet-using urbanites, there is 
some skepticism regarding foreign election observers and considerable opposition to foreign 
funding of Russian observers. Moreover, state-sponsored efforts to shape citizens’ attitudes 
through media campaigns seem to have had some effect. Overall, there seems to be real support 
for domestic and foreign election monitoring, though there is evidence that it is important that 
observers are seen as organically grown from domestic sources rather than imported from 
abroad. 
PURPOSE 
One of the key elements that needs to be taken into account when evaluating the impact of 
supporting an organization such as Golos is the broader political context in the organization 
works and the interpretation within that context of support given to organizations like Golos 
that seek to play a role in the political process. Put bluntly, foreign assistance given to election 
monitoring organizations in semi-authoritarian contexts like Russia could be counterproductive 
if receiving foreign support discredits any organization in the eyes of the public. 
In order to probe the effects of foreign—specifically, U.S.—sources of most of Golos’s funding 
on attitudes toward election observers and the effects of these attitudes, we included a series of 
survey experiments to determine whether providing information on Golos’s funding sources 
made a difference in respondents’ evaluations of election observation. 
DESCRIPTION 
The experiment was administered over two survey rounds. In round 1, conducted two weeks 
before the presidential election of March 2012, respondents were randomly assigned one of 
four texts to read and asked a series of questions about their attitudes toward election observ￾ers. The texts were lightly modified versions of reports that had appeared in Russian newspa￾pers around the period of the election. The first text was a short, neutral text stating that elec￾tions were held to the Duma, noting the number of candidates, parties and voters and the fact 
that all parties in the out-going Duma were represented in the new one. Respondents reading 
this text are represented in the tables has receiving the “Neutral” treatment. The second text 
mentioned Golos specifically, detailing some criticisms leveled at the elections and noting that 
Golos is a Russian organization and had been working on Russian elections since 2000. This text 
is referred to in the table as the “Golos” treatment. In the third text this descriptive information 
on Golos was replaced with a paragraph from a Russian tabloid story that actually appeared on 
the eve of the elections describing Golos as having close ties to the US State Department and 
receiving not just moral support but also detailed instructions and money. This is the “GosDep” 
GOLOS IMPACT EVALUATION—FINAL REPORT D-2 
treatment. Finally for round 1, the fourth text was identical to the second but instead the criti￾cisms were presented as coming from the OSCE and some descriptive information on OSCE 
monitoring was provided. This is the “OSCE” treatment. 
In round 2, respondents were randomly assigned to receive either just the questions, with no 
framing (“Control”), or a two-minute introductory clip of a “documentary” aired on the eve of 
the Duma elections on the pro-government news station, NTV. Entitled “Voice from Nowhere” 
(“Golos Niotkuda”), the clip represents Golos as engaged in illegal, extremist activities and as 
being a tool of the US, evaluating elections negatively in accordance with pre-ordered instruc￾tions from Washington. This treatment is referred to as “NTV.”
The range of treatments allows us to examine several different aspects. By comparing the “Neu￾tral,” “Golos,” and “OSCE” treatments, we get a sense of whether the brand name “Golos” has 
any effect on respondents’ opinions of election observers and whether this effect is similar to, 
or different from, the associations of the “OSCE” brand. Similarly, comparing these treatments 
with the “Golos – GosDep” and “Golos – NTV” treatments allows us to see whether evalua￾tions are affected by drawing negative attention to Golos’s association with and funding from the 
US government. Moreover, we can compare the impact of different forms of propaganda—the 
written text and attacks made by Russian television. 
RESULTS 
The first thing to note about attitudes to election observation among the SPARCs group is that 
there are high and rising levels of support. In both waves more than 80 percent of respondents 
supported either free or only lightly regulated access to polling stations. Moreover, there is 
some evidence that attitudes to election observers grew even more positive between the two 
waves of the survey. Patterns over time can be gauged by comparing lines 1 and 2 of each ta￾ble.18 While in both rounds 84 percent of respondents expressed a positive attitude to access to 
polling stations for election observers, the balance within that shifted somewhat with the pro￾portion of respondents saying observers should have free access to polling stations growing 
from 56 to 61 percent. 
Furthermore, respondents not only believed that observers should have access to polling sta￾tions, a majority in both waves believed that the presence of observers actually make elections 
more free and fair (Table 2). In the first wave, 60 percent of respondents agreed with this posi￾tion, while only 12 percent disagreed. In wave 2, 56 percent agreed and 9 percent disagreed. 
Nevertheless, despite high levels of support for observer access to polling stations, even highly 
educated, upper income, internet using urbanites in Russia treat the announcements of observ￾ers with a significant degree of skepticism. As Table 3 shows, both before and after the presi￾dential elections, about half of the respondents (51 percent before and 48 percent after) said 
they trusted election observer reports either completely or somewhat. On the other hand, in 
both waves only 11 percent expressed suspicion. More than four in ten respondents remain to 
be convinced either way. 
 
18 Though the treatments here are not absolutely identical they are very close in that we are comparing people in R1 
who read a neutral text and respondents in R2 who simply answered the same set of questions with no prompt. 
GOLOS IMPACT EVALUATION—FINAL REPORT D-3 
FOREIGN OBSERVERS 
Interestingly, however, attitudes among SPARCs to foreign involvement in Russian elections are 
considerably more skeptical. While support for domestic observers is very strong, the prospect 
of foreign involvement, unsurprisingly, elicits a more complex set of responses. However, once 
we separate OSCE observers out from other kinds, including observers from the CIS and 
Shanghai Cooperation Organization, we find levels of support and trust similar to that of domes￾tic observation groups. 
In wave 1, 47 percent of respondents supported either no or only lightly regulation of access for 
foreign observers to polling stations, while the same proportion wanted either strict regulation 
or the outright banning of foreign observers. By wave 2 after the presidential elections, attitudes 
seem to have improved considerably in regard to foreign observers. Then 59 percent supported 
free or lightly regulated access, as opposed to only 35 percent supporting strict regulations or a 
ban (Table 4). Even with the improvement, however, support for strict regulation of access for 
foreign observers remains considerably higher that it is for domestic observation groups. 
One possible source of complication in Russian attitudes towards foreign election observers is 
that there are really two quite distinct kinds of foreign election observation missions that typi￾cally play a role. On the one hand there are observers invited from neighbors like Belarus or 
states such as China and Kazakhstan that, like Russia, are members of the Shanghai Cooperation 
Organization (SCO). On the other hand, there are observers from the OSCE, organized 
through the Office for Democratic Institutions and Human Rights Institute (ODIHR). It is quite 
plausible that respondents would see the former as being sympathetic to the Russian authorities, 
while the latter is more critical. 
In order to try and clarify this issue for respondents, in wave 1 we randomly prompted one 
quarter of respondents with a text providing descriptive information on OSCE monitoring and 
noting that the OSCE was critical of the Duma elections. Those who were specifically prompted 
to think about OSCE observation (OSCE) gave very similar evaluations of observers as those 
who were not prompted to think about any particular groups (Neutral). OSCE prompted re￾spondents gave the strongest endorsement of free access for observers (64 percent compare to 
56 percent of neutrals), though overall positive view on access was only marginally more posi￾tive than the “Neutral” frame—(87 percent compared to 84 percent). Similarly trust in observer 
reports was no different between the OSCE-prompted respondents (49 percent) and the neu￾tral-prompted (51 percent). View of the effect of observers was the same—60 percent of both 
groups felt that observers had at least some impact on making elections more free and fair. 
Consequently, we can conclude from the analysis that international observation missions from 
the OSCE enjoy just as high levels of support from SPARCs as domestic observers. They do not, 
however, appear to be better thought of than domestic observation missions. This underlines 
the importance of domestic political actors in the Russian context. 
FOREIGN FINANCING 
While there is considerable support for election monitors in general, foreign participation in 
supporting Russian domestic observation teams was greeted, even by SPARCs with a much 
higher degree of skepticism. In wave 1, 44 percent of respondents thought that foreign financing 
of domestic election monitoring organizations should be banned completely and a further 26 
percent thought it ought to be tightly regulated. Only 22 percent of respondents felt that such 
assistance should be able to be given either freely or only subject to light regulation. There is 
GOLOS IMPACT EVALUATION—FINAL REPORT D-4 
evidence, however, that attitudes softened somewhat between the two waves of the survey. 
After the presidential elections, the proportion of respondents thinking that foreign support 
should be banned fell to 36 percent and some 29 percent now felt that such assistance could be 
freely given or only lightly regulated. 
GOLOS 
To test the extent to which respondents think differently about election observers associated 
with Golos from observers in general, one quarter of respondents in wave 1 were given a text 
to read that gave descriptive information about Golos’s history, activities. Overall, it appears 
that mentioning Golos specifically has a small positive effect on respondents’ evaluations of elec￾tion observation. More consequentially, the Golos brand seems unaffected by efforts to paint 
them at tools of the US and other foreign powers. In part these results are both likely to be 
consequences of the relatively small number of people who are familiar with Golos’s work—
both positive and negative associations are limited. 
In terms of access, 64 percent of “Golos”-prompted respondents thought that monitors should 
have free access, compared to 56 percent of “Neutral”-promptees, though there was no statis￾tical difference in overall positive responses (87 percent for “Golos” and 84 percent for “Neu￾tral). Negative attitudes to observers were somewhat less common among those who received 
the “Golos” text—only 7 percent said observers should be strictly regulated or banned, while 
the proportion for the “Neutral” group was 11 percent. Similarly, there was marginally more 
trust in election observer reports among those prompted with information on Golos than with￾in the unprompted group. For the “Golos” group 57percent said they trusted election observer 
reports at least somewhat, while 51 percent said so in the “Neutral” group. With regard to the 
effect of election observers, again responses among those receiving the “Golos” frame were 
marginally more positive than among those receiving a “Neutral” frame—63 percent and 60 
percent respectively felt observers made elections at least somewhat more free and fair. Conse￾quently, we can conclude that the Golos brand in itself may improve very slightly attitudes to 
observers, and that it does not by itself have a negative effect. 
MEDIA EFFECTS 
Finally, it is important to bear in mind that Golos activities have been taking place in an increas￾ingly hostile political context as state sponsored and allied media have stepped up attacks on 
Golos. The attacks have primarily focused around charges less of incompetence than of political 
bias and being agents of a foreign power. Charges that Golos’s evaluations are preconceived ide￾as ordered up by foreigners, particularly the United States are common. The data presented in 
the previous section, however, suggest that the impact of these allegations is limited. Most of 
the SPARC respondents show high levels of trust for and faith in the effects of elections observ￾ers, even when Golos is mentioned by name. In this section, we probe more deeply into the 
effects of the Russian state’s media offensive against Golos in the period of the elections. 
To assess the impact of the Russian government’s charges, we introduced two different ele￾ments to the experimental set-up. In wave 1, one quarter of respondents received a text 
adapted from a Russian tabloid alleging that Golos, in addition to monitoring and criticizing elec￾tions, received financial support and instructions from the US Department of State (GosDep). In 
wave 2, we randomly assigned half of the respondents to watch two minutes of video from the 
Russian TV channel NTV making similar allegations. 
GOLOS IMPACT EVALUATION—FINAL REPORT D-5 
The results show almost no effect for those receiving the text framing Golos as foreign agents, 
but quite strong effects for those exposed to the television coverage. Fewer respondents receiv￾ing the GosDep framing felt that observers should have free access to polling stations than any 
other group (52 percent). However, this is still a majority of respondents, and when combined 
with those who think light regulation is appropriate, the total reached 83 percent, about the 
same as the proportion of those who received a “Neutral” frame. Similarly, the GosDep framing 
made little or no difference to trust in observers (49 percent reporting at least some trust) or 
to views of the observers’ efficacy (62 percent still thought that they made at least some positive 
difference). 
By contrast, respondents who were exposed to the video offensive on Golos were much more 
influenced in their responses. Compared to the control group in wave 2, there was a 15 point 
reduction in the proportion of respondents who thought observers should have free access to 
polling stations—46 percent compared to 61. However, even here the impact was somewhat 
limited in the sense that almost all of the difference is accounted for by a shift towards the view 
that there should be light regulation—35 percent among the NTV compared to 23 percent in 
the control group. 
The proportions believing in strict regulation or banning of observers was unchanged. Trust in 
observer reports was quite significantly reduced by exposure to the NTV footage. Only 36 per￾cent of this group reported completely or somewhat trusting observer reports compared to 48 
percent in the control group. Part of the difference is accounted for an increase in those who 
neither trust nor distrust (41 percent compared to 34 percent in the control group), but here 
there was an increase in distrust with 17 percent of respondents reporting some or complete 
distrust, relative to only 11 percent in the control. 
There was also a slight decrease in belief in the efficacy of observers (the proportion thinking 
they make at least some contribution to more free and fair elections fell from 56 percent in the 
control group to 48 percent. There was also a sizeable and significant increase in the proportion 
of respondents who thought foreign contributions to domestic monitoring organizations should 
be banned or tightly regulated (74 percent compared to 62 percent in the control group). 
Probing attitudes to observers further, we asked respondents in wave 2 who they thought had 
funded the unprecedentedly large number of observers who turned out to supervise the March 
presidential elections. This question was asked of all respondents with 608 answering in the con￾trol group and 610 after watching the NTV video. Here again differences between the two 
groups were marked (see Table 6). While in the control group half of all respondents thought 
that private Russian citizens paid most of the expenses of observers. Only 35 percent of those 
who had watched the NTV footage agreed. The other big difference was in the proportion who 
felt the protesters were paid for by ill-intentioned foreigners. While in both groups the propor￾tion who felt the observers were paid for by foreign governments who want to support free and 
fair elections was essentially the same (10 and 11 percent respectively), in the NTV group 23 
percent believed that the observers were mostly funded by foreign governments who wanted to 
destabilize Russia—this was nearly double the 13 percent in the control group who thought the 
same. Similarly, when asked why people had volunteered to monitor elections, the two groups 
gave almost identical answers except for the option “because they were paid to do by foreign 
GOLOS IMPACT EVALUATION—FINAL REPORT D-6 
governments” – for this question, 21 percent of those who had seen the NTV clip agreed with 
this suggestion, while only 11 percent of the control group agreed.19 
CONCLUSION: FOREIGN INFLUENCE, ELECTION MONITORS AND IN￾FLUENTIAL RUSSIAN OPINION 
What should we conclude from these internet survey experiments about trust in Golos and 
other election monitors, and the effect of foreign (specifically US) financial support on that trust? 
The preceding data allows us to draw some rather nuanced conclusions about what is certainly a 
complex and interesting picture. The lessons can be summarized as follows: 
 It is clear that, at least by the time of the presidential elections in March 2012, educated, 
urban Russians had strongly positive attitudes towards election monitors. 
 These positive attitudes extended to wishing observers to have largely unfettered access 
to polling stations, to trusting monitors reports and to thinking that monitors can have a 
positive effect on elections. 
 
 We find that this support also extends to observers from the OSCE, though foreign ob￾servers as a whole are quite a heterogeneous group and are not all thought of as being 
the same. 
 
 SPARCs are, however, quite skeptical of foreign funding of Russian observers, though at￾titudes may be softening over time. 
 Within the various observer groups, the effects of Golos as a “brand” are difficult to 
pick up. Responses did not change significantly when Golos was mentioned specifically, 
in either a positive or a negative direction. 
 Nevertheless, there is some evidence that the Russian government’s attacks on Golos 
can have an effect on attitudes to observers, especially attacks on TV. Respondents ex￾posed to anti-Golos propaganda videos do exhibit significantly more negative attitudes 
than those not exposed (Work Plan Hypothesis 11). 
 
 
19 This result is not an artifact of people who believe foreign governments paid monitors being more likely to watch 
NTV and have seen the video before. Only 3 of the 64 respondents who agreed that foreign money was at play had 
previously seen the video on TV or the internet. 
GOLOS IMPACT EVALUATION—FINAL REPORT D-7 
Table 1: What kind of access should observers have to polling stations? 
Treatment Free Access Lightly 
Regulated
Strictly 
Regulated Forbidden
Don’t 
Know/Won’t 
Answer
Control (R2) 61 23 8 3 5
Neutral (R1) 56 28 7 4 6
Golos (R1) 64 23 6 1 7
GosDep 52 31 8 3 7
NTV (R2) 46 35 9 3 7
OSCE (R1) 64 23 5 1 7
Table 2: To what extent do you agree that elections are more free and fair when 
observers are present? 
Treatment Strongly Somewhat
Neither 
Agree nor 
Disagree
Somewhat 
Disagree
Completely 
Disagree
Don’t 
Know/Won’t 
Answer
Control 
(R2) 20 36 29 5 4
7
Neutral 
(R1) 20 40 23 8 4
5
Golos (R1) 25 38 24 7 2 4
GosDep 23 39 23 7 4 4
NTV (R2) 17 31 34 9 4 5
OSCE (R1) 24 36 29 4 2 4
Table 3: Trust in Observer Reports 
Treatment Completely 
Trust
Somewhat 
Trust
Neither 
Trust nor 
Don’t 
Trust
Somewhat 
Distrust
Completely 
Distrust
Don’t 
Know/Won’t 
Answer
Control 
(R2) 13 35 34 7 4
8
Neutral 
(R1) 11 40 35 7 4
4
Golos (R1) 15 42 32 7 2 3
GosDep 11 38 34 9 4 4
NTV (R2) 8 28 41 11 6 7
OSCE (R1) 14 35 38 6 3 4
GOLOS IMPACT EVALUATION—FINAL REPORT D-8 
Table 4: Since the Duma elections, several changes in the electoral law have been 
considered. How do you think each of the following should be regulated? 
Treatment Freely Lightly 
Regulated
Strictly 
Regulated Forbidden
Don’t 
Know/Won’t 
Answer
Control (R2) 36 23 31 4 6
Neutral (R1) 29 18 39 8 7
Golos (R1) 35 19 36 6 4
GosDep 30 19 39 7 5
NTV (R2) 34 19 39 4 4
OSCE (R1) 36 18 36 3 8
Table 5: Foreign Financing of Election Monitoring Organizations 
Can foreign governments give money to domestic election monitoring organiza￾tions? 
Treatment Freely Lightly 
Regulated
Strictly 
Regulated Forbidden
Don’t 
Know/Won’t 
Answer
Control (R2) 13 16 26 36 9
Neutral (R1) 9 13 26 44 9
Golos (R1) 10 12 23 47 9
GosDep 9 10 29 45 7
NTV (R2) 6 14 30 44 7
OSCE (R1) 9 11 26 44 12
GOLOS IMPACT EVALUATION—FINAL REPORT E-1 
ANNEX E: EVALUATION STATE￾MENT OF WORK 
SECTION C: DESCRIPTION SPECIFICATIONS/STATEMENT OF WORK
Democracy Surveys in Russia and the Middle East 
I. PURPOSE 
Understanding citizen attitudes toward democracy in general and toward specific proposed changes in a 
political context is essential for host country governments as well as U.S. policymakers and development 
professionals. Host country governments require reliable data for use in shaping the responsive govern￾ments they desire in times of great change and the U.S. government requires sound data to shape pro￾grams and implement policies that understand and respond to citizen attitudes. The purpose of this pro￾ject is to provide a statistically rigorous measure of public opinion to help gage attitudes and understand 
popular demand through large, random sample surveys is critical to the democracy, human rights and 
governance programs of the U.S. Agency for International Development and U.S. Government. 
This project will be divided into two discrete activities, one focusing on Russia and the other on select 
countries of the Middle East including Egypt, Tunisia and Morocco. 
II. RUSSIA IMPACT EVALUATION 
A. PURPOSE 
USAID’s Bureau for Democracy, Conflict and Humanitarian Assistance is soliciting the services of a con￾tractor to conduct an evaluation of select components of an approximately $9.2 million portfolio of U.S. 
Government-funded electoral and political process programs, including programs related to Russia’s par￾liamentary and presidential elections in December 2011 and March 2012, respectively. The evaluation 
shall feature a quasi-experimental impact evaluation design, and utilize mixed methods to complement 
the impact evaluation and conduct a performance evaluation. The USG bas supported elections-related 
programs in Europe, Eurasia and Central Asia for two decades, but has rarely used rigorous methods to 
independently evaluate the impact of these programs. Under the Obama Administration, the U.S. Gov￾ernment has identified improving the quality of evaluation of democracy and governance programs in 
Russia as a priority. This evaluation of elections-related programming will be consistent with that guid￾ance and help guide future elections and political process programs in Russia -- and potentially other 
countries. 
Evaluation of elections-related programs in Russia must be conducted with a clear understanding of the 
considerable constraints on programs in this sector, which will require a realistic set of expectations 
directed to reasonably set parameters for performance results, and precautions in attempting to gener￾alize regional results to the nation as a whole. A number of factors will influence the design, selected 
methodologies and evaluation modalities, including the state’s expressed policy of “managing” the politi￾cal system and political competition; suspicion toward external donors and foreign-funded NGOs; cur￾rent limitations placed by state actors and organizations on independent political activity; and prevalent 
political attitudes by important stakeholders which do not favor representative democracy. In addition, 
GOLOS IMPACT EVALUATION—FINAL REPORT E-2 
juxtaposed against the considerable resources of opponents of democratic liberalization, are the rela￾tively modest levels of resources available for programs. 
The overall deliverable under this Statement of Work (SOW) will be a comprehensive report to pro￾vide information for USAID, the Department of State, implementing partners and others related to the 
performance of select USG elections-related programs. The evaluation report should present a set of 
findings and recommendations for future electoral and political processes related programming in Russia, 
as well as any inferences that might be drawn for such programming in similar environments. The evalua￾tion's findings will also improve the U.S. Government’s understanding of how best to conduct monitor￾ing and evaluation activities of elections-related programming. 
B. BACKGROUND 
U.S. Foreign Policy in Russia 
The U.S. Government has historically sought to engage the Russian Government to pursue policy goals 
of common interest while in parallel engaging directly with Russian society to promote our economic 
interests, enhance mutual understanding between our nations, and advance universal values. As an im￾portant focus of these efforts, the United States works to support Russia in becoming a more democrat￾ic and open partner that increasingly moves towards a free-market, democratic system built on checks 
and balances, while protecting and promoting the principles of universal human rights. U.S. priorities 
include encouragement of accountable and participatory democratic political institutions, an active civil 
society, independent media, and the rule of law. 
U.S. Government Democracy and Governance Programming in Russia 
In support of this policy, the majority of bilateral U.S. foreign assistance in Russia is targeted to advance 
democratic development. This includes programs focused to help Russians increase civil society devel￾opment and civic participation, including in the political process; strengthen independent media and ac￾cess to information; reduce corruption; bolster the rule of law; promote human rights; increase elec￾toral transparency; and encourage transparent, accountable and participatory local governance. These 
programs are leveraged by activities carried out by organizations which serve as a legacy of U.S. assis￾tance, including the U.S.- Russia Foundation for Economic Advancement and the Rule of Law as well as 
the New Eurasia Foundation. 
U.S. Government Elections and Political Process Programming in Russia 
Over the last decade U.S. Government-funded technical assistance programs contributed to the 
strengthening of a domestic election monitoring network, encouraged public debate and assisted Rus￾sian-led initiatives attempting to uphold and promote fair and transparent political processes in Russia. 
With powerful special interests working against these processes, the upcoming parliamentary and presi￾dential elections will serve as an important bellwether to mark the trends in Russia’s future democratic 
development. Previous elections have been characterized by Russian organizations and much of the in￾ternational community as seriously flawed, and democratic opposition parties continue to face significant 
barriers to competition. However, there are growing opportunities for citizens to participate in advanc￾ing the country towards freer and fairer elections, particularly due to the advent of new information and 
communication technologies which are increasingly helping organizations to monitor, document, report 
and engage in dialogue on national issues of public interest, including electoral and political processes. 
Within this context, the objectives of U.S. Government programs related to the upcoming elections in 
2011 and 2012 will be to: 
GOLOS IMPACT EVALUATION—FINAL REPORT E-3 
 Promote civic participation and awareness of electoral issues and processes, e.g., increased 
knowledge and changed attitudes; 
 Ensure oversight and monitoring of the elections; 
 Increase the quantity and quality of independent media coverage and electoral discourse; 
 Strengthen rule of law and accountability in the electoral process; and 
 Enhance national initiatives to strengthen political pluralism. 
USAID/Russia has awarded several grants and cooperative agreements in support of electoral initiatives 
related to the upcoming State Duma and presidential elections in 2011-2012. 
Some of these grants and cooperative agreements are primarily focused on sub-sectors not directly re￾lated to elections, such as anti-corruption and independent media, but include some election-related 
activities as described below. The first two in the list below are the primary focus of this evaluation: 
Observation: USAID/Russia’s Agreement no. 118-A-0010-00070 with the Foundation 
GOLOS, a Russian non-governmental organization, has three main goals: 
(a) Institutional Development. GOLOS will expand the role and scope of responsibility of its Interre￾gional Foundations. GOLOS will also concentrate further on developing the organizational improve￾ments, its PR policy and its volunteers and activists. In addition, and related to USAID efforts to pro￾mote Russian civil society organizations moving towards adoption of ISO 9001type improved Manage￾ment System Standards (MSS) to institutionalize total quality control for administration, management 
and operations processes; 
(b) Improvements of the Russian Electoral Practices and Legislation. For the upcoming Federal elections 
GOLOS is planning and getting prepared to use new methodology of elections monitoring which will 
bring the quality of data it collects and publishes to a new level. GOLOS will continue training in new 
regions on SBO (partial parallel vote tabulation methodology) and get prepared to utilize it in as many 
regions of the Russian Federation as possible, subject to funding possibilities. GOLOS will also continue 
to improve its long- term methodology and mobile observation during elections day. GOLOS will con￾tinue advocating for the draft Electoral Code that aims to address current problems of the Russian elec￾toral process. Golos will conduct a series of round table discussions in the regions involving Moscow￾based and regional experts, political parties, civic groups and electoral commissions. 
(c) Citizen Involvement at the Grass-roots Level. GOLOS will continue its program to involve citizens in 
constructive joint efforts with regional and local administrations and deputies to address issues that citi￾zens face in their daily lives. GOLOS expects to win citizens’ support at the grass-roots level, build bet￾ter contacts with local and regional decision-makers to facilitate its election monitoring efforts, increase 
its base of volunteers and activists, and improve its reputation as a credible independent civic group. 
A major focus of the Golos program during the 2011-2012 elections cycle will be organizing long-term 
and short-term election monitoring which includes both observation of political action leading up to 
election day and election day itself. GOLOS plans to provide short- term monitoring in 30 regions 
throughout Russia with USAID funding and an additional to regions with EC funding, for a total of 40 
regions. Approximately 3,000 observers will be trained for these elections. GOLOS intends to field 
long-term observers and implement Sample Based Observation (SBO) in 20 regions. This type of moni￾toring, which is oriented around a statistically representative sample of polling stations, allows the ob￾serving organization to extrapolate findings and trends across the sampled region and make broader as￾sessments about the process in that region. GOLOS will also operate election hotlines, conduct over 
four press conferences, and employ "new media" and Information Communication Technology (ICT) 
GOLOS IMPACT EVALUATION—FINAL REPORT E-4 
tools a Russian nongovernmental organization, has three main goals: to conduct long-term and short￾term election monitoring which includes both observation of political action leading up to election day 
and election day itself. 
Civic Oversight: Under USAID/Russia' s current Agreement (no. 118A-0006-00082) with Transparency 
International-Russia (TI-R) that will end in September 2011, TI-R engages with and empowers citizens to 
participate in monitoring and reporting of irregularities and the misuse of public resources using innova￾tive ICT tools to allow activists and concerned citizens to monitor, document and provide real-time in￾put to an interactive map of public official abuses or misuse of public resources for elections. Under a 
new agreement (pending negotiations) USAID/Russia plans to further expand TI-R’s election monitoring 
efforts, such as through an Internet-based portal to promote civic engagement and public debate about 
elections. 
Party Development and Monitoring: USAID/Russia awarded Agreement no. 118-A-0009- 
00078 to the National Democratic Institute (NDI) and Agreement no. 118-A-0009-00076 to the Inter￾national Republican Institute (IRI) in support of party development and election monitoring. In addition 
to providing technical assistance to GOLOS in election monitoring, NDI promotes activities to address 
civic advocacy participation and domestic election monitoring based on a peer-to-peer approach, expos￾ing Russian groups to best international practices in these particular areas. NDI and IRI will engage in 
multidisciplinary seminars for young leaders to build a network of citizens well-versed in democratic 
norms and able to advocate for democratic practices in public policy and processes. Both institutes also 
facilitate and support increased engagement and bilateral exchange between political and elected officials 
of the United States and Russia at the federal, state/regional, and local levels for joint discussion of sub￾stantive issues of governance, constituent outreach, and transparency. 
Under recently approved USAID/Washington Election and Political Process (EPP) supplemental funding, 
USAID/Russia authorized the following: 
 Expansion of the number of regions where Golos will conduct long-term observation to 48 (40 
regions with USAID funding and 8 regions with EC funding); the number of regions with short￾term observation will increase from 30 to 40; Statistically-Based Observation (SBO) on Election 
Day will be conducted in 20 regions instead of 12. 
 NDI will design and produce a comprehensive series of web-based multimedia election observa￾tion training modules, which will be accessible to the public on GOLOS’S website. 
Media and Public Engagement: USAID/Russia awarded Agreement no. 118-A-0004-00061 to the Founda￾tion for Information Policy Development (FIPD), a resource for media editors, reporters and journalists 
to engage in discussion and dialogue on how best to cover the political issues of the day. FIPD promotes 
the idea of participatory print media, especially in localities where newspapers may be the only source of 
communication between the citizens and authorities. New technologies available on the Internet, mobile 
platforms, and video reporting will be implemented by newspapers to establish immediate two-way 
communication channels with its audience that is currently impossible. FIPD will continue to host elec￾tion-related on-line discussions on their website, and educational seminars for regional media represent￾atives will address professional and unbiased coverage of electoral campaigns. 
Documentation and Public Information: Under Agreement no. 118-A-0011-00005 with the New Eurasia 
Foundation a limited set of interventions is planned for documentation of the election campaigns. Specif￾ically, video crews follow party candidates on campaigns to document how they may be more effective 
when more engaged with citizens on local issues. Similar documentaries by New Eurasia Foundation 
GOLOS IMPACT EVALUATION—FINAL REPORT E-5 
have highlighted successes of local candidates in overcoming significant obstacles and winning against all 
odds, when running a citizen-focused (as opposed to special-interests focused) campaign. 
C. METHODOLOGY 
The contractor will utilize a variety of data collection and analysis methods determined in part by data 
collection challenges including cost, time and issues of validity and reliability. 
Impact Evaluation: The contractor is required to conduct an impact evaluation of select Golos and 
Transparency International election-related activities. The main source of data for the impact evaluation 
will be from a series of three or four public opinion surveys conducted prior to the parliamentary elec￾tions, after the parliamentary elections and before the presidential elections, and after the presidential 
elections. Through the review of project agreements, work plans and reports, and from interviews with 
USAID and partner staff, the contractor will determine the project outcomes to be evaluated and asso￾ciated outcome indicators to be measured using surveys. The contractor shall then design a quasiexper￾imental impact evaluation to measure the outcome indicators. For the evaluation of Golos activities, the 
contractor will contract for surveys in 5-10 regions (or sub-regions-- the exact number and geographic 
unit to be determined during the design) in which Golos works (or has targeted), and in 5-l0 compari￾son/control regions. (Contractor should identify the means for selecting the comparison units such that 
they are as similar as possible to the target units). Sample size will likely be in the range of 600-700 re￾spondents per region. For the evaluation of Transparency International activities (which are national in 
scope without any geographic target regions), the contractor will use a quasi-experimental design, if 
possible using the same surveys employed for the Golos evaluation. This might be achieved by over￾sampling in those same regions to have a large enough sample size for TI’s intended target audience, 
such as Internet users or youth. Alternatively, or additionally, the contractor might oversample in a sep￾arate national survey to the same effect. The contractor is encouraged to utilize the most rigorous qua￾si-experimental design(s) appropriate for this task. 
The surveys will provide the primary data source for the impact evaluations, and will be conducted prior 
to the start of the parliamentary campaign (approximately October), immediately (i.e., within 1 week) 
after the parliamentary election, and immediately (i.e., within 1 week) after the presidential election. If 
within budget, and with approval of the COTR, another round of surveys could be conducted prior to 
the start of the presidential campaign (approximately January-February). 
USAID, in consultation with the Department of State (EURJACE), will provide key issues/areas and illus￾trative questions to the contractor at the time of the award. The contractor will also meet with 
USAID/Russia for input, and then develop a draft questionnaire in the Russian language, review for sur￾vey sufficiency, and test for operational efficacy. Focus groups will be used to inform the survey instru￾ment and help shape the survey questions. Outcomes must be linked to specific activities of the imple￾menting partners. Illustrative examples of outcomes to be evaluated include: 
 Citizen awareness of acts that constitute violations of the election law; 
 Citizen reporting of alleged violations of the election law; 
 Citizen awareness of reported violations of the election law; 
 Citizen awareness/acceptance of the role of non-partisan election monitors; 
 Citizen support for a new electoral code; 
 Citizen engagement in public debate about electoral processes; 
 Citizen awareness of voting rights/procedures; 
 Interaction between local electoral bodies and civic groups. 
GOLOS IMPACT EVALUATION—FINAL REPORT E-6 
In addition to the outcome indicator questions, the surveys will include other (non-impact) questions 
related to the activities or otherwise useful for informing programmatic and policy decisions for future 
elections-related programs. For example, questions might assess: 
 Public support for civil society oversight of elections; 
 Whether people plan to vote, or whether they report having voted; 
 Participation in other aspects of the political process; 
 Support for a new election code. 
The evaluation team shall create a sampling methodology based on best practices for statistically repre￾senting the population from which the sample is drawn. The contractor shall propose sample sizes, con￾fidence level, and expected confidence interval for the surveys in its proposal, but the final details will be 
completed by the contractor after the award is made and upon approval from the COTR. The final de￾sign should propose whether and how to examine differences among key demographic groups, such as 
urban populations, youth, and Internet users. To the extent possible the survey should utilize questions 
from other surveys previously fielded in Russia in order to extend the longitudinal base upon which to 
make comparisons. The Evaluation team shall also develop code books and instructions for interviewers 
and submit the final questionnaire with the evaluation design for USAID approval. 
The contractor will hire a Russian survey firm to conduct and assist with the design of the surveys. The 
Russian survey firm must have experience developing sophisticated sampling plans and in fielding regional 
and nationwide public opinion surveys that include questions about political preferences. The firm 
should have a strong record of objectivity, independence and professionalism. Experience in conducting 
surveys for donor-funded programs is highly desirable. The firm will be identified following the project 
award in coordination with USAID and upon approval of the COTR. The firm will be engaged by the 
contractor prior to arrival in country and will take direction from the evaluation team leader. 
The surveys will be supplemented \with field observations, key informant interviews and additional data 
collection on the activities of GOLOS and Transparency International during and after the parliamentary 
and presidential elections. 
The evaluation team and the Russian survey firm will meet with the Moscow offices of 
Golos and Transparency International, and with USAID/Moscow, within the first weeks of the task or￾der to obtain more detailed information about program activities and objectives for the purpose of iden￾tifying and finalizing impact indicators and the survey and focus group questionnaires. The implementing 
partners will be expected to provide self-collected monitoring data and other project documentation to 
the evaluation team. 
Performance Evaluations: In addition to the impact evaluation, the contractor shall use other evaluation 
methods to complement the impact evaluation and conduct a performance evaluation of Golos, TI, and, 
to the extent possible, other USG-supported election activities. A package of briefing materials related 
to USG elections programming will be made available to the contractor upon award of the task order. In 
addition to reviewing these background documents, the contractor will use implementer data, inter￾views, site visits and other pertinent information. The team will also draw upon the results of the impact 
evaluation surveys and focus groups, and, possibly, other existing survey research on public opinion in 
Russia regarding electoral processes and civic oversight. 
In addition to visiting the Moscow offices of Golos and Transparency International (and, if necessary, 
other USG election assistance partners), the evaluation team will travel to a limited number of activity 
sites as determined by the evaluation team leader in consultation with USAID for face-to-face key in-
GOLOS IMPACT EVALUATION—FINAL REPORT E-7 
formant interviews, direct observation, and discussions with local stakeholders to examine implementa￾tion of the implementer activities. 
The primary purpose of the performance evaluations will be to complement and supplement the impact 
evaluations of Golos and Transparency International, but will allow the evaluation team to draw addi￾tional findings and recommendations about program components that cannot be evaluated through the 
impact evaluation. 
D. TASKS 
The contractor will employ mixed methods, including quasi-experimental impact evaluation design, to 
evaluate elements of select USG election-related programming. In particular, the evaluation will focus on 
examining the electoral observation activities of GOLOS and the elections-related civic oversight activi￾ties of Transparency International, as described above. (More detailed program descriptions will be pro￾vided to the contractor at the start of the task order period). While other elections related program￾ming by other implementers may be addressed in the evaluation, they will not be a primary focus. The 
specific tasks are: 
1. Impact Evaluation: Conduct an impact evaluation of select Golos and Transparency International elec￾tion-related activities, primarily through the use of public opinion surveys using the methodology guid￾ance described below. 
2. Performance Evaluation: Conduct a performance evaluation of the election-related activities of Golos 
and Transparency International, and, if possible and in consultation with the COTR and USAID/Moscow, 
of other USG election-related activities. 
GOLOS IMPACT EVALUATION—FINAL REPORT F-1 
ANNEX F: DISCLOSURE OF ANY 
CONFLICTS OF INTEREST 
GOLOS IMPACT EVALUATION—FINAL REPORT F-2 
U.S. Agency for International Development 
1300 Pennsylvania Avenue, NW 
Washington, DC 20523