1 Evaluation of the iChange e-Petition Platform: Final Report 1 OCTOBER 2020 Submitted to: USAID/LAB/HESN USAID/GEORGIA Submitted by: Daniel Maliniak Assistant Professor, William & Mary Member, AidData Research Consortium dxmali@wm.edu Elizabeth H. Miller William & Mary ehmiller@email.wm.edu Elizabeth K. Sutterlin William & Mary eksutterlin@email.wm.edu 1 This report is submitted to USAID/USAID Georgia and USAID/LAB/HESN by AidData (AID-OAA-A-12-00096), headquartered at the William & Mary. We are grateful to the staff of USAID Georgia, especially Lina Panteleeva and David Stonehill, the GGI staff, in particular Tengiz Chumburidze, Mikheil Darchiashvili, Marika Gorgadze, Levan Samadashvili, and David Smith, and the Administration of the Government of Georgia, especially Giga Paichadze, for their cooperation and assistance throughout the evaluation. We are also recognize the research assistance Carolyn DeRoster, Deb Kornblut, and Fallon O’Malley. 2 Table of Contents Executive Summary 2 Evaluation Purpose and Questions 5 Project Background 5 Evaluation Methods and Limitations 7 Findings 9 Recommendations Error! Bookmark not defined.3 Annexes Error! Bookmark not defined. Acronyms API Application programming interface AoG Administration of Georgia CCE Centers for Civic Engagement CI Civic Initiative CiDA Civil Development Association CSI Civil Society Institute CSO Civil society organization DRG Democracy, Rights, and Governance EWMI East West Management Institute E&E Education & Encouragement GGI Good Governance Initiative GoG Government of Georgia GYLA Georgian Young Lawyers' Association IDFI Institute for Development of Freedom of Information IDP Internally displaced person KRDF Kakheti Regional Development Foundation LGBT Lesbian, gay, bisexual, and transgender MP Member of Parliament NGO Non-governmental organization NLP Natural language processing OSGF Open Society Foundation - Georgia PD Project Description STM Structural topic modeling TI Transparency International 3 Executive Summary This report outlines the findings of an evaluation of the iChange online petition platform. In the spring of 2015, Dr. Daniel Maliniak traveled to Tbilisi to be part of the co-creation activities for elements of Georgia’s E-Governance tool for citizen engagement, iChange. The tool would be developed as part of Georgia's Good Governance Initiative, a five-year, USAID-sponsored program that aimed to “strengthen transparency and effectiveness of governance by improving public administration and policy making, increasing civic engagement, and strengthening parliamentary oversight.” iChange was considered a key piece of the broader GGI program, as it would allow both citizen engagement and the opportunity for the government to provide feedback. In addition to the petition portion of the platform, plans included tools for discourse and discussion between citizens as well as direct communications between the government and citizens that would be implemented in the future. The original goal of the evaluation was to determine the effectiveness of various aspects of an education and encouragement (E&E) campaign focused on increasing engagement with the online petition platform iChange. The rollout of the campaign was to be closely coordinated with the evaluation team to allow for a rigorous impact evaluation. Due to issues in the rollout of the iChange platform outside the control of the evaluation team, the scope of the evaluation and research question changed. We provide analysis to address the question: Why is it that some e-petition platforms receive more engagement than others? We include analysis of iChange compared to Manifest.ge, a platform developed by civil society, as well as the Tbilisi municipal platform, Idea Tbilisi, (შენი იდეა ქალაქის მერს). These two platforms have overlapped in existence with iChange, and have seen variation in citizen engagement. These three platforms vary in the types of petitions that qualify as germane and in the number of signatures required to deem a petition successful. While our analysis is limited in providing a definitive answer to this question, we rule out some explanations for differences in citizen engagement. The original plan involved a careful evaluation of the education and encouragement campaign, utilizing geospatial data from where these activities took place and matching them with data on where petition authors and signatories were located. In late summer of 2015, the Mission reported the Government of Georgia’s (GoG) reticence to enact the legislation prior to an upcoming election. Additional delays pushed back the launch date significantly. The changes in the launch dates made the agreed timeline in the original PD obsolete. A number of additional iterations with new timelines were agreed upon, but again made obsolete due to continued delays in the launch dates. During this time, MPs and the AoG worked on additional changes to the iChange plan. These amendments included an increase in the number of signatures from 5,000 to 10,000 to consider a petition successful and require a response from the government. The AoG also decreased the window of time in which the petition could collect signatures. Since the launch of the platform and the activities of grantees ran their course, a general look at the usage of the iChange platform up to the end of 2019 revealed an underwhelming performance. The average petition received 0.3% of the required signatures, with only one petition barely breaking the 5% threshold. The number of petitions on the platform remains low. In the time between then and now, one petition did receive the requisite number of signatures, but it is by far the outlier. It was clear that an analysis based on the original PD, even were it to overcome the data and logistical constraints, would not be very helpful in explaining variation in usage of the platform. We utilize a multi-method approach combining qualitative interview data with advanced statistical analysis of the petitions themselves. Our qualitative interviews produced a number of potential explanations for the lack of engagement with iChange. Many interviewees had more than one factor they believe contributed to the failure to gain consistent users. Some of the arguments placed 4 the blame on changes made to the number of signatures required for a petition to be successful and the shorter window to gather signatures. Other arguments focused on the constraints to what petitions were deemed appropriate by moderators for iChange, and that those who submitted petitions that did not have the proper scope became disillusioned with iChange. Our quantitative method relies on using the petitions as data. We collected data on every petition successfully submitted to the three e-petition portals over the course of their existence, including the number of signatures, when the petition was created, and the text of the petitions. Using natural language processing tools, we are able to categorize the petitions to allow for better comparison across petition platforms. We then present a variety of descriptive results about the petitions to give a sense of what Georgians are interested in. We validate the topics used to categorize the petition and then discuss how these topics vary across the different platforms. We then present evidence suggesting that the topics themselves do not seem to be the limiting factor for iChange, and that some of the concerns about people posting topics outside of iChange’s mandate seem overblown. Our results do suggest that users likely skew younger. Based on the findings from our evaluation we have a number of recommendations. First, that iChange’s effort to police topics was misplaced. Data from Manifest suggests that in the overwhelming majority of cases, people will still discuss serious political and economic issues even with less supervision. Rather than strictly policing the content that is allowed to move to the voting stage on iChange, future administrators of the program could do more to provide feedback and assistance to those drafting iChange petitions. Second, personalized feedback aimed at helping users to improve their submissions could go a long way in terms of both increasing engagement with the platform and reducing the share of submitted petitions that are ultimately rejected. Third, while work was put into an E&E campaign about how to utilize the iChange platform when it was first launched, such one-off trainings did not reach a large enough base of users to generate public knowledge and excitement about the platform. Outside of the civil society sphere in Georgia, few people the interviewers interacted with day￾to-day had heard of the platform at all. Continuous efforts to work with petition authors to improve and develop their ideas would encourage rather than discourage new users, who are key for sharing their petitions and thus spreading knowledge of iChange organically among their own social networks. Fourth, future efforts should work to improve organic, "word of mouth" knowledge of the iChange platform to make it easier to publish, sign, and discuss petitions with one's network. Manifest stands out from Georgia's government-operated petition sites with its use of Facebook plugins, making it possible for users to leave comments using their Facebook profile without even leaving Manifest, and enabling users to share a petition with just a click. Given the immense popularity of Facebook in Georgia, this feature has proven to be a powerful tool for driving more people to the site to sign petitions with ease. 5 Evaluation Purpose and Questions The purpose of the evaluation and the questions it is meant to answer have morphed over the life of the project. The original goal of the evaluation was to determine the effectiveness of various aspects of an education and encouragement (E&E) campaign focused on increasing engagement with the online petition platform iChange. The rollout of the campaign was to be closely coordinated with the evaluation team to allow for a rigorous impact evaluation. Due to issues in the rollout of the iChange platform outside the control of the evaluation team (detailed below), the scope of the evaluation and research question changed. To provide the most value to the Mission and to future efforts related to e￾governance of this type, and e-petition efforts overall, we use the case of iChange’s successes and failures embedded in the broader e-petition space in Georgia. We provide analysis to address the question: Why is it that some e-petition platforms receive more engagement than others? We include analysis of iChange compared to Manifest.ge, a platform developed by civil society, as well as the Tbilisi municipal platform, Idea Tbilisi, (შენი იდეა ქალაქის მერს). These two platforms have overlapped in existence with iChange, and have seen variation in citizen engagement. These three platforms vary in the types of petitions that qualify as germane and in the number of signatures required to deem a petition successful. While our analysis is limited in providing a definitive answer to this question, we rule out some explanations for differences in citizen engagement. Project Background In the spring of 2015, Dr. Daniel Maliniak traveled to Tbilisi to be part of the co-creation activities for elements of Georgia’s E-Governance tool for citizen engagement, iChange. The tool would be developed as part of Georgia's Good Governance Initiative, a five-year, USAID-sponsored program that aimed to “strengthen transparency and effectiveness of governance by improving public administration and policy making, increasing civic engagement, and strengthening parliamentary oversight.”2 These initial meetings were held in conjunction with members of the Global Development Lab at USAID. The concept of iChange came about through input and suggestions from civil society organizations (CSOs) in Georgia. iChange was considered a key piece of the broader GGI program, as it would allow both citizen engagement and the opportunity for the government to provide feedback. In addition to the petition portion of the platform, plans included tools for discourse and discussion between citizens as well as direct communications between the government and citizens that would be implemented in the future. The initial PD was signed in May 2015. That document reflected conversations and compromises between David Stonehill at the Mission, David Smith and Tengiz Chumburidze at GGI, Giga Paichadze at the Administration of Government (AoG), and various representatives of the Georgia Young Lawyers Association (GYLA) consortium of CSO grantees. The original PD outlined a 2-3 year evaluation of the iChange platform using the Democracy, Human Rights, and Governance (DRG) survey that was believed to have many of the variables of interest for the evaluation, and would be flexible enough to include relevant future questions. Many of the variables in these data were meant for the specific purpose of comparing information on users and petition creators, along with their geographic location, with efforts of grantees in educating and encouraging members of the public to take advantage of iChange. Ideally, 2 https://www.tetratech.com/en/projects/georgia-good-governance-initiative 6 the rollout of the campaign would provide insights on what communities were actually using the platform when receiving the campaign compared to those groups who either did not receive the campaign, or those who were not targeted. Particular care was taken to address differences between men and women, as well as ethnic minority communities and IDPs within Georgia. In late summer of 2015, the Mission reported the Government of Georgia’s (GoG) reticence to enact the legislation prior to an upcoming election. Those in the AoG were hopeful that the passage of necessary legislation to authorize the iChange platform would come soon after the election. These initial delays compounded with objections by members of parliament and other political considerations. Additional delays pushed back the launch date significantly. As negotiations about passing the necessary legislation to enact iChange became protracted, additional breaks in the work dragged on. The changes in the launch dates made the agreed timeline in the original PD obsolete. A number of additional iterations with new timelines were agreed upon, but again made obsolete due to continued delays in the launch dates. During this time, MPs and the AoG worked on additional changes to the iChange plan. These amendments included an increase in the number of signatures from 5,000 to 10,000 to consider a petition successful and require a response from the government. The AoG also decreased the window of time in which the petition could collect signatures. Over this time, there were also a number of false starts on a baseline report. These drafts were paused when it became clear that the launch of the platform was not imminent, and for that reason, new survey data, for a later round of the DRG survey— would likely be available for a baseline close to the launch. For these reasons, an official baseline analysis was never submitted because delays in the launch made subsequent analysis obsolete. The website for iChange finally launched in 2017. At that time, grantees rapidly deployed their plans for the E&E campaign. A number of the people involved in the initial conversations about data collection to aid in the evaluation moved on from their positions, and understandably, maintaining those requirements was not a top priority. The Mission informed Dr. Maliniak that the DRG survey may not run, and without knowing what data would be available for the endline, there were practical concerns with designing the baseline analysis. Moreover, contacts at the AoG expressed new concerns about what data they had or could share about users (petition creators and signatories), making promised analyses in the PD impossible to conduct. For instance, the AoG determined that sharing the geographic breakdowns and demographics of users was no longer feasible and potentially illegal, despite an earlier agreement. The Mission made it clearer there was neither additional funding nor the will to run an additional survey to recover some of the aspects of the original PD. Since the launch of the platform and the activities of grantees ran their course, a general look at the usage of the iChange platform up to the end of 2019 revealed an underwhelming performance. The average petition received 0.3% of the required signatures, with only one petition barely breaking the 5% threshold. The number of petitions on the platform remains low. In the time between then and now, one petition did receive the requisite number of signatures, but it is by far the outlier. It was clear that an analysis based on the original PD, even were it to overcome the data and logistical constraints, would not be very helpful in explaining variation in usage of the platform. Given these structural limitations on the initial plan and questions, Dr. Maliniak suggested a revised plan in 2019 with more modest goals. In particular, the lack of success in the usage of iChange stood in contrast to the relative success of a municipal government-run platform, Idea Tbilisi, and of one created by civil society, Manifest.ge. With a team of researchers who were in the country in the summer of 2019, Dr. Maliniak and his team collected large amounts of publicly available data from all three platforms and conducted interviews with people involved in the iChange E&E campaign. A new PD was 7 agreed on whereby Dr. Maliniak and his team would address the aforementioned question using a mix of quantitative and qualitative analysis. Evaluation Methods and Limitations Due to the changes listed above, our analysis takes two complementary approaches. The first was qualitative interviews with parties involved in the campaign. Our interviews took place from June through August 2019 in Tbilisi, with a few interviews conducted via phone call when meeting in-person was infeasible (this was primarily an issue when we spoke to participants from regions far from Tbilisi, or when interviewees were only available to speak after the interview team had returned to the U.S. A full list of interview participants is provided in Annex 1. All but two interviews were conducted in English, while the remaining two were conducted in Russian for participants who lacked sufficient English knowledge. These interviews are more meant to give context to some of the reasons why the platform seemed to be ineffective, as many of the individuals had experience in the field. Interviews are also helpful in determining where participants believed the failures occurred, and to focus our attention to mentioned factors in the data analysis. These interviews had the goal of understanding what aspects of the E&E campaign were successful or not in the eyes of those involved with the rollout of the iChange platform. In determining why some platforms are more successful than others are, the views of individuals involved help to shape some of the arguments we sought to test using the data. Our qualitative strategy comes with limitations. First, some key figures in the process had moved on to other positions or were unavailable to us for interviews. This was true mostly in the CSO community, but we were still able to meet with members of key players, even if they were not involved with the initial discussions. Second, given the larger failure of the E&E campaign, there is a clear potential that those involved wanted to present explanations for the broader lack of success that were unrelated to aspects of the project where they were involved. We saw some evidence of this in our conversations. Some interviewees took a defensive or combative tone, suggesting they were concerned about how an evaluation of their role in the E&E could be seen as failing. Overall, we believe by talking to people at different levels and with different roles in the process, we have prevented this potential confound dominating the results overall. Some factors were consistently mentioned as decreasing engagement, like the higher threshold for signatures, but different interviewees were willing to offer critical evaluations of some of the actors in the process. A third related limitation of this method was that some of our interview sampling was done through recommendations and suggestions of previous interviewees. Some of our interviews then are subject to a selection bias that might similarly be meant to present a vision of the E&E campaign that paints our other actors in a better light. While we do not rule out this potential, most of our interviewees were frank with their assessment of the others with whom they worked in the project. The second piece of analysis used focuses on the petitions themselves as data. To determine why one platform is more successful than another, the key measurement is the engagement with the platform through the petitions. However, the petitions themselves may also vary between platforms, so comparing engagement is not straightforward. Petition creators assign a category to their petition upon submission. One option would be to compare petitions in the same category across platforms, but in many cases, the exact categories were not consistent across websites. Moreover, while all three platforms allow individuals to categorize their petitions, reported categories may not represent the actual topic of the petition. For example, does a petition to rename a street belong in the "Road Infrastructure" category or the "Cultural, Sport, and Youth Issues" category? Does a petition about reducing emissions from taxi cabs belong in "Transportation," or "Environmental Protection," or "Health 8 and Social Affairs?" On Manifest, where there were no moderators to enforce whether petitions belonged in the selected category, the user was left with what appeared to them an inconsequential choice. Given each platform's different rules around what issues were permissible and relevant, two petitions in the same category on different platforms could represent dramatically different issues (e.g. a petition on Idea Tbilisi about installing dust filtration systems in kindergarten buildings to protect children's health3 vs. a petition on iChange to relocate an open air landfill further from populated areas4 ), as well as differ drastically within topic (e.g. a petition to create a dog park in Vake for smaller dogs5 vs. a petition about reduction or elimination of interests rates during the pandemic6 both being assigned to “Social Issues - სოციალური საკითხები”). Because any difference in success and public engagement with these platforms could be the result of whether or not Georgians believe their issue is germane to a specific platform, we need to consider what role the content of the actual petitions plays. Rather than evaluate each petition by hand, we use a machine learning method, topic modeling, to analyze the petition text and identify common topics across all three platforms. Topic modeling is a growing method for evaluating text data that reveals latent topics through statistical evaluation of word and document co-occurrences within a corpus of documents. It is particularly well suited to tasks like this where there exists a relatively large corpus of text data, and the need to categorize documents by the words and phrases used. We collected a total of 2779 petitions, 1264 from Manifest, 114 from iChange, and 1371 from Idea Tbilisi. Our data from Manifest ranged from November 2014, when the platform was launched, through July 2019. Our data from Idea Tbilisi ranged from February 12, 2018, when the platform was launched, to June 2020. Our data from iChange also ranged from the platform's launch until June 2020.7 Along with the raw text of the petition, we collected the name of the petition author, the title of the petition, the date of submission, the number of signatures the petition received, and the number of signatures required for the petition to “pass”.8 We performed some manual steps to clean the raw text of the petitions before analysis. This included removing tokens and phrases that would not pass cleanly into the topic model. These included emojis as well as links to YouTube, Facebook, and other websites. We utilized the STM package in R.9 While the software is agnostic to language and script, we make some unique adaptations to accommodate for petitions written in Georgian. Topic modeling has been generally optimized for English, as well as some other languages, but has not been widely utilized 3 https://idea.tbilisi.gov.ge/idea-details/394 4 https://ichange.gov.ge/13375 5 https://ichange.gov.ge/13410 6 https://ichange.gov.ge/13400 7 We extended the period of time over which petitions were collected due to iChange's first successful petition from November 4, 2019. There is reason to expect that a signature that garnered 10,000 signatures could have a spillover effect for other petitions on iChange; that once that many users had created accounts to sign this petition and seen its success, they would be more likely to return and add signatures to other petitions, thus boosting the numbers of more recent petitions. We updated the Idea Tbilisi data at the same time to maintain a fair comparison for the successes of a local vs. national petition platform over the same period of time, but we chose to leave Manifest alone due to the number of successful petitions we had already captured in our data through 2019. 8 In the case of petitions from Manifest, the creator of the petition could set the number of signatures required for the petition. It is not clear what would happen to a petition on Manifest if it met the requirement whereas with the government sponsored petition websites, the government was supposed to respond in some way if the threshold of signatures was met. 9 http://www.luigicurini.com/uploads/6/7/9/8/67985527/stmvignette.pdf 9 in Georgian. In order to create optimal topics within the model, we translated the petitions from Georgian into English before running the model. In order to prevent our mechanical translation from imposing meaning on the sentences, we translated words individually and ran a substitution algorithm to replace the Georgian word with its English translation using the Google Translate API. This operation allows the computer to more accurately complete the stemming and stopwords process and allows for the better development of topics.10 Stemming and stopwords are two important preprocessing steps in the creation of topic models. Stopwords are removed before topic modeling in order to provide clearer topics, as they do not generally add to the content of what is being said. We use a standard set of English “stopwords”, words such as “the”, “like”, and “this” which, while required for the structures of a sentence are not important for understanding the topic of a petition. We developed a similar list of Georgian stopwords from a list that was verified by a Georgian speaker. We compared the translations of frequent Georgian words in the dataset of words appearing in the petition with the standard English stopwords list in order to identify Georgian stopwords missing from the initial list. We also added as English stopwords: “will”, “please”, “signature”, “Georgia”, “Georgian”, “petition”, and “sign” as these appear too frequently in the petitions and provide little insight on the petition topics. After cleaning the data using these stopwords, we had 6135 terms and 134,195 tokens for the 2779 total documents. After the data are cleaned, we use natural language processing (NLP) and structural topic models (STMs) to identify latent topics within the petitions, instead of the preset categories authors could choose from. The topic modeling process identifies latent themes and topics across all the documents. The output is a matrix of scores of words to topics, as well as a matrix of topic by document (petition). After testing, we settled on 20 topics for this analysis.11 We are then able to determine what topics appear in petitions generally, as well as what topics appear in certain platforms over others. Findings, Conclusions and Recommendations Discussion of qualitative findings Our interviews took place from June through August 2019, and included meetings with representatives from fourteen organizations with varying levels of involvement in the iChange project. Some of our participants represented organizations that were directly involved in the development, implementation, and administration of the platform, while others were experts in Georgian civil society and democracy projects, who could provide an outside perspective on the project in context of other development programs in Georgia. Our semi-structured interview format ensured that all participants 10 Stemming is the practice of abbreviating words with common roots in order to help to define the topics. For example, educate, educator, and education are all stemmed to educat- because the words are all related to the same content and topic. Because of the nature of the Georgian language, stemming in Georgian is a very complicated process, and beyond the scope of this project. The process described above is consistent with other efforts to utilize these tools in less used languages. 11 There is no established method for determining the “correct” number of topics. Instead, we ran a number of models with different parameters and a different number of potential topics. While models with many more topics fit the data better in some sense, the results are unwieldy and unhelpful for analysis. With a smaller number of topics, we saw clear evidence that distinct topics were being lumped together. 10 were asked about the iChange project with the same priming, while providing interviewers with the flexibility to ask relevant follow up questions as necessary. These interviews provided us with a wealth of qualitative information about the factors at play impacting both iChange and e-democracy efforts in Georgia as a whole. From all of our interview notes, we were able to distill 11 hypotheses that participants felt had led to iChange's lackluster performance. It is important to note that participants were able to list as many reasons as they could, and were not limited to choosing one they felt was most important. Table 1 below summarizes these arguments and ranks them by how frequently they occurred in our notes (how many organizations cited them.) Reason for iChange failure Suggested by 1 Signature requirement of 10,000 was too high GGI, AoG, CSI ,IDFI, GYLA, CCE, EWMI, CiDA, KRDF, TI, training participant (11) 2 Familiarity with Manifest made people unwilling to switch platforms AoG, IDFI, CI, GYLA, EWMI, TI, CiDA, OSGF (8) 3 Excessive hurdles placed by government to sign up for iChange and post petitions (registration difficulty, long moderation wait times) IDFI, GGI, CI, GYLA, CCE, EWMI, TI, OSGF (7) 4 Lack of government and political leadership support for iChange and the E&E campaign GGI, IDFI, AoG, CiDA, GYLA, TI (6) 5 Low citizen trust in government-led programs generally Manifest, CI, EWMI, CiDA, TI, OSGF (6) 6 Insufficient time to gather signatures GGI, CSI, EWMI, GYLA, training participant (5) 7 The platform was "ahead of its time," civic activism is not yet fully a part of citizen life in Georgia CSI, CCE, CiDA, TI (4) 8 No "success story" driving more users to the platform KRDF, EWMI, CiDA, OSGF (4) 9 Lack of computer skills in remote regions/rural areas made many people too difficult to reach CSI, CCE, CiDA, TI (4) 10 Low citizen understanding of or interest in national issues CCE (1) 11 Insufficient E&E campaign TI (1) Table 1.Themes to explain lack of engagement with iChange emerging from interviews After consolidating this list of possible causes for iChange's performance, we attempted to find evidence for them in our dataset of petitions. There were a few causes, however, that we did not evaluate as closely. Almost every participant we spoke to mentioned the 10,000-signature count being 11 too high. The high signature count certainly played a role in the platform's lack of success, and it seems like an obvious answer. However, considering the high signature count alone is a surface-level explanation, especially when our data showed that multiple Manifest petitions gathered over 10,000 (and in a select few cases, over 20,000) signatures. Now that iChange has finally had a petition succeed in reaching the signature target, it is indisputable that online petitions in Georgia can achieve that level of popular support. We thus turn to other explanations of iChange's failures to understand why so many of its petitions did not. In addition to paying little attention to explanation 1, we also spend little time assessing explanation 6, which claimed that a one-month period was too little time to gather 10,000 signatures. A previous study of e-petition initiatives evaluated thousands of petitions in the United States and UK and found that there was a rapid decay in the rate of new signatures over time; researchers found that after a day or two, a petition's fate was "virtually set" and it became exponentially more difficult to garner new signatures.12 This pattern led us to discard explanation 6, as additional time to gather signatures beyond the thirty day iChange period would likely lead to very few, if any, new signatures. Finally, it is worth noting that the only party explicitly to criticize the amount of effort or quality of the E&E campaign was Transparency International, who were also the least involved in the E&E campaign. There are clearly ways in which the other arguments made could compound or potentially overcome the failures of a poorly executed E&E campaign. In this case, we cannot rule out that the lack of engagement with iChange was primarily the result of issues related to the delayed campaign. General Descriptives on Petitions After cleaning the data, our dataset consisted of 2779 petitions from the three petition sites. On average, each petition received 372 signatures or 12% of the “threshold” number, calculated by petition.13 However, 521 petitions, all from the Idea Tbilisi platform, received zero signatures. Excluding these petitions that received no signatures, the average number of signatures was 458 or 15% of the “threshold” number. One hundred ninety-eight petitions in the dataset received 100% of the “threshold” signatures or more. Two hundred ninety-three petitions received 50% or more of the “threshold” signatures. The average word count for petitions was 101 words. Looking at the petition word count by platform, seen in Figure 1, the petitions on iChange had a higher median word count than the petitions on Manifest or Idea Tbilisi, although the petition with the largest number of words was from Manifest. The higher median word count for iChange may reflect that we were able to include all petitions from 12 Yasseri, Hale, and Margetts. “Rapid rise and decay in petition signing.” (2017) https://epjdatascience.springeropen.com/articles/10.1140/epjds/s13688-017-0116-6 13 While the threshold number was standard for iChange and Idea Tbilisi, on Manifest the author of the petition could set the number. 12 Manifest and Idea Tbilisi, regardless of quality, while poorly written or extremely short petitions from iChange have been screened out by government moderation.14 Figure 1. Boxplot of Petition Word Count divided by Petition Platform Discussion of Topic Model Results Our use of a structural topic model allows us to present a variety of valuable and interesting statistics and analyses. The STM algorithm determines topics by looking for words that co-occur commonly with each other in the same documents. The matrix and word cluster topic model results group together the latent themes within the dataset of petitions, but the algorithm cannot qualitatively determine the topic. We are left to label and identify these topics manually based upon the words within them. We focus on words with high “FREX” scores, which are those most frequent to a given topic and exclusive to that topic. Figure 2 shows the frequency of the given topics out of the entire corpus (represented by the solid horizontal lines). The topics are relatively evenly distributed throughout the corpus, as every topic proportion is greater than one percent. The most prevalent topic generally focuses on Infrastructure, discussed below, and the least prevalent topic focuses on Labor issues. Other topic names were determined by looking at the FREX words and the petitions most associated with the topic. Importantly, we see words throughout that relate to a variety of issues covered across petitions of all sites, and clear 14 See figure 13 below and associated discussion for some evidence that moderation may play a role. 13 evidence that these categories are more specific than those assigned by users themselves at the time of submission. Figure 2. Topics labeled and ranked by topic proportion with associated high ‘FREX’ words15 15 Georgian words ‘ნინო’ and ‘მწერალი’ appear in the Domestic Politics topic. These words were not translated in the processing due to infrequency and threshold level factors in the vocabulary list creation process. Once translation occurred, they became included within the new translated vocabulary. Previous model attempts included more than one translation step in an attempt to minimize these words, but the results were time intensive and not significant in reducing some Georgian words appearing. Though infrequent, they are shown because they are very exclusive to the topic. 14 Discussion of Infrastructure Topic Figure 3. Infrastructure (Topic 15) by Platform Proportion Figure 4. Infrastructure (Topic 15) over Time After identifying latent topics within the petition data, we turn to how these topics vary across platforms and over time. We can think of the topics as representing the major issues on the minds of Georgians that rise to the level for petitioning the government, or even more importantly, are relevant in appealing to their fellow Georgians to band together to illustrate to the government the interest in an issue. For that reason, seeing what the population looks like can give insights into why iChange was relatively 15 sparsely used. If the types of transportation issues of interest to petitioners have more to do with crosswalks than highways, potholes than railroads, then the iChange’s lack of use may just stem from interest in other issues. To illustrate the outputs of our analysis, we start by discussing the topic we refer to as Infrastructure. Infrastructure is the most prevalent topic within the corpus of petitions. As seen in Figure 3, petitions in the Infrastructure topic are relatively more frequent on Idea Tbilisi, with a much higher and significant mean topic proportion than Manifest or iChange. This large difference is likely because Idea Tbilisi is by nature a local level petition platform, designed to receive petitions from residents about projects to improve the city. Infrastructure improvements on houses, buildings, and other city structures are a central focus of the Tbilisi platform, while users are less focused on this issue for iChange and Manifest, which collect signatures from across the country.16 Infrastructure projects are generally under local rather than national jurisdictions, which means some infrastructure projects may have been rejected outright by iChange moderators who discarded any petition that was outside a national scope. The slightly higher mean topic proportion for Infrastructure on Manifest suggests that the lack of moderation contributed to slightly more infrastructure related petitions, but that most users may have recognized that an unofficial website was not the most effective avenue for petitioning for infrastructural improvements in their local cities. We can also see that the prevalence of the Infrastructure topic, as seen in Figure 4, has been relatively stable over time. 16 While Manifest and iChange can gather signatures from the whole country, there is nothing stopping petitioners from directing their petitions to the municipal government, even in Tbilisi. Manifest allows you to write in your intended audience, so you could theoretically get local petitions on that platform. Moreover, some petitions on both Manifest and iChange frame their local issue in a national manner, like a petition to the government to repair a road destroyed in disaster (https://ichange.gov.ge/12235). 16 Topic Connectedness Figure 5. Topic Connectedness Web In addition to understanding which topics are prevalent, our methods allow us to determine what topics tend to co-occur. A single petition can cover multiple topics or issue areas. The way authors link topics might be due to a strategic choice on the part of those drafting petitions, to link one topic to another to garner more support. It may reflect how those writing petitions think about the interconnected nature of the issues they support. It may also be due to the inherent co-occurrences of certain issues---for example, a petition about homeless people living in a park would link social services & parks/green spaces topics. By understanding the way petition authors purposely and unintentionally interact these topics, we also gain broader insights on how citizens are using these platforms. Figure 5 shows how topics are interconnected using the petition data for all platforms. Topics that are connected frequently between petitions are grouped together with lines connecting related topics. We color topics that are similar thematically the same. The trends observed in the topic connectedness plot also help to reinforce the topic labeling as expected related topics are connected. For example, the “Youth” topic connects to Entertainment, Education, Tourism, and the Internet. Infrastructure is linked to Utility, Recreation, Landmarks (which includes a number of street names), Parks, Tourism, and Internet. Regulation is linked to a variety of other topics which likely call on changes in regulation. Labor, the least prevalent topic by expected topic proportion, is not connected to any other topic. 17 Topic variation across platforms Having established a sense of the overall ecosystem of the petition text, topics, and interconnectedness, we now turn to the variation across platforms. In our interviews, some parties attributed iChange’s failure to engage users to a lack of interest in national issues, and to the excessive burden placed on topic content was considered germane for the platform. We looked at variation across topics by platform to examine how frequently petition topics that were appropriate and allowed on iChange appeared in the data of other Georgian petitioning websites. Fig 6.Education (Topic 20) by Platform Proportion Fig 8. Financial (Topic 14) by Platform Proportion Fig 7.Healthcare (Topic 17) by Platform Proportion Fig. 9. Entertainment (Topic 6) by Platform Proportion 18 Petitions relating to Healthcare, Finance, and Education are all more prevalent by mean topic proportion on iChange than on Manifest or Idea Tbilisi, as shown in Figures 6 - 9 above. By their nature, these are topics we would expect to appear on iChange because they are generally relevant to the country and involve national-level legislation or attention to address. Entertainment has a high mean topic proportion on Manifest compared to the other two platforms. When officials created official e￾petitioning platforms, they likely did not want them to become bogged down with insufficiently serious petitions. Those in charge of moderating petition content took "joke" petitions extremely seriously, and as a result, there are far fewer entertainment-related petitions on iChange and Idea Tbilisi than there are on Manifest, which had little to no moderation of content. Perhaps contrary to officials' expectations, however, the high proportion of less "serious" topics on Manifest does not mean Georgians who use it did not take it seriously. The platform also has a higher mean topic proportion for Finance and Education when compared to Idea Tbilisi. While a lack of moderation allowed more unrealistic and lighthearted petitions through the platform, users were not dissuaded from publishing petitions regarding serious economic and social issues important to them. It may be the case that extensive policing of petitions on iChange lead to a decrease in the visibility of the platform. The high proportion of petitions with the Education topic on all three platforms has important implications for demographics using online petitioning in Georgia. Appeals to the nationwide Ministry of Education were frequently made on both iChange and Manifest about exams, textbooks, building schools, and other related issues. Typically, those who are most vocal about issues in education are parents of young students and adolescents currently in the school system. Not only are young adults seeing things they want changed in their education system, but also they are typically a highly active age group socially and politically. University students and other young people featured prominently in 2019 protests that erupted when a Russian MP gave a speech from the Georgian parliament. In addition, in the age of social media, young people are able to tap into their networks of classmates, friends, and families with a few clicks to generate change, whether through a petition or a protest. The prevalence of the Education topic across multiple platforms suggests that young people, through activism and higher technological knowledge compared to older generations, are talking about their priorities enough to shape the conversations on e-petition sites. 19 Topic and success by platform Figure 10. Petition Success Rate by Percentage out of ‘Necessary’ Signatures Received Ultimately, success of petitions to reach the required number of signatures is a key element in the success of the platforms. Our dataset of petitions confirmed the trend observed at the outset: compared to Manifest and Idea Tbilisi, not only did iChange have fewer petitions, but it had far fewer successful ones. At the time of writing, iChange has had only one petition succeed in reaching its signature count goal. When comparing the data on petition success across platforms, it is important to note that Manifest allowed users to set their own signature count goal when they created a petition, where iChange and Idea Tbilisi had fixed goals of 10,000 and 2,500 signatures respectively. This made it easier for petitions on Manifest to reach 100% of their goal if it was set lower. Irrespective of the signature count goal, however, Figure 10 above reveals that there was far less of a spillover effect on iChange than what occurred on Manifest or Idea Tbilisi. Other than the one petition that achieved its signature goal on iChange, almost all petitions remained stuck at close to 0% of their goal. No non-successful petition broke even the 20% of the goal for signatures. For Manifest and Idea Tbilisi, however, there is much more variation in the percent of signature goals achieved by petitions. On Manifest, petitions that did not succeed were not merely stuck at 0%, but they managed to achieve a significant portion of the required signatures, some upwards of 80% of their stated goal number. On Idea Tbilisi, the data is less evenly spread, but there are still clear instances where petitions failed with between 20 and 60 percent of required signatures, meaning many petitions collected between 500 and 1500 signatures each. While they may not have been considered successful enough to require a government response, such petitions are still successfully generating interest among hundreds 20 of people who also care about local issues. Other than its one successful petition, iChange did not generate the same interest among users for posted issues that these other platforms did; the vast majority of petitions received no more than a handful of signatures. Next, we examined the characteristics of successful petitions. It could be the case that iChange’s lack of success and engagement is due to the restrictions on what constitutes a germane petition. If successful petitions on Manifest primarily address issues that lack the same level of seriousness as ones on iChange, then this variation in success would not be a good indication of an important difference in their performance. Similarly, were we to focus only on petition on Manifest that would be appropriate for iChange, we would expect to see similar levels of success. We present a statistical model to illustrate what factors correlate with total signatures.17 We use a negative binomial model due to the over dispersion of the count variable, signatures. The model coefficient is given the change in expected log-count of signatures. Figure 11 below illustrates the results of the model graphically, with the point estimates represented by the blue dots, and the 90 and 95 percent confidence intervals represented by the thick and thin blue lines, respectively. Figure 11: Results of a Negative Binomial count model using topic scores to predict signature counts 17 We ran similar models predicting success, and results are largely similar even considering that Manifest petition creators can set their own standard for success. 21 Once we control for the topics, petitions on Idea Tbilisi are not statistically different from those on iChange in terms of number of signatures. Petitions on Manifest do gain more signatures than those on either government-run petition website. A petition that received a “1” for the Youth category would be expected to receive about seven times as many signatures as one solely about education, the baseline category. This finding provides support that e-petition users are predominantly young people, writing and signing petitions on topics that directly impact them. As they are the first generation to grow up with the internet, young Georgian e-petition writers are likely able to garner digital support much more effectively than older generations who have less interest in the Youth topic and more interest in topics that garner lower signature counts. The younger generation's practical understanding of new digital media is likely bolstered by their experience growing up in post-Soviet Georgia, where freedom of expression is more prevalent than it was for older generations. The topic that received the strongest negative coefficient were petitions labeled with the "miscellaneous" topic. The model predicts that a petition placed entirely in the miscellaneous category would receive only two percent of the expected signatures of one solely about education. This finding is consistent with what experts involved in the development of the iChange project said about e-petitions before the platform was launched. They argued that more specific, targeted, and well-written petitions were much more likely to catch people's attention and garner more signatures. Petitions that were vague or not categorized under a topic, however, were unlikely to garner as much support. That was a large part of the impetus for the E&E campaign targeted at getting the word out about how to use iChange prior to the site launch. Not only was it important that users knew how to sign up with their emails and utilize the site from a technical standpoint, but it was also crucial that users had some primer on how to produce a quality petition that would stand a chance of reaching 10,000 signatures. In a country where civic and political knowledge is highly unequal, providing in-person training in regions outside of Tbilisi was a necessary prerequisite for the platform to gain traction across the country. This argument for the importance of specific, targeted petitions is supported by the data in the low signature counts we see on petitions labeled "miscellaneous." Finally, these results help counter the sense that the types of issues being discussed on iChange are those that generally do not garner interest. Youth topics are clearly of interest to those who log on anywhere to sign petitions. It is not the case that entertainment petitions (discussed in more detail below) are far more popular. Issues related to public spaces, regulation, waste and utilities all receive higher signature counts than the baseline category, education. Many of the specifics of petitions within these categories fit the goals of iChange. While there are some outliers on Manifest (petitions for various popular boy bands to perform in Georgia, or a petition to Georgian distributors of Coca Cola to make the cherry flavor available18) the majority of petitions focus on the same serious issues as those on Idea Tbilisi and iChange. This suggests that in retrospect, iChange may not have needed such a high level of content moderation as was put in place at its inception. In interviews, representatives from NGOs that worked on the iChange project noted that partners in the Georgian government feared being overrun with petitions online. This was part of the rationale for setting the signature limit so high: government officials sought to limit the strain on resources for processing successful petitions by placing significant roadblocks along the way. 18 https://manifest.ge/main/item/255 22 According to NGO representatives, this may have also played into the lack of support government partners contributed to the E&E campaign that aimed to get the word out about iChange. Figure 12. Rejection reasons for iChange petitions Representatives from the government side of the project, however, viewed the failures of the iChange project as rooted in a lack of popular understanding about the purpose of the platform. The majority of rejected petitions were outside the scope of the central government's jurisdiction (59% of rejected submissions), despite the E&E campaign efforts executed by GYLA and CiDA to promote high quality petition submissions. According to government officials who worked on the project, 17% of rejected petitions are duplicates of existing petitions (possibly the result of technical difficulties with submission, or users not understanding how to update a published petition with an image or new information), 18.6% of rejected petitions are unfinished or clearly "jokes," and 4% called for actions that were against the law (see Figure 12). In an interview, a government representative gave the example of a submission that asked the government to ban Pride and included homophobic hate speech. Government officials also rejected petitions that were only for 1 person's interest or benefit, and petitions that simply called for a minister to resign. For better or for worse, petitions demanding the resignation of political figures and petitions calling for anti-LGBTQ laws are prevalent on Manifest. In Manifest's human rights category, there is a petition that call for a ban on "LGBT propaganda" and another that demands better protections of the rights of transgender Georgian citizens, reflecting the divided public opinion on such issues. Each received around 1000 signatures, engaging many others to sign and show support in their beliefs. This example highlights a fundamental difference between Manifest and iChange, and a challenge that the government e-petition platform has faced from the outset. With its limited content moderation, Manifest is a democratic free-for-all. Anyone can post any petition, and anyone can sign. This helped to quickly popularize Manifest after its launch in 2015, even though government officials were under no obligation to consider petitions on the site. iChange, on the other hand, is moderated carefully and thoroughly, to prevent any unprofessional or offensive petition from gaining signatures. While this practice has benefits (keeping the platform free from hate speech, or preventing a situation in which the 23 Georgian government was required to draft a law inviting One Direction to perform), it also limited the popularity and exposure that iChange received. That exposure in turn may have been able to make more people aware of how iChange could be used to petition the government for serious changes. Such a situation occurred on the U.S.'s own e-petition platform: in 2013, a petition on We The People asking the White House to build a Death Star went viral and received the requisite number of signatures to earn a response. The Obama administration reviewed the petition and politely rejected it,19 but the entire situation put the website into the public eye. That is not to say, however, that such potential benefits of such a stunt would be worth the potential costs in Georgia. The Obama administration may have been able to reject the absurd Death Star request with good humor because of the high level of trust in the U.S. government to respond meaningfully to realistic e-petitions and initiatives. The administration's track record of seriously responding to citizen needs allowed it to take the time to give a formal "no" response to a successful joke petition. But in a younger democracy working to get an e-petition site (and a myriad of other transparency reforms) off the ground, time wasted replying to successful but preposterous petitions might undermine the intent of an official petition website, and even call into question the efficacy of a government to enact meaningful democratic reforms. Recommendations As described above, one of the key challenges that iChange has faced since its inception has been striving to become popular while remaining credible. Those involved in the platform's creation and maintenance chose to enforce stringent content moderation standards to achieve that balance. These strict standards severely limited the number of people who actually used the iChange platform to submit or sign petitions. The success of Idea Tbilisi has demonstrated that certain segments of Georgian society are ready to actively engage with an e-petition platform. Outside of Tbilisi, however, the results are much more mixed. Manifest allows users to give their name and location when they sign, and the overwhelming majority of petition signatories list their location as Tbilisi, despite Manifest's reach as an (unofficial) national platform. While it is tempting to write off the success of Idea Tbilisi as evidence for the use of e-petitions at more local levels of government, a closer look suggests this is not the case. Through interviews and conversations with experts at the Good Governance Initiative, we learned that there were eight other municipalities with online petition platforms--but of those eight, only Idea Tbilisi succeeded in drawing in thousands of repeat users to write and sign petitions. Batumi's municipal platform drew a few petitions, but did not generate any momentum, and the websites for the other six municipalities seem to not have been used at all--the only petitions listed are "test" petitions created when the platforms began. In our qualitative interviews, those we spoke to all pointed to the success of Idea Tbilisi in the wake of iChange's disappointing performance. Many of our interviewees suggested that the reason iChange failed to take off in the same way was just because of the difficulty of garnering ten thousand signatures for a project that would benefit a village of two hundred people. However, the success of 19 https://obamawhitehouse.archives.gov/blog/2015/07/23/look-back-we-people-petitions-2010-today 24 Idea Tbilisi, the lack of engagement with other municipal petition websites, and the lackluster engagement with iChange suggest something else. The challenge may not be to bring nationwide attention to problems in the regions, but to bring people in the regions' attention to nationwide issues through online platforms. Those administering iChange have poured resources into the monitoring and filtering of petition submissions to prevent incomplete or inappropriate petitions from garnering signatures. Data from Manifest suggests that in the overwhelming majority of cases, people will still discuss serious political and economic issues even with less supervision. Rather than strictly policing the content that is allowed to move to the voting stage on iChange, future administrators of the program could do more to provide feedback and assistance to those drafting iChange petitions. The lack of use of regional municipal petition sites suggests that residents of smaller Georgian cities and towns may not even be aware that e￾petition resources exist--much less how to use them effectively to gather signatures for a cause. While users from Tbilisi are more likely to have knowledge of both technological resources and civic engagement, for many in further reaches of Georgia, iChange may be their first foray into e-democracy and political participation. If their petitions are rejected outright, whether for being outside of central government scope or too closely resembling an existing petition, they are unlikely to try using the platform again. However, personalized feedback aimed at helping users to improve their submissions could go a long way in terms of both increasing engagement with the platform and reducing the share of submitted petitions that are ultimately rejected. While work was put into an E&E campaign about how to utilize the iChange platform when it was first launched, such one-off trainings could not reach a large enough base of users to generate public knowledge and excitement about the platform. Outside of the civil society sphere in Georgia, few people the interviewers interacted with day-to-day had heard of the platform at all. Continuous efforts to work with petition authors to improve and develop their ideas would encourage rather than discourage new users, who are key to spreading knowledge of iChange organically among their own social networks. Another potential recommendation to improve organic, "word of mouth" knowledge of the iChange platform (this wider audience being the key to increased petition signature counts) is to make it easier to publish, sign, and discuss petitions with one's network. Manifest stands out from Georgia's government-operated petition sites with its use of Facebook plugins, making it possible for users to leave comments using their Facebook profile without even leaving Manifest, and enabling users to share a petition with just a click. Given the immense popularity of Facebook in Georgia, this feature has proven to be a powerful tool for driving more people to the site to sign petitions with ease. While as a government platform, iChange administrators must be cautious of potential fake accounts and other inauthentic behavior that could impact the e-petition process, promoting better integration between iChange and social media popular throughout Georgia (especially among young people) could drive a larger audience towards the site and improve the numbers of signatures each petition posted to iChange receives. 25 Annexes Annex 1: List of Persons Interviewed Currently excluded. Annex 2: Bibliography of Documents Reviewed Yasseri, Hale, and Margetts. “Rapid rise and decay in petition signing.” (2017) https://epjdatascience.springeropen.com/articles/10.1140/epjds/s13688-017-0116-6 Tetra Tech ARD. Good Governance Initiative (GGI) in Georgia (AID-114-C-15-0001). "Request for Expressions of Interest to co-create a Civic Engagement Campaign for the iChange Electronic Platform." 2014 Good Governance Initiative. "GGI Co-Creation PIA Meeting Notes." June 11, 2015. Good Governance Initiative. "Justification to Restrict Eligibility (JRE) for CiDA." August 12, 2015. Good Governance Initiative. "iChange Rejection Form Letter Ver1.1." Good Governance Initiative. "iChange Print Poster." Good Governance Initiative. "iChange Moderator Responsibilities and Duties." Eds. Khatuna Kviralashvili, Irakli Svanidze, and Sulkhan Saladze. "Campaign for Raising Public Awareness and Education of iChange Electronic Platform. Final Report, Version 1.0." Georgian Young Lawyers' Association, ACT Research Group, and the U.S. Agency for International Development. June 2018. Civil Development Association. "iChange Print Brochure." Government of Georgia. "Decree #245 On the Creation of an e-Petitions portal (iChange.gov.ge) and the Approval of the portal Use Regulations and Conditions." May 18, 2017. Good Governance Initiative. "GGI Grants Program Final Evaluation, Grantee: GYLA." 2018. Good Governance Initiative. "Quarterly Performance Report (QPR) No. 16 January-March, 2019." April 2019. Good Governance Initiative. "General Concept on iChange Implementation (GYLA and CiDA)." August 2017. Good Governance Initiative. "Fourth Draft Rejection Form iChange." 2015. Eds. Khatuna Kviralashvili, Irakli Svanidze, and Sulkhan Saladze. "iChange.gov.ge Campaign for Raising Awareness and Public Education: Final Research Report and Recommendations." Georgian Young Lawyers' Association. June 2018. Salome Zurabishvili. "Final Performance Report." Civil Development Agency. July 5, 2018. Government of Georgia. Procedural Manual for Managing Citizen Platform iChange (E-Petitions Section). Institute for the Development of the Freedom of Information (IDFI) and the Center for Post-Soviet Studies (CPSS). "Concept of citizen platform - www.ichange.ge." April 2013. U.S. Global Development Lab. "Development Innovation Accelerator Fact Sheet." The Good Governance Initiative. "iChange Official Launch. "December 22, 2017. Good Governance Initiative. "8th Interim Program Performance Report." June 2018. "I-Change Performance Monitoring Matrix." April 2018. 26 Annex 3: Databases "iChange." ichange.gov.ge "Manifest." manifest.ge "Idea Tbilisi." idea.tbilisi.gov.ge Annex 4: Semi-Structured Interview Questions The following was a template question list we followed for each interview we conducted in summer 2019. Questions were phrased the same way in all of our interviews, only changing to reflect the organization our interviewee represented. Because we followed a semi-structured format, we included space after our nine main questions to ask additional questions about each interviewee and organization's role in the process, as well as to ask follow-up questions as necessary in each interview. 1. What was your specific role within the iChange framework? What about your organization as a whole? a. # of people working on the program, whether it’s a set team, type of tasks, etc. 2. Can you talk about the process of developing the online platform? How did the project initially get off the ground? 3. How did your role evolve throughout the course of the project? What about the role of your organization? 4. How was this project prioritized by your organization? 5. Could you talk about how you collaborated with other organizations when developing and promoting the platform? 6. What was the feedback you received about the platform? How long did it take for people to understand the platform? 7. What does the future of iChange look like from the perspective of your organization? 8. Is there anyone else at your organization that you think would be useful for us to talk to about iChange? Would you be able to put us in touch? 9. Is there anything else you think is important for us to know? 10. [Space for additional questions for interviewee depending on their role in the project] Annex 5: Legal Text Establishing iChange Article 4. Electronic petition 1. All adult citizens of Georgia can initiate an electronic petition before the Government of Georgia and have the right to sign an electronic petition. The right to initiate an electronic petition entitles the customer to request a review of the petition in the form prescribed by this Rule and Conditions. 27 2. To initiate or sign an electronic petition, a citizen must register on the ichange portal, where he / she indicates the information specified in this rule and conditions. The user must agree to the terms and conditions given on the website. 3. An electronic petition will be received and published on the ichange portal in case: A) it contains a clearly formulated request for a specific response by the Government of Georgia; B) the decision on the request provided by the electronic petition falls within the authority of the Government of Georgia; C) it does not contain a request similar to the one reflected in the e-petition once considered by the Government of Georgia; D) it is submitted in the state language; E) is initiated by an adult citizen of Georgia; F) meets the other criteria set forth in this Resolution. 4. Will not be accepted and published on the ichange portal: A) petitions containing opinions / proposals, the subject of which does not fall within the competence of the Government of Georgia; B) Petitions that do not clearly state the opinion, idea / suggestion; C) petitions containing political, propaganda and commercial content; D) petitions that include offensive, obscene, provocative content, as well as petitions initiated for the purpose of joking; E) petitions containing calls for hatred, discrimination, threats of violence, harm to an individual or any group; F) Petitions that contain the overthrow or violent change of the constitutional order of Georgia, the violation of the country's independence and territorial integrity, or calls such as propaganda of war and violence, incite national, sectarian, religious or social strife and create a clear direct action under this subparagraph. And contains a substantial threat, as well as calls for disobedience to Georgian law; G) petitions that concern only the interests, personal circumstances or issues of one particular person; H) other petitions that contradict the requirements of this Resolution and the legislation of Georgia. 5. An electronic petition shall be received and published on the ichange portal if it meets all the requirements set forth in paragraph 3 of this Article together and does not belong to the petition containing the content referred to in paragraph 4 of this Article. 28 6. If the e-petition does not comply with the requirements of this resolution, it will not be accepted by the ichange moderator and will not be published on the ichange portal. 7. Each customer has the right to submit a new petition in accordance with the requirements of the electronic petition portal. 8. In order to consider an electronic petition and receive an official response to it, it is necessary for at least 10,000 (ten thousand) users to support the electronic petition. The e-petition must collect this number of signatures within 30 calendar days after the publication of the initiated e-petition on the ichange portal. If the petition fails to obtain the required number of signatures within the specified time, it will be considered expired, it will be closed on the ichange portal and it will be impossible to add a signature.