DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY Impact Evaluation of USAID/Georgia’s Momavlis Taoba (MT) Civic Education Initiative (CEI) TASKING 015 April 2019 Contract No. GS-10F-0033M/AID-OAA-M-13-00013 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY IMPACT EVALUATION OF USAID/GEORGIA’S MOMAVLIS TAOBA (MT) CIVIC EDUCATION INITIATIVE (CEI) (APRIL 2019) Prepared under Contract No.: GS-10F-0033M/AID-OAA-M-13-00013 Submitted to: Morgan Holmes Submitted by: NORC at the University of Chicago Attention: Renee Hendley, Program Manager Bethesda, MD 20814 Tel: 301- 634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 CONTENTS Tables ............................................................................................................................................................ii Acronyms ....................................................................................................................................................iii EXECUTIVE SUMMARY.......................................................................................................................... 1 1. INTRODUCTION ............................................................................................................................. 4 2. LITERATURE REVIEW...................................................................................................................... 6 3. DESCRIPTION OF MOMAVLIS TAOBA ................................................................................... 14 4. EVALUATION QUESTIONS AND HYPOTHESES ................................................................. 17 5. METHODOLOGY............................................................................................................................ 22 6. RESULTS ............................................................................................................................................. 31 7. CONCLUSIONS............................................................................................................................... 47 REFERENCES............................................................................................................................................ 50 ANNEX 1 IMPACT EVALUATION DESIGN REPORT................................................................. 58 ANNEX 2 SURVEY QUESTIONNAIRE...........................................................................................109 ANNEX 3 DESCRIPTIVE STATISTICS FOR OUTCOME MEASURES ....................................128 ANNEX 4 BALANCE TESTS..............................................................................................................130 IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | i DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 TABLES Table 1. Summary of MT Civic Education Variations......................................................................22 Table 2. Number of Schools per Group, by Region........................................................................23 Table 3. Structure of the Data..............................................................................................................25 Table 4. Summary of Data Collected by Group, per Grade and Wave......................................25 Table 5. Hypotheses and Hypothesis Tests for EQ1 and EQ2 .....................................................31 Table 6. Effects of Civic Education Programming.............................................................................33 Table 7. Hypotheses and Hypothesis Tests for EQ3.......................................................................35 Table 8. Heterogeneous Effects: Boys vs. Girls................................................................................38 Table 9. Heterogeneous Effects: Non-minority vs. Minority Students........................................40 Table 10. Heterogeneous Effects: Non-low-SES vs. Low-SES Students......................................42 Table 11. Hypotheses and Hypothesis Tests for EQ4 ....................................................................43 Table 12. Heterogeneous Effects: One Year vs. Two Years of MT Programming ...................44 Table 13. Hypotheses and Hypothesis Tests for EQ5 ....................................................................45 Table 14. Heterogeneous Effects: 9th-graders vs. 10th-graders...................................................46 Table A3.1. Descriptive Statistics for Outcome Measures ..........................................................129 Table A2.1. Multinomial Logistic Regression of Variation Assignment......................................131 IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | ii DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 ACRONYMS ACETT Applied Civic Education and Teacher Training ATET Average Treatment Effect on the Treated CEI Civic Education Initiative CRRC Caucasus Research Resource Centers DRG Democracy, Human Rights, and Governance EQ1, 2, … Evaluation Question 1, 2, ... FDR False Discovery Rate H1, 2, … Hypotheses 1, 2, … IE Impact Evaluation IEA International Association for the Evaluation of Educational Achievement IPW Inverse Probability Weighting ITT Intent-to-Treat LER Learning, Evaluation, and Research MI Multiple Imputation MOES Ministry of Education and Science MT Momavlis Taoba NGO Non-governmental Organization NORC National Opinion Research Center (NORC at the University of Chicago) OLS Ordinary Least Squares RCT Randomized Control Trial SES Socioeconomic Status USAID United States Agency for International Development V1, 2, … Variation 1, 2, … Y1, 2, … Year 1, 2, … IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | iii DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 EXECUTIVE SUMMARY The U.S. Agency for International Development Mission in Georgia (USAID/Georgia) funded a civic education initiative (CEI) in Georgia called Momavlis Taoba (MT) which was implemented by PH International between June 2014 and December 2017. The main goals of MT were to promote greater civic engagement of young people, and expand and institutionalize secondary school civics education curricula and practical applications throughout the country. To assess what aspects of the MT design had positive effects on civic outcomes, PH International implemented a substantial portion of MT following a randomized controlled trial (RCT) design whereby schools were randomly assigned to one of three intervention variations or a control (or comparison) group:  Variation 1 (V1), enhanced teacher training and supplementary civics curricula.  Variation 2 (V2), enhanced teacher training and supplementary civics curricula plus voluntary civics clubs plus.  Variation 3 (V3), enhanced teacher training and supplementary civics curricula plus mandatory class civics projects.  Control, no MT programming (receives national civics curriculum). MT’s three variations respectively represent a pure traditional classroom model, a traditional classroom and voluntary experiential learning hybrid, and a traditional classroom and compulsory experiential learning hybrid. This impact evaluation (IE) report assesses the absolute and relative effectiveness of these three variations in bringing about desired changes in civic, democratic, and prosocial attitudes and behaviors inside and outside schools. The IE report also examines whether the variations contributed to closing ‘democratic achievement gaps’ between boys and girls, majority and minority students, and better-off and worse-off students. NORC at the University of Chicago (NORC) and its local research partner Caucasus Research Resource Center in Georgia (CRRC/Georgia) administered three waves of data collection for the IE of MT—in Spring 2016, Fall 2016, and Spring 2017—among 26,350 students (average of 8,784 students per wave) in 240 schools throughout the country. The survey was administered in a sample of schools receiving each of the three variations and control schools. The survey instrument included 135 items. These items are used to construct the IE’s outcome measures following a theoretically-informed, data-driven process. The process starts by classifying items according to the level of change they intent to measure; outcome or impact. Outcome-level items are those gauging behaviors and attitudes about objects inside school that are likely to change in the short and medium term as a result of the interventions. Impact-level items are those gauging behaviors and attitudes about objects outside school that are also likely to change, but in the medium and long term. Factor analysis and item reliability analysis are then used to construct the following nine outcome measures: IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 1 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013  Outcome-level measures: o Attitudes related to school government and politics o Civic behaviors and teachings in the classroom o Engagement in school government and politics  Impact-level measures: o Prosocial and democratic attitudes o Attitudes about corruption and dictatorship o Internal political efficacy and beliefs about citizenship o External political efficacy o Current civic engagement and practices o Future engagement and political participation The survey also included a measure of uptake of the experiential learning components offered as part of V2 and V3. It asked students whether, as part of a civics class, they had worked on a civics or volunteer project in the last two years. This uptake-level outcome measure is included in the analysis, for a total of ten outcome measures. As mentioned above, the IE followed an RCT design. Balance tests are conducted to check if the randomization produced groups with balanced control covariates. The results indicate that it did not. While random assignment can produce unbalanced groups due to chance, it is not possible to rule out that imbalances are due to improper randomization (either at design or implementation). To minimize the possibility of bias in the results, effects aree estimated using inverse probability weighting (IPW). This technique models treatment assignment and estimates the probability (or propensity score) of being assigned to a given group for each student, and it then uses this probably to produce estimates that account for nonrandom treatment assignment. The following are the major findings of the IE report:  MT programming had a relatively successful uptake of experiential learning activities. Surprisingly, students in schools assigned to V1 participated in voluntary or civics projects at similar rates than those in schools assigned to V2, but given that the effects of V3 on project uptake are much stronger, the results show that V2 and V3 together are more effective on enhancing participation than V1.  MT programming had a significant effect on outcome-level measures gauging attitudes related to school government and politics, and civic behaviors and teachings in the classroom. Regardless of variation, students in MT schools were more likely to express their opinions, learn about problems in society, discuss current events, have positive views about participation in school politics, and believe that positive changes can happen in schools when students mobilize in groups than students in control schools. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 2 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013  When it comes to effecting change on impact-level measures (i.e., behaviors and attitudes about objects outside school that are likely to change in the medium to long term), MT programming had mixed results. On the one hand, MT programming did not have any effects on students’ civic, democratic, and prosocial attitudes. On the other hand, all three MT variations, but especially V2 and V3, had a positive effect on current civic engagement and practices. There is a concern, however, that these results might reflect uptake of experiential learning projects rather than more spontaneous engagement and practices that happened outside of MT programming.  With regard to the democratic achievement gap between genders, findings from control schools indicate that, contrary to expectations, girls fare better than boys on civic, democratic, and prosocial attitudes and behaviors. MT programming had a stronger effect on uptake of experiential learning activities and current civic engagement and practices among girls, increasing the gap between genders in favor of girls.  MT programming did not contribute to closing democratic achievement gaps between students from ethnoreligious minorities and other students, and between low-SES students and other students.  Lastly, two years of exposure to MT programming had similar effects than one year of exposure, and MT programming had similar effects among 9th-graders and 10th￾graders. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 3 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 1. INTRODUCTION Georgia is continuing its transition to a full democracy and has enormous potential as a strong U.S. ally and leader in the Caucasus and beyond, making it a good example for other countries transitioning to democracy. USAID/Georgia recently sponsored members of the Georgian Bar Association to travel to Iraq to partner in training with Iraqi lawyers. While Georgia has and continues to make great strides, a plurality (43 percent) of those surveyed in the most recent Caucasus Barometer concede that Georgia is “a democracy but with major problems.” Given the cooperation of the government of Georgia, now is a critical time to continue institutionalizing democracy in Georgia, ensuring as best as possible its irreversibility. One way to guarantee Georgia’s democratic future, and indeed that of any country, is by creating a strong demand for democracy from its citizens. In 2003, the people of Georgia banded together and peacefully, yet powerfully, demanded that their government respect democratic principles and forced the government to heed its calls. Despite this incredible democratic spirit, the most recent Caucasus Barometer reports that of those surveyed, a plurality (40 to 48 percent) neither trust nor distrust major institutions of government including local government, the executive, parliament and political parties. Although improving public institutions may be one way to address this apathy in trust, another is to empower citizens, holding them accountable and confident in their responsiveness. Increased civic knowledge is likely to increase trust in public institutions in a democracy (Galston 2001, 224). On measures of tolerance, Georgian society also has some room to improve. Between 20 and 24 percent of Georgians surveyed disapproved of doing business with ethnic minority populations in the country. However, those disapproval numbers jump to between 60 and 70 percent when asked whether Georgian women should marry members from those communities. Recent research suggests that lack of integration of ethnic minorities into the national identity makes a country susceptible to ethnic conflict even if a majority of people prefer peace and only a few extremists aim to provoke violence (Sambanis and Shayo 2013). Given the regional concentration of Georgia’s ethnic minority populations and that regional concentration of ethnic minorities in general consistently appears as a risk factor for violent conflict in studies of interethnic violence (Toft 2002, Connell 2002, Lichbach 1995, Goemans 2006), it is important to take this lack of integration seriously. With all of this in mind, Georgia is ripe for improved civic education to further institutionalize its democratic gains. While there are a myriad of approaches to improving civic education and behavior of citizens, several of which have been implemented in Georgia, educating young people may potentially pay great dividends over the short and long terms. The Georgian school system is sufficiently poised to implement a high-quality civic education initiative; the Ministry of Education and Science (MOES) has also expressed its support for teaching such classes and for evaluating their effectiveness. The USAID Center of Excellence on Democracy, Human Rights and Governance, together with USAID/Georgia and the Momavlis Taoba (MT) Civic Education Initiative (CEI) implementing partner PH International, designed and conducted an impact evaluation (IE) of three civic education interventions: (1) enhanced teacher training and supplementary civic curricula; (2) IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 4 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 voluntary student civic clubs; and (3) class civic projects. These interventions were bundled into the following three variations:  Variation 1 (V1): Enhanced teacher training and supplementary civic curricula  Variation 2 (V2): V1 plus voluntary student civic clubs  Variation 3 (V3): V1 plus mandatory class civic projects The 240 schools participating in the IE were randomly assigned to receive one of these variations or to a control (or comparison) group receiving no MT programming. The goal of the IE is to rigorously measure the effectiveness of these bundles in improving civic education among Georgian youth and in producing desired changes in civic and political behaviors and social and political attitudes. Under the Democracy Human Rights and Governance Learning Activity (DRG-LER), NORC at the University of Chicago was requested to carry out data collection in participating schools. NORC was also tasked with carrying out the final analysis of the data and producing this final report. The remainder of the report is organized as follows. The next section reviews the literature on civic education. The third section describes the MT project’s interventions and how these interventions are bundled for the IE. The fourth section lays out the evaluation questions and hypotheses guiding the study. The fifth section presents the IE’s empirical strategy; it describes the IE design, data collection, outcome measure construction, and estimation. The sixth section presents and discussed the estimation results pertaining to each research question. The last section summarizes the main findings and concludes. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 5 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 2. LITERATURE REVIEW It is a nearly universally held belief that an informed and engaged citizenry is good for democracy. Such a public can elect competent leaders and hold them accountable for the policies and decisions they pursue and exercise legal mechanisms to remove them from office when they fail to live up to democratic principles. Scholars agree that “well-designed institutions are not enough, that a well-ordered polity requires citizens with appropriate knowledge, skills and traits of character […] And, it is reasonably clear that good citizens are made, not born” (Galston 2001, 217). The question is: what is the best way to go about creating an informed and engaged citizenry? Over the past fifty years, several waves of scholarly interest in civic education have emerged. In the mid-1990s the American politics literature showed renewed interested in answering this question. Interest had lain dormant for some time given a seminal report in 1968 that found that in the U.S., civics curriculum was not “even a minor source of political socialization” (Langton and Jennings 1968, 865). Comparative politics literature showed increased interest in the topic starting in the 2000s, likely as a result of the desire to examine the outcomes of a decade of democracy promotion in the post-Cold War era. For instance, Kim, Flanagan, and Pykett (2015) find that in post-communist countries, families are not supportive of social responsibility and students are less likely to participate in extracurricular activities. Others note that socialist societies were based on “values, norms and institutions that were antithetical and openly hostile to political and social pluralism, voluntaristic citizen participation, respect for individual rights and the rule of law” among other democratic principles (Slomczynski and Shabad 1998, 750). They also note a culture of “privatism, distrust outside one’s immediate family and close social circle and an emphasis on personal favors and ties” (Ibid., 750). Another division in this literature is whether studies target adults or adolescent students. This distinction is important, because evidence suggests that civic participation and its manifestation vary greatly by age group as does the type of civic engagement activity (Pavlova et al. 2015). Finally, the type of the civic education intervention used—classroom education, workshops, public information campaigns, or service learning—differentiates the research along another axis. Indeed, political environment, maturity of the audience, and method of delivery, all have substantial impacts on learning outcomes in addition to duration and type of deliverer. This literature review outlines the most relevant studies on civic education to date and highlights the theories the IE examines. Adult Civic Education Much of the literature on adult-targeted civic education comes from studies in developing democracies where international aid agencies have sponsored programs. Finkel (2002) theorizes that adult civic education programs can have direct, indirect, and conditional effects on civic participation. He finds that in most of the programs examined in the Dominican Republic and South Africa, direct and indirect effects occurred but that in all of them, the more frequently individuals receive training in civic education and the more participatory the methods, the greater the effects (ibid., 1012). In a parallel study of an adult civic education program in the Dominican Republic, Finkel, Sabatini, and Bevis (2000) found that the program IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 6 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 had a negative effect on participants’ trust in the institutions of government and theorize that this is because the non-governmental organizations (NGOs) that were involved with implementing the program were in opposition to the government and because the government was functioning poorly. They explain that people’s increase in political knowledge about the government and their increased sense of personal efficacy made them more aware of governments’ shortcomings and how they might overcome them. In another paper in which he adds an adult civic education program in Poland to the Dominican Republic and South Africa analysis, Finkel (2003) finds overwhelming support for the programs increasing participation especially when they had strong local problem-solving components. A study of the same three programs also found that they helped to close the participation gap between elites and non-elites but that overall gains were modest (Blair 2003). However, an earlier study of a USAID-funded adult civic education program in Zambia found that elites benefited more than non-elites (Bratton et al. 1999). A US-based study of young adults in college found that civic education activities can reduce the democratic achievement gap and promote political empowerment among those with lower socioeconomic status (Beaumont 2011).1 Adolescent Civic Education Much research attention has been given to studying civic education and knowledge among high school students in the U.S., with comparative studies of students in other countries becoming increasingly common. The focus of much of the research in adolescent civic education has addressed the roles that family, schools, peers, and associational life play (Amna 2012). Generally speaking, children inherit their parents’ sense of social trust as well as knowledge, values, attitudes, and actions (Wray-Lake and Flanagan 2012, Serek, Lacinova and Macek 2012). While parents’ social resources play a pivotal role, other research has indicated that civic education programs in schools can overcome deficits in parental resources by creating opportunity for those youth to participate in civic activities (Flanagan and Levine 2012, Melchior et al. 1999). Others have pointed to adolescence as a crucial period in which individuals develop and consolidate their own political points of view (Eckstein et al. 2012, Sears and Levy 2003; Avery et al. 1992) and civic skills (Verba et al. 1995). While the Langston and Jennings (1968) report painted a dire picture of the state of U.S. civic education, other subsequent research has been more optimistic. One of the problems in measuring the impact of civic education efforts at the national level is that the education system is decentralized, giving the states and local school districts most of the control over the curriculum. While textbooks and classroom instruction vary little (Avery and Simmons 2000/2001), activities and opportunities outside the classroom have wide variations (Hahn 2003). National-level surveys are able to measure political knowledge, civic skills, political participation, and social and political values, but they cannot necessarily attribute them to civic education classes or programs, nor distinguish effects of general civic socialization that come from simply living in a democratic society. Some researchers, however, do find the amount of civic instruction and the shorter the time since the course was taken increase the level of 1 As explained in section 3, ‘democratic achievement gap’ refers to the deficit that some groups might face on various dimensions of civic, democratic, and prosocial attitudes and behaviors. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 7 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 political knowledge students display (Niemi and Junn 1998), although their methodology has been questioned (Torney-Purta et al. 1999; Greene 2000). Another study focused on four elements of the high school experience, including a sense of school community and level of political discussion. The study found those elements to significantly impact high school students’ civic consciousness, and that civic education also occurs in classes other than civics classes (Conover and Searing 2000) Recent research also has examined the levels of civic knowledge and engagement and the effects of civic education for adolescents cross-nationally. In 1999, the International Association for the Evaluation of Educational Achievement (IEA) undertook an extensive study surveying representative samples of approximately 3,000 students in the modal grade for 14-year-olds in each of 28 countries (Torney-Purta et al. 2001). Of the many findings from IEA’s 1999 survey, the most relevant for this IE is that there is a positive relationship between civic knowledge and reported likelihood of voting in the future; participatory and open school environments correlate with higher civic knowledge and engagement; and trust in government appears to mirror the levels reported by adults in their respective countries (Torney-Purta et al. 2001). In 2009, the survey implementation was updated and expanded to include more than 140,000 students from 38 countries or regions (Schulz et al. 2010). While the surveys in 1999 and 2009 did not ask all of the same questions, the 2009 iteration included several of the original survey questions for comparison purposes and found that civic knowledge had declined significantly in several countries since 1999. Perceptions of openness during classroom discussions of political and social issues continued to be positively correlated with civic knowledge and engagement (Ibid. 19). Other civic education research outside the U.S. has focused on programs in individual developing countries. A study conducted among adolescents in Poland, for example, found that civic education seemed to moderate students’ opinions at both ends of the ideological spectrum. Slomczynski and Shabad (1998) found that while those who participated in an updated participatory civic education curriculum were less likely to exhibit extreme anti￾democratic and anti-market positions, they were also less likely to assert extreme pro￾democracy and pro-market attitudes. The authors theorize that this effect likely is due to the curriculum encouraging students to think critically about government, regardless of its type. A study of a program in Argentina in which students in grades 6 and 7 debated current events and read news articles found that students who participated in the program outperformed those who did not in terms of political knowledge and tolerance (Morduchowicz et al. 1996). A comparative analysis of indigenous practices related to civic engagement in Madagascar and the Sahel suggests that people in Africa often see democratization and civic education programs as externally imposed or Western imports but that many African cultures have civic engagement traditions that could be harnessed to bring the concepts better in line with indigenous culture (Antal and Easton 2009). Classroom Learning While traditional classroom instruction may be less pedagogically exciting than some newer extracurricular innovations, it remains a substantial part of the way civic education is delivered to students, both in the U.S. and abroad. Unsurprisingly, research shows that more interactive teaching techniques increase the retention of civic knowledge (Niemi and Junn 1998; Torney￾IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 8 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Purta et al. 2001). However, instruction generally emphasizes the federal government in the U.S. and devotes little time to local government where students have a greater opportunity to engage in civic issues and apply their knowledge (Neimi and Junn 1998). Other studies also point to the effectiveness of openness in classroom discussions (Alivernini and Manganelli 2011; Youniss 2011; Campbell 2008, McDevitt and Kiousis 2006; Morduchowicz et al. 1996). Training of teachers also has received attention in recent research. Those wishing to understand the effects of particular curricula realized that those delivering the content were a significant part of the treatment. In many countries until very recently, or even presently, the main method of teacher instruction was one way communication of information which students memorized. Civic education that encourages participation and engagement is particularly poorly suited for this pedagogical tradition. For example, one study of the reformed curriculum in Mexico found that the biggest problem with the program as reported by teachers and administrators alike is the lack of teacher training in more democratic styles of instruction, classroom dialogue, and encouragement of critical analysis among students (Levinson 2004). Another study of a specialized civic education program in the U.S. found that providing teachers with specialized civic education training improved students’ civic knowledge and engagement values (Owen 2015). Service Learning While the traditional classroom model remains a popular method of delivery of civic education content, service learning has become a popular way of promoting student engagement in their communities. In the early 1990s, the U.S. federal government enacted legislation to fund activities that promoted community service; universities as well as primary and secondary schools started new programs and continued to expand existing ones to encourage student evolvement in their communities (Markus et al. 1993). The theory upon which the hypothesized effectiveness of service learning is based appears to have its roots implicitly in experiential learning. This approach to learning emphasizes learning as a process that allows individuals to apply abstract concepts to concrete problems, gather data, and reevaluate the concepts, either confirming, altering, or disconfirming them (Kolb 1984). Additionally, the field of developmental psychology has found that during adolescence, students form connections to their communities, along with their character and moral commitment to society (Lerner et al. 2000; Pittman et al. 2001). Therefore, “students who form connections to their school and community (and to their ethnic group) are likely to care about issues pertaining to these institutions, and therefore to contribute to addressing these issues” (Torney-Purta et al. 2007). For those who want to test the effectiveness of service learning, there is a distinct normative bias toward finding positive results. Who wants to hear that doing community service does not improve engagement in the community and strengthen democratic norms? Markus, Howard, and King (1993) highlight the significant differences in students who were required to do community service as part of a political science college course, but neglect to discuss that students who did not participate in community service as part of the class reported an increase in concern regarding social justice in the post-course evaluation, while the community service students showed no change. They also fail to point out that neither group showed an improvement in their perception of being able to make a difference the world or of individuals being able to have an impact. Also, by measuring the effects immediately following the course, IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 9 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 they did not have much to say about whether any positive effects would be lasting. Additionally, merely reporting having a positive experience or liking the activity article reports (Bennion 2006) does not necessarily translate into learning outcomes. Another problem with measuring the effect of service learning is the diversity of programs. As some have pointed out, the type and quality of service in which one is engaged likely has substantial impacts on learning outcomes (Reinders and Youniss 2006). Stuffing envelopes for a membership organization mailing versus serving the homeless dinner at a soup kitchen are likely to have different learning outcomes for students. How well the service activity is tied to learning activities in a classroom also impacts student outcomes. A three-part study of adolescents in Portugal found that those who experienced low quality participatory activities had more negative political attitudes post-treatment than those in the control group (Fereira et al. 2012). Another study comparing three models of service learning in college students— placement optional, placement mandatory, and class collectively assigned to consult on a community issue—found that for all three groups, students’ assessment of the importance of community service was lower after the end of the class (Parker-Gwin and Mabry 1998). However, there is some evidence of positive effects of service learning, as well. In a 1998 study of a large US-based service learning program, researchers found several positive outcomes, specifically that a program longer than one year had significantly stronger civic outcomes and that older high school students retained longer-term impacts than middle school students (Melchior et al. 1999). The same study also found that the program appeared to benefit disadvantaged and non-white students more academically in general (Ibid.). Others have found that hybrid classroom and service learning programs that are closely connected in content produce positive results in terms of values, interests and participation in civic life (Westheimer and Kahne 2002; Parker-Gwin and Mabry 1998). In Finkel’s (2003) study of three adult civic education programs mentioned above, while he does not call it service learning, he finds that programs that had an applied component focusing on local problem solving and being in contact with local officials have better civic education outcomes particularly with regard to civic participation. This may indicate that solving a problem about which one cares in his/her community as part of the learning experience enhances desired civic outcomes (Youniss 2011, Soule 2000). Others assert that one component often missing from service learning programs is a focus on acquiring civic skills (Kirlin 2002; Verba et al. 1995). Additionally, there is evidence that student-led service learning projects induce greater anticipated civic participation in the future and tolerance toward outgroups (Morgan and Streb 2002). Another debate within the literature and the practitioner community is about whether and how the outcomes of service learning vary between compulsory programs and voluntary ones. On the one hand, some research indicates that required service has a negative effect on future civic behaviors. Helms, et al. (2009) find that when American college students in an experiment were forced to contribute a fixed amount of time to a charity-benefitting task, their contribution was substantially less than that of students who were given no minimum time requirement to contribute. Additionally, a study of high school graduates from Maryland, where community service has been a requirement for graduation since 1992, showed that while the requirement boosted community service among eighth-graders, it reduced volunteering among older students (Helms 2013). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 10 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 On the other hand, a fair amount of research does support the claim that all service learning leads to the intended positive outcomes. Hart et al. (2008) find that both compulsory and voluntary service learning for high school students correlates with higher levels of adult voting and volunteerism. In a college setting, others find that requiring service as part of course work also increased some measures of civic values and engagement sentiments among participants (Westheimer and Kahne 2002, Parker-Gwin and Mabry 1998). As mentioned below, it is likely that with the variety of service programs delivered, there is some problem with comparative measurement. It is very likely that how well the service requirement is connected to a learning program has a large impact on the likelihood of successful outcomes. A 2010 study of American high school students found that participation in clubs like debate, drama, and music significantly increased the likelihood of voting in presidential elections in the future, but that most sports clubs had no effect or a negative effect on future voting (Thomas and McFarland 2010). This is particularly interesting in that much of the literature about adult civic participation stresses the positive role that group memberships of any kind play in promoting political action (Putnam 2000; Booth and Richard 1998). While not an issue to be discussed at length here, others do have an ethical problem with forcing students to perform community service as a requirement for a course or for graduation, claiming that compelling service is antithetical to the principles upon which it is based. A Myriad of Desired Outcomes Another way to examine the research and literature on civic education is through the lens of desired outcomes. While researchers characterize their findings in different ways, the general approach emphasizes three categories: knowledge, values and efficacy, and participation. Generally speaking, there is more evidence analyzing the impact of civic education programs on civic and political knowledge (e.g. see Tonge, Mycock, and Jeffery 2012; Whiteley 2012; Finkel, Horowitz, and Rojo-Mendoza 2012; Syvertsen 2008; Finkel and Ernst 2005; Claes, Hooghe, and Stolle 2009; McDevitt and Chaffee 2000; Hartry and Porter 2004; Vontz, Metcalf, and Patrick 2000; Finkel 2014). The reason behind this is that civic and political knowledge are relatively easy to measure compared to the other two categories. Program implementers and researchers can simply give students or program participants a test in the form of self￾administered surveys in countries with high literacy rates, or can take the slightly more arduous step of using enumerator-administered surveys. That is, they could survey participants before the instruction begins, survey them again after the instruction is over, survey those who did not receive any instruction, and then compare the results. This method is very useful when measuring knowledge and even critical thinking skills if the questions are well constructed, but it can suffer from selection bias when the participants and non-participants are not randomly assigned. If participation in a civic education course is voluntary, there are likely endogenous factors which taint the analysis. For example, if a person volunteers to participate in a civic education program, he or she very likely has a heightened awareness of or interest in things related to civics or politics. Thus, if the study were to find differences between participants and not participants, it would not be possible to determine with certainty whether these are due to the training or to pre-existing differences in awareness or interest. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 11 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 This selection bias may be even more problematic when attempting to examine the effects of civic education programs on improving support for democratic values and civic engagement or political participation. Those who volunteer for civic education programs probably see some value in civic action and value democratic principles more so than those who do not volunteer. Additionally, while knowledge and skills may be relatively easily acquired, norms, values and attitudes are influenced by many experiences and external forces and may be more difficult to change and require a substantial amount of time to do so. Another, potentially bigger problem with measuring the effect of civic education interventions on democratic norms and values is the prevalence of social desirability bias. Even if subjects do not have a fealty toward democratic values or norms of tolerance, they likely know that they are supposed to in a democratic society. Therefore, attempting to measure norms and values in a survey format can be particularly problematic, and for measures to be valid, questions must be carefully constructed. Despite these substantial challenges and caveats, studies that may suffer from some or all of these problems have generally found that civic education programs marginally improve participants’ sense of political efficacy (Keating et al. 2010, Whitely 2012, Finkel et al. 2012, Soule 2002, Hartry and Porter 2004, Hope and Jagers 2014). Generally speaking, however, these studies find little if any evidence that such programs improve societal levels of democratic values, attitudes, and tolerance. Assessing the impact of civic education on civic engagement and political participation is not easy. One of the most difficult problems to overcome is that a civic education program administered in one time period (t) is not likely to lead to immediate civic participation or engagement, especially if it is a short-term intervention or if the participatory behavior being measured is voting and those receiving civic education are younger than 18 years of age. What researchers and practitioners really want to know is whether civic education programs promote civic participation and engagement in the future (i.e., at t + 1, t + 2, t + 5, t + 10 or t + 20). While it is possible to conduct a longitudinal study in which one surveys this population repeatedly over time after an intervention (see, e.g., Keating et al. 2010), such research may be difficult to undertake, particularly in developing countries, and takes many years to complete, making it less desirable for testing outcomes to inform program design. Therefore, researchers generally ask civic education program participants about planned future behavior. Social desirability bias is often a problem in this kind of measure, but as big a problem is the inability for individuals to predict how they will feel about civic participation in the future, especially adolescents and younger students. In short, it is very difficult to measure long-term effects of civic education programs which is their very purpose. No one ever designed a program with the goal of improving civic participation among those involved for six months, with the possible exception of a “get out the vote” intervention targeting a specific election. Despite the many problems with assessing the impact of civic education on civic engagement and political participation, many studies have attempted to test this relationship using these methods and have found mixed results. Many have found a correlation between civic knowledge and self-reported likelihood of future voting (Cohen and Chaffee 2013, Torney-Purta et al. 2001, Maeillo et al. 2003, Hart et al. 2007). Others have found that taking civics courses increases the self-reported likelihood of future voting (Berson et al. 2013) and general participation in civic life (Soule 2005, Hartry and Porter 2004, Vontz et al. 2000, Tonge et al. 2011, Whiteley 2012). Anoter set of studies has found civic education to have no effect on civic participation (McDevitt and Kiousis 2006, Pasek et al. 2008, Bers and Chau 2010). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 12 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Conclusion While there is much research measuring the effects of civic education, both in the U.S. and abroad, among adults and adolescents, on classroom-based programs, experiential learning programs, and hybrids, and on particular outcomes of interest, the findings are often contradictory, conditional, or biased. Prior research and theorizing does, however, lead us to a number of important hypotheses related to the relative effectiveness of the traditional classroom model and the compulsory or voluntary experiential learning models which this IE report tests. Before laying the evaluation questions and hypotheses, the following section describes the MT project. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 13 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 3. DESCRIPTION OF MOMAVLIS TAOBA The main objectives of MT were to promote greater civic engagement of young people and to expand and institutionalize secondary school civic education curricula and practical applications in the country of Georgia. The project reached approximately 480 secondary schools, and was designed to improve the quality and scope of school-based civic education as a means to positively influence the attitudes and behaviors of youth as active participants in Georgia’s democratic society. MT was implemented by PH International from June 2014 to December 2017. Overall, MT sought to improve democratic capacity among students in targeted schools. Democratic capacity, generally speaking, incorporates the three elements of knowledge, attitudes, and behaviors important for a functioning and thriving democratic society. Specifically, democratic capacity captures students’ perception of their own efficacy in solving community and societal problems both now and in the future; beliefs about what makes a good/active citizen; knowledge of rights and duties of citizens; and willingness to give of themselves for the greater good through volunteering or other prosocial collective action. ‘Democratic achievement gap’ is a closely-related concept that refers to the deficit that some groups might face on these constructs. For instance, girls, minority groups, or those lower on the socioeconomic spectrum might have a lower perception of their own efficacy to solve community problems. Besides improving democratic capacity across the board, MT sought to contribute to reducing gaps in democratic achievement. 3.1 MT INTERVENTIONS To accomplish these objectives, MT offered several different interventions to participating schools. As described in section 3.2, these interventions were bundled intro three ‘variations’ and randomly assigned to a set of participating schools. Enhanced Teacher Training and Supplementary Civics Curricula This intervention includes a four-day pedagogical training course for civics teachers called “Teaching Democratic Citizenship.” As part of this training, teachers receive additional instruction regarding the Georgian National Curriculum’s civic education requirements.2 The training also includes modules on teaching civic participation, lesson plans, community service projects and project planning, student assessment and creating rubrics, and interactive teaching methods. Teachers also participate in a one-day training course on how to use the PH International-developed supplemental textbooks and schools receive supplemental textbooks for each student enrolled in the 9th or 10th grades. Teachers are offered professional development opportunities, such as participation in open lessons, regional roundtables, and national civics conferences. The main focus of this intervention is on giving teachers tools to 2 This “core” training includes national curriculum requirements and modules, which cover topics and tools included in the national curriculum, teacher professional standard, and other documents of the MOES. These topics include the role of school in civics teaching, identification and development of additional teaching materials, assessment, project-based teaching, and interactive teaching methodologies. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 14 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 enhance their ability to give effective classroom instruction without relying on much external support throughout the school year. Students in six regions were involved in social media trainings and students in five additional regions received training by January 2016. While not all students received this training, the intent was for those who did to return to their schools and train their peers. Many students also took part in other social media activities (e.g., Facebook quizzes, online conferences, blog contests) and took part in civics camp during summer 2015. Schools receiving this intervention had access to 20 small grants per year for classroom or school improvements; teachers were responsible for grant applications. Voluntary Civics Clubs This intervention provides schools with NGO support to organize voluntary student civics clubs. Teachers received a one-day training on how to use the civics club toolbox, “Instructions for the Work of School-based Civics Clubs.” Each school received 20 copies of the toolboxes for students, two for teachers, and one for the school library. This intervention aimed to replicate the model of the previous USAID-supported Applied Civic Education and Teacher Training (ACETT) project with some modification in the role of NGOs in supporting activities of student clubs by not specifying the number of deliverables from the NGOs, but rather allowing the students to guide the process. Schools received help from local NGOs to facilitate the establishment and functioning of school-based civics clubs and to assist with planning and implementation of civics clubs’ projects as needed. The civics clubs had student leaders who carried out activities with the assistance from the teacher-facilitator and an NGO. For example, if students planned a project that required advocacy of some sort, the NGO offered them training in advocacy skills. Student participation in civics clubs was entirely voluntary and does not count toward their final evaluation in the civics class. The civics clubs were driven largely by student leaders, but with teacher-facilitator and NGO assistance. Students in the clubs identified a problem in their communities, determined how they could themselves address the problem, tried to mobilize partners among local decision-makers/agencies, and carried out activities to address the problem. Additionally, students from school civics clubs participated in group visits to state and non-state agencies and media outlets, meetings with expert guests, and trainings organized by partner NGOs on civic education topics. Schools receiving this intervention had access to 20 small grants per year to support the civics clubs; students were responsible for initiating grant applications. Class Civics Projects This intervention includes a one-day training for teachers on how to lead classroom projects and use the toolbox “Instructions for Classroom Projects in Civic Education.” The toolbox give guidance on how to use a six-step process to implement a classroom project: investigation, preparation, action, reflection, demonstration, and evaluation. Schools receive toolboxes for all students enrolled in 9th grade and 10th grade, plus two for teachers and one for the library. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 15 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Each civics teacher was asked to carry out at least one classroom project during the academic year with the class that she or he taught, with participation from all students in the class. Partner NGOs provided the needed resources for implementation of classroom projects, when requested by teachers or occasionally by students if the teacher was passive. Service learning was conducted as a class project in which all students in the class participated. Each class completed projects as a class; teachers did not break up the class into smaller groups of students or encourage students to pursue individual projects. Since the types of projects were defined by the teachers with student input and interest taken into account, the only limit on the “type” of project was that it had to be connected to the curriculum. Twenty additional small grants to support classroom projects were awarded in this intervention each year. NGOs played similar kinds of project backstopping roles in this intervention as they did in the clubs intervention. NGOs supported implementation of classroom projects based on the needs identified by a teacher leading a particular project with her/his students. For example, if transportation was needed, NGOs arranged it for all students in the class, or if some stationery was needed, the NGO provided it. Additionally, if meeting with state or non-state agencies or expert guests related to the classroom project, then the NGO facilitated such meetings. 3.2 IMPACT EVALUATION VARIATIONS MT offered three intervention bundles or variations to schools participating in the IE. These variations allow implementing partner PH International and the MOES of Georgia to know which civic education curricula, applied learning techniques, or combination of pedagogical tools work best and for which students. These variations are based on the best practices of high school civic education programs internationally and in Georgia. They reflect the original intentions of MT and maximize the value of MT for policy decisions made by the Government of Georgia. The bundles benefited students enrolled in the 9h and 10th grades in participating schools.  Variation 1 (V1), enhanced teacher training and supplementary civics curricula: Schools assigned to this variation had their teachers trained in enhanced teaching techniques and use of supplementary textbooks on civics topics as described above.  Variation 2 (V2), voluntary civics clubs plus enhanced teacher training and supplementary civics curricula: This variation includes V1 plus teacher training in developing civics clubs along with NGO support as described above.  Variation 3 (V3), mandatory class civics projects plus enhanced teacher training and supplementary civics curricula: This variation includes V1 plus a teacher training on how to lead classroom projects along with NGO support as described above. These three variations respectively represent a pure traditional classroom model, a traditional classroom and voluntary experiential learning hybrid, and a traditional classroom and compulsory experiential learning hybrid. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 16 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 4. EVALUATION QUESTIONS AND HYPOTHESES 4.1 EVALUATION QUESTIONS In line with the IE Design Report (Ilirjani and Inman 2015, see Annex 1), this IE report answers the following evaluation questions: 1. Evaluation Question 1 (EQ1): Does civic education programming increase students’ civic, democratic, and prosocial attitudes? What is the relative effectiveness of the three civic education variations? 2. Evaluation Question 2 (EQ2): Does civic education programming increase students’ civic, democratic, and prosocial behaviors? What is the relative effectiveness of the three civic education variations? 3. Evaluation Question 3 (EQ3): Does civic education programming contribute to closing the democratic achievement gap between students of different genders, ethnoreligious groups, and socioeconomic status (SES)? Are these effects different across the three civic education models? The IE report also addressed two additional evaluation questions: 4. Evaluation Question 4 (EQ4): Does receiving two years of civic education programming produce greater effects than one year of programming? 5. Evaluation Question 5 (EQ5): Does civic education programming have similar effects among 9th-graders and 10th-graders? 4.2 HYPOTHESES3 Classroom instruction remains the main venue for delivering civic education to students. This environment has many advantages in terms of reaching many students and holding them accountable for acquiring knowledge. However, such instruction must be interactive and incorporate democratic principles into the functioning of the classroom to have positive and lasting effects. Teachers must be trained in interactive teaching methods and encourage open discussion of civic principles, especially those that may be controversial. In addition to acquiring formal knowledge about local governance and the principles of democracy, students learn by 3 In addition to the hypotheses presented here, the IE Design Report also included hypotheses related to ‘knowledge.’ In the political behavior and public opinion literature, political knowledge refers to factual knowledge about the way a polity works and about current political events (e.g., how long is the presidential term and who the current president is, respectively). The questionnaire contains no questions about political knowledge so any hypotheses related to this construct cannot be assessed. The literature review references public health and public education studies that analyze ‘civic knowledge.’ However, the measures of civic knowledge in these studies (e.g., following a definition of ‘public debate,’ respondents are asked for their views on how can public debate benefit society; Schultz et al. [2010]) are attitudes at their core. The questionnaire does contain questions whose likely purpose was to measure this type of civic knowledge. However, given that the IE Design Report provides no specific guidance on which items should be included in a scale measuring this construct, they are treated as attitudes in the analysis. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 17 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 emulating their teachers and peers. If teachers model behavior that is accepting of opinions that differ from their own, then students will be more likely to accept such differences as well. Requiring students to think critically about what it means to be an active citizen in a democracy is also essential for effective learning. When it comes to promoting desirables civic and political behaviors (e.g., civic engagement, voting, and other forms of political participation), experiential leaning is likely to make a bigger difference than traditional classroom instruction. Community-oriented service projects, for example, give students a template for other forms of engagement and future participation. Encouraging students to care about a particular community problem and involving them in the process of resolving that problem, allows them to develop stronger ties to their community and encourages them to take an interest in improving it by engaging in community service, voting for political leaders who will pursue policies that help their communities, petitioning government officials for the enactment of such policies, or partaking in other forms of political participation. Engaging students to participate in resolving issues within their communities is likely to create the momentum to continue their participation. Open, democratically-run classrooms and interactive teaching facilitate the shaping of prosocial attitudes, like social trust and tolerance, but so, too, does putting civic principles in practice through applied learning tools measures. Students who take part in student-led community service projects are more likely to see that they can bring about positive change in their communities as well as on a larger scale when they become engaged in civil society. Research suggests that these activities work best when tied to learning objectives as opposed to service for service’s sake. It also asserts that student-organized activities have greater effects than those lead by teachers. The research is mixed about whether compulsory and voluntary service projects have better outcomes for social trust. Voluntary projects have student buy-in and may increase motivation, but they suffer from lack of participation by those from under-resourced backgrounds, those who may not feel comfortable volunteering, and those whose parents do not want them participating in after-school activities. Compulsory projects do not get as much buy-in and may suffer from lower participant motivation, but they include many more segments of the population, including those who might be left out of a voluntary project. Regardless of the type, civics projects that are democratically run allow students to gain confidence in their ability to effect change and are likely to increase social trust. The discussion suggests that when it comes to changes in social and political attitudes, all three variations should result in improvements, but it is not clear which variation, if any, should be more effective. When it comes to producing desirable changes in civic and political behaviors, the variations that include an experiential learning component (V2 and V3) should be more effective than the variation that includes classroom learning only (V1). The discussion also suggests that the variations that include an experiential learning component (V2 and V3) should be more effective than then the variation that includes classroom learning only (V1) in closing the democratic achievement gap. These expectations are formalized as follows: Hypothesis 1 (H1): Specialized teacher training in interactive techniques and civic education principles improves students’ civic, democratic, and prosocial attitudes. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 18 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Hypothesis 2 (H1): Experiential learning, either voluntary or compulsory, paired with specialized teacher training in interactive techniques and civic education principles will result in greater improvements in students’ civic, democratic, and prosocial attitudes than specialized teacher training alone. Hypothesis 3 (H3): Specialized teacher training in interactive techniques and civic education principles improves students’ civic, democratic, and prosocial behaviors. Hypothesis 4 (H4): Experiential learning, either voluntary or compulsory, paired with specialized teacher training in interactive techniques and civic education principles will result in greater improvements in students’ civic, democratic, and prosocial behaviors than specialized teacher training alone. The preceding discussion and literature review do not offer clear guidance when it comes to developing expectations about the relative effectiveness of Variation 2 and Variation 3. Variation 2 is more likely to result in greater improvements among voluntary civics clubs participants, but absolute effectiveness will depend on the proportion of students who decide to participate. Variation 3 is likely to produce a smaller effect among participants, but all students assigned to this variation should participate. Given this uncertainty, no hypotheses about this issue are made explicit, though the issue is addressed in the analysis.4 Those in more privileged positions within society will likely have higher initial levels of desirable civic and political attitudes and behaviors, exposing all to a civic education programming should contribute to closing the gap between the privileged and the less so. As indicated by research question 5, the IE analyses three types of gaps: (1) those between boys and girls; (2) those between Georgian and Orthodox majority students and ethnoreligious minority students (i.e., those whose mother tongue is not Georgian or whose religion is not Orthodox); and (3) those between students from high socio-economic status and their less privileged peers. While all variations might contribute to closing the democratic achievement gap, Variation 3 should be particularly effective given its compulsory experiential component. The following six hypotheses summarize this discussion: Hypothesis 5 (H5): Civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among girls than among boys. Hypothesis 6 (H6): Compulsory experiential learning paired with specialized teacher training in interactive techniques and civic education principles will result in the greatest improvements among girls relative to boys. Hypothesis 7 (H7): Civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among students from ethnoreligious minorities than among other students. 4 As explained in section 5.4 below, the analysis focuses on the intent-to-treat (ITT) effect of V2. In plain words, the analysis estimates the effects of V2 using information from all students assigned to this variation and without differentiating between voluntary civics clubs participants and non-participants. Because it is not possible to identify the effect of V2 among participants using random assignment alone, some hypotheses related to this effect that were included in the IE Design Report are not considered here. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 19 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Hypothesis 8 (H8): Compulsory experiential learning paired with specialized teacher training in interactive techniques and civic education principles will result in the greatest improvements among students from ethnoreligious minorities relative to other students. Hypothesis 9 (H9): Civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among low-SES students than among other students. Hypothesis 10 (H10): Compulsory experiential learning paired with specialized teacher training in interactive techniques and civic education principles will result in the greatest improvements among low-SES students relative to other students. Regarding the question about length of exposure (i.e., receiving two years of civic education programming rather than a single year), intuition suggests than two years of civic education programming should be more effective than one year. Lastly, regarding the question about timing of exposure (i.e., receiving civic education in 9th grade vs. 10th grade), the political socialization literature suggests that earlier exposure should be more effective that later exposure, though this literature compares coarser periods in the life cycle (e.g., childhood vs. adolescence vs. early adulthood). Hence the following hypotheses:5 Hypothesis 11 (H11): Two years of civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors than one year of civic education programming. Hypothesis 12 (H12): Exposure to civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among students in the 9th grade than among students in the 10th grade. Statistically, for any given outcome measure, the hypotheses laid out below could be used to address questions about absolute effectiveness (i.e., comparison of a given variation to control) and relative effectiveness (i.e., comparisons across variations). When presenting results in section 6, the report tailors statistical hypothesis tests to the needs of the research questions and their corresponding theoretical hypotheses (H1 through H12). 1. V1 has a significant effect on the outcome; this effect is estimated by comparing students assigned to V1 and those assigned to receive no programming. 2. V2 has a significant effect on the outcome; this effect is estimated by comparing students assigned to V2 and those assigned to receive no programming. 3. V3 has a significant effect on the outcome; this effect is estimated by comparing students assigned to V2 and those assigned to receive no programming. 5 Hypothesis 11 was included in the IE Design Report even though no question about length of exposure was included. Hypothesis 12 and its corresponding evaluation question (EQ5) were not originally included in the IE Design Report. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 20 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 4. For EQ1 and EQ2: The effect of the variations that include experiential learning, V2 and V3, is significantly higher that the effect of the variation that does not include experiential learning, V1; a joint test is performed by comparing the estimated effects of V2 and V1, and of V3 and V1. 5. For EQ3: The effect of the compulsory experiential learning variation, V3, on disadvantaged subgroups is significantly higher than the effect of the variations that do not include compulsory experiential learning, V1 and V2; a joint test is performed by comparing the estimated effects of V3 and V1, and of V3 and V2 for the three subgroups. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 21 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 5. METHODOLOGY 5.1 IMPACT EVALUATION DESIGN The IE followed a randomized controlled trial (RCT) design whereby eligible schools were randomly assigned to receive one of the three variations or to a control group that received no MT programming. As previously explained, these three variations respectively represent a pure traditional classroom model, a traditional classroom and voluntary experiential learning hybrid, and a traditional classroom and compulsory experiential learning hybrid. Table 1 summarizes the intervention components that students assigned to each variation received. Table 1. Summary of MT Civic Education Variations Interventions V1 V2 V3 Contro l Enhanced teacher training and supplementary civic curricula X X X Voluntary civics clubs X Class civics project X MT implementers intended to target 480 secondary schools out of a total of 2,329 schools in Georgia. The interventions were to be offered in all regions of the country. Per the recommendation of the implementer, the IE design team adopted a clustered randomization approach. This approach consisted in grouping all 2,329 high schools into 368 school districts with the help of the regional implementing partners. These school districts served as the unit of randomization, with all schools in a given district being assigned to the same group (V1, V2, V3, or Control) to mitigate spillover effects. The criteria for including a given school district in the IE was to have at least one school with at least 10 students in 9th grade and at least 25 students in the 9th and 10th grades combined. This was determined to be the minimum number of students needed for purposes of statistical power. Of the 368 school districts, only 252 satisfied these criteria and were assigned to one of the three variations or to the control group.6 MT implementers sought to offer civic education programming to all schools included in the IE and according to the randomization schedule. However, ten schools had to be dropped from the study at inception due to administrative reasons. In addition, one school that agreed to receive MT programming refused to take part in the baseline survey. In the end, a total of 241 schools in 237 school districts were included in the first wave of data collection. In subsequent rounds of data collection, a few schools had to be replaced because they lost accreditation and thus became no longer eligible to receive MT programming or because they refused to participate in the surveys. The IE team attempted to replace each of these schools with schools of comparable size from the same region that were assigned to the same group. Between the first and second rounds of data collection, three schools were replaced, but an additional school that lost accreditation was not, bringing the total number of schools included 6 It is worth noting that no school districts in the Racha-Lechkhumi region met the criteria for inclusion in the IE, but school therein will be invited to participate in the MT project. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 22 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 in the IE to 240. And additional three schools had to be replaced between the second and third rounds of data collection. Table 2 provides the number of schools assigned to each group broken down by region. Table 2. Number of Schools per Group, by Region Region V1 V2 V3 Control Total Adjara 4 5 4 5 18 Guria 2 3 2 2 9 Imereti 12 12 11 12 47 Kakheti 6 7 6 6 25 Kvemo Kartli 6 6 7 7 26 Mtskheta-Mtianeti 2 2 2 1 7 Samegrelo-Zemo Svaneti 6 6 6 5 23 Samtskhe-Javakheti 3 3 3 3 12 Shida Kartli 6 7 6 7 26 Tbilisi 12 12 12 11 47 Total Schools 59 63 59 59 240 One potential problem that the design attempted to avoid is spillover effects, in which students from the different variations or control group share their knowledge from class with one another. One way the design attempted to mitigate spillover is by assigning each school district to the same variation or control group, so that schools that are near one another receive same programming. There is some risk that students from schools receiving one variation transferred at some point to a school receiving another variation or no programming. While it is likely this occurred, it would be nearly impossible for the researchers to track since it is illegal to collect identifying student information. However, there is no reason to believe that such transfers would be systematic and introduce bias. It is also possible that some students may be exposed to other civics programming by PH International prior to the MT project or that they may have other contact with students of different variations over the two years of the IE. Again, exposure and contact would be unlikely to be systematic and introduce bias. 5.2 DATA COLLECTION The IE team, NORC, USAID, and implementing partner PH International developed the survey instrument in coordination with the Government of Georgia’s MOES over several months; the process started in July 2015 and ended in January 2016. Because of various delays in finalizing the survey, no baseline data was collected for the first year (Y1) of the IE. Wave 1 of the data collection, which captured endline data for Y1, was administered from April 25th until June 6th, 2016 in 241 schools throughout the country. A total of 8,621 students in the 9th and 10th grades completed the survey in Wave 1. Of those participating in the survey, 25 percent of students attended schools receiving V1; 28 percent of students attended schools receiving V2; 23 percent of students attended schools receiving V3; and 24 percent of students attended schools receiving no programming (control group). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 23 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Wave 2 of the survey was administered between October 9 and December 9, 2016. While these wave was supposed to collect baseline data for the second year of the IE (Y2), the national elections occurred on October 8 and a decision was made to delay data collection until immediately after the elections to avoid any appearance of influencing the election results. During Wave 2, 8,937 students completed the survey, of which 25 percent attended schools receiving V1; 27 percent attended schools receiving V2, 24 percent attended schools receiving V3, and 24 percent students attended schools receiving no programming (control group). The final wave of data collection, Wave 3, captured endline data for Y2. It included 8,742 students who completed the survey between April 3 and June 5, 2017, of which 26 percent attended schools receiving V1; 27 percent attended schools receiving V2; 24 percent attended schools receiving V3; and 23 percent students attended schools receiving no programming (control group). Due to laws in Georgia preventing the collection of students’ private information, the IE was not able to track individual student performance over time. Nevertheless, the data do track the same schools and grades across the three waves of data collection. Given that there are two grades receiving programming over the course of two years and three waves of data collection, the structure of the data is somewhat complex. Students who were in 9th grade during Y1 moved up to the 10th grade for Y2. This group of students thus received two years of MT programming (one as nine-graders and the other as 10th-graders) and is included in all three waves of data collection. Students who were starting 9th grade when they participated in Wave 2 of data collection are the only students for whom there is data that is close to being baseline, since they had received no prior civics curriculum. They were still in 9th grade when they were surveyed again at the end of the year in Wave 3 (Y2 endline). For the students who were in 10th grade during the first year of the IE, there is only one measure from Wave 1 (Y1 endline). They then dropped out of the study as they entered 11th grade. Table 3 (overleaf) illustrates the structure of the data. One cohort of students (highlighted in light blue) consists of those who were enrolled in 9th grade during Y1. A second cohort of students (highlighted in light grey) consists of students who were enrolled in 9th grade during Y2. Student who were enrolled in 10th grade during Y1 were only surveyed once (no highlighting). Wave 2 data is referred to as pseudo-baseline data for Y2 because, as explained above, these data were collected after programming had already started. Table 4 (overleaf) provide more detailed information by breaking down the number of students interviewed by groups, grades, and waves. As in Table 3, light blue highlighting is used to indicate when cell entries correspond to the cohort of students who was enrolled in 9th grade Y1 and continued to be followed during Y2. Light grey highlighting indicates that cell entries correspond to cohort of students who was enrolled in 9th grade during Y2. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 24 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 3. Structure of the Data Wave 9th Grade 10th Grade Wave 1 (Y1 Endline) 4,999 4,275 Wave 2 (Y2 Pseudo-baseline) 4,998 4,433 Wave 3 (Y2 Endline) 4,467 4,275 Note: Light blue highlighting indicates that cell entries correspond to the cohort of students who was enrolled in 9th grade during Y1 and continued to be followed during Y2. Light grey highlighting indicates that cell entries correspond to the cohort of students who was enrolled in 9th grade during Y2. Table 4. Summary of Data Collected by Group, per Grade and Wave Wave 9th Grade 10th Grade V1 V2 V3 C V1 V2 V3 C Wave 1 (Y1 Endline) 1,061 1,255 1,098 1,085 1,070 1,167 918 967 Wave 2 (Y2 Pseudo-baseline) 1,134 1,286 1,018 1,060 1,146 1,176 1,104 1,011 Wave 3 (Y2 Endline) 1,179 1,200 1,043 1,045 1,131 1,155 1,029 960 Note: Light blue highlighting indicates that cell entries correspond to the cohort of students who was enrolled in 9th grade during Y1 and continued to be followed during Y2. Light grey highlighting indicates that cell entries correspond to cohort of students who was enrolled in 9th grade during Y2. All data collection for the IE was carried out by Caucasus Research Resource Center in Georgia (CRRC/Georgia) under subcontract with NORC at the University of Chicago, with enumerator training and oversight provided by NORC. The survey questionnaire is included as Annex 2 to this report. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 25 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 5.3 OUTCOMES MEASURES7 The survey included a total 135 substantive items, which were often organized in testlets, that is, groups of items under a common instruction or question and response scale. The outcome measures analyzed in this report were developed following a theoretically-informed, data-driven process. The process started by classifying items according to the level of change they intent to measure; outcome or impact. Outcome-level items are those gauging behaviors and attitudes about objects inside school that are likely to change in the short and medium term as a direct result of the interventions. Impact-level items are those gauging behaviors and attitudes about objects outside school that are also likely to change, but in the medium and long term. Within each level, items were further classified into two groups; those that measure attitudes and those that measure behaviors. Lastly, impact-level items gauging behaviors outside school were further subdivided into two groups; those that measure current behaviors and those that measure future behaviors. To summarize, this classification resulted in the following five groups of items:  Outcome-level items gauging attitudes about objects inside school.  Outcome-level items gauging behaviors inside school.  Impact-level items gauging attitudes about objects outside of school.  Impact-level items gauging current behaviors outside school.  Impact-level items gauging future behaviors outside school. Next, factor analysis was conducted for each of these groups of items to reveal their underlying structure. In short, the goal of factor analysis is to uncover the set of latent, unobserved variables that account for a large set of observed items. Given that items analyzed have an ordinal response scale, the factor analysis was performed on the polychoric correlation matrix. All items were rescaled so that higher response scores corresponded to desired outcomes, and only data from schools assigned to the control group was used to guard against the possibility that the MT variations could produce changes in the underlying structure of the items. For each factor analysis, factors with the highest eigenvalues were retained for scale construction. Factors with eigenvalues higher than one and which could easily be viewed as corresponding to a construct that should be impacted by civic education were also retained. 7 The IE Design Report listed eight ‘core civic education indicators’ as the main outcome variables to be examined; civic behaviors, civic skills, conventional and alternative civic engagement, political efficacy, citizenship types, political conversation and critical thinking, values and personal beliefs, and assessments of school climate. Rather than examining those eight indicators as outcome measures, this report constructs alternative ones through the process that is described in this section. Examining the original eight indicators is not possible because of two reasons. First, the IE team initially proposed a survey with more items; however, several items were removed as a consequence of negotiations with the MOES and internal editing. Second, the IE Design Report included no guidance on how to map the survey items onto the eight indicators. Note that the process followed here produced outcome measures that are similar to the ones originally proposed in the Design Report. The outcomes measures developed below allow for addressing all the evaluation questions and assessing all the hypotheses laid out in section 3. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 26 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 One factor was retained for outcome-level items gauging attitudes about objects inside school, for impact-level items gauging current behaviors outside school, and for impact-level items gauging future behaviors outside school. Two factors were retained for outcome-level items gauging attitudes about objects inside school; and four factors were retained for impact-level items gauging attitudes about objects outside of school. The final step consisted in producing the outcome measures using factor-based scores, after rescaling all items so that they ranged from 0 to 1. After factor-based scores were calculated, their reliability was checked using Cronbach’s alpha. At this stage, specific items were removed from a score whenever doing so improved scale reliability. Factor-based scores are calculated by averaging the items that load highly into the retained factor, while more traditional factor scores are obtained from weighting the different items by their factor loadings. Using factor-based scores as outcome measures in the analysis is justified for two reasons: First, when the factor loadings of included items are relatively homogenous as is the case here, the two procedures result in equivalent scores. Second, the factor score method produces a missing score for each student that has one or more items with missing values, while factor-based scores are calculated disregarding missing values. By doing this, missingness in the outcome variables is minimized while using as much information as possible from the original dataset.8 The resulting nine measures are listed below. The survey also included a measure of uptake of the experiential learning components offered as part of V2 and V3 (survey item Q27). It asked students whether, as part of a civics class, they had worked on a civics or volunteer project in the last two years. This uptake-level outcome measure is included in the analysis below, for a total of ten outcome measures. Annex 3 presents the descriptive statistics for these measures.  Outcome-level measures: o Attitudes related to school government and politics (survey items included in this scale are Q13a Q13b Q13c Q13d Q13e). o Civic behaviors and teachings in the classroom (survey items included in this scale are Q12b Q12c Q12e Q32e Q33a Q33b Q33c Q33d Q33e Q33f Q33g Q33h Q37a Q37b Q37c Q37d Q37e Q37f). o Engagement in school government and politics (survey items included in this scale are Q11c Q11d Q11e Q11f).  Impact-level attitudes measures: o Prosocial and democratic attitudes (survey items included in this scale are Q14a Q14d Q17a Q17b Q17d Q18a Q18b Q18c Q24a Q34a Q36a Q36b Q36c Q36d Q36e Q35c). 8 In addition, it has been shown that using factor scores as dependent variables produces inconsistent estimates, at least in the context of structural equation models. See, e.g., Skrondal and Laake (2001). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 27 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 o Attitudes about corruption and dictatorship (survey items included in this scale are Q34b Q34c Q35a Q35b). o Internal political efficacy and beliefs about citizenship (survey items included in this scale are Q15a Q15c Q15d Q15e Q15f Q15g Q15h Q15i Q15j Q15k Q15l Q15m Q15n Q16b Q16c Q16d Q22* Q24b Q29a Q29b Q29c Q29d). o External political efficacy (survey items included in this scale are Q25a, Q25b, Q25c).  Impact-level behavior measures: o Current civic engagement and practices (survey items included in this scale are Q9a Q9b Q9c Q10a Q10b Q10c Q10d Q31a Q31c). o Future civic engagement and political participation (survey items included in this scale are Q19a Q19b Q19c Q19d Q20a Q20b Q20c Q20d Q20e Q20f Q21a Q21b Q21c Q21d Q21e). 5.4 ESTIMATION The effects of the three variations on a given outcome measure are estimated using the following equation: where 𝑂𝑂𝑂𝑂𝑖𝑖𝑖𝑖 is the outcome variable corresponding to student i in school j; 𝑉𝑉1, 𝑉𝑉2, and 𝑉𝑉3 are indicator variables identifying the variation to which school j was assigned; 𝛽𝛽0 estimates the mean of the outcome measure among students in control schools; 𝛽𝛽1, 𝛽𝛽2, and 𝛽𝛽3 respectively estimate the effects of V1, V2, and V3; X is a vector of control covariates with it corresponding vector of coefficients, θ, and 𝜀𝜀 is a student-specific error term. Equation 1 is estimated using ordinary least squares (OLS) and statistical inference is based on standard errors clustered at the school level to account for intra-class correlation. Note that Equation 2 identifies V2’s intent-to-treat (ITT) effect (i.e., its effect among all students enrolled in schools assigned to this variation, regardless of whether they participated in the voluntary civics clubs) rather than its average treatment effect on the treated (ATET) (i.e., its effect among students who ultimately participated in the voluntary civics clubs). The ITT effect is of chief interest here because it provides information about the changes that donors and implementers can expect to effect by implementing programming similar to V2.9 While the ATET of V2 is also of interest, it cannot be identified by randomization alone. 9 It is worth noting that, to a certain extent, the analysis also estimates ITT effects for V1 and V3. That is because effects are estimated without attempting to determine if and account for students who may have skipped civics class, not received the instruction for any other reason (e.g., teachers refused to implement what they learned in MT-provided training), or decided not to participate in in-class civics projects in the case of V3. From a practical perspective, this kind of information would likely be very difficult to obtain. From an analytic perspective, and in line with what was argued for the effects of V2, ITT effects are of chief interest here. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 28 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Estimates of 𝛽𝛽1, 𝛽𝛽2, and 𝛽𝛽3 directly assess the hypotheses related to each variation’s absolute effectiveness. In turn, additional hypothesis tests to test the equality of a given pair of estimated coefficients (e.g., 𝛽𝛽1 = 𝛽𝛽2 or 𝛽𝛽1 = 𝛽𝛽3) are used to assess the hypotheses about the relative effectiveness of the variations. Lastly, Equation 1 can be modified by interacting the variation variables with other variables to assess the various hypotheses related to heterogeneous effects. Students’ scores at baseline were to be used for covariate adjustment. Since it was not possible to collect baseline data, the analysis below uses a set of covariates that are time-invariant or expected to change little or not at all as a result of civic education programming, and are known to be correlated with civic, democratic, and prosocial attitudes and behaviors. The covariates used are gender, an ethnoreligious minority indicator, and an index of SES. 10 Grade and the number of students surveyed in a given student’s school are also included as adjustment covariates. The latter variable is used as a proxy for the size of the i-th student’s school. Handling of Missing Values Missingness is not a concern in the case of outcome measures. The proportion of missing values is very low—it is 3.4 percent for the external political efficacy measure and less than 0.4 percent for the other outcome measures. Moreover, there is no evidence that missingness in the outcome measures is correlated with assignment to the various groups (i.e., V1, V2, V3, and Control), as indicated by the results of a multinomial logistic regression of variation assignment.11 In the case of the control covariates included in the survey (i.e., gender, age, native language, religion, father’s education, mother’s education, and ownership of five assets), the proportion of missing values is particularly high for the mother’s and fathers’ education variables (16.7 percent and 18.8 percent, respectively). While there is no indication that missingness in control covariates is correlated with assignment to the various groups, 12 missingness is a concern here because case-wise deletion would render 28 percent of the data unusable. Multiple imputation (MI) is used to create complete datasets. Given that most covariates are nominal or ordinal, conditional MI is used (Kropko et al. 2017). This procedure models each variable conditional on all others, with imputation being implemented sequentially via chained equations. Following standard practice, MI is used to generate ten imputed, complete datasets. 10 The ethnoreligious minority indicator identifies students whose mother tongue is not Georgian or whose religion is not Orthodox. The index of SES is the first principal component obtained from performing polychoric principal components analysis of the following covariates: father’s education, mother’s education, and ownership of five assets (motor vehicle, television, personal computer or laptop, flat or house, and washing machine). The survey also captured age, but this covariate is not used for adjustment because it is highly correlated with grade. 11 The regression includes a set of indicator variables (one for each outcome measure) identifying cases with missing values as predictors of variation assignment. Following estimation, it is not possible to reject the null hypothesis of joint orthogonality (i.e., all estimated coefficients are equal to zero) with p=0.9912. 12 Following the estimation of a multinomial logistic regression of variation assignment with a set of indicator variables (one for each control covariate) identifying cases with missing values as predictors, it is not possible to reject the null hypothesis of joint orthogonality with p=0.7946. A similar regression with a single indicator variable identifying cases with a missing value on at least one control covariate leads to the same conclusion; in this case it is not possible to reject the null hypothesis of orthogonality with p=0.9440. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 29 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Balance Tests Balance tests were conducted to check if the randomization had produced groups with balanced control covariates. The test consisted of estimating a multinomial logistic regression of variation assignment with all previously mentioned control covariates (i.e., gender, age, native language, religion, father’s education, mother’s education, asset ownership, grade, and number of students surveyed). Models were estimated with both the original data (using case-wise deletion) and the multiply-imputed data. The results indicate that random assignment did not produce balanced groups. The results of these regressions are presented in Annex 4. Following estimations with the original and the multiply-imputed data, the null hypothesis of joint orthogonality is rejected with p=0.02 and p=0.06, respectively. While random assignment can produce unbalanced groups due to chance, it is not possible to rule out that imbalances are due to improper randomization (either at design or implementation). To minimize the possibility of bias, Equation 1 is estimated using inverse probability weighting (IPW).13 This approach consists of two steps. First, model the selection process into the various groups and estimate the probability (or propensity score) of being assigned to a given group for each student. Then, the inverse of the predicted probability of being assigned to the group to which subject i was indeed assigned to (𝐼𝐼𝐼𝐼𝐼𝐼𝑖𝑖𝑖𝑖 = 1/𝐼𝐼𝑖𝑖𝑖𝑖) is used to estimate Equation 1 via weighted OLS. 14 Multiple Hypothesis Testing Following regression, hypothesis tests are used for a given outcome measure to examine whether each variation has a significant effect when compared to the control as well as to detect significant differences in the magnitudes of these effects. Such multiple hypothesis testing increases the probability of making Type I errors—i.e., falsely detecting a non-existing effect (or difference between effects). The false discovery rate (FDR) is used to control the expected proportion of false discoveries in the analysis below. Results tables include q-values, which are the minimum FDR at which each test is significant (Benjamini and Hochberg, 1995). 13 Three alternatives to estimating Equation 1 with IPW (Specification 4) were also considered: a naïve comparison (i.e., Equation 1 with no control covariates) (Specification 1); Equation 1 (Specification 2); and a naïve comparison with IPW (Specification 3). Overall, covariate adjustment resulted in similar or smaller point estimates and greater precision compared to the naïve alternative, both with and without IPW (Specification 1 vs. Specification 2, and Specification 3 vs. Specification 4, respectively). Estimations with and without IPW (Specification 4 and Specification 2, respectively) produced very similar results, but the former (Specification 4) produced slightly more conservative results, and was therefore chosen for presentation below. 14 Predicted probabilities are estimated using a multinomial logistic regression with age, gender, native language, religion, father’s education, mother’s education, asset ownership, grade, and number of students surveyed as predictors. Following Mitra and Reiter (2012), the following procedure is employed to implement IPW after MI: First, estimate separate multinomial logit models of treatment assignment for each of the ten multiply-imputed datasets and used to predict group assignment probabilities. And, second, calculate weights after averaging the ten predicted probabilities. Similarly, control covariates’ averages across the ten multiply-imputed datasets are used for estimating Equation 1. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 30 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 6. RESULTS The subsections below report the estimation results addressing the evaluation questions laid out in section 4. The first subsection addresses the first two evaluation questions, namely does MT civic education programming increase students’ civic, democratic, and prosocial attitudes (EQ1) and behaviors (EQ1)? The second subsection examines EQ3 pertaining to the contribution of MT programming to closing the democratic achievement gap between students of different genders, ethnoreligious groups, and SES. The third and fourth subsections respectively address questions related to the effects of two years vs. one year of programming (EQ4) and the effects of programming among 9th-graders and 10th-graders (EQ5). All subsections also address questions related to the relative effectiveness of the three civic education variations. All estimations use endline data collected at the end of each year of MT (i.e., data from Wave 1 and Wave 3). 6.1 EVALUATION QUESTIONS 1 AND 2 This section addresses EQ1, Does civic education programming increase students’ civic, democratic, and prosocial attitudes? What is the relative effectiveness of the three civic education variations? and EQ2, Does civic education programming increase students’ civic, democratic, and prosocial behaviors? What is the relative effectiveness of the three civic education variations? Table 5 (overleaf) presents the theoretical hypotheses corresponding to these questions as well as the statistical hypothesis tests used to examine them for a given outcome measure. The expectation is that any given variation of MT programming should result in desired changes in attitudes and behaviors, and that variations including experiential learning (V2 and V3) should be more effective than the variation including only specialized teacher training (V1). Table 5. Hypotheses and Hypothesis Tests for EQ1 and EQ2 Theoretical Hypotheses Statistical Hypothesis Tests Specialized teacher training in interactive techniques and civic education principles improves students’ civic, democratic, and prosocial attitudes (H1)  Effect of V1 > 0  Effect of V2 > 0  Effect of V3 > 0 Experiential learning, either voluntary or compulsory, paired with specialized teacher training will result in greater improvements in students’ attitudes than specialized teacher training alone (H2) Joint test of:  Effect of V2 > Effect of V1  Effect of V3 > Effect of V1 Specialized teacher training in interactive techniques and civic education principles improves students’ civic, democratic, and prosocial behaviors (H3)  Effect of V1 > 0  Effect of V2 > 0  Effect of V3 > 0 Experiential learning, either voluntary or compulsory, paired with specialized teacher training will result in greater improvements in students’ behaviors than specialized teacher training alone (H4) Joint test of:  Effect of V2 > Effect of V1  Effect of V3 > Effect of V1 IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 31 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 6 (overleaf) presents the estimation results. The table reports the outcome measures’ estimated means after regression adjustment for the control schools (“Control” row) as well as the estimated differences between these means and the means for the groups receiving the MT variations (“V1,” “V2,” and “V3” rows). These differences are the impact estimates. To calculate an outcome measure’s estimated mean for a given MT group, the difference for said group is added to the mean for the control group. For example, the results for uptake indicate that 57 percent of students in control schools participated in civics or voluntary projects, while participation in V1 and V2 schools was 70 percent (57 percent+13 percent) and participation in V3 schools was 77 percent (57 percent+20 percent). As indicated in the table’s notes, asterisk are used to indicate conventional statistical significance (p-values) and circumflex accents are used to indicate statistical significance after adjustment for multiple hypotheses testing (q￾values). Uptake Each of the three variations had a significant effect on participation in civics or voluntary projects after adjustments for multiple hypothesis testing. Participation in V1 and V2 schools was 22.8 percent higher than in control schools; in V3 school, participation was 35.1 percent higher than in control schools. As expected, the largest effects are observed in V3 schools, where required classroom-based civics projects were implemented. Not surprisingly, the joint test of differences in effects indicates that the experiential learning variations (V2 and V3) led to greater participation in projects than the training only variation (V1). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 32 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 6. Effects of Civic Education Programming Independent Uptake Outcome Impact - Attitudes Impact - Behaviors Variables (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Control 0.57*** (0.02) 0.74*** (0.01) 0.67*** (0.01) 0.31*** (0.01) 0.68*** (0.00) 0.44*** (0.01) 0.70*** (0.01) 0.65*** (0.01) 0.40*** (0.01) 0.63*** (0.01) V1 0.13***^^^ (0.03) 0.02***^^^ (0.01) 0.02**^ (0.01) 0.02 (0.02) 0.01 (0.00) -0.01 (0.01) 0.00 (0.01) 0.01 (0.01) 0.02**^ (0.01) 0.01 (0.01) V2 0.13***^^^ (0.03) 0.02***^^ (0.01) 0.01* (0.01) 0.01 (0.01) 0.01 (0.00) -0.01 (0.01) 0.00 (0.01) -0.00 (0.01) 0.05***^^^ (0.01) 0.00 (0.01) V3 0.20***^^^ (0.03) 0.01* (0.01) 0.02**^ (0.01) 0.02 (0.02) 0.01 (0.01) -0.00 (0.01) 0.00 (0.01) 0.01 (0.01) 0.06***^^^ (0.01) 0.01 (0.01) Joint test ***^^^ ***^^^ Observations 17,296 17,316 17,350 17,312 17,349 17,294 17,348 16,777 17,350 17,345 R-squared 0.04 0.07 0.11 0.03 0.11 0.02 0.05 0.02 0.05 0.03 Key: (1) Participation in civics projects; (2) Attitudes related to school government and politics; (3) Civic behaviors and teachings in the classroom; (4) Engagement in school government and politics; (5) Prosocial and democratic attitudes; (6) Attitudes about corruption and dictatorship; (7) Internal political efficacy and beliefs about citizenship; (8) External political efficacy; (9) Current civic engagement and practices; and (10) Future civic engagement and political participation. Notes: Results correspond to IPW regressions with gender, an ethnoreligious minority indicator, an index of SES, grade, and number of students surveyed as controls. Standard errors clustered at the school level in parentheses; *** p<0.01, ** p<0.05, * p<0.1; ^^^ q<0.01, ^^ q<0.05, ^ q<0.1. All p- and q-values correspond to two-tailed tests. Adjustments for multiple hypothesis testing are conducted for each group of outcome measures: 1 uptake measure (4 hypotheses), 3 outcome-level measures (12 [4x3] hypotheses), 4 impact-level attitude measures (16 [4x4] hypotheses), and 2 impact-level behavior measures (8 [4x2] hypotheses). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 33 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Two additional findings are worth noting. First, V1 resulted in higher participation in civic or voluntary projects. Assuming there were no deviations in the assignment of variations (i.e., instances of V1 schools receiving experiential learning programming) or spillovers (e.g., instances of V1 teachers voluntarily using projects to emulate their V2 or V3 counterparts), specialized teacher training alone seems to have had incentivized teachers to use experiential leaning tools. Second, uptake of experiential learning activities was rather disappointing for the variation including compulsory experiential learning; by design all students assigned to this variation should have reported participation, but the participation rate was only 77 percent. Outcome Level Overall, after adjustment for multiple hypothesis testing, MT programming also had a positive effect on desirable attitudes related to school government and politics, and civic behaviors and teachings in the classroom. Students in MT schools were more likely to express their opinions, learn about problems in society, discuss current events, have positive views about participation in school politics, and believe that positive changes can happen in schools when students mobilize in groups to resolve critical issues than students in control schools. However, MT programming did not have an effect on engagement in school government and politics. The attitudes related to school government and politics measure was 2.7 percent higher in V1 schools than in control schools, while the civic behaviors and teachings in the classroom measure was 3 percent higher. In the case of V2 schools, the attitudes measure was 1.4 percent higher than in control school. In V3 schools, the behaviors and teachings measure was 3 percent higher than in control schools. None of the variations had a significant effect on the engagement in school government and politics measure. Lastly, and contrary to expectations, the results of the join test indicate that, overall, the experiential learning variations did not produce greater improvements than the training only variation. Impact Level Attitudes These estimates directly examine H1 and H2. No MT variation has a significant effect on impact-level attitudes and the experiential learning variations did not produce greater improvements than the training only variation. The results do not support the expectations that specialized teacher training should improve students’ civic, democratic, and prosocial attitudes (H1) and that experiential learning paired with specialized teacher training should result in greater improvements in students’ attitudes than specialized teacher training alone (H2). Impact Level Behaviors These estimates directly examine H3 and H4. While no variation enhanced future civic engagement and political participation, all three MT variations had a positive and significant effect on current civic engagement and practices. The current civic engagement and practices measure in MT schools was higher than in control schools by 5 percent in V1 schools, 12.5 percent in V2 schools, and 15 percent in V3 schools. The results of the join test also indicate that the experiential learning variations produced greater improvements than the training only variation. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 34 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 While encouraging, these results should be taken with caution; given the way in which questions about current civic engagement and practices were posed to students, it is not possible to differentiate between engagement and practices that take place inside and outside of experiential learning projects. Said in other words, this outcome measure might be reflecting uptake of experiential learning projects rather than more spontaneous engagement and practices that happened outside—and as a result—of MT programming. The current civic engagement and practices measure includes items asking respendents to indicate whether they were involved in activities of a environmental organization (Q10a), a human rights organization (Q10b), a voluntary group doing something to help the community (Q10c), or a charity (Q10d) in the last 12 months. As written, it is not possible to determine if a student participated in any such organization or group as part of the voluntary civics clubs of V2 or the mandatory class civics projects of V3, or not. In sum, the results offer tentative support for the expectations that specialized teacher training should improve students’ civic, democratic, and prosocial behaviors (H3) and that experiential learning paired with specialized teacher training should result in greater improvements in students’ behaviors than specialized teacher training alone (H4). 6.2 EVALUATION QUESTION 3 This section addresses EQ3, Does civic education programming contribute to closing the democratic achievement gap between students of different genders, ethnoreligious groups, and socioeconomic status (SES)? Table 7 presents the theoretical hypotheses corresponding to this question as well as the statistical hypothesis tests used to examine them for a given outcome measure. Table 7. Hypotheses and Hypothesis Tests for EQ3 Theoretical Hypotheses Statistical Hypothesis Tests Civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among girls than among boys (H5)  Effect of V1 for boys (base) > 0  Effect of V1 for girls > 0  Effect of V2 for boys (base) > 0  Effect of V2 for girls > 0  Effect of V3 for boys (base) > 0  Effect of V3 for girls > 0 Compulsory experiential learning paired with specialized teacher training will result in the greatest improvements among girls relative to boys (H6) Joint test of:  Effect of V3 for girls > Effect of V1 for girls  Effect of V3 for girls > Effect of V2 for girls IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 35 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Theoretical Hypotheses Statistical Hypothesis Tests Civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among students from ethnoreligious minorities than among other students (H7)  Effect of V1 for non-minority (base) > 0  Effect of V1 for minority > 0  Effect of V2 for non-minority (base) > 0  Effect of V2 for minority > 0  Effect of V3 for non-minority (base) > 0  Effect of V3 for minority > 0 Compulsory experiential learning paired with specialized teacher training will result in the greatest improvements among students from ethnoreligious minorities relative to other students (H8) Joint test of:  Effect of V3 for minority > Effect of V1 for minority  Effect of V3 for minority > Effect of V2 for minority Exposure to civic education programming will have a greater effect both civic, democratic, and prosocial attitudes and behaviors among low-SES students than among other students (H9)  Effect of V1 for non-low SES (base) > 0  Effect of V1 for low SES > 0  Effect of V1 for non-low SES (base) > 0  Effect of V1 for low SES > 0  Effect of V1 for non-low SES (base) > 0  Effect of V1 for low SES > 0 Compulsory experiential learning paired with specialized teacher training will result in the greatest improvements among low-SES students relative to other students (H10) Joint test of:  Effect of V3 for low SES > Effect of V2 for low SES  Effect of V3 for low SES > Effect of V2 for low SES Given these hypotheses, the expectation is that for each of the three comparisons across subgroups (i.e., boys vs. girls, non-minority vs. minority students, and non-low-SES vs. low-SES students), any given MT variation should result in desired changes among groups with lower democratic achievement, and that the variation including compulsory experiential learning (V3) should be more effective than the other two variations. The subsections below present the results for each of the subgroups. In all tables reporting difference in effects across groups (tables 8, 9, 10, 12, and 14), the coefficients for the group indicators are the estimated differences in means between the groups of interest and the reference category in control schools. The groups of interest are “Girls” in Table 8, “Minority” in Table 9, “Low SES” in Table 10, “Two Years of MT” in Table 12, and “10th-graders” in Table 14. The reference groups are boys in Table 8, non-minority students in Table 9, non-low SES students in Table 10, students who received one year of MT in Table 12, IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 36 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 and 9th-graders in Table 14.15 The coefficients for the MT variation indicators are the impact estimates for the reference group. The coefficients of particular interest are those corresponding to the interaction between the MT variation indicators and the groups indicators (i.e., “V1(2/3)*Girls” in Table 8, “V1(2/3)*Minority” in Table 9, “V1(2/3)*Low SES” in Table 10, “V1(2/3)*Two Years of MT” in Table 12, and “V1(2/3)*10th-graders” in Table 14). These coefficients estimate the differences in effects between the groups of interest and their respective reference groups. A significant coefficient for the interaction indicates that the MT variation had different effects across groups. Conversely, a non-significant coefficient for the interaction indicates that the variation did not have different effects across groups. 6.2.1 Gender Table 8 (overleaf) presents the estimation results. Before analyzing the estimates, it is important to highlight that, contrary to expectations, the results indicate that girls are not disadvantaged in comparison to boys when it comes to democratic achievement. In facto, seven of the ten coefficients in the “Girls” column gauging differences between girls and boys in control schools are positive and significant, while only one is negative and significant. Girls significantly outperform boys in the following measures: attitudes related to school government and politics; civic behaviors and teachings in the classroom; engagement in school government and politics; prosocial and democratic attitudes; attitudes about corruption and dictatorship; internal political efficacy and beliefs about citizenship; and future civic engagement and political participation. Girls significantly underperform boys in the external political efficacy measure. There are no statistically significant differences in the performance of girls and boys when it comes to participation in civics projects and current civic engagement and practices. Turning to estimates, after adjustments for multiple hypothesis testing, only some of the effects on uptake and on current civic engagement and practices are significant. In the case of uptake, the three variations had a significant effect on students’ participation in civics or voluntary projects. For the reference group (boys), estimated effects range from 16.9 percent in the case of V1 to 27.1 percent in the case of V3. Effects on uptake are significantly larger among girls in the case of V1 (28.1 percent vs. 16.9 percent) and V3 (42 percent vs. 27.1 percent). More generally, however, the joint test of difference between effects indicates that the compulsory experiential learning variation (V3) did not result in greater participation in projects among girls than the compulsory experiential learning variation (V2) or the training only variation (V1). 15 Note that estimates corresponding to the reference groups and the groups of interest in control schools were not considered in adjustments for multiple hypothesis testing because the design report did not include specific evaluation questions or hypotheses related to the existence of the democratic achievement gap across the various groups. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 37 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 8. Heterogeneous Effects: Boys vs. Girls Independent Uptake Outcome Impact - Attitudes Impact - Behaviors Variables (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Control 0.59*** (0.02) 0.75*** (0.01) 0.67*** (0.01) 0.32*** (0.01) 0.68*** (0.01) 0.44*** (0.01) 0.70*** (0.01) 0.65*** (0.01) 0.41*** (0.01) 0.64*** (0.01) V1 0.10***^^^ (0.03) 0.02*** (0.01) 0.02** (0.01) 0.01 (0.02) 0.00 (0.01) -0.00 (0.01) 0.00 (0.01) -0.00 (0.01) 0.01 (0.01) -0.00 (0.01) V2 0.11***^^^ (0.03) 0.01 (0.01) 0.01 (0.01) 0.01 (0.01) 0.00 (0.01) -0.01 (0.01) 0.00 (0.01) -0.00 (0.01) 0.03***^^ (0.01) -0.00 (0.01) V3 0.16***^^^ (0.03) 0.00 (0.01) 0.01 (0.01) 0.02 (0.02) -0.00 (0.01) -0.00 (0.01) 0.00 (0.01) 0.00 (0.01) 0.04***^^^ (0.01) 0.00 (0.01) Girls -0.02 (0.02) 0.04*** (0.01) 0.06*** (0.01) 0.04*** (0.01) 0.07*** (0.01) 0.04*** (0.01) 0.03*** (0.00) -0.03*** (0.01) 0.01 (0.01) 0.02*** (0.01) V1*Girls 0.06**^^ (0.02) 0.00 (0.01) 0.01 (0.01) 0.02 (0.02) 0.01 (0.01) -0.01 (0.01) 0.00 (0.01) 0.03** (0.01) 0.02 (0.01) 0.01 (0.01) V2*Girls 0.04 (0.03) 0.01 (0.01) 0.01 (0.01) 0.00 (0.02) 0.00 (0.01) 0.00 (0.01) 0.00 (0.01) -0.00 (0.01) 0.04***^^ (0.01) 0.01 (0.01) V3*Girls 0.08***^^^ (0.03) 0.02 (0.01) 0.02** (0.01) -0.00 (0.02) 0.02** (0.01) 0.00 (0.01) 0.00 (0.01) 0.01 (0.01) 0.02** (0.01) 0.01 (0.01) Joint Test * ** Observations 17,296 17,316 17,350 17,312 17,349 17,294 17,348 16,777 17,350 17,345 R-squared 0.04 0.07 0.11 0.03 0.11 0.02 0.05 0.02 0.05 0.03 Key: (1) Participation in civics projects; (2) Attitudes related to school government and politics; (3) Civic behaviors and teachings in the classroom; (4) Engagement in school government and politics; (5) Prosocial and democratic attitudes; (6) Attitudes about corruption and dictatorship; (7) Internal political efficacy and beliefs about citizenship; (8) External political efficacy; (9) Current civic engagement and practices; and (10) Future civic engagement and political participation. Notes: Results correspond to IPW regressions with an ethnoreligious minority indicator, an index of SES, grade, and number of students surveyed as controls. Standard errors clustered at the school level in parentheses; *** p<0.01, ** p<0.05, * p<0.1; ^^^ q<0.01, ^^ q<0.05, ^ q<0.1. All p- and q-values correspond to two-tailed tests. Adjustments for multiple hypothesis testing are conducted for each group of outcome measures: 1 uptake measure (7 hypotheses), 3 outcome-level measures (21 [7x3] hypotheses), 4 impact-level attitude measures (28 [7x4] hypotheses), and 2 impact-level behavior measures (14 [7x2] hypotheses). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 38 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Effects on outcome-level measures and impact-level attitudes are not significant after multiple hypothesis testing. Moving on to the effects on impact-level behaviors, each of the two experiential learning variations had a significant effect on the current civic engagement and practices measure. For the reference group (boys), effects for V2 and V3 are 7.3 percent and 9.8 percent, respectively. Only the effect of V2 on this outcome measure is significantly larger among girls (16.7 percent vs. 7.3 percent). Like before, these results should be taken with caution, as it is not possible to differentiate between engagement and practices that take place inside and outside of experiential learning projects. Lastly, the compulsory experiential learning variation (V3) did not have greater effects than the other two variations among girls. Overall, thsee results lend some support to H5; MT programming had greater effects on two outcome measures among girls than among boys. However, given that girls fare better than boys on most outcome measures in the control group, MT increased the gap between genders in favor of girls. The results do not lend support to H6; compulsory experiential learning paired with specialized teacher training did not result in the greatest improvements among girls relative to boys. 6.2.2 Ethnoreligious Minority Status Table 9 (overleaf) presents the estimation results. The results indicate that students from the ethnoreligious minority do seem to be disadvantaged in comparison to non-minority students when it comes to democratic achievement. Five of the ten coefficients in the “Minority” column gauging differences between minority and non-minority students in control schools are negative and significant, while only one is positive and significant. Minority students significantly underperform majority students in the following measures: attitudes related to school government and politics; civic behaviors and teachings in the classroom; engagement in school government and politics; prosocial and democratic attitudes; and future civic engagement and political participation. Minority students significantly outperform majority students in the external political efficacy measure. After adjustments for multiple hypothesis testing, only some of the effects on uptake and on current civic engagement and practices are significant. In the case of uptake, the three variations had a significant effect on students’ participation in civics or voluntary projects. For the reference group (non-minority students), estimated effects range from 23.1 percent in the case of V1 and V2 to 35.7 percent in the case of V3. However, no effects are significantly different among students belonging to the ethnoreligious minority and the joint test consequently indicates that the compulsory experiential learning variation (V3) did not have greater effects than the other two variations among this group of students. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 39 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 9. Heterogeneous Effects: Non-minority vs. Minority Students Independent Uptake Outcome Impact - Attitudes Impact - Behaviors Variables (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Control V1 V2 V3 Minority V1*Minority V2*Minority V3*Minority 0.56*** (0.02) 0.13***^^^ (0.03) 0.13***^^^ (0.03) 0.20***^^^ (0.03) -0.06 (0.04) -0.02 (0.06) 0.02 (0.05) -0.02 (0.05) 0.75*** (0.01) 0.02*** (0.01) 0.01** (0.01) 0.01** (0.01) -0.06*** (0.01) 0.01 (0.01) 0.01 (0.02) -0.00 (0.01) 0.67*** (0.01) 0.02** (0.01) 0.01 (0.01) 0.02* (0.01) -0.06*** (0.01) -0.00 (0.01) 0.00 (0.01) 0.01 (0.01) 0.32*** (0.01) 0.01 (0.02) 0.01 (0.02) 0.01 (0.02) -0.06*** (0.02) 0.02 (0.02) 0.03 (0.02) 0.01 (0.02) 0.68*** (0.00) 0.00 (0.01) 0.01 (0.01) 0.00 (0.00) -0.03*** (0.01) 0.01 (0.01) -0.00 (0.01) 0.00 (0.01) 0.44*** (0.01) 0.00 (0.01) 0.00 (0.01) 0.00 (0.01) -0.00 (0.01) -0.03** (0.02) -0.03* (0.02) -0.02 (0.01) 0.70*** (0.01) 0.00 (0.01) 0.00 (0.01) 0.00 (0.01) -0.02 (0.01) -0.00 (0.01) 0.01 (0.01) -0.00 (0.01) 0.65*** (0.01) 0.01 (0.01) -0.00 (0.01) 0.01 (0.01) 0.02* (0.01) 0.01 (0.02) 0.00 (0.02) -0.01 (0.02) 0.40*** (0.01) 0.02* (0.01) 0.05***^^^ (0.01) 0.06***^^^ (0.01) -0.01 (0.01) 0.00 (0.02) 0.02 (0.02) 0.00 (0.01) 0.64*** (0.01) 0.00 (0.01) -0.00 (0.01) 0.01 (0.01) -0.03** (0.01) 0.01 (0.02) 0.03 (0.02) 0.00 (0.02) Joint Test Observations R-squared 17,296 0.04 17,316 0.07 17,350 0.11 17,312 0.03 17,349 0.11 17,294 0.02 17,348 0.05 16,777 0.02 17,350 0.05 17,345 0.03 Key: (1) Participation in civics projects; (2) Attitudes related to school government and politics; (3) Civic behaviors and teachings in the classroom; (4) Engagement in school government and politics; (5) Prosocial and democratic attitudes; (6) Attitudes about corruption and dictatorship; (7) Internal political efficacy and beliefs about citizenship; (8) External political efficacy; (9) Current civic engagement and practices; and (10) Future civic engagement and political participation. Notes: Results correspond to IPW regressions with gender, an index of SES, grade, and number of students surveyed as controls. Standard errors clustered at the school level in parentheses; *** p<0.01, ** p<0.05, * p<0.1; ^^^ q<0.01, ^^ q<0.05, ^ q<0.1. All p- and q-values correspond to two-tailed tests. Adjustments for multiple hypothesis testing are conducted for each group of outcome measures: 1 uptake measure (7 hypotheses), 3 outcome-level measures (21 [7x3] hypotheses), 4 impact-level attitude measures (28 [7x4] hypotheses), and 2 impact￾level behavior measures (14 [7x2] hypotheses). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 40 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Effects on outcome-level measures and impact-level attitudes are not significant. Moving on to impact-level behaviors, the experiential learning variations had a significant effect on the students’ current civic engagement and practices. For the reference groups (non-minority students), V2’s and V3’s estimated effects are 12.5 percent and 15 percent, respectively. Once again, no effects are significantly different among students belonging to the ethnoreligious minority and the join test consequently indicates compulsory experiential learning variation (V3) did not have greater effects than the other two variations among this group of students. These results do not support H7 and H8. MT programming did not have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among students from ethnoreligious minorities than among other students. MT programming did not contribute to closing the democratic achievement gap between these two groups of students. 6.2.3 Socioeconomic Status Table 10 (overleaf) presents the estimation results. Like minority students, the results indicate that low-SES students are disadvantaged in comparison to non-low-SES students when it comes to democratic achievement. Six of the ten coefficients in the “Low SES” column gauging differences between low-SES and non-low-SES students in control schools are negative and significant. Minority students significantly underperform majority students in the following measures: attitudes related to school government and politics; civic behaviors and teachings in the classroom; engagement in school government and politics; prosocial and democratic attitudes; internal political efficacy and beliefs about citizenship; and current civic engagement and practices. After adjustments for multiple hypothesis testing, the effects on uptake, on some of the outcome-level measures, and on current civic engagement and practices are significant. In the case of uptake, the results indicate that the three variations had a significant impact on students’ participation in civics or voluntary projects. For the reference group (non-low-SES students), estimated effects range from 21.1 percent in the case of V1 to 35.1 percent in the case of V3. Effects on uptake are not significantly different for low-SES students and the joint test of difference between effects indicates that the compulsory experiential learning variation (V3) did not produce greater participation in projects than the other two variations among this group of students. Effects on outcome-level measures are significant in the case of desirable attitudes related to school government and politics, and civic behaviors and teachings in the classroom. The three variations have a positive effect on the former, with estimated effects for the reference group (non-low-SES students) ranging from 1.3 percent (V2) to 2.7 percent (V1 and V3). In addition, fof the reference group, V1 has a positive estimated effect of 2.9 percent on in-classroom civic behaviors and teaching. Effects on outcome-level measures are not significantly different for low-SES students. Furthermore, the compulsory experiential learning variation (V3) did not have greater impacts than the other two variations among this group of students. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 41 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 10. Heterogeneous Effects: Non-low-SES vs. Low-SES Students Independent Uptake Outcome Impact - Attitudes Impact - Behaviors Variables (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Control V1 V2 V3 Low SES V1*Low SES V2*Low SES V3*Low SES 0.57*** (0.03) 0.12***^^^ (0.03) 0.13***^^^ (0.03) 0.20***^^^ (0.03) -0.03 (0.03) 0.02 (0.04) 0.01 (0.04) -0.01 (0.04) 0.75*** (0.01) 0.02***^ (0.01) 0.01**^ (0.01) 0.02**^ (0.01) -0.04*** (0.01) 0.01 (0.01) 0.00 (0.01) -0.02* (0.01) 0.68*** (0.01) 0.02**^ (0.01) 0.01* (0.01) 0.02** (0.01) -0.04*** (0.01) -0.01 (0.01) -0.01 (0.01) -0.00 (0.01) 0.33*** (0.01) 0.02 (0.02) 0.01 (0.02) 0.02 (0.02) -0.06*** (0.01) -0.01 (0.02) 0.03 (0.02) 0.01 (0.02) 0.68*** (0.00) 0.01 (0.01) 0.01 (0.01) 0.01* (0.01) -0.03*** (0.01) -0.00 (0.01) -0.01* (0.01) -0.02* (0.01) 0.44*** (0.01) -0.01 (0.01) -0.01 (0.01) -0.00 (0.01) -0.01 (0.01) -0.00 (0.01) -0.01 (0.01) 0.01 (0.02) 0.70*** (0.01) 0.01 (0.01) 0.01 (0.01) 0.01 (0.01) -0.01** (0.01) -0.02* (0.01) -0.01 (0.01) -0.02** (0.01) 0.65*** (0.01) 0.02 (0.01) 0.00 (0.01) 0.01 (0.01) 0.00 (0.01) -0.00 (0.02) -0.02 (0.02) -0.01 (0.02) 0.40*** (0.01) 0.03**^ (0.01) 0.06***^^^ (0.01) 0.06***^^^ (0.01) -0.02*** (0.01) -0.02* (0.01) -0.03* (0.01) -0.03** (0.01) 0.63*** (0.01) 0.01 (0.01) 0.00 (0.01) 0.01 (0.01) -0.01 (0.01) -0.01 (0.01) 0.01 (0.01) -0.01 (0.01) Joint Test ** Observations R-squared 17,296 0.04 17,316 0.07 17,350 0.10 17,312 0.03 17,349 0.10 17,294 0.01 17,348 0.04 16,777 0.02 17,350 0.05 17,345 0.03 Key: (1) Participation in civics projects; (2) Attitudes related to school government and politics; (3) Civic behaviors and teachings in the classroom; (4) Engagement in school government and politics; (5) Prosocial and democratic attitudes; (6) Attitudes about corruption and dictatorship; (7) Internal political efficacy and beliefs about citizenship; (8) External political efficacy; (9) Current civic engagement and practices; and (10) Future civic engagement and political participation. Notes: Results correspond to IPW regressions with gender, an ethnoreligious minority indicator, grade, and number of students surveyed as controls. Standard errors clustered at the school level in parentheses; *** p<0.01, ** p<0.05, * p<0.1; ^^^ q<0.01, ^^ q<0.05, ^ q<0.1. All p- and q-values correspond to two-tailed tests. Adjustments for multiple hypothesis testing are conducted for each group of outcome measures: 1 uptake measure (7 hypotheses), 3 outcome-level measures (21 [7x3] hypotheses), 4 impact-level attitude measures (28 [7x4] hypotheses), and 2 impact-level behavior measures (14 [7x2] hypotheses). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 42 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 All variations had a positive impact on the current civic engagement and practices measure, although there is no significant effect on any other impact-level measures. For the reference group (non-low-SES studentes), estimated effects range from 7.5 percent (V1) to 15 percent (V2 and V3). Like before, the effects on this outcome measure are not significantly different for low-SES students. Finally, the compulsory experiential learning variation (V3) did not have greater impacts than the other two variations among this group of students. Overall, the results do not support H9 and H10. Civic education programming did not have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among low-SES students on any of the outcome measures. MT programming did not contribute to closing the democratic achievement gap between low-SES students and their more affluent peers. 6.3 EVALUATION QUESTION 4 This section addresses EQ4, Does receiving two years of civic education programming produce greater effects than one year of programming? Table 11 presents the theoretical hypothesis corresponding to this question as well as the statistical hypothesis tests used to examine it for a given outcome measure. The expectation is that any given MT variation programming should be more effective when students are exposed to it for two years than for one year. Table 11. Hypotheses and Hypothesis Tests for EQ4 Theoretical Hypotheses Statistical Hypothesis Tests Two years of civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors than one year of civic education programming (H11)  Effect of V1 for 1 year of MT programming (base) > 0  Effect of V1 for 2 year of MT programming > 0  Effect of V2 for 1 year of MT programming (base) > 0  Effect of V2 for 2 years of MT programming > 0  Effect of V3 for 1 year of MT programming (base) > 0  Effect of V3 for 2 years of MT programming > 0 Table 12 (overleaf) presents the estimation results. After adjustments for multiple hypothesis testing, only some of the effects of one year of programming are significant. In the case of uptake, the results indicate that each of the three variations had a significant effect on participation in civics or voluntary projects. V1 and V2 increased participation by 23.2 percent, while V3 increased participation by 35.7 percent. As expected, the largest increases are observed in schools assigned to the compulsory experiential learning variation. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 43 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 12. Heterogeneous Effects: One Year vs. Two Years of MT Programming Independent Variables Uptake (1) (2) Outcome (3) (4) (5) Impact - Attitudes (6) (7) (8) Impact - Behaviors (9) (10) Control 0.56*** (0.02) 0.74*** (0.01) 0.67*** (0.01) 0.32*** (0.01) 0.68*** (0.00) 0.44*** (0.01) 0.69*** (0.01) 0.65*** (0.01) 0.39*** (0.01) 0.63*** (0.01) V1 V2 V3 Two Years V1*Two Years V2*Two Years V3*Two Years 0.13***^^^ (0.03) 0.13***^^^ (0.03) 0.20***^^^ (0.03) 0.00 (0.03) -0.01 (0.04) -0.01 (0.03) 0.01 (0.04) 0.02***^^ (0.01) 0.01** (0.01) 0.02***^^ (0.01) -0.00 (0.01) 0.02 (0.01) 0.01 (0.01) -0.02 (0.01) 0.02** (0.01) 0.02** (0.01) 0.02***^^ (0.01) -0.02*** (0.01) 0.02* (0.01) -0.01 (0.01) -0.01 (0.01) 0.02 (0.02) 0.01 (0.02) 0.01 (0.02) 0.01 (0.01) -0.01 (0.01) 0.01 (0.02) 0.01 (0.02) 0.01 (0.01) 0.01 (0.01) 0.01 (0.01) 0.00 (0.01) 0.00 (0.01) -0.01 (0.01) -0.01 (0.01) -0.01 (0.01) -0.00 (0.01) 0.00 (0.01) -0.01 (0.01) -0.00 (0.01) -0.01 (0.01) -0.01 (0.01) 0.01 (0.01) 0.01 (0.01) 0.00 (0.01) -0.01*** (0.01) -0.00 (0.01) -0.01 (0.01) -0.00 (0.01) 0.01 (0.01) -0.00 (0.01) 0.01 (0.01) -0.01 (0.01) 0.01 (0.01) 0.01 (0.01) -0.00 (0.01) 0.02**^ (0.01) 0.05***^^^ (0.01) 0.06***^^^ (0.01) -0.02*** (0.01) -0.01 (0.01) 0.00 (0.01) -0.00 (0.01) 0.01 (0.01) 0.01 (0.01) 0.01 (0.01) 0.00 (0.01) -0.00 (0.01) -0.01 (0.01) -0.01 (0.01) Observations R-squared 17,296 0.04 17,316 0.07 17,350 0.11 17,312 0.03 17,349 0.11 17,294 0.02 17,348 0.05 16,777 0.02 17,350 0.05 17,345 0.03 Key: (1) Participation in civics projects; (2) Attitudes related to school government and politics; (3) Civic behaviors and teachings in the classroom; (4) Engagement in school government and politics; (5) Prosocial and democratic attitudes; (6) Attitudes about corruption and dictatorship; (7) Internal political efficacy and beliefs about citizenship; (8) External political efficacy; (9) Current civic engagement and practices; and (10) Future civic engagement and political participation. Notes: Results correspond to IPW regressions with gender, an ethnoreligious minority indicator, an index of SES, grade, and number of students surveyed as controls. Standard errors clustered at the school level in parentheses; *** p<0.01, ** p<0.05, * p<0.1; ^^^ q<0.01, ^^ q<0.05, ^ q<0.1. All p- and q-values correspond to two-tailed tests. Adjustments for multiple hypothesis testing are conducted for each group of outcome measures: 1 uptake measure (7 hypotheses), 3 outcome-level measures (21 [7x3] hypotheses), 4 impact-level attitude measures (28 [7x4] hypotheses), and 2 impact-level behavior measures (14 [7x2] hypotheses). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 44 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 One year of MT programming also had some positive effects on outcome-level measures. V1 significantly improved desirable related to school government and politics by 2.7 percent; and V3 significantly improved attitudes related to school government and politics, and civic behaviors and teachings in the classroom by 2.7 percent and 3 percent, respectively. At the impact level, one year of MT programming only had significant effects on the current civic engagement and practices measures. Improvements range from 5.1 percent in the case of V1 to 15.4 percent in the case of V3. The previously made caveat regarding the extent to which this measure gauges engagement and practices that happen outside of MT programming also applies. Summing up, the results do not support H11; two years of MT programming did not have a greater effect on on both civic, democratic, and prosocial attitudes and behaviors than one year of MT programming. 6.4 EVALUATION QUESTION 5 This section addresses EQ5, Does civic education programming have similar effects among 9th￾graders and 10th-graders? Table 13 presents the theoretical hypothesis corresponding to this question as well as the statistical hypothesis tests used to examine it for a given outcome measure. The expectation is that all MT variations should be more effective among 9th-graders than among 10th-graders.16 Table 13. Hypotheses and Hypothesis Tests for EQ5 Theoretical Hypotheses Statistical Hypothesis Tests Exposure to civic education programming will have a greater effect on both civic, democratic, and prosocial attitudes and behaviors among students in the 9th grade than among students in the 10th grade (H12)  Effect of V1 for 9th- graders (base) > 0  Effect of V1 for 10th-graders > 0  Effect of V2 for 9th-graders (base) > 0  Effect of V2 for 10th-graders > 0  Effect of V3 for 9th-graders (base) > 0  Effect of V3 for 10th-graders > 0 Table 14 (overleaf) presents the estimation results. After adjustments for multiple hypothesis testing, the effects on uptake, on some of the outcome-level measures, and on current civic engagement and practices are significant. In the case of uptake, the results indicate that the three variations had a significant effect on participation in civics or voluntary projects. Estimated effects range from 19.3 percent in the case of V1 to 35.1 percent in the case of V3. However, effects on uptake are not significantly different for 10th-graders. 16 Differences in the effects across grades were also examined using only Wave 1 (Y1 endline). The results are similar to the ones presented here. This estimations were performed because students who were in the 10th grade in Wave 3 (Y2 endline) received two years of programming and this could muddle differences in effects estimated using data from Wave 1 and Wave 3. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 45 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table 14. Heterogeneous Effects: 9th-graders vs. 10th-graders Independent Variables Uptake (1) (2) Outcome (3) (4) (5) Impact - Attitudes (6) (7) (8) Impact - Behaviors (9) (10) Control 0.57*** (0.03) 0.74*** (0.01) 0.67*** (0.01) 0.32*** (0.01) 0.67*** (0.00) 0.44*** (0.01) 0.69*** (0.01) 0.65*** (0.01) 0.40*** (0.01) 0.63*** (0.01) V1 0.11***^^^ (0.03) 0.02**^ (0.01) 0.02**^ (0.01) 0.02 (0.02) 0.01* (0.01) -0.01 (0.01) 0.01 (0.01) 0.01 (0.01) 0.02* (0.01) 0.01 (0.01) V2 0.14***^^^ (0.03) 0.02***^ (0.01) 0.02**^ (0.01) 0.01 (0.02) 0.01* (0.01) -0.00 (0.01) 0.01 (0.01) -0.01 (0.01) 0.05***^^^ (0.01) 0.00 (0.01) V3 0.20***^^^ (0.03) 0.02**^ (0.01) 0.03***^^ (0.01) 0.01 (0.02) 0.01 (0.01) -0.00 (0.01) 0.00 (0.01) 0.00 (0.01) 0.06***^^^ (0.01) 0.01 (0.01) 10th-graders 0.07*** (0.02) 0.02*** (0.01) 0.03*** (0.01) 0.02** (0.01) 0.02*** (0.00) 0.02*** (0.01) 0.01 (0.01) -0.04*** (0.01) 0.03*** (0.00) 0.00 (0.01) V1*10th-graders 0.02 (0.03) 0.01 (0.01) -0.00 (0.01) -0.01 (0.01) -0.01 (0.01) 0.01 (0.01) -0.01 (0.01) 0.01 (0.01) 0.00 (0.01) 0.00 (0.01) V2*10th-graders -0.01 (0.02) -0.01 (0.01) -0.01* (0.01) 0.01 (0.01) -0.01 (0.01) -0.01 (0.01) -0.00 (0.01) 0.01 (0.01) 0.01 (0.01) -0.00 (0.01) V3*10th-graders -0.00 (0.03) -0.01 (0.01) -0.02** (0.01) 0.01 (0.01) -0.01 (0.01) -0.00 (0.01) -0.00 (0.01) 0.01 (0.01) 0.00 (0.01) -0.00 (0.01) Observations R-squared 17,296 0.04 17,316 0.07 17,350 0.11 17,312 0.03 17,349 0.11 17,294 0.02 17,348 0.05 16,777 0.02 17,350 0.05 17,345 0.03 Key: (1) Participation in civics projects; (2) Attitudes related to school government and politics; (3) Civic behaviors and teachings in the classroom; (4) Engagement in school government and politics; (5) Prosocial and democratic attitudes; (6) Attitudes about corruption and dictatorship; (7) Internal political efficacy and beliefs about citizenship; (8) External political efficacy; (9) Current civic engagement and practices; and (10) Future civic engagement and political participation. Notes: Results correspond to IPW regressions with gender, an ethnoreligious minority indicator, an index of SES, and number of students surveyed as controls. Standard errors clustered at the school level in parentheses; *** p<0.01, ** p<0.05, * p<0.1; ^^^ q<0.01, ^^ q<0.05, ^ q<0.1. All p- and q-values correspond to two-tailed tests. Adjustments for multiple hypothesis testing are conducted for each group of outcome measures: 1 uptake measure (7 hypotheses), 3 outcome-level measures (21 [7x3] hypotheses), 4 impact-level attitude measures (28 [7x4] hypotheses), and 2 impact-level behavior measures (14 [7x2] hypotheses). IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 46 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Effects on outcome-level measures are significant in the case of desirable attitudes related to school government and politics, and civic behaviors and teachings in the classroom. Each of the three variations has a positive effect on attitudes equivalent to 2.7 percent percent, while their effects on civic behaviors and teaching range from 3 percent (V1 and V2) to 4.5 percent (V3). Nevertheless, the effects on the outcome-level measures are not significantly smaller (or larger) for 10th-graders. The results also indicate that two of the three variations had a positive effect on students’ current civic engagement and practices, but there is no significant effect on other impact-level variables. Estimated effects on the current civic engagement and practices measure are 12.5 percent in the case of V2 and 15 percent in the case of V3. Like before, the effects on this outcome measure are not significantly different for 10th-graders. Summing up, the results do not support H12; MT programming did not have a greater effect on on both civic, democratic, and prosocial attitudes and behaviors among 9th-graders than among 10th-graders. 7. CONCLUSIONS PH International’s implementation of the MT CEI was a herculean effort among hundreds of teachers, staff and NGO partner workers throughout Georgia, as well as outside consultants and monitors working diligently to ensure implementation was going as planned for the IE. The IE report provides a complex and interesting picture of the project’s impacts; it also offers some tentative lessons regarding future civic education programming for both the implementation team and USAID. The IE followed an RCT design, whereby participating schools were randomly assigned to receive one of three MT variations (enhanced teacher training and supplementary civics curricula, V1; voluntary civics clubs plus enhanced teacher training and supplementary civics curricula, V2; and mandatory class civics projects plus enhanced teacher training and supplementary civics curricula, V3) or to a control group receiving no programming. Balance tests were conducted to check if the randomization had produced groups with balanced control covariates. The results indicate that it did not. While random assignment can produce unbalanced groups due to chance, it is not possible to rule out that imbalances are due to improper randomization (either at design or implementation). To minimize the possibility of bias in the results, effects were estimated using inverse probability weighting (IPW). This technique models treatment assignment and estimates the probability (or propensity score) of being assigned to a given group for each student, and it then uses this probably to produce estimates that account for nonrandom treatment assignment. Overall, MT civic education programming had a relatively successful uptake of experiential learning activities. Surprisingly, students in schools assigned to the enhanced teacher training only variation (V1) participated in voluntary or civics projects at similar rates than those in schools assigned to the variation including voluntary experiential learning (V2). Given that the effects of V3 on project uptake were much stronger, the results show that the two experiential learning variations together were more effective in enhancing participation than the training only variation. It is also worth noting that uptake of experiential learning activities was rather disappointing for the variation including compulsory experiential learning. By design all students assigned to this variation should have reported participation, but this was not the case by far. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 47 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 MT programming also had a significant effect on outcome-level measures gauging attitudes related to school government and politics, and civic behaviors and teachings in the classroom. Regardless of variation, students in MT schools were more likely to express their opinions, learn about problems in society, discuss current events, have positive views about participation in school politics, and believe that positive changes can happen in schools when students mobilize in groups to resolve critical issues than students in control schools. This indicates that the project was effective at achieving its immediate objectives of promoting the use of experiential learning tools and more democratic, open, and participatory teaching styles in civics classes. The project was also successful at nurturing more positive attitudes toward school government and politics, although this change in attitudes was not accompanied by measurable changes in engagement. When it comes to effecting change on impact-level measures (i.e., behaviors and attitudes about objects outside school that are likely to change in the medium to long term), MT programming had mixed results. On the one hand, MT programming did not have any effects on students’ civic, democratic, and prosocial attitudes. On the other hand, all three MT variations, but especially the ones that include experiential learning, had a positive effect on current civic engagement and practices. There is a concern, however, that these effects might reflect uptake of experiential learning projects rather than more spontaneous engagement and practices that happened outside of MT programming. The current civic engagement and practices measure includes items asking respendents to indicate whether they were involved in activities of an environmental organization (Q10a), a human rights organization (Q10b), a voluntary group doing something to help the community (Q10c), or a charity (Q10d) in the last 12 months. As mentioned before, given the way in which questions about current civic engagement and practices were posed to students, it is not possible to differentiate between engagement and practices that take place inside and outside of experiential learning projects. Even though the results pertaining to impact-level measures might be disappointing given the implementation effort, it is important to keep in mind that effecting measurable short-term changes in the impact-level measures considered here would be quite a feat. This is especially the case for the enhanced teacher training only variation because of its sole focus on traditional classroom learning. A follow-up study would be needed to evaluate if impact￾level effects are observed in the medium to long term. With regard to the democratic achievement gap between genders, findings from control schools indicate that, contrary to expectations, girls fare better than boys on civic, democratic, and prosocial attitudes and behaviors. The results indicate that MT programming had larger effects among girls on participation in civics and voluntary projects and on current civic engagement and practices, increasing the gap between genders in favor of girls. MT programming did not contribute to closing gaps between students from ethnoreligious minorities and other students, and between low-SES students and other students. Lastly, exposure to two years of MT programming had similar effects than one year of exposure, and MT programming had similar effects among 9th-graders and 10th￾graders. These results suggest at least two lessons for future civic education programming. First, assuming there were no deviations in variation assignment or spillover effects, results regarding MT programming’s impact on uptake and outcome-level measures indicate that specialized teacher training alone might incentivize teachers to use experiential leaning tools. Since the training only variation is presumably less costly than the variations including IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 48 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 experiential learning, and the impacts of the former on uptake and outcome-level measures are quite similar to those of the voluntary experiential learning variation, it would be important to reflect on whether it is worth pursuing the voluntary variation in the future. Second, and contrary to what one would expect, the results also indicate that two years of programming did not produce greater impacts than a single year of programming. In light of this finding, it would be important to reflect on whether it is worth providing two years of civic education programming starting when students are in the 9th grade. It would also be important for future studies to examine if this finding holds when students receive civic education programming at lower grades, as it is possible that younger students might benefit from two years of civic programming. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 49 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 REFERENCES Alivernini, Fabio, and Sara Manganelli. 2011. “Is There a Relationship between Openness in Classroom Discussion and Students’ Knowledge in Civic and Citizenship Education?” Procedia-Social and Behavioral Sciences 15: 341–45. Amnå, Erik. 2012. “How Is Civic Engagement Developed over Time? Emerging Answers from a Multidisciplinary Field.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 611–27. Antal, Carrie, and Peter Easton. 2009. “Indigenizing Civic Education in Africa: Experience in Madagascar and the Sahel.” International Journal of Educational Development 29 (6): 599–611. Avery, Patricia G., Karen Bird, Sandra Johnstone, John L. Sullivan, and Kristina Thalhammer. 1992. “Exploring Political Tolerance with Adolescents.” Theory & Research in Social Education 20 (4): 386–420. Avery, Patricia G., and Annette M. Simmons. 2001. “Civic Life as Conveyed in United States Civics and History Textbooks.” International Journal of Social Education 15 (2): 105–30. Beaumont, Elizabeth. 2011. “Promoting Political Agency, Addressing Political Inequality: A Multilevel Model of Internal Political Efficacy.” The Journal of Politics 73 (01): 216–31. Benjamini Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society. Series B (Methodological): 289-300. Bennion, Dr Elizabeth A. 2006. “Civic Education and Citizen Engagement: Mobilizing Voters as a Required Field Experiment.” Journal of Political Science Education 2 (2): 205–27. Bers, Marina Umaschi, and Clement Chau. 2006. “Fostering Civic Engagement by Building a Virtual City.” Journal of Computer-Mediated Communication 11 (3): 748–70. Berson, Michael J., Liliana Rodríguez-Campos, Connie Walker-Egea, Corina Owens, and Aarti Bellara. 2013. “Youth Engagement in Electoral Activities: A Collaborative Evaluation of a Civic Education Project.” Journal of Education and Training Studies 2 (1): 81–87. Blair, Harry. 2003. “Jump-Starting Democracy: Adult Civic Education and Democratic Participation in Three Countries.” Democratization 10 (1): 53–76. Booth, John A., and Patricia Bayer Richard. 1998. “Civil Society, Political Capital, and Democratization in Central America.” The Journal of Politics 60 (03): 780–800. Bratton, Michael, Philip Alderfer, Georgia Bowser, and Joseph Temba. 1999. “The Effects of Civic Education on Political Culture: Evidence from Zambia.” World Development 27 (5): 807–24. Camino, Linda, and Shepherd Zeldin. 2002. “From Periphery to Center: Pathways for Youth Civic Engagement in the Day-to-Day Life of Communities.” Applied Developmental Science 6 (4): 213–20. Campbell, David E. 2008. “Civic Side of School Choice: An Empirical Analysis of Civic Education in Public and Private Schools, The.” BYU L. Rev., 487. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 50 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Cicognani, Elvira, Bruna Zani, Bernard Fournier, Claire Gavray, and Michel Born. 2012. “Gender Differences in Youths’ Political Engagement and Participation. The Role of Parents and of Adolescents’ Social and Civic Participation.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): Claes, Ellen, Marc Hooghe, and Dietlind Stolle. 2009. “The Political Socialization of Adolescents in Canada: Differential Effects of Civic Education on Visible Minorities.” Canadian Journal of Political Science 42 (03): 613–36. Cogburn, Derrick L., and Fatima K. Espinoza-Vasquez. 2011. “From Networked Nominee to Networked Nation: Examining the Impact of Web 2.0 and Social Media on Political Participation and Civic Engagement in the 2008 Obama Campaign.” Journal of Political Marketing 10 (1-2): 189–213. Cohen, Alison, and Benjamin Chaffee. 2013. “The Relationship between Adolescents’ Civic Knowledge, Civic Attitude, and Civic Behavior and Their Self-Reported Future Likelihood of Voting.” Education, Citizenship and Social Justice 8 (1): 43–57. Conover, Pamela Johnston, and Donald D. Searing. 2000. “A Political Socialization Perspective.” Rediscovering the Democratic Purposes of Education, 91–124. Crocetti, Elisabetta, Parissa Jahromi, and Wim Meeus. 2012. “Identity and Civic Engagement in Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 521–32. Eckel, Catherine, Philip J. Grossman, Cathleen A. Johnson, Angela CM de Oliveira, Christian Rojas, and Rick Wilson. 2011. “Social Norms of Sharing in High School: Teen Giving in the Dictator Game.” Journal of Economic Behavior & Organization 80 (3): 603–12. Eckstein, Katharina, Peter Noack, and Burkhard Gniewosz. 2012. “Attitudes toward Political Engagement and Willingness to Participate in Politics: Trajectories throughout Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 485–95. Erentaitė, Rasa, Rita Žukauskienė, Wim Beyers, and Rasa Pilkauskaitė-Valickienė. 2012. “Is News Media Related to Civic Engagement? The Effects of Interest in and Discussions about the News Media on Current and Future Civic Engagement of Adolescents.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 587–97. Ferreira, Pedro D., Cristina N. Azevedo, and Isabel Menezes. 2012. “The Developmental Quality of Participation Experiences: Beyond the Rhetoric That ‘participation Is Always Good!’” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 599–610. Finkel, Steven E. 2002. “Civic Education and the Mobilization of Political Participation in Developing Democracies.” The Journal of Politics 64 (4): 994–1020. Finkel, Steven E. 2003. “Can Democracy Be Taught?” Journal of Democracy 14 (4): 137–51. Finkel, Steven E. 2014. “The Impact of Adult Civic Education Programmes in Developing Democracies.” Public Administration and Development 34 (3): 169–81. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 51 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Finkel, Steven E., and Howard R. Ernst. 2005. “Civic Education in Post-Apartheid South Africa: Alternative Paths to the Development of Political Knowledge and Democratic Values.” Political Psychology 26 (3): 333–64. Finkel, Steven E., Jeremy Horowitz, and Reynaldo T. Rojo-Mendoza. 2012. “Civic Education and Democratic Backsliding in the Wake of Kenya’s Post-2007 Election Violence.” The Journal of Politics 74 (01): 52–65. Flanagan, Constance, Wim Beyers, and Rita Žukauskienė. 2012. “Political and Civic Engagement Development in Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 471–73. Flanagan, Constance, and Peter Levine. 2010. “Civic Engagement and the Transition to Adulthood.” The Future of Children 20 (1): 159–79. Galston, William A. 2004. “Civic Education and Political Participation.” PS: Political Science and Politics 37 (2): 263–66. Gastil, John. 2004. “Adult Civic Education through the National Issues Forums: Developing Democratic Habits and Dispositions through Public Deliberation.” Adult Education Quarterly 54 (4): 308–28. Gil de Zúñiga, Homero, Nakwon Jung, and Sebastián Valenzuela. 2012. “Social Media Use for News and Individuals’ Social Capital, Civic Engagement and Political Participation.” Journal of Computer-Mediated Communication 17 (3): 319–36. Glennerster, Rachel and Kudzai Takavarasha. 2013. Running Randomized Evaluations. A Practical Guide. Princeton University Press. Goemans, Hein. 2006. “Bounded Communities: Territoriality, Territorial Attachment, and Conflict.” Territoriality and Conflict in an Era of Globalization, 25–61. Greene, Jay P. 2000. “Niemi and Junn, Civic Education.” Social Science Quarterly 81 (2): 696– 696. Hahn, Carole L. 2003. “Democratic Values and Citizen Action: A View from US Ninth Graders.” International Journal of Educational Research 39 (6): 633–42. Helms, Sara, Erik Angner, Brian Scott, and Sarah Culver. 2009. “Mandated Volunteering: An Experimental Approach.” http://economics.lafayette.edu/files/2010/03/AngnerPaper1.pdf. Helms, Sara E. 2013. “Involuntary Volunteering: The Impact of Mandated Service in Public Schools.” Economics of Education Review 36: 295–310. Hope, Elan C., and Robert J. Jagers. 2014. “The Role of Sociopolitical Attitudes and Civic Education in the Civic Engagement of Black Youth.” Journal of Research on Adolescence 24 (3): 460–70. Kahne, Joseph E., and Susan E. Sporte. 2008. “Developing Citizens: The Impact of Civic Learning Opportunities on Students’ Commitment to Civic Participation.” American Educational Research Journal 45 (3): 738–66. Keating, Avril, David Kerr, Thomas Benton, Ellie Mundy, and Joana Lopes. 2010. “Citizenship Education in England 2001-2010: Young People’s Practices and Prospects for the Future: The IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 52 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Eighth and Final Report from the Citizenship Education Longitudinal Study (CELS).” http://dera.ioe.ac.uk/11579/. Kim, Taehan, Constance A. Flanagan, and Alisa A. Pykett. 2015. “Adolescents’ Civic Commitments in Stable and Fledgling Democracies: The Role of Family, School, and Community.” Research in Human Development 12 (1-2): 28–43. Kirlin, Mary. 2002. “Civic Skill Building: The Missing Component in Service Programs?” Political Science & Politics 35 (03): 571–75. Kolb, David. 1984. Experiential Learning as the Science of Learning and Development. Englewood Cliffs NPH, editor. Kropko, Jonathan, Ben Goodrich, Andrew Gelman, and Jennifer Hill. 2017. “Multiple Imputation for Continuous and Categorical Data: Comparing Joint Multivariate Normal and Conditional Approaches.” Political Analysis 22 (4): 497-519. Langton, Kenneth P., and M. Kent Jennings. 1968. “Political Socialization and the High School Civics Curriculum in the United States.” American Political Science Review 62 (03): 852–67. Lerner, Richard M., Celia B. Fisher, and Richard A. Weinberg. 2000. “Toward a Science for and of the People: Promoting Civil Society through the Application of Developmental Science.” Child Development 71 (1): 11–20. Levinson, Bradley AU. 2004. “Hopes and Challenges for the New Civic Education in Mexico: Toward a Democratic Citizen without Adjectives.” International Journal of Educational Development 24 (3): 269–82. Lichbach, Mark Irving. 1995. The Rebel’s Dilemma. Ann Arbor: University of Michigan Press. Maiello, Carmine, Fritz Oser, and Horst Biedermann. 2003. “Civic Knowledge, Civic Skills and Civic Engagement.” European Educational Research Journal 2 (3): 384–95. Markus, Gregory B., Jeffrey PF Howard, and David C. King. 1993. “Notes: Integrating Community Service and Classroom Instruction Enhances Learning: Results from an Experiment.” Educational Evaluation and Policy Analysis 15 (4): 410–19. Martínez, M. Loreto, Pilar Peñaloza, and Cristina Valenzuela. 2012. “Civic Commitment in Young Activists: Emergent Processes in the Development of Personal and Collective Identity.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 474–84. Marzana, Daniela, Elena Marta, and Maura Pozzi. 2012. “Social Action in Young Adults: Voluntary and Political Engagement.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 497–507. McDevitt, Michael, and Steven Chaffee. 2000. “Closing Gaps in Political Communication and Knowledge Effects of a School Intervention.” Communication Research 27 (3): 259–92. McDevitt, Michael, and Spiro Kiousis. 2006. “Deliberative Learning: An Evaluative Approach to Interactive Civic Education.” Communication Education 55 (3): 247–64. Melchior, Alan, Joseph Frees, Lisa LaCava, Chris Kingsley, Jennifer Nahas, Jennifer Power, Gus Baker, et al. 1999. “Summary Report: National Evaluation of Learn and Serve America.” IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 53 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 http://digitalcommons.unomaha.edu/slceeval/36/?utm_source=digitalcommons.unomaha.edu percent2Fslceeval percent2F36&utm_medium=PDF&utm_campaign=PDFCoverPages. Mitra, Robin and Jerome P Reiter. 2012. “A Comparison of Two Methods of Estimating Propensity Scores after Multiple Imputation.” Statistical Methods in Medical Research 25 (1): 188-204. Morduchowicz, Roxana, Edgardo Catterberg, Richard G. Niemi, and Frank Bell. 1996. “Teaching Political Information and Democratic Values in a New Democracy: An Argentine Experiment.” Comparative Politics 28 (4): 465–76. Morgan, William, and Matthew Streb. 2002. “Promoting Civic Activism: Student Leadership in Service-Learning.” Politics & Policy 30 (1): 161–88. Niemi, Robert, and Jane Junn. 1998. What Makes Students Learn. New Haven: Yale University Press. Nucci, Larry, Tobias Krettenauer, and Darcia Narvaez. 2015. Handbook of Moral and Character Education. Routledge. Owen, Diana. 2015. “High School Students’ Acquistition of Civic Knowledge: The Impact of We the People.” Center for Civic Education. Parker-Gwin, Rachel, and J. Beth Mabry. 1998. “Service Learning as Pedagogy and Civic Education: Comparing Outcomes for Three Models.” Teaching Sociology, 276–91. Pasek, Josh, Lauren Feldman, Daniel Romer, and Kathleen Hall Jamieson. 2008. “Schools as Incubators of Democratic Participation: Building Long-Term Political Efficacy with Civic Education.” Applied Development Science 12 (1): 26–37. Pasek, Josh, eian more, and Daniel Romer. 2009. “Realizing the Social Internet? Online Social Networking Meets Offline Civic Engagement.” Journal of Information Technology & Politics 6 (3- 4): 197–215. Pavlova, Maria K., Astrid Körner, and Rainer K. Silbereisen. 2015. “Perceived Social Support, Perceived Community Functioning, and Civic Participation Across the Life Span: Evidence from the Former East Germany.” Research in Human Development 12 (1-2): 100–117. Pearson, Frederic S. 2001. “Dimensions of Conflict Resolution in Ethnopolitical Disputes.” Journal of Peace Research 38 (3): 275–87. Pérez, J. Carola, and Patricio Cumsille. 2012. “Adolescent Temperament and Parental Control in the Development of the Adolescent Decision Making in a Chilean Sample.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 659– 69. Putnam, Robert. 1995a. “Bowling Alone: America’s Declining Social Capital.” Journal of Democracy 6 (1): 65–78. Putnam, Robert. 2000. Making Democracy Work. Princeton: Princeton University Press. Reinders, Heinz, and James Youniss. 2006. “School-Based Required Community Service and Civic Development in Adolescents.” Applied Developmental Science 10 (1): 2–12. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 54 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Sambanis, Nicholas, and Moses Shayo. 2013. “Social Identification and Ethnic Conflict.” American Political Science Review 107 (02): 294–325. Schmid, Christine. 2012. “The Value ‘social Responsibility’ as a Motivating Factor for Adolescents’ Readiness to Participate in Different Types of Political Actions, and Its Socialization in Parent and Peer Contexts.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 533–47. Sears, David O., and Sheri Levy. 2003. “Childhood and Adult Political Development.” http://psycnet.apa.org/psycinfo/2003-88243-003. Šerek, Jan, Lenka Lacinová, and Petr Macek. 2012. “Does Family Experience Influence Political Beliefs? Relation between Interparental Conflict Perceptions and Political Efficacy in Late Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 577–86. Skrondal, Anders and Petter Laake. 2001. “Regression among Factor Scores.” Psychometrika 66 (4): 563–575. Slomczynski, Kazimierz M., and Goldie Shabad. 1998. “Can Support for Democracy and the Market Be Learned in School? A Natural Experiment in Post-Communist Poland.” Political Psychology 19 (4): 749–79. Soule, Suzanne. 2000. “Beyond Communism and War: The Effect of Civic Education on the Democratic Attitudes and Behavior of Bosnian and Herzegovinian Youth.” http://eric.ed.gov.proxy.library.georgetown.edu/?id=ED447046. Schulz, Wolfram, John Ainley, Julian Fraillon, David Kerr, and Bruno Losito. 2010. Initial Findings from the IEA International Civic and Citizenship Education Study. International Association for the Evaluation of Educational Achievement (IEA). Syvertsen, Amy K. 2008. Best Practices in Civic Education: Changes in Students’ Civic Outcomes. DIANE Publishing. Toft, M. D. 2003. The Geography of Ethnic Violence: Identity, Interests, and the Indivisibility of Territory. Princeton Univ Press. Tonge, Jon, Andrew Mycock, and Bob Jeffery. 2012. “Does Citizenship Education Make Young People Better-Engaged Citizens?” Political Studies 60 (3): 578–602. Torney-Purta, Judith. 2002. “Patterns in the Civic Knowledge, Engagement, and Attitudes of European Adolescents: The IEA Civic Education Study.” European Journal of Education, 129– 41. Torney-Purta, Judith, and Jo-Ann Amadeo. 2003. “A Cross-National Analysis of Political and Civic Involvement among Adolescents.” Political Science and Politics 36 (02): 269–74. Torney-Purta, Judith, Carolyn H. Barber, and Britt Wilkenfeld. 2007. “Latino Adolescents’ Civic Development in the United States: Research Results from the IEA Civic Education Study.” Journal of Youth and Adolescence 36 (2): 111–25. Torney-Purta, Judith, Rainer Lehmann, Hans Oswald, and Wolfram Schulz. 2001. Citizenship and Education in Twenty-Eight Countries: Civic Knowledge and Engagement at Age Fourteen. http://eric.ed.gov.proxy.library.georgetown.edu/?id=ED452116. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 55 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Torney-Purta, Judith, and John Schwille. 1986. “Civic Values Learned in School: Policy and Practice in Industrialized Nations.” Comparative Education Review, 30–49. Torney-Purta, Judith, John Schwille, and Jo-Ann Amadeo. 1999. “The IEA Civic Education Study: Expectations and Achievements of Students in Thirty Countries. ERIC Digest.,” December. http://eric.ed.gov/?id=ED435585. van Goethem, Anne A. J., Anne van Hoof, Marcel A. G. van Aken, Quinten A. W. Raaijmakers, Jan Boom, and Bram Orobio de Castro. 2012. “The Role of Adolescents’ Morality and Identity in Volunteering. Age and Gender Differences in a Process Model.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 509– 20. Verba, Sidney, Kay Lehman Schlozman, Henry E. Brady, and Henry E. Brady. 1995. Voice and Equality: Civic Voluntarism in American Politics. Volume 4. Cambridge Univ Press. http://journals.cambridge.org/production/action/cjoGetFulltext?fulltextid=6272664. Vontz, Thomas S., Kim K. Metcalf, and John J. Patrick. 2000. “‘ Project Citizen’ and the Civic Development of Adolescent Students in Indiana, Latvia, and Lithuania.” http://eric.ed.gov/?id=ED447047. Westheimer, Joel, and Joseph Kahne. 2002. “Educating the‘ Good’ Citizen: The Politics of School-Based Civic Education Programs.” http://eric.ed.gov/?id=ED474166. Whiteley, Paul. 2012. “Does Citizenship Education Work? Evidence from a Decade of Citizenship Education in Secondary Schools in England.” Parliamentary Affairs, gss083. Wilson, Rick K. 2011. “The Contribution of Behavioral Economics to Political Science.” Annual Review of Political Science 14: 201–23. Wray-Lake, Laura, and Constance A. Flanagan. 2012. “Parenting Practices and the Development of Adolescents’ Social Trust.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 549–60. Yamamoto, Masahiro, Matthew J. Kushin, and Francis Dalisay. 2013. “Social Media and Mobiles as Political Mobilization Forces for Young Adults: Examining the Moderating Role of Online Political Expression in Political Participation.” New Media & Society, 1461444813518390. Youniss, James. 2011a. “Civic Education: What Schools Can Do to Encourage Civic Identity and Action.” Applied Developmental Science 15 (2): 98–103. Youniss, James. 2011b. “Civic Education: What Schools Can Do to Encourage Civic Identity and Action.” Applied Developmental Science 15 (2): 98–103. Youniss, James, Susan Bales, Verona Christmas-Best, Marcelo Diversi, Milbrey Mclaughlin, and Rainer Silbereisen. 2002. “Youth Civic Engagement in the Twenty-First Century.” Journal of Research on Adolescence 12 (1): 121–48. Zhang, Weiwu, Thomas J. Johnson, Trent Seltzer, and Shannon L. Bichard. 2009. “The Revolution Will Be Networked: The Influence of Social Networking Sites on Political Attitudes and Behavior.” Social Science Computer Review. http://ssc.sagepub.com/content/early/2009/06/12/0894439309335162.short. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 56 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Zhang, Xiaoyun, Haiping Wang, Yan Xia, Xiaohong Liu, and Eunju Jung. 2012. “Stress, Coping and Suicide Ideation in Chinese College Students.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 683–90. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 57 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 ANNEX 1 IMPACT EVALUATION DESIGN REPORT IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 58 Educating for Democracy: An Impact Evaluation of the Momavlis Taoba (Future Generation) Project Impact Evaluation Design Prepared by Altin Ilirjani, Ph.D., USAID DRG Center Molly Inman, Ph.D., USAID DRG Center & Georgetown University USAID DRG Learning Division October 2015 Table of Contents Project Team ................................................................................................................................................. 3 Introduction .................................................................................................................................................. 4 Civic Education Literature Review ................................................................................................................ 5 Adult Civic Education ................................................................................................................................ 5 Adolescent Civic Education....................................................................................................................... 6 Classroom Learning ................................................................................................................................... 7 Service Learning ........................................................................................................................................ 8 A Myriad of Desired Outcomes................................................................................................................. 9 Conclusion............................................................................................................................................... 11 Theory of Change and Hypotheses ............................................................................................................. 12 Knowledge .............................................................................................................................................. 12 Democratic Values, Attitudes and Efficacy ............................................................................................. 12 Civic Engagement and Participation ....................................................................................................... 13 Democratic Achievement Gap ................................................................................................................ 13 Policy Outcomes ......................................................................................................................................... 15 Experimental Design ................................................................................................................................... 15 Assignment of Schools to Treatment Variations ........................................................................................ 17 Measurement and Statistical Power........................................................................................................... 18 Surveys.................................................................................................................................................... 18 Statistical Power...................................................................................................................................... 19 Threats to Impact Evaluation ...................................................................................................................... 21 References .................................................................................................................................................. 22 Appendix One: GIS Maps of MT Impact Evaluation Schools ...................................................................... 29 Appendix Two: Survey Questionnaire ........................................................................................................ 33 2 Educating for Democracy: An Impact Evaluation of Momavlis Taoba (Future Generation) Project Project Team MT Activity Implementer PH International Performance period June 2014 – October 2017 (40 months) Danielle Reiff USAID/Georgia, DG Office Director (2013-2015) David Stonehill USAID/Georgia, Deputy DG Office Director Nino Buachidze USAID/Georgia, Momavlis Taoba AOR Marina Ushveridze MT Chief of Party, PH International Altin Ilirjani Principal Investigator, USAID DRG Center Molly Inman co-Principal Investigator (IIE Democracy Fellow) Nina Kakauridze Impact Evaluation Field Coordinator (Tbilisi, Georgia) This document describes the impact evaluation design for the Momavlis Taoba (MT) civic education activity of USAID/Caucasus in Georgia and PH International. The main objectives of the MT project are to promote greater civic engagement of young people and to expand and institutionalize secondary school civics education curricula and practical applications. The MT activity will reach 480 secondary schools (or 20 percent of all secondary schools) in Georgia, improving the quality and scope of school-based civic education as a means to positively influence the knowledge, attitudes, and behaviors of youth as active participants in Georgia’s democratic society. The project was awarded in June 2014 with a period of performance through October 2017. The impact evaluation will follow civic education achievements of students in grades 9 and 10 during 2015-16 and 2016-17 academic years. 3 Introduction Georgia is continuing its transition to a full democracy and has enormous potential as a strong US ally and a leader in the Caucasus and beyond. As a strong example for other countries transitioning to democracy, Georgia has much to offer. For instance, USAID recently sponsored members of the Georgian Bar Association to travel to Iraq to partner in training with Iraqi lawyers. While Georgia has and continues to make great strides, a plurality (43%) of those surveyed in the most recent Caucasus Barometer concede that Georgia is “a democracy but with major problems.” Given the cooperation of the government of Georgia, now is a critical time to continue institutionalizing democracy in Georgia, insuring as best as possible its irreversibility. One way to guarantee Georgia’s democratic future, and indeed that of any country, is by creating a strong demand for it from its peoples. In 2003, the Georgian people banded together and peacefully yet powerfully demanded that their government respect democratic principles and forced the government to heed its calls. Despite this incredible democratic spirit, the most recent Caucasus Barometer reports that of those surveyed, a plurality (40-48%) neither trust nor distrust major institutions of government including local government, the executive, parliament and political parties. Although improving the institutions may be one way to address this apathy in trust, another is to empower citizens to hold them accountable and be confident in their responsiveness. Increased civic knowledge is likely to increase trust in public institutions in a democracy (Galston 2001, 224). On measures of tolerance, Georgian society also has some room to improve. Between twenty and twenty-four percent of Georgians surveyed disapproved of even doing business with various minority populations in the country. However, those disapproval numbers jump to between sixty and seventy percent when asked whether Georgian women should marry members of those communities. Recent research suggests that lack of integration of minorities into the national identity makes a country susceptible to ethnic conflict even if a majority of people prefer peace and only a few extremists aim to provoke violence (Sambanis and Shayo 2013). Given the regional concentration of Georgia’s minority populations which consistently appears as a risk factor for violent conflict in studies of interethnic violence (Toft 2002, Connell 2002, Lichbach 1995, Goemans 2006), it is important to take this lack of integration seriously. With all of this in mind, Georgia is ripe for improved civic education to further institutionalize its democratic gains. While there are myriad approaches to improving civic education and behavior of citizens, several of which have been implemented in Georgia, educating young people can potentially pay great dividends over the short and long-term. The school system is sufficiently developed to implement a high-quality program and the Ministry of Education and Science has also expressed its support for teaching such classes and for evaluating their effectiveness. The random assignment of different variations of civic education to 480 schools across the country will allow for a robust, scientific study of what is most effective when it comes to civic education of youth in Georgia. 4 Civic Education Literature Review It is a nearly universally held belief that an informed and engaged citizenry is good for democracy. Such a public can elect competent leaders and hold them accountable for the policies and decisions they pursue and exercise legal mechanisms to remove them from office when they fail to live up to democratic principles. Scholars agree that “well-designed institutions are not enough, that a well￾ordered polity requires citizens with appropriate knowledge, skills and traits of character…And, it is reasonably clear that good citizens are made, not born” (Galston 2001, 217). The question is: what is the best way to go about creating an informed and engage citizenry? Over the past fifty years of research, trends in interest in civic education have emerged. In the mid￾1990s the American government literature showed renewed interested in answering this question. This interest lay dormant for some time given a seminal report in 1968 that found that in the US civics curriculum was not “even a minor source of political socialization” (Langton and Jennings 1968, 865). The comparative politics literature showed increased interest in the topic starting in the 2000s, likely as a result of the desire to examine the outcomes of the decade of democracy promotion in the post-Cold War era. For instance, Kim, Flanagan and Pykett (2015) find that in post-communist countries, families are not supportive of social responsibility and students are less likely to participate in extracurricular activities. Others note that socialist societies were based on “values, norms and institutions that were antithetical and openly hostile to political and social pluralism, voluntaristic citizen participation, respect for individual rights and the rule of law” among other democratic principles (Slomczynski and Shabad 1998, 750). They also note a culture of “privatism, distrust outside one’s immediate family and close social circle and an emphasis on personal favors and ties” (ibid., 750). Another division in both of these literatures especially in empirical studies is whether they target adults or adolescent students. This distinction is important, because evidence suggests that civic participation and its manifestation vary greatly by age group as does the type of civic engagement activity (Pavlova et al. 2015). Finally, the type of the civic education intervention used, such as classroom education, workshops, public information campaigns or service learning, differentiates the research along another axis. Indeed, political environment, maturity of audience and method of delivery, all have substantial impacts on learning outcomes in addition to duration and type of deliverer. This literature review will outline the most relevant literature on civic education to date and will highlight the promising theories that this impact evaluation will examine. Adult Civic Education Most of the literature on adult-targeted civic education comes from studies in developing democracies in which international aid agencies have sponsored programs. Finkel (2002) theorizes that adult civic education programs can have direct, indirect and conditional effects on civic participation. He finds that in most of the programs examined in the Dominican Republic and South Africa, direct and indirect effects occurred but that in all of them, the more frequently individuals received training in civic education and the more participatory the methods, the greater the effects (ibid., 1012). In a parallel study of a Dominican Republic adult civic education program Finkel, Sabatini and Bevis (2000) found that the program had a negative effect on participants trust in the institutions of government and theorize that this is because the NGOs that were involved with implementing the program were in opposition to the government and because the government was functioning extremely poorly. They explain that people’s increase in political knowledge about the government and their increased sense of personal efficacy made them more aware of government shortcomings and how they might overcome these 5 problems. In a third paper in which he adds an adult civic education program in Poland to the Dominican Republic and South Africa analysis, Finkel (2003) finds overwhelming support for the programs increasing participation especially when they had strong local problem solving components. However, the effects on support for democratic values and civic competence were much weaker when present at all. A study of the same three programs also found that they helped to close the participation gap between elites and non-elites but that overall gains were modest (Blair 2003). However, an earlier study of a USAID-funded adult civic education program in Zambia found that elite benefitted more than non-elites (Bratton et al. 1999). A US-based study of college students found that civic education activities can reduce the democratic achievement gap and promote political empowerment among those with lower socioeconomic status (Beaumont 2011). Adolescent Civic Education Much research attention has been given to studying civic education and knowledge among high school students in the US, with comparative studies of children in other countries becoming increasingly common. The focus of much of the research in adolescent civic education has addressed the roles that family, schools, peers and associational life play (Amna 2012). Generally speaking, children inherit their parents’ sence of social trust as well as knowledge, values, attitudes and actions (Wray-Lake and Flanagan 2012, Serek, Lacinova and Macek 2012). While parents’ social resources play a pivotal role, other research has indicated that civic education programs in schools can overcome deficits in parental resources by creating opportunity for those youth to participate in civic activities (Flanagan and Levine 2012, Melchior et al. 1999). Others have pointed to adolescence as a crucial period in which individuals develop and consolidate their own political points of view (Eckstein et al. 2012, Sears and Levy 2003; Avery et al. 1992) and civic skills (Verba et al. 1995). While the 1968 report from Langston and Jennings painted a dire picture of the state of US civic education, other subsequent research has been more sanguine. One of the problems in measuring the impact of civic education efforts at the national level is that the education system is decentralized, giving the states and local school districts most of the control over the curriculum. While textbooks and classroom instruction vary little (Avery and Simmons 2000/2001), activities and opportunities outside the classroom vary widely (Hahn 2003). National-level surveys are able to measure civic knowledge, skills, participation and values, but they cannot necessarily attribute them to civic education classes or programs and cannot distinguish effects of general civic socialization that comes from simply living in a democratic society. Some, however, do find that amount of civic instruction and the shorter the time since the course was taken increase the level of civic knowledge students display (Niemi and Junn 1998) although, their methodology has been questioned (Torney-Purta et al. 1999; Greene 2000). Another study focused on four elements of the high school experience, including a sense of school community and level of political discussion, found that these significantly impact civic consciousness of high school students, and that civic education occurs in classes other than civics classes as well (Conover and Searing 2000). Recent research has also examined the levels of civic knowledge and engagement and the effects of civic education for adolescents cross-nationally. In 1999, the International Association for the Evaluation of Educational Achievement (IEA) undertook a massive study surveying representative samples of approximately 3,000 students in the modal grade for 14-year-olds in each of twenty-eight countries (Torney-Purta et al. 2001). Of the many findings of the 1999 IEA survey, the most relevant for this impact evaluation is that there is a positive relationship between civic knowledge and reported likelihood of voting in the future; participatory and open school environments correlate with higher civic 6 knowledge and engagement; and trust in government appears to mirror the levels reported by adults in their respective countries (Torney-Purta et al. 2001). In 2009, the survey implementation was updated and expanded to include over 140,000 students from thirty-eight countries or regions (Schultz et al. 2010). While the surveys in 1999 and 2009 did not ask all of the same questions, the 2009 iteration did include several of the 1999 iteration questions for comparison purposes and found that civic knowledge had declined significantly in several countries since 1999. Perceptions of openness during classroom discussions of political and social issues continued to be positively correlated with civic knowledge and engagement (ibid. 19). Other civic education research outside the US has focused on programs in individual developing countries. A study conducted of adolescents in Poland found that civic education seemed to moderate students opinions at both ends of the ideological spectrum. Slomczynski and Shabad (1998) found that while those who participated in an updated participatory civic education curriculum were less likely to exhibit extreme anti-democratic and anti-market positions, they were also less likely to assert extreme pro-democracy and pro-market attitudes. The authors theorize that this effect is likely due to the curriculum encouraging students to think critically about government, regardless of its type. A study of a program in Argentina in which sixth and seventh grade students debated current events and read news articles found that students who participated in the program outperformed those who did not in terms of civic knowledge and tolerance (Morduchowicz et al. 1996). A comparative analysis of indigenous practices related to civic engagement in Madagascar and the Sahel suggests that people in Africa often see democratization and civic education programs as externally imposed or Western imports but that many African cultures have civic engagement traditions that could be harnessed to bring the concepts better in line with indigenous culture (Antal and Easton 2009). Classroom Learning While traditional classroom learning may be less pedagogically exciting than some newer extracurricular innovations, it remains a substantial part of the way civic education is delivered to students, both in the US and abroad. Unsurprisingly, research shows that more interactive teaching techniques increase the retention of civic knowledge (Niemi and Junn 1998; Torney-Purta et al. 2001). However, instruction generally emphasizes the federal government in the US and gives short shrift to local government where students have a greater opportunity to engage in civic issues and apply their knowledge (Neimi and Junn 1998). Other studies also point to the effectiveness of openness in classroom discussions (Alivernini and Manganelli 2011; Youniss 2011; Campbell 2008, McDevitt and Kiousis 2006; Moduchowicz et al. 1996). Training of teachers has also received attention in recent research. Those wishing to understand the effects of particular curricula realized that those delivering the content were a significant part of the treatment. In many countries until very recently or even continuing in the present, the main method of teacher instruction was one way communication of information which students memorized. Civic education that encourages participation and engagement is particularly poorly suited for this pedagogical tradition. One study of the reformed curriculum in Mexico found that the biggest problem with the program as reported by teachers and administrators alike was lack of teacher training in more democratic styles of instruction, classroom dialog and encouraging critical analysis (Levinson 2004). Another study of a specialized civic education program in the US found that providing teachers with specialized civic education training improved students’ civic knowledge and engagement values (Owen 2015). 7 Service Learning While the traditional classroom model remains a popular method of delivery of civic education content, service learning has become a popular way of promoting student engagement in their communities. In the early 1990s, the US federal government enacted legislation to fund activities that promoted community service, and universities as well as primary and secondary schools started new programs and continued and expanded existing ones to encourage student evolvement in their communities (Markus et al. 1993). The theory upon which the hypothesized effectiveness of service learning is based appears to have its roots implicitly in experiential learning. This approach to learning emphasizes learning as a process that allows individuals to apply abstract concepts to concrete problems, gather data and reevaluate the concepts, either confirming, altering or disconfirming them (Kolb 1984). Additionally, the field of developmental psychology has found that during adolescence, students form connections to their communities, character and moral commitment as well as contributions to society (Lerner et al. 2000; Pittman et al. 2001). Therefore, “students who form connections to their school and community (and to their ethnic group) are likely to care about issues pertaining to these institutions, and therefore to contribute to addressing these issues” (Torney-Purta et al. 2007). For those who want to test the effectiveness of service learning, there is a distinct normative bias toward finding positive results. Who wants to hear that doing community service does not improve engagement in the community and strengthen democratic norms? For instance, Markus, Howard and King (1993) highlight the significant differences in students who were required to do community service as part of a political science college course, but neglect to discuss that students who did not participate in community service as part of the class reported an increase in concern regarding social justice in the post-course evaluation, while the community service students showed no change. They also fail to point out that neither group showed an improvement in their perception of being able to make a difference the world or of individuals being able to have an impact. Also, by measuring the effects immediately following the course, they did not have much to say about whether any positive effects will be lasting. Additionally, merely reporting having a positive experience or liking the activity article reports (Bennion 2006) does not necessarily translate into learning outcomes. Another problem with measuring the effect of service learning is the diversity of programs. As some have pointed out, the type and quality of service in which one is engaged likely has substantial impacts on learning outcomes (Reinders and Youniss 2006). Stuffing envelopes for a membership organization mailing versus serving the homeless dinner at a soup kitchen are likely to have different learning outcomes for students. How well the service activity is tied to learning activities in a classroom also impacts student outcomes. A three part study of adolescents in Portugal found that those who experienced low quality participatory activities had more negative political attitudes post-treatment than those in the control group (Fereira et al. 2012). Another study which compared three models of service learning in college students, placement optional, placement mandatory, and class collectively assigned to consult on a community issue, found that for all three groups over all, students’ assessment of the importance of community service was lower after the end of the class (Parker-Gwin and Mabry 1998). However, there is some evidence of positive effects of service learning, as well. In a 1998 study of a large US-based service learning program, researchers found several positive outcomes, specifically that a program longer than one year had significantly stronger civic outcomes and that older high school students retained longer-term impacts than middle school students (Melchior et al. 1999). The same study also found that the program appeared to benefit disadvantaged and non-white students more 8 academically in general (ibid.). Others have found that hybrid classroom and service learning programs that are closely connected in content produce positive results in terms of values, interests and participation in civic life (Westheimer and Kahne 2002; Parker-Gwin and Mabry 1998). In Finkel’s (2003) study of three adult civic education programs mentioned above, while he does not call it service learning, he finds that programs that had an applied component that focused on local problem solving and being in contact with local officials had better civic education outcomes particularly with regard to civic participation. This may indicate that solving a problem about which one cares in his/her community as part of the learning experience enhances desired civic outcomes (Youniss 2011, Soule 2000). Others assert that one component that is often missing from service learning programs is a focus on acquiring civic skills (Kirlin 2002; Verba et al. 1995). Additionally, there is evidence that student-led service learning projects induce greater anticipated civic participation in the future and tolerance toward outgroups (Morgan and Streb 2002). Another debate within the literature and practitioner community is about whether and how the outcomes of service learning vary between compulsory programs and voluntary ones. On one hand, some research indicates that required service has a negative effect on future civic behaviors. Helms, et. al. (2009) find that when American college students in an experiment were forced to contribute a fixed amount of time to a charity-benefitting task that they contributed substantially less than students who were given no minimum time requirement to contribute. Additionally, a study of high school graduates from Maryland, where community service has been a requirement for graduation since 1992, showed that while the requirement boosted community service among eighth graders, it reduced volunteering among older students (Helms 2013). On the other hand, a fair amount of research does support the claim that all service learning leads to the intended positive outcomes. Hart et al. (2008) find that both compulsory and voluntary service learning for high school students correlates with higher levels of adult voting and volunteerism. In a college setting, others find that requiring service as part of course work also increased some measures of civic values and engagement sentiments among participants (Westheimer and Kahne 2002, Parker-Gwin and Mabry 1998). As mentioned above, it is likely that with the variety of service programs delivered, there is some problem with comparative measurement. It is very likely that the how well the service requirement is connected to a learning program has a large impact on the likelihood of successful outcomes. A 2010 study of American high school students found that participation in clubs like debate, drama and music significantly increased the likelihood of voting in presidential elections in the future but that most sports clubs had no effect or a negative effect on future voting (Thomas and McFarland 2010). This is particularly interesting in that much of the literature about adult civic participation stresses the positive role that group memberships of any kind play in promoting political action (Putnam 2000; Booth and Richard 1998). While not an issue to be discussed at length here, others do have an ethical problem with forcing students to perform community service as a requirement for a course or for graduation, claiming that compelling service is antithetical to the principles upon which it is based. A Myriad of Desired Outcomes Another way to examine the research and literature on civic education is through the lens of desired outcomes. While researchers characterize their findings in different ways, the general approach emphasizes three categories: knowledge; values and efficacy; and participation. Generally speaking, there is more evidence showing that civic education programs promote civic and political knowledge (e.g. see Tonge, Mycock, and Jeffery 2012; Whiteley 2012; Finkel, Horowitz, and Rojo-Mendoza 2012; Syvertsen 2008; Finkel and Ernst 2005; Claes, Hooghe, and Stolle 2009; McDevitt 9 and Chaffee 2000; Hartry and Porter 2004; Vontz, Metcalf, and Patrick 2000; Finkel 2014). However, this likely for two related reasons. First it is relatively easy to measure compared to the other two categories. Program implementers and researchers can simply give students or program participants a test in the form of self-administered survey in countries with high literacy rates, or can take the slightly more arduous step of using enumerator administered surveys. Survey before the instruction begins, survey after the instruction is over, survey those who did not receive any instruction, and compare the results. This method is very useful when measuring knowledge and even critical thinking skills if the questions are well constructed, but it can suffer from selection bias when the participants and non￾participants are not randomly assigned. If participation in a civic education course is voluntary, there are likely endogenous factors which taint the analysis. For instance, if a person volunteers to participate in a civic education program, he or she very likely has a heightened awareness or interest in things related to civic knowledge and thus the differences in the effect of the instruction on knowledge acquisition will be smaller than the true effect if the instruction were applied randomly to the general population. Conversely, those who volunteer but have low civic knowledge may be more motivated to acquire it than the general population and thus show an effect larger than the true effect for the general population. This selection bias can be even more problematic when attempting to examine the effects of civic education programs on improving support for democratic values and civic participation or engagement. Those who volunteer for civic education programs probably see some value in civic action and value democratic principles more than those who do not. Additionally, while knowledge and skills can be relatively easily acquired, norms, values and attitudes are influenced by many experiences and external forces and may be more difficult to change and require a substantial amount of time to do so. Another, potentially bigger problem with measuring the effect of civic education interventions on democratic norms and values is the prevalence of social desirability bias. Even if subjects do not have a fealty toward democratic values or norms of tolerance, they likely know that they are supposed to in a democratic society. Therefore, attempting to measure norms and values in a survey format can be particularly problematic, and for measures to be valid, questions must be carefully constructed. Despite these substantial challenges and caveats, studies which may suffer from some or all of these problems have generally found that civic education programs can marginally improve participants sense of political efficacy (Keating et al. 2010, Whitely 2012, Finkel et al. 2012, Soule 2002, Hartry and Porter 2004, Hope and Jagers 2014). However, generally speaking, these studies find far less evidence if any for such programs improving levels of democratic values and attitudes and tolerance. One way to measure social trust, a significant element of democratic norms, and tolerance is to construct behavioral games which measure participants’ applied level of social trust and tolerance which can differ significantly from self-reported levels (Wilson 2011). Such games typically involve the sequential exchange of small amounts of money between two players in which the best outcome for both players is to cooperate and trust the other. However, as with many behavioral games testing a variety of concepts, the worst outcome for a given player is to fully trust the other person and have that person “defect” or not reciprocate the trust. There are several mid-level outcomes in which some trust and some mistrust still pay some dividends. The first player exhibits trust or mistrust, but the second player exhibits reciprocity or untrustworthiness. If played with co-ethnics or persons of unknown ethnicity, trust is the main focus. If played with individuals of different ethnicities as well, the game can also measure tolerance and discrimination when compared with behavior in co-ethnic or ethnically ambiguous pairs. Such games generally avoid the problem of social desirability bias in that players do not generally know what is being measured ahead of time and potential for economic gain generally trumps any desire to impress or satisfy a stranger implementing the game. 10 Civic participation and engagement is likely the most difficult outcome to measure and also suffers from the likely presence of selection bias in non-random samples. One of the most difficult problems to overcome is that a civic education program administered in one time period (t) is not likely to lead to immediate civic participation or engagement, especially if it is a short-term intervention or if the participatory behavior being measured is voting and the sample is under 18. Additionally, as described above, many civic education programs strongly encourage or require civic participation of some sort in the form of community service or project design, so measuring participation during the program (at time ) would be artificially inflated. What researchers and practitioners really want to know is whether civic education programs promote civic participation and engagement in the future, i.e., at t + 1, t + 2, t+5, t+10 or t+20. While it is possible to conduct a longitudinal study in which one surveys this population repeatedly over time after an intervention and some do (Keating et al. 2010), such research can be difficult particularly in developing countries and takes many years, making it less desirable for testing outcomes to inform program design. Therefore, researchers generally ask civic education program participants about planned future behavior. Social desirability bias is often a problem in this kind of measure, but as big a problem is the inability for individuals to predict how they will feel about civic participation in the future, especially adolescents and younger students. In short, it is very difficult to measure long-term effects of civic education programs which is their very purpose. No one ever designed a program the goal of which was to improve civic participation among those involved for six months, with the possible exception of a “get out the vote” intervention targeting a specific election. Despite the many problems with assessing the impact of civic education on civic participation and engagement, many studies have attempted to test this relationship using these methods and have found mixed results. Many have found a correlation between civic knowledge and self-reported likelihood of future voting (Cohen and Chaffee 2013, Torney-Purta et al. 2001, Maeillo et al. 2003, Hart et al. 2007). Others have found that taking civics courses increased this self-reported likelihood of future voting (Berson et al. 2013) and generally participating in civic life (Soule 2005, Hartry and Porter 2004, Vontz et al. 2000, Tonge et al. 2011, Whiteley 2012). On the other hand, other studies have found civic education to have no effect on civic participation (McDevitt and Kiousis 2006, Pasek et al. 2008, Bers and Chau 2010). One new way potentially to measure civic engagement, although not without its own limitations is, is the use of social media data collection. The biggest limitation is the role that social media itself may play in promoting civic engagement (Yamamoto et al. 2013, Cogburn and Espinoza-Vasquez 2011, Pasek et al. 2009, Zhang et al. 2009). However, some research indicates that using social network sites to seek information is a predictor of civic and political participation (Zuniga et al. 2012). Measuring online civic engagement using social media is a potentially useful way of determining the effects of civic education programs, especially in randomized control trials. Conclusion While there is a lot of research on the effects of civic education, both in the US and abroad, among adults and among adolescents, on classroom-based programs, experiential ones and hybrids, and on particular outcomes of interest, the findings are often contradictory, conditional, biased or context specific. Prior research and theorizing does, however, lead us to a number of important hypotheses which this impact evaluation will test. 11 Theory of Change and Hypotheses The theory of change builds on the decades of literature on how civic education for adolescent students can improve their civic knowledge, instill civic and democratic values and efficacy and increase civic engagement and participation. Knowledge Classroom instruction remains the main venue for delivering civic education to students. This environment has many advantages in terms of reaching many students and holding them accountable for acquiring knowledge. However, such instruction must be interactive and incorporate democratic principles into the functioning of the classroom to have a positive and lasting effect. Teachers must be trained in interactive teaching methods and encourage open discussion of civic principles, especially those that may be controversial. In addition to acquiring formal knowledge about local governance and the principles of democracy, students learn by emulating their teachers and peers. If teachers model behavior that is accepting of opinions that differ from their own, then students will be more likely to accept such differences as well. Requiring students to think critically about what it means to be an active citizen in a democracy is also essential for effective learning. Research has also established that having an open classroom environment has positive effects on acquiring civic knowledge. With regard to civic knowledge acquisition, the more open, interactive, and accepting of diverse opinions the classroom instruction is, the better the learning outcomes. H1: Specialized teacher training in interactive techniques and civic education principles will lead to better civic knowledge among students in the treatment groups. • Comparison variations: 1, 2 and 3 versus 4 H2: Students involved in civic education clubs will exhibit the highest level of civic knowledge and skills over the other treatments and control • Comparison variations: 2 versus 1,3 and 4 Democratic Values, Attitudes and Efficacy As noted in the literature review, changing values and attitudes can be more difficult and take longer than merely imparting knowledge, but it is also essential. Open, democratically run classrooms and interactive teaching facilitate the shaping of these essentials, but so, too, does putting civic principles to the test through applied learning measures. Students who take part in student-lead community service projects are more likely to see that they can instigate positive change in their communities as well as on a larger scale when they become engaged in civil society. Research suggests that these activities work best when tied to learning objectives as oppose to service for service’s sake. It also asserts that student￾organized activities have a better effect than those lead by teachers. The research is mixed about whether compulsory and voluntary service projects have better outcomes. Voluntary projects have student buy-in and may increase motivation, but they suffer from lack of participation by those from under resourced backgrounds, those who may not feel comfortable volunteering and those whose parents do not want them participating in after school activities. Compulsory projects do not get as much buy-in and may suffer from lower participant motivation, but they include many more segments of the population, including those who might be left out of a voluntary project. Regardless of the type, civics projects which are democratically run allow students to gain confidence in their ability to effect change and are likely to increase positive attitudes toward the democratic process. 12 H3: Students in the voluntary club variation and classroom project variation will express higher levels of perceived efficacy and support for democratic norms over the classroom only and control group. • Comparison variations: 2 and 3 versus 1 and 4 H4:Students in the voluntary club variation will express the highest levels of perceived efficacy and support for democratic norms. • Comparison variations: 2 versus 1, 3 and 4 H5: Students in all three treatment groups will exhibit higher levels of social trust and lower levels of intolerance than students in the control group. • Comparison variation: 1,2,3 versus 4 Civic Engagement and Participation Modeling what civic engagement looks like through community oriented service projects gives students a template for other forms of engagement and for future participation. Getting students to care about a particular community problem and attempting to remedy it, allows them to develop strong ties to their community and to continue to take an interest in improving it, whether through continued service or through active voting for political leaders who will pursue policies that help their communities. Getting students into the practice of engaging and participating in their communities is likely to create the momentum to keep participating. H6: Both voluntary afterschool civics clubs and in-class civics projects will increase students’ interest in participating in civic activities over those in the classroom instruction only group and the control group. • Comparison variations: 2 and 3 versus 1 and 4 H7: Students in the voluntary afterschool civics clubs will express the greatest interest in civic engagement. • Comparison variations: 2 (just club members or all?) versus 1, 3 and 4 H8: Those in the classroom project group will express greater interest in civic engagement than those in the program with civics clubs but who did not participate in afterschool civics clubs. • Comparison variation: 3 versus 2-non-participants Democratic Achievement Gap H9: Students lower on the socio-economic spectrum will see greater benefits in terms of civic knowledge, perceived efficacy and engagement than those at the higher end in all groups. • Comparison: lower SE 1,2,3 and 4 versus higher SE 1,2,3 and 4 H10: Minority and female students will benefit more in terms of perceived efficacy and civic participation than male majority group members in the classroom project treatment. • Comparison variation: minority and female 3 versus non-minority and male 3 H11: Minority and female students will participate at lower levels in the voluntary treatment, but those that do participate will benefit more in terms of perceived efficacy and civic participation. 13 • Comparison: minority and females 2 versus non-minority and females 2 H12: Urban students will start with higher levels of civic knowledge, participation and democratic values, but rural students will see more gains in these areas in all treatment groups. • Comparison: Urban 1,2,3 and 4 versus rural 1,2, 3 and 4 All hypotheses H13: Gains will be greater after two years of civic education in all treatment groups than after one year and will be greater than in the control group. • Comparison: 1, 2, 3 Y1 versus 1, 2, 3 Y2 (only 9th Y1 and 10th Y2) Box One. Hypotheses List by Grouping V1, 2, 3 vs. control • H1: knowledge • H5: Social trust and tolerance • H9 (low socio-econ vs high socio-econ): Democratic achievement gap—econ V2 vs. V1, 3 and control • H2: Best knowledge • H4: Best efficacy and norms • H7 (just club members): Best engagement V2 and 3 vs. V1 and control • H3: Efficacy and norms • H6: Service learning V3 vs. V2 • H8(V3 all students vs.V2 non-participants): Service learning, compulsory vs voluntary V3 • H10-minority and female vs non-minority and male: Democratic achievement gap— compulsory service learning V2 • H11-minority and female vs non-minority and male: Democratic achievement gap— voluntary service learning V1, 2, 3 Urban vs V1, 2, 3 Rural • H12: Democratic achievement gap—urban vs rural (need baseline for this) Y1 vs Y2 • H13-V1, 2 and 3 (9th Y1 and 10th Y2): duration 14 Policy Outcomes The impact evaluation will answer the following policy questions identified by USAID/Georgia and the activity implementer, PH International: I. Can civic education contribute measurably to students’ democratic capacity?1 II. Can civic education promote informed social and political involvement of students in their communities? III. Can civic education help close the “democratic achievement gap” between students from different socioeconomic backgrounds?2 IV. What civic engagement pedagogical tools have the strongest effect in improving students’ civic skills? Experimental Design The MT project offers three treatment variations to participating schools. These variations allow PH International and the Ministry of Education of Georgia to assess which civic education curricula, applied learning techniques, or combinations of pedagogical tools work best and for which students. These treatment variations are based on the best practices of high school civic education programs internationally and in Georgia. They reflect original intentions of the MT project and maximize the value of the project for policy decisions by the Government of Georgia. 1. Treatment One (T1) → Enhanced teacher training and supplementary civic curricula. This intervention includes a four day pedagogical training course for civics teachers called “Teaching Democratic Citizenship.” As part of this training, teachers receive additional instruction regarding the Georgian National Curriculum’s civic education requirements.3 The training also includes modules on teaching civic participation, lesson plans, community service projects and project planning, student assessment and creating rubrics, and interactive teaching methods. Teachers also participate in a one day training course on how to use the PH-developed supplemental textbooks, and each school receives the supplemental textbooks for each student in grades nine and ten. Teachers are offered professional development opportunities, such as participation in open lessons, regional roundtables and national civics conference. The main focus in this treatment is on giving the teacher tools to enhance their ability to give effective classroom instruction without relying on much external support throughout the school year. Students in 6 regions were involved in social media trainings and students in the 5 additional regions will receive training by January 2016. While not all students receive this training, the intent is for those who do to return to their schools and train their peers. Many students also take part in other social media activities (facebook quizzes, online conferences, blog contests) and took part in civics camp during summer 2015. Schools participating in this treatment have access to small grants initiatives for 1 Democratic capacity is defined as a mix of knowledge and skills related to influence, action, collaboration, critical thinking, leadership and communication. 2 Democratic achievement gap refers to inequalities in family and community backgrounds of high school students that may hinder their ability to become effectively engaged citizens. 3 This “core” training includes national curriculum requirements and modules, which cover topics and tools included in the national curriculum, teacher professional standard and other documents of MES. These topics include: the role of school in civics teaching, identification and development of additional teaching materials, assessment, project-based teaching, interactive teaching methodologies, among others. 15 school/classroom improvement. Teachers are responsible for applying for the grants and 20 are awarded per year. 2. Treatment Two (T2) → T1 plus voluntary student civic clubs. This treatment includes all elements of T1 plus NGO support to organize voluntary student civic education clubs in participating schools. Teachers receive an additional one day training in how to use the civics club toolbox, “Instructions for the Work of School-based Civics Clubs.” Each school receives 20 copies of the toolboxes for students to use, 2 for teachers and 1 for the school library. This intervention aims to replicate the model of the previous USAID-supported Applied Civic Education and Teacher Training (ACETT) project with some modification in the role of NGOs in supporting activities of student clubs by not specifying the number of deliverables from the NGOs, but rather allowing the students to guide the process. Schools receive help from local NGOs to facilitate establishment and functioning of school-based civics clubs and to assist with planning and implementation of civic club projects as needed. The civics clubs have student leaders, who carry out activities with assistance from the teacher-facilitator and NGO. For instance, if students plan a project that requires advocacy of some sort, the NGO will offer them training in advocacy skills. Student participation in civics clubs is entirely voluntary and does not count toward their final evaluation in the civics class. The civics clubs are driven largely by student leaders but with teacher-facilitator and NGO assistance. Students in the clubs identify a problem in their communities, determine how they can themselves address the problem, try to mobilize partners among local decision-makers/agencies and carry out activities to address the problem. Additionally, students from school civics clubs participate in group visits to state and non-state agencies and media outlets, meetings with expert guests, trainings organized by partner NGOs on civic education topics. The small grants applications in this treatment are initiated by students in the civics clubs, and an additional 20 grants are awarded to schools in this treatment. The main focus of this treatment is active, voluntary student engagement in civic activities with the support of teachers and NGOs. 3. Treatment Three (T3) → T1 plus required class civics projects. This treatment includes all of the elements of T1 plus an additional one day training for teachers in how to lead classroom projects and in how to use the toolbox, “Instructions for Classroom Projects in Civic Education.” The toolboxes give guidance on how to use a 6 step process to implement a classroom project: investigation, preparation, action, reflection, demonstration and evaluation. Schools receive toolboxes for each student in grades 9 and 10 to use so that every student has access to her/his own copy, plus 2 for teachers and 1 for the library. Each civics teacher is asked to carry out at least 1 classroom project during an academic year with the class that she/he teaches, with participation of all students in a class. Partner NGOs provide the needed resources for implementation of classroom projects, when requested by teachers or occasionally by students if the teacher is passive. Service learning is conducted as a class project in which all students in the class participate. Each class completes projects as a class; teachers do not break up the class into smaller groups of students or encourage students to pursue individual projects. Since the types of projects are defined by the teachers with student input and interest taken into account, the only limit on the “type” of project is that is be connected to the curriculum. Twenty additional small grants are awarded in this treatment to support classroom projects. 16 NGOs play similar kinds of project backstopping role in Treatment 3 as they do in Treatment 2. NGOs support implementation of classroom projects based on the needs identified by a teacher leading a particular project with her/his students. For example, if transportation is needed, NGOs shall arrange it for all students in the class, or if some stationery is needed, the NGO will provide it. Additionally, if meeting with state or non-state agencies or expert guests relates to the classroom project, then the NGO will facilitate such meetings. The main focus of this treatment is required participation by the entire class in a teacher-led civics project, with assistance from the NGO as needed. 4. Control Group Control group schools meet the same eligibility criteria as treatment schools and are randomly assigned. 63 school districts are included in the control group. It is important that control group schools are not informed of their status and that regional NGOs make absolutely no contact with these schools. Table One. Summary of Momavlis Taoba Civic Education Treatments T1 T2 T3 Control Enhanced teacher training & supplemental civic education textbooks Voluntary student civic clubs In-class “service-learning” project Assignment of Schools to Treatment Variations The MT activity will target 480 secondary schools out of a total of 2,329 schools in Georgia. The interventions will be offered in all regions in Georgia and, per the recommendation of activity implementer, the impact evaluation team adopted a random sample selection that is stratified at the regional level. With the help of regional implementing partners, all 2400 schools in Georgia were grouped into 368 school districts which serve as the unit of randomization for impact evaluation purposes. Of the 368 school districts only 252 have at least one secondary school with at least 10 students in grade 9 and 25 students in grades 9 and 10 combined, which is the minimum number of students needed for purposes of statistical power analysis. All these 252 school districts were included in the impact evaluation and were randomly assigned to one of the three treatments of the MT program or to the control group. All participating schools within a school district will receive the same treatment variation. Schools in Racha-Lechkhumi do not meet the criteria for the impact evaluation but will be invited to participate in the project. 17 Table Two. Momavlis Taoba Treatment Assignment by Region T1 T2 T3 Control Total Adjara 5 5 5 5 20 Guria 3 3 3 3 12 Imereti 12 12 12 12 48 Kakheti 7 7 7 7 28 Kvemo Kartli 6 6 6 6 24 Mtskheta-Mtianeti 2 2 2 2 8 Samegrelo-Zemo Svaneti 6 6 6 6 24 Samtskhe-Javakheti 3 3 3 3 12 Shida Kartli 7 7 7 7 28 Tbilisi 12 12 12 12 48 Racha-Lechkhumi 0 0 0 0 0 Total Schools 63 63 63 63 252 Measurement and Statistical Power Surveys This impact evaluation will use surveys and monitoring reports of regional NGOs to measure civic education achievements of students in schools participating in the activity. Below is a list of core civic education indicators. ● Civic Behaviors – Indicators will assess students’ perceived ability to engage in civic action (e.g., organize a meeting), express their voice (e.g., sign a petition) and think critically. ● Civic Skills – Indicators will assess students’ skills relevant for civic participation, especially those related to influence and action in their communities. ● Conventional and Alternative Civic Engagement – Indicators will measure students expected likelihood of participation in various types of civic activities. Given that students in MT schools are ineligible to vote and political activities in secondary schools in Georgia are forbidden, the student surveys will target alternative community activities like expressed interest in joining special interest groups and service-learning. ● Political Efficacy -- Self-efficacy directly influence how people feel about themselves and the types of activities they choose to engage in. ● Citizenship Types -- Different survey measures will target the various aspects of the concept of a "good citizen" such as: personally responsible, participatory, and justice-oriented. ● Political Conversation and Critical Thinking – These measures will assess students’ communication with parents, teachers, friends and classmates about current events ● Values and Personal Beliefs -- Values play an important role as standards for personal behavior and as a basis for individual views and positions on public policies. These indicators will measure the importance students place on improving ethnic relations in Georgia, tolerance, gender equality, helping others, protecting the environment, civic participation, social trust, etc. ● Students Assessment of School Climate -- Students assessments of the climate at their school are associated with their academic achievement and with their civic dispositions and values. 18 The survey questionnaires for students and teachers are based on previous studies by the Center for Information and Research on Civic Learning and Engagement (CIRCLE) of the University of Maryland’s School of Public Policy, and on the International Civic and Citizenship Education Study of the International Association for the Evaluation of Educational Achievement. Table Three. Evaluation Survey Timeline, 2015-2017 Student and Teacher Baseline One (2015- 16) October 2015 - February 2016 Endline One May-June 2016 Student and Teacher Baseline Two (2016- 2017) October 2016 Endline Two May-June 2017 Statistical Power Power analyses for the impact evaluation design are done with the Optimal Design statistical software. Given the nature of the experimental design and available school and student data, the effect size approach is most applicable for power analysis in our case. The experimental design used in this study is a two-level cluster randomized trial (2-CRT) with students nested within schools where schools are the unit of randomization. We make the following assumptions for power calculations: Based on results of adolescent civic education literature, for purposes of our power calculations, we expect the cluster-level covariate to explain 50% of the variation in civic achievements of students. i.e., last year’s (or baseline) scores explain 50 percent of the variation in students’ civic achievements at the end of the academic year. Baseline and endline surveys during 2015-2016 academic year will provide precise estimates of R2 L2, L1. Statistical power, P=80%. Desired level of statistical significance, α=95%. At least 25 students combined in grades 9 and 10 in each school, n=25. Based on the design, we expect about 15% of the variation to lie between schools, i.e., correlation coefficient, ρ=0.15. The chart below shows how the MDES changes as a function of the total number of schools per treatment, holding all above parameters constant. We find that for J = 63 per each treatment and control group, the MDES is 0.22 effect size units. 19 Figure One. Momavlis Taoba Impact Evaluation: Effect Size Approach to Power Analysis (Student-level Outcomes). Momavlis Taoba Impact Evaluation: Effect Size Approach to Power Analysis (Student-level Outcomes). E f f e c t S i z e - M D E S 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 = 0.050 n = 25 P= 0.80, = 0.15 P= 0.80, = 0.15,R2 L2 = 0.50 30 55 80 105 130 Number of schools 20 Threats to Impact Evaluation Implementation threats for Momavlis Taoba impact evaluation include attrition, spillover and compliance. The table below summarizes threats that were identified during the impact evaluation scoping trip and solutions suggested by PH International and USAID. The Impact Evaluation Team will continue to work closely with PH International to monitor the implementation of the intervention and propose strategies for how to deal with these issues during the project implementation. Threat Mitigation Strategy Attrition Schools participation in the program is entirely voluntarily and they may drop out of the project along the way. Spillover Schools near each other or in the same communities may compare what they are doing and begin mixing approaches, which hinders measurement of impact. Media campaigns as well as social media and website increase awareness of other treatments/approaches and may alter the behavior of some schools, or may inspire schools in the control group to begin implementing some of the project components on their own which would lessen our ability to compare impacts. Compliance Schools and civic education teachers may not do what they are supposed to do, may add other activities or may stop doing activities that are important for their treatment approach. PH International will coordinate with the Ministry of Education to secure their support for the project. USAID and PH already have strong relations with the Ministry of Education and it is not expected that attrition will be managed successfully. Randomization at the school district level means that much of the spillover from geographic proximity will be contained within the school district. However, other aspects related to use of social media, TV, radio, and internet dissemination of information (especially through the www.civics.ge and www.initiatives.ge websites) are being evaluated on a regular basis. The impact evaluation team will coordinate closely with PH International to mitigate these issues. PH and USAID have proposed several reasonable solutions to most compliance issues to incentivize and monitor compliance with program protocols by civics teachers and NGOs. The main compliance issue is that schools in T1 and T3 will implement T2 due to civic teachers’ familiarity with T2 from previous iterations of civic education programs in Georgia. 21 References Alivernini, Fabio, and Sara Manganelli. 2011. “Is There a Relationship between Openness in Classroom Discussion and Students’ Knowledge in Civic and Citizenship Education?” Procedia-Social and Behavioral Sciences 15: 341–45. Amnå, Erik. 2012. “How Is Civic Engagement Developed over Time? Emerging Answers from a Multidisciplinary Field.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 611–27. doi:10.1016/j.adolescence.2012.04.011. Antal, Carrie, and Peter Easton. 2009. “Indigenizing Civic Education in Africa: Experience in Madagascar and the Sahel.” International Journal of Educational Development 29 (6): 599–611. Avery, Patricia G., Karen Bird, Sandra Johnstone, John L. Sullivan, and Kristina Thalhammer. 1992. “Exploring Political Tolerance with Adolescents.” Theory & Research in Social Education 20 (4): 386–420. Avery, Patricia G., and Annette M. Simmons. 2001. “Civic Life as Conveyed in United States Civics and History Textbooks.” International Journal of Social Education 15 (2): 105–30. Beaumont, Elizabeth. 2011. “Promoting Political Agency, Addressing Political Inequality: A Multilevel Model of Internal Political Efficacy.” The Journal of Politics 73 (01): 216–31. Bennion, Dr Elizabeth A. 2006. “Civic Education and Citizen Engagement: Mobilizing Voters as a Required Field Experiment.” Journal of Political Science Education 2 (2): 205–27. doi:10.1080/15512160600686555. Bers, Marina Umaschi, and Clement Chau. 2006. “Fostering Civic Engagement by Building a Virtual City.” Journal of Computer-Mediated Communication 11 (3): 748–70. Berson, Michael J., Liliana Rodríguez-Campos, Connie Walker-Egea, Corina Owens, and Aarti Bellara. 2013. “Youth Engagement in Electoral Activities: A Collaborative Evaluation of a Civic Education Project.” Journal of Education and Training Studies 2 (1): 81–87. Blair, Harry. 2003. “Jump-Starting Democracy: Adult Civic Education and Democratic Participation in Three Countries.” Democratization 10 (1): 53–76. doi:10.1080/714000108. Booth, John A., and Patricia Bayer Richard. 1998. “Civil Society, Political Capital, and Democratization in Central America.” The Journal of Politics 60 (03): 780–800. Bratton, Michael, Philip Alderfer, Georgia Bowser, and Joseph Temba. 1999. “The Effects of Civic Education on Political Culture: Evidence from Zambia.” World Development 27 (5): 807–24. doi:10.1016/S0305-750X(99)00031-5. Camino, Linda, and Shepherd Zeldin. 2002. “From Periphery to Center: Pathways for Youth Civic Engagement in the Day-to-Day Life of Communities.” Applied Developmental Science 6 (4): 213– 20. Campbell, David E. 2008. “Civic Side of School Choice: An Empirical Analysis of Civic Education in Public and Private Schools, The.” BYU L. Rev., 487. Cicognani, Elvira, Bruna Zani, Bernard Fournier, Claire Gavray, and Michel Born. 2012. “Gender Differences in Youths’ Political Engagement and Participation. The Role of Parents and of Adolescents’ Social and Civic Participation.” Journal of Adolescence, Political and Civic 22 Engagement Development in Adolescence, 35 (3): 561–76. doi:10.1016/j.adolescence.2011.10.002. Claes, Ellen, Marc Hooghe, and Dietlind Stolle. 2009. “The Political Socialization of Adolescents in Canada: Differential Effects of Civic Education on Visible Minorities.” Canadian Journal of Political Science 42 (03): 613–36. Cogburn, Derrick L., and Fatima K. Espinoza-Vasquez. 2011. “From Networked Nominee to Networked Nation: Examining the Impact of Web 2.0 and Social Media on Political Participation and Civic Engagement in the 2008 Obama Campaign.” Journal of Political Marketing 10 (1-2): 189–213. Cohen, Alison, and Benjamin Chaffee. 2013. “The Relationship between Adolescents’ Civic Knowledge, Civic Attitude, and Civic Behavior and Their Self-Reported Future Likelihood of Voting.” Education, Citizenship and Social Justice 8 (1): 43–57. Conover, Pamela Johnston, and Donald D. Searing. 2000. “A Political Socialization Perspective.” Rediscovering the Democratic Purposes of Education, 91–124. Crocetti, Elisabetta, Parissa Jahromi, and Wim Meeus. 2012. “Identity and Civic Engagement in Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 521–32. doi:10.1016/j.adolescence.2011.08.003. Eckel, Catherine, Philip J. Grossman, Cathleen A. Johnson, Angela CM de Oliveira, Christian Rojas, and Rick Wilson. 2011. “Social Norms of Sharing in High School: Teen Giving in the Dictator Game.” Journal of Economic Behavior & Organization 80 (3): 603–12. Eckstein, Katharina, Peter Noack, and Burkhard Gniewosz. 2012. “Attitudes toward Political Engagement and Willingness to Participate in Politics: Trajectories throughout Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 485–95. doi:10.1016/j.adolescence.2011.07.002. Erentaitė, Rasa, Rita Žukauskienė, Wim Beyers, and Rasa Pilkauskaitė-Valickienė. 2012. “Is News Media Related to Civic Engagement? The Effects of Interest in and Discussions about the News Media on Current and Future Civic Engagement of Adolescents.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 587–97. doi:10.1016/j.adolescence.2011.12.008. Ferreira, Pedro D., Cristina N. Azevedo, and Isabel Menezes. 2012. “The Developmental Quality of Participation Experiences: Beyond the Rhetoric That ‘participation Is Always Good!’” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 599–610. doi:10.1016/j.adolescence.2011.09.004. Finkel, Steve E., Christopher A. Sabatini, and Gwendolyn G. Bevis. 2000. “Civic Education, Civil Society, and Political Mistrust in a Developing Democracy: The Case of the Dominican Republic.” World Development 28 (11): 1851–74. Finkel, Steven E. 2002. “Civic Education and the Mobilization of Political Participation in Developing Democracies.” The Journal of Politics 64 (4): 994–1020. ———. 2003. “Can Democracy Be Taught?” Journal of Democracy 14 (4): 137–51. ———. 2014. “The Impact of Adult Civic Education Programmes in Developing Democracies.” Public Administration and Development 34 (3): 169–81. Finkel, Steven E., and Howard R. Ernst. 2005. “Civic Education in Post-Apartheid South Africa: Alternative Paths to the Development of Political Knowledge and Democratic Values.” Political Psychology 26 (3): 333–64. 23 Finkel, Steven E., Jeremy Horowitz, and Reynaldo T. Rojo-Mendoza. 2012. “Civic Education and Democratic Backsliding in the Wake of Kenya’s Post-2007 Election Violence.” The Journal of Politics 74 (01): 52–65. Flanagan, Constance, Wim Beyers, and Rita Žukauskienė. 2012. “Political and Civic Engagement Development in Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 471–73. doi:10.1016/j.adolescence.2012.04.010. Flanagan, Constance, and Peter Levine. 2010. “Civic Engagement and the Transition to Adulthood.” The Future of Children 20 (1): 159–79. Galston, William A. 2004. “Civic Education and Political Participation.” PS: Political Science and Politics 37 (2): 263–66. Gastil, John. 2004. “Adult Civic Education through the National Issues Forums: Developing Democratic Habits and Dispositions through Public Deliberation.” Adult Education Quarterly 54 (4): 308–28. Gil de Zúñiga, Homero, Nakwon Jung, and Sebastián Valenzuela. 2012. “Social Media Use for News and Individuals’ Social Capital, Civic Engagement and Political Participation.” Journal of Computer￾Mediated Communication 17 (3): 319–36. Glennerster, Rachel and Kudzai Takavarasha. 2013. Running Randomized Evaluations. A Practical Guide. Princeton University Press. Goemans, Hein. 2006. “Bounded Communities: Territoriality, Territorial Attachment, and Conflict.” Territoriality and Conflict in an Era of Globalization, 25–61. Greene, Jay P. 2000. “Niemi and Junn, Civic Education.” SOCIAL SCIENCE QUARTERLY-AUSTIN- 81 (2): 696–696. Hahn, Carole L. 2003. “Democratic Values and Citizen Action: A View from US Ninth Graders.” International Journal of Educational Research 39 (6): 633–42. Helms, Sara, Erik Angner, Brian Scott, and Sarah Culver. 2009. “Mandated Volunteering: An Experimental Approach.” http://economics.lafayette.edu/files/2010/03/AngnerPaper1.pdf. Helms, Sara E. 2013. “Involuntary Volunteering: The Impact of Mandated Service in Public Schools.” Economics of Education Review 36: 295–310. Hope, Elan C., and Robert J. Jagers. 2014. “The Role of Sociopolitical Attitudes and Civic Education in the Civic Engagement of Black Youth.” Journal of Research on Adolescence 24 (3): 460–70. Kahne, Joseph E., and Susan E. Sporte. 2008. “Developing Citizens: The Impact of Civic Learning Opportunities on Students’ Commitment to Civic Participation.” American Educational Research Journal 45 (3): 738–66. Keating, Avril, David Kerr, Thomas Benton, Ellie Mundy, and Joana Lopes. 2010. “Citizenship Education in England 2001-2010: Young People’s Practices and Prospects for the Future: The Eighth and Final Report from the Citizenship Education Longitudinal Study (CELS).” http://dera.ioe.ac.uk/11579/. Kim, Taehan, Constance A. Flanagan, and Alisa A. Pykett. 2015. “Adolescents’ Civic Commitments in Stable and Fledgling Democracies: The Role of Family, School, and Community.” Research in Human Development 12 (1-2): 28–43. doi:10.1080/15427609.2015.1010344. Kirlin, Mary. 2002. “Civic Skill Building: The Missing Component in Service Programs?” Political Science & Politics 35 (03): 571–75. Kolb, David. 1984. “Experiential Learning as the Science of Learning and Development.” Englewood Cliffs NPH, editor1984. 24 Langton, Kenneth P., and M. Kent Jennings. 1968. “Political Socialization and the High School Civics Curriculum in the United States.” American Political Science Review 62 (03): 852–67. doi:10.2307/1953435. Lerner, Richard M., Celia B. Fisher, and Richard A. Weinberg. 2000. “Toward a Science for and of the People: Promoting Civil Society through the Application of Developmental Science.” Child Development 71 (1): 11–20. Levinson, Bradley AU. 2004. “Hopes and Challenges for the New Civic Education in Mexico: Toward a Democratic Citizen without Adjectives.” International Journal of Educational Development 24 (3): 269–82. Lichbach, Mark Irving. 1995. The Rebel’s Dilemma. Ann Arbor: University of Michigan Press. Maiello, Carmine, Fritz Oser, and Horst Biedermann. 2003. “Civic Knowledge, Civic Skills and Civic Engagement.” European Educational Research Journal 2 (3): 384–95. Markus, Gregory B., Jeffrey PF Howard, and David C. King. 1993. “Notes: Integrating Community Service and Classroom Instruction Enhances Learning: Results from an Experiment.” Educational Evaluation and Policy Analysis 15 (4): 410–19. Martínez, M. Loreto, Pilar Peñaloza, and Cristina Valenzuela. 2012. “Civic Commitment in Young Activists: Emergent Processes in the Development of Personal and Collective Identity.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 474–84. doi:10.1016/j.adolescence.2011.11.006. Marzana, Daniela, Elena Marta, and Maura Pozzi. 2012. “Social Action in Young Adults: Voluntary and Political Engagement.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 497–507. doi:10.1016/j.adolescence.2011.08.013. McDevitt, Michael, and Steven Chaffee. 2000. “Closing Gaps in Political Communication and Knowledge Effects of a School Intervention.” Communication Research 27 (3): 259–92. McDevitt, Michael, and Spiro Kiousis. 2006. “Deliberative Learning: An Evaluative Approach to Interactive Civic Education.” Communication Education 55 (3): 247–64. Melchior, Alan, Joseph Frees, Lisa LaCava, Chris Kingsley, Jennifer Nahas, Jennifer Power, Gus Baker, et al. 1999. “Summary Report: National Evaluation of Learn and Serve America.” http://digitalcommons.unomaha.edu/slceeval/36/?utm_source=digitalcommons.unomaha.edu %2Fslceeval%2F36&utm_medium=PDF&utm_campaign=PDFCoverPages. Morduchowicz, Roxana, Edgardo Catterberg, Richard G. Niemi, and Frank Bell. 1996. “Teaching Political Information and Democratic Values in a New Democracy: An Argentine Experiment.” Comparative Politics 28 (4): 465–76. doi:10.2307/422053. Morgan, William, and Matthew Streb. 2002. “Promoting Civic Activism: Student Leadership in Service￾Learning.” Politics & Policy 30 (1): 161–88. Niemi, Robert, and Jane Junn. 1998. What Makes Students Learn. New Haven: Yale University Press. Nucci, Larry, Tobias Krettenauer, and Darcia Narvaez. 2015. Handbook of Moral and Character Education. Routledge. Owen, Diana. 2015. “High School Students’ Acquistition of Civic Knowledge: The Impact of We the People.” Center for Civic Education. Parker-Gwin, Rachel, and J. Beth Mabry. 1998. “Service Learning as Pedagogy and Civic Education: Comparing Outcomes for Three Models.” Teaching Sociology, 276–91. 25 Pasek, Josh, Lauren Feldman, Daniel Romer, and Kathleen Hall Jamieson. 2008. “Schools as Incubators of Democratic Participation: Building Long-Term Political Efficacy with Civic Education.” Applied Development Science 12 (1): 26–37. Pasek, Josh, eian more, and Daniel Romer. 2009. “Realizing the Social Internet? Online Social Networking Meets Offline Civic Engagement.” Journal of Information Technology & Politics 6 (3-4): 197–215. doi:10.1080/19331680902996403. Pavlova, Maria K., Astrid Körner, and Rainer K. Silbereisen. 2015. “Perceived Social Support, Perceived Community Functioning, and Civic Participation Across the Life Span: Evidence from the Former East Germany.” Research in Human Development 12 (1-2): 100–117. doi:10.1080/15427609.2015.1010351. Pearson, Frederic S. 2001. “Dimensions of Conflict Resolution in Ethnopolitical Disputes.” Journal of Peace Research 38 (3): 275–87. doi:10.1177/0022343301038003001. Pérez, J. Carola, and Patricio Cumsille. 2012. “Adolescent Temperament and Parental Control in the Development of the Adolescent Decision Making in a Chilean Sample.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 659–69. doi:10.1016/j.adolescence.2011.09.002. Putnam, Robert. 2000. “D.(1993). Making Democracy Work.” Princeton: Princeton University Press.•(1995a). Bowling Alone: America’s Declining Social Capital. Journal of Democracy 6 (1): 65–78. Reinders, Heinz, and James Youniss. 2006. “School-Based Required Community Service and Civic Development in Adolescents.” Applied Developmental Science 10 (1): 2–12. doi:10.1207/s1532480xads1001_1. Sambanis, Nicholas, and Moses Shayo. 2013. “Social Identification and Ethnic Conflict.” American Political Science Review 107 (02): 294–325. doi:10.1017/S0003055413000038. Schmid, Christine. 2012. “The Value ‘social Responsibility’ as a Motivating Factor for Adolescents’ Readiness to Participate in Different Types of Political Actions, and Its Socialization in Parent and Peer Contexts.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 533–47. doi:10.1016/j.adolescence.2012.03.009. Sears, David O., and Sheri Levy. 2003. “Childhood and Adult Political Development.” http://psycnet.apa.org/psycinfo/2003-88243-003. Šerek, Jan, Lenka Lacinová, and Petr Macek. 2012. “Does Family Experience Influence Political Beliefs? Relation between Interparental Conflict Perceptions and Political Efficacy in Late Adolescence.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 577–86. doi:10.1016/j.adolescence.2011.10.001. Slomczynski, Kazimierz M., and Goldie Shabad. 1998. “Can Support for Democracy and the Market Be Learned in School? A Natural Experiment in Post-Communist Poland.” Political Psychology 19 (4): 749–79. “Social Media Use for News and Individuals’ Social Capital, Civic Engagement and Political Participation - Gil de Zúñiga - 2012 - Journal of Computer-Mediated Communication - Wiley Online Library.” 2015. Accessed July 28. http://onlinelibrary.wiley.com/doi/10.1111/j.1083- 6101.2012.01574.x/full. Soule, Suzanne. 2000. “Beyond Communism and War: The Effect of Civic Education on the Democratic Attitudes and Behavior of Bosnian and Herzegovinian Youth.” http://eric.ed.gov.proxy.library.georgetown.edu/?id=ED447046. 26 Syvertsen, Amy K. 2008. Best Practices in Civic Education: Changes in Students?` Civic Outcomes. DIANE Publishing. https://books.google.co.jp/books?hl=en&lr=&id=27- Gzt8VWUEC&oi=fnd&pg=PA2&dq=Syvertsen,+Amy+K.+2008.+Best+Practices+in+Civic+Educatio n:+Changes+in+Students%3F%60+Civic+Outcomes.+&ots=L3_vkLpZ4C&sig=_ejYagXsLNg5ThPaS 7UMBitg81Q. Thomas, Reuben, and Daniel McFarland. 2010. THE CENTER FOR INFORMATION & RESEARCH ON CIVIC LEARNING & ENGAGEMENT. Toft, M. D. 2003. The Geography of Ethnic Violence: Identity, Interests, and the Indivisibility of Territory. Princeton Univ Pr. Tonge, Jon, Andrew Mycock, and Bob Jeffery. 2012. “Does Citizenship Education Make Young People Better-Engaged Citizens?” Political Studies 60 (3): 578–602. Torney-Purta, Judith. 2002. “Patterns in the Civic Knowledge, Engagement, and Attitudes of European Adolescents: The IEA Civic Education Study.” European Journal of Education, 129–41. Torney-Purta, Judith, and Jo-Ann Amadeo. 2003. “A Cross-National Analysis of Political and Civic Involvement among Adolescents.” Political Science and Politics 36 (02): 269–74. Torney-Purta, Judith, Carolyn H. Barber, and Britt Wilkenfeld. 2007. “Latino Adolescents’ Civic Development in the United States: Research Results from the IEA Civic Education Study.” Journal of Youth and Adolescence 36 (2): 111–25. Torney-Purta, Judith, Rainer Lehmann, Hans Oswald, and Wolfram Schulz. 2001. Citizenship and Education in Twenty-Eight Countries: Civic Knowledge and Engagement at Age Fourteen. ERIC. http://eric.ed.gov.proxy.library.georgetown.edu/?id=ED452116. Torney-Purta, Judith, and John Schwille. 1986. “Civic Values Learned in School: Policy and Practice in Industrialized Nations.” Comparative Education Review, 30–49. Torney-Purta, Judith, John Schwille, and Jo-Ann Amadeo. 1999. “The IEA Civic Education Study: Expectations and Achievements of Students in Thirty Countries. ERIC Digest.,” December. http://eric.ed.gov/?id=ED435585. van Goethem, Anne A. J., Anne van Hoof, Marcel A. G. van Aken, Quinten A. W. Raaijmakers, Jan Boom, and Bram Orobio de Castro. 2012. “The Role of Adolescents’ Morality and Identity in Volunteering. Age and Gender Differences in a Process Model.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 509–20. doi:10.1016/j.adolescence.2011.08.012. Verba, Sidney, Kay Lehman Schlozman, Henry E. Brady, and Henry E. Brady. 1995. Voice and Equality: Civic Voluntarism in American Politics. Vol. 4. Cambridge Univ Press. http://journals.cambridge.org/production/action/cjoGetFulltext?fulltextid=6272664. Vontz, Thomas S., Kim K. Metcalf, and John J. Patrick. 2000. “‘ Project Citizen’ and the Civic Development of Adolescent Students in Indiana, Latvia, and Lithuania.” http://eric.ed.gov/?id=ED447047. Westheimer, Joel, and Joseph Kahne. 2002. “Educating the‘ Good’ Citizen: The Politics of School-Based Civic Education Programs.” http://eric.ed.gov/?id=ED474166. Whiteley, Paul. 2012. “Does Citizenship Education Work? Evidence from a Decade of Citizenship Education in Secondary Schools in England.” Parliamentary Affairs, gss083. Wilson, Rick K. 2011. “The Contribution of Behavioral Economics to Political Science.” Annual Review of Political Science 14: 201–23. 27 Wray-Lake, Laura, and Constance A. Flanagan. 2012. “Parenting Practices and the Development of Adolescents’ Social Trust.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 549–60. doi:10.1016/j.adolescence.2011.09.006. Yamamoto, Masahiro, Matthew J. Kushin, and Francis Dalisay. 2013. “Social Media and Mobiles as Political Mobilization Forces for Young Adults: Examining the Moderating Role of Online Political Expression in Political Participation.” New Media & Society, 1461444813518390. Youniss, James. 2011a. “Civic Education: What Schools Can Do to Encourage Civic Identity and Action.” Applied Developmental Science 15 (2): 98–103. ———. 2011b. “Civic Education: What Schools Can Do to Encourage Civic Identity and Action.” Applied Developmental Science 15 (2): 98–103. doi:10.1080/10888691.2011.560814. Youniss, James, Susan Bales, Verona Christmas-Best, Marcelo Diversi, Milbrey Mclaughlin, and Rainer Silbereisen. 2002. “Youth Civic Engagement in the Twenty-First Century.” Journal of Research on Adolescence 12 (1): 121–48. Zhang, Weiwu, Thomas J. Johnson, Trent Seltzer, and Shannon L. Bichard. 2009. “The Revolution Will Be Networked: The Influence of Social Networking Sites on Political Attitudes and Behavior.” Social Science Computer Review. http://ssc.sagepub.com/content/early/2009/06/12/0894439309335162.short. Zhang, Xiaoyun, Haiping Wang, Yan Xia, Xiaohong Liu, and Eunju Jung. 2012. “Stress, Coping and Suicide Ideation in Chinese College Students.” Journal of Adolescence, Political and Civic Engagement Development in Adolescence, 35 (3): 683–90. doi:10.1016/j.adolescence.2011.10.003. 28 Appendix One: GIS Maps of MT Impact Evaluation Schools 29 Figure One. Distribution of MT IE schools by region. 30 Figure Two. Distribution of MT IE schools by municipality population. 31 Figure Three. Distribution of MT IE schools by ethnicity. 32 Appendix Two: Survey Questionnaire (Attached as a separate file) 33 CIVIC EDUCATION ASSESSMENT STUDENT QUESTIONNAIRE SCHOOL ID GRADE / SURVEY NUMBER / 1 Dear Student, Your school has been selected to participate in a nation-wide survey that will ask you for your opinions about civics and citizenship in Georgia. This survey is conducted in coordination with the Georgian Ministry of Education and Science to assess and strengthen civic education in Georgia. It will be administered by survey monitors from the Caucasus Research Resource Centers (CRRC -Georgia) twice each school year for the next two years. However, agreeing to participate now does not mean that you must participate when asked to participate again. The survey will take up to one hour to complete. There are no correct or incorrect answers. You should provide the response that is right for you. Your teachers will not be able to see how you completed this questionnaire, and your grade does not depend on answering the questions. This survey is completely anonymous and confidential. This means that no one will know how you answered the questions, and that your responses to the questions will not be shared with anyone. You will not be asked to provide your name or student identification number. Once all of the surveys are completed, your responses will be put together with the responses of all students taking the survey and will be used to write a report that will help the Ministry of Education and Science improve civic education instruction in Georgia. We ask you to take this survey seriously and respond with honest answers so that the report truly represents what you have to say. Please address the monitors if you have any questions. If you agree to participate in this survey, please complete the questionnaire. CRRC Georgia monitors now will provide instructions. 2 PLEASE READ EACH QUESTION CAREFULLY AND ANSWER IT AS WELL AS YOU CAN. Q1 When were you born? (Please write month and year.) Month: Year: Q2 Are you a girl or a boy? Girl ............................................................................................................................................................ Boy ........................................................................................................................................................... Q3 Which of the following levels of education do you expect to complete? (Please mark only one box.) I do not expect to complete secondary school......................................................................................... Secondary school (complete grade 12) ................................................................................................... Technical / Vocational education .............................................................................................................. University degree (for example, a bachelor's degree) ............................................................................. Post-graduate education (for example, a master's degree or doctorate) ................................................ Questions 4 to 7 ask information about your family. If you do not want to answer these questions, please continue to question 8. Q4 What language do you speak at home most of the time? Georgian................................................................................................................................................... Armenian .................................................................................................................................................. Azerbaijani................................................................................................................................................ Russian .................................................................................................................................................... Other......................................................................................................................................................... 3 Q5 Please think of your mother (or, if your mother is not living with you, of a female guardian you live with – a grandmother, a stepmother, an aunt). What is the highest level of education she completed? (If you are not sure which box to choose, please raise your hand and ask the survey monitor for help.) No formal education ................................................................................................................................. Primary education .................................................................................................................................... Secondary education................................................................................................................................ Technical / Vocational education .............................................................................................................. Higher education - BA .............................................................................................................................. Higher education - MA or Specialist degree............................................................................................. Post-graduate degree .............................................................................................................................. I do not know ............................................................................................................................................ Q6 Now please think of your father (or, if your father is not living with you, of a male guardian you live with – a grandfather, a stepfather, an uncle). What is the highest level of education he completed? (If you are not sure which box to choose, please raise your hand and ask the survey monitor for help.) No formal education ................................................................................................................................. Primary education .................................................................................................................................... Secondary education................................................................................................................................ Technical / Vocational education .............................................................................................................. Higher education - BA .............................................................................................................................. Higher education - MA or Specialist degree............................................................................................. Post-graduate degree .............................................................................................................................. I do not know ............................................................................................................................................ Q7 How often do your parents or guardians discuss community and social issues at home? (Please mark only one box in each row.) Never or At least once a At least once a Daily or Not hardly ever month week almost daily applicable Mother / Female Guardian Father / Male Guardian 4 Q8 On a normal school day, how much time do you spend doing each of the following activities outside of school? (Please mark only one box in each row.) Less than About 30-60 About 1-2 More than 2 No time 30 minutes minutes hours hours Watching television Homework or study for school Reading for enjoyment Chatting with friends over the phone Spending time with friends Attending classes with tutor(s) Participating in sports or attending dance/arts classes Q9 How often are you involved in each of the following activities outside of school? (Please mark only one box in each row.) Never or At least once a At least once a Daily or hardly ever month week almost daily Talking with your family members about community or social issues Using television, newspapers, or the internet to inform yourself about the news Talking with friends about community and social issues Q10 Have you ever been involved in activities of any of the following organizations, clubs, or groups, whether formal or informal? (Please mark only one box in each row.) Yes, within Yes, but the last 12 more than months a year ago No, never Environmental organization Human rights organization A voluntary group doing something to help the community A charity 5 Q11 At school, have you ever done any of the following activities? Please think about all schools you have been enrolled at since the fifth grade. (Please mark only one box in each row.) Yes, within Yes, but the last 12 more than months a year ago No, never Voluntarily participated in school-based activities outside of regular lessons Actively participated in a debate club or civics club Voted for representatives for student government or other student organizations Took part in decision-making about how the school is run (for example, through a meeting with a school administration or Ministry of Education (Educational Resource Center) representative) Took part in discussions at a student organization (for example a meeting to plan an event, organize a trip, or address a problem) Became a candidate for class representatives or other student organization Q12 Please think about the period since you started school this September. When discussing political and social issues during any school classes, how often do the following things happen? (Please mark only one box in each row.) Never Rarely Sometimes Often Students are able to openly disagree with their teachers Teachers encourage students to make up their own minds Teachers encourage students to express their opinions Students bring up current political events for discussion in class Students express opinions in class even when their opinions are different from most of the other students Q13 How much do you agree or disagree with the following? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Student participation in how schools operate can make schools better Lots of positive changes can happen in schools when students work together Organizing groups of students to express their opinions could help solve problems in schools All schools should have student self-government committees Students should mobilize themselves in groups to resolve critical issues for the school 6 Q14 Please read the statements below and tell us how much you agree or disagree with each of the following statements: (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Everyone should have the right to express their opinions freely Political leaders should be allowed to give government jobs to their family members No single company or government should be allowed to own the majority of media outlets in a country (for example, the majority of television stations, newspapers, radio stations) People should be free to criticize the government publicly without being afraid of punishment Q15 How important are the following behaviors for being a responsible citizen? (Please mark only one box in each row.) Very Quite Not very Not at all important important important important Voting in every national election Joining a political party Learning about Georgia's history Following political issues in the news Discussing politics Participating in peaceful political protests Participating in activities to benefit people in your community Working hard Obeying the law Meeting with decision makers to discuss critical issues for people living in your settlement Meeting with decision makers to discuss critical issues for Georgia Taking care of the environment Following traditions Respecting the rights of others Q16 How much do you agree or disagree with the following statements? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree I know more about politics in Georgia than most people my age When political issues are being discussed, I usually have something to say I am able to understand most political issues easily I have political opinions worth listening to 7 Q17 Please read the statements below and tell us how much you agree or disagree with each of the following statements: (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Men and women should have the same rights in every way Men and women should get equal pay when they are doing the same job Men are more qualified than women to be political leaders Women's only priority should be raising children and caring for her family Q18 Please read the statements below and tell us how much you agree or disagree with each of the following statements: (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Members of all ethnic groups should have an equal chance to get a good education in Georgia Members of all ethnic groups should have an equal chance to get good jobs in Georgia Schools should teach students to respect members of all ethnic groups Q19 There are many ways Georgian citizens may protest against things they believe are wrong. If you feel strongly that a government decision is wrong, how willing would you be to take part in any of the following forms of protest? (Please mark only one box in each row.) I would I would I would I would certainly do probably probably not certainly not this do this do this do this Writing a letter to a newspaper, television station, or other media organization Contacting the member of parliament representing my district Taking part in a peaceful march or rally Organizing a petition (this means collecting signatures from individuals who agree with you about the problem) Blocking traffic 8 Q20 Listed below are various ways adult citizens actively participate in civic life. When you are an adult, what do you think you will do? (Please mark only one box in each row.) I would I would I would I would certainly do probably probably not certainly not this do this do this do this Vote in parliamentary elections Vote in local elections (local council officials' elections and elections of mayors for self-governing cities) Get information about candidates before voting in an election Join a political party Join an NGO working on national or local issues Run for elected office Q21 Listed below are various activities that you, as a young person, could take part in during the next few years. What do you expect you will do? (Please mark only one box in each row.) I would I would I would I would certainly probably probably not certainly not do this do this do this do this Volunteer time to help people in need in my community Talk to others about my views on civic and social issues Write to a newspaper, television station, or other media outlet about political or social issues Contribute to an online discussion forum or social media about social and civic issues Join an organization for a civic or social cause 9 Q22 If you notice a problem in your community (people around you) that you would like to help resolve (for example, absence of speed bumps or traffic lights in the street in front of the school; absence of a playground for children or no park in the village/borough; littered environment; unavailability of internet), how well do you think you can do each of the following? (Please mark only one box in each row.) Very well Well Poorly Not at all Create a plan to address the problem Get other people to care about the problem Organize and run a meeting Express your views in front of a group of people Identify individuals or groups of individuals who could help solve the problem Write a letter to a local newspaper, television station, or other media outlet Call someone on the phone you never met to ask for their help with the problem Contact an elected official about the problem Organize a petition (this means collecting signatures from individuals who agree with you about the problem) Q23 Thinking of problems in your community (people around you), how likely is it that you would do each of the following after you graduate from school? (Please mark only one box in each row.) Not at all Somewhat Extremely likely Unlikely Maybe likely likely Contact or visit someone in government who represents your community to solve a problem Contact a newspaper, radio, or television station about a problem Sign an email, Facebook, or written petition to resolve a problem Q24 How much do you think each of the following describes you? (Please mark only one box in each row.) Not very Not at all much Somewhat Very much I listen to people's views on politics even when I disagree with them When I hear news about politics, I try to figure out what the truth actually is 10 Q25 Please indicate how much you agree or disagree with each statement. (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Don't know Most elected officials consider opinions of the citizens they represent In general, elected officials devote a lot of their time to make the community a better place In general, elected officials are concerned with serving their fellow citizens Q26 When you think about life after high school how likely or unlikely is it that you would do each of the following? (Please mark only one box in each row.) Not at all Somewhat Extremely likely Unlikely likely likely Volunteer to help people in need Become involved in issues like health or safety that affect your community Work with a group to resolve a problem in the community where you live THE QUESTIONS BELOW ASK ABOUT YOUR EXPERIENCES IN THE LAST 2 YEARS. Q27 As part of a civics class, have you worked on a civics or volunteer project? Yes............................................................................................................................................................ No ............................................................................................................................................................. IF YOU RESPONDED "YES" TO Q27, PLEASE RESPOND TO Q28. IF YOU RESPONDED "NO" TO Q27, PLEASE PROCEED TO QUESTION Q29. Q28 Please respond to the following questions about your project. (Please mark only one box in each row.) Yes No Did you talk about your experience with other students in the class? Did you apply information learned in civics class to your project? Did you learn about possible causes of problems that your civics or volunteer project was about? Did you learn about possible solutions to problems that your civics or volunteer project was about? Did you discuss what the government or other relevant parties (organizations, groups, businesses) could do to resolve the problem? 11 Q29 How much do you agree or disagree with each of these statements? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree I can make a difference in my community By working with people living around me I make things better in the settlement People should assist those in need It is important for people to follow laws I am willing to help others without being paid Some people in Georgia live in poor living conditions I am concerned about the poor living conditions of some people in Georgia It hurts me to hear about people being treated unjustly Q30 Please indicate how much you agree or disagree with the statement below: At least one of my close family members is active in the community (for example they participate in clubs or organizations, volunteer, or are members of local NGOs). Strongly agree .......................................................................................................................................... Agree ........................................................................................................................................................ Disagree ................................................................................................................................................... Strongly disagree ..................................................................................................................................... Q31 Here are some questions about your discussions with others. Please indicate how much you agree or disagree with each statement. (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree I talk with my family members about community issues My family members want to hear my opinions about community issues, even if their views are different from mine I talk to my friends about community issues My friends want to hear my opinions about community issues, even if their views are different from mine 12 Q32 How much do you agree or disagree with the following statements? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Everyone at my school tries to keep the school clean Students feel they are an important part of my school Students feel proud to be part of this school I have opportunities to work in groups on projects with students who are different from me (different race, nationality, economic status, social status, religion, etc.) We talk about ethnic discrimination, gender-based discrimination, or other forms of discrimination in our classes or other school activities Q33 Please indicate how much you agree or disagree with each statement about your school experience. (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree I learn about people or groups who work to make society better in Georgia I learn about people or groups who work to make society better abroad I learn about things in our society that need to be changed I learn about problems in our society I learn about the causes of problems in our society We discuss current events in my region We discuss current events in Georgia We discuss current events abroad Q34 How much do you agree or disagree with the following statements about governments and their power? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree The most acceptable form of government is when power is distributed among the three branches of government (Executive, Judicial, and Legislative) Dictatorships (leadership by one person with total power) are justified when they bring order and safety Dictatorships (leadership by one person with total power) are justified when they bring economic benefits 13 Q35 How much do you agree or disagree with the following statements about the civil service (საჯარო მოხელეები) and government? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Paying an additional amount to a civil servant, in addition to the official fee, to gain a personal benefit is acceptable It is acceptable for a civil servant to help his/her friends by giving them employment in his/her office The government should tell people how much they spend on what The government must answer to society Q36 How much would you like or dislike having neighbors belonging to the following groups? (Please mark only one box in each row.) I would I wouldn't I would like it care dislike it People of a different ethnic group than yours People of a different economic situation than yours People of a different religion than yours People who come from another region of the country People with physical disabilities People with mental disorders Q37 At your school during civics classes, how much have the following issues been discussed? (Please mark only one box in each row.) Not at all Little Sometimes Often Rights and duties of a citizen Consequences of consuming illegal drugs Equally protecting the rights of all people in the community regardless of gender, ethnicity, social and economic background, and abilities The role of non-governmental organizations (NGOs) operating in a democratic country Respect for people of different religions Facilitating education and employment of the people with special needs 14 Q38 Which of the following items does your household have: Yes No A motor vehicle Television Personal computer or laptop Flat or house Washing machine Q39 What is your religion? (Please mark only one box.) No religion ................................................................................................................................................ Orthodox................................................................................................................................................... Armenian Apostolic................................................................................................................................... Roman Catholic ........................................................................................................................................ Islam ......................................................................................................................................................... Judaism .................................................................................................................................................... Other......................................................................................................................................................... Decline to answer..................................................................................................................................... Thank you for participating in this survey! 15 FIELD CONTROL FORM - [STUDENT LEVEL] [ASSESSMENT TEAM COMPLETES ONE FORM FOR EACH STUDENT] [A field control form MUST be filled out by the assessment team for each student in the session.] SCHOOL NUMBER GRADE / SURVEY CODE SUPERVISOR NUMBER INTERVIEWER ID SCHOOL NAME DATA ENTERER ID / 16 VISIT 1: DATE OF VISIT GRADE SESSION START TIME END TIME DISPOSITION 1 VISIT 2: DATE OF VISIT GRADE SESSION START TIME END TIME DISPOSITION 2 / / : : / / : : FINAL DISPOSITION 17 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 ANNEX 2 SURVEY QUESTIONNAIRE IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 109 1 CIVIC EDUCATION ASSESSMENT STUDENT QUESTIONNAIRE SCHOOL ID GRADE / SURVEY NUMBER / 1 2 Dear Student, Your school has been selected to participate in a nation-wide survey that will ask you for your opinions about civics and citizenship in Georgia. This survey is conducted in coordination with the Georgian Ministry of Education and Science to assess and strengthen civic education in Georgia. It will be administered by survey monitors from the Caucasus Research Resource Centers (CRRC -Georgia) twice each school year for the next two years. However, agreeing to participate now does not mean that you must participate when asked to participate again. The survey will take up to one hour to complete. There are no correct or incorrect answers. You should provide the response that is right for you. Your teachers will not be able to see how you completed this questionnaire, and your grade does not depend on answering the questions. This survey is completely anonymous and confidential. This means that no one will know how you answered the questions, and that your responses to the questions will not be shared with anyone. You will not be asked to provide your name or student identification number. Once all of the surveys are completed, your responses will be put together with the responses of all students taking the survey and will be used to write a report that will help the Ministry of Education and Science improve civic education instruction in Georgia. We ask you to take this survey seriously and respond with honest answers so that the report truly represents what you have to say. Please address the monitors if you have any questions. If you agree to participate in this survey, please complete the questionnaire. CRRC Georgia monitors now will provide instructions. 2 3 PLEASE READ EACH QUESTION CAREFULLY AND ANSWER IT AS WELL AS YOU CAN. Q1 When were you born? (Please write month and year.) Month: Year: Q2 Are you a girl or a boy? Girl ............................................................................................................................................................ Boy ........................................................................................................................................................... Q3 Which of the following levels of education do you expect to complete? (Please mark only one box.) I do not expect to complete secondary school......................................................................................... Secondary school (complete grade 12) ................................................................................................... Technical / Vocational education .............................................................................................................. University degree (for example, a bachelor's degree) ............................................................................. Post-graduate education (for example, a master's degree or doctorate) ................................................ Questions 4 to 7 ask information about your family. If you do not want to answer these questions, please continue to question 8. Q4 What language do you speak at home most of the time? Georgian................................................................................................................................................... Armenian .................................................................................................................................................. Azerbaijani................................................................................................................................................ Russian .................................................................................................................................................... Other......................................................................................................................................................... 3 4 Q5 Q6 Q7 Please think of your mother (or, if your mother is not living with you, of a female guardian you live with – a grandmother, a stepmother, an aunt). What is the highest level of education she completed? (If you are not sure which box to choose, please raise your hand and ask the survey monitor for help.) No formal education ................................................................................................................................. Primary education .................................................................................................................................... Secondary education................................................................................................................................ Technical / Vocational education.............................................................................................................. Higher education - BA .............................................................................................................................. Higher education - MA or Specialist degree............................................................................................. Post-graduate degree .............................................................................................................................. I do not know ............................................................................................................................................ Now please think of your father (or, if your father is not living with you, of a male guardian you live with – a grandfather, a stepfather, an uncle). What is the highest level of education he completed? (If you are not sure which box to choose, please raise your hand and ask the survey monitor for help.) No formal education ................................................................................................................................. Primary education .................................................................................................................................... Secondary education................................................................................................................................ Technical / Vocational education.............................................................................................................. Higher education - BA .............................................................................................................................. Higher education - MA or Specialist degree............................................................................................. Post-graduate degree .............................................................................................................................. I do not know ............................................................................................................................................ How often do your parents or guardians discuss community and social issues at home? (Please mark only one box in each row.) Never or At least once a At least once a Daily or Not hardly ever month week almost daily applicable Mother / Female Guardian Father / Male Guardian 4 5 Q8 On a normal school day, how much time do you spend doing each of the following activities outside of school? (Please mark only one box in each row.) Less than About 30-60 About 1-2 More than 2 No time 30 minutes minutes hours hours Watching television Homework or study for school Reading for enjoyment Chatting with friends over the phone Spending time with friends Attending classes with tutor(s) Participating in sports or attending dance/arts classes Q9 How often are you involved in each of the following activities outside of school? (Please mark only one box in each row.) Never or At least once a At least once a Daily or hardly ever month week almost daily Talking with your family members about community or social issues Using television, newspapers, or the internet to inform yourself about the news Talking with friends about community and social issues Q10 Have you ever been involved in activities of any of the following organizations, clubs, or groups, whether formal or informal? (Please mark only one box in each row.) Yes, within Yes, but the last 12 more than months a year ago No, never Environmental organization Human rights organization A voluntary group doing something to help the community A charity 5 6 Q11 Q12 Q13 At school, have you ever done any of the following activities? Please think about all schools you have been enrolled at since the fifth grade. (Please mark only one box in each row.) Yes, within Yes, but the last 12 more than months a year ago No, never Voluntarily participated in school-based activities outside of regular lessons Actively participated in a debate club or civics club Voted for representatives for student government or other student organizations Took part in decision-making about how the school is run (for example, through a meeting with a school administration or Ministry of Education (Educational Resource Center) representative) Took part in discussions at a student organization (for example a meeting to plan an event, organize a trip, or address a problem) Became a candidate for class representatives or other student organization Please think about the period since you started school this September. When discussing political and social issues during any school classes, how often do the following things happen? (Please mark only one box in each row.) Never Rarely Sometimes Often Students are able to openly disagree with their teachers Teachers encourage students to make up their own minds Teachers encourage students to express their opinions Students bring up current political events for discussion in class Students express opinions in class even when their opinions are different from most of the other students How much do you agree or disagree with the following? (Please mark only one box in each row.) Strongly agree Agree Disagree Strongly disagree Student participation in how schools operate can make schools better Lots of positive changes can happen in schools when students work together Organizing groups of students to express their opinions could help solve problems in schools All schools should have student self-government committees Students should mobilize themselves in groups to resolve critical issues for the school 6 7 Q14 Q15 Q16 Please read the statement below and tell us how much do you agree or disagree with each of the following statements? (Please mark only one box in each row.) Strongly agree Agree Disagree Strongly disagree Everyone should have the right to express their opinions freely Political leaders should be allowed to give government jobs to their family members No single company or government should be allowed to own the majority of media outlets in a country (for example, the majority of television stations, newspapers, radio stations) People should be free to criticize the government publicly without being afraid of punishment How important are the following behaviors for being a responsible citizen? (Please mark only one box in each row.) Very important Quite important Not very important Not at all important Voting in every national election Joining a political party Learning about Georgia's history Following political issues in the news Discussing politics Participating in peaceful political protests Participating in activities to benefit people in your community Working hard Obeying the law Meeting with decision makers to discuss critical issues for people living in your settlement Meeting with decision makers to discuss critical issues for Georgia Taking care of the environment Following traditions Respecting the rights of others How much do you agree or disagree with the following statements? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree I know more about politics in Georgia than most people my age When political issues are being discussed, I usually have something to say I am able to understand most political issues easily I have political opinions worth listening to 7 8 Q17 Please read the statement below and tell us how much do you agree or disagree with each of the following statements? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Men and women should have the same rights in every way Men and women should get equal pay when they are doing the same job Men are more qualified than women to be political leaders Women's only priority should be raising children and caring for her family Q18 Please read the statement below and tell us how much do you agree or disagree with each of the following statements? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Members of all ethnic groups should have an equal chance to get a good education in Georgia Members of all ethnic groups should have an equal chance to get good jobs in Georgia Schools should teach students to respect members of all ethnic groups Q19 There are many ways Georgian citizens may protest against things they believe are wrong. If you feel strongly that a government decision is wrong, how willing would you be to take part in any of the following forms of protest? (Please mark only one box in each row.) I would I would I would I would certainly do probably probably not certainly not this do this do this do this Writing a letter to a newspaper, television station, or other media organization Contacting the member of parliament representing my district Taking part in a peaceful march or rally Organize a petition (this means collecting signatures from individuals who agree with you about the problem) Blocking traffic 8 9 Q20 Listed below are various ways adult citizens actively participate in civic life. When you are an adult, what do you think you will do? (Please mark only one box in each row.) I would I would I would I would certainly do probably probably not certainly not this do this do this do this Vote in parliamentary elections Vote in local elections (local council officials' elections and elections of mayors for self-governing cities) Get information about candidates before voting in an election Join a political party Join an NGO working on national or local issues Run for elected office Q21 Listed below are various activities that you, as a young person, could take during the next few years. What do you expect you will do? (Please mark only one box in each row.) I would I would I would I would certainly probably probably not certainly not do this do this do this do this Volunteer time to help people in need in my community Talk to others about my views on civic and social issues Write to a newspaper, television station, or other media outlet about political or social issues Contribute to an online discussion forum or social media about social and civic issues Join an organization for a civic or social cause 9 10 Q22 If you notice a problem in your community (people around you) that you would like to help resolve (for example, absence of speed bumps or traffic lights in the street in front of the school; absence of a playground for children or no park in the village/borough; littered environment; unavailability of internet), how well do you think you can do each of the following? (Please mark only one box in each row.) Create a plan to address the problem Very well Well Poorly Not at all Get other people to care about the problem Organize and run a meeting Express your views in front of a group of people Identify individuals or groups of individuals who could help solve the problem Write a letter to a local newspaper, television station, or other media outlet Call someone on the phone you never met to ask for their help with the problem Contact an elected official about the problem Organize a petition (this means collecting signatures from individuals who agree with you about the problem) Q23 Thinking of problems in your community (people around you), how likely is it that you would do each of the following after you graduate from school? (Please mark only one box in each row.) Not at all Somewhat Extremely likely Unlikely Maybe likely likely Contact or visit someone in government who represents your community to solve a problem Contact a newspaper, radio, or television station about a problem Sign an email, Facebook, or written petition to resolve a problem Q24 How much do you think each of the following describes you? (Please mark only one box in each row.) Not very Not at all much Somewhat Very much I listen to people's views on politics even when I disagree with them When I hear news about politics, I try to figure out what the truth actually is 10 11 Q25 Please indicate how much you agree or disagree with each statement. (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Don't know Most elected officials consider opinions of the citizens they represent In general, elected officials devote a lot of their time to make the community a better place In general, elected officials are concerned with serving their fellow citizens Q26 When you think about life after high school how likely or unlikely is it that you would do each of the following? (Please mark only one box in each row.) Not at all Somewhat Extremely likely Unlikely likely likely Volunteer to help people in need Become involved in issues like health or safety that affect your community Work with a group to resolve a problem in the community where you live THE QUESTIONS BELOW ASK ABOUT YOUR EXPERIENCES IN THE LAST 2 YEARS. Q27 As part of a civics class, have you worked on a civics or volunteer project? Yes............................................................................................................................................................ No............................................................................................................................................................. IF YOU RESPONDED "YES" TO Q27, PLEASE RESPOND TO Q28. IF YOU RESPONDED "NO" TO Q27, PLEASE PROCEED TO QUESTION Q29. Q28 Please respond to the following questions about your project. (Please mark only one box in each row.) Yes No Did you talk about your experience with other students in the class? Did you apply information learned in civics class to your project? Did you learn about possible causes of problems that your civics or volunteer project was about? Did you learn about possible solutions to problems that your civics or volunteer project was about? Did you discuss what the government or other relevant parties (organizations, groups, businesses) could do to resolve the problem? 11 12 Q29 Q30 Q31 How much do you agree or disagree with each of these statements? (Please mark only one box in each row.) Strongly agree Agree Disagree Strongly disagree I can make a difference in my community By working with people living around me I make things better in the settlement People should assist those in need It is important for people to follow laws I am willing to help others without being paid Some people in Georgia live in poor living conditions I am concerned about the poor living conditions of some people in Georgia It hurts me to hear about people being treated unjustly Please indicate how much you agree or disagree with the statement below: At least one of my close family members is active in the community (for example they participate in clubs or organizations, volunteer, or are members of local NGOs). Strongly agree .......................................................................................................................................... Agree........................................................................................................................................................ Disagree ................................................................................................................................................... Strongly disagree ..................................................................................................................................... Here are some questions about your discussions with others. Please indicate how much you agree or disagree with each statement. (Please mark only one box in each row.) Strongly agree Agree Disagree Strongly disagree I talk with my family members about community issues My family members want to hear my opinions about community issues, even if their views are different from mine I talk to my friends about community issues My friends want to hear my opinions about community issues, even if their views are different from mine 12 13 Q32 Q33 Q34 How much do you agree or disagree with the following statements? (Please mark only one box in each row.) Strongly agree Agree Disagree Strongly disagree Everyone at my school tries to keep the school clean Students feel they are an important part of my school Students feel proud to be part of this school I have opportunities to work in groups on projects with students who are different from me (different race, nationality, economic status, social status, religion, etc.) We talk about ethnic discrimination, gender-based discrimination, or other forms of discrimination in our classes or other school activities Please indicate how much you agree or disagree with each statement about your school experience. (Please mark only one box in each row.) Strongly agree Agree Disagree Strongly disagree I learn about people or groups who work to make society better in Georgia I learn about people or groups who work to make society better abroad I learn about things in our society that need to be changed I learn about problems in our society I learn about the causes of problems in our society We discuss current events in my region We discuss current events in Georgia We discuss current events abroad How much do you agree or disagree with the following statements about governments and their power? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree The most acceptable form of government is when power is distributed among the three branches of government (Executive, Judicial, and Legislative) Dictatorships (leadership by one person with total power) are justified when they bring order and safety Dictatorships (leadership by one person with total power) are justified when they bring economic benefits 13 14 Q35 Q36 Q37 Q38 How much do you agree or disagree with the following statements about the civil service (საჯარო მოხელეები) and government? (Please mark only one box in each row.) Strongly Strongly agree Agree Disagree disagree Paying an additional amount to a civil servant, in addition to the official fee, to gain a personal benefit is acceptable It is acceptable for a civil servant to help his/her friends by giving them employment in his/her office The government should tell people how much they spend on what How much would you like or dislike having neighbors belonging to the following groups? (Please mark only one box in each row.) I would I wouldn't I would like it care dislike it People of a different ethnic group than yours People of a different economic situation than yours People of a different religion than yours People who come from another region of the country People with physical disabilities People with mental disorders At your school during civics classes, how much have the following issues been discussed? (Please mark only one box in each row.) Not at all Little Sometimes Often Rights and duties of a citizen Consequences of consuming illegal drugs Equally protecting the rights of all people in the community regardless of gender, ethnicity, social and economic background, and abilities The role of non-governmental organizations (NGOs) operating in a democratic country Respect for people of different religions Facilitating education and employment of the people with special needs Which of the following items does your household have: Yes No A motor vehicle Television Personal computer or laptop Flat or house Washing machine 14 15 Q39 What is your religion? (Please mark only one box.) No religion ................................................................................................................................................ Orthodox................................................................................................................................................... Armenian Apostolic................................................................................................................................... Roman Catholic........................................................................................................................................ Islam......................................................................................................................................................... Judaism .................................................................................................................................................... Other......................................................................................................................................................... Decline to answer..................................................................................................................................... Thank you for participating in this survey! 15 16 FIELD CONTROL FORM - [STUDENT LEVEL] [ASSESSMENT TEAM COMPLETES ONE FORM FOR EACH STUDENT] [A field control form MUST be filled out by the assessment team for each student in the session.] SCHOOL NUMBER GRADE / SURVEY CODE SUPERVISOR NUMBER INTERVIEWER ID / DATA ENTERER ID SCHOOL NAME 16 __________________________________________________________ 17 VISIT 1: DATE OF VISIT / / GRADE SESSION START TIME : END TIME : DISPOSITION 1 VISIT 2: DATE OF VISIT GRADE SESSION / / START TIME : END TIME : DISPOSITION 2 FINAL DISPOSITION 17 Disposition Codes: 01 – # of Completes 02 – # Broke off - will continue later 03 – # Broke off - will NOT continue later 04 – # Refusal 05 – # Absent/Ill/Limited cognitive abilities 18 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 ANNEX 3 DESCRIPTIVE STATISTICS FOR OUTCOME MEASURES IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 128 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table A3.1. Descriptive Statistics for Outcome Measures Variable Obs Mean Std. Dev. Min. Max. Uptake Participation in civics projects 17,296 0.66 0.47 0 1 Outcome Attitudes related to school government and politics Civic behaviors and teachings in the classroom Engagement in school government and politics 17,316 17,350 17,312 0.77 0.68 0.32 0.17 0.16 0.29 0 0 0 1 1 1 Impact - Attitudes Pro-social and democratic attitudes 17,349 0.71 0.13 0 1 Attitudes about corruption and dictatorship Internal political efficacy and beliefs about citizenship External political efficacy 17,294 17,348 16,777 0.47 0.69 0.59 0.22 0.13 0.25 0 0 0 1 1 1 Impact - Behavior Current civic engagement and practices 17,350 0.42 0.19 0 1 Future civic engagement and political participation 17,345 0.61 0.18 0 1 IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 129 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 ANNEX 4 BALANCE TESTS IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 130 DRG-LER – GS-10F-0033M/AID-OAA-M-13-00013 Table A2.1. Multinomial Logistic Regression of Variation Assignment Independent Variables V1 Original Data V2 V3 Multi V1 ply Imputed V2 Data V3 -0.02 Female -0.02 0.05 -0.01 0.01 0.05 (0.07) (0.06) (0.06) (0.06) (0.06) (0.05) -0.07 -0.01 -0.05 -0.05 -0.04 -0.07 Age (0.08) (0.09) (0.11) (0.08) (0.08) (0.10) 0.41 0.04 -0.50 0.40 0.02 -0.36 Not Georgian (0.42) (0.43) (0.49) (0.36) (0.38) (0.39) -0.06 Not Orthodox 0.01 0.04 0.00 0.03 -0.04 (0.27) (0.23) (0.30) (0.21) (0.19) (0.23) 0.03 Mother's education 0.10** 0.07 0.02 0.09** 0.07 (0.05) (0.04) (0.05) (0.05) (0.04) (0.05) -0.02 Father's education 0.06 0.03 -0.01 0.06 0.03 (0.04) (0.05) (0.05) (0.04) (0.05) (0.05) 0.15 Household has a motor vehicle 0.27*** 0.19** 0.10 0.26*** 0.13* (0.09) (0.08) (0.08) (0.09) (0.08) (0.08) -0.02 Household has a television -0.23 -0.26 -0.15 -0.26 -0.34 (0.24) (0.22) (0.25) (0.20) (0.20) (0.22) Household has a personal 0.26 0.05 0.17 0.24 0.05 0.15 computer or laptop (0.17) (0.14) (0.15) (0.16) (0.13) (0.14) 0.15 Household has a flat or house 0.37** 0.25 0.26* 0.39*** 0.31** (0.18) (0.18) (0.18) (0.15) (0.15) (0.15) Household has a washing 0.08 0.15 0.05 -0.02 0.15 0.00 machine (0.14) (0.13) (0.14) (0.13) (0.13) (0.13) 0.17* 0.04 0.03 0.14 0.07 0.06 Tenth grade (0.11) (0.11) (0.12) (0.10) (0.10) (0.12) 0.00 Number of students 0.00 -0.00 0.00 0.00 -0.00 (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 0.21 Constant -1.12 0.15 0.24 -0.51 0.70 (1.34) (1.42) (1.70) (1.20) (1.20) (1.55) Joint orthogonality test statistic 58.30** 1.38* Observations 12,790 17,364 Notes: Base category is control group. Standard errors clustered at the school level in parentheses; *** p<0.01, ** p<0.05, * p<0.1. IMPACT EVALUATION OF USAID/GEORGIA’S CIVIC EDUCATION INITIATIVE | 131 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523