August 2012 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by William Mishler, Steven Finkel, Leslie Schwindt-Bayer, and Mateusz Pucilowski through Social Impact. EVALUATION Performance Evaluation of the Democratic Indicators Monitoring Survey PERFORMANCE EVALUATION OF THE DEMOCRATIC INDICATORS MONITORING SURVEY August, 2012 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by William Mishler, Steven Finkel, Leslie Schwindt-Bayer, and Mateusz Pucilowski through Social Impact. PERFORMANCE EVALUATION OF THE DEMOCRATIC INDICATORS MONITORING SURVEY August 30, 2012 Contract No. AID-OAA-I-10-00003, AID-OAA-TO-11-00057 This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (http://dec.usaid.gov). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Boulevard Suite 1000 Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i ACRONYMS AB AmericasBarometer CA Cooperative Agreement CAM Central America and Mexico CARSI Central American Regional Security Initiative CNC Centro Nacional de Consultoría DIMS Democratic Indicators Monitoring Survey DNP Colombian National Department of Planning DOS Department of State DRG Center of Excellence on Democracy, Human Rights, and Governance EDGE-IE Evaluating Democracy and Governance Effectiveness – Impact Evaluation FORPRIDEH Federación de Organizaciones No Gubernamentales para el Desarrollo de Honduras ICP Instituto de Ciencias Politícas IE Impact Evaluation IRI International Republican Institute LAC Latin America and Caribbean Bureau of USAID LAC USAID’s Bureau for Latin America and the Caribbean (also USAID/LAC) LAPOP Latin American Public Opinion Project M&E Monitoring and Evaluation NDI National Democratic Institute NGO Non-Governmental Organization NGO Non-Governmental Organizations PDA Personal Digital Assistant PMP Performance Management (or Monitoring) Plans SI Social Impact SOUTHCOM United States Department of Defense Southern Command SOW Statement of Work UNDP United Nations Development Program USAID US Agency for International Development USAID/DCHA/DRG USAID’s Bureau of Democracy, Conflict and Humanitarian Assistance, Center of Excellence on Democracy, Human Rights, and Governance USAID/W US Agency for International Development in Washington USG United States Government WVS World Values Survey ii DIMS Evaluation TABLE OF CONTENTS ACRONYMS ........................................................................................................................................................................ i LIST OF FIGURES AND TABLES..................................................................................................................................iii EXECUTIVE SUMMARY ..................................................................................................................................................1 Background and Purpose .............................................................................................................................................1 Main Findings...................................................................................................................................................................1 Recommendations .........................................................................................................................................................3 I. INTRODUCTION .........................................................................................................................................................5 Background .....................................................................................................................................................................5 Evaluation Team Members ..........................................................................................................................................5 Intended Users and Uses of the Report ..................................................................................................................6 Scope and Methodology ...............................................................................................................................................6 Limitations .......................................................................................................................................................................8 II. EVALUATION TOPIC #1: QUALITY .....................................................................................................................9 Survey Methodology .....................................................................................................................................................9 Questionnaire Content ............................................................................................................................................. 11 III. EVALUATION TOPIC #2: ACCESS AND UTILIZATION ............................................................................ 13 Data Accessibility ........................................................................................................................................................ 14 Data Utilization ........................................................................................................................................................... 14 IV. EVALUATION TOPIC #3: CAPACITY BUILDING ........................................................................................ 17 Technical Competence .............................................................................................................................................. 18 Human Capital ............................................................................................................................................................. 19 Organizational Durability .......................................................................................................................................... 20 V. EVALUATION TOPIC #4: SATISFACTION ...................................................................................................... 21 USAID ........................................................................................................................................................................... 21 Academics ..................................................................................................................................................................... 22 VI. EVALUATION TOPIC #5: COST-EFFECTIVENESS ....................................................................................... 23 Costs .............................................................................................................................................................................. 23 Benefits .......................................................................................................................................................................... 26 VII. EVALUATION TOPIC #6: GENDER ................................................................................................................. 27 Gender Equality in LAPOP Governance and Administration .......................................................................... 27 Training and Development of Survey Research Staff ......................................................................................... 27 Design and Content of the Survey Questionnaires ........................................................................................... 27 Gender in DIMS Reports and Analyses ................................................................................................................. 28 iii VIII. RECOMMENDATIONS........................................................................................................................................ 29 1. Funding ................................................................................................................................................................. 29 2. Technical Quality of DIMS Surveys ............................................................................................................... 29 3. LAPOP Governance .......................................................................................................................................... 30 4. Capacity Building................................................................................................................................................ 30 5. Management of the DIMS Cooperative Agreement ................................................................................. 30 Figures and Tables ........................................................................................................................................................... 31 Appendix A. Scope of Work ........................................................................................................................................ 56 Appendix B. Persons Contacted .................................................................................................................................. 63 Appendix C. Data Collection Tools ........................................................................................................................... 68 Appendix D. Intensive Case Studies ........................................................................................................................... 97 Colombia ...................................................................................................................................................................... 97 Honduras .................................................................................................................................................................... 100 Appendix E. Confidential Expert Evaluations ......................................................................................................... 105 External Review 1 ..................................................................................................................................................... 105 External Review 2 ..................................................................................................................................................... 108 Appendix F. Survey Data ............................................................................................................................................. 111 Academic Survey ....................................................................................................................................................... 111 Mission Survey ........................................................................................................................................................... 128 Appendix G. Overview of Survey Sampling Methods .......................................................................................... 145 Appendix H. AmericasBarometer Studies By Year And Funding Source ........................................................ 148 Appendix I. DIMS Core Questionnaire .................................................................................................................. 149 Appendix J. LAPOP Policies On Subscriber And Repository Fees .................................................................. 176 LIST OF FIGURES AND TABLES Figure 1.1: Academic Survey ......................................................................................................................................... 31 Figure 1.2: Academic Survey by Regional Specialization ........................................................................................ 31 Figure 1.3: Academic Survey by Type of Institution: .............................................................................................. 32 Figure 1.4: Academic Survey by Location of Institution: ........................................................................................ 32 Figure 1.5: Academic Survey by Gender: ................................................................................................................... 33 Figure 1.6: Academic Survey by Familiarity with AB Data .................................................................................... 33 Figure 1.7: Academic Survey ......................................................................................................................................... 34 Figure 1.8: Scholarly Articles Citing AB Data ........................................................................................................... 34 Figure 1.9: Academic Survey ......................................................................................................................................... 35 iv DIMS Evaluation Figure 1.10: Academic Survey ...................................................................................................................................... 35 Figure 1.11: Academic Survey ...................................................................................................................................... 36 Figure 1.12: Academic Survey by Regional Specialization ...................................................................................... 36 Figure 1.13: Academic Survey by Type of Institution ............................................................................................. 37 Figure 1.14: Academic Survey by Location of Institution ...................................................................................... 37 Figure 1.15: Academic Survey by Gender ................................................................................................................. 38 Figure 1.16: Academic Survey by Familiarity with AB Data .................................................................................. 38 Figure 1.17: Academic Survey ...................................................................................................................................... 39 Figure 2.1: Mission Survey ............................................................................................................................................. 40 Figure 2.2: Mission Survey ............................................................................................................................................. 40 Figure 2.3: Mission Survey: ............................................................................................................................................ 41 Figure 2.4: Mission Survey ............................................................................................................................................. 41 Figure 2.5: Mission Survey ............................................................................................................................................. 42 Figure 2.6: Mission Survey ............................................................................................................................................. 42 Figure 2.7: Academic Survey ......................................................................................................................................... 43 Figure 2.8: Academic Survey ......................................................................................................................................... 43 Table 2.1: Mentions of AmericasBarometer in USAID Mission Documents .................................................... 44 Table 2.2: AB Media Citations in DIMS Countries ................................................................................................. 45 Table 2.3: Scholarly Publications on DIMS Countries ............................................................................................ 46 Figure 3.1: Mission Survey ............................................................................................................................................. 47 Figure 3.2: Academic Survey ......................................................................................................................................... 47 Figure 3.3: Mission Survey ............................................................................................................................................. 48 Table 3.1: Survey Firms by Location ........................................................................................................................... 48 Table 3.2: LAPOP Trained Students from Central and South America ............................................................. 49 Figure 4.1: Mission Survey: ............................................................................................................................................ 51 Figure 4.2: Mission Survey ............................................................................................................................................. 51 Figure 4.3: Mission Survey: ............................................................................................................................................ 52 Figure 4.4: Mission Survey ............................................................................................................................................. 52 Figure 4.5: Mission Survey ............................................................................................................................................. 53 Table 6.1: Partners by Gender ..................................................................................................................................... 53 Table 6.2: Core Gender-Related Questions ............................................................................................................ 54 Table 6.3: Number of Gender-Related Items in Country Questionnaires ....................................................... 54 Table 6.4: Mentions of Gender in LAPOP Reports ................................................................................................ 55 1 EXECUTIVE SUMMARY BACKGROUND AND PURPOSE The Democratic Indicators Monitoring Survey (DIMS) Cooperative Agreement (CA), signed between USAID and the Latin American Public Opinion Project (LAPOP), is a $14.76 million, eight-year agreement that includes several components. The largest component of the agreement is a series of nationally-representative public opinion surveys that have been conducted biennially since 2006 in 16 Central American, South American, and Caribbean countries (hereafter the “DIMS Countries”): Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Guatemala, Guyana, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, Paraguay, Peru and Uruguay. Other components of the agreement include survey oversamples in DIMS countries, a range of reporting and dissemination activities, and special studies, most notably including the Central American Regional Security Initiative (CARSI) impact evaluation (IE). The DIMS surveys are a part of the broader AmericasBarometer (AB) project conducted by LAPOP, a research institute at Vanderbilt University. In addition to DIMS funding for the 16 aforementioned countries, LAPOP receives funding from private sources to conduct DIMS-style surveys in 10 additional countries: Argentina, Brazil, Belize, Bolivia, Canada, Chile, Suriname, Trinidad and Tobago, the United States and Venezuela. USAID does not fund activities through DIMS in these countries. The DIMS public opinion surveys are designed to measure citizen perceptions of and experiences with several aspects of governance and democracy. Their purpose is to: (1) track longitudinal changes in democratization in each country individually as well as the region as a whole; (2) support USAID strategic planning at both Mission and Bureau levels; (3) allow USAID to make cross-national comparisons and develop common indicators of progress; and (4) assess the outcomes of USAID’s democracy programming. Social Impact (SI) has been contracted by USAID to conduct a rigorous performance evaluation of the public opinion survey component of the DIMS agreement. SI was asked to evaluate a) the technical quality of the DIMS surveys; b) their accessibility and utilization; c) their contribution to building local capacity; d) their cost-effectiveness; and e) their attention to gender issues and the inclusion of women; as well as f) the level of satisfaction with the surveys among diverse intended constituencies, including USAID, local governments, NGOs, the media, and academic researchers. The purpose of this evaluation is to assess the performance of DIMS and to advise USAID on ways to strengthen the design and implementation of a possible new iteration of the DIMS CA. MAIN FINDINGS Technical Quality DIMS has provided an abundance of high quality data to USAID/Washington (USAID/W), USAID country Missions, host governments, NGOs, media and researchers. Data are centered on public attitudes, democratic values and behavior, governance quality, the rule of law, individual rights, and other important social, economic, and political issues. DIMS methodology adheres to the highest professional standards.1 Its sampling frames are carefully constructed to produce surveys that are nationally representative, with margins of error that are 1 Appendix G provides an overview of some technical sampling terms and concepts used in this report. 2 DIMS Evaluation typically less than +/- 3%. DIMS fieldwork is closely monitored and consistently of high quality. LAPOP has also been a leader in the use of survey technology, implementing one of the first cross-national surveys using hand-held Personal Digital Assistants (PDAs) for data collection. The DIMS questionnaire is carefully constructed, with a core time-series of standard questions across countries combined with country-specific and special topic questions. Though well-constructed, the questionnaire has become increasingly lengthy, creating concerns about respondent fatigue. In addition, different constituencies have opposing opinions about the appropriate mix of core versus country￾specific questions and about the relative emphasis given to specific topics. There are also concerns, especially among academic researchers, about the process used to make decisions on questionnaire content. LAPOP has increased the number of interviews per municipality in 2012, which will facilitate more sophisticated statistical analyses and permit the analysis of regional and other contextual influences on specific attitudes, values, and behaviors, further enhancing the quality of DIMS data. Utilization and Accessibility DIMS is used by USAID both in the Latin America and Caribbean (LAC) Bureau and in individual country Missions. DIMS data has principally been used in program development, but it is also increasingly being used as an assessment tool. With some notable exceptions,2 DIMS data is not used frequently for systematic evaluations of USAID programs. USAID officials at all levels are highly satisfied with DIMS and strongly support its continued funding, with various improvements. There are several examples of the effective use of DIMS by host country governments, local media and NGOs, but the overall utilization record by local organizations is disappointing. Poor local utilization is due to inconsistent outreach by LAPOP and country Missions. Also, the size and density of the country reports and their biennial publication discourages more widespread local use. DIMS has been used extensively and with increasing frequency by researchers studying Latin America. Beginning with the 2012 survey, DIMS datasets will be available without charge or embargo to anyone who wants to use them. This should satisfy current concerns about data access and encourage more DIMS-based research by scholars outside of the LAPOP community. Printed country and regional reports are easily accessible but not sufficiently user-friendly. They are well written and analytically rigorous, but their size, density, infrequent publication, and methodological emphasis have discouraged more extensive use by host governments, local NGOs and the media. Capacity Building DIMS has increased the survey research and analytical capabilities of individuals and firms throughout Latin America. Affiliation with DIMS has helped firms attract new business, and LAPOP’s training of local team leaders in most countries has increased the number and quality of local survey research analysts. In countries with little existing survey capacity, DIMS capacity-building contributions have been substantially less effective. Several countries continue to lack in-country team leaders. Many of these countries lack survey research firms deemed capable by LAPOP of conducting the DIMS survey. 2 These exceptions include USAID’s CARSI IE (four countries), the Merida Initiative (Mexican border), local governance projects in Honduras and Nicaragua, and the DOD-funded 1207 stabilization project in Panama’s Darien region. 3 Satisfaction LAPOP and its local partners have been highly responsive to USAID/LAC and DIMS Missions, virtually all of whom praise LAPOP’s cooperation. The little dissatisfaction that Missions express is focused on dissemination issues and a desire for DIMS to include more country-specific questions. Academic users of DIMS are highly satisfied with the quality of the data but not as satisfied with the decisions LAPOP makes regarding questionnaire design, nor are they pleased with the limited transparency and inclusiveness of the DIMS survey design process. Academic users have limited means to communicate their concerns to LAPOP and to participate in the process of questionnaire design. Cost-effectiveness DIMS costs are broadly in line with those of other regional barometers and surveys in Latin America. While significant, DIMS costs cannot be significantly reduced without adverse effects on data quality. Moreover, small additional investments in DIMS dissemination would produce large increases in utilization and have positive net benefits for the cost-effectiveness equation. Gender LAPOP has extensive female participation with respect to top leadership and staff positions at Vanderbilt. There is less female involvement among LAPOP partners and contractors, over which LAPOP has less direct control. Gender has not been a principal focus of DIMS research, and DIMS questionnaires do not include many questions about women. This is a point of considerable dissatisfaction among several DIMS constituencies. RECOMMENDATIONS A. Funding 1. It is highly recommended that USAID continues and, if possible, expands its financial support of DIMS. The surveys play a central role in strategic planning and assessment of DRG activities in a large number of USAID Missions throughout Central and South America and the Caribbean. 2. LAPOP and country Missions should devote more time, effort and money to the dissemination of DIMS data, especially to local media and host country governments. For LAPOP, this means producing more short, readable, country-level analyses on topics of contemporary interest. For USAID, this means investing in public relations experts who can better and more regularly communicate and develop relationships with the media and government officials. 3. LAPOP should work with USAID/LAC, country Missions, and other governments and donors to broaden and diversify DIMS funding sources. B. Technical Quality of Dims Surveys 4. LAPOP should rethink its use of quota sampling at the household level and reconsider the cost￾effectiveness of converting to full probability methods. 5. LAPOP should take greater advantage of innovative survey and survey-experimental methods to ensure that it maintains high-quality data, remains cost-effective, and retains its reputation as a state-of-the-art enterprise. 6. LAPOP should continue to increase the number of interviews per municipality to facilitate multi￾level analyses and allow DIMS to be used more effectively for program evaluation. Funding needs to be increased to allow this. 4 DIMS Evaluation 7. LAPOP should reduce the length of the core time-series questionnaire to provide more space for the inclusion of country-specific questions. It needs to re-focus the core and consider whether there are new topics (for example, questions on women and marginalized groups) that merit more inclusion over existing topics. 8. LAPOP should have a fully trained team leader living full-time in each DIMS country to maximize data quality and utilization. C. LAPOP Governance 9. The process by which decisions are made regarding questionnaire construction should be made more transparent and inclusive. This should be accomplished by creating a larger, more active and influential Advisory Board that is more representative of DIMS’ multiple constituencies. 10. LAPOP should be encouraged to build on its current leadership structure and make more firm plans for leadership succession. In particular, the role of the current Associate Director should be enhanced and publicized to DIMS stakeholders. A more active, effective, and representative Advisory Board should be formed to make LAPOP more transparent, collaborative and publicly accountable. D. Capacity Building 11. LAPOP should do more to build survey capacity in the four countries where outside survey research contractors are used. LAPOP should consider contracting with a local survey research contractor in each country, even if this means partnering with inexperienced contractors until they have the capacity to conduct independent surveys. Though additional costs are associated with this approach, the potential long-term benefits are well worth the expenditure, especially if USAID provides financial support for the activity. 12. LAPOP should increase its efforts to promote gender equality among team leaders and survey partners, and make the inclusion of questions on gender and other marginalized groups a higher priority. 13. LAPOP and the local Missions should broaden capacity building by sponsoring summer workshops in DIMS countries. This will create an important constituency of local students and young faculty with an awareness of DIMS data and the basic skills to use it in their classes and research. E. Management of the DIMS Cooperative Agreement 14. USAID/W should improve the central collection of data on DIMS activities and spending such that country costs can be meaningfully compared over time. Having an easily-accessible central repository of CA tasks and a detailed and consistent accounting of CA costs would greatly improve evaluations of DIMS performance. 5 I. INTRODUCTION USAID’s Center of Excellence on Democracy, Human Rights, and Governance in the Bureau for Democracy, Conflict and Humanitarian Assistance (USAID/DCHA/DRG) contracted Social Impact (SI) to conduct a performance evaluation of the DIMS Cooperative Agreement under the Enhancing Democracy and Governance Effectiveness – Impact Evaluation (EDGE-IE) mechanism. DIMS has been implemented since 2006 by the Latin American Public Opinion Project (LAPOP), a research institute at Vanderbilt University. The primary component of the DIMS CA, and the focus of this evaluation, is a series of nationally-representative public opinion surveys measuring citizen perceptions of and experiences with various aspects of governance, democracy, and citizen security across 16 Central American, South American, and Caribbean countries.3 BACKGROUND In 2003, USAID’s Bureau for Latin America and the Caribbean (LAC) adopted the Central America and Mexico (CAM) Strategy, designed “to promote a more democratic and prosperous Central America and Mexico and bolster citizen confidence in democratic governments.” In 2004, the region’s democracy officers decided to conduct a public opinion survey in the CAM countries tailored to USAID’s performance monitoring requirements. The surveys were designed to: (1) track changes in democratization in each country; (2) support USAID Missions’ Performance Monitoring Plans; (3) allow USAID to make cross-national comparisons and develop common indicators; and (4) assess the outcomes of USAID’s democracy programming. The DIMS CA, signed on March 29, 2006, substantially extended and expanded the work begun under CAM. The CA has been amended several times, expanding both the scale and scope of work to a current funding ceiling of $14.76 million. To date, $7.7 million has been committed for central DIMS surveys, with the balance of CA funding being committed to DIMS oversamples as well as a variety of DIMS and non-DIMS special studies, including CARSI. DIMS has funded four rounds of data collection in 16 countries: Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Guatemala, Guyana, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, Paraguay, Peru, and Uruguay.4 EVALUATION TEAM MEMBERS The SI evaluation team is comprised as follows: 1. William Mishler (Team Leader) is a Professor of Government and Public Policy at the University of Arizona, twice former Political Science Program Director at the National Science Foundation, and Associate Director of the New Democracy Barometer and the New Russia Barometer. He is also editor of the Journal of Politics. 3 DIMS is the name of USAID’s cooperative agreement (CA) with LAPOP that funds survey work in 16 countries. AmericasBarometer (AB) is the name of LAPOP’s survey research project covering 26 countries throughout the Western Hemisphere, including the 16 DIMS countries. Most people refer to DIMS as the “AmericasBarometer” or the “LAPOP [or Vanderbilt] survey.” Vanderbilt/LAPOP receives additional support from other donors to fund countries or modules that USAID does not fund. USAID also provides funds for other LAPOP activities under the DIMS CA, such as an impact evaluation of the Central American Regional Security Initiative (CARSI) project. In accordance with the evaluation SOW, this evaluation focuses solely on the 16 national surveys conducted under the DIMS CA. 4 Not all 16 countries were included in every survey round. As the CA expanded, new countries were added. In Colombia, DIMS data have been collected annually from 2006 thru 2012. See Appendix H for a table of DIMS Country surveys by year and funding source. 6 DIMS Evaluation 2. Steven Finkel (Team Member) is the Daniel Wallace Professor and Chair of the Department of Political Science at the University of Pittsburgh. 3. Leslie Schwindt-Bayer (Team Member) is an Associate Professor of Political Science at the University of Missouri and author of Political Power and Women’s Representation in Latin America. 4. Mateusz Pucilowski (Team Member) is a Program Manager at Social Impact, where he provides technical support to a global portfolio of evaluation contracts and routinely leads monitoring and evaluation (M&E) trainings for USAID and other United States Government (USG) personnel. 5. Cassilde Schwartz (Research Assistant) is a Doctoral Candidate in Political Science at the University of Pittsburgh, specializing in Latin American Government and Politics. INTENDED USERS AND USES OF THE REPORT This report is intended for use by the USAID/LAC and DCHA/DRG Bureaus, USAID Missions and US Embassies in DIMS countries, as well as the leadership of Vanderbilt University and LAPOP. The purpose of this evaluation is to provide recommendations for possible future agreements with similar purposes and activities. SCOPE AND METHODOLOGY The Evaluation Statement of Work (SOW) charges the evaluation team with providing assessments of the quality, impact, sustainability, and cost-effectiveness of DIMS, with particular attention to:  The quality of the DIMS survey design, implementation and management, the data sets produced, and the analyses of those data conducted by the DIMS cooperative partners.  The accessibility and utilization of the data and analyses by key audiences including USAID, host governments, civil society, media and academics, and the impact of the data and analyses on USAID strategies and programs.  The extent to which DIMS has contributed to local capacities for survey research and analysis, including the technical competence of local survey firms and the supply of human capital trained in survey research methods.  The satisfaction of USAID missions and other targeted audiences with DIMS products, services and policies.  The cost-effectiveness of DIMS compared to other USAID survey projects.  The extent to which DIMS takes gender into account in survey design and analysis, in the training and development of staff, and in the governance and administration of DIMS. The team utilized a mixed-methods evaluation approach to answer evaluation questions, triangulating findings and leveraging the strengths of a variety of qualitative and quantitative methods, including: Washington-Based Interviews Team members interviewed USAID/LAC staff members regarding their relationship with DIMS, their use of DIMS data, the responsiveness of LAPOP, their satisfaction with and perceptions of DIMS’ strengths and weaknesses, and their recommendations for improvements. The team also conducted interviews with individuals working on Latin American governance issues at the State Department, the Department of Defense, the World Bank, and the National Endowment for Democracy. LAPOP Site Visit Team members visited LAPOP at Vanderbilt University. The team conducted interviews with LAPOP directors and key technical staff. The team also collected reports, technical documents, financial data, 7 and information on the current administrative and governance structure of the AB and on plans to sustain and strengthen the enterprise. Phone Interviews with LAPOP Team Leaders and Survey Firms At least two members of the evaluation team conducted phone interviews with at least one LAPOP team leader and/or survey research firm representative for 12 of the 16 DIMS countries (13 interviews in total). These interviews lasted approximately one hour each and provided invaluable information on DIMS functioning and perceptions of the strengths and weaknesses of the project. Intensive Case Studies Two of the 16 DIMS country programs were selected for intensive case studies. The selection was based on recommendations from LAC officials and LAPOP leaders regarding how robust and institutionalized LAPOP operations were in the various DIMS countries.5 On this basis, Colombia was selected as one of the stronger programs and Honduras was selected as a relatively less effective program.6 These countries were visited by two-person teams who interviewed DRG Mission staff, US Embassy staff, host government officials, non-governmental organization (NGO) and civil society leaders, and academics. Local survey research firms partnering with DIMS were visited as well. A total of 26 people across 12 different institutions were interviewed in Colombia, and 12 people in 6 different institutions were interviewed in Honduras. Mission Survey USAID/DRG personnel in DIMS countries were asked to complete an internet survey soliciting the same categories of information collected in the site visits. Thirty-one of the maximum-possible 36 staff members completed the survey, representing a very high response rate of 86%. Academic Survey An internet survey gauging satisfaction with and utilization of AB7 information was administered to American and international scholars who self-identify as specialists in Latin American politics or in comparative politics.8 Of the 1966 individuals contacted, 370 responded, representing a 19% response rate.9 5 Defined as exhibiting strong support (financial and otherwise) from the USAID Mission, the existence of well￾established local partners, and the demonstrated utilization of DIMS data by Missions, NGOs, local academics, and government. 6 A third intensive case study planned for Ecuador, another relatively strong program, was cancelled because of Embassy/Mission concerns about local political sensitivities. Given time constraints, it was not possible to arrange a replacement. 7 Given the lack of broad-based understanding of differences between DIMS and AB in the public domain, external reviewers and academics were asked about the broader AB project to limit confusion. As DIMS surveys are a subset of the AB, this approach is methodologically appropriate. 8 Scholars were identified through two primary sources: A list of reviewers from The Journal of Politics(JOP) who self-identified their research specializations as : (1) “Latin American Politics and Caribbean Politics”, “Comparative Political Behavior”, “Comparative Political Institutions”, “Comparative Transitions toward Democracy”, and “Developing Countries”, and (2) a list of reviewers from the journal Latin American Politics and Society (LAPS). The JOP provides an extensive list of political scientists who might conceivably use DIMS survey data, In fact, a very large percentage of the scholars in Comparative Politics who are not Latin America specialists are likely to have little or no interest in DIMS data. Similarly, LAPS draws in scholars from other disciplines who are interested in various aspects of Latin America, but many of whom have no interest in public opinion or politics. The low response rate to this survey reflects the Team’s decision to cast a wide net to maximize the number and diversity 8 DIMS Evaluation External Expert Review Confidential assessments of AB quality were solicited from two external experts in cross-national public opinion research. See Appendix E for anonymized text of the external reviews. Document Review A variety of USAID Mission documents were analyzed to assess the utilization of DIMS information. These included DRG Assessments, Mission Performance Management Plans (PMPs) and Results Frameworks. Additionally, internet searches were used to identify other uses of AB data and analyses in the public domain. LIMITATIONS The SOW for this evaluation was very ambitions, particularly given the time allocated for its completion. Evaluating a variety of DIMS activities across 12 years and 16 countries is a substantial undertaking requiring parsimonious data collection and analysis. The evaluation was designed to solicit input from all major stakeholders, triangulate findings through diverse data collection methods, and blend quantitative and qualitative evidence to produce a balanced assessment of DIMS. Although internet surveys were used to solicit information from all DIMS Activity Managers and DG Officers in the 16 DIMS Missions, the team could only visit two countries for more intensive case studies. Similarly, the team lacked the time and resources to interview all potentially relevant NGOs, academics, government representatives, and media in DIMS countries, though every effort was made to interview samples of each. Aside from academics and USAID Mission staff, for whom relatively comprehensive sampling frames were constructed, the team could not collect systematic data from other stakeholders. Aside from internet searches, the most rigorous qualitative data collection on these other stakeholders was confined to the two intensive case studies. Regarding cost-effectiveness, the team was limited to a sample of DIMS costs provided by LAPOP. Due to the decentralized and complex nature of the CA, LAC was unable to provide comprehensive cost data on DIMS. The data shared with the team were incomplete and were reported in different formats with data coded differently for different countries and years. Additionally, comprehensive cost information from comparable large-scale surveys was unavailable. As detailed in Section VI, these factors severely limited systematic cost comparisons and analyses. A concern with all surveys is that those who choose to respond may have the strongest feelings either in support or in opposition to the subject matter in question. Many respondents either had a personal relationship to LAPOP (especially with LAPOP founder and director, Dr. Mitch Seligson), or held a financial stake in DIMS data collection firms, which may have introduced bias into the evidence collection. The evaluation team tried to mitigate these problems by using multiple methods and triangulating findings. While the evaluation team is confident in the findings, conclusions and recommendations presented in this report, it is not possible to assess all facets of an undertaking of DIMS’ size, scope and duration. The team utilized cost and time-effective methods to maximize evaluation rigor while sampling as broad a range of evidence as was feasible. of scholars who might have an interest in Latin American public opinion and the DIMS data, which also means the survey was sent to a very large number of scholars with no interest in Latin American politics. 9 Because the sample of academics is not a probability sample (individuals self-selected rather than being selected randomly), tests of statistical significance are inappropriate and are therefore not used. What matters in interpreting the data is simply the proportions of respondents answering a question in different ways. 9 II. EVALUATION TOPIC #1: QUALITY The DIMS surveys are conducted to very high scientific standards, producing rigorous, high-quality data, and addressing a series of critically relevant issues across Latin America and the Caribbean, including citizen attitudes, values, and engagement with democracy and the political process. Nevertheless, there are several ways in which DIMS methods and measures can be strengthened. SURVEY METHODOLOGY The technical quality of the DIMS surveys and the quality of the data and supporting analyses are strong, and LAPOP is committed to their continued improvement as documented at several points below. Great care is taken by LAPOP in designing the samples that appear to be highly representative of the overall country populations as judged by available census data.10 LAPOP’s detailed discussion of the surveys’ margins of error and how they are calculated shows an unusually high level of methodological sophistication in cross-national survey research.11 The team found no similar publicly available documents on either the Afrobarometer or Arab Barometer web sites, while the Latinobarometer website provides sampling errors for their national samples as if the samples were simple random samples, which they are not. LAPOP updated their sampling frames in 2012 to reflect post-2000 Census information, another very positive development. LAPOP and its local partners provide extensive training to interviewers before data collection begins in order to maximize response quality and ensure consistency of responses across countries.12 They are trained in the use of the PDAs to ask questions, and they conduct several practice interviews. 10 LAPOP uses multi-stage stratified cluster sampling, with units selected within strata until the block level on a proportional to population basis. The highest strata is sometimes the natural geographic or administrative divisions within the country (e.g., "Sierra, Costa, and Oriente" in Ecuador, the 9 major administrative units in Bolivia), and other times the size of the municipality (e.g. “fewer than 25,000, between 25 and 100,000, and more than 100,000” in Guatemala). Within each of these strata, units are then stratified further (by Urban/Rural population if the first stratum was geographic, and by geography if the first stratum was population). A fixed number of Census segments within each of these strata are then selected in proportion to population size. The Census segments are then divided into three or more "blocks", and one is selected at random for inclusion in the sample. The block represents the "cluster" of households which will be selected systematically for inclusion (e.g., every third household on the block is initially approached). A quota system, discussed at further length below, is then imposed at the final stage of individual selection within the household. For more information on survey methodology, see Appendix G. 11 As explained in Appendix G, the use of cluster methods introduces larger sampling errors than would be obtained under “simple random sampling” (or SRS). Deviations from the errors that a given sampling method produces compared to what would be obtained under SRS are called “design effects”. LAPOP, unlike the other regional barometers, routinely reports these errors. 12 Interviewers are trained in Spanish or Portuguese according to standardized procedures outlined in an extensive 28 page manual. They are also provided a short version of the interviewer manual to guide them in the field, and a separate document explaining the procedures for registering the outcomes of visits to households selected for inclusion in the sample using the Personal Digital Assistants (PDAs) that LAPOP has interviewers use for data collection. The training sessions are led by country partners and/or personnel from either LAPOP Central or trusted members of the LAPOP community. Trainers work from a detailed Training Manual that again works to ensure standardization across country contexts. 10 DIMS Evaluation Furthermore, supervisors randomly back check a sample of respondents to insure the interviewer actually showed up and conducted the interview submitted. LAPOP’s training and monitoring of the data collection process adheres to the highest professional standards. The decision to replace paper surveys with Personal Digital Assistants (PDAs) for data collection purposes, starting on a limited basis in 2006, almost certainly has reduced coding and data entry errors and improved the flow and overall quality of the interviews. Increasing the number of interviews in 2012 to at least 12 respondents per municipality was an important development that will facilitate rigorous analyses of municipal-level factors related to important concerns such as crime and poverty. A wide range of testimony was collected regarding the high scientific quality and rigor of DIMS. One survey research expert who provided an external review writes that “the project shows a strong appreciation of the technical issues surrounding comparative research in an economically underdeveloped region. The project has made a series of perfectly reasonable compromises [such as using quota samples at the household level] to ensure that the data quality is the best that can be achieved with the resources at hand.” Another expert reviewers adds, “Setting the bar higher (for all of us), LAPOP continues to pave the road to achieve the highest quality work.” Assessments by scholars studying Latin American politics or comparative politics are similarly positive. As Figures 1.1-1.6 show, the perceived “scientific quality of the AB data” was viewed as “high” (54%) or “extremely high” (33%) by almost 90% of the 369 respondents. Individuals who consider themselves to be specialists in Latin American politics or who are most familiar with the AB were even more positive, with nearly half of these individuals (45% and 48%, respectively) estimating the scientific quality of the data to be “extremely high”. Equally telling are responses to a question asking individuals to rate the relative usefulness of the AB data compared with “other sources of public opinion data in Latin America”; nearly three-quarters (72%) reported that the DIMS data were “more useful”, with only 3% of respondents reporting DIMS to be “less useful” than other sources (Figure 1.7). As a further indication of the quality of DIMS data, eleven articles using data collected from DIMS countries (13 using data more broadly from AB countries) have passed a rigorous double-blind peer￾review process and been published in the top journals in political science and sociology (Figure 1.8), with many others appearing in the top English and Spanish language journals devoted specifically to Latin American politics. Notwithstanding the overall high quality of the DIMS surveys, there are several areas where improvements could be made. One regards the use of quota methods at the final (household) stage of the sampling frame to identify the individual within the household who will be interviewed. Quotas are imposed at the last stage of the sampling process, where the interviewer systematically selects households within the Census-type blocks that have been included in the sample, and then conducts interviews with a single person per household such that a pre-determined number (based on the country's known population figures) of men and women of particular ages are eventually interviewed on the block. LAPOP claims that this procedure reduces interviewing costs without significant loss of sampling accuracy. Some survey partners even claim that the quota procedure is more accurate because it reduces the overrepresentation of women and the elderly who are more likely to be at home when interviewers are working. It is the case, however, that the quota method departs from current “best practices” in cross-national survey research. The latter mandates probability selection at all stages of the multi-stage process, normally using either the “Kish grid” or “next birthday” method (both discussed in Appendix G) for selecting the individual to interview within the household. Cost is an important concern, but the use of household probability samples should not greatly increase data collection costs: One of the principal survey companies employed by DIMS reports that it uses the “next birthday” method for other national surveys, with an estimated cost “surcharge” of approximately 5%, although it is unclear if this would also be the case in all DIMS countries 11 The total number of interviews conducted in DIMS surveys, 1500 in most countries, is also problematic for conducting meaningful sub-group analyses in countries with more heterogeneous populations, a point echoed by two LAPOP partners. The 2012 sampling plan increases the number of interviews per municipality and is a step in the right direction. LAPOP estimates that the number of interviews per municipality should be closer to 30 than the current 12 to facilitate meaningful regional and municipal level analyses. Larger numbers of interviews would have the additional benefit of providing enough cases to enable DIMS data to be used more effectively for program evaluation purposes. While the change to 12 per municipality has been achieved within the constraints of the current overall 1500 interview samples, moving to 30 interviews per municipality will likely require significantly larger overall samples and somewhat higher costs. QUESTIONNAIRE CONTENT The “core” DIMS time-series questions relating to government legitimacy, mass support for democracy and the political system, and individual participation in politics and civil society are essential aspects of democratization and are of keen interest to USAID and other stakeholders. The information provided by DIMS on these topics over time and across a wide range of Latin American and Caribbean countries is one of its great strengths. Nearly three-quarters (72%) of the academics surveyed agree that the DIMS surveys are “extremely relevant” to the study of public opinion and/or democratization processes in Latin America (Figure 1.9). Moreover, the “signature” topics of legitimacy, democratic attitudes, and political engagement are the subject of the vast majority of scholarly articles written using DIMS data. This scholarship is viewed by a slight majority of academics (54%) as having “greatly enhanced our understanding of Latin American public opinion and/or democratization” (Figure 1.10). In recent rounds, questions have been added to measure attitudes about local government, local service provision, crime victimization and security issues, and experiences with, and perceptions of, political corruption. In some cases, such as the questions on corruption experiences, LAPOP has been an international leader in questionnaire design. For example, LAPOP has moved away from questions asking people about their perceptions of crime and corruption and has asked them instead about their direct experience with crime and corruption. The two types of questions produce dramatically different results and add to the strength of the project. At the same time, the addition of new questions to the core without judicious pruning of existing questions raises a number of concerns.13 Foremost is the length of the questionnaire. This issue was raised by virtually all of the local partners, who variously described the questionnaire as too “heavy” and as leading to respondent fatigue and resultant “response set” or “acquiescence” biases. Quantitative evidence supplied by LAPOP show a mixed picture in this regard. On the one hand, the average (i.e., mean) interview time in 2012 across all DIMS countries is just under 50 minutes. However, average interview times are longer than 50 minutes in five of the DIMS countries, and, given that median interview times are longer than mean times, interviews of more than one hour are common. These interview times are at or near the point at which data quality is likely to decline. The problem of length is caused not only by the growing survey core but also by the increasing use of “buy-in” modules from new clients and other initiatives that are not always relevant to central DIMS concerns. For example, two partners raised questions about the value of the 2012 module on Chinese influence in the region, claiming that the series of questions neither translated well cross-nationally nor 13 The question of what to cut in a questionnaire when adding new content is highly controversial; all questions have their supporters and detractors. The evaluation team’s position, discussed elsewhere in the evaluation, is that these decisions should be made through an open and collaborative process and not by LAPOP leaders alone or peremptorily by the members of this evaluation team. 12 DIMS Evaluation resonated with respondents. While the broadening of funding sources is desirable and feasible, funding sources seeking to purchase marginally relevant one-time questions should be avoided. LAPOP responded to the problem of questionnaire length in 2012 by utilizing “split-samples” where some questions are asked only to one-half of the respondents (thus allowing more questions to be asked to different subsets of respondents). This is a reasonable strategy for reducing length but has other adverse consequences, since the number of individuals available for analysis drops by one-half of the total sample whenever these questions are considered. The number of interviews conducted in most countries already constrains analysis of many regional, ethnic or other sub-groups. Dividing the sample in half exacerbates the problem and undermines LAPOPs laudable efforts to increase sample sizes for this purpose. Another frequently expressed concern relates to the appropriate mix of “core” versus “local” items in the questionnaire. While most users agree that “core” DIMS questions tap essential elements of democratization, many local partners and USAID Missions want to devote a greater share of the DIMS questionnaire to country-specific issues. For example, one LAPOP team leader said that the limit on country-specific questions is a problem. Another mentioned that his/her country was particularly interested in justice issues but that these were not sufficiently covered in the survey. In Colombia, Mission staff pointed out that they were only permitted ten questions on the 2012 survey and would have preferred 15-20. Four Mission respondents also noted a desire for more attention to country￾specific issues or questions on DIMS. These responses indicate that local content in DIMS would better illuminate country-specific opinions and also lead to greater commitment by local stakeholders and increased levels of dissemination and utilization of the DIMS data and reports. The content and length of the DIMS questionnaire are clearly viewed more critically by project stakeholders and end-users than are other aspects of the survey methodology. The survey of academics asked whether the questionnaire contained “too little”, “too much”, or “just about the right amount” of emphasis on a series of topics. Figure 1.11 shows that one-quarter or more of the self-selected academics who responded said that the questionnaire has “too little” emphasis on issues relating to marginalized peoples, corruption, women issues, democratic commitments among political elites, and international relations. These proportions rise to over 40% in some sub-groups, notably women and non-academic respondents. Nearly two-thirds (62%) of women, for example, say that there is “too little” emphasis on issues related to minority inclusion (Figures 1.12 to 1.16). By contrast, 17 and 20 percent of academic respondents believe that “too much” emphasis is placed on mass legitimacy and political and social tolerance, respectively, two elements that constitute the historical “core” of DIMS (Figure 1.17). These figures are even larger for Latin American specialists and respondents outside of the United States. Similar views were expressed in the open-ended questions, where approximately one-third of respondents recommended that the survey update its questionnaire to give greater attention to more current topics, such as clientelism, security and elections. Some dissatisfaction with questionnaire content was expressed by Mission personnel, whose survey responses are reported separately from the academic survey. In the open-ended question asking Mission respondents about content areas they felt are under-emphasized, seven respondents mention at least one of the following issues: citizen security, crime victimization, the environment, youth and food security items, national elections, issues of inter-ethnic relations, and violence. Separately, local partners suggested the following issues as needing greater attention in the survey: political behavior, marginalization and inequality, states and state development, political institutions, and parties and political elites. They recommended less attention to the more traditional areas of mass political culture that currently dominate the core. In Colombia, the most popular suggestion made by interviewees was more attention to perceptions of political institutions and parties (see Appendix D). 13 All surveys face hard choices about which questions to include and which to exclude. The issues LAPOP confronts are no different and no worse than those faced by all large-scale survey research projects. What is different is the process by which these decisions are made. While various constituencies are invited to comment on the questionnaire and are allowed to suggest topics or questions for inclusion, real decision-making is vested solely in LAPOP Central at Vanderbilt University. LAPOP needs to address these issues in a less centralized and more transparent fashion. This top-down decision-making process was appropriate when LAPOP was a private enterprise. U.S. Government funding, however, creates an obligation for greater transparency and greater public collaboration. Other major surveys make effective use of advisory boards or other collaborative mechanisms to decide such fundamental matters as what to keep in the core and what to replace. As successful as the DIMS surveys have been, steps are needed to open the process and allow more innovation—for example, by facilitating pilot projects from partners or others in “off years,” 14 more aggressively soliciting the views of Latin American-based academics, expanding representation of various stakeholder constituencies in the LAPOP Advisory Board, and expanding the Advisory Board’s role in determining questionnaire content. III. EVALUATION TOPIC #2: ACCESS AND UTILIZATION DIMS data are made available in three primary formats. First, users can access the raw datasets for each country or access a merged, cross-national dataset in electronic files that work with most common statistical software programs. Second, users can do simple statistical manipulations with the survey questions using an online data analysis website maintained by LAPOP.15 Third, users can view results in printed region-wide and country-specific reports, also available online. Accessibility and utilization of the data vary across countries and users, as reported in interviews and the online surveys. Academics use the data primarily in its raw form, whereas USAID Missions, host governments, NGOs, and the media rely mostly on printed reports. When individuals from these latter groups want additional analyses, they usually ask LAPOP or the local team leader to produce them. The heavy dependence on the country reports is confirmed by the Mission survey. Of the 30 respondents, 26 “read reports prepared by Vanderbilt LAPOP or others which use AB data,” compared with 14 who “asked someone else at the Mission to do some analyses of AB data” and only four (4) who have “personally . . . created figures or tables from DIMS data” (Figure 2.1). 14 The Afro-Colombian focus of the 2011 off-year survey in Colombia is an excellent example. As discussed in the intensive case study in Appendix D, this study has generated widespread interest and enthusiasm among multiple stakeholders. Among other large scale national survey projects, the American National Election Survey (ANES), for example, has instituted on “On-Line Commons” where members of the academic community can suggest questions for inclusion on future surveys; the best proposals are sometimes then included in “off year” or pilot surveys and then evaluated for possible inclusion in the main data collection phase. 15 Website: http://lapop.ccp.ucr.ac.cr/Lapop_English.html. Although the instructions for manipulating the data are in English or Spanish, the survey questions themselves are only in the language of the host country. 14 DIMS Evaluation DATA ACCESSIBILITY The CA between USAID and LAPOP states that “the surveys, their methodology, and databases should be publicly accessible, within the US Government guidelines for research on human subjects.” LAPOP makes the DIMS data publicly available in multiple formats, without an embargo, and in strict compliance with guidelines on the protection of human subjects. The LAPOP website provides copies of all questionnaires,16 and a wealth of detail on sampling and other methodological concerns. In these regards, DIMS is a model for other publicly funded surveys. There is one important exception, however, which concerns existing restrictions on public access to the raw data generated by DIMS, although LAPOP is taking steps to correct this. Access Policies LAPOP’s policy since 2006 limits access to DIMS datasets to individuals and institutions that are LAPOP subscribers or repositories. An exception is granted to citizens of DIMS countries, who are allowed free access to their country’s dataset but not the datasets of other DIMS or AB countries. Subscribers must pay an annual fee to access the data; repositories become permanent owners of the datasets they purchase.17 Fifty-one institutions are subscribers, 27 are repositories and 10 are both. Seventeen of these institutions are Latin American academic institutions (5 in Mexico, 2 in Peru, and 1 in Colombia) and four are non-academic institutions. The charges for DIMS are modest, and the funds generated are used by LAPOP to help pay for AB surveys that are conducted in the non-DIMS countries (which are not funded by USAID). Still, the policy contrasts with other publicly funded regional surveys, such as the AfroBarometer, which provides free access to all of their data. LAPOP is changing its data access policy. Beginning with the release of the 2012 survey, all individual country datasets for all years will be available free to all users. LAPOP has also pledged that merged datasets for the 16 DIMS countries will be free to anyone willing to sign the standard consent form. Merged datasets that include all 26 AB countries, however, will only be available through “premium access” to subscribers and repositories. Data from before 2004 and from special populations not funded with public money will be available only to repositories. All of the AB data (both DIMS and non-DIMS) can be analyzed by any user via LAPOP’s online data analysis website. Websites of this sort are common and are useful for low-intensity users, but they provide limited analytical power and usually are insufficient for academic users. DATA UTILIZATION DIMS data have been used by USAID, other USG agencies, international donors, host governments, civil society, media and academics to varying degrees and in a variety of ways. The most extensive uses have been by USAID Missions for program development and by academics for research. USAID and USG USAID/LAC and USAID Missions make extensive use of DIMS. The survey of Mission staff shows that most respondents are familiar with DIMS, use it in their work, and support its continuation. Ninety percent of Mission respondents had “moderate,” “high,” or “extremely high” familiarity with DIMS (see Figure 2.2). Seventy percent use it in their day-to-day activities at least “several times a year,” and 10% use it “at least once a month” (see Figure 2.3). 16 The questionnaires are often only in the language in which they were asked in the field, usually Spanish. 17 For more information on the fee structures for subscribers and repositories, see Appendix J. 15 Mission staff say they use DIMS more often for program development than for program evaluation (Figures 2.4 and 2.5). Of the thirty Mission staff who responded to the survey, 87% report using it for program development at least “once or twice”. In contrast, 47% report using DIMS at least “once or twice” for evaluation purposes, while an identical percentage said they had “never” used DIMS for program evaluation. Two respondents reported that DIMS “was central to the DRG Assessment process,” and 54% said that “it played an important role but there are more important sources of information.” However, 39% felt it played only a limited role or no significant role at all in DRG Assessments (Figure 2.6). Twenty of the 30 Mission respondents described their uses of the data in their own words in response to an open-ended question as follows:  “The design of our political party program was influenced by the data collected through the AB survey which demonstrated the lack of credibility, accountability and democracy within the political party structures and its leaders.”  “The Municipal Governance Program. We are focusing our citizen participation efforts on local budget because we learned from the survey that the participation was very low.”  “Crime prevention program design efforts use the data as part of the overall description of the problems.”  “We did an elections program trying to change voters' attitudes toward democracy and used AB for baseline data.”  “The data was utilized in the evaluation of [a] former justice project. Evaluators found data relevant to compare overall perceptions and trust in justice during the last years of our support.”  “We tracked the corruption victimization index as a proxy to evaluate the Transparency Program that ended in 2009.” The greater use of DIMS for program development is also seen in the Mission documents analyzed for this evaluation (see Table 2.1). Of the 47 program-focused “mentions” of DIMS in the 29 documents provided, only 12 referred to the evaluation of specific programs, while 35 mentions served to justify new projects or provide political context. This may be because of the two-year timing of DIMS surveys and because DIMS samples are not large enough to evaluate the programs initiated by Missions in specific regions. According to one Performance Monitoring Plan, “While the timing of the survey is convenient to our needs, unfortunately we cannot use this survey to measure activity impacts in our target regions. The survey methodology makes the results meaningful at the national level, at strata level . . . but not at the individual department/regional level” (Peru PMP, 2010). DRG officers in LAC report extensive use of the DIMS data. The data was said to “identify burgeoning areas of attention at the regional and bilateral levels”, such as corruption, crime and security, it “helps drive programs,” and the release of DIMS results provides USAID a platform to insert DRG-related issues into inter-agency discussions with entities such as the Department of State and the National Security Council. Officers cited several examples where LAPOP data were used for monitoring and evaluation purposes, though these related more to CARSI and other special initiatives than to the core DIMS data. DRG officers also report using the data on a “daily” or “weekly” basis, asking LAPOP Central frequently for additional information and making frequent requests for LAPOP Central staff to conduct briefings for visiting officials or at venues such as the Foreign Service Institute. Other USG agencies also use DIMS. For example, the Department of Defense’s Southern Command (SOUTHCOM) is engaged in a “critical mapping exercise” that overlays aggregated DIMS data on citizen trust in security forces and the judiciary on top of objective indicators of crime and drug-related activity in Central America and the Northern Triangle. SOUTHCOM officials expressed a high degree of satisfaction with DIMS and hope to obtain more complete sub-national data from future surveys. 16 DIMS Evaluation A respondent from the Department of State (DOS) reported that DIMS data and reports are used exclusively as background information. While DIMS is used for various briefings and by the Foreign Service Institute to strengthen Foreign Service Officer training, the data is not used in the development of policy or programming because information is only available every two years. While the respondent praised LAPOP for improving the frequency of reporting through the creation of the Insights Series, a bi￾weekly publication written by LAPOP scholars and graduate students that analyzes specific topics using AmericasBarometer data,18 the informational needs of the Department are such that DOS has to conduct its own surveys. A similar sentiment was expressed in one of the country visits, where an embassy official reported that DIMS was one of several sources of background data used in preparing briefings for the Ambassador and cables for visiting dignitaries. International Donors DIMS has also been used by a variety of other international organizations. Information from DIMS surveys form part of the widely-utilized World Bank Worldwide Governance Indicators for Latin American and Caribbean countries. The Inter-American Development Bank includes responses from several DIMS questions relating to perceptions of insecurity and trust in its interactive web site on cross-national governance indicators (“DataGov”). Interviews with officers at the National Endowment for Democracy indicated that they (and their National Democratic Institute [NDI] and International Republican Institute [IRI] affiliates) make frequent use of DIMS, nearly always in the form of the country reports, for background on developments in countries where they are working. The United Nations Development Program (UNDP) has also used the data in preparation for its annual Human Development Reports. Host Country Governments Evidence regarding data usage by DIMS country governments is limited mostly to indirect reports from Mission staff and local team leaders, and to the Honduran and Colombian case studies. In Honduras, no specific use of DIMS by the government was observed, although Mission staff said that various government agencies are aware of the data. Colombian government agencies and political parties have used DIMS relatively more. Two Colombian parties report having used the survey results to inform their programming and policymaking toward the minority Afro-Colombian community. The National Department of Planning (DNP) also reports using the data in their programming and in efforts to educate government departments about public perceptions of government corruption, perceptions of presidential performance, and public participation in politics. Civil Society Information on the use of DIMS in civil society is limited primarily to team leader recollections of their use and to the Honduran and Colombian case studies. In Honduras, a partner NGO, FORPRIDEH (Federación de Organizaciones No Gubernamentales para el Desarrollo de Honduras), has been contracted by USAID/Honduras to disseminate DIMS results. It has held several regional meetings and shared the country report with local governments and other NGOs, but has not used the information for other purposes. In Colombia, usage by NGOs has been more common, but nearly all interviewees also mentioned that wider dissemination is needed (see Appendix D for more detail). For example, the Instituto de Ciencias Políticas (ICP), IRI, and NDI report using it for their programming on youth (ICP) and political parties (IRI and NDI). 18 Reports are 4-5 pages with a goal to provide short, relevant analysis of important topics in Latin America public opinion. The reports are disseminated through a LAPOP listserve, their website, and Twitter. 17 Media Media usage of DIMS varies across countries and time. According to an internet search conducted by the evaluation team, references to DIMS appeared in host country media reports 206 times between 2006 and 2012, with usage increasing over time (Table 2.2). Among DIMS country programs, the Dominican Republic has been the most successful in getting its data covered by the media, with 21% of the 206 media reports using Dominican DIMS data. Colombia was second, with its data appearing in 12% of the reports. The least-covered countries were Costa Rica and Haiti. Interviews in both Honduras and Colombia suggested that media usage of DIMS data is insufficient. In Honduras, Mission staff maintain that the media regularly uses DIMS data and graphics, though the analysis of media coverage in Table 2.2 found only 9 news stories in Honduras using DIMS since 2006. The survey research firm that conducts the survey in Honduras said that they intentionally do not communicate with the media out of concern that the company’s reputation will suffer if its data are compared in the media to the often unscientific and frequently biased political polls promoted by self￾interested political parties or government officials (see Appendix D). In Colombia, opinions also varied on how much media attention the survey receives, although the media search uncovered 25 stories in print since 2006. In hopes of attracting more media attention, the Colombia LAPOP organization provides press releases, a “graph of the week” on their website, and a report series. Academics Academic use of the DIMS data has been more extensive than among other stakeholders. According to data compiled by LAPOP, 356 peer-reviewed publications have used the AB data since 2006, with 306 of those publications based on at least one DIMS country (Table 2.3). Of these, 53% were authored by at least one person affiliated with LAPOP. Approximately two-thirds have been published or presented at conferences in the United States. In Colombia, the data have appeared in 15 academic publications (8 articles, 2 theses, 2 book chapters, and 3 books) since 2004. In contrast, there does not appear to be any academic usage of DIMS in Honduras, where local academic capacity is limited, as discussed at length in the case study (see Appendix D). The survey of academics shows reasonable familiarity with DIMS from scholars with a wide-range of substantive and regional interests. Sixty-six percent of respondents reported “moderate,” “high,” or “extremely high” familiarity with the AB (Figure 2.7). Ninety-one percent have read a scholarly article using the data, 67% have viewed or downloaded the data, 46% have used the data in their own research, 42% have included research using the data in their classes, and 62% have read one of the LAPOP Insights Series reports (Figure 2.8). Fifty percent of respondents felt the DIMS data were more useful than other Latin American public opinion resources, compared to 18% who said it was “about the same” and 2% who said it was “less useful” (Figure 1.7). IV. EVALUATION TOPIC #3: CAPACITY BUILDING A principal requirement for LAPOP set forth in the initial DIMS Cooperative Agreement (p 10) is to “promote and enhance . . . the capacity of citizens in the [DIMS] region to conduct, analyze, and disseminate surveys”. At the start of the CA in 2006, survey research and analysis capacity was distributed unevenly across DIMS countries. In some countries, such as Mexico and Colombia, local survey research capacity was strong; scholars, experienced in the analysis of survey data, taught and had 18 DIMS Evaluation active survey research agendas in national universities with strong political science, sociology or economics departments. In other countries, however, such as Haiti and Honduras, few if any professional survey research firms existed, and local universities either did not have political science departments or those departments did not include many faculty with substantial training in survey or quantitative methods. Given this, LAPOP has identified and collaborated with qualified partners (data collection firms, research institutions, and individual researchers) in countries where they existed, while being forced to contract with regional firms in those countries without sufficient capacity. TECHNICAL COMPETENCE All DIMS field work is sub-contracted to survey research firms based in DIMS countries, though not always in the countries where data collection takes place. In twelve countries, the survey research firm is based in the country where the survey is conducted (see Table 3.1). In four countries, Haiti, Honduras, Nicaragua, and Panama, the perceived weakness of indigenous survey research firms has led LAPOP to contract with Borge y Asociados, headquartered in Costa Rica, to conduct the field work. For the 2012 survey year, thirteen countries used professional for-profit data collection firms, whereas three countries used university-based data collection facilities (see Table 3.1). LAPOP is not the only large client of the survey firms in DIMS countries. Unlike most clients, however, who contract for a survey and provide a questionnaire but leave the survey firm to design the sampling frame, train the interviewers, pretest the survey, hire and train field supervisions, carry out the field work, and code and clean the data, LAPOP works with the survey firm on every step. LAPOP determines the sampling frame; LAPOP or its country leaders also supervise all of the interviewer training and the pretests. LAPOP is in frequent contact with the survey firm to deal with questions or problems that arise during interviewing, and they carefully verify the data afterward. LAPOP and DIMS receive high marks from USAID Missions for improving the local capacity of survey firms: 70% of the Mission staff surveyed “strongly agree” or “agree” that “the AB has contributed to the quantitative research capacity of local data collection firms” (Figure 3.1). Interviews conducted with local team leaders and the survey firm directors confirm this view. One director noted that the firm’s use of statistical analysis packages has increased as a result of DIMS. Where they work with indigenous survey research firms, LAPOP has helped to strengthen and enhance the reputations of those firms. In a number of countries, LAPOP has helped the firms develop better, more representative sampling frames. In Colombia, for example, Centro Nacional de Consultoría was a thriving survey research firm even before partnering with LAPOP. The director of the firm reports, however, that LAPOP helped to sharpen their sampling frame and increase its accuracy. The firm now offers clients a menu of different sampling frames of varying precision and price. They call the highest quality and most expensive sampling frame “the AmericasBarometer” frame. Technologically, as well, LAPOP’s use of PDAs has improved survey research in Latin America. PDAs not only enable survey firms to conduct interviews in the field with greater accuracy and lower cost, but they also have been invaluable to interviewers in countries with multiple indigenous languages, such as Guatemala, Peru, and Ecuador, since PDAs allow interviewers to switch back and forth easily between different language versions of the questionnaire. Most interviewed firms report that they now routinely use PDAs for all of their surveys. Borge y Asociados was deeply involved in the development of the LAPOP PDA software, which the Regional General Manager said their firm ‘co-developed.’ This experience built the capacity of Borge with regard to computer programming. Another way in which survey firms report that their connection with LAPOP has contributed to capacity is in terms of reputation. The AB is viewed as the “gold standard” for political public opinion surveys not only in Latin America but also among other regional barometers. Among academics surveyed, 85% reported that the AB reputation among Latin American political and social scientists is 19 “high” or “extremely high” (Figure 3.2). Interviewed survey firms also expressed strong support for the AB. For example, two large and prominent LAPOP-affiliated survey firms (Borge y Asociados in Central America and the Centro Nacional de Consultoría [CNC] in Colombia) commented on the AB’s strengths in terms of its coverage and in terms of quality control. Additionally, participation in the AB brings associational benefits for survey firms, with CNC noting that they conduct the AB more for its reputational benefits than its financial ones. LAPOP has not identified a local survey firm with which to partner in four DIMS countries. They contract instead with Borge y Asociados, located in Costa Rica. There is no indication that the quality of the survey data collected in these four countries is of any lesser quality than the data collected by local firms in other countries. Still, this arrangement means that the countries that most need local survey research capacity-building are the ones that gain the least local capacity from DIMS, as the use of local staff is limited to lower-level positions. HUMAN CAPITAL Local capacity to analyze and interpret survey data also varies across DIMS countries in close relationship to the nature and quality of their universities and social science departments. LAPOP has made concerted efforts to recruit promising young individuals from local survey research firms or local universities to enroll in the Political Science graduate program at Vanderbilt University. LAPOP had a similar policy when it was based at the University of Pittsburgh. A number of the DIMS team leaders are Pittsburgh and Vanderbilt alumni. LAPOP efforts in these regards are viewed positively by Mission staff and others. The Mission survey, for example, showed that 56% of respondents feel “very strongly” or “strongly” that DIMS “has built the capacity of local researchers to conceptualize and conduct research,” compared to 22% who disagreed (Figure 3.3). In Colombia, an academic not affiliated with LAPOP argued that AB has been very important for bringing more systematic, theoretical, and empirically sound research to political science departments throughout Bogotá. As illustrated in Table 3.2, fifteen Central and South American students received PhDs while associated with LAPOP either at Pittsburgh or Vanderbilt over the past fifteen years. Another nine students from the region currently are enrolled at Vanderbilt (Table 3.3), supported financially by LAPOP, Vanderbilt, or in a few cases, USAID Missions. This is in addition to another ten individuals from the United States and other countries outside Latin America who have been affiliated with LAPOP while working on their PhDs in Political Science at Pittsburgh or Vanderbilt. The value of these efforts is clearly shown by the number of LAPOP alumni who hold positions within the organization. Team leaders in seven of the DIMS countries are LAPOP alumni with PhDs from Vanderbilt or Pittsburgh or are currently working on PhDs at Vanderbilt. These team leaders further contribute to local capacity by bringing local research assistants into their work with LAPOP and encouraging them to consider studying survey methods in the United States. LAPOP and Vanderbilt have invested a considerable amount of their own money to make this process happen given that Vanderbilt spends more than $20,000 per student per year on graduate stipends alone. The strength of the recruitment and training system developed by LAPOP is that it identifies talented individuals, gives them a high level of training in political science and survey research methods, and encourages strong professional values and a commitment to LAPOP and DIMS. What the system cannot do is ensure that the human capital thus nurtured will return to the countries from where the students were recruited. Eight of the fifteen individuals identified by LAPOP as Latin American alumni of their program, including the current LAPOP Team Leaders for Guatemala, Peru and Venezuela, are employed and live permanently in the United States. While the human capital cultivated through LAPOP is not lost to the students’ home countries, those countries benefit less than they would if the individuals had 20 DIMS Evaluation returned home to full-time positions in local universities and/or survey research firms. Individuals, of course, are free to pursue careers of their choosing, in places of their choosing. But there is a strong argument to be made for providing graduate funding to students from DIMS countries with a proviso that those students agree to return home for a period of time following the completion of their studies, perhaps equal to the number of years for which they received funding. Additionally, while the strategy of bringing a small number of elite students to the United States to earn doctorates has worked superbly in producing a core of Latin American scholars with training that is among the best in the world, the numbers produced in absolute terms are small. There is a case to be made for augmenting this program with a lower-cost, wider, but broader-impact program. For example, LAPOP might hold intensive summer workshops at selected universities in DIMS countries where larger numbers of graduate students and younger faculty in local universities can be introduced to survey research methods and be given the opportunity to analyze DIMS data. Such outreach is likely to have a greater short-term impact in DIMS universities than awarding a small number of PhDs to students who may not return home. ORGANIZATIONAL SUSTAINABILITY LAPOP is a highly successful organization whose value is widely recognized by USAID/LAC, USAID Missions, NGOs and academics. While there is room for improvement, nearly all of those who were interviewed in connection with this evaluation consider it vital that LAPOP and DIMS be sustained and strengthened. Nevertheless, concerns were registered in various quarters about the nature of the LAPOP organizational structure, its dynamism and ability to adapt, and its long-term prospects of sustainability. One important concern focuses on the future leadership of LAPOP. While Professor Seligson has created an impressive multi-national organization filled with able people, his influence is visible in all aspects of the organization and its operation; LAPOP is very much a top-down organization. The very close association of Professor Seligson with all aspects of LAPOP raises questions about the leadership of LAPOP “after Mitch.” LAPOP and Vanderbilt are aware of this and have addressed the matter by hiring Professor Elizabeth Zechmeister and installing her as Associate Director of LAPOP. Professor Zechmeister is an outstanding political scientist and one of the leading Latin American scholars of her generation. In nine years since receiving her PhD from Duke, she has published two books and more than 20 scholarly articles and book chapters. In her four years with LAPOP, she has established herself within the organization in various ways. However, her roles and responsibilities do not appear to be well known beyond LAPOP Central. Her role needs to become more visible and more public to facilitate an easy transition from Professor Seligson when the time comes. Given the increasing size and scope of the organization, LAPOP needs to evolve into a more collaborative enterprise, regardless of who serves as Director. In the same way that the Michigan Election Study was transformed by its founder into the National Election Study, in which the founder’s role evolved from direct leadership to chairing a nationally representative Board of Advisors with real decision-making authority, it is time for AB/DIMS to develop a Board of Advisors that is truly representative of its multiple constituencies and which has real authority over such important matters as questionnaire composition, data dissemination policies, sampling, etc. The sustainability of DIMS, the AB, and LAPOP more generally also depends upon the identification of future funding that is sufficient to both continue the biennial DIMS surveys and to: 1) expand sample sizes, allowing for more fine-grained regional and sub-group analyses; 2) incorporate new, cutting edge survey research methods and techniques; 3) substantially increase DIMS dissemination activities including the production of many more shorter, country specific reports and press releases; 4) to establish country teams and survey firms in countries where they currently do not exist; and 5) broaden and strengthen the decision making role of the LAPOP Advisory Board. 21 While continued funding from USAID/W should be a major source of financial support for these activities, additional sources of support need to be identified. If DIMS is as critical to the USAID Missions as they overwhelmingly report, then they should be expected to contribute to the support of the core survey in addition to their current periodic contributions to the country reports, dissemination, over￾samples or special topics. Additionally, if the DIMS data and the associated capacity-building contributions are truly important to host governments, they should also be expected to contribute financially to the surveys. Lastly, an exploration of multi-lateral funding, akin to the manner in which Afrobarometer is supported, is worth exploring. LAPOP is well aware of the need for additional funding. In the past six years they report having raised approximately $9 million dollars from a variety of public and private sources. Vanderbilt’s support of the enterprise is substantial as well, totaling almost $2 million by the university’s estimates. V. EVALUATION TOPIC #4: SATISFACTION USAID Overall satisfaction with DIMS within the LAC Bureau at USAID and within USAID Missions is very high. LAC officials who were interviewed had high praise for the project, remarking that “the agreement [with LAPOP] is the cornerstone of the DRG team’s work”, it “drives our agenda”, “it is the one dataset we count on to provide evidence on how to adapt projects” in the field, “it provides measures of DG outcomes that are very hard to measure”, and “it gives citizens a voice in between elections.” The survey of USAID Mission personnel found that over two-thirds of the 30 respondents rated DIMS 8 or higher on a 1-10 scale, with “10” indicating that DIMS was “essential and must be continued” and “1” indicating that DIMS was “of minimal value and the money spent on it should be reprogrammed” (Figure 4.1). The intensive case studies in Colombia and Honduras triangulated the quantitative findings (for more detail see Appendix D), indicating widespread support for DIMS among USAID personnel in the DIMS countries. USAID staff also expressed satisfaction with their role in developing the content and design for DIMS surveys, with their relationship with LAPOP Central, and, to a lesser extent, with in-country LAPOP partners. DIMS “themes” for the various rounds were adopted in consultation with LAC officers at USAID/W, and with Mission personnel as well. In the Mission survey, 87% report that their Mission had contracted with LAPOP to develop country-specific questions, and more than 75% report that the Mission participated “to a great extent” in the content of those questions” (Figures 4.2 and 4.3). Over one-third of Mission respondents report “extensive” input into the core questionnaire, and another 38% report that they review the questionnaire and provide comments during the biennial “kick-off” meetings (Figure 4.4). Mission input into questionnaire development was also emphasized in local LAPOP partner interviews, with some partners lamenting the weaker influence they had compared with USAID. Finally, LAC had extensive praise for the responsiveness of LAPOP Central, with one LAC official claiming that LAPOP was the “most responsive partner ever.” Seventy-two percent of the Mission respondents report that they are “extremely” satisfied with LAPOP’s responsiveness to Mission concerns about the country-specific questions or other questionnaire content, with the remainder saying they are “somewhat” satisfied (Figure 4.5). Nevertheless, USAID personnel voiced several criticisms. As noted above, one concerned the allocation of core versus country-specific questions on the DIMS surveys, with preferences expressed by several 22 DIMS Evaluation USAID interviewees for a greater number of country-specific items. The Mission survey identified a variety of topics, discussed in the section on Quality, that Missions would like included or given greater emphasis ranging from environmental and youth issues to non-traditional forms of political participation. The aspect of DIMS that prompted the most concern among USAID respondents was the dissemination of results. Several open-ended responses in the Mission survey criticized DIMS dissemination, the general theme being that the country reports are too long and not sufficiently “user friendly to engage civil society and other relevant actors in the country.” As one respondent put it: “There is a need to improve the dissemination of the survey within USAID, with government officials and with the public in general. The information contained in the survey is very valuable, but there is a need to make it more accessible to all so that it's used more frequently.” Several others urged “user-friendly, more periodic survey results” and “some reader friendly versions of the report that could be distributed to the communities and the ordinary man in the streets.” These comments resonated with the views of Mission personnel in both Colombia and Honduras as reported in the intensive case studies in Appendix D. They also echo the views of many local partners, who claim that the country reports fail to resonate with Mission personnel and that the amount of interest and engagement that DIMS dissemination events generated was often disappointing. One partner claimed their report “fell with a thud” upon its release; another said the reports are “speaking over the heads of the Missions”; others claimed that the relatively “rigid” format mandated by LAPOP as well as the relatively few country-specific versus regionally-oriented chapters detracted from the local interest. Responsibility for the dissemination of survey results does not rest solely with LAPOP; USAID Missions share responsibility for report dissemination. Whoever is responsible, changes in the dissemination strategy are needed. Improving dissemination is critical to increasing the effective the utilization of DIMS by NGOs, local media and host country governments. ACADEMICS The survey of academics specializing in Latin American politics and/or comparative political behavior shows high levels of satisfaction with the AB, which is seen as producing rigorous, high quality data on issues of importance to Latin American democracy and governance. Academics view the data as “more useful” than other sources of Latin American political opinion data, such as the Latinobarometer. They agree overwhelmingly that the data are “extremely relevant” to the study democratization and public opinion, and believe that scholarship using DIMS has significantly enhanced academic understanding of the field. Academics had two areas of dissatisfaction with the AB, however. The first concerns the accessibility of the survey data, as discussed in the utilization section above. Nearly a third (30%) of academics said that accessing the data was “difficult”, and a good many mentioned easier data access in response to the open-ended question soliciting suggestions for “anything that you think should be changed . . . to improve [the surveys] usefulness”, as indicated below. These responses were given without knowledge of the forthcoming changes in the LAPOP data access policy, but as the letters from outside experts in Appendix E indicate, the new policy, while clearly an improvement, is not likely to satisfy everyone.  “Given the public/federal money it receives, free access to the data should be the baseline. Having to subscribe to obtain the data does not seem right or fair.”  “Should learn from (follow lead of) the Afrobarometer in terms of making data easily accessible.”  “The data should become public. Not only are other surveys (NES, European Social Survey, Afrobarometer) open, but even Latinobarometer data is public now. LAPOP data will only be as valuable as the number of people analyzing the information.”  “Getting access to the data is a pain and so far the data have not been as widely used as they could/should be given their quality and I think that's an issue of access. It is also frustrating that 23 we've had to spend some significant time in getting the data into formats where it could be used when we did get it.” A second concern among academics is questionnaire content and the openness of the process by which new content may be added. The report already has presented evidence regarding areas that the academic community feels are relatively under- or over-emphasized in the AB questionnaire. The open￾ended responses provide additional comments in these regards and also show some dissatisfaction with the process by which questionnaire content is decided:  “Engage more with scholars in Latin America rather than having the surveys always designed by a small number of players”  “Easier access . . . and a clearer process for . . . scholars to request new survey items or retain time￾series items planned for removal”  “Annual open consultation to academic community on ideas on how to improve specific batteries of questions included in the survey”  “It would be great if LAPOP had some formal process . . . to consider the inclusion of survey questions that are of interest to the scholarly community” LAPOP and USAID should consider instituting some additional inclusive and collaborative mechanisms to engage the academic community and enable new, potentially innovative ideas for survey content to be incorporated in the surveys. Finally, a number of scholars suggested that DIMS could benefit from the incorporation of innovative methods that are increasingly used in survey research projects elsewhere. The best available surveys now routinely incorporate experimental methodologies to investigate, among other things, question wording effects, to obtain information on sensitive issues (with so-called “list experiments”), and to explore the impact of elite rhetoric and framing on mass opinion and behavior. Experimental methods investigating “cutting edge” themes in cross-national political behavior such as clientelism, vote-buying, and ethnic political violence have been used effectively in Africa and elsewhere. Other projects have incorporated panel or longitudinal designs in order to more intensively investigate the sources of individual change over time. LAPOP has made some use of experimental methods in questionnaire development and in some special studies for individual USAID missions. There would be value in LAPOP’s expanding its energies in these directions, and in thinking creatively about how to include innovative survey and survey-experimental approaches into the core DIMS design. VI. EVALUATION TOPIC #5: COST￾EFFECTIVENESS COSTS USAID has obligated $11,985,360 through the DIMS Cooperative Agreement with LAPOP, representing 81% of the total estimated cost of $14,760,000. The agreement covers three components: four rounds of DIMS data collection across 16 countries19; oversampling modules and special studies requested and paid for by USAID Missions; and publication and dissemination of the country and regional reports. At 19Surveys were conducted biennially in even-numbered years, 2006-2012, in all countries except Colombia where three additional surveys also were conducted in the odd numbered years. 24 DIMS Evaluation $7.7 million, DIMS data collection represented the largest component, with oversamples and special studies contributing $4.2 million.20 DIMS funding came from three USAID sources: the Bureau for Latin America and Caribbean’s Regional Sustainable Development office (USAID/LAC/RSD) supported the core DIMS survey and the CARSI impact evaluation; USAID Missions funded oversamples and special studies; and the Bureau for Democracy, Conflict, and Humanitarian Assistance's Center of Excellence on Democracy, Human Rights, and Governance (USAID/DCHA/DRG) contributed to the core survey and funded a website improvement for Costa Rica University’s Centro Centroamericano de Población. The largest constituent costs of the Cooperative Agreement, rank ordered by size, were: sub-awards (40%), indirect costs (31%), and labor21 (14%). Core DIMS data collection, reporting and dissemination involves a total cost of $1,925,000 per survey round, or an average of $120,313 per country survey. In response to a request for more detailed budgetary and contractual information from a sample of DIMs countries, LAPOP sent the evaluation team the original Cooperative Agreement, all contract modifications, and detailed cost information for seven DIMS countries: Costa Rica, El Salvador, Guatemala, Haiti, Honduras, Nicaragua, and Panama. However, since the sample excluded countries from South America, the representativeness of the data is not known. While additional data were requested from and provided by USAID/LAC, these data were highly aggregated, making meaningful comparisons impossible. The lack of a central, detailed and standardized accounting of CA costs at USAID/W made it impossible to triangulate LAPOP cost information using Agency data. The following analysis should be read with these limitations in mind. For the sample of DIMS countries provided by LAPOP, an average of 49% of sub-award costs were allocated to data collection, with an average survey unit cost of $23.61. On this basis, 20% or roughly $2,350,000 of the total Cooperative Agreement cost (or 30% of the $7.7 DIMS component cost) was spent directly on DIMS data collection. Other subcontracted costs as a proportion of the CA total include: report writing (23%*), report printing (17%), report dissemination (12%*) and pre￾testing/training (7%*).22 Total DIMS costs varied widely across countries and years. The largest range in total country costs, or difference between most and least expensive countries, occurred in 2010 ($69,377), while the smallest was in 2006 ($15,596). In both cases, Guatemala was the most expensive country, while Costa Rica and Nicaragua were the least expensive.23 There was less variability in data collection costs. The difference between the most and least expensive countries ranged from $13,203 in 2008 to $17,450 in 2010. The largest mean deviation24 in data collection costs across countries occurred in the 2010 ($5,851), with 20 Of this total, $3.2 million was spent on the CARSI impact evaluation and around $1 million was spent on other special studies and oversamples. Due to the decentralized CA accounting practices, it is not possible to disaggregate the costs of the components. 21 Due to the manner in which the CA is structured, labor costs are inclusive of fringes. 22 Figures marked with an asterisk are drawn from activities that are sometimes directly paid for and implemented by LAPOP personnel. Accordingly, these figures are derived from a smaller number of observations. 23 Much of this variation is explained by the fact that DIMS does not support all data collection and reporting tasks consistently in each country. Costa Rica, for instance, did not receive DIMS financing for non-data collection activities after the 2006 round. For Guatemala, LAPOP has needed to translate surveys into multiple indigenous languages and use multi-lingual enumerators. In addition, LAPOP has worked with a local NGO that only uses university-educated enumerators, further increasing costs. 24 Mean deviation is the average of the absolute values of the differences between each value in a data set and the average of all the values of that set. This measure, which has an intuitive appeal as compared to standard deviation 25 Guatemala the most expensive country and Costa Rica, Honduras, Nicaragua and Panama tied for the least expensive countries with respect to data collection. The largest increase in data collection costs over time occurred in Guatemala, which experienced 37, 13 and, 8 percent budget increases from 2006 to 2012. Conversely, El Salvador and Honduras experienced the smallest increases over time. Given the large geographic scope, the six-year time horizon, and the fact that data collection partners changed for certain countries, cost variability is relatively limited. This is especially true if one removes the countries with omitted costs from the calculation. While essential for a rigorous analysis of cost-effectiveness, direct cost comparison to other large-scale public opinion surveys was not possible due to the proprietary nature of their budgetary information. The evaluation team attempted to solicit data collection costs from other regional barometers through direct communication with directors and donors. Although systematic data was not provided, the Director of the Afrobarometer reported that Afrobarometer surveys typically cost between $60,000- 80,000 per country for a sample size of 1,500, or between $40 and $60 per interview. The Director of the Arab Barometer reported that field work for a recent survey in Morocco cost $35,000 for 1,200 interviews, a per unit cost of just under $30. It is not possible to determine the exact composition of these figures using this highly aggregated data, and the figures represent data collection costs from substantively different contexts. Still, DIMS unit costs appear broadly comparable. In an attempt to increase analytical rigor, the evaluation team acquired budgets for four surveys implemented in Central and South America. On the basis of this limited sample, DIMS unit costs compared favorably.25 To supplement the quantitative cost analysis, stakeholders were asked their perceptions of and satisfaction with DIMS costs. A high-ranking DRG Officer in USAID/LAC reported satisfaction with LAPOP cost controls and stated that, if half of his program budget were to be cut, he would choose to continue funding DIMS. During the site visits, a DIMS Activity Manager indicated that she thought the cost of DIMS oversamples were very reasonable26. A DIMS data collection firm also testified to the effectiveness of LAPOP in negotiating survey budgets and complained that that profit margin for the firm for DIMS was thin relative to other clients and projects. Lastly, a respondent from the State Department, who is involved in the design and implementation of Latin American surveys, compared the data collection costs of DIMS quite favorably to those of DOS surveys.27 These qualitative findings, particularly triangulated with the other evidence, support the conclusion that DIMS is undertaken in a cost-conscious manner with costs that are broadly consistent with other data collection undertakings in the region and elsewhere. or variance, is used when the number of values is small and subsequent analyses are not required. The formula is: 25 A number of these budgets were for studies with much smaller sample sizes. There is an inherent economy of scale with data collection, so generally speaking the larger the sample (N), the lower the unit cost. 26 When the team reviewed the associated data collection budget, it compared very closely with that of the core DIMS survey. As the two surveys are negotiated by separate Operating Units, cost comparability is important. 27 It should be noted that DOS contracts surveys through local data collection firms and does not pay the overhead or fringe costs of contracting a US-based institution. Regardless, the respondent estimated that after including all associated DIMS costs, the surveys would still be comparable. 26 DIMS Evaluation BENEFITS In order to value DIMS relative to its costs, it is also necessary to define outcomes, or benefits. Unlike analyses of development interventions with tangible and well-defined benefits (number of schools built, increase in literacy rates, etc.), it is difficult to quantify the varied and largely intangible benefits provided by a longitudinal, nationally representative, yet regionally comparable public opinion survey. As the primary stakeholder, USAID has derived very practical benefits from DIMS, as discussed in the Access and Utilization section of this report. DRG officers report using DIMS for programming and, to a lesser extent, in evaluation and DRG Assessments. As evidenced by a review of DG Assessments of countries and Missions’ PMPs, as well as in country interviews with DG Officers, DIMS is utilized for strategic planning, performance monitoring and program evaluation purposes as well.28 The fact that all but two of 16 bilateral LAC Missions have directly financed one or more special DIMS studies and/or oversamples further confirms the fact that missions value DIMS services. Most Missions fund special studies or oversamples every survey round. In addition to these localized benefits, DIMS surveys produce significant positive externalities. DIMS is the most comprehensive and rigorous public opinion dataset in the LAC region. The data is a global public good that is used by host country governments, other USG agencies, local and international NGOs, local and international academics, and media from around the world. Additionally, as discussed in the Capacity Building section, DIMS creates value beyond the data and reports it generates. The institutional and individual experience gained by the large number of data collection firms, partner organizations, and LAPOP employees contributes toward building local social science capacity. Lastly, and perhaps most difficult to quantify, is the extent to which DIMS data drives public debate in host countries. As discussed in the Utilization section of this report, there is evidence that DIMS data feed into national-level conversations about the state of democratization and support for political institutions, whether in newspaper editorials about low levels of public trust or public debates in the legislature regarding crime and human security. Given stakeholder support for DIMS, the benefits reported by different constituencies, the significance of regional time-series data, and the lack of any reasonable alternatives,29 the continuation of DIMS is highly desirable. While increasing the involvement of host country collaborators would reduce US-based labor and indirect costs, the paucity of human and institutional capacity in many countries and the challenges of coordinating large-scale surveys across countries make the continued involvement of LAPOP critical to the quality of DIMS. Similarly, while DIMS’ costs are significant, direct data collection costs cannot be significantly reduced without adverse effects on data quality. Aside from a cost-sharing venture akin to the multilateral approach used for funding the Afrobarometer, the best way to increase the cost-effectiveness of DIMS would be to increase funding for dissemination activities. A more concerted effort to publicize findings would do much to increase utilization and enhance the benefit side of the cost-effectiveness analysis. 28 During the Honduras site visit it was learned that DIMS data precipitated a shift in focus of the Mission’s Rule of Law programming. The result was the creation of a $10 million transparency program. 29 Other regional surveys do exist but are insufficient in several ways. The Latinobarometer is the closest possible stand-in for DIMS; however, it does not compare favorably with regard to coverage, it does not provide consistent time-series data, and, until quite recently, there were serious questions about the quality of its sampling frames. 27 VII. EVALUATION TOPIC #6: GENDER LAPOP and DIMS have a mixed record with regard to promoting gender awareness, and more specifically, gender equality. While LAPOP has done an excellent job ensuring women’s representation among core staff at LAPOP Central, women are less represented among the advisory boards, team partners, and survey firm staff. The DIMS surveys provide only limited attention to gender issues, which in turn leads to limited attention to gender in reporting and dissemination. GENDER EQUALITY IN LAPOP GOVERNANCE AND ADMINISTRATION The administration of DIMS by LAPOP involves several levels both in the United States and in DIMS countries. Gender equality varies across the different levels. At Vanderbilt, the director of LAPOP is male and the associate director is female. Below that is the central LAPOP staff who handle the day-to￾day responsibilities for DIMS. According to the most recent LAPOP brochure (July 2012), the staff of five is all-female. Among LAPOP’s Faculty Fellows, faculty members from various Vanderbilt departments who are unpaid but affiliated with LAPOP, three are women and three are men. This is an impressive record of gender equity. LAPOP also has two advisory boards: the International Advisory Board and the Scientific Support Group. The International Advisory Board has 16 members, two of whom are women. The Scientific Support Group has six members, one of whom is a woman. However, neither board is reported to be very active or influential in LAPOP activities. At the country-level of LAPOP’s administrative structure, LAPOP has team leaders who are responsible for each of the countries where they administer DIMS. At the time of the 2006, 2008, and 2010 surveys, 22, 29 and 27 percent, respectively, of team leaders were female (some countries have co-team leaders). For the 2012 survey, 32% (7 of 22) team leaders were female (see Figure 6.1).30 Systematic data are unavailable regarding the staff who may work with the team leaders in most countries. TRAINING AND DEVELOPMENT OF SURVEY RESEARCH STAFF DIMS surveys are typically conducted by different survey research firms in each country, although there are several Central American countries where surveys are conducted by the same company. Only limited data is available on the gender composition of the survey firms. Among the primary LAPOP contacts at the DIMS survey firms, three were female out of a total of 23 women. Systematic data do not exist on the gender composition of the survey enumerators/interviewers who carry out the survey in each country, though everyone reported that their interviewers are disproportionately women. This is intentional; the assumption typically is that people are more willing to open their doors to female interviewers and feel more comfortable talking with them. DESIGN AND CONTENT OF THE SURVEY QUESTIONNAIRES Issues of gender can be addressed in survey research in two ways. One is by including a question asking the respondent’s sex, which allows analyses of men’s and women’s differences in their responses to the 30 These percentages are roughly the same as the percentage of women who are Assistant Professors of Political Science in the United States, although this is not an especially high standard and invokes the question of gender equality (50% women, 50% men). 28 DIMS Evaluation full range of questions on the survey. The other is by including survey questions specifically addressing issues about women and gender. On the first dimension, all four DIMS surveys included a question asking for the sex of the survey respondent. On the second dimension, however, the DIMS surveys (particularly until 2012) had very few questions on women’s and gender issues (Tables 6.2 and 6.3). For the 2006, 2008, and 2010 DIMS core questionnaires, only two questions appeared on women’s and gender issues, and there was only one survey year that included both of these questions in the same survey (Table 6.2). In 2012, the theme for the DIMS survey was “Marginalization in the Americas,” and four questions on women’s and gender issues were included. In all but one case, the country-specific portions of DIMS questionnaires fare no better.31 Table 6.3 illustrates that only the Dominican Republic includes an extensive set of items on women and gender, with 10 questions in 2006, 11 questions in 2008, and 14 in 2010. No other country has included more than 4 women and gender questions in any given year, and even that occurred only in Colombia (2010), Nicaragua (2008), and Panama (2008). The limited attention to women’s and gender issues on the DIMS surveys was lamented by several academics, team leaders, and civil society leaders and was noted in the internet survey of academics. As reported earlier, 36% of academic respondents perceived too little attention to “the inclusion of marginalized communities” and 28% perceived too little attention to “women’s rights and women’s political issues” (Figure 1.11). These were two of the top four categories of issues that respondents felt were underrepresented on the DIMS survey questionnaire. Both male and female survey respondents thought too little attention was paid to gender issues, while female respondents were significantly more likely than males (62% to 26%) to report that too little attention was paid to issues related to marginalized communities (Figure 1.15). One of the academics surveyed additionally lamented that “questions…related to women/gender are only available for certain years.” Overall, the coverage of women’s issues and concerns in the DIMS questionnaires is poor. Few questions are asked and virtually none are repeated consistently across the surveys to allow gender issues to be tracked over time. Especially disappointing is the fact that in the 2012 survey where the theme of “Marginalization in the Americas” only four gender questions were included. GENDER IN DIMS REPORTS AND ANALYSES Attention to gender in the reports occurs in several ways: one is by “controlling” for a respondent’s sex as one of several possible influences on political outcomes, another is to highlight sex differences that emerge in the analyses, and the final is to explicitly analyze questions on women’s and gender issues. In regards to the first, nearly all of LAPOP’s analyses include a control variable for respondent sex. In regards to the second, LAPOP reports sex differences in the rare instances that they are relevant. In terms of analysis of the questions on women’s and gender issues, the central Report on the Americas and individual country reports are sparse, due to the limited number of questions on women’s and gender issues in the surveys. The 2010 comprehensive report analyzed one question on women’s issues. Attention to women’s and gender issues varies in the country reports. The Dominican Republic has given the most attention to gender in its reports. Their questionnaire has 14 questions about women and 126 mentions of women in the report. Most countries fall far below this outlier, however (Table 6.4). The paucity of attention to gender was noted by several interviewees. LAPOP has promoted gendered analysis of DIMS data in other ways. In 2010-2011, they published two Insights series reports focusing on gender. Then, in preparation for the 2012 survey, LAPOP ran a grant competition for scholars conducting research on “discrimination, marginality, gender, and democracy” in the Americas to use the AB data for their research. They held a half-day academic conference at the 31 This claim does not necessarily apply to the 2012 country questionnaires, to which we did not have access. 29 2012 start-up meeting in Miami, Florida, where six scholars presented research on marginalization in the Americas, three of which covered gender issues. Academic analysis of gender using the DIMS data has also been minimal–only five studies were identified. Consideration of gender has been limited among USAID missions as well. Three mission documents and one respondent to the mission survey referenced programs that have used gendered analyses of the DIMS data in their work. VIII. RECOMMENDATIONS A. FUNDING OF THE DIMS SURVEYS 1. It is highly recommended that USAID continues and, if possible, expands its financial support of DIMS. The Democratic Indicators Monitoring Surveys play a critical role for USAID’s LAC Bureau and for the Missions in virtually all of the DIMS countries. DIMS is also highly valued by a number of other constituencies and its uses have still to be fully realized. 2. LAPOP and the Missions should devote considerably more time, effort and money to the dissemination of DIMS data and analyses, especially to local media and DIMS country governments. For LAPOP, this means producing more and more frequent short, readable, country-level analyses on topics of contemporary interest in particular countries. Providing these analyses will require that LAPOP ensure the existence of high quality country teams in all DIMS countries. For USAID, this means investing in public relations experts for Missions who can better and more regularly communicate with the media and government officials and develop stronger formal and informal ties. Getting Missions and foreign governments to “buy in” to DIMS is critical to increasing the utilization of DIMS data and analyses in order to improve both government and Mission performance. 3. LAPOP should work with USAID/Washington, USAID Missions, other government and donors to broaden and diversify the sources of DIMS funding. B. TECHNICAL QUALITY OF DIMS SURVEYS 4. LAPOP should rethink its use of quota sampling at the household level and reconsider the cost￾effectiveness of converting to full probability methods. 5. LAPOP should take advantage of innovative survey and survey-experimental methods, as they did with the adoption of PDAs, not only to maintain its reputation as a leader in survey research, but also because the use of such methods would enhance the DIMS’ capacity to be used for strategic planning, performance monitoring and evaluation. It also would facilitate more and higher quality academic research. 6. LAPOP should further increase the number of interviews conducted per municipality and with under-represented subgroups to facilitate regional analyses and allow DIMS to be used more effectively for program evaluation. Additional interviews are costly and USAID/W or local Missions would need to increase DIMS funding to enable this. 7. LAPOP should reduce the length of the questionnaire. Repetitious and irrelevant questions added as a result of “buy-ins” or other special considerations should be culled. At the same time, the time-series core needs to be reduced in length, and more space needs to be created, within a shorter DIMS, for the inclusion of country-specific questions. LAPOP also needs to revisit the focus of the core time-series and consider whether there are other topics (for example, questions on women and marginalized groups) that merit inclusion over existing ones. 30 DIMS Evaluation 8. LAPOP should have a fully trained team leader living full-time in each DIMS country to maximize data quality and utilization. C. LAPOP GOVERNANCE 9. The process by which LAPOP makes decisions regarding questionnaire construction should be made more transparent and inclusive. This should be accomplished by, at minimum, the creation of a larger, more active and influential Advisory Board that is representative of DIMS’ multiple constituencies, including USAID/W, USAID Missions, and academics. 10. Continuing efforts are required to build on LAPOP’s strong leadership structure and to plan for succession. The value of DIMS to so many constituencies commends the project’s continuation, but this requires that LAPOP remain a vibrant and dynamic organization. Vanderbilt University has taken an important step to address its inevitable issue of leadership succession with the hiring of an Associate Director of LAPOP. Further steps are needed to enhance the Associate Director’s role and promote her visibility both within LAPOP and among its many constituencies. LAPOP should also begin a transformation into a more collaborative, transparent, publicly-accountable institution. The development of a more active and influential Advisory Board that is more representative of DIMS’ multiple constituencies would help substantially in this regard. D. CAPACITY BUILDING 11. LAPOP should do more to build survey capacity in the four countries where outside survey research contractors are used. LAPOP should consider contracting with a local survey research contractor in each country, even if this means partnering with inexperienced contractors until they have the capacity to conduct independent surveys. Though additional costs are associated with this approach, the potential long-term benefits are well worth the expenditure, especially if USAID provides financial support for the activity. 12. LAPOP should better promote gender equality among team leaders and survey partners. It also needs to make the inclusion of questions on gender and other marginalized groups a high priority for the DIMS core time-series. 13. LAPOP and the local Missions should broaden human and institutional capacity building by sponsoring workshops in various DIMS countries, paying local students and young faculty to attend in order to acquire an awareness of DIMS data and the basic skills to use it in their classes and research. These types of workshops are quite common and are held every summer at the University of Michigan, Syracuse University, Virginia Polytechnic Institute, the University of Essex in Great Britain, the Central European University in Budapest, Academia Sinica in Taiwan, and other global locations. This activity should be coordinated by local team leaders with additional funding provided by USAID Missions. E. MANAGEMENT OF THE DIMS COOPERATIVE AGREEMENT 14. USAID/W should improve the central collection of data on DIMS activities and spending such that country and activity costs can be meaningfully compared over time. While the decentralized manner in which the Agreement is currently managed facilitates direct communication between Missions and LAPOP, detailed explanations of DIMS activities funded through Missions should be systematically and consistently shared with USAID/W. Having an easily-accessible central repository of all constituent tasks, as well as a detailed and consistent accounting of CA costs would greatly improve the ability to evaluate DIMS performance. 31 FIGURES AND TABLES EVALUATION TOPIC #1: QUALITY Figure 1.1: Academic Survey Figure 1.2: Academic Survey by Regional Specialization 0 10 20 30 40 50 60 Low to Moderate High Extremely High Percentage of Valid Responses Perceived Quality Would you say that the scientific quality of the AmericasBarometer data is: 0 10 20 30 40 50 60 70 80 Low to Moderate High Extremely High Percentage of Valid Responses Perceived Quality Would you say that the scientific quality of the AmericasBarometer data is: Latin America: Primary Field Latin America: Secondary Field No Latin American Specialization 32 DIMS Evaluation Figure 1.3: Academic Survey by Type of Institution: Figure 1.4: Academic Survey by Location of Institution: 0 10 20 30 40 50 60 70 80 Low to Moderate High Extremely High Percentage of Valid Responses Perceived Quality Would you say that the scientific quality of the AmericasBarometer data is: Academic Not Academic 0 10 20 30 40 50 60 Low to Moderate High Extremely High Percentage of Valid Responses Perceived Quality Would you say that the scienfic quality of the AmericasBarometer data is: US-Based Not US-Based 33 Figure 1.5: Academic Survey by Gender: Figure 1.6: Academic Survey by Familiarity with AB Data 0 10 20 30 40 50 60 Low to Moderate High Extremely High Percentage of Valid Responses Perceived Quality Would you say that the scientific quality of the AmericasBarometer data is: MALE FEMALE 0 10 20 30 40 50 60 70 80 90 Low to Moderate High Extremely High Percentage of Valid Responses Perceived Quality Would you say that the scientific quality of the AmericasBarometer data is: LOW MODERATE HIGH-EX HIGH 34 DIMS Evaluation Figure 1.7: Academic Survey Figure 1.8: Scholarly Articles Citing AB Data 0 10 20 30 40 50 60 70 80 Less Useful About the Same More Useful Percentage of Valid Responses Perceived Usefulness Would you say that the AmericasBarometer surveys provide more useful information than other sources of public opinion in Latin America, less useful information, or is the information about the same as other sources of data? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 11 131 88 54 82 60 Top Journal Not Top Journal US Publication Outside Publication LAPOP Central Affiliation Not Affiliated Percentage of Total Scholarly Articles Total N for DIMS Countries: 142 35 Figure 1.9: Academic Survey Figure 1.10: Academic Survey 0 10 20 30 40 50 60 70 80 Completely Irrelevant Not Very Relevant Somewhat Relevant Extremely Relevant Percentage of Valid Responses Perceived Relevance In your view, how relevant are the AmericasBarometer surveys to the study of public opinion and/or democratization processes in Latin America? 0 10 20 30 40 50 60 Somewhat Enhanced Greatly Enhanced Percentage of Valid Responses Perceived Enhancement of Latin American Public Opinion and Democratization Thinking about scholarship that uses AmericasBarometer data or analyses, would you say that this work has greatly enhanced, somewhat enhanced, or not enhanced the understanding of public opinion and/or democratization in Lati 36 DIMS Evaluation Figure 1.11: Academic Survey Figure 1.12: Academic Survey by Regional Specialization 0 5 10 15 20 25 30 35 40 Percentage of Valid Responses Thinking now about the concent of the AmericasBarometer surveys, please tell us if there is too little attention paid to the following areas: 0 5 10 15 20 25 30 35 40 45 Percentage of Valid Reponses Thinking now about the content of the AmericasBarometer surveys, please tell us if there is too little attention paid to the following areas: Latin America: Primary Field Latin America: Secondary Field No Latin American Specialization 37 Figure 1.13: Academic Survey by Type of Institution Figure 1.14: Academic Survey by Location of Institution 0 5 10 15 20 25 30 35 40 45 Percentage of Valid Responses Thinking now about the content of the AmericasBarometer surveys, please tell us if there is too little attention paid to the following areas: Academic Not Academic 0 5 10 15 20 25 30 35 40 Percentage of Valid Responses Thinking now about the content of the AmericasBarometer surveys, please tell us if there is too little attention paid to the following areas: US-Based Not US-Based 38 DIMS Evaluation Figure 1.15: Academic Survey by Gender Figure 1.16: Academic Survey by Familiarity with AB Data 0 10 20 30 40 50 60 70 Percentage of Valid Responses Thinking now about the content of the AmericasBarometer surveys, please tell us if there is too little attention paid to the following areas: MALE FEMALE 0 5 10 15 20 25 30 35 40 Percentage of Valid Responses Thinking now about the content of the AmericasBarometer surveys, please tell us if there is too little attention paid to the following areas: LOW MODERATE HIGH-EX HIGH 39 Figure 1.17: Academic Survey 0 5 10 15 20 25 Percentage of Valid Responses Thinking about the content of the AmericasBarometer surveys, please tell us if there is too much attention paid to the following areas: 40 DIMS Evaluation EVALUATION TOPIC #2: ACCESS AND UTILIZATION Figure 2.1: Mission Survey Figure 2.2: Mission Survey 0 5 10 15 20 25 30 35 Read reports prepared by Vanderbilt LAPOP or others which use AmericasBarometer data? Asked someone else at the mission to do some analyses of AmericasBarometer data? Personally used a spreadsheet program, such as Excel, to create figures or tables from DIMS data? Used AmericasBarometer data, tables, or other information in a public presentation you’ve given as part of your official duties? Seen AmericasBarometer data, tables or other information in a presentation given by others as part of their official duties? Which of the following ways have you used AmericasBarometer data? Have you: Yes No extremely high 20% high 50% moderate 20% low 10% have not heard of AmericasBarometer (or LAPOP) at all 0% How familiar are you with the AmericasBarometer surveys? Would you describe your familiarity with the AmericasBarometer surveys and their contents as: 41 Figure 2.3: Mission Survey: Figure 2.4: Mission Survey At least once a month 10% Several times a year 70% Not more than once or twice a year 17% Virtually never 3% In your day-to day activities, about how often do you use or make reference to AmericasBarometer / DIMS data? Often 37% Once or twice 50% Never 3% Don’t recall 10% How often would you say that you have used AmericasBarometer data or analyses in DEVELOPING specific DRG programs or targeting their focus? 42 DIMS Evaluation Figure 2.5: Mission Survey Figure 2.6: Mission Survey Often 13% Once or twice Never 33% 47% Don't recall 7% How often would you say that you have used AmericasBarometer data or analyses as part of an EVALUATION of specific DRG programs? It was central to the DG assessment process 7% It played an important role but there are more important sources of information 54% It played a role, but a very limited one 31% It didn’t play any significant role at all 8% What role has AmericasBarometer played in previous DG assessments in your mission? 43 Figure 2.7: Academic Survey Figure 2.8: Academic Survey Never Heard of the Surveys 12% Low Familiarity 22% Moderate Familiarity 30% High Familiarity 21% Extremely High Familiarity 15% Familiarity with the Surveys 0 10 20 30 40 50 60 70 80 90 100 Read Article Using AB Data Viewed Information on LAPOP Website Read an Insight report Used AB Data in Own Research Assigned an Article Using AB Data Received a Grant from LAPOP Percentage of Respondents Please tell us if you have done the following activities in your academic career: 44 DIMS Evaluation Tables: Table 2.1: Mentions of AmericasBarometer in USAID Mission Documents Country # Documents # Substantial Mentions # Semi￾Substantial Mentions # Not Substantial Mentions Colombia 6 3 1 1 Dominican Republic 6 0 0 0 Ecuador 1 0 0 1 El Salvador 3 0 0 9 Guatemala 2 1 0 5 Haiti 0 0 0 0 Honduras 3 1 2 7 Jamaica 0 0 0 0 Mexico 3 1 1 0 Nicaragua 2 1 2 5 Panama 0 0 0 0 Paraguay 0 0 0 0 Peru 3 0 3 12 Notes: A: "Substantial" means that several pages (more than 2) are devoted to a discussion of LAPOP. These mentions tend to be misleading, as they are not really "mentions." Instead, they tend to be documents entirely about LAPOP. So there cannot be more than one "substantial mention" in the same document because the mention is the document. B: "Semi-Substantial" indicates a more-or-less thorough discussion of the data within a document about a program or an assessment. I code the discussion as "semi-substantial" if it lasts for more than a paragraph (even if it just means that there is a detailed footnote LAPOP to further explain a thorough paragraph about LAPOP). So the minimum of "semi-substantial is one paragraph + a detailed footnote, and the maximum is 2 pages. C: "Not Substantial" indicates a sentence to a brief paragraph about the data. This tends to be a very brief description of some relevant finding in a report. 45 Table 2.2: AB Media Citations in DIMS Countries Country/Year Number Percentage Dominican Republic 43 20.87% Colombia 25 12.14% Mexico 18 8.74% Uruguay 17 8.25% Jamaica 15 7.28% Guatemala 14 6.80% Ecuador 12 5.83% Nicaragua 11 5.34% Peru 11 5.34% El Salvador 10 4.85% Honduras 9 4.37% Panama 9 4.37% Paraguay 8 3.88% Guyana 2 0.97% Costa Rica 1 0.49% Haiti 1 0.49% 2006 2 0.98% 2007 30 14.71% 2008 27 13.24% 2009 29 14.22% 2010 34 16.67% 2011 67 32.84% 2012 (incomplete year) 15 7.35% Total Media Mentions 206 100% 46 DIMS Evaluation Table 2.3: Scholarly Publications on DIMS Countries Publication Type Number Scholarly Article 142 Top Journal 11 Book/Monograph 38 Book Chapter 24 Conference 61 Dissertation 30 Total 306 Number of U.S.-based publications 207 Number of foreign publications 99 Number of publications by LAPOP affiliates (team partners, LAPOP faculty, and LAPOP graduate students) 189 47 EVALUATION TOPIC #3: CAPACITY BUILDING Figure 3.1: Mission Survey Figure 3.2: Academic Survey 0 5 10 15 20 25 30 AB has contributed to the quantitative research capacity of local data collection firms. AB has built the capacity of local researchers to conceptualize and conduct research. Thinking about your experiences with LAPOP partners, please state the extent to which you agree or disagree with the following statements: Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don't know 0 10 20 30 40 50 60 Low-Moderate High Extremely High Percentage of Valid Responses Perceived Reputation In your view, what is the overall reputation of the AmericasBarometer surveys among the Latin American political and social science communities? 48 DIMS Evaluation Figure 3.3: Mission Survey Tables: Table 3.1: Survey Firms by Location 2006 2008 2010 2012 In-House Number 4 3 3 3 Percent of Total 25% 19% 19% 19% In-Country Number 8 8 8 8 Percent of Total 50% 50% 50% 50% Multi-Country Number 4 5 5 5 Percent of Total 25% 31% 31% 31% 0 5 10 15 20 25 30 AB has contributed to the quantitative research capacity of local data collection firms. AB has built the capacity of local researchers to conceptualize and conduct research. Thinking about your experiences with LAPOP partners, please state the extent to which you agree or disagree with the following statements: Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don't know 49 Table 3.2: LAPOP Trained Students from Central and South America NAME COUNTRY CURRENT POSITION OSE RENE ARGUETA Honduras Political Science Research Consultant, Pittsburgh, PA. ARIEL ARMONY Argentina Weeks Professor of International Studies and Director of the Center for Latin American Studies, University of Miami, Florida SILVIA LUCRECIA DEL CID AVALOS Guatemala Deceased DINORAH AZPURU Guatemala Associate Professor of Political Science, Wichita State University, Kansas; Associate Member of ASIES (Guatemala); Team Leader, LAPOP Guatemala MARIA FERNANDA BOIDI Uruguay Professor, Universidad de Montevideo, Uruguay, LAPOP Program Coordinator-Field Operations ERNESTO CABRERA Argentina Senior Vice President, Market Fusion Analytics, New York City, New York DAMARYS CANACHE Venezuela Associate Professor, Department of Political Science, University of Illinois; Team Leader, LAPOP Venezuela JULIO FRANCISCO CARRION Peru Associate Professor and Director, Center for Latin American Studies, University of Delaware, Team Leader, LAPOP Peru ANNABELLE CONROY Bolivia Assistant Professor, University of Central Florida, Orlando ABBY B. CORDOVA GUILLEN El Salvador Post-doctoral Fellow and Director of Field Experiment Research, Vanderbilt University RICARDO CORDOVA El Salvador Executive Director, FundaUngo, El Salvador, Team Co-leader, LAPOP El Salvador JUAN CARLOS DONOSO Ecuador Assistant Professor, Universidad San Francisco de Quito, Ecuador , Adjunct Director, Prime Consulting, Team Leader, LAPOP Ecuador JULIANA FRANZONI MARTINEZ Costa Rica Professor of Social Sciences, Institute of Social Investigation, University of Costa Rica JORGE DANIEL MONTALVO Ecuador CEO, Prime Consulting, Associate Professor, International Relations and Political Science, Universidad San Francisco de Quito, Ecuador DIANA ORCES Ecuador Coordinator, School of Political Science and International Relations, Universidad de las Americas, Ecuador 50 DIMS Evaluation Table 3.3: Current LAPOP Graduate Students in Residence at Vanderbilt from Central and South America NAME HOME COUNTRY MARCO A. F. ARAUJO Brazil ALEJANDRO DIAZ DOMINGUEZ Mexico TED ENAMORADO Honduras, Team Leader LAPOP Honduras ARTURO MALDONADO Brazil FREDERICO BATISTA PEREIRA Brazil RAFAEL PINEIRO Uruguay JUAN CAMILO PLATA Colombia MARIANA RODRIGUEZ Venezuela GUILHERME AZZI RUSSO Brazil DANIEL ZIZUMBO-COLUNGA Mexico 51 EVALUATION TOPIC #4: SATISFACTION Figure 4.1: Mission Survey: Figure 4.2: Mission Survey 1 minimum value 4% 2 3 3% 3% 4 3% 5 3% 6 7% 7 7% 8 28% 9 28% 10 essential 14% On a Scale of 1-10 where 10 means DIMS is essential and must be continued and 1 means it is of minimum value and the money spent on it should be reprogrammed, how important is it to you that the contract for DIMS should be renew Yes 86% No 7% Don't know 7% Has your mission contracted with Vanderbilt LAPOP to develop special batteries of questions specifically focused on your country or to increase the sample size for certain sub-populations in the AmericasBarometer surveys? 52 DIMS Evaluation Figure 4.3: Mission Survey: Figure 4.4: Mission Survey To a great extent 77% To a limited extent 23% To virtually no extent 0% To what extent did your mission participate in the design of those questions? We participate extensively, suggesting both general topics and specific questions for inclusion in the survey 34% We review and comment generally on the survey during the Vanderbilt LAPOP organized “kick-off” meetings 38% Vanderbilt LAPOP mostly designs the survey with little input from us. 14% I don't have enough information to answer this question. 14% To what extent does your mission participated in the design of the core AmericasBarometer questions (i.e., the questions that are asked in all of the AmericasBarometer countries surveyed)? 53 Figure 4.5: Mission Survey EVALUATION TOPIC #6: GENDER Table 6.1: Partners by Gender 2006 2008 2010 2012 Male Number 18 15 16 15 Percent of Total 78% 71% 73% 68% Female Number 5 6 6 7 Percent of Total 22% 29% 27% 32% Extremely satisfied 72% Somewhat satisfied 28% Somewhat dissatisfied 0% Extremely dissatisfied 0% How satisfied were you with the responsiveness of the Vanderbilt LAPOP staff and senior personnel to your mission’s concerns about the country-specific questions or other content you wanted to include in the survey? 54 DIMS Evaluation Table 6.2: Core Gender-Related Questions Questions in Core Questionnaires Question # Years Included [Women only] Meetings of associations or groups of women or home makers. Do you attend them… [how frequently] CP20 2008, 2010, 2012 Generally speaking, men are better political leaders than women. How much do you agree with that statement? VB50 2008, 2012 Changing the subject again, some say that when there is not enough work, men should have a greater right to jobs than women. To what extent do you agree or disagree? GEN1 2012 The state ought to require that political parties reserve some space on their lists of candidates for women, even if they have to exclude some men. How much do you agree or disagree? GEN6 2012 Table 6.3: Number of Gender-Related Items in Country Questionnaires 2006 2008 2010 Core Questionnaire 0 2 1 Colombia 1 3 4 Costa Rica 0 2 1 Dominican Republic 10 11 14 Ecuador 1 2 3 El Salvador n/a n/a 1 Guatemala n/a 2 3 Guyana 0 2 1 Haiti 0 2 1 Honduras 0 2 1 Jamaica 0 2 1 Mexico 0 2 3 Nicaragua 0 4 1 Panama 0 4 1 Paraguay 0 2 1 Peru 0 2 3 Uruguay n/a 2 1 55 Table 6.4: Mentions of Gender in LAPOP Reports Women Gender Colombia 16 7 Costa Rica 23 10 Dominican Republic 126 74 Ecuador 26 28 El Salvador 23 10 Guatemala 18 13 Guyana n/a n/a Haiti 23 12 Honduras 6 13 Jamaica 10 17 Mexico 54 28 Nicaragua 41 14 Panama 8 5 Paraguay n/a n/a Peru 21 13 Uruguay n/a n/a MEAN 30 19 MEDIAN 23 13 56 DIMS Evaluation APPENDIX A. SCOPE OF WORK STATEMENT OF WORK Evaluation of USAID’s Democratic Indicators Monitoring SURVEY 1. BACKGROUND The US Agency for International Development (USAID) is planning to conduct an evaluation of its Democratic Indicators Monitoring Survey (DIMS) cooperative agreement, implemented since 2006 by the Latin American Public Opinion Project (LAPOP), a research institute at Vanderbilt University. DIMS, better known as AmericasBarometer (AB), is a series of nationally representative public opinion surveys that measures citizen perceptions of and experiences with various aspects of governance, democracy, and citizen security across the Americas. This evaluation will review the 2006, 2008 and 2010 surveys and accompanying analysis, and their development impact upon and utilization by USAID and other stakeholders and audiences. The findings of this evaluation will contribute significantly to the design of the next iteration of the DIMS agreement. Project Information 1. Project to be Evaluated: Democratic Indicators Monitoring Survey (DIMS) 2. Cooperative agreement #: 598-A-00-06-00061-00 3. Award Dates: March 2006 – March 2014 4. Funding: $14,760,000 5. Implementing Organization: Vanderbilt University’s Latin American Public Opinion Project (LAPOP) 6. Agreement Officer’s Representative: Vanessa Reilly (LAC Bureau) Context In 2003, USAID’s Bureau for Latin America and the Caribbean (LAC) adopted, for the first time, a regional strategy for Mexico, Central America and Panama. This Central America and Mexico (CAM) Strategy (2004-2008) aimed “to promote a more democratic and prosperous Central America and Mexico and bolster citizen confidence in democratic governments.” The strategy emphasized a need “to monitor closely the impact on … the confidence in democratic institutions and processes.” In 2004, the region’s democracy officers decided to conduct a public opinion survey in each of the CAM countries tailored to USAID’s performance monitoring requirements. The purposes of the survey were: (1) to track changes in democratization over time and in each country; (2) to provide a basis for developing USAID Mission Performance Monitoring Plans; (3) to allow USAID to make cross-national comparisons and develop common indicators of progress; and (4) finally, to understand the outcomes or effects of USAID’s democracy programming. In 2006, USAID awarded Vanderbilt University/LAPOP with a cooperative agreement to continue conducting the surveys, now called DIMS 32. 32 Democratic Indicators Monitoring Survey (DIMS) is the name of USAID’s cooperative agreement with Vanderbilt/LAPOP; it funds survey work in 16 countries currently. AmericasBarometer (AB) is the name of the broader survey research project run by Vanderbilt/LAPOP, and covers about 25 countries 57 The DIMS agreement has been extended and expanded over time, to $14.76 million over eight years (2006-2014). USAID funds DIMS in 16 countries, primarily where USAID has missions: Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Guatemala, Guyana, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, Paraguay, Peru, and Uruguay. Vanderbilt/LAPOP averages 1,500 nationally representative surveys per country per survey year, or more than 40,000 across the Hemisphere for each survey round. DIMS surveys include survey modules that cover a broad range of democracy, governance and citizen security topics, such as support for democracy, perceptions of insecurity, crime victimization, corruption, trust in government institutions, interpersonal trust, civic participation and protest, voting behavior, political tolerance, and topical areas such as youth and racism. Approach and Implementation The agreement’s activities include a few key components33: 1) The “core” survey, which asks the same questions of respondents in every country in which Vanderbilt/LAPOP operates, providing time-series data going back to at least 2006 for most countries in the Hemisphere. 2) Country-specific questions and survey modules added to the core survey in each country, at the request of USAID missions. 3) Other “special services” ordered by USAID missions, which include oversamples, special samples, country reports, shorter reports on special topics, and dissemination services. 4) Analysis, including a regional report, topical Insights Reports, and ad hoc data analysis for USAID. USAID funds the cost to gather “core” survey data in each of the 16 countries and missions fund any special services they request. Country-specific questions are generally free to missions, but are limited by the length of the survey. Vanderbilt/LAPOP works through subcontracts with local institutions in each country (professors and university research institutes, survey research firms, think tanks, and civil society organizations, depending on the country). These institutions conduct the surveys, identify and train survey-takers, produce country reports and other deliverables using Vanderbilt/LAPOP’s templates, and oversee dissemination of country reports and other deliverables in the field. Intended Target Audiences of DIMS project The primary intended audiences for DIMS data and analysis are:  USAID: Generally regarded as the primary intended audience, USAID’s objective has been to use the DIMS data and analyses to inform strategic planning and project design, to measure throughout the Western Hemisphere, including the 16 funded by USAID. Most people refer to USAID’s DIMS project as “AmericasBarometer” or the “LAPOP [or Vanderbilt] survey.” Vanderbilt/LAPOP receives additional support from other donors to fund countries or modules that USAID does not fund. 33 The current cooperative agreement also includes impact evaluations of USAID’s crime prevention programs in Central America and Mexico, which will not be a focus of this evaluation. 58 DIMS Evaluation program results, and as a tool for dialogue between the US Government and host country stakeholders.  Host country governments: DIMS data is intended to help recipient country governments to strengthen their evidence-based policy making. The expectation has been that greater access to information would enable governments to develop responsive policies and guide budget priorities.  Host country civil society, media and citizens: The expectation has been that this group would use DIMS data to inform their own projects, advocacy campaigns, and public policy work. The evaluation should define and measure the utilization of DIMS by all these intended audiences. In addition, there are important audiences:  US Embassies and other US Government agencies: The US Government may use the data to understand trends in the region.  Development banks, international donors and international civil society  Academia: Researchers are expected to use the data in academic and policy papers, which can enhance USAID’s and the public’s understanding of regional trends in the LAC region. USAID, through DIMS, is the primary funder of the broader AmericasBarometer project. Other funders for AmericasBarometer include the Inter-American Development Bank, the United Nations Development Programme, Vanderbilt University, and others listed on the project website: http://www.vanderbilt.edu/lapop/. 2. EVALUATION Purpose and Timing If it decides to continue this type of survey research, USAID will use findings from the evaluation to design the next iteration of this agreement. The current cooperative agreement ends in March 2014, with 2012 being the last survey round included. USAID would have to design the next iteration in mid￾2012 to allow time for USAID to award a new agreement and for the next implementer to prepare adequately for the early-2014 round. Questions to Answer This evaluation should review DIMS survey design, implementation, dissemination and utilization of survey results and datasets by all intended users. The evaluation team will decide how to evaluate the outcomes of DIMS and the extent and effects of its use by all intended audiences of this activity. Evaluation should assess the following issues: 1) The quality of survey design, field implementation, project management and long-term sustainability of the project. 2) The quality of survey findings and analysis developed by the cooperative agreement partner and their utilization by USAID. 3) The dissemination and use of the survey results and datasets by different stakeholders including host country governments, civil society/media and academia. 4) Provide recommendation on how to improve the next iteration of the project. USAID expects the evaluation team to present strong quantitative and qualitative analysis that clearly addresses the above issues. While the evaluation team will develop a robust evaluation methodology for 59 this project, the following questions may guide the evaluators in their work to further focus their research. Suggested Evaluation Questions Question category Question or issue to be addressed Methods: Quality of survey design, implementation, survey findings and analysis  Has the survey methodology been conducted to the highest standards?  Has the survey management and administration been conducted to the highest standards?  Has the survey data been sufficiently accessible during the entire project time? How does current accessibility compare to industry standards or other similar cross-national survey projects? Effects and relevance: Utilization of data and analysis by key audiences; dissemination  What has been the effect of DIMS upon host government and civil society counterparts?  What has been the effect of DIMS upon USAID’s strategies and programs?  How are missions and USAID using the data?  Is the core questionnaire asking the right questions to address the needs of USAID and all target audiences? Local capacity development  How and to what extent has local capacity for survey research and analysis been strengthened? Client satisfaction  How do missions use DIMS products? Do missions feel DIMS is useful to them?  How are missions engaged in the survey process?  Do other targeted audiences find DIMS useful to them? Cost-effectiveness  Are the costs for DIMS survey reasonable and within the norms for similar survey projects?  Is DIMS an effective use of USAID resources to improve policy, strategy and project design? In addition, the evaluation team shall provide recommendations, related to each of the questions above, about how to improve the next iteration of the project. Methodology USAID is looking for creative suggestions regarding this evaluation and the evaluation team should propose a methodology for carrying out the work. The evaluation team may propose data collection methods in addition to these suggestions: 60 DIMS Evaluation  Country visits: The evaluation team shall make country visits to two to three countries in which USAID funds DIMS. During these field visits, evaluators may conduct interviews with the USAID Mission, host country counterparts, civil society, media, development partners and academia and consult other sources of information as needed.  Web-based surveys of DIMS stakeholders.  Blind peer reviews of the DIMS survey design and products by renowned experts.  Review of the DIMS publications to assess their quality and utility.  Review of DIMS internal documents such as survey questionnaires and sampling methodology documents, reports on capacity building, survey dissemination and outreach events, etc.  Review of USAID Mission documents to review effects of DIMS data and findings.  Review of civil society reports, campaigns and activities to review effects of DIMS data and findings.  Review of publications not funded by DIMS (i.e., academic articles) that rely on DIMS data.  Interviews with select respondents in national and international research centers and institutes to assess the value, and use of DIMS data, results and publications. Existing Data and Resources The evaluation team will have access to DIMS datasets from 2006, 2008 and 2010. Related analytical documents developed by Vanderbilt/LAPOP are available on its website (http://www.vanderbilt.edu/lapop) and on USAID’s Development Experience Clearinghouse. USAID and Vanderbilt/LAPOP will provide access to information and documentation regarding data sampling and survey methods, survey management processes and other related documents as needed by the evaluation team in order to evaluate all issues and questions of this evaluation. USAID will also facilitate the evaluation team’s communication with Mission staff, and Vanderbilt/LAPOP will do the same for its own staff and local subcontractors. Gender Integration It is the policy of USAID to integrate gender in all projects and programs across sectors. As such, the evaluation should consider how the DIMS surveys take gender into account in designing and carrying out the surveys as well as in the analytical reports produced for USAID. A gender analysis is required for all USAID projects. For further information on gender integration and analysis see section IV.3 and http://www.usaid.gov/policy/ads/200/201sab.pdf. 3. EVALUATION PRODUCTS Deliverables and Timeline 1. Evaluation plan 2. Presentation of the draft evaluation findings to USAID 3. Draft evaluation report 4. Final evaluation report Within 10 days of receiving this Scope of Work, the evaluation implementer shall submit a proposed list of key personnel for this evaluation, along with their resumes or CVs. USAID must approve the Evaluation Team Leader. 61 Within three weeks of receiving this Scope of Work, the evaluation team shall produce an evaluation plan that details its understanding of the assignment, the evaluation methodology, the evaluation timeline and an evaluation budget. USAID must approve the evaluation plan. The evaluation team shall submit a draft report to USAID within 30 days of completing data gathering and field work. USAID will provide comments to the consultant within two weeks of the submission of the draft evaluation report. The evaluation team will present its draft findings to USAID. Ideally this will be conducted in person, but may be conducted via video conference call if the evaluation team members are not located near Washington, DC. The evaluation team will submit the final report to USAID within two weeks after receiving USAID’s comments. USAID will make the evaluation report public and will submit it to the Development Experience Clearinghouse (http://dec.usaid.gov). The evaluation team is strongly encouraged to review methods and evaluation reports of other cross-national survey research projects supported by development or academic institutions, such as the AfroBarometer or World Bank supported surveys, when designing the DIMS project evaluation. The evaluation design and the final evaluation report must fully comply with USAID’s Evaluation Policy (www.usaid.gov/evaluation). Final evaluation report The final report shall contain a maximum of thirty pages (not including annexes) and shall answer each of the evaluation questions above, with clear references to sources. It shall include findings and recommendations for improvements. A 3-5 page executive summary shall form part of the final report. The format for the final evaluation report should be as follows: 1. Table of contents 2. Executive summary – Concisely state the most salient findings and recommendations 3. Introduction 4. Background of DIMS and evaluation purpose 5. Evaluation methodology 6. Evaluation findings 7. Issues – Provide a list of key technical or administrative issues related to the implementation of the DIMS surveys 8. Recommendations for improvements to the next iteration of the project, including suggestions for how evaluation criteria could be included in the next project design 9. Annexes – Include separate annexes for the evaluation scope of work; evaluation work plan; evaluation design and tools, including any questionnaires or interview questions; references and succinct lists of all sources of data and information; and any other additional information judged pertinent by the evaluation team. All deliverables should be submitted in electronic form to the Contracting Officer’s Representative for the EDGE-IE Task Order in DCHA/DRG. The final evaluation report should be submitted electronically to USAID in Microsoft Word format, in addition to five printed and bound copies. 62 DIMS Evaluation 4. KEY PERSONNEL / EVALUATION TEAM COMPOSITION: The Evaluation Team will consist of a team leader and at least two technical experts. USAID must approve the Evaluation Team Leader. Per USAID’s Evaluation Policy all evaluation team members are required to disclose any conflict of interest. 1. Team Leader/Senior Evaluation Specialist – He/she should be highly experienced in conducting cross-national surveys and evaluations of such projects, using quantitative and qualitative methods. Ideally he/she would have been involved in conducting surveys similar to AmericasBarometer in order to best appreciate appropriate methods for its evaluation. He/she should have a post graduate degree in political science, public policy, or another social science field. Experience working in developing country contexts and with donors, is preferable. 2. Technical Experts – Between the two experts the following requirements should be covered:  The technical experts should be experienced in conducting project evaluations in the international development context, including quantitative and qualitative methods.  At least one person in the team should have expertise as documented through publication records on issues of democratization and other focal areas of the DIMS surveys. At least one person should have a record of scientific publications that include the use of survey results and advanced statistical methods.  All team members should have advanced academic degrees in the social sciences (i.e., political science, public policy, economics, etc.).  The team must also have access to Spanish speakers, or to French speakers if they plan to visit Haiti, during field work in the region. 5. EVALUATION MANAGEMENT Logistical Support DCHA/DRG, in coordination with the LAC Bureau, will assist the evaluation team in making the necessary arrangements to facilitate their work in Washington, DC, and at the USAID Missions or with Vanderbilt/LAPOP. The evaluation contractor is responsible for arranging travel logistics (e.g., plane tickets, vehicle rental and drivers, translators, hotel accommodations) as needed for site visits and field work. USAID sponsors of the evaluation in DCHA/DRG and LAC will be available to the team for consultations regarding sources and technical issues before and during the evaluation process. Period of Performance The period of performance for this evaluation is April 12 – July 15, 2012. Budget USAID should receive a proposed budget for this evaluation within three work weeks of receiving this Scope of Work as part of the evaluation plan. 63 APPENDIX B.PERSONS CONTACTED COLOMBIA Centro Nacional de Consultoria (CNC) Cristina Querubín Borrero, Staff Carlos Lemoine, President Cifras y Conceptos César Caballero Reinoso, Gerente Department of Political Science at the Pontificia Universidad de Javeriana – Bogotá Andrés Casas-Casas, Profesor Grupo de Gobierno Departamento Nacional de Planeación (DNP) Paloma Solano, Asesor en Sujetos Congresos Instituto de Ciencia Política Andrea Benavides Romero, Directora de Proyectos Instituto Nacional Democrata (NDI) Francisco Herrero, Director para Colombia International Republican Institute (IRI) Gabriela Serrano, Resident Country Director – Colombia Carlos Muñoz, Resident Program Officer – Colombia Misión de Observación Electoral (MOE) Alejandra Barrios Cabrera, Directora Nacional Felipe Jimenez, Staff Observatorio de la Democracia (Universidad de Los Andes) Felipe Botero, Associate Professor 64 DIMS Evaluation Juan Carlos Rodríguez Raga, Associate Professor and LAPOP Co-Team Leader Miguel García, Assistant Professor and LAPOP Co-Team Leader Gabriel Camargo, LAPOP Research/Program Assistant Natalia Garbiras, LAPOP Research/Program Assistant Partido Conservador Alexander Ruiz H., General Director for Race, Ethnicity, and Cooperation Partido MIRA Luis Olave Valencia, Director Nacional Elizabeth Bonilla, Staff Nicolas Falla, Staff Ana Belsú Rodríguez, Secretaria General del Partido USAID/Colombia Donald Chisholm, Deputy Director Office of Democracy, Human Rights and Democracy Lucy Malo Rodríguez, Gerente Programa de Elecciones y Procesos Políticos Catalina, Program Officer Edwin, Program Officer Dominican Republic University of Tennessee Jana Morgan, Assistant Professor, LAPOP Partner Ecuador Universidad San Francisco de Quito Daniel Montalvo, Assistant Professor, LAPOP Partner/Data Collector El Salvador Florida International University Jose Miguel Cruz, Visiting Professor, LAPOP Partner Guatemala Wichita State University Dinorah Azpuru, Associate Professor, LAPOP Partner 65 Haiti Iowa State University Amy Erica Smith, Assistant Professor, LAPOP Partner Honduras Borge y Asociados Victor Borge Gonzalez, Regional General Manager Claudia Canton Schiffman, Regional Marketing Director Centro de Investigación y Promoción de los Derechos Humanos en Honduras (CIPRODEH) Wilfredo Mendez, Executive Director Hagamos Democracia/FOPRIDEH Rolando Bu, Director General Ronald Fiallos, Project Coordinator Elgardo Vargas, Project Coordinator Jimena Carias, Communications Universidad Nacional Autonoma de Honduras (UNAH) Belinda Flores de Mendoza, Dean of Economics USAID/Honduras Denia Chavez, Activity Manager Sonia Zacapa, Program Officer Gabriela Leva, Program Officer Brioni James, Democracy & Governance Officer Jamaica University of West Indies at Mona Balford Lewis, Assistant Lecturer, LAPOP Partner Mexico Data Opinion Publica y Mercados Pablo Paras, President and Founder, LAPOP Partner 66 DIMS Evaluation Nicaragua University of North Texas John Booth, Regents Professor, LAPOP Partner Panama Central Michigan University Orlando Perez, Professor and Chairperson, LAPOP Partner Peru Instituto de Esudios Peruanos Patricia Zarate, Researcher, LAPOP Partner/Data Collector United States USAID/W Christopher Cushing, Program Officer LAC/RSD Eric Kite, Team Leader, Democracy & Human Rights LAC/RSD Lacy Kilraine, Program Officer LAC/RSD Lawrence Rubey, Director, Office of Sustainable Development LAC/RSD Vanessa Reilly, Democracy and Governance Specialist (DIMS AOR) LAC/RSD SOUTHCOM Scott Taylor, USAID Liaison National Endowment for Democracy Miriam Kornblith, Director – Latin America & the Caribbean Aimel Rios Wong, Program Assistant – Latin America & the Caribbean USDOS Andrew Stein, Latin American Analyst, OPN LAPOP John Geer, Chair of Political Science Mitch Seligson, Founder and Director Liz Zechmeister, Associate Professor and Associate Director Fernanda Boidi, Coordinator of Field Operations 67 Georgina Pizzolitto, Coordinator of Special Samples Rubi Arana, Data Subscription and Repository Manager, Webmaster Abby Cordova, Coordinator of CARSI Cindy Kam, Professor 68 DIMS Evaluation APPENDIX C.DATA COLLECTION TOOLS Latin America Scholars Questionnaire 69 70 DIMS Evaluation 71 72 DIMS Evaluation 73 74 DIMS Evaluation 75 76 DIMS Evaluation 77 Survey template 2 78 DIMS Evaluation 79 80 DIMS Evaluation 81 82 DIMS Evaluation 83 84 DIMS Evaluation 85 86 DIMS Evaluation 87 88 DIMS Evaluation 89 90 DIMS Evaluation Interview Script for NGOs We are contacting you as part of a United States Agency for International Development (USAID) evaluation of the AmericasBarometer (AB, also known as the Democratic Indicators Monitoring Survey, DIMS), conducted since 2006 by Vanderbilt University’s, Latin American Public Opinion Project (LAPOP). We are reaching out to NGO’s who may use the AmericasBarometer data in their work to determine the extent to which NGO’s are familiar with the survey, how and to what effect they use it, their opinions of the quality and usefulness of the survey, and their suggestions future improvements. Possible Questions: 1. How familiar are you with the AmericasBarometer surveys? a. How did you first learn about them? b. How frequently do you manipulate the data internally? c. How important are the reports that LAPOP publishes? d. How does AB compare to other survey data in your country that you might use? 2. In what specific ways do you make use of the data and the reports that are provided by the AmericasBarometer? a. Which themes in the surveys are most useful to you? b. Are there any themes on which you would like to see more AB questions? c. Have the AmericasBarometer surveys, data, and reports influenced the discourse among civil society groups in the region? d. How have the surveys helped you effect positive changes? 3. How accessible are the AmericasBarometer data and reports to you? a. Have you had any difficulties getting access to the data? b. How responsive has LAPOP been to questions your questions about the data and reports? 4. What influence do you have, if any, over the content of the surveys? Has LAPOP been responsive to your suggestions for questions in future surveys? 5. Do you perceive any discernible biases – political, ideological, or otherwise – in the information that is provided by the AmericasBarometer? 6. In your view, how important is the continuation of the AB survey? 7. How could the AmericasBarometer project better serve the goals of NGO’s in Latin America? 91 Interview Script for Survey Firms We are contacting you as part of a USAID evaluation of the Democratic Indicators Monitoring Survey (DIMS), conducted since 2006 by the Vanderbilt University’s, Latin American Public Opinion Project (LAPOP). The formal name of LAC/RSD’s agreement is Democracy Indicators Monitoring Survey, DIMS, but is better known as AmericasBarometer (AB). We are contacting all of the survey firms that conducted one the DIMS/ AB surveys for LAPOP in 2012 to ask you about your experiences working on this project. LAPOP Director, Dr. Mitch Seligson, identified you as the primary AB contact at your firm. We thank you very much for agreeing to speak with us. It should take no more than 45 minutes of so of your time. We’d like to assure you at the outset that your comments will be held in strict confidence and not associated or identified in any way with you by name, or with your country, firm or organization. 1. Could we start by asking you to tell us a little bit about your firm – how long have you been conducting public opinion polling in the country, how large the firm is, and how many surveys does the firm typically conduct each year? How many of these surveys are governance/democracy related, how many are for USAID or other donors, and how many for DIMS/AmericasBarometer in your country and perhaps other countries? 2. Let’s focus now on your work with LAPOP/AB. How did you come to be involved in the AB data collection? Were you approached by the LAPOP partner or by LAPOP Central in the US, did you submit a bid in a competitive process, or what exactly? 3. Could you tell us about your relationship with the LAPOP partner in terms of carrying out the surveys. How much do you work with the partner in developing the questions and sampling and other technical procedures? Is it a collaborative process or one where your firm mostly carries out the instructions given to you by LAPOP? How about LAPOP central in the US? How much contact do you have and how much technical assistance is provided by the Vanderbilt staff? How responsive have they been to any issues, problems or concerns you have raised? Were you able to attend the Miami conference with the LAPOP partners where the questionnaire and procedures were finalized in 2011? Could you tell us about your experiences there and how much input you felt you had on the process? Can you tell us any specific ways that you or the firm have suggested changes in the procedures used in order to improve the data collection process in your country? How satisfied in general are you with your relationship with a) the LAPOP partner in your country; and b) LAPOP Central in the United States? Is there anything you could suggest to change the ways that you interact with these groups to improve the process? [POSSIBLE ASK ABOUT USAID MISSION INVOLVEMENT IN THE PROCESS, CONTACT WITH USAID, AND SATISFACTION, THOUGH THIS IS LIKELY TO BE RARE FOR THE FIRMS] 4. Now we’d like to ask about some of the technical details of the survey process. a. SAMPLING: When you carry out the LAPOP/AB surveys, what kind of sampling procedures do you use, and are they the same procedures that you use for other national surveys? Better/Worse in your view than the ones you use for other surveys? 92 DIMS Evaluation Did LAPOP partner and/or LAPOP Central provide adequate technical assistance in carrying out the sampling for the study? Could you tell us how you selected members of households to be interviews? (Kish, next birthday, household quota (official LAPOP policy). What would you think of the feasibility of any alternatives, like using the Kish or next birthday method? Other suggestions for improvement? b. INTERVIEWER RECRUITMENT AND TRAINING: When you carry out the LAPOP/AB surveys, how are the interviewers recruited? Who are they? Your normal pool of people or special skills required? Do you aim for gender balance among the interviewer pool, ethnic/indigenous populations, language skills, etc.? How are the interviewers trained – by you or LAPOP partners? Satisfied with the technical assistance you are provided by LAPOP in this regard? Suggestions for improvement? c. PRE-TEST AND DATA COLLECTION: Could you please tell us a little about the pre-testing, if any, that went into the process. Are you required by LAPOP to conduct a pre-test, how much supervision and technical assistance was provided, and how useful was it to the entire data collection process? How are the interviewers instructed to handle instances where no one is at home? How often do the interviewers return to the same household before other household is substituted? How are the interviewers instructed to handle non-response during the survey itself? Are there probes used to move “no opinion/don’t know/refuse to answer” respondents? Please tell us a little about the use of the PDA handheld devices that LAPOP is using for data collection – are the interviewers and is your firm comfortable with them? Have they led to any significant changes in the quality of the data collection process in your view? (IF NOT USING PDAS, tell us a bit more about the data entry process and how the data is delivered to LAPOP) Suggestions for improvement? 5. Now we’d like to ask about how being involved in the AB/LAPOP has influenced your company/organization. To what extent has working on the AmericasBarometer surveys influenced the quality of your other survey work? [ASK FOR SPECIFIC EXAMPLES: better sampling procedures, better training, know more about technical aspects of surveys, etc.] To what extent has working for AB enhanced your visibility and reputation among the survey research/polling community in your country? [DITTO] To what extent has AB involvement allowed your company/organization to grow? Has AB involvement allowed your company/organization to be more competitive in securing contracts with other democracy/governance related surveys for USAID or other international donors? [SPECIFICS needed] Has AB involvement increased the number of women who work for your organization, and the stature of the women you have on your staff? 6. OVERALL ASSESSMENTS 93 How would you compare the quality of the DIMS AB survey with other public opinion surveys you have done for other firms? Compared to other clients for whom you have done public opinion surveys, would you say that the standards set by LAPOP were higher or lower than those demanded by other clients? Do you have any other suggestions for improving the AB surveys in the future in your country? How likely is it that you will seek to continue your partnership with LAPOP/AB in the 2014 round of interviews and beyond? 94 DIMS Evaluation Interview Script for LAPOP Partners We are contacting you as part of a USAID evaluation of the Democratic Indicators Monitoring Survey (DIMS), conducted since 2006 by the Vanderbilt University’s, Latin American Public Opinion Project (LAPOP). The formal name of LAC/RSD’s agreement is Democracy Indicators Monitoring Survey, DIMS, but is better known as AmericasBarometer (AB). We are contacting all of the local partners involved in the DIMS/ AB surveys for LAPOP in 2012 to ask you about your experiences working on this project. LAPOP Director, Dr. Mitch Seligson, identified you as the primary local partner for [COUNTRY]. We thank you very much for agreeing to speak with us. It should take no more than 45 minutes of so of your time. We’d like to assure you at the outset that your comments will be held in strict confidence and not associated or identified in any way with you by name, or with your country, team or organization. 1. Could we start by asking you to tell us a little bit about yourself and how you happened to get involved in the LAPOP project? [Did you study with Mitch at Vanderbilt or Pittsburgh, or with someone else who studied with him?] What kinds of experiences with conducting public opinion surveys had you had before working with LAPOP? [How many of those surveys were governance/democracy related, how many are for USAID or other donors] What kinds of other survey research projects are you currently involved in? Do you happen to know if you competed with other people or organizations to be selected to join the LAPOP project? Why do you think you were chosen? 2. Could you tell us about your role in the AB surveys? What are your responsibilities? Do you conduct the surveys with your own staff or just serve as a link between LAPOP central and the firm conducting the surveys? What role have you played in the writing and dissemination of either the country or regional reports that have been produced from the AB data? How have you or your organization carried out the task of writing these reports? Do you have sufficient resources to carry out these tasks effectively? Are you satisfied with the guidance and technical assistance that you have been given by LAPOP for carrying out these tasks? 3. Could you tell us about your relationship with LAPOP central in terms of carrying out the surveys. How much do you work with Mitch and Liz and the others at Vanderbilt in developing the questions and sampling and other technical procedures? Do you feel you have enough say in the content of the core questions for the survey? Do you feel you have enough say in the content of the country-specific portion of the survey? How responsive would you say that LAPOP central has they been to any issues, problems or concerns you have raised? Were you able to attend the Miami conference with the LAPOP partners where the questionnaire and procedures were finalized in 2011? Could you tell us about your experiences there and how much input you felt you had on the process? 95 Can you tell us any specific ways that you suggested changes in the procedures used in order to improve the data collection process in your country? How satisfied in general are you with your relationship LAPOP Central in the United States? Is there anything you could suggest to change the ways that you interact with these groups to improve the process? 4. How much contact have you had with the USAID mission in your country, and specifically the Democracy/Rights/Governance officers who liaise with the LAPOP project? Could you tell us in what ways these individuals have helped your work, and what ways they perhaps have been less helpful? How satisfied in general are you with your relationship with the USAID mission in your country? Is there anything you could suggest to change the ways that you interact with these individuals to improve the process? 5. Now we’d like to ask about the data collection process. Do you contract with a survey company in carrying out the data collection, or do you hire your own interviewers and handle the data collection yourself? Please tell us how this decision came about. [IF USE SURVEY COMPANY]: How did you choose this particular firm? Was there a competitive bidding process? How do you negotiate the contract – yourself or through LAPOP Central? How satisfied in general are you with the survey firm that you have selected? Is there anything you would want to change for the next round of interviewing? [IF USE OWN INTERVIEWERS/OWN PROCESS]: Please tell us what you see are the strengths and weaknesses of the process that you have instituted – do you think the data collection process would be improved if a commercial firm, NGO or some other entity worked with you in the process? 6. Now we’d like to ask about some of the technical details of the survey process. a. SAMPLING: How did you arrive at the sampling procedures for the LAPOP survey? Was it based on your input, on LAPOP Central, or on the survey research firm [if applicable], or some combination of all of these? Are these procedures Better/Worse in your view than the ones you may be using for other surveys? Did LAPOP Central provide adequate technical assistance in carrying out the sampling for the study? Could you tell us how you [and the survey company] selected members of households to be interviews? (Kish, next birthday, household quota (official LAPOP policy). What would you think of the feasibility of any alternatives, like using the Kish or next birthday method? Other suggestions for improvement? b. INTERVIEWER RECRUITMENT AND TRAINING: When you carry out the LAPOP/AB surveys, how are the interviewers recruited? Who are they? Your [the survey company’s] normal pool of people or special skills required? Do you aim for gender balance among the interviewer pool, ethnic/indigenous populations, language skills, etc.? How are the interviewers trained – by you or LAPOP partners? Satisfied with the technical assistance you are provided by LAPOP in this regard? Suggestions for improvement? 96 DIMS Evaluation c. PRE-TEST AND DATA COLLECTION: Could you please tell us a little about the pre-testing, if any, that went into the process. Are you required by LAPOP to conduct a pre-test, how much supervision and technical assistance was provided, and how useful was it to the entire data collection process? How are the interviewers instructed to handle instances where no one is at home? How often do the interviewers return to the same household before other household is substituted? How are the interviewers instructed to handle non-response during the survey itself? Are there probes used to move “no opinion/don’t know/refuse to answer” respondents? Please tell us a little about the use of the PDA handheld devices that LAPOP is using for data collection – are the interviewers and is your firm comfortable with them? Have they led to any significant changes in the quality of the data collection process in your view? (IF NOT USING PDAS, tell us a bit more about the data entry process and how the data is delivered to LAPOP) Suggestions for improvement? 7. Now we’d like to ask about how being involved in the AB/LAPOP has influenced you and your organization. To what extent has working on the AmericasBarometer surveys influenced the quality of your other survey work? [ASK FOR SPECIFIC EXAMPLES: better sampling procedures, better training, know more about technical aspects of surveys, etc.] To what extent has working for AB enhanced your visibility and reputation among the survey research/polling community in your country? [DITTO] To what extent has AB involvement enhanced your visibility and reputation in your university or among the academic community? Have you been able to secure additional academic funding for other projects based on your AB involvement, or secured additional academic/consulting/policy-oriented work for other governmental agencies or international donors?[SPECIFICS needed] Has AB involvement enhanced the training you have been able to offer to graduate students in the field of public opinion, survey research, democratization, or other areas of social science? Has AB involvement increased the number of women who work for your organization, and the stature of the women you have on your staff? 8. OVERALL ASSESSMENTS How would you compare the quality of the DIMS AB survey with other public opinion surveys you have done for other firms? Compared to other clients for whom you have done public opinion surveys, would you say that the standards set by LAPOP were higher or lower than those demanded by other clients? Do you have any other suggestions for improving the AB surveys in the future in your country? How likely is it that you will seek to continue your partnership with LAPOP/AB in the 2014 round of interviews and beyond? 97 APPENDIX D.INTENSIVE CASE STUDIES As outlined in the evaluation report, two of the 16 DIMS country programs were selected by the evaluation team for intensive case studies. These countries were visited by two-person teams for the purpose of conducting face-to-face interviews with a range of DIMS stakeholders: Mission staff, Embassy staff, LAPOP country leaders, local data collection firms, representatives of local government, NGOs and academics. The selection of these cases, conducted on the basis of program documents and interviews with USAID/W and LAPOP, was implemented so as to allow the team to visit one high performing and one relatively low-performing country. A comparison of the two programs was intended to provide insights into the possibilities and limitations of the LAPOP structure as it applies to the DIMS program. These case studies detail findings about the quality and utilization of the survey, the DIMS contributions to capacity building, and overall satisfaction with the DIMS. While this information has been incorporated into the body of the evaluation report, we present this appendix for a more robust discussion of the site visits. COLOMBIA Colombia was selected as an example of a strong, well-functioning LAPOP program given its strong support (financial and otherwise) from the USAID Mission, its well-established team of local partners, and the extensive utilization of its data by missions, NGOs, local academics, and, increasingly, by government. In addition to the biennial DIMS survey conducted in even numbered years supported by funding from the USAID Cooperative Agreement, DIMS surveys also were conducted in 2007, 2009 and 2011 with funding from the USAID Mission. Evaluation team members visited Bogota from July 4-6, 2012, interviewing 26 people from 12 different local entities. The case of Colombia represents, in many ways, the “gold standard” of how LAPOP and USAID operate through DIMS: the survey is regarded by many as perhaps the best technical survey instrument in the country, the cooperation between LAPOP and USAID has generated significant local academic capacity in the form of the Universidad de Los Andes’ Observatorio de la Democracia (LAPOP’s local presence in Colombia), and the survey has been utilized by a number of local entities including the USAID Mission, government agencies, local NGO’s, and academics. Overall, interviewees reported high levels of satisfaction with the DIMS survey, the USAID Mission, and LAPOP. Quality The overarching sentiment from the NGOs, the USAID Mission, government agencies, local academics, and survey firms was that the DIMS survey is of high quality and compares very favorably to other surveys conducted in Colombia and other regional surveys (i.e., the Latinobarometer). The questionnaire is thorough and covers questions that allow regional comparisons as well as country￾specific analyses, and nearly everyone praised the regionally comparative nature of the DIMS survey (particularly as compared to other regional surveys). Unlike other surveys conducted in Colombia, DIMS includes rural areas that are more expensive and more challenging to reach, yielding a better sample than most other surveys. One interviewee noted that the DIMS is one of the best in Colombia for producing a national sample. Several interviewees praised the time-serial nature of the survey saying it is very beneficial for tracking changes in public opinion. The comparative core questions also were viewed favorably because they not 98 DIMS Evaluation only provide information about Colombia but show how Colombia compares to other countries in Latin America. One respondent summed up the benefits of the DIMS as being particularly useful because “policy groups need rigorous academic information to inform public policy.” A few concerns were raised about the survey instrument. First, several people noted that the questionnaire is too long, leading to respondent fatigue and thus reduced quality. Some people saw repetition in certain questions, the consolidation of which could shorten the survey or make room for other questions. One interviewee saw the questionnaire more as a “patchwork quilt” of questions than a coherent theme-driven instrument. Second, many users of the data highlighted the most useful questions as being those dealing with the country’s political institutions—presidency, congress, and judiciary—whereas those about political culture were less informative. One group that focuses on congressional issues noted that more questions related to public perceptions of the national legislature would be very helpful as the congress is a very important political institution in Colombia. Another organization suggested that they would benefit from more questions about political knowledge in the country and region. The questions on corruption, citizen participation, and party ideology were also popular among interviewees. One interviewee lamented the limited data on women’s political participation. A final concern was the number of questions allocated to the country. The survey questionnaire includes both “core” questions that are asked across all DIMS countries and “country-specific” questions that the LAPOP team leaders and local USAID missions derive together. The Mission in Colombia noted that they were only allocated 10 questions for the 2012 survey and really would have preferred 15-20. Utilization DIMS has been used by the USAID Mission, government agencies, local NGOs, and Colombian academics in a myriad of ways. According to the Observatorio de la Democracy, 44 Colombian publications have used the DIMS data between 2004 and 2012. The 2011 DIMS survey that oversampled the AfroColombian community was very popular by all users because it was one of the first surveys of the minority group’s public opinions. The Colombian USAID Mission is one of the primary users of the data and regularly uses the data in their Performance Management Plans (PMP). Their AfroColombian unit has used the survey to inform their programming related to AfroColombian political participation, with a particular interest in whether AfroColombians feel that they can not only participate in elections but run for office themselves. Other areas of interest for the Mission have included human rights, elections and electoral processes, and the justice system. They have used the survey to track perceptions of the declining credibility of the justice sector to generate greater support for programs in this area. Political parties in Colombia have also begun using the DIMS results in recent years. The two political parties that we interviewed have used the 2011 AfroColombian survey results to inform their programming and policymaking toward the minority AfroColombian community. Partido MIRA noted that the survey has helped them better understand inequality in the country. The Partido Conservador used the data to help justify the need to create a party leadership position dedicated to AfroColombian issues. The National Department of Planning (DNP) has become very interested in DIMS over the past year. They have several projects underway for which they have found the DIMS useful. Specifically, they have been interested in public perceptions of government corruption and the national congress, perceptions of presidents, and better understanding of public participation in politics. They have hosted a series of meetings over the past few months with government officials, such as representatives from the Ministry of the Interior and members of congress, where a LAPOP representative has presented some of the DIMS findings about various topics. They have plans to do more of this over the second half of this year 99 as it has been quite popular and they have several staff members interested in pursuing these efforts further. Local Colombian NGOs have also found uses for the DIMS. The Instituto de Ciencias Políticas used recent DIMS reports to inform their youth programs aimed at promoting democratic citizenship. They have also set up a legislative observatory that follows legislative bills and has used the survey results to compare that legislation to civil society’s preferences on issues such as health, economic development, and justice. The International Republican Institute has used the data in their efforts to help Colombian political parties better understand how citizens view them and to inform party platforms. The National Democratic Institute has used the survey to inform their work with Colombian political parties that aims to improve the political party transparency inclusion. DIMS has also informed the research and teaching of Colombian academics. Those we interviewed said that they use it in their own papers, articles, and books. Several also use it in their teaching, assigning papers for students to write with the data. The academic uses have been greatest among scholars at the Universidad de Los Andes, Pontificia Universidad de Javeriana – Bogotá, and Universidad Sergio Arboleda, where scholars have the training and skills needed to use survey data. Utilization of the DIMS data has primarily been through the annual report produced by LAPOP and USAID. Except for academics, most other users said that their familiarity and use of the data was from the printed reports rather than analysis of the raw data using statistical software or the LAPOP data website. All interviewees also praised the Observatorio de la Democracia for being readily willing and able to provide additional analyses of questions that they wanted to use in their programming. Capacity Building The DIMS survey has helped to build local capacity in Colombia, particularly in academic institutions and survey firms. The USAID Mission in Colombia provided funds to create the Observatorio de la Democracia (OD) at the Universidad de Los Andes with a goal to “transfer capacity to the country” (Interview with the OD). The OD contracts with the survey firm, Centro Nacional de Consultoria (CNC) to conduct the DIMS survey, and is currently staffed by two full-time professors, three graduate research assistants, and four undergraduate student volunteers. The OD maintains an active website with information on the DIMS and a location for users to request the data (www.obsdemocracia.org). LAPOP’s presence at Los Andes has encouraged more systematic, data-driven research in social science departments in Bogotá. As a result, one academic at an institution outside of Los Andes has been asked to lead the next round of the World Values Survey in Colombia. This said, several people noted that the survey is still used predominantly by academics at Los Andes. In part, this is a function of the relatively small (although growing) numbers of academics in the country interested in survey research and possessing the statistical skills to manipulate the data. The survey has also helped build the capacity and reputation of the CNC survey firm that conducts the survey. The CNC noted that one of their primary reasons for doing the DIMS is because of its strong reputation and the benefit that being associated with LAPOP and the DIMS brings to their company. Dissemination of the DIMS Results The primary area where concerns were raised about DIMS was its dissemination. Although everyone interviewed was familiar with the project and has used the data in some way, nearly all lamented the limited reach of the DIMS. Everyone, from the LAPOP-Colombia staff, to the USAID Mission, to the survey firm that conducts the DIMS, to outside survey firms and academics, agreed that DIMS findings are not disseminated widely enough, especially through the media or among elected public officials. This appears to be changing, particularly in the past year with the release of the 2011 DIMS report on AfroColombians, but more efforts are needed in this area. One person said that the problem is that the 100 DIMS Evaluation survey makes a “splash” during the week it is released and a series of press conferences held, but that the splash quickly dissipates and little effort is made to disseminate the data to the media thereafter. Among dissemination concerns, several individuals argued that dissemination efforts were too limited to Bogotá and not sufficiently focused on areas outside of the capital. Several people also noted that the knowledge of the survey tends to be limited to the academic world, and more specifically, to Los Andes. Many of the people familiar with the survey knew about it because of a personal connection to Los Andes or the LAPOP staff. A smaller number learned about DIMS by receiving a mailing about it or attending a recent dissemination event. Even within the USAID Mission, the Mission staff felt that they do not use the data as much as they should. When they have held events to disseminate information within the Mission and Embassy, turnout often is low; at one event, only ten of thirty invitees attended. Most of those interviewed expressed satisfaction with the DIMS annual report. One person noted that the reports are very easy to understand, and another praised them for being technically sound and a serious publication. One person noted, however, that the reports are too large and technical, making it difficult to reach important outlets such as the media, members of congress, non-academic NGO’s, etc. This interviewee suggested small fliers, press releases, and roundtables as better mechanisms for DIMS outreach. Almost everyone highlighted the need for more media-friendly publications above and beyond the report to stimulate ongoing interest in the DIMS surveys. The USAID Mission in Colombia agreed that dissemination is the weakest part of the DIMS and suggested that the Mission bore some of the responsibility for this. They have been talking with the LAPOP-Colombia staff about possibly providing a grant to pay for hiring a communications or marketing specialist to handle dissemination of the results more widely. One possibility in these regards would be to expand the role of on the survey firm, CNC, which indicated during an interview that they would be eager to do more media relations work for DIMS. Summary The intensive case study of the DIMS project in Colombia revealed that the DIMS survey in Colombia is of high quality, is familiar to multiple audiences, and has a strong reputation among those familiar with it. The USAID Mission expressed very strong support of the DIMS and its continuation. The DIMS report has been used in numerous ways by a variety of local organizations and has enhanced the capacity of the local survey firm and several universities in Bogota, especially Los Andes. The primary weakness of DIMS in Colombia is its limited dissemination, especially in the media, among elected government officials, and outside the capital. While LAPOP might do more with regard to dissemination beyond the initial release of the data, the Mission needs to take ownership of the data and play a more central role in dissemination, even if this means contracting to provide a media expert, perhaps in the Observatorio or through the CNC survey firm. HONDURAS Honduras was selected as a contrasting case, where the identification and development of local partners has been problematic and the use of the data by individuals and groups beyond the Mission has been limited. Evaluation team members visited Tegucigalpa from July 9-11, 2012, interviewing 12 people from 6 different local entities. Quality Honduras has been included in the DIMS surveys since the beginning of the Cooperative Agreement (CA), with field work carried out in 2006, 2008, 2010, and 2012. In addition to the funding provided by USAID/W through the CA with Vanderbilt University, the project received supplemental funding from the USAID Mission in 2010 and 2012. In 2012, the Mission paid for an oversample of 1300 cases focused on six targeted and six control municipalities. Additionally, the Mission provided funds to the Federación 101 de Organizaciones No Gubernamentales para el Desarrollo de Honduras (FORPRIDEH), an NGO which serves as a local LAPOP partner, for organizing DIMS dissemination workshops where the country results are presented to an audience of government officials, NGO and media representatives, academics and other interested parties. Field work for all four surveys has been provided by Borge y Asociados (Borge). This multi-national polling firm has its primary office in Costa Rica and regional offices in Guatemala, El Salvador, Nicaragua and Panama. Borge also conducts AmericasBarometer surveys for several other DIMS countries in Central America and Haiti and views LAPOP as “more of a partner than a client.” The personal relationship between Borge founder, Victor Borge, and Mitch Seligson of LAPOP goes back more than 20 years. Borge does not maintain a permanent office in Honduras but rents temporary space in Tegucigalpa during periods when field work is being conducted. Because of limited local capacity and the ease with which residents of neighboring countries can work in Honduras, DIMS data collection is led by senior staff from the Nicaraguan office. However, Borge maintains a pool of experienced Honduran interviewers, primarily women, who are supervised by the permanent staff. The sampling frame and questionnaire for Honduras are both provided to Borge by LAPOP central at Vanderbilt. The sampling frame is the LAPOP standard: namely, a multi-stage probability sample down to the level of household, where a quota sample is employed. However, in other surveys conducted in Honduras, Borge has used the ‘last birthday’ method for selecting the household member to interview. Borge estimates that the use of random, rather than quota samples at the household level for DIMS would add approximately 5% to the cost of conducting the survey in Honduras. Unlike some other countries, LAPOP has never had strong local partners based in Honduras. The local team leader for Honduras has usually been a graduate student or academic based in another country, often the United States. In part this is because Honduran universities do not have strong social science programs. Currently, there are few Honduran scholars who have sufficient training in research methods and statistics to partner with LAPOP. Accordingly, LAPOP has found it difficult to identify individuals who might be interested and able to take on the role of country leader. Since the premier university in Honduras does not have a Political Science Department (government courses are taught in Sociology and Law) and because the polling firm for Honduras is not local and does not employ local citizens except as interviewers, there also have been no obvious places for LAPOP to identify prospective students to enroll in the graduate program at Vanderbilt where they might develop the skills necessary to be a country leader. LAPOP did manage to identify and recruit one individual in 2011 who has enrolled in the Political Science PhD program at Vanderbilt with LAPOP funding. That individual currently serves as the team leader for Honduras, although, as a first year graduate student located in the United States, he does not yet possess the training and experience needed to be fully effective in the role. Moreover, given the dearth of Political Science programs in Honduras, it is not certain that this individual will be able to find work as an academic in Honduras and begin the task of developing local institutional capacity even when his studies are complete. Utilization Members of the DRG team at USAID/Honduras express considerable enthusiasm for DIMS, describing it as “our bread and butter.” The Mission uses DIMS data and publications for strategic planning, performance monitoring, and program evaluation purposes. In terms of planning, DIMS is used primarily to develop relevant and mutually-reinforcing DRG interventions, while targeting programming with regard to geography and critical beneficiary groups (women and youth). DIMS data are cited several times in the 2007 DRG Assessment to justify Mission programming. In addition to providing the empirical foundation strategic planning, DIMS is used to justify the creation of development hypotheses. 102 DIMS Evaluation DIMS was cited as a primary data source that led to program development or modification in three concrete cases. First, when the 2008 country report showed very low levels of youth engagement in politics, the Mission responded by designing an integrated program utilizing media campaigns, youth electoral observers, and youth participation in quick-counts to address the problem. Second, after the 2009 coup, the DRG office was asked to revisit the results framework (RF) to better align it with changing Honduran realities. Through a combination of using raw DIMS data and the preliminary report from 2010, the third component of the DRG RF was reworked to increase political knowledge and civic awareness. Third, DIMS data on the lack of public trust in political institutions led to a shift in the focus of USAID Rule of Law programming of 20 years duration. The Mission realized that capacity building for government institutions alone was insufficient to reduce corruption and increase utilization. On the basis of DIMS and other assessments, programming shifted from an institutional focus to one of citizen empowerment, including a $10 million NGO monitoring program intended to foster transparency. DIMS is also used by the Mission for performance management purposes. DIMS data comprise the principal performance indicators in one of the DRG RF Intermediate Results, and have done so at least since August 2010 (the first performance monitoring document available to the evaluation team). The DRG office has had success including customized, country-specific questions into DIMS surveys. The USAID activity manager has been overseeing the Honduras surveys since the CA began in 2006 and has reported successful negotiations with LAPOP over the content of the 2010 and 2012 DIMS questionnaires. For the 2012 survey, LAPOP proposed cutting several questions central to the Mission’s assessment activities, though they ultimately agreed to keep the questions. The Mission also lobbied for the inclusion of a variety of other country-specific questions and reports succeeding in deleting 15 core questions and adding 35 special country questions in the 2012 DIMS questionnaire. This is more than three times the number of country questions that the Colombia Mission reports they were allocated in 2012. DG Mission staff report a very close and productive relationship with LAPOP Central but much less of a relationship with the various individuals who have served as country partners or team leaders over the years. Their principal point of contact appears to be LAPOP Director, Dr. Mitch Seligson, and to a lesser extent the LAPOP business manager at Vanderbilt. In addition to what was described as “lots of back and forth” with Vanderbilt over the content of the questionnaire, the Mission also reports extensive discussions over the content of the country report, including not only big issues such as the amount of attention to give to different subjects but also smaller issues such as how graphs are presented and the level and accessibility of the language used in the report. The Political Section Counselor in the US Embassy is aware of DIMS and has access both to the biennial country report and to the shorter, more frequent, Insight reports distributed by LAPOP Central based on DIMS data. The Embassy uses these sources as general background data on the political context in Honduras and blends knowledge gained from these report with those from other sources in preparing briefings for the Ambassador and cables for visiting dignitaries. Utilization of DIMS by others actors in Honduras beyond USAID appears much more limited. Among NGOs, FORPRIDEH, an umbrella organization representing more than 80 national and international NGOs in Honduras, was contracted by USAID to distribute DIMS country reports to its member organization and beyond. FORPRIDEH reports that it held a number of regional meetings around the country to disseminate DIMS findings. Dissemination usually consisted of holding a conference where key DIMS findings were reported and then distributing copies of the printed country report to attendees to take back to their organizations. FROPIDEH also subscribes to the Insight series and distributes these selectively to member NGOs as well. Academic usage of DIMS in Honduras appears to be extremely limited. There are few academics interested in or able to conduct quantitative research on government and politics; nor do there appear 103 to be many ‘think tanks’ in Honduras where such individuals could find a home, given the absence of a Political Science Department at the central, National Autonomous University of Honduras (UNAH). An Interview with the Dean of Economics at UNAH indicated that she had no real knowledge about DIMS and was skeptical that it would be of interest within the university except possibly in courses on government and politics, which she could not identify but thought probably existed in Sociology or Law. Capacity Building Capacity building in Honduras clearly has been a challenge for LAPOP. While LAPOP can build on and enhance existing capacity in a country, it is hard for the organization to create capacity from nothing. This is a problem for LAPOP with respect to finding a local partner, which in most countries is an established Political Science professor in a local university. In Honduras, the absence of a Political Science department makes an institutional partnership with a university almost impossible. Similarly, while LAPOP can help increase the quality and capacity of an existing survey research firm such as Borge, which it has done in Costa Rica, it cannot create a survey research firm in a country such as Honduras where autonomous firms do not appear to exist. Even commercial firms, such as Monsanto, Coca Cola, Nestle, and McDonalds, use Borge and other regional organizations as their Honduras survey firm, which suggests that there may not even be any high quality commercial polling firms in Honduras with which LAPOP can reasonably partner and help develop a political polling capacity. Dissemination Media usage of DIMS is difficult to assess since none of the principal newspapers has online archives of back issues and there are no radio or television archives. However, USAID Mission staff, members of interviewed NGOs and the Political Section Counselor all maintain that the media regularly uses DIMS data and graphics, including one printed in the newspaper the day before our interviews with Mission staff took place. We have no independent evidence of the extent of media use, however, and given the fact that dissemination of survey data to media is conducted primarily through the handing out of country reports, we have questions about the extent to which media make use of the findings. Busy reporters are not likely to be willing and able to search and extract information from a 200+ page printed book, especially one in which data can be up to two years old. The survey research firm, Borge, highlighted dissemination as an area in need of additional strengthening. The Regional General Director indicates that Borge makes no effort to communicate with the media out of concern that the company’s reputation will suffer if its data are compared in the media to the often unscientific and frequently biased political polls promoted by self-interested political parties or government officials. Overall, the USAID Mission is highly satisfied with DIMS and strongly supports its continuation. When asked what it would mean to the Mission if USAID decided not to renew the DIMS CA, a high-ranking DRG official said they would “absolutely have to contract for [DIMS] ourselves.” Still, the Mission and others interviewed had a number of recommendations for improving and strengthening DIMS going forward. One recommendation everyone agreed on was the need to develop local partners. The contract with FORPRIDEH is a good start, and there is real hope that the current country leader and Vanderbilt student will quickly complete his PhD and return to Honduras to work as a local intermediary. Since LAPOP country leader is not a full time job, this will depend upon the job prospects this individual will face in Honduras. Another recommendation from the Mission staff was for LAPOP to provide greater freedom for missions to include country-specific questions in DIMS, reducing the number of common (i.e. cross￾country) core questions in the process. Reducing the overall length of the DIMS questionnaire was a recommendation made both by the Mission and by the survey research firm. Greater use of the Insight series, or some other mechanism through which to create timely, concise analyses relevant to the Honduran context was another recommendation. The challenges associated 104 DIMS Evaluation with relying on a large hard-copy report printed every two years were highlighted by many respondents. Targeting key stakeholders such as media and local politicians with short, targeted documents would go far towards increasing dissemination and utilization of survey findings. It should be noted, however, that multiplying the number of insight reports for all 16 DIMS countries would add greatly to LAPOP’s scope and to the overall cost of the CA. A related concern, and perhaps the most frequent recommendation we heard in Honduras, was to increase DIMS penetration of the Honduras media and its use by Honduran national and municipal governments. While there may be more that LAPOP can do in this regard, media and government relations in country would logically seem to be more of a Mission or Embassy responsibility than that of a survey research contractor. If and when LAPOP has a permanent country presence in Honduras, the country leader may be able to do more with the media and government, but we have not seen much evidence in any of the DIMS countries that local partners, most of whom are academics with little experience working with media or government, are very effective at media or government outreach. In order to strengthen the extent to which DIMS data penetrate local institutions, either the follow-on CA needs to be augmented so as to provide funding to LAPOP to hire public outreach directors in each country or missions need to take on this responsibility directly. Summary The DIMS survey in Honduras is technically of high quality, and LAPOP works closely with the Mission staff in designing the surveys and preparing the country reports. While the Mission makes considerable use of the printed country report for programming and evaluation, and there are claims of significant usage by the media, DIMS does not appears to be widely used by NGOs or local researchers. LAPOP and the Mission contract with FORPRIDEH to disseminate DIMS among a wide section of the NGO community, but the lack of a local team leader with whom NGOs can easily communicate and obtain assistance in analyzing and interpreting the data limits DIMS’ utility to them. Similarly, the scarcity of social scientists trained in survey research methods and statistics limits DIMS’ utility in the academic community. Borge appears to be a very capable survey research firm with long ties to LAPOP, but it has little local presence in Honduras and few connections to the Mission, NGOs or academics. An out-of￾country survey firm compounds the problems created by the lack of an experienced in-country LAPOP team leader. The general absence of human capital in Honduras is a principal obstacle to more effective use of DIMS. 105 APPENDIX E. CONFIDENTIAL EXPERT EVALUATIONS Two accomplished experts in cross-national survey research were asked for confidential evaluations of the DIMS/AmericasBarometer surveys. These reviewers are both well-known and respected in the field of public opinion research and have led or participated in dozens of surveys. Combined, the reviewers have several books and over 100 publications relating to survey research in various parts of the world. The text of their confidential reviews follows: EXTERNAL REVIEW 1 As requested, I have read the materials you sent me, and also conducted some research of my own on the web about the Latin American Public Opinion Project (LAPOP) and the AmericasBarometer. The evaluation that follows is organized around the five headings you requested feedback on in your 19 June 2012 email to me. By way of background, I am not a user of these surveys, nor am I a Latin American specialist. My own work is broadly comparative in focus and for **** years I was the chair of the ****. I therefore have some expertise in conducting/overseeing comparative surveys in a range of country settings, and familiarity with the issues surrounding the analysis of comparative data by a wide range of users with differing skill levels. 1. Technical and substantive quality The technical information supplied by the project is generally clear, wide-ranging and up to date (though see my comments about access under (4) below). There is an awareness in the project of the technical issues surrounding a large, comparative survey exercise of this kind. Overall, the documentation and the discussions contained therein were very good. (a) Sample size. The sample sizes for the individual country surveys are adequate for most purposes, at (usually) around 1,500 respondents. In a small number of cases the sample sizes are greater than this (presumably because of oversamples being drawn from particular subgroups within the electorate of interest). (b) Sample extraction. In countries that are economically under-developed, there are major challenges in drawing what would, in the established democracies, be considered a reliable sample. The documentation quite rightly considers this problem in depth. The sample design that is used (and I assume has been used for the surveys in the 1980s and 1990s, though that is not clear) is based on household selection, and within the household, based on a quota, rather than on a Kish grid or similar method. There are obviously some compromises here, since what is being extracted is not a true probability sample. However, the documentation does discuss this, and I think the case that is made for the approach that is used is generally convincing, namely, that this is the most practical approach in the context of the countries being surveyed. Applying weights should remedy any sampling biases, although—again as the documentation points out—official statistics are not always available and/or sufficiently reliable in order to generate appropriate weights. It would be useful to know what weights are used (age, gender, ethnicity, language for example?) and what weight factors are applied. A general rule of thumb is that a weight factor of more than two is placing too much pressure on the sample. In addition, do we know how much the survey differs from 106 DIMS Evaluation known parameters, for example, voting statistics (turnout, party support etc)? These are usual indicators of sample reliability. (c) Survey instrumentation. The surveys are appropriately pre-tested, an important consideration where there are multiple languages/ethnic groups to be surveyed. It would be useful to know if there is any back translation of the questionnaires. The arguments in favor of the use of hand held devices in order to deliver the surveys, and their preference ahead of CATI or CAPI techniques, is convincing. (d) Survey content. I am in less comfortable in evaluating the content of the surveys, since I am not a user/analyst of the data. I did however browse several of the questionnaires and they look to me to include a good range of questions covering such areas as economic expectations, social capital, views of democracy, law enforcement, political issues and evaluations of government. Others will be able to comment on how valuable these questions are for research. However, I was struck by two things. First, few of the questions seemed to be the familiar cross-national questions. That may be because these questions are covered in the LatinoBarometer or elsewhere. Second, few of the questions seemed to be the sort of tracking questions that usually make up a significant minority of questions within any survey. It would have been useful to have had access to the cumulative trend file to see what questions are asked on a continuing, longitudinal basis (see comments on access in (4) below). Summary: Overall, the project shows a strong appreciation of the technical issues surrounding comparative research in an economically under-developed region. The project has made a series of perfectly reasonable, practical compromises to ensure that the data quality is the best that can be achieved in the light of the resources at hand. A better evaluation of the content of the surveys would be possible if it I could see what the trend questions are; this, however, is apparently available only to ‘subscribers’. 2. Reputation among survey experts in the fields of democratization and /or Latin American politics While this is not my own field of research, we do have a Center for Latin American Studies at my university. I consulted the two political scientists on the staff, and they were familiar with the AmericasBarometer. While they did not use the surveys themselves, they cited, without prompting, a series of important articles that had been published from the data, in quality journals. Both of them also emphasized that the surveys had an important policy influence, and again cited some of the development studies work done in the area which had been based on the AmericasBarometer data. Within the ****, which is my own direct experience of cross-national data collection, our project committee was aware of the AmericasBarometer and it was discussed in passing on several occasions in formal planning meetings. ****. 3. Visibility and usage among both scholars and practitioners In talking to my Latin American studies colleagues, my impression is that the AmericasBarometer has good policy outreach. The importance of the surveys was stressed to me as providing a basic indicator of change, since official statistics are often highly unreliable in many of the countries in the region. The surveys are often used by policy makers, I was told, to monitor household composition, socioeconomic status, educational attainments, and a range of other basic measures. These indicators are, in turn, important in allocating resources for reducing social problems, and for the allocation of resources to conduct further research. 107 4. Accessibility including both its availability for use by scholars and the ease of using the data when it is obtained The project’s website is well designed, but regrettably not especially user friendly. I looked around the website and the documentation, and it appeared to be the case that country studies were freely available. However, another webpage contradicted this and said that only Panama 2004 was publicly accessible without charge; after 10 minutes I gave up trying to track the dataset down. Other documentation referred to ‘premium access’ which apparently requires a subscription, either by an individual or an institution. Yet more documentation referred to cumulative files from which trend analyses could be carried out but they too require some form of subscription. And one webpage refers to country studies being available only to those who live in the country in question. Overall, this appears to the outside observer as a complete and unnecessary muddle which inhibits scholars accessing the data for research. I found the policies about public use of the surveys confusing, and what is on the website does not appear to match the statements about ‘public good’ and ‘public use’ that are presented elsewhere at some length. As a matter of principle, I feel these data should be freely available, in a simple way, especially if they involve the expenditure of public money. The purpose of collecting data like this is to get people to use it for academic and policy work. If the investigators want to commercialize it they should say so up front and not run the operation through a university. Most serious international surveys (WVS, CSES, ISSP, EES etc) are all freely available without embargo and easily downloaded from their websites. The AmericasBarometer should do the same. I do realize that other survey exercises in the region, such as the LatinoBarometer, do charge for their data, but it seems to me that given the public funding of the AmerciasBarometer and its academic focus, it needs to set a clear principle about the public use of its data. I did manage to experiment with the online analysis tool. That worked well, though it was less user friendly than others I have used. Since I was unable to download any data it was not possible to evaluate the codebooks or other user documentation for the surveys. I did look at some of the past questionnaires and technical documents on the website and they downloaded without difficulty. In using any comparative dataset, issues of weighting and design effects loom large. The dilemma here is to provide a lot of information and let users decide on weighting procedures themselves, or minimize information and provide weighted data that takes account of design effects. Both approaches have their advantages and disadvantages and it was a problem we discussed on many occasions in the ****. While I was unable to run the combined AmericasBarometer dataset, the documentation does appear to strike a reasonable balance between these two extremes ****. 5. Strengths and weaknesses compared especially to other regional barometers such as the LatinoBarometer, the AfroBarometer and the AsiaBarometer The main advantages of the AmericasBarometer over its regional comparators are: (1) its greater longevity and country coverage and (2) generally greater concern with methodological uniformity. The latter is certainly the case in comparison with the AfroBarometer. However, the data from the AfroBarometer is much more accessible. Any comparisons with the LatinoBarometer are difficult because it is an annual survey with a limited range of questions and has a generally greater commercial and policy focus than the AmericasBarometer. However, the most recent LatinoBarometer data are also not freely available. Overall, the AmericasBarometer compared well in quality with other regional barometers, but its comparative advantage in this area is undermined by problems of accessibility and the complex policies that surround it. 108 DIMS Evaluation EXTERNAL REVIEW 2 The Latin American Public Opinion Project (LAPOP) is a state of the art source of public opinion data on Latin America. Under the directorship of Mitch Seligson, LAPOP has established itself as the most comprehensive source of public opinion data on Latin America. The data are collected on a two year cycle. In 2010, over 40,000 citizens from 26 different Latin American countries were surveyed. The data tap into the orientations and public opinion attitudes of ordinary citizens across the region. The surveys query attitudinal orientations and behavioral patterns crucial for understanding citizenship and democratic governance. It’s no surprise that policy makers and academics make tremendous use of this data source. In the remainder of this evaluation I will comment on the specific strengths of LAPOP. 1. Technical and substantive quality LAPOP employs the necessary technology and adopts very stringent training criteria to achieve its remarkable status as a premier survey data project. As a **** on the **** project, I have often looked to LAPOP as the model to emulate. In fact, we have adopted some of the training techniques and pre‐ survey activities that LAPOP employs. Setting the bar higher (for all of us), LAPOP continues to pave the road to achieve the highest quality work. The techniques that LAPOP employs ensure the minimization of sampling and nonsampling errors. Again these techniques are considered cutting edge procedures that guarantee high quality data. Some of these techniques include the following:  Extensive pre‐testing before the administration of surveys.  Translation (and back translation) of survey instruments.  Expert sampling designs of national probability samples.  Use of PDAs to upload data immediately (to minimize data entry and data merging mistakes)  Standard Training Manuals for all researchers on the project. Substantively LAPOP asks core questions visible in all global barometers and pertinent to both theoretical and policy debates. These substantive areas speak directly to themes and debates that emerge in studies of political culture, democratization, good governance, accountability, transparency, citizenship and civil society for example. Some of the substantive areas that LAPOP emphasizes in its surveys include the following.  Trust in institutions  Political tolerance  Civil society participation  Evaluations of the economy  Support for democracy  Political legitimacy  Electoral behavior  Corruption and crime victimization  Engagement in and attitudes toward local government 2. Reputation LAPOP’s reputation is solid among policy makers and scholars of comparative and Latin American politics. For example, I served as a discussant on **** panel last year, where almost each of the panelists made use or referenced LAPOP. It’s considered by many the most authoritative source of public opinion data on Latin America and indeed influences many studies concerned with comparative development and comparative political behavior as well. LAPOP data are utilized throughout the academy and in the policy world. The data are accessible (available for free on the website), comprehensive and reliable. This is a tremendous service to policy and academia. 109 LAPOP’s mission to provide accurate and high‐quality data reverberates across all networks (policy‐ making and academic alike). Furthermore, LAPOP’s Consortium based model, which has led to the inclusion of country partners from across Latin America means that this project is not seen as an “imposed” external project. The Consortium guarantees that Latin American experts influence and shape the content of the survey instruments in ways that reflect the needs and priorities of Latin Americans. In addition to administering a standard instrument in all countries—LAPOP allows for “flexibility” at the country‐level for the inclusion of survey items specific to local concerns. This tailored research design allows local policy makers and academics the precision they need to address local phenomena. Thus, LAPOP has earned for itself a solid reputation in Latin America itself. This is no small task to accomplish and LAPOP should be commended for its tremendous efforts on this score. 3. Visibility and Usage: As indicated above, the data are very visible and widely used. Anybody working on public opinion knows of LAPOP. The data are not only the most authoritative source of attitudes and behaviors on citizenship in Latin America, but they are extraordinarily useful to policy makers as well. These data serve the reform and democratization agendas by providing crucial information about how citizens prioritize their political and economic needs and how they view reform initiatives. Furthermore, LAPOP’s continued and impressive success in its ability to secure funding (from sources like USAID, UNDP, and the Inter‐ American Development Bank) is a testament of its fine and outstanding reputation. 4. Accessibility: LAPOP rises above the call of duty in making the data accessible to as a wide of an audience as possible. LAPOP now grants unrestricted access to the AmericasBarometer surveys (2004, 2006, 2008, 2010, and 2012; a total of 113 datasets). For subscribers, the data is available in merged datasets across time and space. The LAPOP website also includes an interactive program in both English and Spanish that allows for simple tabulations and even more rigorous multivariate analyses. Not only does LAPOP make its data accessible but also provides the following services as well:  Access to LAPOP‐created Stata do files designed especially for use with the AmericasBarometer data in order to generate user‐friendly graphs of a variety of statistical analyses.  Access to tech support (for subscribers only). Currently, LAPOP responds on a daily basis to countless technical questions about the database from its user community regarding such issues as sample design, data weights, question comparability, translation issues, etc. 4. Strengths and Weaknesses /Areas for Improvement: LAPOP is a leader among the Global Barometers. It consistently sets the bar higher for all of us to emulate. The team leaders have produced a very fine and outstanding source of data. As noted above, the projects strengths are numerous and impressive. There are really no weaknesses associated with this project. I would like to offer two suggestions, however, that may be useful to the project as it moves forward. First, LAPOP should play a more concerted effort in engaging and working with the other global barometers. There are real opportunities for collaboration which may include instrument design and development, sharing of new technology and setting even more rigorous standards for field workers, etc. In terms of outputs LAPOP should consider coordinating the release of findings in ways that speak to findings in other barometers as well. My second suggestion is an easier concern to address (and I believe important for LAPOP moving forward). Tech support should be provided for all inquiries (regardless of subscriber status). Otherwise, it will remain difficult for grad students and other individuals/groups (especially in Latin America) who 110 DIMS Evaluation can’t afford to pay subscriber fees to obtain clarifying and accurate information about the data. I would urge the PIs to include a line budget item for the fees related to a full‐time technical support staff to respond to queries from all individuals (subscribers and non‐subscribers) using these data. 111 APPENDIX F. SURVEY DATA ACADEMIC SURVEY (N = 370) Question 1: Question 2: your primary area of interest 37% an important but secondary area specialization 23% not really an area in which you specialize atdf all 40% To what extent is Latin America (by which we mean South America, Central America, Mexico, and the Caribbean) an area of academic interest to you? Is it: 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% Latin American public opinion and political behavior political institutions in Latin America political economy democratization What are your primary areas of substantive specialization? (check up to three) 112 DIMS Evaluation Question 3: Question 4: Working at a research-oriented U.S. university. 53% Working at a teaching-oriented US university. 7% Working in a non-academic position in the United States. 2% Working at a university in the Latin American region. 6% Working at a non￾academic position in the Latin American region. 0% Working at some other position. 25% Currently working toward an academic degree. 7% What is your current employment status? extremely high 15% high 21% moderate 30% low 22% have not heard of AmericasBarometer (or LAPOP) at all 12% How familiar are you with the AmericasBarometer surveys? Would you describe your familiarity with the AmericasBarometer (or LAPOP) surveys and their contents as: 113 Question 5: Question 6: 0 50 100 150 200 250 300 350 Read all or part of a report from the Americas Barometer Insights series. Viewed or downloaded information from the AmericasBarometer (LAPOP) web site. Read a scholarly article that made use of the AmericasBarometer data. Made use of the AmericasBarometer data in your own research. Received a grant from LAPOP to use the AmericasBarometer data. Included a published article using AmericasBarometer data as part of a class syllabus. For each of the following activities, please to tell us if you have done the following more than once or twice in your academic career, once or twice in your academic career, or never in your academic career: more than once or twice once or twice never 114 DIMS Evaluation Question 7: Question 8: extremely high 22% high 37% moderate 7% low 1% extremely low 1% you don’t feel you have enough information to make this assessment 32% Please tell us your overall assessment of the scientific quality of the AmericasBarometer data. Would you say that the scientific quality of the data is: Yes 47% No 53% Have you used AmericasBarometer data from multiple countries and/or multiple years? 115 Question 9: Question 10: Consistent 50% Somewhat consistent 29% Somewhat inconsistent 7% Inconsistent 1% No opinion 13% How consistent is the quality of AmericasBarometer data across countries? Consistent 41% Somewhat consistent 39% Somewhat inconsistent 5% Inconsistent 1% No opinion 14% How consistent is the quality of AmericasBarometer data across years? 116 DIMS Evaluation Question 11: Question 12: extremely relevant 61% somewhat relevant 22% not very relevant 2% completely irrelevant 0% you don't feel you have enough information to make this assessment 15% In your view, how relevant are the AmericasBarometer surveys to the study of public opinion and/or democratization processes in Latin America? extremely relevant 21% relevant 40% irrelevant 8% extremely irrelevant 1% you don't feel you have enough information to make this assessment 30% How relevant would you say the AmericasBarometer surveys are to your own research in the field of Latin American politics? 117 Question 13: Question 14: extremely high 25% high 42% moderate 11% low 1% extremely low 0% you don't feel you have enough information to make this assessment 21% In your view, what is the overall reputation of the AmericasBarometer surveys among the Latin American political and social science communities? More useful 72% Less useful 3% About the same 25% Would you say that the AmericasBarometer surveys provide more useful information than other sources of information? 118 DIMS Evaluation Question 15: Question 16: extremely easy 13% easy 59% difficult 25% extremely difficult 3% Based on your own experiences or what you know of others’ experiences, how easy or difficult is it for scholars to access the AmericasBarometer data for use in their own research? extremely responsive 13% responsive 24% unresponsive 4% extremely unresponsive 0% never needed assistance 59% Based on your own experiences or what you know of others’ experiences, how responsive or unresponsive is the LAPOP staff to scholars who have questions or problems with the AmericasBarometer data? 119 Question 17: extremely responsive 7% responsive 17% unresponsive 5% extremely unresponsive 0% never needed assistance 71% Based on your own experience or what you know of others’ experiences, how responsive or unresponsive is the Vanderbilt staff to scholars who seek to add questions or new content areas to the AmericasBarometer surveys? greatly enhanced the understanding of public opinion and/or democratization in Latin America 54% somewhat enhanced the understanding of democratization in Latin America 44% not enhanced the understanding of democratizatio n in Latin America 2% Thinking about scholarship that uses AmericasBarometer data or analyses, would you say that this work has: 120 DIMS Evaluation Question 18: Question 19: 0 50 100 150 200 250 elections, voter… political institutions participation beyond… political parties civil society groups… the economy and… crime, drugs and… corruption and vote-… mass political… mass political and… democratic… the judiciary and the… Latin American… women’s rights and … the inclusion of… international politics… Thinking now about the content of the AmericasBarometer surveys, please tell us if there is too much, just about the right amount, or too little attention paid to the following areas: Too much attention Just about the right amount of attention Too little attention 0 50 100 150 200 250 300 Civil society groups who seek to further democracy around the region. International donors like USAID who sponsor democracy assistance programs around the region. Academics who study democracy in Latin America. As far as you can assess, how useful would you say the AmericasBarometer data and analyses are to the following groups? Very useful Somewhat useful Not useful 121 Question 20: Open-Ended Responses for Academic Survey Could you please tell us anything that you think should be changed in the AmericasBarometer surveys in order to improve their usefulness to the academic and policy communities? 1 My undergraduates have used the data for their senior projects and I seem to remember that some of the data were free and some were not. The cost was prohibitive and limited their work, if I remember correctly. 2 Make more of the datasets freely available. 3 More attention should be given to the role of foreign countries, especially the US, in Latin American politics. 4 There is a trade-off between maintaining the same questions to build a time-series and changing questions to adapt to new issues of interest and new survey techniques that can improve the quality of the data and its usefulness. I believe there has been too much emphasis on the former. The consequence is that in order to add the new the questionnaires have become way too long, affecting the overall reliability of the answers. I would get rid of some old fashioned questions and prioritize the new, even at the cost of sacrificing time-series data. 5 More questions about policymaking and policy preferences 6 It should be made more available to the academic community. There should be no fees for its use. 7 Making the data even more accessible for analysis via the project's webpage -- making more sophisticated use of the data possible for "visiting" researchers. 8 More frequent general comparative surveys versus current two-year pattern 9 So far I have only needed to use the data for descriptive statistics that can be calculated with the online analysis tool (which is pretty clunky, but has worked for my purposes). But by far, the biggest improvement in usefulness would come from improvements in access to the data. 10 make it more accessible (free) 11 It's a tricky balancing act to maintain a core of important questions over time but to also respond to changing mission interests and academic research questions. I think it goes pretty well. 12 Too much focus in the questionnaires on issues of democratic legitimacy and political culture. Most of the young generation of Latin Americanists doing survey research do not care about these attitudes, and indeed there is a lot of evidence to show they do not matter. The topics of interest and importance these days are things like clientelism, issue cleavages, mass media, etc. 13 The surveys are generally quite helpful for assessing public opinion about key democratic institutions over time. The problem is that they tend to be (over)sold and viewed as a means to measure demcoratic institutional development, which is a mistake. LAPOP should make this clear. 14 My biggest suggestion would be to simplify access to LAPOP surveys/data. Currently it is VERY expensive for individual academics (and institutions) to pay the rather exorbitant (and prohibitive) fees to gain access to all country/year surveys. This is why LAPOP has had so little impact/visibility and is not as widely used as it should. As of now, one can perhaps pay the fee to gain acces to one or two country/year surveys, but it'd be impossible to pony up the money to gain access to all of 122 DIMS Evaluation them. As a result, the data are severely underutilized. 15 I really like that LAPOP asks insightful questions about law and the judiciary. I would like it to go a bit further by asking specific questions about opinions about different levels of courts. ie. high courts vs. lower courts and specific questions about certain areas of law such as the protection of property rights. I may be wrong on this as I have not looked at LAPOP in awhile, but I did use it in two articles and found it very helpful 16 should include more about citizen's attitudes about international phenomena such as military intervention, democracy promotion, climate change, international organizations, etc. 17 more comparative data on relations between shantytown residents and the police 18 I rarely use public opinion data in my research. My preferred databases for public opinion are GSS (US), ISSP (OECD) because of the broad range of co-variates that one can include in an analysis (in addition to what appear to be good sampling techniques). The more AmericasBarometer can look like ISSP/GSS, the better. 19 New methodological approaches of the sort ANES usually includes 20 It might be a good idea to make the questions of the AB more - or exactly - similar to questions in surveys from other regions (e.g. Afrobarometer, Asia etc.). This would help increase comparability of things like attitudes to democracy across regions. Also, I think it would be good to have more information about voter evaluations of the operation of democracy. For instance, things like voters experience with vote-buying, electoral malpractice and party intimidation, perceptions of ballot secrecy etc. 21 1. Access to data is very restricted and very expensive, particuarly for multi-country or multi-year studies 2. Questions are somewhat inconsistent over time 3. More questions on opinions about public policy. 22 It would be nice to have representative samples of subnational regions within countries beyond Mexico. 23 I am sorry to be of so little help. I have only recently begun to be interested in Latin American politics. My current interests pertain to institutions, but from earlier work on public opinion in Europe and the OECD, I welcome the prospect of crossregional comparison that the Americasbarometer data can give. 24 More accessible for scholarly research. 25 I think the series is great, and my "too little" attention choices above are matters of emphasis rather than any sort of complaint... 26 Not really. The cross-national nature of surveys is both a pro (comparability) and a con (often country-specific questions are missed). 27 Fuller description of the sampling procedures 28 Data should be deposited at ICPSR, it is wrong that they are not publicly available. 29 I was told, and this could be false, that the Americas Barometer were the least accessible of the various Barometers series. However, I have never used the data myself, and I have a high opinion of the scholars involved. 30 If part of the effort is to make these surveys more easily available, then surveys prior to 2004 should be publicly available. Releasing only five waves is not that helpful longitudinally. Even at a Research I university, we had to petition (and find funds) to have access to the funds. 123 31 The template used in the published reports is too repetitive; the same difficult-to-read tables and graphs over and over. There is no need to use such a restrictive template. Produce country reports that deviate from the rather boring model used in recent years; allow authors to prepare customized graphics and tables; try to make the reports a bit less dry and lifeless. 32 I haven't used the surveys myself, but a number of my graduate students have used or tried to use them. It has been a frustrating experience for them. The survey needs to include more items that ask what the respondent knows about the political system so that analysts can control for political sophistication/information. It should also include more items about current politics and attitudes toward parties, economic policies, and crime. Democratization happened a long time ago in Latin America. Questions about popular attitudes toward democratic values aren't relevant to what most Latin Americanists study now. Some of the questions on the survey I just filled out didn't make sense to me so I didn't answer them. For example, what does it mean for an individual-level survey to include more about political institutions? Do you mean items about citizens' attitudes toward PR? If so, I don't see much point. Or do you mean a link to a website that details the political institutions of each country at the time the survey was done? That would be a useful convenience, though it's possible to find descriptions of Latin American political institutions on other websites. 33 Focus more on elections-who would the person vote for if the election were held today (the previous vote and partisanship questions are across the political spectrum, but that question is much more closed), more information on policy preferences and perceived policy responsiveness 34 Include policy communities in the events launching the country reports; organize multi-country edited volumes on specific themes. 35 Make the data available at no charge. 36 Yes, the access barriers are too high and frustrating, LAPOP says the data are in the public domain but the hoops and barriers to access make that a practical fiction....unless you get USAID to "unlock" the data.. Latinobarometer and World Values data are more valuablke consistent for cross-country and cross-time benchmark comparisons. 37 Provide more a core of common questions every year with the addition of the rotating questions for scholars' particular needs or for new and timely topics. 38 more focus on day to day politics and issues and less focus on democratic legitimacy 39 Perhaps expanding the number of grants available to graduate students and scholars in the U.S. would make it more useful. Currently, the fee structure is a bit of a barrier for scholars based at teaching universities, where research funds are scarce. 40 AB should be annual and should include questions that gauge vote choice, not just turnout. Also it should have standard questions about economic assessments used in economic voting literaure. 41 It simply needs a longer time series. 42 It may be beneficial to try to standardize ways of allowing others to contribute questions, rather than just through personal contacts. For example, wanting more questions on environmental issues, how would I go about seeking their placement? 43 I have two broad areas of comment. The first is accessibility of the data. Here, I express my wish that the data were made fully, readily, and freely available to all interested researchers. Of course, I understand that there may exist cost or ownership issues that prevent this. Second, I have some brief comments on substantive issue areas. ACCESIBILITY: I have not used AmericasBarometer data extensively in my own research.To the best of my recollection, when I 124 DIMS Evaluation did use these data, I found that it was not straightforward to access it. I was able to generate summary tables online. However, the process of obtaining the raw data -- which is what, as an academic, I really need -- required a subscription by my university or by me, which cost money (this is what I recall, apologies if this is not the case and I am simply not remembering correctly). This limits the likelihood that the data will be used by the following categories of researchers: (1) researchers for whom these data are not essential to their research projects, and therefore cannot justify the cost of full access; (2) researchers who have no ready access to funds for a subscription, or whose university does not subscribe. The problem is compounded by the fact that, for some issue areas, there exists publicly-available public-opinion survey data of high quality for Latin America (e.g. the Comparative Study of Electoral Systems, which is easily, freely, and readily available, among others). Of course, coverage is different insofar as questions, issue areas, etc. are concerned. Full access means: access to the full set of surveys in one single merged file covering all survey years, in a variety of formats including, at least, SPSS and STATA, as well as a searchable, complete, codebook in one file (as an example, see the General Social Survey). I see from your website that free access will be implemented beginning in December 2012, but again not providing the full dataset. If the LAPOP data are financed with taxpayer money, perhaps it is worth considering releasing them fully to researchers, as is the practice with other data generated with public money (e.g. the General Social Survey, the American National Election Study, or the Mexico Panel Study, all of which can be downloaded straight from the web in their complete form). SUBSTANTIVE COVERAGE: In terms of substantive content, one issue are that I feel requires further emphasis is electoral manipulation -- encompassing not only vote buying, but also the full panoply of tactics of manipulation that are sometimes used, media freedom in connection with elections, journalist repression in connection with elections, and public confidence in the fairness of elections and in electoral results. Again, I do not intimately know the data, so perhaps these areas are already covered and I am simply not aware of it. A second issue area that I would recommend emphasizing is corruption. I have not yet looked into the current or past coverage of corruption-related issues, but it is an issue area where improvements in "measurement" are being made, that may not yet be reflected in all data-collection efforts (for example, the literature has emphasized the difference between asking about corruption perceptions and corruption experiences). Finally, a third suggestion is to consider implementing surveys that are representative on the subnational level, even if this is only possible in a reduced set of countries. I realize that this is an order of magnitude more complicated and expensive than conducting nationally-representative surveys. However, the unevenness of "democratization" within democratic countries, when comparing states, provinces, or municipalities, is a striking fact that is becoming increasingly prominent and important in the political lives of citizens in many, if not all, countries of Latin America. Moreover, these data would be entirely novel (I know of no subnationally-representative dataset with follow-up waves for any Latin American country), and open very fruitful windows for research and for democracy-related foreign aid efforts. 44 No. It's working great. Extremely valuable resource. 45 Several universities decide not to invest their monies in buying access to LAPOP. This is entirely the University's responsibility, but scholars and students get ultimately harmed. If there was at least a reduced version available (or old databases, like Latinobarometro does), it'd be helpful 46 Engage more with scholars in Latin America rather than having the surveys always designed by a small number of players. 47 perhaps more pre- and post-election surveys 48 Nothing comes to mind. 49 Pay more attention to the academic interests of scholars living and working in LA. 125 50 To the extent possible, consistency in repeating questions with the same wording and categories across survey years. 51 place likely interested parties on email listserv for LAPOP updates 52 The data should become public. Not only other surveys (NES, European Social Survey, Afro￾Barometer) are open, but even Latinobarometro data is public now. LAPOP data will only be as valuable as the number of people analyzing the information. 53 Easier access for scholars and graduate students and a clearer process for non-core LAPOP scholars to request new survey items or retain time-series items planned for removal. 54 more behavioral data 55 Not enough knowledge of the survey to make a real assessment. 56 Reduce the size of the questionnaire to improve data quality. 57 Translate everything in English 58 This is a superb survey. Its methodology and content are far superior to the main rival, Latinobarometer. I am in the process of gaining an institutional subscription for my university, as there are several faculty and students in American and Comparative politics who wish to use the LAPOP data. 59 improve the questions. I think many people do not understand them due to the way they are posed. 60 The AmericasBarometer surveys have set the bar for survey research. This is an excellent source of data, collected with the highest level of concern about quality. The only thing that could improve the surveys would be to increase their accessibility. In 2012 I believe there is a pilot available to make the data freely accessible completely, and it would be wonderful to maintain this after 2012. 61 maintain questions across all countries and years so that better comparisons can be made. 62 Would be good to include survey / list experiments. 63 This survey has a reputation as an overpriced and underperforming tool captured by a handful of scholars at a single institution. 64 Annual open consultation to academic community on ideas on how to improve specific batteries of questions included in the survey. 65 The reports sent to me in the monthly (?) newsletter always seem to be very superficial. I stopped reading them because the conceptual and analytical framing is about 10-15 years behind the debate in Latin America. this may be a function of the public opinion field, but I no longer read the newsletters. I am uncertain about the intended audience. 66 It would be great if LAPOP had some formal process by which it would consider the inclusion of survey questions that are of interest to the scholarly community. 67 Getting access to the data is a pain and so far the data have not been as widely used as they could/should be given their quality and I think that's an issue of access. It is also frustrating that we've had to spend some significant time in getting the data into formats where it could be used when we did get it. It would be nice if especially for cases where it was possible to pool across countries it was an easier process to assemble the dataset because that represents a major use of time. Data are good but getting them into usable form seems to be more time consuming than it 126 DIMS Evaluation should be. 68 More a Mexico-specific researcher than a regional one, I find that state-society relations tend to be conflated with topics like corruption, ethnic minorities, and rule-of-law. It would be useful to have survey data on concrete levels of state presence in outlying communities and districts. For example, in Mexico, my experience in the southern region is that state presence is circumscribed to public schools, IFE, Oportunidades and sometimes a clinic. does a similar pattern hold for the underdeveloped parts of other Latin American countries? 69 Some questions are poorly designed and entire sections (e.g., political trust) prone to response bias. Also, the methodological reports are extremely light and, quite frankly, I don't believe in "self￾weighted" samples. 70 Should learn from (follow lead of) Afrobarometer in terms of making data easily accessible. 71 La información de prensa no resulta muy relevante. Tal vez falta algo de iniciativas multilaterales y su impacto en las políticas nacionales 72 Some questions, for example those related to women/gender are only available for certain years. Other questions are only available in some countries. Consistency in questions over time and breadth of countries would greatly enhance the ability to measure change over time and compare across the region. One deficit of LAPOP is that it is relatively new; thus making some longer-term surveys more useful for some questions. Consistency in asking a broad variety of questions, in multiple countries over time will make the survey and its data ever more useful. 73 It is an outstanding and comprehensive source on all aspects of democratic transition and consolidation. 74 reputation for ease of access is low. more anchoring vingettes for cross-national comparability. reputation for sample representativeness is low, although I myself have never verified this. 75 Given the public/federal money it receives, free access to the data should be the baseline. Having to subscribe to obtain the data does not seem right or fair. 76 Given my interests: -Make samples representative by electoral district. -More extensive collaboration with Salamanca to ask same questions of elites and masses. -Consultation with a broader audience regarding content. 77 In terms of the academic community, the AmericasBarometer needs to be decentralized. The current AB commitment to a single questionnaire (with a few local questions) administered to a cross-section means that there is no money left for panels or for subnational designs, and the AB's focus on political culture makes it of little use for research on elections. The AB has a commitment to keeping all the questions used in its past, so the questionnaires are very long. some of the questions are poor, but they have to stay in. And the emphasis on trust in various institutions is pretty much a bore to academics. USAID is spending a lot of money for what would appear to be little more than very soft measures of change in fuzzy measures of legitimacy, trust in institutions, etc. The scarcity of real panel studies in Latin America is related to the concentration of survey money in LAPOP. 78 Better data on vote choice in legislative and local elections 79 Include a more elaborate battery of policy positions beyond economic distribution. Include questions about clientelism, from vote buying through partisan based access to social insurance benefits or patronage jobs, etc. 80 Lack of analysis of intra-Latin American relations. Not everything is internal or flows from the 127 region to the US and viceversa. 128 DIMS Evaluation MISSION SURVEY (N=30) Question 1: Question 2: extremely high 20% high 50% moderate 20% low 10% have not heard of LAPOP at all 0% How familiar are you with the AmericasBarometer surveys? Would you describe your familiarity with the AmericasBarometer surveys and their contents as: At least once a month 10% Several times a year 70% Not more than once or twice a year 17% Virtually never 3% In your day-to day activities, about how often do you use or make reference to AmericasBarometer / DIMS data? 129 Question 3: Question 4: Open-Ended Responses on Program Development in Mission Survey Could you please describe a specific program (or programs) and explain how AmericasBarometer data was used in its development or targeting? 1 Many of the indicators that constitute the DG Office's PMP are taken from the information of the LAPOP survey (corruption perception, crime victimization, government responsiveness, etc.) 2 The barometer was used for setting the indicators, monitoring and evaluating our Active Democracy program which promoted democratic values in key regions with a particular focus on youth and women 3 The DIMS survey has helped the **** for several purposes: first, to measure progress (through indicators) towards the achievement of expected results in different elements of good governance such as transparency, elections, security, rule of law, local development, legislative development etc; second, to see the trend of changes and depending on the results, to focus in specif areas to strengthen the ****democratic process; and third to show partners that the democracy surveys are scientific and that the data is reliable. 4 Design of the 5-year DG strategy for the mission including the specific projects in transparency, elections, and decentralization 5 We have used information in LAPOP in our Local Government Program. We also used the survey as a base to do an oversampling in ***** - using many of the crime victimization questions. We used this information to inform a local program that focuses on building the capacity of police, prevention efforts and rule of law. Often 37% Once or twice 50% Never 3% Don’t recall 10% How often would you say that you have used AmericasBarometer data or analyses in DEVELOPING specific DRG programs or targeting their focus? 130 DIMS Evaluation 6 The Municipal Governance Program. We are focusing our citizen participation efforts on local budget because we learned from the survey that the participation was very low. The participation in public consultations made by municipal government has a trend of going down also and we hope to have an impact in both indicators. The satisfaction with local services is also something that we are focusing on, by supporting service improvement on three areas. 7 Crime prevention program design efforts use the data as part of the overall description of the problems. 8 we did a elections program trying to change voters' attitudes toward democracy and used AB for baseline data. 9 Used in developing two projects, one in the justice sector and the other in the elections and political processes sector. Both involved an analysis of the level of support for government institutions, which complemented field research. 10 We have most recently used the questionnaire to structure our own mission-wide baseline survey in which we will compare the data we collect with that collected by AB, which is nation-wide and perception-focused. 11 Our Justice and HUman Rights Projects used the AmericasBarometers as references for sub￾sectors to target our assistance. 12 Rule of law, good governance and civil society strengthening. 13 for the design of civil society advocacy and related anticorruption programs. 14 The design of our political party program was influenced by the data collectedthrough the AB survey which demonstrated the lack of credibility, accountability and democracy withing the political party structures and its leaders. 15 The data and analysis was used for the definition of USAID/**** Democracy Strategy for the period 2001-2005. Indicators for the performance monitoring plan were defined based on the survey's data. The data also provided the Mission with stronger arguments to focus its democracy work on local governance, justice, and transparency. One of the areas in which USAID/**** has been providing support is civil society and local governance. The data produced by the Americas Barometer have been extremely helpful to either continue USAID support (in civil society), or provide greater attention to specific areas (for example, citizen participation in local governance). The data also helped our justice sector partner evidence the success of its interventions by comparing data at the local level where it had focused project activities vs. data at the national level in terms of citizen trust in the justice system. 16 Information from the survey was used to develop specific indicators for the local governance and justice projects. 17 The Mission has used the survey to inform certain program designs such as the Elections program, the Access to Justice program and the Human Rights Program. 18 When preparing a Request for Application (RFA) or when evaluating proposals that come in to USAID for evaluation. In the first case I have cited the AB several times and in the second cases I have compared the info provided with the AB data. 19 Community based police programs - helped to gauge perception of the police Corruption and strengths of state institutions - to set baselines 20 DG program in ****, high level indicators in framework are the Americas Barometer data 131 21 An institutional strengthening program for electoral authorities used the citizen participation data to demonstrate to members of these bodies and civil society that citizen participation could be improved. 22 Data was used for activity design for the Mission's 2009-2013 DG program and for certain indictors in the PMP. 23 We used the results of the Barometer survey we commissioned in 2009, an off year, for project design and in our D&G programming documents. 24 Justice and Civil Society Program. Also Preventive Security Program 25 Data was used as a baseline for assessing ***** women's political attitudes Question 5: Often 13% Once or twice Never 33% 47% Don't recall 7% How often would you say that you have used AmericasBarometer data or analyses as part of an EVALUATION of specific DRG programs? 132 DIMS Evaluation Question 6: Open-Ended Responses on Program Evaluation in Mission Survey Could you please describe a specific program (or programs) and explain how AmericasBarometer data were used in its evaluation? 1 See previous answer 2 The 2004 and 2006 surveys helped the Mission to design the on-going transparency and Governance Program based on the data and analysis provided in the area of corruption and its impact in a stable democracy. 3 Reports were used in the design of the Mission 5-year strategy. Citizens perception on democratic participation and governance - were critical in identifying work with civil society as a key component of our strategy. 4 We are conducting a rule of law evaluation and a baseline for a new security and justice sector program. We will include LAPOP as one of the required readings in the desk study and something that should be considered for background. We also included it as a key reference for our country development cooperation strategy (CDCS) and a security sector reform assessement. 5 We tracked the corruption victimization index as a proxy to evaluate the Transaparency Program that ended in 2009 6 data has been used to broadly determine if change has occurred. 7 in our elections/voter behavior change program when we measured voters' attitudes on democracy. 8 We recently looked at the data on elections and voting as we are planning an election security assessment for upcoming local elections. However, it seemed that the data was much more optimistic than results of other evaluations we have recently read so it was hard to compare. 9 Elections and Political Parties Program. 10 AB data in the area of transparency was used to explore more in depth citizen perceotion of the cost of corrupton (Our Mission conducted an additonal study) 11 The data was utilized in the evaluation of USAID/**** former justice project. Evaluators found data relevant to compare overall perceptions and trust in justice during the last years of our support. The data, being part of our Performance Monitoring Plan, was also utilized/analyzed in 2006 by auditors participating in an OIG Performance Audit of AID/**** Democracy and Governance Activities. Auditors considered that a couple of the indicators that the Mission had selected were not appropriate because our activities were too limited in scope to influence planned results. 12 I wrote it in question # 4. 13 Community based policing and anticorruption programs 14 AMericasBarometer data are part of DG framwework in **** 133 Question 7: Question 8: It was central to the DG assessment process 7% It played an important role but there are more important sources of information 54% It played a role, but a very limited one 31% It didn’t play any significant role at all 8% What role has AmericasBarometer played in previous DG assessments in your mission? 0 5 10 15 20 25 30 35 Read reports prepared by Vanderbilt LAPOP or others which use AmericasBarometer data? Asked someone else at the mission to do some analyses of AmericasBarometer data? Personally used a spreadsheet program, such as Excel, to create figures or tables from DIMS data? Used AmericasBarometer data, tables, or other information in a public presentation you’ve given as part of your official duties? Seen AmericasBarometer data, tables or other information in a presentation given by others as part of their official duties? Which of the following ways have you used AmericasBarometer data? Have you: Yes No 134 DIMS Evaluation Question 9: Question 10: Yes 86% No 7% Don't know 7% Has your mission contracted with Vanderbilt LAPOP to develop special batteries of questions specifically focused on your country or to increase the sample size for certain sub-populations in the AmericasBarometer surveys? To a great extent 77% To a limited extent 23% To virtually no extent 0% To what extent did your mission participate in the design of those questions? 135 Question 11: Question 12: Extremely satisfied 72% Somewhat satisfied 28% Somewhat dissatisfied 0% Extremely dissatisfied 0% How satisfied were you with the responsiveness of the Vanderbilt LAPOP staff and senior personnel to your mission’s concerns about the country-specific questions or other content you wanted to include in the survey? We participate extensively, suggesting both general topics and specific questions for inclusion in the survey 34% We review and comment generally on the survey during the Vanderbilt LAPOP organized “kick￾off” meetings 38% Vanderbilt LAPOP mostly designs the survey with little input from us. 14% I don't have enough information to answer this question. 14% To what extent does your mission participated in the design of the core AmericasBarometer questions (i.e., the questions that are asked in all of the AmericasBarometer countries surveyed)? 136 DIMS Evaluation Question 13: Question 14: Highly responsive to the Mission? 55% Somewhat responsive? 31% Not very responsive? 0% Not at all responsive? 0% I don't have enough information to answer this question. 14% How would you assess the responsiveness of Vanderbilt LAPOP to your mission’s general concerns about the content or implementation of the AmericasBarometer surveys? Would you describe Vanderbilt LAPOP as: A great deal of contact 39% Some contact 32% Virtually no contact. 29% How much contact do you have with the LAPOP partners in your country, that is, the people at the university, civil society groups, or survey firms that carry out the AmericasBarometer survey on behalf of LAPOP? 137 Question 15: Question 16: 0 5 10 15 20 25 30 AB has contributed to the quantitative research capacity of local data collection firms. AB has built the capacity of local researchers to conceptualize and conduct research. Thinking about your experiences with LAPOP partners, please state the extent to which you agree or disagree with the following statements: Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don't know 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Host country government officials Other foreign donors International NGOs Local NGOs Others Have you seen AmericasBarometer data used by any of the following organizations? (select all that apply) 138 DIMS Evaluation Question 17: Question 18: More Interaction 24% About the same Interaction 76% Less Interaction 0% In terms of your relationship with LAPOP local partners, would you prefer less interaction, about the same amount of interaction, or more interaction in the future? More Interaction 3% About the same Interaction 93% Less Interaction 4% In terms of your relationship with Vanderbilt LAPOP Central, would you prefer less interaction, about the same amount of interaction, or more interaction in the future? 139 Question 19: Question 20: Rely on the country report and other analyses routinely provided by Vanderbilt LAPOP and its partners 46% Ask academic professionals at LAPOP to do a special analysis for you 39% Ask someone in your mission to analyze the data for you 4% Use a spreadsheet such as EXCEL to generate any tables or figures you might need on your own. 11% Other 0% If you wanted to take advantage of AmericasBarometer data would you be mostly likely to: AmericasBaromete r is a highly valuable tool that we could not do without 28% It’s a useful tool but we could do our work almost as well without it 55% AmericasBaromete r has little or no value to the DRG work of the mission. 3% I don't have enough information to answer this question. 14% How would you assess the overall value of AmericasBarometer for the DRG work of your mission? 140 DIMS Evaluation Question 21: Question 22: Open-ended Responses on Content in Mission Survey Could you tell us about any content areas where you think there should be more emphasis in the AmericasBarometer surveys? In other words, can you elaborate on things that you think missions need to know that are currently not asked at all or not asked in enough detail in the AmericasBarometer questionnaires? 1 Citizen security is a real issue in **** and in the neighbor countries, but this is not asked in enough details. The questionnaire is already too long so maybe the number of surveyed areas (corruption, local governments, etc) should be reduced and concentrate or focus in each country's priorities. The analysis for each country (combining different variables) can be done more in depth. I understand that this may be in detriment of the comparative analysis among participating countries. 2 working closely with Vanderbilt has allowed us to incorporate areas that the Mission needed - hence I dont have any additional recommendations. 3 There could be more detailed questions on crime victimization. 4 We would like youth at risk and food security questions. We would like it if you incorporate the amount that corruption victims have paid. 5 I think the questions and data collected is good, and Vanderbilt has been very responsive to tweaking the questions we ask them to, the problem is our Mission needs better training on how to use the data. We are also so stretched that it is hard to sometimes even get DG officers to 1 minimum value 4% 2 3% 3 4 3% 3% 5 3% 6 7% 7 7% 8 28% 9 28% 10 essential 14% On a Scale of 1-10 where 10 means DIMS is essential and must be continued and 1 means it is of minimum value and the money spent on it should be reprogrammed, how important is it to you that the contract for DIMS should be renewed? 141 read the report or input on the questionnaire. I think **** is a special case with a very difficult strategy to carry out and sometimes M&E is put to the wayside. Perhaps when things slow down, people will have more time to appreciate the survey, but for now, it is hardly utilized to its potential. 6 There are certainly a number of areas that could be explored in more depth, however, the AB survey it's already too long to be expanded and if we eliminate questions we lose comparability with the set of data (data bese )we already have. That said, I believe the survey could explore including more on youth. 7 The battery of questions for measuring citizen participation needs to be strengthened and up-dated to capture more non-traditional mechanisms for participating in democratic processes. Incorporate questions on areas such as Environment and Global Climate Change which are key issues of concern to many countries in the region. We need more data which could be relevant and useful to strengthen USAID 's Democracy and Environment cross-sector work. 8 I think we have included the questions that are relevant for this context. We have discussed them with both, Vanderbilt and local partner. 9 we do have an input in the content areas and those might change from time to time depending on the issues facing the country. 10 the why sometimes in questions of all types is left out, and is left to interpretation by LAPOP. 11 For regional comparative and trend analyses, there is need for a common set of information (as is currently done). In this regard, the current content areas are adequate. However, there are country specific issues for which 'perception' is not sufficient - e.g forms/incidences of corruption, ethno-political hositlity and violence, issues relating to periodic events such as national elections. 12 Environment area and relationship between poor and democracy culture 13 Annual frequency is low, could be higher. It would be interesting to see effect of pending elections on attitudes (if any). 142 DIMS Evaluation Question 23: Open-Ended Responses on Usefulness in Mission Survey Can you think of any other ways that the AmericasBarometer surveys could be made more useful for your work? 1 Again, more in depth analysis of priority topics of each country 2 friedlier reports that policy makers and congress could readily use/apply 3 The surveys are a great tool. The challenge is taking advantage of them, sharing them with partners and ensuring the information is informing our programs. They generate excitement during the presentation but often are not cited or used afterwards. This is a challenge for the DG team given everything else that is going on. I found that the oversampling for specific areas was really critical and we plan to do that again for our new Local Government program. 4 Yes. If the reports could be written earlier (three months after the field work), **** would be more excited and open to learn about the results. Since there are periodic surveys here**** and they have similar democracy questions, many of the questions we have become outdated and superseded and usually disseminate results within a few weeks. This is a common complain that my colleagues have and that they hear from their partners. 5 Again, see the comment above. We should be using it to design new programs and evaluate our performance, but it is barely use at all. I'm not sure people have faith in it and there are so many other surveys going on. Our Mission is doing their own yearly survey across all sectors using many of the same questions as AB so we will then perhaps be forced to use that data over AB. 6 the design of the Agency and Missions DG strategies should be based more on the political trends found through the AB. 7 Develop more frequent in-house presentations to USAID and Embassy officials on key issues covered in the study. Enhance the utilization of the results and analysis to specific DG projects that USAID is implementing. Take more advantage of LAPOP's partners in the field. The Mission is not taking advantage of this great opportunity. Promote greater media involvement for the dissemination of results. This could be sensitive but it will be important to come up with more effective strategies for disseminating the results of the surveys. 8 There is a need to improve the dissemination of the survey both within USAID and with government officials and the public in general. The information contained in the survey is very valuable, but there is a need to make it more accesible to all so that it's used more frecuently. 9 I want to mention that on Question # 15 I don't agree with the statements because the local firms that we work with here are excellent organizations with lots of experience in surveys, analysis and research. Answering this question (# 23), I think we need to use more the data and results within USAID and with partners from different projects. If we use it more and encourage others to use it, it will become more useful and indispensable for our work. 10 Provide more user-friendly, more periodic survey results. 11 develop some reader friendly versions of the report that could be distributed to the communities and the ordinary man in the streets 12 Be used for evaluations of DG programs. 13 AB surveys can be more useful to Missions if the data collected are more in sync with the Foreign Assistance Framework indicators. Although not practical, AB surveys can be more useful if they coincide with country strategy planning periods and program implementation timelines. AB looks 143 at high/impact level responses and thus is not suitable for annual reporting requirements of USAID. 14 Next stages of AB should focus on building Local Capacity to do this survey in country in the future. This will guarantee sustainability and phase out Vanderbilt. Question 24: Colombia 11% Dominican Republic 7% Ecuador 7% Guatemala 7% Guyana 7% Haiti 7% Honduras 7% Mexico 7% Nicaragua 7% Panama 7% Paraguay 4% Peru 7% El Salvador 11% Venezuela 4% Which country does your mission support? 144 DIMS Evaluation Question 25: Question 26: 1-2 years 45% 2-3 years 10% more than 3 years 45% How long have you worked in the mission? DIMS Activity Manager 28% Democracy and Governance Officer (not Activity Manager) 58% Other 14% What is your job function? 145 APPENDIX G.OVERVIEW OF SURVEY SAMPLING METHODS The goal of any survey sampling method is to draw a sample of individuals whose responses will accurately represent what would have been obtained had all members of a population been interviewed. The most important distinction in sampling methodology is between a probability and a non-probability sample, with the former being defined as a sample where each individual from a population has a known chance of being included. With probability samples, researchers may, after calculating the percentages of individuals in a sample who have some characteristic or who respond a certain way to a particular question, then calculate an interval within which the overall population percentages are likely to lie, subject to a known degree of confidence (usually 95%). This interval is called the sampling error for a given survey. With large probability samples, the sampling error is typically in the range of 3%, which means that adding 3% and subtracting 3% from the aggregate response observed for a question in a given sample will produce an interval within which the researcher estimates that the population response would lie with 95% confidence (or a 5% chance of being wrong). The specific sampling error in a given instance will depend on the exact number of individuals interviewed, as well as any “design effects” (explained below) that produce deviations in practice from a “pure” random sample. With non￾probability samples, sampling errors cannot be calculated; it is difficult if not impossible to generalize the results obtained in a survey sample to what the values are likely to be in the overall population. We list the kinds of samples typically used in practice in survey research below. The ideal (“pure”) probability sample is Simple Random Sampling (SRS), in which every member of the population has a known, non-zero, and equal chance of being selected. This is obviously not possible to implement when sampling large national populations, given the absence of lists of individuals in the population that would be available for researchers to use as a sampling frame. Most large-scale national samples, including LAPOP’s sample, are multi-stage stratified cluster samples, which divide the population into smaller geographic strata, sample these strata at random with “proportional to population” selection weights such that the likelihood of a unit within the strata being selected is based on the size of its population), then divide those selected units into smaller units, sample them at random using proportional to population weights, and so on until the smallest stratum is selected, usually the city or village block. Because individuals living on the same block are likely to be similar to one another (or at least more similar than randomly selected individuals from the country at large), clustering introduces more error into the sampling process, the more so the more homogeneous the blocks and the smaller the number of interviews per block. The increases in sampling error that result from multi-stage clustering versus a simple random sample are called design effects, and LAPOP conducts, to our knowledge, the only regional barometer that calculates these design effects and includes them in their official documents about the samples, sampling errors, and other kinds of data quality issues. Our main concern with the LAPOP samples is in their use of quota methods based on age and gender characteristics when selecting an individual to interview within a chosen household, as opposed to using a probabilistic method such as the Kish or “next birthday” method. See Sections II and III below and the discussion in the main text for more details. 146 DIMS Evaluation I. Probability Sample Designs 1. Simple Random Sampling: Every member of the population has a known, non-zero, and equal chance of being selected. The quintessential example is drawing names at random from a hat with no replacement. Everyone’s name is included, and they all have the same chance of being drawn. 2. Systematic Random Sampling: Selection is based on a randomly-ordered sampling frame. It is an approximation to an SRS in the sense that every member of the population has an equal chance of selection. However, it requires an actual, randomly-ordered listing of the population members. A random starting point is chosen, and then every element on the list is chosen for the sample. The random start and the sampling interval are calculated as the population size (N) divided by the sample size (n) required. 3. Stratified Random Sampling: In a heterogeneous population, it may be useful to group the population in relatively homogenous subpopulations and then sample each subpopulation, or stratum, independently. In a stratified random sample, the strata must be mutually exclusive and exhaustive. Separate simple random samples are then drawn within each stratum. This design is more appropriate in populations with natural and easily identified groupings. a. Proportionate Stratified Sampling: Sampling fractions (n/N) are the same for all strata. In this design, the distribution of cases across the strata in the sample effectively reflects the distribution across strata in the population. Proportionate stratified sampling ensures that all sub-groups are represented in the sample, which might not occur in a SRS just by chance. b. Disproportionate Stratified Sampling: Sampling fractions (n/N) differ across strata. In this design, the distribution of cases across the strata in the sample is different from the distribution across strata in the population. Some strata are over-sampled relative to their numbers in the population, while others are sampled at lower rates than they occur in the population. This design allows for statistical comparisons of rare sub￾groups, which would be represented by too few cases for reliable analysis in SRS or proportionate stratified samples. 4. Cluster Sampling: Population is divided into self-evident sub-groups, or clusters, and a random sample of clusters is subsequently selected. Clusters are often based on geographical areas in order to make the execution of the survey more cost-effective in a geographically dispersed population. However, cluster samples are less precise than SRS or stratified samples because selections in the sample are not independent of one another and this generally results in higher standard errors for statistics. 5. Multi-Stage Cluster, or Multi-Stage Stratified Cluster Sampling: Represents a complex form of cluster sampling. In multi-stage samples, clusters are chosen at a first stage (referred to as primary units) and are further broken down into smaller clusters (referred to as secondary units), which may be sampled randomly again. The units at each level are usually stratified by population, with the likelihood of selection for each unit being proportional to their population size. By the final stage, only a fairly small cluster is actually included in the sample. Multi-stage sampling can be extended to two, three, or more stages. II. Probability-Based Selection at Household Level 1. Kish Method: The interviewer lists all men in the household and orders them by decreasing age, then similarly lists all women. The interviewer then identifies a respondent by using a selection table with rotations of possible combinations of sex and age. This method is useful because it allows for truly random selection at the final stage of the sampling design. However, it is time consuming and difficult outside of face-to-face interviews. 147 2. Age-Order: The interviewer lists the household’s adults by age and then identifies a respondent by generate a random number (from one to the total number of individuals in the household) and choosing the individual’s name that corresponds to that number. It is not as complex as the Kish method because it does not organize the list by gender. 3. Full Enumeration: The interview itemizes each adult by name and then generates a random number. The interviewer selects the respondent whose name corresponds with the random number. This technique is not very prevalent in the literature. 4. Next Birthday: The interviewer selects the adult with the next birthday in the household. 5. Last Birthday: The interviewer selects the adult who had the last birthday in the household. It tends to be easier for respondents to identify the most recent birthday than the next birthday. III. Non-Probability Sampling 1. Convenience Sampling: Selection based on availability (e.g., students in a classroom, patients at a particular clinic on a particular day). 2. Quota Sampling: Selection based on established quotas of easily identified sub-sections of the population. Convenience samples are typically used to complete the sample (e.g., 25 males and 25 females). The quotas may be set so that the sample yields the proportions of the broader population. 3. Purposive Sampling: Selection based on a convenience sample from a population with a specific set of characteristics (e.g., students in a cafeteria in order to examine their eating habits). 4. Snowball Sampling: Selection based on the recommendations of other participants. The surveyor asks participants to identify other potential participants with a specific set of characteristics and then adds them to the sample. Resources: CDM – Executive Board. “General Guidelines for Sampling and Surveys for Small-Scale CDM Project Activities.” EB 50 Report: Annex 30. http://cdm.unfccc.int/Reference/Guidclarif/ssc/methSSC_guid20.pdf Gaziano, Cecilie. 2005. “Comparative Analysis of Within-Household Respondent Selection Techniques.” Public Opinion Quarterly. 69(1): 124-157. Groves, Robert M., Floyd J. Fowler, Jr., Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2009. Survey Methodology 2nd Edition. Hoboken, NJ: John Wiley & Sons. 148 DIMS Evaluation APPENDIX H. AMERICASBAROMETER STUDIES BY YEAR AND FUNDING SOURCE 149 APPENDIX I. DIMS CORE QUESTIONNAIRE 150 DIMS Evaluation 151 152 DIMS Evaluation 153 154 DIMS Evaluation 155 156 DIMS Evaluation 157 158 DIMS Evaluation 159 160 DIMS Evaluation 161 162 DIMS Evaluation 163 164 DIMS Evaluation 165 166 DIMS Evaluation 167 168 DIMS Evaluation 169 170 DIMS Evaluation 171 172 DIMS Evaluation 173 174 DIMS Evaluation 175 176 DIMS Evaluation APPENDIX J. LAPOP POLICIES ON SUBSCRIBER AND REPOSITORY FEES The fee structure for a data repository is $250 per STATA/SPSS dataset and $300 per ASCII dataset. For the 2010 data, the cost would be $6500- $7800 for all 26 countries (http://www.vanderbilt.edu/lapop/become-repository.php). The same charges apply to specialized surveys on specific countries or subpopulations which are only “available for purchase on an individual, repository basis.” The fee structure for subscriptions varies by type of institution (http://www.vanderbilt.edu/lapop/become-subscriber.php):34 - “If the Subscribing institution is located in the US, Canada, Japan, Western Europe or in a European Union member country, and if the Subscribing institution is a government institution, multi-lateral public institution, think tank, or a degree-granting institution of higher learning that offers a Ph.D. or Masters in political science (or government), international relations, or public policy, the subscription is an all-inclusive license fee of $1,500 fee per year. If the Subscribing institution is located in any other part of the world, or if the Subscribing institution is a degree-granting institution of higher learning that does not offer a Ph.D. or Masters in political science (or government), international relations, or public policy, the subscription is a flat $500 fee per year.” 34 Subscription rates as of 2011. Rates for individuals are identical to those for institutions. U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, DC 20523