PHOTO CREDIT: E+E ECONOMÍA Y ESTADÍSTICAS PARA EL DESARROLLO IMPACT EVALUATION OF THE DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT FINAL REPORT Prepared under Contract No. GS-10F-0033M/AID-OAA-M-13-00013, Tasking N018 This publication was produced for review by the United States Agency for International Development. It was prepared by Gustavo Setrini (Principal Investigator), Cyrus Samii (Co-Investigator, Methodology), and Cynthia González (Co-Investigator, Data Analysis) for NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | i DRG LEARNING, EVALUATION, AND RESEARCH ACTIVITY IMPACT EVALUATION OF THE DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT Final Report February 2020 Prepared under Contract No. GS-10F-0033M/AID-OAA-M-13-00013, Tasking N018 Submitted to: Brandy Witthoft, USAID COR Submitted by: Gustavo Setrini, Principal Investigator Cyrus Samii, Co-Investigator, Methodology Cynthia González, Co-Investigator, Data Analysis Contractor: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Renée Hendley Tel: 301- 634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | ii TABLE OF CONTENTS ACRONYMS...............................................................................................................................................................IV ACKNOWLEDGMENTS .........................................................................................................................................V EXECUTIVE SUMMARY........................................................................................................................................... 1 I. BACKGROUND ................................................................................................................................................. 3 Goals and Approach of the Inclusive Value Chain Project........................................................................ 3 Incorporating a Municipal Governance Focus.............................................................................................. 3 The Role of Municipalities in Economic Development and Governance Failures............................... 4 The Role of Farmer Organizations and Their Challenges......................................................................... 5 Clientelism and the Governance of Small-Farmer Value Chains............................................................. 6 II. THEORY OF CHANGE.................................................................................................................................... 8 III. METHODOLOGY............................................................................................................................................10 Intervention Design ..........................................................................................................................................10 Randomization Procedure...............................................................................................................................11 IV. BASELINE CHARACTERISTICS ...................................................................................................................14 V. ENDLINE SAMPLING AND DATA COLLECTION...............................................................................20 VI. RESULTS .............................................................................................................................................................21 Improving the Position of Producers vis-à-vis Municipal Authorities...................................................21 Improving the Position of Producers vis-à-vis Buyers..............................................................................23 Aligning Mayors’ Priorities with Producers’ Priorities.............................................................................24 Improving Distribution and Quality of District Services.........................................................................25 Quality of the Municipal Development Plans .............................................................................................26 Downstream Outcomes..................................................................................................................................26 Agricultural Inputs and Upgrading ..........................................................................................................................26 Agricultural Income......................................................................................................................................................27 VII. DISCUSSION AND LESSONS ......................................................................................................................28 ANNEX 1: TECHNICAL DETAILS OF RESEARCH DESIGN......................................................................29 ANNEX 2: EVALUATION TIMELINE.................................................................................................................35 ANNEX 3: PROJECT INVESTIGATOR PROFILES..........................................................................................37 ANNEX 4: IMPLEMENTING PARTNER PROFILE ..........................................................................................39 CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | iii LIST OF TABLES Table 1. Treated and Control Municipalities...............................................................................................12 Table 2. Pre-treatment Covariate Balance...................................................................................................14 Table 3. Allocation Outcomes for Pre-intervention Group Deliberation Exercises.........................16 Table 4. Deliberation Style Measures for Pre-intervention Group Deliberation Exercises.............18 Table A1. Basic Power Calculations .................................................................................................................32 LIST OF FIGURES Figure 1. Theory of Change ................................................................................................................................ 8 Figure 2. Map of Treated and Control Municipalities.................................................................................13 Figure 3. Effects on Producers’ Position vis-à-vis Municipal Authorities ...............................................22 Figure 4. Effects on Producers’ Position vis-à-vis Buyers...........................................................................24 Figure 5. Effects on Alignment between Mayors’ and Producers’ Priorities.........................................25 Figure 6. Effects on Prioritization of Rural Producers in Development Plans......................................26 CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | iv ACRONYMS CIRD Centro de Información y Recursos para el Desarrollo (Center for Information and Resources for Development) DRG-LER Democracy, Human Rights, and Governance Learning, Evaluation, and Research FECOPROD Federación de Cooperativas de Producción (Federation of Production Cooperatives) ICC Intra-Class Correlation IVC Inclusive Value Chains MDES Minimum Detectable Effect Size NGO Non-Governmental Organization NORC NORC at the University of Chicago RCT Randomized Controlled Trial STP Secretaría Técnica de Planificación (Technical Planning Secretariat) USAID United States Agency For International Development USD United States Dollars CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | v ACKNOWLEDGMENTS This report was written by Gustavo Setrini, Cyrus Samii, and Cynthia González for NORC at the University of Chicago. Lidia Valenzuela contributed with data preparation and implementation of the sampling plan, and Antonella Bandiera contributed analysis of the endline data. Fieldwork was carried out by NORC subcontractor E+E Economía y Estadísticas para el Desarollo under the direction of Cynthia González. Luis Camacho, Katharine Mark, Jeffrey Telgarsky, and Clifford Zines from NORC provided input on the research design and endline analysis. Rocío Duarte assisted with qualitative interviews, Nino Cricco contributed in assembling administrative data, and Pier Fuster and Nico Fernandez coded the development plans. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 1 EXECUTIVE SUMMARY As part of the Democracy, Human Rights, and Governance Learning, Evaluation, and Research (DRG-LER) Activity, the United States Agency for International Development (USAID) requested that NORC design and budget for an impact evaluation of the Democracy, Human Rights, and Governance (DRG) component of USAID/Paraguay’s Inclusive Value Chains (IVC) project, a multi￾year initiative implemented by the Federación de Cooperativas de Producción (FECOPROD) under a Cooperative Agreement. A confluence of negative formal and informal institutional incentives within municipal government and farmer organizations infuse the interactions among farmer organizations, local government, and private firms involved in small-farmer value chains with a strong clientelist dynamic. Farmers, their leaders, and the institutions that fund them use development resources to fuel the exchange of political support and short-term subsistence guarantees rather than to support long-run investment and growth. Clientelism and the misallocation of development resources are broadly generalized problems in developing countries. The IVC project and its municipal governance component are conceived as a set of interventions to alter this dynamic, improving development governance to better direct development resources toward profitable investments that increase the income and well-being of small farmers. More specifically, it seeks to increase the technical capacities of municipal and farm organization leaders and increase democratic accountability in the exchanges between farmers and their leaders and between farmer organizations and municipalities. The evaluation considers institutional strategies for promoting responsiveness to smallholder constituents and public goods provision by municipal leaders. In particular, we ask: What is the effect of a collective participatory budgeting campaign on public goods provision to rural constituents? We hypothesize two main mechanisms through which the participatory budgeting process could increase public goods provision. The first is by democratizing farmer and municipal government interactions, through which participatory budgeting equalizes the power of rural constituent groups (that is, producers’ organizations) over municipal budget decision-making. The second is by institutionalizing the interactions between farmers, buyers, and municipal governments, through which informal two-way interactions give way to public, formal negotiation that includes all three parties. Each of these mechanisms serves to improve the position of producers vis-à-vis municipal (i.e. district) authorities and buyers. Beyond this first order interest in public goods provision, the evaluation also seeks to address the second order question of whether integrating a governance dimension can enhance the effect of the IVC project on rural producers’ income and well-being. Thus, a secondary evaluation question is whether participatory budgeting, and any benefits that result, translate into increased access to markets and incomes among rural producers. We conducted a randomized controlled trial (RCT) to investigate these questions. We randomly selected a treatment group of 16 municipalities from the IVC project’s population of 32 municipalities and the participating farmer organizations within them. A subcontractor (CIRD) implemented a governance intervention with these 16 municipalities, and the other 16 served as a control group that did not receive the governance intervention. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 2 The intervention took place over approximately two years from 2016-2018, and consisted of 1) training in participatory planning, 2) a participatory rural development planning exercise facilitated by the secretary of agriculture of each municipality, 3) and support to municipalities to set up their municipal development councils and to create and designate a “Secretary of Production” within the mayor’s office to be in charge of the planning process and of agricultural development policy in general. To measure the effects of our intervention, we conducted two endline producer-household surveys in the treatment and control groups. In addition, we conducted a short survey of the complete population of 32 mayors at the time of the second endline producer-household survey (about one year following the end of the intervention) to elicit mayors’ priorities and budgeting outcomes. We used qualitative interviews with buyers to assess the development of any new relationships with producers, and we obtained the municipality development plans from each of the 32 municipalities and coded them according to a rubric that assessed prioritization of needs of small rural producers. Our study found significant positive effects of the intervention in terms of improving the position of producers vis-à-vis municipal authorities, improving the position of producers vis-à-vis buyers, aligning mayors’ priorities with producers’ priorities, and improving the quality of the municipal development plans. We found no significant effects for the indirect downstream results, such as the distribution and quality of district public services, the upgrading of agricultural production, or agricultural income. Beyond the results reported for specific indicators, this study points toward two broader lessons. First, much of the emphasis in development programing and the development literature on participation (for example, the now vast amount of work done on participatory budgeting) focuses on the determinants, the processes, and the effects of individual citizen participation. To the extent the intervention succeeded in boosting the political participation of small farmers and democratizing relations between farmers and local government, it did so by engaging farmer organizations. These organization articulate farmer interests collectively and through the mediation of their leaders. Second, the role of the development NGO, CIRD, was most likely crucial in achieving this outcome through its ability to persuade mayors to open up this space for dialogue and undertake institutional changes that permit greater participation and responsiveness. It is not clear that producers could have achieved similar results on their own, without the financial, political, and knowledge resources that an NGO with many years of experience in development work can offer to a municipal government. In conclusion, a “collective action” model for citizen participation, such as the one that came into focus through this research project, can improve the position of small producers and their interests in the process of municipal planning. While these changes in political participation and representation did not produce any immediate changes in farmers’ incomes, it is probably too soon to draw conclusions about these indirect effects. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 3 I. BACKGROUND As part of the Democracy, Human Rights, and Governance Learning, Evaluation, and Research (DRG-LER) Activity, the United States Agency for International Development (USAID) requested that NORC design and budget for an impact evaluation of the Democracy, Human Rights, and Governance (DRG) component of USAID/Paraguay’s Inclusive Value Chains (IVC) project, a multi￾year initiative implemented by the Federación de Cooperativas de Producción (FECOPROD) under a Cooperative Agreement. GOALS AND APPROACH OF THE INCLUSIVE VALUE CHAIN PROJECT The IVC project is designed to address rural poverty and inequality by improving the incomes of small farmers through their incorporation into agro-industrial supply chains in Paraguay’s northern region. The project operates in the departments of San Pedro, Concepción, Amambay, and Canindeyu and aims to reach 20,000 small farmers. The project’s approach has four defining characteristics: • A Focus on Inclusion: the expected outcome of the current project is to increase small producers’ incomes, rather than increase private-sector sales. This requires mechanisms to help small farmers overcome the multiple barriers they face in establishing relationships with agro-industrial buyers. Conversely, it requires mechanisms that reduce the elevated transaction costs that agro￾industrial buyers face when sourcing from small farmers relative to larger agricultural enterprises. • A Market-driven Focus: the project aims to identify already existing and unmet market demand and work backward, designing and implementing development interventions to allow small farmers to overcome supply constraints. This means supporting farmers to produce what can be sold, as opposed to helping them sell what they produce. • A Focus on Strengthening Collective Action within Farmer Organizations: the project focuses on strengthening the capacities of farmers to cooperate within formal organizations to manage the investment, production, and commercial processes required to become suppliers of agro￾industrial enterprises. • A Focus on Improving Coordination between Farmers and “Anchor Firms:” the project identifies agro￾industrial processors and exporters as anchor firms with unmet demand for raw materials that small farmer organizations are capable of providing. It then works to coordinate the flow of information, knowledge, and resources between these companies and farmer organizations to bring the demands of companies into alignment with the production of small farmers. INCORPORATING A MUNICIPAL GOVERNANCE FOCUS Creating and consolidating inclusive value chains requires addressing a series of barriers related to local governance. In particular, municipal governments are crucial providers of public goods and investment resources that can potentially complement and attract private-sector investment toward small farmer development. For this reason, USAID/Paraguay made an additional USD 1 million available to incorporate a municipal governance-strengthening intervention into the project. NORC’s investigators co-designed the intervention with the Centro de Información y Recursos para el Desarrollo (CIRD), a local non-governmental organization (NGO) that was also responsible for implementing the governance intervention under a subcontract with FECOPROD. The investigators and CIRD designed the governance intervention to be evaluated using an RCT and as a complement and an extension of the activities already planned by FECOPROD. In the initial design, CIRD would CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 4 provide support to producer organizations and municipal governments to carry out a participatory planning process with municipal development councils. Control municipalities would not carry out any participatory planning process. However, in the course of designing the intervention and the RCT, Presidential Budget Decree number 4774 (January 2016) made local development planning mandatory, establishing that the Ministry of Finance would use the plans to audit local expenditures made with budget transfers from the National Fund for Public Investment and Development (FONACIDE) to municipal governments. The decree also established that the Technical Planning Secretariat (Secretaría Técnica de Planificación, STP) would define procedures and provide guidance for the creation of these plans, ensuring that they aligned with the priorities defined by the 2030 National Development Plan, while also employing a participatory process.1 In this context, our study examines the impacts of the governance intervention given the mandatory requirement of local development planning. This may have reduced our intervention’s effect size, given that control municipalities now had incentives to produce development plans. On the other hand, the requirement made for an interesting comparison, by shifting our focus from the effects of planning itself, to the effects of the support given to producers and municipalities to make the planning process meaningfully participatory. THE ROLE OF MUNICIPALITIES IN ECONOMIC DEVELOPMENT AND GOVERNANCE FAILURES Paraguay has undergone a substantial degree of decentralization over the last decades, receiving new resources as well as new responsibilities for economic development. In addition to local property tax revenue, municipalities now receive a direct budget allocation from the royalties earned by the two bi-national hydroelectricity authorities. Moreover, departmental and municipal governments are key actors in allocating and distributing budgetary resources that come from the central government and the various national-level ministries that fund and implement economic development initiatives. Decentralization received further formal legal backing with a revised municipal organic law, which in 2010 granted municipalities new responsibilities for economic development including providing technical assistance and support for small and medium enterprises, contributing to national, regional, and local development planning, and specifically developing employment plans in coordination with national-level ministries. This set of changes means that municipalities have both new legal imperatives and new fiscal means to support local economic development. At the same time, municipal government must overcome a difficult set of challenges so that these resources translate into broad improvements in the incomes and livelihoods of citizens. Three factors may impede the effective and efficient allocation of development resources at the local level. First, municipal governments generally lack planning skills and technical capacities to allocate resources to the most developmentally valuable investments. Many municipalities’ structures consist of no more than the mayor, the city council, and one or two administrative and custodial staff. Only the larger and more developed municipalities have program staff dedicated to agriculture or economic development. Moreover, even where permanent, dedicated staff exist, they may lack the technical skills and knowledge to accurately judge the costs, benefits, and risks of alternative investments and to coordinate complex transactions within global value chains. 1 Setrini, Gustavo, and Liliana Rocío Duarte-Recalde. “The development of participatory institutions in Paraguay: a tool for democratization or state control?” Cahiers des Amériques Latines 90 (2019): 39-57. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 5 Second, municipal governments may misallocate resources because they are insufficiently accountable to their constituents or insufficiently informed about citizen priorities. Constituents seek individual private accountability from their elected officials as opposed to collective public accountability. Consequently, elected officials use development resources to distribute individual benefits and secure short-term political support, as opposed to investing in public goods that generate long-term growth. As a result, development resources are stolen or used clientelistically and diverted from their most developmentally rational use. Finally, media reports suggest that an informal, illicit political economy that is sustained by marijuana cultivation and drug trafficking has arisen in northern Paraguay, providing a barrier to improved development governance. This illicit economy is highly stable and profitable relative to existing formal economic opportunities that are available to economic and political actors in rural and agricultural regions. It absorbs private resources, specifically land and capital, and generates illicit profits which in turn provide a source of immediate payoffs for public officials. In addition to subverting the rule of law, this illicit economy creates a major disincentive for municipal governments to invest in other forms of economic development. For elected officials, the investment and administrative costs of developing new formal agro-industrial value chains are high and the economic and political payoffs are uncertain when compared to maintaining the already consolidated and high-value marijuana value chain. THE ROLE OF FARMER ORGANIZATIONS AND THEIR CHALLENGES Farmer organizations play a key role in consolidating small-famer value chains, serving as vehicles for economic and political collective action among smallholder farmers. First, farmer organizations allow farmers to reach economies of scale in production and commercial activities. They may, for example, coordinate individual (farm-level) investments and pool capital to undertake collective investments in productive and administrative infrastructure. Such activities lower production and commercialization costs and support collective efficiency and competitiveness among an organization’s members. Second, farmer organizations aggregate the interests and articulate the needs of their members to the external actors with whom farmers engage in economic and political exchanges. Conversely, they mediate and articulate external economic and political demands back onto their members and may organize collective responses. Chief among these external actors are private firms within agro-industrial value chains. These firms are typically monopsonistic (i.e., they face little or no competition as buyers of agricultural raw materials). This gives them the power to set prices and impose costs and risk onto suppliers. This is particularly attractive where suppliers are cash-strapped or risk averse and wish to avoid fixed capital investments. On the other hand, agro-industrial firms are under increasing pressure to meet escalating demands from their clients for product quality, environmental protection, and social responsibility. This makes a reliable supply base an important competitive advantage and creates incentives for investing in suppliers. Thus, farmer-firm relationships are often ambiguous, ranging from competitive to collaborative. Where they exist, farmer organizations are a key institution and their leaders are key actors in mediating this relationship and coordinating transactions between farmers and firms. In Paraguayan agricultural value chains, political and informal means of coordination (negotiation and informal agreements among organized farmers) are becoming more common than individual spot￾market (arm’s-length) transactions. Legal forms of coordination (i.e., individual or collective contracts) are still rare. In this process of negotiation and coordination, farmer organizations can CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 6 serve both as a source of countervailing power vis-à-vis monopsonistic firms, as well as a means of disciplining the supply base to the demands of buyers. Second, farmer organizations aggregate and articulate farmers’ demands and interests vis-à-vis the multiple public sector institutions that provide development resources. These include municipalities, departmental governments, the Ministry of Agriculture and Livestock, and donor agencies. They provide a range of benefits such as technical assistance, subsidized credit, agricultural inputs, and production and processing equipment. Most of these programs require smallholder farmers to be organized into committees or associations to qualify for assistance. Overall, public investment in small farm development is low. However, smallholder farmer organizations play an important role in shaping how these resources are used. There are three reasons farmer organizations may misallocate development resources. First, they may make poor use of development resources because their leaders are not held accountable for their decisions. This may allow leaders to use organizational resources in self-interested ways, either appropriating them or directing them toward personal as opposed to collective ends. Democratic participation in farmer organizations’ decision-making processes may be low because members lack sufficient market and political information to form and articulate rational preferences about the use of development resources and to hold their leaders accountable. Alternatively, democratic participation in farmer organizations’ decision-making processes may be low because widespread authoritarian norms of deference to leaders discourage open deliberation and debate over the use of resources even in the presence of good information. Second, farmer organizations may misallocate resources because their leaders lack the technical and managerial skills to accurately judge the costs, benefits, and risks of alternative investments and to coordinate complex transactions within agricultural value chains. In this case, the problem is not the self-interested use of resources by leaders, but instead the technical barriers they face in making rational decisions about their use. Third, farmer organizations may misallocate resources because of the subsistence risks faced by their members. Climate variability, crop disease, and market volatility make crop production a risky investment, particularly when they involve crops, production techniques, and markets that are unfamiliar to farmers. Moreover, development initiatives that require collective economic action (such as pooled investments or collective commercialization) introduce an additional set of risks. Having observed the failure of many similar schemes, rational farmers are aware of these risks and accurately estimate the likely payoffs of many small-farmer development initiatives to be low. In the absence of social welfare policies that guarantee a minimal level of subsistence income, poor farmers may prefer to receive a stream of individual material benefits and look to leaders to provide access to development projects as a source of individual benefits to meet short-term needs, rather than as a source of investment capital for collective infrastructure and long-term growth. If this is the case, then increased democratic participation in organizational decision-making may worsen rather than improve the allocation of resources toward developmental ends. CLIENTELISM AND THE GOVERNANCE OF SMALL-FARMER VALUE CHAINS The confluence of governance pathologies within municipal government and farmer organizations infuse the interactions among farmer organizations, local government, and private firms involved in small-farmer value chains with a strong clientelistic dynamic. Farmers, their leaders, and the institutions that fund them use development resources to fuel the exchange of political support and short-term subsistence guarantees rather than support long-run investment and growth. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 7 Clientelism and the misallocation of development resources are broadly generalized problems in developing countries. It is widely believed that this governance problem prevents the efficient use of public resources and the effective supply of public goods and services. This, in turn, drives poor economic performance and reinforces inequality by concentrating resources among the politically influential and preventing them from reaching the poor. At the same time, as in Paraguay, fiscal decentralization and the fragmented governance structure of contemporary economic development projects give local actors all over the developing world increasing influence over the use of development resources. The IVC project and the municipal governance component are conceived as a set of interventions to alter this dynamic, improving development governance to better direct development resources toward profitable investments that increase the income and well-being of poor farmers. More specifically, it seeks to increase the technical capacities of municipal and farm organization leaders and increase democratic accountability in the exchanges between farmers and their leaders and between farmer organizations and municipalities. The project aims to address these goals in a way that creates impacts that are measurable and quantifiable using an RCT. In turn, employing this methodology will permit the research team to address a broader set of theoretical questions about whether participatory planning processes, when coupled with capacity building at the municipal level, can curb the clientelistic use of public resources and improve economic development outcomes. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 8 II. THEORY OF CHANGE The evaluation considers institutional strategies for promoting responsiveness to smallholder constituents and public goods provision by municipal leaders. In particular, we ask: What is the effect of a collective participatory budgeting campaign on public goods provision to rural constituents? Figure 1 displays our theory of change, which is based on the intervention and existing political science theories.2 The location of the governance intervention is shown in green. The key outcomes that the governance intervention targets are shown in blue. Potential moderating factors, which the discussion above emphasized, are shown in gray. Then, downstream outcomes that are of interest to the broader IVC project are shown in brown. Figure 1. Theory of Change3 We hypothesize two main mechanisms through which the participatory budgeting process could increase public goods provision. The first is by democratizing farmer and municipal government interactions, through which participatory budgeting equalizes the power of rural constituent groups (that is, producers’ organizations) over municipal budget decision-making. The second is by institutionalizing the interactions between farmers, buyers, and municipal governments, through which informal two-way interactions give way to public, formal negotiation that includes all three parties. Each of these mechanisms serves to improve the position of producers vis-à-vis municipal (i.e. district) authorities and buyers. This is the first step in the hypothesized causal chain. The democratization of farmer-municipality interactions could improve public goods provision through the alignment of mayor’s priorities with those of producers, the second step in the causal 2 For a useful synthesis of recent theoretical and empirical research, see Jonathan A. Fox (2015), “Social Accountability: What Does the Evidence Really Say?” World Development 72:346-361. 3 This theory of change incorporates some simplifications and modifications relative to the design document, based on an updated understanding of the context of the intervention, as described above. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 9 chain. This could occur through a variety of mechanisms. First, it may alter the efficiency calculus by which municipal leaders decide the most efficient way to use public resources. Facing a much broader set of interests, mayors may come to view public goods provision, as opposed to the distribution of individual and private goods, as an efficient means of satisfying their constituents. Second, the participatory budgeting process may generate information about farmers’ needs in their municipality. This allows municipal leaders to target such needs with their expenditures. Third, the participatory budgeting forums will be conducted in a manner that essentially promotes an open discourse of broad-based development and public goods provision. This may alter the norms that govern political demand-making, reducing clientelism by encouraging mayors to make commitments in public. The institutionalization of buyer-farmer-municipality interactions could improve the information shared among these three groups about the needs and resources available for rural development. It may also induce buyers to provide resources and public goods directly to localities that are home to farmers in their supply chain. Clearly, these different mechanisms can interact as well. The end result of this alignment is hypothesized to be an improvement in the distribution and quality of municipal services with respect to the needs of producers. Beyond this first order interest in public goods provision, the evaluation also seeks to address the second order question of whether integrating a governance dimension can enhance the effect of the IVC project on rural producers’ income and well-being. Thus, a secondary evaluation question is whether participatory budgeting, and any benefits that result, translate into increased access to markets and incomes among rural producers. We must also consider the possibility of adverse effects. For example, bringing producers into the participatory budgeting process may cause them to relinquish the use of conventional, and potentially more practical, means of influencing municipal leaders. This could free leaders to pursue their particularistic interests at the expense of the community. Hetherington, for example, documents how Paraguayan campesinos’ submission to the “pro-transparency” and “anti-corruption” agenda to deal with the problem of misappropriated land in the 1990s and early 2000s took the steam out of the movement to pursue broad-based land reform.4 Such perverse consequences should never be ruled out, although our intention is to work with the implementing NGO to minimize this possibility. We hypothesize that these various effects will be moderated by three factors, based on our discussion above about the context. The first is farmer organizations’ institutions, and in particular their cohesiveness, inclusiveness, and civic mindedness, which could affect the intensity and broad￾based nature of the pressure that farmer organizations will put on municipal mayors as a result of their participation in the budgeting process. The second is the mayor’s political circumstances and developmental motivation, and in particular the extent of local competition he faces, whether he is aligned with the departmental and national leadership, and his background in working for development-related causes, all of which may affect the extent to which he will feel the need to be responsive to farmer-constituent concerns. The third is the municipality’s institutional capacity, which may affect the ability of the mayor to respond to increased farmer-constituent pressure by increasing public goods. These three moderating factors appear as the gray boxes in Figure 1. 4 Hetherington, Kregg. Guerrilla auditors: the politics of transparency in neoliberal Paraguay. Duke University Press, 2011. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 10 III. METHODOLOGY The methodology for this impact evaluation involved three steps. First, the investigators co-designed the intervention with implementing partner CIRD. Annex 4 provides background on CIRD. Second, the project investigators randomly selected a treatment group of 16 municipalities from the IVC project’s population of 32 municipalities and the participating farmer organizations within them. CIRD implemented the governance intervention with these 16 municipalities, and the other 16 served as a control group that did not receive the governance intervention. Third, the project investigators gathered baseline administrative data and then two waves of endline surveys with samples of producers’ households, the municipal mayors, and then administrative data from the treatment and control groups to estimate the impact of the governance intervention. Annex 2 details the implementation timeline for the intervention and impact evaluation. INTERVENTION DESIGN Based on findings from a pre-intervention scoping trip, observations of the project sites, and a co￾design workshop with CIRD, the project investigators decided on the following components for the intervention: 1. Pre-intervention workshops (Oct-Nov 2015, Jan 2016) in both treatment and control groups. Workshops were held with farmer groups in 28 of the 32 targeted municipalities in October and November 2015, prior to the 15 November 2015 municipal elections. They were provided with information about the functions and responsibilities of municipal government and asked to prioritize ten different potential municipal functions related to agricultural development (ranging from private goods, like input distribution, to public goods, like road maintenance). Participants were given individual entry and exit surveys to measure their individual preferences. They were also asked to engage in a collective prioritization exercise, assigning 100 budget ‘points’ among the ten selected areas. The deliberation exercise was recorded and transcribed, and facilitators were asked to fill out an instrument to give a qualitative assessment of the discourse and interaction. This exercise gave the evaluation team data on producers’ individual preferences, their correspondence with the collectively expressed preferences, authority relations among producers as they are manifest in deliberation, and on how preferences change as a consequence of deliberation. The results indicated strong preferences for three areas of public investment by the municipality: road maintenance and upgrading, rural electrification, and agricultural extension services. These insights were used to fine-tune the endline measurement strategy. In the month of January 2016, CIRD conducted workshops with the newly elected mayors in both treatment and control localities to discuss how they prioritize municipal investments based on their perception of their constituents’ needs and preferences. The mayors were also informed about the IVC Project (which was occurring in all municipalities), and the randomized selection of municipalities for the governance intervention at these workshops. This ensured consent at the mayoral level and also helped to control for any changes to mayors’ behavior that may have been due to merely having been aware of CIRD’s presence, as distinct from the actual intervention components. Data on mayors’ professional backgrounds, their staffing, and other capacity indicators were also collected for use in blocking the randomized selection of the treatment and control groups. The remaining activities took place only in the treatment group and constitute the core of the governance intervention. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 11 2. Training in Participatory Planning (Mar-May 2016). CIRD provided farmer organizations with training to improve their capacities to meaningfully participate in the nationally mandated participatory planning process (see the discussion above regarding the law mandating participatory planning). This included a) training on the functions and responsibilities of local government with regard to rural development, b) training on citizenship rights to participation, and c) basic technical training related to rural road maintenance, rural electrification, and rural extension services so that farmers understand the range of technically feasible options. CIRD also provided a parallel set of training to municipal officials (mayors or their delegates) to provide them with the basic skills (budgeting, technical knowledge) necessary to participate in the mandated planning process. 3. Participatory Rural Development Planning Exercise (Jun-Sep 2016). As described above, in early 2016, the Government of Paraguay passed a law mandating all municipalities to come up with participatory development plans by June 2016. The law also established guidelines for the planning process and tied the disbursement of specific transfers to the completion of these plans. These plans would determine the allocation of funds in the subsequent budgets. CIRD’s work plan was tailored to the law and its timeline. To support this planning process, CIRD facilitated the creation of rural development tables (mesas de desarrollo rural) where they did not exist or their reactivation where they existed only nominally. These tables included representatives from the municipal government, the local offices of national-level ministries, and of local civil society. The tables convened a participatory planning process with the goal of defining priorities for the municipal rural development plan that municipalities were mandated to furnish to the STP. 4. Support to Municipalities (2016-2017). CIRD provided support to the treatment municipalities to set up their municipal development councils and to create and designate a “Secretary of Production” within the mayor’s office to be in charge of the planning process and of agricultural development policy in general. RANDOMIZATION PROCEDURE The impact evaluation used a municipality-level, blocked cluster-randomized design, with blocking on moderator variables as well as other important social and development indicators. We used municipality, organization, and farmer covariates to pair-match municipalities, and then used a restricted randomization to assign treatments within the matched pairs. The baseline data came from FECOPROD’s 2015-16 survey of producers, pre-intervention data described above, and various administrative data. The restricted randomization allowed only assignment profiles that maintain a high level of overall covariate balance.5 Table 1 shows the treated and control municipalities, along with the number of IVC project organizations within each municipality and then the average number of producers per organization. Figure 2 is a map of the locations of these municipalities. 5 For discussion of such “restricted randomization” methods, see Miriam Bruhn and David McKenzie (2009) “In Pursuit of Balance: Randomization in Practice in Development Field Experiments,” American Economic Journal: Applied Economics 1(4):200-232. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 12 Table 1. Treated and Control Municipalities DEPARTMENT DISTRICT BLOCK ASSIGNMENT NO. IVC ORGS. AVG. PROD/ORG. Amambay Bella Vista 14 Treated 4 28 Amambay Pedro Juan Caballero 15 Control 10 46 Amambay Zanja Pyta 11 Control 2 12 Canindeyu Curuguaty 13 Control 5 63 Canindeyu Jasy Kany 1 Control 8 45 Canindeyu Villa Ygatimi 10 Control 2 92 Canindeyu Yvy Pyta 10 Treated 4 12 Concepcion Azotey 11 Treated 3 28 Concepcion Belen 9 Treated 5 29 Concepcion Concepcion 15 Treated 6 26 Concepcion Horqueta 13 Treated 8 26 Concepcion Loreto 9 Control 2 36 Concepcion Yby Yau 7 Control 6 23 San Pedro 25 de diciembre 8 Treated 3 69 San Pedro Antequera 6 Treated 1 19 San Pedro Capiibary 12 Treated 7 47 San Pedro Chore 2 Treated 4 58 San Pedro Gral. E. Aquino 2 Control 7 57 San Pedro Gral. Resquin 4 Control 4 80 San Pedro Guayaivi 8 Control 5 68 San Pedro Itacurbi del Rosario 16 Control 2 111 San Pedro Liberacion 4 Treated 3 24 San Pedro Lima 3 Treated 7 54 San Pedro Nueva Germania 5 Treated 5 30 San Pedro San Estanislao 7 Treated* 5 48 San Pedro San Pablo 1 Treated 2 19 San Pedro San Pedro del Ycuamandyyu 16 Treated 11 49 San Pedro Santa Rosa del Aguaray 12 Control 7 74 CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 13 DEPARTMENT DISTRICT BLOCK ASSIGNMENT NO. IVC ORGS. AVG. PROD/ORG. San Pedro Tacuati 5 Control 4 18 San Pedro Union 14 Control 1 44 San Pedro Yataity del Norte 6 Control 1 55 San Pedro Yryvukua 3 Control 4 76 * In San Estanislao, the program was rejected. Nonetheless, the analysis uses an “intention to treat” approach to incorporate San Estanislao into the average for those assigned to treatment. Figure 2. Map of Treated and Control Municipalities CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 14 IV. BASELINE CHARACTERISTICS Table 2 displays means and standard deviations for pre-treatment covariates for treated and control municipalities. We see that study municipalities have, on average, about 200-300 producer households and total populations around 25,000 people (though with a high level of variation). Producer organizations score toward the middle of FECOPROD’s organizational capacity index (maximum value of 60). A slight majority of the municipalities have mayors aligned with the national incumbent party and the average margin of victory is 16-17 percentage points, suggesting moderately high margins of victory on average, although here too we have a good deal of variation (ranging from less than one percentage point to more than 60). The municipalities exhibit moderate levels of industrialization on average (the figures imply on the order of a few hundred small enterprises per municipality), and they are diverse in terms of their agricultural outputs—this diversity is captured by the fact that the agricultural factors scores, which measure different clusters of agricultural activities, are all well balanced. The figures in the table imply that annual municipal budgets are, on average, on the order of 4 to 5 billion Guaraníes (equivalent to about USD 700,000 to 900,000 based on the current exchange rate). Table 2. Pre-treatment Covariate Balance CONTROL N=16 DISTRICTS TREATMENT N=16 DISTRICTS MEAN SD MEAN SD P - VALUE Number of producer households in district 258.94 213.46 273.69 309.82 0.88 Median FECOPROD organization score (0-60) 31.44 5.89 33.56 5.73 0.31 Mayor’s margin of victory 0.17 0.16 0.16 0.13 0.78 Population 28113 26213 25835 23159 0.80 Mean producer HH distance to main road (in UTM degrees) 15432 14108 10925 12659 0.35 Alignment of mayor to national incumbent party 0.63 0.50 0.56 0.51 0.73 Agricultural product factor 1 score -0.11 0.79 0.11 1.17 0.55 Agricultural product factor 2 score -0.24 0.91 0.24 1.02 0.17 Agricultural product factor 3 score -0.03 1.12 0.03 0.85 0.85 Agricultural product factor 4 score -0.20 0.89 0.20 1.02 0.26 Log(Number of small enterprises) 5.78 1.02 5.87 0.99 0.81 Log(Annual district budget) 8.45 0.74 8.39 0.68 0.81 Index of availability of district services (z-score) -0.12 0.96 0.12 1.06 0.51 CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 15 Table 3 and Table 4 present results from the pre-intervention deliberation and spending prioritization exercises, which were used to measure both producers’ preferences over municipal spending priorities as well as the degree of cohesion and inclusiveness in producer organizations’ decision-making. Table 3 shows how producers in the different municipalities chose, as a group, to allocate hypothetical resources over municipal spending items. The dominant interest, apparent when one looks at the mean values reported at the bottom of the table, is investment in roads. Investment in agricultural machines comes second, and then priorities are quite diverse beyond that. The “prioritization measures” on the right of the table use a Herfindahl index to measure the extent to which the producers prioritized one or another spending item, rather than spreading the resource allocation evenly across all categories. This is measured in terms of the group allocation that resulted from the deliberation exercise as well as the individual allocations as measured through entry and exit surveys administered to participants before and after the deliberation. The index varies between zero and one, with one indicating concentration of all attention onto one item, and zero indicating even spread onto all items. The group level values tend to be quite low, indicating a lack of very strong prioritization. The individual level measures are even lower, and there is no indication that going through the deliberation exercise increases concentration on a few priority items. The last two columns measure the extent to which the allocation decided upon by the group corresponds to the priorities indicated by producers on the entry and exit surveys. The measure itself does not have a straightforward interpretation, but rather will be used, in future analyses, as a way of ranking communities in terms of the correspondence between group- and individual-level preferences. Table 4 shows various measures of deliberation style, which were meant to capture the inclusiveness of the deliberation process in each municipality. We scored the deliberations in terms of the method of assent (with verbal consent receiving a lower score than actual show of hands), extent of participation, and level of agreement. Measures of these styles of consent were taken at two moments in the deliberation: the initial discussion of how to modify the priorities and the final approval of the list of priorities. The findings suggest that both at the start and end of the deliberation, assent tended to be given verbally but that participation tended to be quite inclusive (typically including all individuals present or all leaders at least, rather than only some of the leaders present). Levels of agreement were moderate when deciding on initial modifications but grew to be closer to unanimity by the time the final allocation was reached. The deliberations tended to last about 40 minutes. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 16 Table 3. Allocation Outcomes for Pre-intervention Group Deliberation Exercises Municipality Treat- ment status Group deliberation allocation Total Prioritization measures Preference divergence credit agri￾cultural inputs agri￾cultural mach- ines agri￾cultural exten- sion public infra- struc- ture roads trans￾port electr- icity private partner- ship public partner- ship Group Herfin- dahl index* Mean Entry Survey Herfin- dahl Index* Mean Exit Survey Herfin- dahl Index* Mean Entry Survey Distance Score** Mean Exit Survey Distance Score** Curuguaty Control 3 19 16 5 15 30 5 3 2 2 100 0.18 0.10 0.09 0.29 0.28 General Aquino Control 0 35 0 0 0 45 20 0 0 0 100 0.37 0.12 0.27 0.50 General Resquin Control 5 3 30 10 10 20 7 5 5 5 100 0.17 0.11 0.12 0.24 0.23 Itacurubi del Rosario Control 0 0 25 0 25 50 0 0 0 0 100 0.38 0.11 0.05 0.52 0.53 Jasy Kañy Control 10 0 40 0 0 50 0 0 0 0 100 0.42 0.11 0.10 0.56 0.55 Loreto Control 10 3 10 20 2 25 1 25 2 2 100 0.19 0.11 0.11 0.30 0.26 Pedro Juan Caballero Control 0.25 10 25 10 14.25 20 5 15 0.25 0.25 100 0.17 0.08 0.11 0.27 0.24 Santa Rosa Aguaray Control 15 5 15 10 15 10 10 5 5 10 100 0.12 0.10 0.10 0.12 0.11 Tacuati Control 5 5 25 10 15 15 5 5 10 10 105 0.13 0.10 0.10 0.17 0.18 Ybi Yau Control 5 5 25 5 10 35 5 3 2 5 100 0.21 0.10 0.07 0.33 0.32 Yrybucua Control 0 0 5 60 0 35 0 0 0 0 100 0.49 0.08 0.07 0.61 0.62 Azotey Treated 3 15 18 10 15 25 5 3 3 3 100 0.16 0.13 0.08 0.23 0.24 Belen Treated 1 3 1 2 3 35 3 25 25 2 100 0.25 0.19 0.15 0.40 0.36 Bella Vista Treated 2 25 30 5 10 15 10 1 1 1 100 0.20 0.10 0.12 0.31 0.26 Caapiibary Treated 0 0 30 0 0 50 0 20 0 0 100 0.38 0.12 0.14 0.49 Chore Treated 0 0 0 0 35 30 0 0 35 0 100 0.34 0.11 0.09 0.49 0.47 Concepción Treated 8 2 7 25 3 20 5 25 4 1 100 0.18 0.17 0.11 0.29 0.28 CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 17 Municipality Treat￾ment status Group deliberation allocation Total Prioritization measures Preference divergence credit agri￾cultural inputs agri￾cultural mach￾ines agri￾cultural exten￾sion public infra￾struc￾ture roads trans￾port electr￾icity private partner￾ship public partner￾ship Group Herfin￾dahl index* Mean Entry Survey Herfin￾dahl Index* Mean Exit Survey Herfin￾dahl Index* Mean Entry Survey Distance Score** Mean Exit Survey Distance Score** Horqueta Treated 0.5 9.5 23.5 15 20 25 5 0.5 0.5 0.5 100 0.19 0.10 0.09 0.30 0.26 Liberación Treated 0 0 30 0 0 40 0 0 30 0 100 0.34 0.14 0.12 0.49 0.50 Lima Treated 10 5 5 5 10 15 10 15 15 10 100 0.12 0.10 0.09 0.14 0.14 Nueva Germania Treated 10 0 20 20 20 30 0 0 0 0 100 0.22 0.16 0.15 0.35 0.35 Puerto Antequera Treated 25 2 3 20 1 25 3 2 10 10 101 0.18 0.08 0.11 0.29 0.28 San Pablo Treated 10 5 20 3 2 20 5 10 20 5 100 0.15 0.10 0.10 0.23 0.25 San Pedro Treated 2 3 20 20 3 20 5 2 20 5 100 0.17 0.11 0.11 0.26 0.26 Ybypyta Treated 0 0 20 0 0 45 35 0 0 0 100 0.37 0.10 0.10 0.50 0.52 Sargento José Felix Lopez (N/A) 5 15 20 5 12 25 10 2 3 3 100 0.16 0.10 0.08 0.24 0.24 Control Mean 4.84 7.73 19.64 11.82 9.66 30.45 5.27 5.55 2.39 3.11 0.26 0.10 0.11 0.36 0.33 Control SD 5.01 10.59 11.68 17.07 8.25 13.87 5.87 7.78 3.15 3.89 0.13 0.01 0.06 0.17 0.17 Treated Mean 5.11 4.96 16.25 8.93 8.71 28.21 6.14 7.39 11.68 2.68 0.23 0.12 0.11 0.34 0.32 Treated SD 7.03 7.19 10.97 9.19 10.45 10.85 8.94 9.72 12.38 3.56 0.09 0.03 0.02 0.12 0.11 Diff. in Means p-value 0.91 0.47 0.47 0.62 0.80 0.66 0.77 0.60 0.01 0.78 0.60 0.03 0.78 0.80 0.84 Source: Authors’ calculations based on data collected at group deliberation events facilitated by CIRD. * For the Herfiindahl index, higher scores mean a higher level of concentration on one or a few priorities, whereas lower scores mean allocations are more scattered. ** The distance scores use Euclidean distance to measure how much survey respondents’ prioritizations differ from the group deliberation outcome. Higher scores mean that on average, preferences differed more, implying that the group deliberation outcome was less representative of members’ preferences than would be the case for lower scores. Empty entries in the table mean that necessary data to compute the entry were not collected. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 18 Table 4. Deliberation Style Measures for Pre-intervention Group Deliberation Exercises Municipality Treat￾ment status Method of Assent for Initial Modification Extent of Participation in Initial Modification Level of Agreement for Initial Modification Method of Assent for Final Proposal Extent of Participation for Final Proposal Level of Agreement for Final Proposal Duration of Deliberation (minutes) Curuguaty Control 0-verbal consent 1-all leaders 1-strong majority 0-verbal consent 1-all leaders 1-strong majority 30.00 General Aquino Control 0-verbal consent 1-all leaders 2-unanimous 0-verbal consent 1-all leaders 2-unanimous 60.00 General Resquin Control 0-verbal consent 2-all 1-strong majority 0-verbal consent 2-all 2-unanimous 43.00 Itacurubi del Rosario Control 0-verbal consent 1-all leaders 1-strong majority 0-verbal consent 1-all leaders 1-strong majority 40.00 Jasy Kañy Control 0-verbal consent 2-all 1-strong majority 1-show of hands 2-all 1-strong majority 40.00 Loreto Control 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 15.00 Pedro Juan Caballero Control 0-verbal consent 0-some leaders 1-strong majority 0-verbal consent 0-some leaders 1-strong majority 45.00 Santa Rosa Aguaray Control 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 60.00 Tacuati Control 0-verbal consent 2-all 1-strong majority 0-verbal consent 2-all 1-strong majority 50.00 Ybi Yau Control 0-verbal consent 1-all leaders 1-strong majority 0-verbal consent 1-all leaders 1-strong majority 40.00 Yrybucua Control 0-verbal consent 0-some leaders 0-weak majority 0-verbal consent 0-some leaders 0-weak majority 35.00 Azotey Treated 0-verbal consent 1-all leaders 1-strong majority 0-verbal consent 1-all leaders 1-strong majority 30.00 Belen Treated 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 5.00 Bella Vista Treated 0-verbal consent 1-all leaders 1-strong majority 0-verbal consent 1-all leaders 1-strong majority 40.00 Caapiibary Treated 0-verbal consent 0-some leaders 0-weak majority 0-verbal consent 0-some leaders 1-strong majority 79.80 Chore Treated 1-show of hands 2-all 1-strong majority 0-verbal consent 1-all leaders 2-unanimous 45.00 Concepción Treated 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 30.00 CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 19 Municipality Treat￾ment status Method of Assent for Initial Modification Extent of Participation in Initial Modification Level of Agreement for Initial Modification Method of Assent for Final Proposal Extent of Participation for Final Proposal Level of Agreement for Final Proposal Duration of Deliberation (minutes) Horqueta Treated 0-verbal consent 0-some leaders 1-strong majority 0-verbal consent 0-some leaders 1-strong majority 50.00 Liberación Treated 0-verbal consent 2-all 2-unanimous 0-verbal consent 2-all 2-unanimous 85.00 Lima Treated 0-verbal consent 2-all 1-strong majority 0-verbal consent 2-all 2-unanimous 10.00 Nueva Germania Treated 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 30.00 Puerto Antequera Treated 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 5.00 San Pablo Treated 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 30.00 San Pedro Treated 1-show of hands 2-all 1-strong majority 1-show of hands 2-all 2-unanimous 30.00 Ybypyta Treated 0-verbal consent 0-some leaders 0-weak majority 0-verbal consent 1-all leaders 1-strong majority 45.00 Sargento José Felix Lopez N/A 0-verbal consent 1-all leaders 1-strong majority 0-verbal consent 0-some leaders 1-strong majority 25.00 Control Mean 0.18 1.27 1.00 0.27 1.27 1.27 41.64 Control SD 0.40 0.79 0.45 0.47 0.79 0.65 12.86 Treated Mean 0.50 1.43 0.93 0.43 1.43 1.64 36.77 Treated SD 0.52 0.85 0.47 0.51 0.76 0.50 23.96 Diff. in Means p-value 0.10 0.64 0.70 0.44 0.62 0.13 0.52 Source: Authors’ calculations based on data collected at group deliberation events facilitated by CIRD. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 20 V. ENDLINE SAMPLING AND DATA COLLECTION The outcomes that we measured included both the outcomes directly targeted by the governance intervention (the blue boxes in the theory of change) as well as downstream development outcomes targeted by the broader IVC project (the brown boxes). We also used evidence from interviews and participant observation to trace out the causal process linking the intervention to the observed outcomes. We used the following data sources: • Producer Surveys. We conducted two endline producer-household surveys to collect household￾level data from the treatment and control groups. The first endline survey took place just at the end of the governance intervention, from September to November of 2018, and the second one took place roughly seven months later, from June to July 2019. The two endlines allow us to track trends and any medium-term enduring effects of the program. The survey sample size was 1,800 producer households (i.e., 50-60 producers per municipality), drawn from IVC project beneficiaries. This sample size was set on the basis of a calculation targeting an effect size of 0.22 standard deviations (or ten percentage points for a dichotomous outcome) at 80 percent power and 95 percent confidence, as described in detail in Annex 1 below. • Mayor Surveys. We conducted a short survey of the complete population of 32 mayors at the time of the second endline producer-household survey (about one year after the end of the intervention). The survey elicited mayors’ priorities and budgeting outcomes. • Buyer Interviews. We used qualitative interviews with buyers to assess the development of any new relationships with producers. • Municipality Administrative Data. We obtained the municipality development plans from each of the 32 municipalities and coded them according to a rubric that assessed prioritization of needs of small rural producers. The scoring methodology follows current best practices for systematic qualitative assessments.6 6 See Nicholas Bloom and John Van Reenen (2007), “Measuring and Explaining Management Practices Across Firms and Countries,” The Quarterly Journal of Economics 122(4): 1351-1408. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 21 VI. RESULTS We now present results, proceeding through the steps in the causal chain shown in Figure 1. For results drawn from the survey of producers’ households, we present graphs of mean values (with confidence intervals) for treated municipalities (black dots) and control municipalities (hollow dots) at each of the two endlines. This allows us to see how outcomes evolved over the course of a year following the end of the intervention. Given the clear differences in authority positions and proactivity of producer organization leaders versus non-leaders, we show effects separately for these two subgroups. As we will see below, doing so offers important insights into the effects of the intervention and also the nature of producers’ representation in this context. For outcomes measured at the municipality level (based on interviews with mayors or the scored development plans) we show graphs of mean values for treated (black dots) and control (hollow dots) municipalities. IMPROVING THE POSITION OF PRODUCERS VIS-À-VIS MUNICIPAL AUTHORITIES The intervention works with producer organizations in articulating development demands and in creating an opportunity for municipal authorities to hear their voices. Thus, the first step in the theory of change is that the intervention should improve the position of producers vis-à-vis municipal authorities. To assess this, we used indicators capturing the following: (i) Producers’ engagement with municipal development institutions; and (ii) Producers’ perceptions about accessibility and responsiveness of the municipal authorities. The graphs in Figure 3 illustrate key results with regard to these indicators. Beginning with indicators for engagement with development institutions, we first note that we observed very high variation across districts in knowledge of development institutions, and that the treatment substantially increased such knowledge. The first panel in Figure 3 is illustrative of such effects. For leaders, both in the first endline wave and the second wave, we see a substantial increase in knowledge that the municipality has a development council, with only about 25 percent of leaders being aware of such councils in control communities and 50 percent or higher in treated communities. Of course, this reflects the sum of the intervention’s effects on the municipality in activating such councils as well as the effects on producers’ organizations in having them engage such councils. Both of these effects contribute to the goal of improving producers’ position vis-à-vis the municipal authorities. For non￾leader producers, such an increase in knowledge only becomes apparent by the second wave, which is suggestive of information flowing first through leaders and then being passed on to non-leader members of producers’ organizations. Similar effects are apparent for knowledge of other development institutions, such as the development roundtables or municipal secretaries of production. Then, we also see substantial increases in meetings between producers and municipal development authorities. An example is given by the second panel in Figure 3. The rate at which leaders of producers’ organizations met with municipal production roundtables was less than ten percent in control municipalities, but was around 20 percent in the first wave and then increased to about 33 percent by the second wave. We also see an increase for non-leader producers, although the magnitude of the effect is small. Again, this reflects the centrality of producers’ organization leaders in mediating the governance relationship between producers and municipal authorities. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 22 Figure 3. Effects on Producers’ Position vis-à-vis Municipal Authorities Notes: Black dots are treated municipality means, and hollow dots are control municipality means. The line segments passing through the treated municipality dots are treatment effect confidence intervals (thin gray line is for 95 percent confidence, and the thicker darker line is for 90 percent). Using the control means (hollow dots) as a reference point, the confidence interval lines indicate the range of plausible treated municipality means. If the confidence interval segment does not cross beyond the control mean, then the treatment effect is statistically significant. Outcomes are shown for Wave 1 (a few months after the end of the intervention) and Wave 2 (one year later) so as to visualize trends in treated and control municipalities in the year following the intervention. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 23 We do not see such substantial effects on producers’ perceptions about the responsiveness of municipal authorities. Part of the explanation is that producers in control municipalities had quite optimistic views of their municipal authorities. For example, in an index based on survey questions about municipal responsiveness that ranged from zero (highly pessimistic) to three (highly optimistic), the control group means for leaders and non-leaders alike were around two, with almost no difference for the treatment group. A ceiling effect may prevent any impacts on such an indicator. Another way to get at this is to track producers’ perceptions that their priorities are being addressed in municipal development planning. Recall from above that in pre-intervention deliberation exercises, we found that producers overwhelmingly tended to prioritize roads, rural electrification, and agricultural extension services. The third panel in Figure 3 tracks the number of such priorities (from zero to three) that producers sensed were being addressed in municipal planning. In the first wave, leaders in treated areas had a stronger sense than those in control areas that such priorities were being addressed, but this gap closed by the second wave (in both types of localities, there was an increase in leaders’ sense that such needs were being addressed). For non-leader producers, such a perception was also increasing over waves, but in a manner that does not show any differences between treated and control municipalities. In summary, with respect to this first step in the causal chain, we find that the intervention generated new opportunities for municipal engagement for producers and therefore did indeed improve their position vis-à-vis municipal authorities. These effects were mediated through producers’ organization leaders. This did not translate into a strong effect on producers’ perceptions, however this is attributable at least in part to the fact that, even in control conditions, producers’ perceptions were quite optimistic. IMPROVING THE POSITION OF PRODUCERS VIS-À-VIS BUYERS The first step in the theory of change also proposes that the intervention could improve the position of producers’ organizations vis-à-vis buyers. This effect is expected as a consequence of the improved position of producers vis-à-vis municipal authorities, because one of the roles of municipal authorities in rural development, and particularly for production secretaries, is to facilitate contacts with buyers. To track this outcome, we collected survey data from producers on the following indicators: (i) Interactions between producers’ organizations and buyers; and (ii) Facilitation of contracts by district. Here we find a sustained improvement in access to buyers. Figure 4 shows effects on a zero-to￾three index that counts instances within the previous year of meeting with new buyers, negotiating new contracts, and meeting with municipal authorities to discuss new connections with buyers. Among leaders, there is strong indication that such activity increased, with a substantial treatment effect in both waves. For non-leaders, such effects are more muted, although this may be attributable to the fact that such non-leaders rarely engage directly in such interactions. Qualitative interviews with members of farmer committees confirm that the majority of their interactions are with their leaders. Committees hold weekly, biweekly, or monthly meetings depending on their level of activity, where leaders inform the members of their activities, listen to concerns and needs, and make decisions about what actions the leaders will take to advance the interests of the group. Leaders then articulate the demands of their committee in meetings with local and national government officials, private companies, and development actors. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 24 Figure 4. Effects on Producers’ Position vis-à-vis Buyers Notes: Black dots are treated municipality means, and hollow dots are control municipality means. The line segments passing through the treated municipality dots are treatment effect confidence intervals (thin gray line is for 95 percent confidence, and the thicker darker line is for 90 percent). See the caption to Figure 3 for details on interpreting statistical significance. ALIGNING MAYORS’ PRIORITIES WITH PRODUCERS’ PRIORITIES The previous sections provided evidence that the intervention improved producers’ position vis-à￾vis municipal authorities. To what extent did this translate into municipal mayors internalizing producers’ priorities? To assess this next step in the theory of change, we evaluate the results of a rating exercise that we conducted with both producers and mayors. In the exercise, respondents were given a list of ten development spending areas (agricultural credit, material agricultural inputs such as fertilizer, agricultural machinery, extension services, public infrastructure such as market stations, roads, transport services, rural electrification, contracts with buyers, and contracts with public institutions) and asked to allocate 100 points to designate how they would prioritize spending in these areas. The exercise was done with each producer independently as part of the household survey, and then independently with each mayor as part of the mayors’ surveys. We then measure the divergence in the allocations between each producer and his or her respective mayor. The divergence measure is the Euclidean distance between the vectors assigned by each producer and his/her respective mayor; lower values on this scale mean less divergence, and therefore higher alignment. Insofar as producers’ priorities have been internalized by the mayor, we should see that this divergence decreases. Figure 5 shows the results. On top are the mean divergence scores between mayors and non-leader producers in treated (black dot) and control (hollow dot) municipalities. The confidence interval for the treated municipality estimates reflect uncertainty in the treatment effect estimate. Results for leaders are shown below the results for non-leaders. The results show a ten percent decrease in divergence for non-leaders, and then a 15 percent decrease in divergence for leaders (significant at the ten percent level). This represents a meaningful increase in the alignment of mayors’ priorities with those of producers. Note that this was measured only at the second wave, a year after intervention programming had concluded. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 25 Figure 5. Effects on Alignment between Mayors’ and Producers’ Priorities Notes: Black dots are treatment municipality means, and hollow dots are control municipality means. The segments passing through the dots are confidence intervals (thin line is 95 percent, thicker line is 90 percent). See the caption to Figure 3 or details on interpreting statistical significance. IMPROVING DISTRIBUTION AND QUALITY OF DISTRICT SERVICES We focus on roads, electricity infrastructure, and extension services to assess the intervention’s effect on the distribution and quality of local public goods and services. A priori, these three public goods are highly relevant for small farmer integration. In addition, they arose as the three highest priorities among the group of beneficiary farmers who participated in a deliberation and ranking exercise that was conducted in all the control and treatment municipalities. We constructed indicators for: (i) Transport costs, transport time to municipal center and to the nearest paved road, and the perceived quality of the road adjacent to the farm; (ii) Access to the electricity grid and knowledge of improvements to electricity grid connection; and (iii) Access to agricultural extension training through the municipality. The data revealed two distinct groups, one that reported few problems with road reliability and perceptions of high quality, and another that reported poor road quality and a high incidence of problems. The intervention had no effect on the access to roads or the perceived quality of roads; however, a large number of producers reported few problems to begin with. Economic losses due to poor road quality are infrequent. However, qualitative interviews revealed that farmers with poor road access simply forgo investing in commercial production that requires reliable road transport (e.g. dairy production) rather than risk losing their investment. Among the farmers, access to the electricity grid is already high (close to 90 percent), although connection to three-phase current, which is necessary to run processing equipment and farm machinery, is very rare (only about five percent of farmers have it). Given these base rates, it is not surprising that the intervention had no effect on electricity access. In terms of access to agricultural extension and training, leaders reported a large increase between the first and second waves of the endline survey (from 28 percent to 61 percent), however this occurred in both the treatment and control groups, and not as a result of the intervention. Ordinary producers reported no change in their access to extension services. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 26 QUALITY OF THE MUNICIPAL DEVELOPMENT PLANS As the previous section reports, the intervention had no significant effects on producers’ perceived access to the public goods and services they most prioritize. However, it is not clear that enough time transpired during and since the intervention to witness such effects. An outcome that is more likely to change in the short run is the intentions of the municipal government to provide services, as demonstrated by the development plans they prepared. Two research assistants scored the municipal development plans that were submitted to the STP according to seven dimensions, each scored on a zero-to-15 scale, without knowledge of which municipalities received the treatment. The scores revealed large improvements in whether development plans expressed clear commitments to improving the welfare of rural producers, specifically regarding the inclusiveness of the planning process to small farmer groups, evidence of efforts to diagnose challenges for rural development, and the priority given to the needs of rural small producers (see Figure 6). The data make clear that the intervention opened up the planning process to greater farmer participation and also increased formal commitment to rural producers’ needs. Figure 6. Effects on Prioritization of Rural Producers in Development Plans Notes: Black dots are treatment municipality means, and hollow dots are control municipality means. The segments passing through the dots are confidence intervals (thin line is 95 percent, thicker line is 90 percent). See the caption to Figure 3 for details on interpreting statistical significance. The outcome measures here are ratings (on a zero-to-15 scale) produced by two independent coders who were blind to treatment. DOWNSTREAM OUTCOMES AGRICULTURAL INPUTS AND UPGRADING The intervention aimed not only to improve small farmers’ access to municipal services, but to enhance the effects of the broader productive and economic development project executed by FECOPROD. The idea here is that certain local public goods are necessary complements to farm￾and firm-level investments that are geared toward “upgrading” the production strategies, product quality, value-added, or the diversity of agricultural production at the farm level. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 27 To capture whether producer upgrading occurred with greater frequency in the treatment group, we devised indicators for: (i) Land-market transactions, including titling, renting, purchasing, or selling of land. (ii) Upgrading of technologies or production techniques. (iii) Acquisition of international certifications. (iv) Upgrading of functions within the value chain. (v) Diversification of crops. The survey revealed no program effects on this set of indicators. This is not surprising, given that any additional upgrading observed in the control group would be the indirect effect of improved access to services, and, as shown above, the intervention did not yet produce effects beyond the commitments made by municipal governments and buyers. It is possible that producers are waiting to see if districts and buyers honor these commitments before changing their production behavior. AGRICULTURAL INCOME Similarly, the provision of public goods and farm-level upgrading could enhance both agricultural yields and agricultural income. To evaluate these effects, we constructed indicators for: (i) Income from agricultural products. (ii) Number of agricultural transactions and contracts. The intervention had no effect on income. This is not surprising given that upstream effects were not yet observed and given the relatively short duration of the intervention in relation to budgetary cycles. However, the treatment did produce an increase in the use of collective contracts for agricultural transactions, from around five percent in the control group to about ten percent in the treatment group for non-leader producers, and from five percent to 25 percent for leaders. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 28 VII. DISCUSSION AND LESSONS Beyond the results reported above for specific indicators and the evaluation of specific hypothesized effects, this study points toward two broader lessons. First, much of the emphasis in development programing and the development literature on participation (for example, the now vast amount of work done on participatory budgeting) focuses on the determinants, the processes, and the effects of individual citizen participation. To the extent our intervention succeeded in boosting the political participation of small farmers and democratizing relations between farmers and local government, it did so through mediation of farmer leaders that were connected to farmer organizations that articulate interests collectively as opposed to individually. The intervention worked by opening up greater space in public forums and within local government administration for the voices of leaders that represent collective producers’ interests. This, in turn, produced a meaningful shift in the local development priorities expressed by mayors, as evidenced by the greater alignment between mayors’ priorities and the individual priorities expressed by small farmers. This observation suggests that research and interventions related to democratic participation should focus on collective action and local collective action institutions. Second, the role of the development NGO, CIRD, was most likely crucial in achieving this outcome through its ability to persuade mayors to open up this space for dialogue and undertake institutional changes that permit greater participation and responsiveness. It is not clear that producers could have achieved similar results on their own, without the financial, political, and knowledge resources that an NGO with many years of experience in development work can offer to a municipal government. This hypothesis is bolstered by the observation that the development plans in treated municipalities shared a lot of common language, largely reflecting CIRD’s direct input and support of the planning process, and also by the fact that the intervention did not change farmers’ perceptions of the quality of their municipal institutions. On the one hand, this raises additional questions about the types of development interventions that are necessary to create greater civil society capacity for democratic participation and accountability in the long term. However, it also reflects an important reality: in the short run, marginalized actors require both material and political support to get their voices heard in authoritarian and clientelistic settings. In conclusion, a “collective action” model for citizen participation, such as the one that came into focus through this research project, can improve the position of small producers and their interests in the process of municipal planning. While these changes in political participation and representation did not produce any immediate changes in farmers’ incomes, it is probably too soon to draw conclusions about these indirect effects. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 29 ANNEX 1: TECHNICAL DETAILS OF RESEARCH DESIGN CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 30 SAMPLING PLAN We use a two-stage sample procedure. In the first stage, we sample producer organizations in each of the treatment and control districts. The sampling strategy was as follows. For districts with ten or fewer organizations, we take all of the organizations into our sample. For districts with more than ten organizations, we sample ten of them with probability proportional to size. For each organization indexed by t in a municipality indexed by m the design implies a probability of being sampling, Pmt, where Pmt=1 for organizations in municipalities with fewer than ten organizations, and for organizations in municipalities with more than ten organizations, Pmt=Nmt/Nm where Nmt and Nm are the number of producers in organization t in municipality m and the total across all organizations in municipality m, respectively. In the second stage, we select random samples of producers from each of the selected organizations so as to achieve our target of 50-60 producers per municipality. To do so we divide an initial target of 60 evenly across the sampled organizations. Suppose there were Tm organizations sampled in municipality m. Then, the sampling quota for each organization equals Qmt= min {Nmt, 60/Tm} For each individual i in organization t and municipality m, this design implies a probability Rmti=Qmt/Nmt of being sampled, conditional on the individual’s organization having been sampled. Thus, the full design-based sampling probabilities for producers are given by the product Pmt Rmti. To implement this design, we needed lists of producers’ organizations and then organization members to serve as the sampling frames for the first and second stages, respectively. FECOPROD’s recruitment strategy for beneficiary organizations provided a challenge for establishing a complete and relevant sampling frame of individual producers. Over the five-year project, FECOPROD’s field managers continuously approached producer organizations, established initial contacts, and integrated the organizations into the project to the extent they made progress. Organizations that did not make progress or express commitment were labeled ‘inactive,’ and ceased to interact with FECOPROD. Those that continued to work with FECOPROD were labeled ‘active.’ FECOPROD collected administrative information for the organizations that they had initially approached and applied a baseline survey to their members. However, over the course of the project, many of them became inactive. Furthermore, a variety of instruments and methods were used to collect information about the organizations that were recruited later on, and information about their members was not collected in a systematic manner. Because no complete sampling frame of individual producers was available, we proceeded by constructing the organization-level sampling frame out of several of FECOPROD’s databases. We included any organizations, active or inactive, that had had contact with FECOPROD in the course of the intervention. We elected to create a sampling frame as broad as possible, because our intervention included activities that were targeted, but not limited to, producer organizations active in FECOPROD’s project, as well as activities that took place at the level of the entire district. We cross-checked with the complete list of producers that had participated in CIRD’s activities in the treatment districts and found that the vast majority of the organizations in CIRD’s list appeared in the sampling frame. This implied that FECOPROD’s list provided a good basis for identifying potential beneficiaries of CIRD’s work. We also verified with FECOPROD’s field managers that this list of organizations was complete. Next, the sample of organizations was drawn based on the first-stage strategy described above. With this sample of organizations, we proceeded to construct the sampling frame of individual producers within the selected organizations. First, we requested complete lists of the membership with contact information from FECOPROD’s field staff and compiled these into a single database. These were CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 31 available only for the active organizations. For the inactive organizations, we requested contact information for the leaders, contacted the leaders directly, and requested the complete names and contact information for the leaders. FECOPROD’s staff was unable to provide contact information for the leaders of a portion of the selected inactive producer organizations. Instead, we requested contact information for these organizations from the Ministry of Agriculture and Livestock’s local extension offices. These procedures yielded lists of members for just over 70 percent of the selected organizations. For the other 30 percent, we were not able to obtain lists. We examined whether rates of such organization-level missingness varied across municipalities and, importantly, across treatment and control municipalities. We found that 26 percent of organizations were missing in treatment municipalities and 30 percent in control municipalities. Given the amount of heterogeneity in numbers of organizations per district, we are not worried about this four percentage point difference introducing bias. In the second stage of the sampling procedures, we drew a random sample from each of the producer organizations for which we had a complete sampling frame. We also selected the leader of each organization to be part of the sample. Including leaders will allow us to analyze how leaders and non-leaders differ in, for example, their preferences related to municipal spending. POWER CALCULATIONS Our proposed sample of 1,800 producer households was based on power calculations summarized in Table 1 and explained below. As a benchmark for comparison, Table I shows the power that we would have in testing for a given outcome (in standardized units) if we were able to survey the entirety of the set of eligible households (the bottom row). The power for such a test would not change appreciably compared to what our proposed sample provides. This is because the number of municipalities dictates power in this setting. Our assumption in doing the power analysis was that the number of municipalities would be 30. As it turns out, we ended up with 32, and so our analysis is conservative. Nonetheless, in our original analysis, the assumption of 30 municipalities means that a completely randomized clustered design would yield a minimum detectable effect size (MDES) of about 0.42 standard deviations, which is quite large and therefore does not look very promising. We thus draw upon additional methods that will allow us to boost power for detecting effects even without increasing sample size. These additional methods include blocked and restricted random assignment, covariance adjustment, and the use of multiple indicators and index-based omnibus tests. The following paragraphs explain. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 32 Table A1. Basic Power Calculations Sample Munis* Orgs/ Muni* Total Orgs* DE Orgs** Farms/ Org (Target) DE farms** Eff N Farms Total N (Target) Attri￾tion Prob Total N (Actual) Farms/ Org (Actual) MDES*** Proposed 30 7.6 227 2.65 7 3.4 177 1,596 0.1 1,773 8 0.42 Population 30 7.6 227 2.32 44 18.2 208 10,032 n/a n/a n/a 0.39 * These numbers are fixed. The number of organizations per municipality is the average per municipality based on data from FECOPROD. ** DE stands for design effect. With a sample of size N and a design effect of DE, the effective sample size (“Eff N”) is N/DE. We compute DE as 1 + (M-1)*ICC, where M is the number of units per cluster and ICC is the cluster interclass correlation. Given the results of analysis of baseline data, we assume that organizations within municipalities have ICC=0.25, and farms within organizations have ICC=0.40. This implies a design effect of 1+(7.6-1)*.25 = 2.65 for the number of organizations per municipality, in which case the effective number of organizations is 227/2.65≈86. For the proposed sample, it also implies a design effect of 1+(7-1)*.40=3.4 within organization when we sample 7 farms per organization, in which case the effective number of farms per organization is 7/3.4=2.06. Putting these two figures together yields a total effective number of farms equal to 86*2.06≈177. The nominal number of farms would be 30*7.6*7=1596. Given ten percent attrition we would want to sample 1596/.9=1773, or 8 farms per organization. Similar calculations yield the figures for a sample of the entire population. *** The MDES is the “minimum detectable effect size,” expressed in standard deviation units. Assuming lower ICCs will result in more power and thus smaller MDESs. The formula for the MDES is based on H. Bloom Eval Rev 19(5):547-566. Given a balanced design, standardized outcomes (that is, standard deviation of 1), 80 percent power and 95 percent confidence, the formula is: MDES= 5.6/√(Eff N), which yields 5.6/√(177)=0.42 and 5.6/√(208)=0.39 for the proposed and full population samples, respectively. First, we can derive our starting MDES approximation. We have 30 municipalities with which to work, and we intend to randomly assign the treatment to half. Our FECOPROD data suggested an average of 7.6 eligible farmer organizations per municipality, and then an average of 44 farms per organization. Because treatment is assigned at the municipality level, we need to account for the fact that there may be “clustering” in the organizations’ and farms’ behavior, attitudes, and production outcomes. Mathematically, one needs to specify an anticipated level of “intra-class correlation” (ICC), a measure from 0 to 1 that indicates the level of homogeneity (where an ICC of 1 means all units within a cluster are identical, and 0 means that they are totally different). Based on our analysis of available data and our understanding of the types of organizations in the study, baseline data from FECOPROD’s farmer surveys suggested a moderate degree of homogeneity of outcomes at the level of organizations within municipalities—an ICC of 0.25. At the level of farmers within organizations, baseline data suggests using an ICC of about 0.40 (after centering the data on the municipal-level means to control for municipal-level variation). We assume that we will have ten percent attrition. Such clustering and attrition means that even though we start with a nominal sample size of 1,800 farms, the “effective sample size” — that is, the actual information value of the sample—will be considerably smaller. Attrition means that we expect to have a final sample of about 1,600 farms. The ICCs at the two levels described above mean that the information content would be equivalent to being able to run a completely randomized (as opposed to cluster randomized) experiment with 177 farms. At 95 percent confidence and 80 percent power, this yields a “minimum detectable effect size” (or MDES, that is, an effect size for which we will have the power to detect against the null hypothesis) of about 0.42 standard deviations. This sample size also allows us to conduct subgroup analyses that split the sample into subgroups within farmer organizations as small as 455 farms while still being able to detect less than a half of a standard deviation effect. Of course, if we were to split the sample across farmer organizations or across municipalities, the minimum detectable effects would be quite large and most likely beyond the reach of this study. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 33 Conventionally, an MDES of 0.42 standard deviations is considered as a “medium” effect size, but in field experimental research, it would actually be quite large. Moreover, this power analysis is relevant to the producer-level sample, but of course with the organization and municipality level data, we are even more limited. Therefore, we take a number of measures to boost the power and information content of the study despite the limited sample size. There are four ways that one can boost the information context of a study without increasing sample size: (1) efficient methods for treatment assignment, (2) efficient methods of analysis, (3) having multiple follow-up rounds rather than a single endline, and (4) using efficient methods to combine analytical results. We describe how we apply these three approaches in turn. First is pair-matched blocking at the municipality level. McKenzie and Bruhn (2009, p. 222) use simulations with real-world development data to suggest that such pair matching combined with stratum fixed effects can boost power for detecting small effects in a development field experiment by anywhere between about 11 percent to 70 percent with a sample size of 30.7 Second, our analysis will adjust for organization-level and farm-level covariates. These will be gathered from the data collected by our implementing partners (FECOPROD and CIRD). We can also collect at any time covariate information for things like subject’s age and size of land holdings, in which case we may not need to carry out baseline surveys prior to the intervention. This is important with respect to third power-boosting strategy, described below. Third, following the recommendations of McKenzie (2012), our design can use two rounds of outcome follow-up, one following the first year implementation of the budgeting intervention, and another following the second year.8 Adapting McKenzie’s results, we can characterize the power gains obtained by going from a simple difference in means to the combination of adjustment for pair￾matching, adjustment for additional covariates, and using the average of two rounds of post￾treatment follow-up measurements. Such a strategy can reduce the variance of the treatment effect estimator by a factor of (1+ρy)/2-R2, Where Py is the correlation in outcomes and R2 the coefficient of determination that characterizes the variance reduction from the matching and from regressing control or treated outcomes on covariates and matched-pair. Using Py = 0.5, which would be conservative relative to the values presented by McKenzie (p. 215) for income and expenditure, and R2=0.4, this would imply a reduction in estimator variance of about 65 percent. Fourth, to test our evaluation hypotheses, we will use omnibus tests on sets of indicators that capture outcomes and patterns of interest rather than on one single outcome variable. In this way, we can evaluate if patterns of effects correspond to what we hypothesize. By combing a set of outcomes into a test, we can boost power. Methods include the construction of “mean effect” indices, inverse covariance weighted indices, as well as other combination statistics.9 The extent of 7 Bruhn, Miriam, and David McKenzie. 2009. “In Pursuit of Balance: Randomization in Practice in Development Field Experiments.” American Economic Journal: Applied Economics, 1(4): 200-232. 8 McKenzie, David. 2012. “Beyond Baseline and Follow-up: The Case for More T in Experiments.” Journal of Development Economics 99: 210-221. 9 For example, Caughy et al. (2015) reanalyze a campaign field experiment in Benin that involved two treatments and a control condition being randomly assigned over 24 clusters, and so only 8 clusters per treatment group. Single-contrast hypothesis tests yielded p-values of 0.22 and 0.03 for the two treatments, but CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 34 the power boost depends on the degree of correlation between the outcomes, but even a 20 percent boost would be analogous to having an MDES of 0.22 standard deviations with a single outcome variable. We can put this into more tangible terms. The FECOPROD baseline survey measured whether farmers received any material assistance from the municipality, (including municipal, producer organization, or buyers, etc., but excluding what FECOPROD is providing), showing that 37 percent had. A 0.22 standard deviation effect would amount to about a ten percentage point effect—that is, an increase from 37 percent access to 47.3 percent access. a joint test that also accounted for the fact that the two treatments were varying “doses” of the same the underlying treatment yielded p<.01. See Caughy, Devin, Allan Dafoe, and Jason Seawright, 2015, “Global Tests of Complex Hypotheses: A Nonparametric Framework for Testing Elaborate Theories,” Working Papers, MIT/Yale/Northwestern. See also Anderson, Michael L, 2008, “Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool and Early Training projects,” Journal of the American Statistical Association 103(484), 1481–1495; and Casey, Katherine, Rachel Glennerster, and Edward Miguel, 2012, “Reshaping Institutes: Evidence on Aid impacts Using a Preanalysis Plan,” The Quarterly Journal of Economics, 127(4):1755-1812. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 35 ANNEX 2: EVALUATION TIMELINE CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 36 The timeline for evaluation activities is as follows: NOVEMBER – DECEMBER 2015: FINALIZED STUDY DESIGN WITH IMPLEMENTATION SUBCONTRACTOR The evaluation team collaborated with CIRD, FECOPROD, and USAID in Asunción to develop and revise the technical proposal for the governance intervention into a detailed work plan that was consistent with the impact evaluation methodology. To design the intervention the evaluation team travelled to Paraguay for one week in November 2015 to carry out a design workshop. MARCH 2016: SELECTION OF MUNICIPALITIES AND FARMER ORGANIZATIONS FOR GOVERNANCE INTERVENTION The evaluation team assembled municipal-level administrative data for the random assignment of municipalities to the treatment and, with USAID/FECOPROD’s collaboration assemble preliminary farmer and farmer organization data for the random selection of farmer organizations. APRIL 2016 – APRIL 2018: GOVERNANCE INTERVENTION AND COMPLEMENTARY RESEARCH While the governance intervention was being implemented, the evaluation team engaged in two types of complementary research. The first one is desk research on the laws and institutions supporting the intervention—specifically, Decree 4774-16 calling for the articulation of municipal development plans. Guidelines provided by this decree emphasize the role of participatory institutions at the municipal level, and this has served as the primary backdrop to the intervention. This desk research provided crucial contextual information for the study. The second type of research consisted in qualitative interviews with key actors in the municipalities and producers’ organizations in order to assess the quality and reach of the intervention. MARCH 2018 – OCTOBER 2018: PREPARE AND ADMINISTER ENDLINE SURVEY WAVE I The evaluation team piloted and revised the endline survey instrument, selected the sample of farmers for the endline survey, and supervised the selection and training of a team of enumerators hired and managed by the data collection subcontractor. They provided monitoring and troubleshooting during the survey process. Due to the difficulty the evaluation team experienced constructing the sampling frame, the start of the first endline was postponed from March 2018 until September 2018. MAY – JULY 2019: PREPARE AND ADMINISTER ENDLINE SURVEY WAVE II The second endline survey’s primary purpose was to reduce measurement error and improve the statistical power of our experimental design, partially compensating for the small number of cases. This survey was originally planned for November 2018. For both methodological and practical reasons, the evaluation team decided to delay the second endline survey to June 2019. First, the difficulty we experienced constructing the sampling frame required delaying the start of the first endline until September 2018. At least some delay in the second endline is necessary. Second, delaying the survey allowed us to measure the effects of our intervention on a portion of the 2019 budget cycle. This second budget cycle offers an additional round of outcome observation, and by pooling the observations from two budget cycles (2018 and 2019), we boosted the power of our analysis relative to having only one budget cycle from which to gauge effects. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 37 ANNEX 3: PROJECT INVESTIGATOR PROFILES CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 38 Principal Investigator: Gustavo Setrini, Ph.D. Lecturer in Political Economy of Development at FLACSO-Paraguay. From 2013-2019, Setrini was Assistant Professor of Food Studies at New York University’s Department of Nutrition, Food Studies and Public Health. Professor Setrini’s research focuses on the political economy of agricultural globalization and rural development in Latin America and the Caribbean. He is currently writing a book on Fairtrade and Organic certification and smallholder farmer organizations in Paraguay’s sugar industry which investigates the role of local governance in promoting small farmer upgrading in global value chains. His other research has examined the impact of donor-funded, NGO-led small farmer development projects in Paraguay, food system entrepreneurship in Puerto Rico, and cocoa quality upgrading in the Dominican Republic. He holds a Ph.D. in political science from the Massachusetts Institute of Technology (MIT). Co-Investigator, Methodology: Cyrus Samii, Ph.D. Associate Professor of Politics at New York University’s Wilf Family Department of Politics. Professor Samii writes and teaches on quantitative social science methodology, with an emphasis on causal inference, and on substantive topics related to governance in contexts where formal institutions are weak, the political economy of development, and social, economic, and psychological causes of violent conflict. His work has appeared in the American Political Science Review, American Journal of Political Science, Annals of Applies Statistics, Journal of Conflict Resolution, Journal of Labor Economics, Journal of Peace Research, Journal of Politics, and Survey Methodology. He has designed and carried out field studies in Afghanistan, Burundi, Colombia, Cote d’Ivoire, Democratic Republic of Congo, Indonesia, Israel, Liberia, and Nepal. He holds a Ph.D. in political science from Columbia University. Co-Investigator, Data Analysis: Cynthia González. Executive Director, E+E Economía y Estadísticas para el Desarrollo, Asunción Paraguay. Ms. Gonzalez is a labor market and impact evaluation researcher. She has over fifteen of experience as a research consultant and data analyst for academic institutions, international and donor organizations, and the Paraguayan government. She has designed impact evaluations for the Japanese International Cooperation Agency (JICA) and collaborated with Innovations for Poverty Action (IPA) on experimental impact evaluations. Ms. Gonzalez recently served as Vice Minister of Labor and Social Security. She holds a degree in economics from the University of Asunción and has graduate studies in economics at the University of the Andes in Colombia and training in quantitative research design from ORT University, Uruguay. CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 39 ANNEX 4: IMPLEMENTING PARTNER PROFILE CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 40 The Center for Information and Resources for Development (Centro de Información y Recursos para el Desarrollo, CIRD) is a private, non-profit organization with 30 years of experience in managing development projects in Paraguay. As a result of this experience, CIRD has designed a development project management model for complex and difficult contexts. Human and social capital formation for sustainable development is an essential component of CIRD’s mission. Founded in 1988, the CIRD assists organizations and communities in the applications of strategic, organizational, technological or operational solutions to problems that hinder and limit the options of people of the most vulnerable sectors. Through the provision of specialized technical assistance, in a practical and timely manner, CIRD’s work has succeeded in helping these organizations and communities correct their management difficulties, providing concrete organizational and technical knowledge and skills to solve practical and adaptive problems. For more information, go to: https://www.cird.org.py/ CONTRACT NO. GS-10F-0033M/AID-OAA-M-13-00013 USAID.GOV IMPACT EVALUATION OF DRG COMPONENTS OF USAID/PARAGUAY’S INCLUSIVE VALUE CHAINS PROJECT | 41 U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, DC 20523