December 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Ricardo Fort and Manuel Glave from Grupo de Apoyo al Desarrollo (GRADE) through USAID contract AID-527-C-13-00002 with Partners for Global Research and Development LLC (PGRD) Retrospective Impact Evaluation of Alternative Development Program in Huanuco, San Martin and Ucayali (2007-2012) Foto: Agencia de Prensa Ambiental - InfoRegión ii RETROSPECTIVE IMPACT EVALUATION OF ALTERNATIVE DEVELOPMENT PROGRAM IN HUANUCO, SAN MARTIN AND UCAYALI (2007-2012) Peru December 11, 2014 The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS Acronyms ..................................................................................................................................................... iv Executive Summary ...................................................................................................................................... 5 1. Evaluation Purpose & Evaluation Question ..........................................................................................7 2. Project Background...............................................................................................................................8 3. Evaluation Methods & Limitations .......................................................................................................11 1.1. Types of interventions to evaluate ..........................................................................................11 a. The sample of communities selected for the period 2007-2012 .............................................11 b. Identification of ADP and NADP families ................................................................................13 1.2. Description of potential treatment and control groups and possible methods of impact evaluation............................................................................................................................................16 a. Cross-section analysis ............................................................................................................16 b. Difference in difference: Communities that become part of ADP during the period of analysis. 17 c. Difference in difference: ADP communities against NADP communities ...............................17 d. Same communities..................................................................................................................17 e. Different communities .............................................................................................................18 1.3. Recommendations for the evaluation method to use..............................................................19 a. Limitations for the estimation of impact...................................................................................19 b. Evaluation method...................................................................................................................21 4. Findings, Conclusions & Recommendations ......................................................................................22 1.4. Initial comparison between ADP and NADP families..............................................................22 1.5. Cross section analysis ............................................................................................................26 1.6. Panel of communities..............................................................................................................28 1.7. Conclusions.............................................................................................................................30 1.8. Recommendations ..................................................................................................................31 Annex 1: Sampling Frame 2007-2012 ........................................................................................................32 Annex 2: Identification of ADP families in different surveys ........................................................................34 Annex 3: Differences in income ..................................................................................................................35 Annex 4: Statement of work: Retrospective Impact Evaluation ..................................................................38 iv ACRONYMS ADP Alternative Development Program CORAH Special Project Control and Reduction of Illegal Crops in the Upper Huallaga DEVIDA Peru's National Commission for Development and Life without Drug GOP Government of Peru NADP Non-Alternative Development Program PSM Propensity Score Matching SOW Statement of Work USAID United States Agency for International Development USG United States Government VRAEM Valley of the Apurimac, Ene, and Mantaro Rivers 5 EXECUTIVE SUMMARY From 2002-2011, the United States Government (USG) and the Government of Peru (GOP) jointly implemented the Alternative Development Program (ADP) in areas of San Martin, Huanuco, and Ucayali. Drug traffickers and subsistence farmers have used areas of these regions characterized by a lack of infrastructure and government presence for illicit coca cultivation. The program’s goal was to sustain reductions in illicit coca cultivation through the promotion of licit economic alternatives as well as increasing social capital. The ADP encompasses a collection of projects funded by USAID that all contribute to development objectives, and which were implemented by different partners over the intervention period.1 Between 2002 and 2006, interventions focused assistance on communities that agreed to eradicate their own illicit coca in voluntary eradication agreements, known as the “R-376” communities. After 2006, interventions focused on communities that signed agreements not to replant coca following programmed eradication by the GOP, referred to as Plan20XX according to the operational year in which they entered into agreements. Additionally, between 2006 and 2008 forced eradication was implemented in some communities. Assistance provided to those communities is known as ‘post-eradication’. The main objective of this report is to retrospectively measure the impact of the ADP on participating households in San Martin, Huanuco, and Ucayali. To achieve this, the analysis relies on information from household surveys carried out by Peru's National Commission for Development and Life without Drug (DEVIDA) from 2007 – 2011. The DEVIDA survey, funded by USAID, included data collected from both participating and non-participating communities regarding income and perceptions related to coca, community well-being, local and regional government, and other indicators. However, the purpose of the survey was mainly to monitor the progress of the DEVIDA interventions rather than to evaluate its impacts. This report presents a descriptive analysis of the types of interventions implemented by ADP in the aforementioned regions followed by an estimation and analysis of program effects using two different methods: cross-section propensity score matching and a panel data analysis. Various factors hindered the ability to assess the ADP’s impact. Factors such as lack of information, selection criteria and method, different stages of the project’s intervention (voluntary vs. forced eradication), no baseline survey2 and the lack of sufficient control communities, all limited the assessment of impacts. However, we were able to find indications of interesting potential effects attributable to the ADP in these regions. From the cross section analysis, we only found a significant and positive effect on 1 A major ADP activity implemented by Chemonics over the period was also known by the “ADP” acronym. Unless specified, use of ‘ADP’ in this evaluation refers to the overall collection of program interventions. 2 While there is a survey for the first year of the intervention (and previous years) the data collected is not equivalent to a baseline survey because: interventions had been taking place for years before the intervention period; the data collected in the survey is oriented towards monitoring as opposed to attributing causal effects to the intervention; and, the survey questionnaires and methodologies were not consistent over time. 6 family income for ADP communities in Huanuco for the year 2011 due to an important difference in productive income compared to non-ADP (NADP) communities. Additionally, when we separated ADP communities by type of intervention (R-379 and Plan20XX), we found positive program effects for R-379 communities in Huanuco and Ucayali, but not in San Martin, and these positive effects were more significant for the years 2010 and 2011. This fact can be attributed to the longer period of participation that R-379 communities have under the ADP and to the time it takes for alternative crops like cacao and coffee to become highly productive. In fact, exploring the evolution of income from these crops in the case of R-379 communities and NADP communities that were selected as controls, we found an important improvement in income for the ADP families between 2007 and 2011-2012 in all regions in the case of cacao and in Huanuco also in the case of coffee. In San Martin, however, income from coffee production appears to have grown more for NADP communities. This last result, and the fact that households in NADP communities in San Martin initially reported higher income from other cash crops like rice and coca, could explain the absence of a total income effect in this region. Finally, we took advantage of the fact that some communities were repeated between samples of different years to analyze the average difference in income between each pair of years for ADP and NADP communities. In this panel analysis, we found a significantly bigger improvement in ADP communities for Huanuco and Ucayali in most cases, but results in San Martin were not as clear. In this region, there were slightly positive results until 2011 but these results became not significant or negative when we used the year 2012 as a comparison. For the future impact evaluation of ADP in Huanuco, we recommend reviewing and adjusting the survey sample selection criteria and survey methods in order to attain better baselines, improving the collection of data through the use of a panel which will allow for a greater analytical power to capture the complexity of human behavior and social impact. Finally, questionnaires should be redesigned to incorporate more questions, most importantly questions that assist in the construction of variables that will lead to evidence-based analysis. 7 1. EVALUATION PURPOSE & EVALUATION QUESTION The purpose of this retrospective evaluation is to use DEVIDA household surveys for the period (2007- 2012) to measure the effects of the ADP in terms of providing new sustainable economic activities, principally based on the development of new techniques and organizations for the production of alternative crops like cacao, coffee and palm oil, and reducing poverty in selected areas of Huanuco, San Martin, and Ucayali. To accomplish this goal, the evaluation team compiled and organized available information from DEVIDA surveys, for the period 2007-2012 and the regions of Huanuco, San Martin and Ucayali, to better understand the selection process and implementation of ADP and Non-ADP communities and households. This analysis helped to establish what aspects of the programs results could be explored and measured using a retrospective analysis of available data through:  An understanding of the strategy, timeline, and types of interventions of AD projects in the selected areas.  An understanding of the selection criteria and characteristics of ADP communities and households, as well as comparisons with Non-ADP communities and households.  Exploring possible methods for evaluating the impact of AD projects in communities and households, and present some initial results. The main goal of this evaluation was to provide a methodological framework that allowed for an understanding of the mechanisms by which different project components had an impact on households participating in the ADP, compared with similar households that did not participate in the project. In that sense, the evaluation answers the following questions using the information available from the annual DEVIDA impact surveys: 1. How much of the ADP participating households´ income can be attributed to the effect of the program? 2. In comparing ADP subjects with Non-ADP subjects, do we find significant differences in regards to income, productivity and profitability of program crops (coffee, cacao, palm), as well as other sources of income? 3. How long must ADP be present in order to have an impact on these indicators? 8 2. PROJECT BACKGROUND For 30 years, drug producers have promoted and cultivated illicit coca in the valley areas of Huanuco, San Martin and Ucayali as well as throughout the Apurimac river and Ene valley (VRAEM), which includes parts of the Ayacucho, Cusco and Junin regions. New areas have been identified in San Gaban, Upper Inambari and Tambopata in Puno Region, and Kosñipata in Cusco, Putumayo in Loreto, Ongon, Pataza and Gran Chimu in La Libertad. Peru’s densest illicit coca production and processing thrive in impoverished areas with little or no government presence, bringing violence and lawlessness and providing a source of financing for remnants of terrorist groups. Given the charged and threatening atmosphere of narco-trafficking in these areas, government services and development projects cannot rely on police presence, which makes education, health, and private investment projects very costly and difficult to implement. Aside from illicit coca production, these areas typically support subsistence agriculture and livestock production. Licit economic and social development are stymied without access to markets, lack of roads, and limited communications networks.3 The United States Government (USG) and the Government of Peru (GOP) have implemented alternative development (AD) activities since the nineties. From 2002-2006, the USG and the GOP agreed to achieve sustained reductions of illicit coca crops through alternative development in priority areas of Peru using the voluntary eradication of coca plant cultivation as the main strategy. With the support of local and regional governments as strategic allies, interventions focused on communities with low to medium density coca cultivation that, through formal agreements, committed to eliminate illicit coca cultivation in favor of licit development. Within each participating community, households signed agreements with DEVIDA to eradicate their coca. In return, DEVIDA, backed by USAID funding, committed to implement a package of development activities, including licit crop development and infrastructure improvement activities, among others. In 2006, the program entered a new phase of implementation. Communities with a greater dependence on coca cultivation were not disposed to enter into voluntary reduction agreements. The GOP initiated programmed (forced) eradication in areas of high-density illicit coca cultivation. A new post￾programmed eradication approach addressed higher-density coca areas where minimum security and development conditions were in place after programmed coca eradication. The ADP, with the signing of “Agreements of Understanding” with communities willing to participate in the development of new crops, provided the population with incentives to not replant. In parallel, the ADP continued to consolidate coca reduction achieved in ‘voluntary eradication’ communities by maintaining support to ongoing agricultural activities. Efforts added a strong communications component and activities to build social capital to consolidate gains. In accordance with the National Integrated Sustainable Alternative Development Program of the National Anti-Drugs Strategy designed by DEVIDA in alignment with the Assistance Agreement signed with USAID in 2008, the AD program sought to consolidate and extend changes in behavior in the areas 3Macroconsult (2012). Drug Trafficking: The Threat to Peru´s Sustainable Growth. Studies on coca, cocaine, security and development. 1ª. Ed., Lima: MACROCONSULT S.A. 9 prioritized by the ADP, as well as change regional and national public opinion in favor of a licit style of life. This behavioral change involves abandoning the cultivation of coca within a vision of local development directed at achieving sustainable economic development for families, enhancing social capital, strengthened governability and licit lifestyles within the sphere of the program4. The program aims to increase the families’ legal income by carrying out economically viable activities such as the cultivation of cacao, coffee and palm oil. This in turn contributes to establishing conditions that will promote families’ decision to not replant coca. To achieve this result, ADP takes into account the economic viability of the agricultural activity, the efficiency of local producers’ organizations, an increase in private investment, access to credit and financial services, proper management of natural resources, public-private alliances for investment in economic infrastructure such as roads or warehouses, and public services. The program also aims to increase social capital, which is considered an important factor in achieving sustainability in the behavioral changes towards licit lifestyles. Strengthening social capital involves a growing level of trust between the families in the communities, strong leadership, greater trust in the national government and a common vision of the local development. Strengthening governability is achieved with the participation of local and regional governments to promote citizen participation, ensure public investment and lawful ways of living. It implies enhancing the capabilities of local governments, the incorporation of the regional governments as key allies of the ADP, reinforcing the institutional capability of DEVIDA and other relevant national governmental and non-governmental bodies. Previous evaluations of the ADP5,6 carried out have shown favorable results in the wellbeing and lifestyle of the population following the change from coca leaf cultivation to the production of legal crops. Communities that signed the Framework Agreement to eradicate coca production, did so to improve their security, enhance opportunities for investment, improve the communities’ economies, foster a friendly family environment with better services and public and private infrastructure, as well as projects that generate income through productive activities. The communities that participated in the program benefited from infrastructure and productive activities projects, such as roads and bridges, cacao, coffee, and palm oil crops as well as other activities. The success of the ADP is also measured by the reduction in coca production, which is measured by the density of coca leaf production in hectares through satellite images. The DEVIDA survey7 in 2011 showed that the farmers’ financial situation improved in ADP areas. During the 2006 – 2011 period, the perception of wellbeing rose by 12 percentage points, while non￾ADP areas registered an increase of 4.6 percentage points only. Also, between 2007 and 2011, ADP communities increased their income from agriculture by 4.2 percentage points. Average levels of total income rose also during the 2008 – 2011 period; in the case of ADP communities, the increase was 4 Estrategia Nacional de Lucha contra las Drogas 2007-2011 5Weidemann Associates Inc. (2010). Lesson for future programming. USAID Peru´s Alternative Development Program. 6Weidemann Associates Inc. (2006). Relevance and effectiveness of USAID/PERU´S Alternative Development Program and Strategy. 7DEVIDA (2011). Informe Encuesta Impacto PDA 2011. Lima: Dirección de Promoción y Monitoreo, Comisión Nacional para el Desarrollo y Vida Sin Drogas-DEVIDA, Presidencia del Consejo de Ministros. 10 52.7%, and for non-ADP communities, the increase was 21.6%. These initial comparisons give us some indications of the potential effects of the ADP on participating families. 11 3. EVALUATION METHODS & LIMITATIONS 1.1. Types of interventions to evaluate The different stages that ADP has gone through create distinct beneficiary populations that are relevant to our analysis. An important differentiator is participation in post-eradication agreements (which took place in some communities of San Martin), and those that followed a voluntary eradication of coca crops. Another distinction exists between agreements that include a majority of the community and others (since 2010) where a minority proportion of community members participated by signing an Affidavit (which differentiates these two types of communities in their level of acceptance of the program). Thus, we have i) the “R-379” communities that are part of 379 communities which signed an agreement for “gradual and concerted” reduction from 2002 to 2004; ii) forced eradication communities between 2006 and 2008 (post eradication); iii) communities that signed the agreement for concerted gradual reduction from 2005 onwards (designated as “Plan 20XX”); and iv) communities that did not sign the agreement but where a minority of families signed an Affidavit. On the other hand, the program interventions are directed at two levels of beneficiary: • Community: Infrastructure work8 • Family: Productive or non-productive projects9 This raised new differences in the types of intervention to be analyzed. An ADP community with an infrastructure component has an externality that benefits the entire community, not just those families subscribed to the program so these may serve as a control group to compare the impact of the entire program. a. The sample of communities selected for the period 2007-2012 The complete sampling frame for the annual DEVIDA ADP survey per year, department and type of intervention is presented in Annex 1. For the next section we considered only the 3 regions of interest (Huanuco, San Martin and Ucayali) in rural areas10. Table 1 presents the distribution of ADP and NADP communities per year and department, and the percentage each of these groups represent of the total number of communities in the sampling frame. In general there was a tendency to select a higher percentage and number of ADP than NADP 8 By 2012, 444 communities of the sampling frame had benefited from the infrastructure component and 396 had not. Most of them are in San Martin (198), although the percentage of communities with this component is higher in Ucayali (74%) and Huanuco (69%) than in San Martin (40%). 9 The ADP Communities that would have had such projects are identified only in the Sampling Frame of 2009, but it is possible that this information is sufficient to assume that such implementation was constant during the time these communities were intervened. 10 Some communities have been removed since inconsistencies between the geographic location information of the surveys and the sampling frame could not be solved. However, there were few such cases and most of them belong to urban areas. 12 communities from the sampling frame, which limited the chance of finding similar communities as part of the control group needed for a sensitive impact evaluation. Table 1: ADP and NADP communities by region and year. Parentheses indicate the percentages from the total sampling frame for the annual DEVIDA surveys. Huánuco San Martin Ucayali Total ADP NADP ADP NADP ADP NADP ADP NADP 2007 52 15 121 44 36 14 209 73 (41.9) (4.7) (29.0) (6.6) (18.8) (26.9) (28.5) (7.1) 2008 38 23 96 46 37 15 171 84 (28.6) (7.2) (22.1) (7.0) (18.9) (23.4) (22.4) (8.0) 2009 31 21 80 36 25 15 136 72 (19.6) (6.8) (16.1) (6.0) (12.4) (25.4) (15.9) (7.5) 2010 32 19 70 30 37 14 139 63 (18.2) (6.2) (14.1) (5.0) (17.9) (24.6) (15.8) (6.6) 2011 31 24 57 40 31 16 119 80 (17.6) (7.8) (11.4) (6.7) (14.8) (28.1) (13.5) (8.3) 2012 31 24 55 42 31 17 117 83 (17.6) (7.8) (11.0) (7.0) (14.8) (29.8) (13.2) (8.7) Table 2 presents the number of communities by intervention model, with the largest concentrations corresponding to the R-379 and Plan 20xx groups, which pertain to the voluntary eradication strategy. 13 Table 2. Communities by type of intervention, region and year. ADP NADP R-379 Plan 20xx Post Eradication Affidavit Total Huánuco 2007 33 19 0 0 52 15 2008 23 15 0 0 38 23 2009 17 14 0 0 31 21 2010 19 10 0 3 32 19 2011 17 12 0 2 31 24 2012 16 9 3 3 31 24 San Martin 2007 40 72 9 0 121 44 2008 29 59 8 0 96 46 2009 23 41 16 0 80 36 2010 22 32 14 2 70 30 2011 13 30 10 4 57 40 2012 14 26 8 7 55 42 Ucayali 2007 16 20 0 0 36 14 2008 15 22 0 0 37 15 2009 11 14 0 0 25 15 2010 21 15 0 1 37 14 2011 13 14 0 4 31 16 2012 13 15 0 3 31 17 Total 2007 89 111 9 0 209 73 2008 67 96 8 0 171 84 2009 51 69 16 0 136 72 2010 62 57 14 6 139 63 2011 43 56 10 10 119 80 2012 43 50 11 13 117 83 b. Identification of ADP and NADP families Over the six-year period, there have been several changes to the household survey questionnaires used to evaluate the performance of the program. One of the most relevant for our analysis is the change in the way in which families/households are identified as being a participant of the program. In this section we go over these changes and how they might affect the evaluation (this is summarized in Annex 2)11: • All years have a common question prior to identification: "Do you know or have heard of ADP?" This question should have served as a filter for the question that identifies households participating in the program, but it was not always applied in this way, which raised different assumptions for each year. 11 There are, however, some inconsistencies or questions that did not follow the correct filters. If, for example, a family in an ADP community declares to be participating in the program but this question was only meant for ADP communities, it will not be considered as a valid answer. 14 • In 2007 identification question was "Have you or has someone in your family participated in the signing of the framework agreement with DEVIDA for the gradual reduction of coca?” This question was only asked to families in ADP communities. • In 2008 identification question was "Have you or has someone in your family participated in the framework agreement/Agreement Act with DEVIDA for the gradual reduction of coca?” This question was only asked to families in ADP communities. • In 2009 identification question was "Have you or someone in your family participated in the framework agreement/Agreement Act with DEVIDA for gradual reduction of coca crops, or Affidavit to participate in the ADP?” This was asked to all families in ADP communities.12 • Since 2010 the identification question was, "Are you or is someone in your household a participant of an ADP productive project (agriculture / livestock)?” This, in 2010, was asked to all families who claimed to know about the program while in 2011 and 2012 to all families in ADP communities who claimed to know about the program. Although the question itself has changed over the years, it also reflects a change in the program as it incorporates new forms of participation in the project. The main problem is how to deal with groups that, for some reason, are not asked whether they participate in the program, creating uncertainty regarding the status of respondents regardless of their membership in ADP or NADP communities. This leads us to make certain distinctions among the households that are considered "ADP" and "NADP". Table 3. Identification of participant families ADP community NADP community Knows Yes No Yes No Participates Yes No Yes No Yes No Yes No 2007 A1 B1 A1 B2 B3 B3 B3 B3 2008 A1 B1 A1 B2 B3 B3 B3 B3 2009 A1 B1 A1 B2 B3 B3 B3 B3 2010 A1 B1 B2 B2 A2 B3 B3 B3 2011 A1 B1 B2 B2 B3 B3 B3 B3 2012 A1 B1 B2 B2 B3 B3 B3 B3 As presented in Table 3, the shaded areas are those where the question about participation (which is key to identification) was not asked. Thus, we have two different types of “ADP" families (A) and 3 types of “NADP" families (B):  ADP families o A1: Families in ADP communities that participate in the program. o A2: Families in NADP communities that participate in the program. It is important to separate them from the first because they do not benefit from infrastructure projects 12 However, this conclusion is based on the information gathered. According to the survey only to the families of NADP communities that responded negatively to the question of execution of infrastructure work in the community should have been excluded. 15 developed at the community level. Moreover, these families have different community’s characteristics given that these communities do not participate in the program.  NADP families o B1: Families in ADP communities who claim to know the program but choose not to participate. o B2: Families in ADP communities who claim not to know the program. Although this most likely indicates that they are not participants, also gives us information about the level of disconnection of these families from the rest of the community. o B3: Families in NADP communities that are not asked about participation or that say they do not participate in the program. Similar to the case of A2, these families have the characteristics proper to a NADP community. Table 4 presents the number of families we have in each of the five categories by department and year. Table 4: Families by intervention category for region and year Year A1 A2 B1 B2 B3 Total 2007 3,045 0 681 0 1,230 2008 2,661 0 391 0 1,579 2009 2,081 0 458 0 1,382 2010 1,701 102 755 185 1,151 2011 1,010 0 1,013 265 1,706 2012 784 0 1,048 395 1,717 Huánuco 2007 825 0 121 0 251 2008 564 0 114 0 447 2009 375 0 134 0 405 2010 292 34 234 92 334 2011 196 0 295 90 477 2012 185 0 288 132 467 San Martin 2007 1,695 0 475 0 753 2008 1,603 0 142 0 869 2009 1,362 0 153 0 703 2010 1,026 36 276 47 593 2011 572 0 411 121 917 2012 379 0 480 182 897 Ucayali 2007 525 0 85 0 226 2008 494 0 135 0 263 2009 344 0 171 0 274 2010 383 32 245 46 224 2011 242 0 307 54 312 2012 220 0 280 81 353 16 1.2. Description of potential treatment and control groups and possible methods of impact evaluation In assessing possible estimation methodologies, we consider only 'A1' as a possible treatment group (i.e. the ADP families in ADP communities). We do this because A2 is too small. Furthermore, for simplicity, we will take two different types of control: NADP families that are part of an ADP community (B1 and B2) and NADP families that are part of NADP communities (B3). In all cases, the objective is to compare a participating family with another family that is not participating but is similar enough in terms of structural features and in terms of the variables that may have determined its participation in the program. To this end, we will seek to use matching techniques both at the community and household levels. Furthermore, the analysis will seek to differentiate program effects by department, type of intervention strategy (detailed in Table 1) communities with and without infrastructure projects, and families with and without production projects. In the next sections, we present an analysis of different methods of estimating program impacts that we may use with the information obtained and the identification of types of interventions in communities and household classification shown in the previous sections. a. Cross-section analysis In this case, participant families will be compared against non-participating families (in either ADP or NADP communities). Table 5 presents the total treated families and potential controls in both ADP communities (C1) and NADP communities (C2). Table 5. Families by treatment and control group for region and year Huánuco San Martin Ucayali Total T C1 C2 T C1 C2 T C1 C2 T C1 C2 2007 825 121 251 1,695 475 753 525 85 226 3,045 681 1,230 2008 564 114 447 1,603 142 869 494 135 263 2,661 391 1,579 2009 375 134 405 1,362 153 703 344 171 274 2,081 458 1,382 2010 292 326 334 1,026 323 593 383 291 224 1,701 940 1,151 2011 196 385 477 572 532 917 242 361 312 1,010 1,278 1,706 2012 185 420 467 379 662 897 220 361 353 784 1,443 1,717 17 b. Difference in difference: Communities that become part of ADP during the period of analysis. An ideal scenario would be to measure changes over time in households that belong to a community that was not originally part of the program and then joins it. In this case communities whose status changed from NADP to ADP would be compared against communities that remained as NADP. Observations before the intervention would provide a baseline against which the impact of the program can then be measured by the differences in the change of dependent variables. Unfortunately, while there are cases of communities during the period under review that have gone from NADP to ADP in the sampling frame, there is no information in surveys for these communities in 2 years of the sample. Therefore, this option is not viable. c. Difference in difference: ADP communities against NADP communities In this final approach, the objective is to compare the differences between the income growth of households in communities that are part of the program, compared to households in communities that are not. Here there are two possibilities: Analyze information through time for a group of communities that are present in the sample for more than one year, or assume that communities, ADP or NADP, are similar to each other and therefore it is possible to take each annual sample of communities as representative of all communities ADP or NADP. We must take into account that there is no identification of households in each community for different years, so even if a community is sampled in more than one year, the households interviewed are likely to vary. d. Same communities Analyzing data available from the same communities over time restricts the amount of available information, but also reduces the potential for selection bias by controlling for unobserved variables. Assuming unobserved variables remain constant over the observation period, the bias is eliminated by virtue of taking differences measured before and after among the same group. However, because this is a fixed effects model, the analysis loses variance and, thus, precision, resulting in a greater standard deviation. In short, a panel of the subset of repeatedly sampled communities’ benefits from a low risk of selection bias, but will be less able to detect small effects. Table 6. ADP and NADP families by number of years they show up in the sample Huánuco San Martín Ucayali Total Years sampled ADP NADP ADP NADP ADP NADP ADP NADP 1 50 68 170 100 72 25 292 193 2 29 21 74 30 26 9 129 60 3 15 4 25 6 14 5 54 15 4 9 1 13 7 2 9 24 17 5 4 0 6 6 4 0 14 6 6 1 0 0 1 0 0 1 1 18 Table 6 presents the number of years in which ADP and NADP communities were sampled, revealing that 222 ADP and 99 NAPD communities were sampled in more than one year. Aside from having a sufficient number of repeatedly sampled communities, it is important to compare particular sets of communities that are repeated in the same years. Table 7. Communities that are sampled for each pair of consecutive years by region 2007-2008 2008-2009 2009-2010 2010-2011 2011-2012 ADP NADP ADP NADP ADP NADP ADP NADP ADP NADP Huánuco 15 1 9 1 15 3 10 2 10 6 San Martin 37 7 16 11 24 13 16 11 11 18 Ucayali 7 4 6 2 6 7 13 8 10 11 Table 7 reveals the number of ADP and NADP communities that are present in the sample in each pair of consecutive years by department. We see that at the departmental level the number of communities (especially NADP) can be small, making it difficult for an analysis given the reduced number of control families. Table 8. Communities (ADP/NADP) that are sampled for each pair of years 2007 2008 2009 2010 2011 2012 2007 2008 59/12 2009 54/12 31/14 2010 43/12 38/11 45/23 2011 41/16 33/18 37/27 39/21 2012 35/14 31/17 32/27 41/23 31/35 Table 8 presents the number of ADP and NADP communities that are repeated between any two years. e. Different communities This method, by not requiring the same communities to be used over time, utilizes the same subset showed in Table 5 but combined with a temporal analysis. However, the success of this method depends not only on finding appropriate controls in each time period for participating families, but for the whole treatment and control groups to be comparable over time given that the sampled communities and families are not the same. 19 1.3. Recommendations for the evaluation method to use Given the availability of information and the analysis described in the previous sections, the best possible scenario was to use information on ADP families from R-379 and Plan20XX communities as the treatment group and NADP communities as the control group. Regarding the analytical methodology, we use a cross-section analysis and a temporal analysis using ADP and NADP communities for which information is available in at least two different years (Table 8). This sub sample allows us to combine an analysis of cross-sectional effects, comparing ADP and NADP households in the same year, with a look at possible effects over a longer period in households in the same communities with the use of a difference-in-difference technique. This began with an analysis of Propensity Score Matching among community and household-level variables to limit the evaluation to households that are similar to each other in both groups. This analysis determined the final sample used to evaluate the impact of the ADP, as well as their distribution by department and type of intervention. For the temporal analysis, all communities in the subsample were used since the sample is not large and additional pairing would have negatively affected the number of controls. a. Limitations for the estimation of impact Several factors limit the estimation of impacts. First, there is insufficient information at the community level about the main criteria for the selection of ADP communities and families as well as of the components of the intervention.13 This can present a problem because both the selection of communities by the program and the voluntary self-selection of communities and families to participate in it may be related to characteristics that are not observable, or for which no information is available, but that affect the variables measuring impact. For this reason, we used secondary information that is available only at the District level. Figures 1 and 2 show the relationship between poverty and other development indicators (y-axis) and the proportion of ADP communities at the district level (x-axis)14 . Although there are no significant relations, districts with no ADP communities tend to have different characteristics than those districts that have at least one ADP community, as illustrated by the inclined trend lines. For the purpose of increasing similarity between treatment and control groups, the analysis restricts the sample of control communities to those that belong to a district with at least one ADP community. 13 This information was not systematically registered or collected by DEVIDA. 14 Percentage of ADP communities in the district (number of ADP communities/total number of communities) 20 Figure 1. Selection bias by proportion of ADP communities in a district. Huanuco. Figure 2. Selection bias by proportion of ADP communities in a district. San Martin. Second, there is no baseline survey and there are insufficient control communities in the selected sample. This prevents us from knowing the status of communities before the implementation of the program, which would be useful in measuring the improvement that ADP communities had compared to those that have not participated in the program. The lack of a baseline makes it more important to have enough control communities to allow for the identification of comparable NADP communities. 21 Third, the surveys have not had a follow-up strategy that would guarantee that a family is surveyed in different time periods in order to evaluate more accurately the impact of the program. As a result, there is no panel of communities and families over the years. Such a panel would have been useful to assess the evolution of the same families and communities over time. Finally, there is insufficient information in the survey on the determinants of family participation and on impact indicators to evaluate. b. Evaluation method Given the previous analysis, the selected methodology was the Propensity Score Matching with nearest neighbor and repetition. To estimate the probability of participation, the following variables were used:  District variables o % who received technical assistance to improve production in their agricultural unit, o % who assigned at least a crop for agricultural production, o % of producers that reported sufficiency in agricultural practices. o % households with water. o % households with electricity.  Household variables o Hectares owned or managed o Cultivated Hectares o Years in the community (head of household) o Age (head of household) o Sex (head of household) By restricting the controls to common support and choosing the nearest neighbor for each beneficiary, the control group ended up being much more similar to the treatment group than it was before matching, as is show in Figure 3. Figure 3. Population density in control and treatment group for the probability of participation 22 4. FINDINGS, CONCLUSIONS & RECOMMENDATIONS 1.4. Initial comparison between ADP and NADP families Given that there is no baseline, information from the 2007 survey provides a reference for assessing the differences between ADP and NADP communities. This assessment helped to clarify which variables were relevant for the matching. The data also allowed us to measure the status of key variables at the beginning of the evaluation period in preparation for the measurement of change over time. Table 915 presents the differences between ADP families by type of intervention and NADP families. For Huanuco, we observed significant differences among some household variables, such as number of hectares owned or managed, number of hectares cultivated, and percentage with a male head of household. These factors were higher for ADP families. As for income, NADP families have higher income from coca, but lower income from coffee or livestock. Income from coffee or livestock is higher in ADP families. Many of these variables will be used for the PSM analysis reducing the difference between the new comparison groups. 15 Marked results indicate a significant difference (95%) from NADP families. Color indicates positive (green) or negative (red) difference. 23 Table 9. Family and income variables between ADP and NADP communities in 2007. Huanuco. ADP No ADP Total R379 Plan 20XX Total Years in the community 24.01 * 24.73 22.66 22.25 Hectares (owned or managed) * 35.91 * 35.74 * 36.21 22.3 Hectares (cultivated) * 3.15 * 2.892 * 3.638 2.477 Age Head of Household 44.03 44.1 43.89 43.55 Male Head of Household * 0.9297 * 0.9165 * 0.9545 0.8398 Productive income 3883 3657 4309 3273 Agricultural income 2295 2677 1575 1956 Crop income 2293 2676 1570 1932 Coca * 31.1 * 29.27 * 34.54 313.6 Cacao 369.5 536.8 * 54.16 253.7 Coffee * 833 * 844.7 * 811.1 239.1 Irrigated rice 231.1 353.7 0 902.2 Corn 138.5 * 31.35 340.5 157.5 Palm 0 0 0 0 Banana 558.1 * 812.2 * 79.09 233.1 Processed agric. products income 2.425 0.9388 5.227 24.05 Livestock income * 1331 812 * 2309 516.6 Forest Income 224 132.1 * 397.2 42.32 Extractive act. income * 33.09 * 35.75 28.07 757.9 Wage income 2240 2071 2557 2307 Family income 6123 5728 6867 5579 Per capita family income 1698 1655 1780 1370 In the case of San Martin there are also differences in family variables such as years in the community in favor of ADP families and gender of the head of household in Plan20XX families. However, ADP families have significantly lower income than NADP families in its main variables: productive income, wage income, family income and per capita family income, among others. 24 Table 10. Family and income variables between ADP and NADP communities in 2007. San Martin. 16 ADP No ADP Total R379 Plan 20XX Total Years in the community * 29.28 * 29.14 * 29.39 25.67 Hectares (owned or managed) 18.1 18.38 17.91 18.95 Hectares (cultivated) 3.544 3.576 3.521 3.352 Age Head of Household 42.54 43.64 41.76 43.06 Male Head of Household 0.9522 0.929 * 0.9686 0.9452 Productive income * 3708 * 3302 * 3996 5320 Agricultural income * 3103 * 2739 3361 4464 Crop income * 2984 * 2633 * 3233 4376 Coca * 12.58 * 24.43 * 4.162 107.1 Cacao * 619.4 * 790.1 * 498.1 262.9 Coffee 1105 * 449.6 * 1572 1125 Irrigated rice * 166.4 * 79.74 * 228 1498 Corn 304.6 286.7 317.3 275.7 Palm 8.073 0 13.81 0 Banana * 469.5 * 692.2 311.2 218.7 Processed agric. products income 119.2 106.5 128.3 87.79 Livestock income * 460.3 * 391.6 * 509.2 735.2 Forest Income 102.6 136.4 78.66 85.86 Extractive Act. income 41.74 34.55 46.84 34.81 Wage income * 1732 * 1642 * 1797 2504 Family income * 5440 * 4943 * 5793 7823 Per capita family income * 1507 * 1436 * 1557 2156 In the case of Ucayali, the only difference in terms of family variables is the number of years that the average ADP household lives in the community. As in Huanuco, there are no differences in the main income variables, but there are some for more specific incomes such as crop income or income from cacao and coffee, both in favor of ADP families. 16 Marked results indicate a significant difference (95%) from NADP families. Color indicates positive (green) or negative (red) difference. 25 Table 11. Family and income variables between ADP and NADP communities in 2007. Ucayali.17 ADP No ADP Total R379 Plan 20XX Total Years in the community * 20.73 * 22.02 19.78 17.53 Hectares (owned or managed) 27.68 26.26 28.74 29 Hectares (cultivated) 3.87 4.16 3.656 3.531 Age Head of Household 44.92 44.2 45.44 45.94 Male Head of Household 0.939 0.9327 0.9437 0.9381 Productive income 6515 6492 6531 7409 Agricultural income 4885 * 5433 4480 3662 Crop income * 4852 * 5427 4427 2996 Coca 20.58 14.97 24.73 0 Cacao * 531.4 * 1145 78.61 8.85 Coffee * 136.7 * 183.4 * 102.3 15.88 Irrigated rice * 0 * 0 * 0 196.4 Corn 198.9 110.9 * 263.8 93.85 Palm * 1742 * 1309 * 2061 404 Banana 1175 2133 * 467.6 1158 Processed agric. products income * 33.02 * 5.816 * 53.11 665.9 Livestock income * 597.2 * 508.4 * 662.7 2285 Forest Income 880 502.5 1159 1310 Extractive Act. income 152.5 48.42 229.3 152.4 Wage income 4600 4925 4360 5185 Family income 11115 11417 10891 12594 Per capita family income 2798 2795 2800 3080 17 Marked results indicate a significant difference (95%) from NADP families. Color indicates positive (green) or negative (red) difference. 26 1.5. Cross section analysis The complete results for the Propensity Score Matching (PSM) for each year and region are shown in the annexes, including average incomes in Annex 3 and income differences between treated and controls in Annex 4 for productive, wage, family and family per capita income. From this analysis, we only observe a significant and positive effect on family income for ADP communities in Huanuco for the year 2011. This was driven by an important difference in productive income compared to NADP communities. However, when we separate ADP communities by type of intervention (R- 379 and Plan20XX), we find some effects for R-379 communities in Huanuco and Ucayali, but not in San Martin, and these positive effects are clearer for 2010, as presented in Table 12. In order to determine whether these findings have to do with an increase in the income generated by the production of alternative crops like cacao or coffee, which usually takes a few years to become productive, we proceeded to explore the evolution of income from these crops in the case of R -379 communities and NADP communities that were chosen as controls. Table 12. Average income effects for R379 communities by region and year. San Martín Ucayali Productive 1,313 -8,054 -328 Wage + 1,334 -1,250 -1,648 Familiar + 2,647 - -9,303 -1,976 Familiar PC -890 -2,468 514 Productive -2,086 1,051 1,889 Wage -517 71 1,476 Familiar -2,602 1,122 3,365 Familiar PC -208 + 504 67 Productive 1,561 -277 4,451 Wage 497 46 -381 Familiar 2,058 -231 4,070 Familiar PC 490 96 + 1,844 Productive + 5,958 1,628 + 10,133 Wage + 1,610 -467 -460 Familiar + 7,568 1,161 + 9,674 Familiar PC + 2,418 640 + 2,807 Productive + 6,062 -195 - -17,115 Wage 1,636 -318 230 Familiar + 7,698 -512 - -16,885 Familiar PC + 2,530 -563 -2,744 Productive 664 1,021 -464 Wage 517 168 -2,148 Familiar 1,181 1,189 -2,612 Familiar PC 88 609 -206 Year Income Huánuco 2007 2008 2009 2010 2011 2012 R379 R379 R379 27 In Table 13 presents an important improvement in incomes for ADP families between 2007 and 2011- 2012 in all regions in the case of cocoa and in Huanuco also in the case of coffee. In San Martin, however, income from coffee production appear to have grown more for NADP communities. Table 13. Evolution of cacao and coffee incomes in R-379 (ADP) and NADP communities. Huánuco San Martin Ucayali NADP ADP-379 NADP ADP-379 NADP ADP-379 2007 cacao 284 877 235 800 97 1,467 coffee 88 1,365 558 455 36 228 2008 cacao 1,168 1,123 839 2,199 0 1,028 coffee 1,327 923 965 681 0 0 2009 cacao 962 2,506 1,049 2,766 218 2,327 coffee 127 220 517 436 0 83 2010 cacao 2,967 4,146 1,183 5,946 364 3,422 coffee 60 3,739 2,820 331 0 0 2011 cacao 1,097 3,021 1,639 5,665 440 2,082 coffee 2,373 4,332 1,913 1,361 0 0 2012 cacao 2,729 3,136 1,326 4,500 249 3,862 coffee 776 2,069 1,733 799 0 16 28 1.6. Panel of communities Due to the limitations of a cross-sectional analysis with limited controls and few variables to properly choose the most relevant controls, we analyzed the feasibility of a panel impact evaluation using the communities that were sampled more than once over the evaluation period. Table 14 shows the number of ADP and NADP communities that were repeated between four pairs of years (2007/2008 and 2011/2012). Table 14. ADP and NADP communities that were sampled in each pair of years by region. 2007/2011 Huánuco San Martin Ucayali ADP R-379 9 8 4 Plan20XX 2 11 5 NADP 1 11 2 2007/2012 Huánuco San Martin Ucayali ADP R-379 7 7 2 Plan20XX 4 10 5 NADP 1 9 3 2008/2011 Huánuco San Martin Ucayali ADP R-379 8 2 2 Plan20XX 2 13 6 NADP 3 10 2 2008/2012 Huánuco San Martin Ucayali ADP R-379 8 6 2 Plan20XX 3 6 6 NADP 3 8 4 29 Table 15. Change in average incomes for panel ADP and NADP communities between each pair of years by region. Huánuco San Martín Ucayali Income NADP ADP NADP ADP NADP ADP 2011 - 2007 Productive 1,753 7,329 3,431 5,008 -9,364 9,109 Wage 2,788 2,493 1,755 2,474 2,166 -229 Family 4,541 9,822 5,187 7,482 -7,197 8,879 Family per capita 1,369 3,539 1,429 1,809 178 3,480 2011 - 2008 Productive 680 8,759 1,112 3,459 4,803 8,989 Wage 1,775 2,792 649 2,024 1,180 2,014 Family 2,455 11,551 1,761 5,484 5,983 11,002 Family per capita 561 3,399 938 1,069 3,555 3,189 2012 - 2007 Productive -509 3,638 3,011 3,483 -5,513 2,258 Wage 2,676 3,048 4,634 1,693 4,306 979 Family 2,168 6,686 7,645 5,177 -1,207 3,237 Family per capita 1,508 2,253 2,378 1,256 1,286 970 2012 - 2008 Productive 2,276 4,191 2,438 2,200 -2,684 7,941 Wage 677 3,508 2,859 946 4,675 3,960 Family 2,952 7,698 5,297 3,146 1,991 11,901 Family per capita 2,329 1,790 2,053 866 2,619 2,628 Table 15 presents the average difference in income between each pair of years for ADP and NADP communities. Keeping in mind that each pair of years included a different subset of communities, we observe some common patterns. We can see a significantly bigger improvement in ADP communities for Huanuco and Ucayali in most cases, but results in San Martin are not that clear. It seems that there were slightly positive results in this region until 2011 but these results became not significant or negative when we used the year 2012 as a comparison. 30 1.7. Conclusions The sample design and the data available from the annual surveys restricted the selection of analytical approaches and limited the power of the analyses. These limitations hinder the establishment of rigorous conclusions regarding the impacts of the ADP and restrain the provision of unambiguous responses to the main evaluation questions. Although we addressed some of these challenges with a careful reorganization of community data and the application of multiple analytical methodologies, we cannot discard biases in the initial selection of ADP communities and families because of: limited information available regarding the original selection criteria; the lack of an initial strategy to select NADP communities similar to intervention communities; and, the absence of panel data for both groups. However, we were able to observe indications of income changes at the household level that are potentially attributable to ADP interventions in these regions. The cross section analysis reveals that in 2011, only in Huanuco, there was a significant and positive effect on family income for ADP communities due to a substantial difference in productive income compared with NADP communities. However, when considering ADP communities by type of intervention (R- 379 and Plan20XX), we found significant effects for R-379 communities in Huanuco and Ucayali, but not in San Martin. These positive effects were clearer for the years 2010 and 2011. This effect is likely related with a longer period of participation of R-379 communities in the ADP, time that allowed for the long-term alternative crops supported under these interventions, such as cacao and coffee, to reach full productivity. In order to analyze whether these findings are related with an increase in the income generated by the production of alternative crops like cacao or coffee, we proceeded to explore the evolution of income from these crops in the case of R-379 communities and NADP control communities. We found an important improvement in income for ADP families between 2007 and 2011-2012 in all regions in the case of cocoa and in Huanuco also in the case of coffee. In San Martin, however, income from coffee production appears to have increased more among NADP communities. This last result, and the fact that households in NADP communities in San Martin initially reported higher income from other cash crops like rice and coca (see Table 10), would explain the absence of a total income effect in this region. Finally, we took advantage of the fact that some communities were repeated between samples of different years to analyze the average difference in income between each pair of years for ADP and NADP communities. In this panel18 analysis, we found a significantly bigger improvement in ADP communities for Huanuco and Ucayali in most cases, but results in San Martin are not that clear. It seems that in this region there were positive results until 2011, but that these results became insignificant or negative when we used the year 2012 as a comparison. However, given the above mentioned data limitations, and the fact that this was a panel of communities and not a panel of households, we believe that these results should be taken with caution. 18 In putting together this panel of communities, we were aware that it had some serious limitations to render rigorous results. First, because we are not watching a before - after ADP intervention but only the evolution in time of communities “always intervened” compared to communities “never intervened”. Secondly, given the design of the selection of communities, there is no certainty that there is no initial bias in the selection of both intervened and non-intervened communities. (See Section 3.3 Limitations for the estimation of impact in Evaluation Methods & Limitations and Section 4.3 Panel of Communities in Findings, Conclusions and Recommendations.) 31 1.8. Recommendations The study’s analysis and conclusions provide valuable insight into what measures should be taken to ensure a powerful and improved impact evaluation design for the new program in Huanuco:  In order to have a correct baseline for the study, a careful selection of those communities that will be part of the control group and those that will be part of the treatment group should be made at the beginning of the study.  Communities in the control group should share similar characteristics with treatment communities, to avoid, or at least to be able to identify, potential selection bias issues.  It is very important to collect time series data from a panel (i.e. surveying the same communities and/or households over time) in order to estimate the change of status between treatment and control groups over time.  Survey questionnaires need to incorporate more questions that will allow for the calculation of effective impact indicators and to construct variables that permit effective propensity score matching. 32 ANNEX 1: SAMPLING FRAME 2007-2012 The sampling frame contains the universe of communities that were part of the program and others that served as control group. Each year, a sample was taken from this universe for the household surveys. It is therefore important to understand in advance the characteristics of ADP and NADP communities and the way in which the intervention between regions and models of intervention is distributed. In Table 1 we see the amount of ADP and NADP communities that are part of the sampling frame by department and year, and the types of intervention to which they belong. Table 1. ADP and NADP communities by region and year in the sampling frame ADP NADP R-379 Plan 20xx Post eradication Affidavit Total Huánuco 2007 78 46 0 0 124 321 2008 78 55 0 0 133 321 2009 78 55 14 11 158 310 2010 78 55 24 19 176 306 2011 78 55 24 19 176 306 2012 78 55 24 19 176 306 San Martin 2007 84 249 84 0 417 662 2008 84 267 84 0 435 661 2009 84 267 108 37 496 596 2010 84 266 108 40 498 596 2011 84 266 108 41 499 596 2012 84 266 108 41 499 596 Ucayali 2007 124 68 0 0 192 52 2008 124 72 0 0 196 64 2009 124 72 0 5 201 59 2010 123 72 0 12 207 57 2011 123 72 0 14 209 57 2012 123 72 0 14 209 57 Total 2007 286 363 84 0 733 1035 2008 286 394 84 0 764 1046 2009 286 394 122 53 855 965 2010 285 393 132 71 881 959 2011 285 393 132 74 884 959 2012 285 393 132 74 884 959 33 On the other hand, Table 2 shows the number of ADP communities that have an infrastructure component by department and year. Table 2. ADP communities by infrastructure component for region and year in the sampling frame Huánuco San Martin Ucayali Total Infrst. No Infrst. missing Infrst. No Infrst. missing Infrst. No Infrst. missing Infrst. No Infrst. missing 2007 54 67 3 153 187 77 83 105 4 290 359 84 2008 100 33 186 249 106 90 392 372 0 2009 102 42 14 199 297 144 52 5 445 391 19 2010 102 45 29 198 300 144 51 12 444 396 41 2011 102 45 29 198 300 1 144 51 14 444 396 44 2012 102 45 29 198 300 1 144 51 14 444 396 44 34 ANNEX 2: IDENTIFICATION OF ADP FAMILIES IN DIFFERENT SURVEYS Question in the Survey Year of the Survey 2007 2008 2009 2010 2011 2012 ADP family? Everyone Everyone Do you know or have heard of ADP? Everyone Everyone Everyone Everyone Everyone Everyone So far this year, have infrastructure works or development projects in your community been executed, started or managed? Everyone Did you or anyone in your family participate in the signing of the framework agreement with DEVIDA for the gradual reduction of coca, to participate in the Alternative Development Program? ADP Did you or anyone in your family participate in the framework agreement/Agreement Act with DEVIDA for the gradual reduction of coca?" ADP Did you or anyone in your family participate in framework agreement/Agreement Act with DEVIDA for gradual reduction of coca crops, or Affidavit to participate in the ADP? Everyone (for ADP or communities with infrastructure work) Are you or is anyone in your household participant of a productive project (agriculture / livestock) supported by the ADP? All (From those who claim to know about ADP) All (From those who claim to know about ADP) All (From those who claim to know about ADP) 35 ANNEX 3: DIFFERENCES IN INCOME 36 Table A2. Average incomes for ADP and NADP communities chosen as the control group by region and year. Huánuco San Martín Ucayali Income NADP ADP NADP ADP NADP ADP 2007 Productive 5,004 4,197 4,904 3,707 - 11,443 7,506 Wage 1,995 2,378 2,180 1,697 4,965 5,363 Family 6,999 6,574 7,084 5,404 16,409 12,869 Family per capita 1,577 1,841 2,043 1,487 3,722 3,227 2008 Productive 4,342 4,503 4,888 5,919 6,530 6,379 Wage 2,673 2,631 2,550 2,174 3,843 3,767 Family 7,015 7,135 7,438 8,093 10,373 10,146 Family per capita 1,747 2,093 1,976 2,238 2,706 2,726 2009 Productive 9,874 6,802 10,520 5,607 5,447 8,992 Wage 3,607 3,860 2,977 2,687 3,968 4,563 Family 13,481 10,661 13,497 8,294 9,415 13,555 Family per capita 3,345 3,207 3,372 2,269 2,918 3,921 2010 Productive 7,722 8,867 8,102 8,476 8,338 11,485 Wage 6,754 4,300 3,984 3,016 6,916 5,805 Family 14,476 13,167 12,086 11,492 15,254 17,290 Family per capita 5,960 3,780 2,869 3,040 3,859 4,624 2011 Productive + 7,571 11,929 8,199 9,514 18,702 15,795 Wage 4,589 4,828 3,194 3,353 5,911 4,773 Family + 12,160 16,757 11,393 12,867 24,613 20,568 Family per capita + 3,091 5,040 2,708 3,511 6,032 6,013 2012 Productive 8,661 7,903 10,056 9,172 18,449 13,370 Wage 5,675 5,196 4,907 4,947 7,836 5,022 Family 14,336 13,099 14,963 14,119 26,286 18,391 Family per capita 4,678 3,700 3,636 3,580 5,828 4,679 37 Table A3. Differences between incomes of ADP families and their respective NADP families chosen controls. Huánuco San Martín Ucayali Income R379 Plan 20XX Total R379 Plan 20XX Total R379 Plan 20XX Total 2007 Productive 1,313 -1,714 -807 -8,054 -3,131 -1,183 -328 592 - -4,265 Wage + 1,334 339 613 -1,250 -859 -486 -1,648 1,139 407 Family + 2,647 -1,376 -194 - -9,303 -3,990 -1,669 -1,976 1,730 -3,858 Family per capita -890 -225 295 -2,468 -1,441 -555 514 597 -539 2008 Productive -2,086 777 161 1,051 419 994 1,889 -18 -156 Wage -517 590 -42 71 59 -483 1,476 751 -68 Family -2,602 1,366 120 1,122 478 510 3,365 733 -223 Family per capita -208 181 346 + 504 269 226 67 553 20 2009 Productive 1,561 -11,005 -3,063 -277 -2,062 -4,913 4,451 1,439 3,625 Wage 497 -527 316 46 -39 -271 -381 + 2,907 605 Family 2,058 -11,532 -2,748 -231 -2,101 -5,184 4,070 4,346 4,230 Family per capita 490 -2,145 -128 96 -334 -1,101 + 1,844 1,175 1,009 2010 Productive + 5,958 1,424 1,584 1,628 - -5,216 93 + 10,133 1,243 4,102 Wage + 1,610 -3,101 740 -467 150 -888 -460 2,244 -1,077 Family + 7,568 -1,677 2,324 1,161 - -5,066 -794 + 9,674 3,488 3,025 Family per capita + 2,418 -2,401 -364 640 -558 130 + 2,807 1,014 773 2011 Productive + 6,062 1,198 + 4,359 -195 1,819 1,429 - -17,115 4,949 -3,251 Wage 1,636 555 239 -318 -1,390 159 230 -1,344 -1,138 Family + 7,698 1,752 + 4,597 -512 429 1,588 - -16,885 3,605 -4,389 Family per capita + 2,530 1,481 + 1,948 -563 349 840 -2,744 1,481 -68 2012 Productive 664 -4,276 -855 1,021 -3,105 -407 -464 4,976 -5,080 Wage 517 1,763 -479 168 -1,523 7 -2,148 -293 -2,815 Family 1,181 -2,513 -1,335 1,189 -4,627 -400 -2,612 4,683 -7,894 Family per capita 88 -2,672 -1,082 609 -1,033 -14 -206 133 -1,150 38 ANNEX 4: STATEMENT OF WORK: RETROSPECTIVE IMPACT EVALUATION Background Information Illicit coca cultivation occurs in the valleys used by traffickers 30 years ago. Traditional areas include parts of the Regions in Huanuco, San Martin, Ucayali, Apurimac and river Ene valleys (VRAEM) which includes areas in Ayacucho, Cusco and Junin. New areas have been identified in San Gaban, Upper Inambari and Tambopata in Puno Region, and Kosñipata in Cusco, Putumayo in Loreto, Ongon, Pataza and Gran Chimu in La Libertad. In these areas, coca is illicitly cultivated on a large scale by residents and relates directly to their level of poverty. These areas typically support subsistence agriculture and livestock are marginalized without access to the market, lack of roads, and have a limited communications networks as well as weak governmental presence19. Since 1996, the U.S. Government has supported alternative development programs in Peru, and since 2002, the US and Peruvian governments signed the Grant Agreement Special Purpose Nº 527-0404 with the purpose of achieve sustained reductions of illicit coca crops through alternative development in priority areas of Peru, with a focus on the voluntary eradication of coca plant cultivation. This eradication strategy is carried out through the signing of framework agreements, with the support of local and regional governments as strategic allies. Since 2006, progress made under the strategy of voluntary reduction has declined considerably as communities with a greater dependence on coca cultivation have been less favorably disposed to voluntary reductions. As a result, the CORAH Project has had to carry out forced eradication in areas of high coca cultivation density. The ADP, with the signing of ‘Understanding Agreements’, began its support of post￾eradications communities with the aim of preventing the replanting of coca. In accordance with the National Integrated Sustainable Alternative Development Program of the National Anti-Drugs Strategy designed by DEVIDA, in alignment with the Donation Agreement signed with USAID, the ADP seeks to consolidate and extend changes in behavior in the areas prioritized by the ADP, as well as change regional and national public opinion in favor of a licit style of life. This behavioral change involves abandoning the cultivation of coca within a vision of local development directed at achieving sustainable economic development for families, enhancing social capital, strengthened governability and licit lifestyles within the sphere of the program. The program aims to increase the families’ legal income by carrying out economically viable activities such as the cultivation of cacao, coffee and palm oil. This in turn contributes to establishing conditions that will promote families’ decision to not replant coca. To achieve this result, ADP considers the economic viability of the agricultural activity, the efficiency of local producers’ organizations, an increase in private investment, access to credit and financial services, proper management of natural resources, public-private alliances for investment in economic infrastructure and public services. 19Macroconsult (2012). Drug Trafficking: The Threat to Peru´s Sustainable Growth. Studies on coca, cocaine, security and development. 1ª. Ed., Lima: MACROCONSULT S.A. 39 The program also aims to increase social capital, which is considered an important factor in achieving sustainability in the behavioral changes towards licit lifestyles. Social capital involves a growing level of trust between the families in the communities, strong leadership, greater trust in the national government and a common vision of the local development. Strengthening governability is achieved with the participation of local and regional governments to promote citizen participation ensure public investment and lawful ways of living. It implies enhancing the capabilities of local governments, the incorporation of the regional governments as key allies of the ADP, reinforcing the institutional capability of DEVIDA and other relevant national governmental and non-governmental bodies. The ADP evaluations2021 carried out have shown favorable results in the wellbeing and lifestyle of the population following the change from coca leaf cultivation to the production of legal crops. Communities that signed the Framework Agreement to eradicate coca production, did this to improve their security, enhance opportunities for investment to improve the communities’ economies, foster a friendly family environment with better services and public and private infrastructure, as well as projects that generate income through productive activities. These communities who have joined the program have infrastructure projects, mainly roads and bridges, cacao crops, coffee and palm oil, as well as other benefits. The success of the ADP is also measured by the reduction in coca production, as has being demonstrated by the density of coca leaf production in hectares through satellite images. The DEVIDA survey22 in 2011 shows that the farmers’ financial situation has improved in ADP areas. During the period 2006 – 2011 the perception of wellbeing roses by 12 percentage points, while non￾ADP areas register an increase of 4.6 percentage points only. Also, between 2007 and 2011, ADP communities increased their income from agriculture by 4.2 percentage points. Average levels of total income rose also during the period 2008 – 2011; in the case of ADP communities, the increase was 52.7%, and for non-ADP communities, the increase was 21.6%. These initial comparisons give us some indications of the potential effects of the ADP on participating families. Program to be evaluated In 2008, the governments of the United States and Peru signed the USAID Grant Agreement No. 527- 0423, as a result of the positive progress made in the fight against drugs in the previous phase. The new framework of results of the Alternative Development Program (ADP) aimed at consolidating these achievements in communities participating in the voluntary eradication program, and the involvement of new communities wishing to participate under the new program intervention approach. To achieve this objective, the parties seek to consolidate and expand behavioral changes in the target areas, as well as changes in public opinion at the regional and national level in favor of a lawful life style. Behavior change involves the abandonment of illicit coca cultivation within a vision of local development. This program hopes to achieve four outcomes: Outcome 1: Sustainable economic development of the ADP families, seeking to increase legal income, and developing agriculture and viable economic activities that encourage behavior change towards lawful life styles. To achieve this result, the program will consider the economic viability of agriculture, the 20Weidemann Associates Inc. (2010). Lesson for future programming. USAID Peru´s Alternative Development Program. 21Weidemann Associates Inc. (2006). Relevance and effectiveness of USAID/PERU´S Alternative Development Program and Strategy. 22DEVIDA (2011). Informe Encuesta Impacto PDA 2011. Lima: Dirección de Promoción y Monitoreo, Comisión Nacional para el Desarrollo y Vida Sin Drogas-DEVIDA, Presidencia del Consejo de Ministros. 40 effectiveness of local producer organizations, increased private investment, access to credit and financial services, proper management of natural resources, and public-private partnerships investment in economic infrastructure and public services. Outcome 2: Social Capital building in the ADP area. Social capital involves a growing level of trust between the families of the community, a consolidated leadership, greater confidence in the state and a shared vision of local development. Factors that influence this outcome include strengthened communal identity, strengthened community leadership, linkages between the community and the state, and promoting the role of women within the family and community. Outcome 3: Local Governance strengthened in ADP areas. To achieve this result the participation of local and regional governments will seek to promote citizen participation, and public investments as means of facilitating private investments, economic development and ways to promote lawful life. Strengthened governance involves reinforcing the capabilities of local governments, incorporating relevant regional governments as key ADP allies, strengthening the institutional capacity of DEVIDA and other national government agencies and NGOs. Outcome 4: Promote lawful lifestyles. Achieving this result implies increasing awareness with the participation of families and the general population, allowing a greater number of people to pursue lawful life styles. Factors that influence this outcome include: development of community behavior change strategies for leaders and members of the community, communication strategies and actions made available to key local and regional governments and partners, appropriate communication actions at regional and national levels, and the creation of positive public opinion concerning issues of combating drug trafficking. Peru's National Commission for Development and Life without Drugs- DEVIDA is responsible for the implementation of the National Strategy to Combat Drugs. USAID´s Alternative Development Program supports the GOP´s National Drug Control Strategy 2012-2016232425. The USG and GOP use a three￾pronged approach of interdiction, eradication, and AD to counter drug trafficking. USAID´s AD program, in collaboration with the GOP, has planted more than 73,000 hectares of cocoa, coffee, and palm oil in Peru´s San Martin, Ucayali and Huanuco regions since 2002 and provided substantial support for institutional and community development generally. The three-pronged counter narcotics model is now expanding both south into the Huanuco region and east into the Ucayali region, relying upon stepped-up interdiction, eradication and AD efforts. AD is an essential component to sustain any reduction of illicit crop cultivation26. Addressing basic development needs where the GOP has eradicated coca is critical to avoid a return to coca cultivation. Specifically, sufficient and reliable incomes from licit sources is a fundamental factor in reducing the risk of return to coca cultivation by families in eradicated areas. The Project´s development hypothesis is that the GOP will successfully consolidate and expand AD into the primary coca-vulnerable areas by the end of the project, by ensuring that: a) farmers in participant communities improve their family economies by generating income from alternative crops; b) members of participant communities have greater confidence in AD activities; and c) the GOP at national , regional, 23 DEVIDA (2012). National Drug Control Strategy 2012-2016. Lima: DEVIDA 24Macroconsult (2011) 25 UNODC (2012). El Modelo de Desarrollo Alternativo en la Región San Martín 26Weidemann Associates (2010). Lesson for Future Programming, USAID/Peru´s Alternative Development Program. 41 and local levels effectively responds to economic, social, and security development needs in participant communities. Consolidation and expansion of AD requires that economic and social progress (including institutional capacities) act to minimize the risk of a return to illicit crops in former coca-growing communities. USAID is implementing a comprehensive alternative development program in the region of Huanuco including activities to strengthen the regional government, improve health and education, and provide technical assistance to farmers making the transition from illicit coca to licit crops. The objectives of the AD program are to reduce poverty and the replanting of coca in the targeted areas. The following USAID´s Alternative Development projects are being implemented: ADP implemented by UNODC, “Digital Inclusion by CEDRO”, Peru Cocoa Alliance by CARANA Corporation, New Alternatives by New Alternatives Ventures, Development Credit Authority by Caja Nuestra Gente and other agencies, Selva Ganadora by Grupo ACP Inversiones y Desarrollo, and Technoserve. 42 Table 1. Current Alternative Development Activities Activity Title Partner Activity Description Location(s) Estimated Amount Start Date End Date Alternative Development in the Regions of San Martin, Huanuco and Ucayali United Nations Office on Drugs and Crime (UNODC) The activity’s objective is to generate a licit economy in coca-prone areas and prevent the return of coca cultivation. The project supports palm oil, cacao, and coffee production in communities where the GOP has eradicated coca leaf cultivation. San Martin, Ucayali and Huanuco $16,442,17 9 14/07/ 2006 1/31/2014 Digital Inclusion CEDRO This activity promotes social and economic development in alternative development areas. The Technological Centers for the Amazon (TCAs) enable current Information and Communication Technology (ICT) non-users to benefit from internet access. The activity connects citizens with their region, the country, and the world. The TCAs will become the bridge that provides access to services, skills and capabilities, and market opportunities. San Martin, Ucayali and Huanuco $8,000,000 5/4/20 12 5/3/2015 Peru Cocoa Alliance CARANA Corporation The objective of the alliance is to promote alternative economic development in former coca-growing areas. It provides households in the regions of Huanuco, Ucayali and San Martin with sufficiently attractive licit sources of income to prevent a return to coca growing. It also increases standards of living; integrates smallholder farmers and producer organizations into inclusive and sustainable value chains; facilitates access to finance, and builds capacity. San Martin, Ucayali and Huanuco $36,050,77 0 10/9/2 012 10/8/2016 New Alternatives/Nuev as Alternativas New Alternatives Ventures (NAV) To promote and sustain licit local development in communities and regions affected by illicit coca cultivation in Peru. Ucayali, Huánuco, and San Martin Regions. $12,575,94 7 09/04/ 2013 08/04/201 7 Institutional Strengthening of Peru's National Commission for Development and Life without Drugs (DEVIDA) National Commission for Development and Life without Drugs (DEVIDA) To increase DEVIDA's institutional capacity to plan, coordinate, implement, monitor and evaluate Peru's national counternarcotics strategy, in particular alternative development programs implemented with Government of Peru resources. National￾Level $19,000,00 0 01/01/ 2013 01/12/201 6 Development Credit Authority Caja Nuestra Gente, Caja Luren, Financiera Edyficar, EDPYME Proempresa. The activity aims to facilitate loan capital access to producer associations and small enterprise and demonstrate to the financial sector that long-term lending to the agricultural sector is viable. The activity will also create space for financial institutions to create appropriate financial tools to build long-term relationships with producer associations and small enterprises San Martín, Huánuco, and Ucayali $360,000 01/09/ 2010 01/09/201 7 Selva Ganadora Grupo ACP Inversiones y Desarrollo The activity's main objective is to help improve life in rural communities of the Peruvian Jungle through the promotion of entrepreneurial activities (both economic and social), the improvement of their business skills and the connection of their projects with the private and public sector. Lima, Huanuco, San Martin and Ucayali. $300,000 21/05/ 2013 22/05/201 5 Technoserve The main objective is to Improve Peru´s capacity to manage development in vulnerable areas affected by coca production through business practices of targeted producer organizations oriented to viable commercial markets, increase the productivity of AD crops, strengthen primary production, and improve access to high value and high growth markets. 43 USAID NAP USAID New Alternatives Program (NAP) was created to implement a series of activities that support all post-eradication efforts as well as the capacity building and the communications component. The NAP initiated activities on April 8, 2013. To ensure that the post-eradication transition process proceeds efficiently, USAID New Alternatives Program (NAP) implements a series of foundational activities that support all post-eradication efforts as well as the capacity building and the communications component. The foundational activities are achieved through three program components: Component 1: Transition Activities comprise the set of activities that will engage post-eradication communities, secure their commitment to remain free of coca cultivation and coordinate and implement a set of activities designed to facilitate committed communities transition to licit lifestyles. All Transition Activities are implemented in strict coordination with DEVIDA. Component 2: Capacity Building Activities comprise the set of activities designed to strengthen the capacity of public and private stakeholders to carry out activities and functions that contribute to the success of post-eradication activities, either directly or indirectly. They involve support to DEVIDA, regional and local governments, producer associations and cooperatives, SUNAT, MEF, and others. Component 3: Communications Activities encompass a variety of strategic support and operational activities that will directly facilitate the success of transition activities, build support for program objectives among public opinion, and promote the commercialization of licit production from post-eradication areas. Purposes and Use of the Information This design has two purposes:  To measure the magnitude of the effect attributed to the program in terms of providing new sustainable economic activities, reducing poverty and the replanting of coca in the prioritized areas of post-eradication, based in the analysis of DEVIDA surveys in Huanuco, San Martin and Ucayali (2007-2012); and  To develop an impact evaluation design for the ADP in Huanuco (2014-2017), in order to set the guide lines for the implementation and analysis of a baseline and follow-up surveys that should be part of a posterior impact evaluation of the new ADP. To accomplish the first purpose, the evaluation team will compile and organize available information from DEVIDA surveys and other databases, for the period 2007-2012 and the regions of Huanuco, San Martin and Ucayali, in order to provide a better understanding of the selection and evolution of communities and households in ADP and Non-ADP domains. The principal objectives of this analysis will be:  To have a better understanding of the timeline and different types of interventions of AD projects in the areas under study.  Have a better understanding of the selection criteria and characteristics of ADP communities and households, as well as comparisons with Non-ADP communities and households.  Explore possible methods for evaluating the impact of AD projects in communities and households, and present some initial results. 44 The outcome of this analysis will serve as: i) input for the evaluation design of the new projects to be carried on in Huanuco during the 2014-2017 period, ii) development of recommendations to improve the measurement of indicators and the DEVIDA survey, iii) discussions with USAID and DEVIDA on findings, scope and limitations of the study. The main goal of the evaluation design is to provide a methodological framework that allow understanding the mechanisms by which different project components have an impact on households participating in the ADP, compared with similar households which did not participate in the project. This evaluation design will addressed the ADP in Huanuco, where new communities from the Monzon valley are being included in post eradication activities. The specific objectives of this evaluation design are:  Systematize the timeline and different components of all ADP interventions in Huanuco region for the period under study in order to design a methodological strategy and a baseline survey to be implemented during the year 2014, and follow up surveys each year until 2017.  Define an identification strategy for communities and households treated by ADP in Huanuco, and propose potential communities and households to serve as a control group.  Identify all the potential effects developed as result of participating in ADP, and propose impact indicators and control variables.  Adjust the questionnaire used by DEVIDA in previous years, to incorporate new questions and sections in order to recover these new indicators and variables. The study will also propose a design for some interviews and focus groups in the region to pick up qualitative information.  Propose a sample design for communities and households in Huanuco region to be part of the baseline study and follow-up surveys.  Propose a methodological approach to estimate the effects of ADP by comparing treated and control communities and households in the period under study. Evaluation Questions For the retrospective evaluation of impacts of the AD projects in Huanuco, San Martin and Ucayali during the period 2007-2012, the following questions will try to be answered using the available information from the DEVIDA impact surveys: 1. How much of the income of households participating in ADP can be attributed to the effect of the program? 2. Are there significant differences on sources of income, productivity and profitability of program crops (coffee cacao, palm), other incomes, between ADP households and similar non-ADP households? 3. How much time of exposure to ADP will be required to demonstrate impacts on these indicators? The design of the impact evaluation for the ADP in Huanuco (2014-2017) implies identifying potential effects of the program and indicator to measure them, so that the consultants will specify in detail the evaluation questions to be answered as part of the future study. 45 Evaluation Methods Retrospective impact evaluation of ADP in Huanuco, San Martin and Ucayali Development hypothesis The sustainable economic development of the ADP families will be achieved through legal increases in family income, through the development of agriculture and viable economic activities that encourage behavior change towards lawful life styles. To obtain this result, the economic viability of agriculture, the effectiveness of local producer organizations, increased private investment, access to credit and financial services, proper management of natural resources, public-private partnerships investment in economic infrastructure and public services will be taken into consideration. Furthermore, there are other ADP initiatives that contribute to good governance in the post-eradication communities, by improving management and quality of public services in order to improve population wellbeing through better access to education and health services. The retrospective impact evaluation of the ADP in Huanuco, San Martin and Ucayali will try to measure the magnitude of the effect attributed to the program in terms of increasing household income. The outcome indicators to be evaluated are shown in Table 2. Table 2. Outcome indicators of Alternative Development Program Indicator Final indicator value Criteria for assigning the value Source and measurement technique Total income of households Nuevos Soles Income reported by interviewed DEVIDA impact survey, sections D, E, F, G, H, I. years 2007-2017 Income from agricultural activities Nuevos Soles Income reported by interviewed DEVIDA impact survey, sections D, E, F, G, H, I. years 2007-2017 Other income of household head Nuevos Soles Income reported by interviewed DEVIDA impact survey, sections D, E, F, G, H, I. years 2007-2017 Income of other household member(s) Nuevos Soles Income reported by interviewed DEVIDA impact survey, sections D, E, F, G, H, I. years 2007-2017 Design of the retrospective impact evaluation The magnitude of the effect attributed to ADP on household income will be measured after controlling confounding variables employing the Propensity Score Matching technique using the data obtained from the DEVIDA survey in Huanuco, San Martin and Ucayali (2007-2012). The DEVIDA survey aims to assess the effect of the intervention of the ADP in the areas of intervention, gathering information on various aspects relating to living conditions, production levels, incomes, and changes in attitude in favor of legal crops. Respondents are heads of households ordinarily resident in the areas of intervention of the ADP and in areas that are not part of the program. ADP communities are considered those which signed the agreement for the gradual and concerted reduction of coca cultivation (voluntary eradication), and the towns that are under the post- eradication strategy. 46 This survey excludes households in communities considered unsafe (areas with the presence of drugs or terrorism) that do not have the necessary guarantees for the evaluation. Also excluded were those communities not dependent on agricultural activity, as is the case in some provincial and district capitals. The area covered by the ADP consists of nine areas that in some cases coincide with the so-called coca basins, located in the regions of San Martin, Huanuco and Ucayali. The sampling frame used for the survey is the census and is updated by DEVIDA. Table 3 shows the Sampling Frame for the 2011 DEVIDA Impact Survey. Table 3. Geographic Areas of the sampling frame DEVIDA Impact Survey of 2011, according to program participation. Communities ADP Non-ADP Total Cluster Households Cluster Households Cluster Households Huallaga Central 158 12,131 161 9,559 319 21,690 Alto y Bajo Mayo 68 4,929 379 29,314 447 34,243 Juanjui 66 5,537 20 1,122 86 6,659 Bajo Huallaga 105 7,041 14 880 119 7,921 Tocache 134 8,221 24 1,315 158 9,536 Leoncio Prado 72 3,241 115 4,962 187 8,203 Aguaytia 122 5,106 4 88 126 5,194 Campo Verde-Nueva Requena 43 1,338 39 1,255 82 2,593 Pachitea 74 2,881 106 3,164 180 6,045 Total 842 50,425 862 51,659 1,704 102,084 The sampling unit is the household head habitually resident in the ADP areas. The Primary Sampling Units (PSUs) are called clusters. The survey was designed using a systematic, two-stage sampling process, proportional to the size of the community. Sampling is representative of each domain. The sample was designed to provide estimates for each domain (ADP and non-ADP), and area (Huallaga Central, Upper and Lower Mayo, Juanjui, Bajo Huallaga, Tocache, Leoncio Prado, Aguaytía, Campo Verde, Nueva Requena and Pachitea). The retrospective impact evaluation involves selecting ADP and non-ADP households for comparison using the ‘Propensity Score Matching’ Technique27. This technique creates a comparison group that is based on a model of probability of participating in the program, employing observed characteristics. The matching is done based on this probability (propensity score) with the non-participants. The contextual variables recommended for calculating the propensity are: sex, age, resident in the community, marital status, educational level, literacy, born locally, total area of land, raising animals, tree-felling, approval or disapproval of coca leaders and current coca cultivation. This technique allows the measurement of impacts on outcome indicators attributable to the program after checking by matching the possible confounding variables chosen from the contextual variables. The difference attributed to the program following a control on the confounding variables with the matching will be reported each year from 2007 to 2012. However, there are certain characteristics that may not easily be observed or are not collected daily in the DEVIDA impact survey and that cannot be included in the calculation of the propensity score. These variables may be correlated with the program variable and with the dependent variable and may be a 27 Khander S, Koolwal G, Samad H. (2010). Handbook on Impact Evaluation. Washington DC: Banco Mundial. P:53-69 47 source of bias through endogeneity. In addition, there are communities in Huanuco, San Martin and Ucayali not participating in the program concurrently, and which have different times of exposure to the program. To reduce these effects, the PSM will be combined with the Difference-in-Differences (DD) technique28. With this design, bias through endogeneity will be reduced because it is assumed that the heterogeneity of the endogenous variables not observed will be fixed over time. With data on participants and control observations before and after program intervention, a Difference-in-Differences matching estimator can be constructed. To present the DD estimator, see the setup for the cross-section PSM estimator shown in equation 1. (1) With panel data over two time periods t = {1,2}, the local linear DD estimator for the mean difference in outcomes Yit across participants I and nonparticipants j in the common support is given by (2) With only cross-sections over time rather than panel data, TOT can be expressed as Here, and , t = {1,2} are the outcomes for different participant and non-participant observations in each time period t. The DD approach combines traditional PSM and DD approaches. Observed as well as unobserved characteristics affecting participation can thus be accounted for if unobserved factors affecting participation are assumed to be constant over time. Taking the difference in outcomes over time should also difference out time-invariant unobserved characteristics and thus potential unobserved selection bias. One can also use a regression-adjusted estimator. This method assumes using a standard linear model for outcomes and for estimating the TOT (such as ) and applying weights on the basis of the propensity score to the matched comparison group. It can also allow one to control for selection on unobserved characteristics, assuming these characteristics do not vary over time. Applying PSM could help match treatment units with observationally similar control units before estimating the DD impact. Specifically, one would run PSM on the base year and then conduct a DD on the units that remain in the common support. However, during initial data collection, careful attention should be given to characteristics that determine participation. 28 Khander S, Koolwal G, Samad H. (2010). Handbook on Impact Evaluation. Washington DC: Banco Mundial. P:61 48 Limitations of retrospective impact evaluation Measures of changes in income attributable to the programs will be made with the information available in DEVIDA´s survey, which was not designed for an impact assessment, then it will be limited to have the best counterfactual. The PSM technique only reduces biases from observed variables in the survey, however unobserved variables continue to have a potential confounder and endogenous effect. Initial analysis of the DEVIDA survey of 2012 detected that there are few outcome indicators measured in the survey, and lack of social variables, such as access to services and community features (these are useful for obtaining a best PSM). At the same time was detected limitations in measuring the variable "ADP family." Impact evaluation design for the ADP in Huanuco (2014-2017) Given that the ADP have just started in Monzon, Huanuco, the consultants will propose the implementation of a baseline study, including communities and households that have just been incorporated in the ADP or will be in the next few years, and others that share similar characteristics but will not be considered for the ADP yet. In terms of the diagram presented below, this design will allow us to observe the evolution of some indicators for ‘Communities1’, before and after participating in the ADP (B-A), and compare this difference with the one observed in initially similar ‘Communities 2’ which did not participate in the ADP (D-C). In this regard, the effect attributable to ADP can be calculated as (B-A)-(D-C). In order to implement this design the consultants must have an idea in advance of the intervention plan in Huanuco ADP communities, so that they can select a sample of communities that will progressively enter the NAP in the next few years. But even if that plan is not available, and because there is a voluntary participation of communities and households, it is still possible to initially select enough communities with certain characteristics that make them most likely to be part of the ADP in the following years. The analysis of the sample frame for the DEVIDA surveys 2007-2012, as well as the information from the surveys and other documents and interviews with ADP counterparts will help for this purpose. It is important to note that even when a community is selected as part of the ADP, it does not mean all households will participate in the program. This implies that there is another selection process within communities that we must take into account for the evaluation as the main indicators of the program impact will be at the household level. A:Communities1 B:Communities1 Period 1 Period 2 ADP C: Communities2 D:Communities2 49 In 2014, a baseline survey will be conducted with an expanded sample of households in Huanuco. The sample design is the same as that used in the DEVIDA survey. A random sample of households and clusters in ADP and non-ADP areas will be conducted. The same sample size and sample design will be applied for the years 2015, 2016 and 2017. The DD method combined with PSM would be applied to analyze the impact of the ADP on household income and other outcome indicators. The impact of ADP on outcome indicators will be assessed by comparing the years 2014-2015, 2014-2016 and 2014-2017. For each period of analysis, the difference of outcome indicators between the baseline (2014) and the surveys of 2015, 2016 and 2017 will be calculated. The impact would be measured with a model for fixed-effects regression for the difference in outcome between the baseline and post-program survey. In this case, the difference of all explanatory variables between the baseline and after the program will be calculated, the PSM score will be calculated from the data of both surveys together, and this score will be included in the regression model of fixed effects for DD. Data Collection and Instruments The consultants will propose adjusting the DEVIDA impact survey implemented each year, in order to carry on the evaluation design presented before. The questionnaire includes many of the required indicators for the impact evaluation but it lacks some important ones such as the accumulation of household and productive assets, capitalization, and indicators of the quality of life and access to services. All these indicators and others that may appear necessary after a better understanding of mechanisms by which NAP could impact on households, will be proposed for inclusion in the survey for 2014 in Huanuco. Still, the main questionnaire will maintain its basic structure in order to be able to compare indicators with previous years. The design will also include qualitative information collected though other tools, to provide in-depth explanations of the findings. Qualitative information can also capture the synergies between AD strategies and governance improvement and access to public services (health and education). Qualitative data life stories of families that are recently moving from illicit to legal economies and that can be followed up during the 4 years of the ADP in Huanuco can illustrate the decision-making process, the challenges and opportunities and the benefits received, not always translated into income indicators collected by regular surveys. Existing performance information Weidemann Associates Inc. conducted two assessments on Peru USAID’s Alternative Development Program in 2006 and 2010. In 2006, they assessed the relevance and effectiveness of ADP, and in 2010 they identified the lessons learned and made recommendations for the program. An annual survey is conducted by DEVIDA where indicators are measured and compared in ADP and non-ADP areas. Since 2007, this survey has had the same sample design and data collection instrument. This survey is the responsibility of the Promotion and Monitoring Directorate of the National Commission for Development and Life without Drugs - DEVIDA. DEVIDA has reported rapid assessment on the performance of the alternative development programs in certain areas of Huanuco. In addition, each AD project has quarterly and annual progress reports on activities in Huánuco. 50 Deliverables, Timeline and Budget The evaluation team shall present the following deliverables, associated with payments advances. Contents and schedule of deliverables Deliverables Contents Due date 1 Report on descriptive analysis about ADP interventions in San Martin, Huanuco and Ucayali regions during 2007-2012, based in DEVIDA impact surveys, and possible methods for evaluation of their impact on households. 3rd week after start 2 Report on retrospective impact evaluation results for ADP interventions in San Martin, Ucayali and Huanuco during the period 2007-2012 6th week 3 Report on final proposal for an impact evaluation design of ADP in Huanuco (2014-2017) in SOW format 9th week 4 Report final approved by USAID 11th week The timeline for this study is presented below: Timeline for the impact evaluation design Tasks Weeks 1 2 3 4 5 6 7 8 9 10 11 1 Analysis of DEVIDA Impact surveys for the period 2007-2012 Describe possible evaluation methods Review of documents and interviews with DEVIDA and other agents implementing ADP Evaluate possible identification strategy for ADP 2 Initial comparisons of ADP and non-ADP communities and households 2007-2012 Explorative Impact Evaluation results for ADP interventions in San Martin, Ucayali and Huanuco during the period 2007-2012 3 Design of the impact evaluation of ADP in Huanuco Adjust DEVIDA questionnaire Propose sample design for impact evaluation Propose methodological approach for impact evaluation Draft Statement of Work Final Statement of Work reviewed by USAID 51 Evaluation Team Project staff will work with the subcontractor Grupo de Análisis para el Desarrollo (GRADE) to analyze and discuss the results of the impact evaluation design. After the approval of the SOW, the evaluation team will be led by GRADE and the USAID/EVALUATIONS staff will guide and review the reports. GRADE will designate a senior researcher responsible for the project who will report on progress in the activities and results of the evaluation. The preparation of the report will be led by the senior researcher, who will also be responsible for the day to day advance of the study, as well as coordination of different meetings and interviews with DEVIDA and USAID staff. The team will include another experienced Researcher who will participate in the conceptualization of the evaluation design, give feedback on the analysis of data from the DEVIDA impact surveys (2007-2012) as well as on the explorative impact evaluation to be pursue, an collaborate on the writing of the final report. The team will be completed with a Research Assistant in charge of organizing all the information recover from DEVIDA (sample frame, impact surveys, information on interventions, etc.), and pursuing the data analysis and econometric estimations. Reporting and Dissemination The final report on the impact evaluation design for ADP in Huanuco will be presented to USAID staff, DEVIDA and the ADP projects. This report will include the analysis of the explorative impact evaluation for ADP projects in Huanuco and San Martin using the DEVIDA impact surveys since 2007. The report will be composed of:general information, including the title page, executive summary and acronyms list; body, with an introduction describing the purpose of the study; a chapter on the analysis of information and explorative impact evaluation of ADP in Huanuco and San Martin since 2007; a chapter containing the proposal for an impact evaluation design for ADP in Huanuco (2014-2017); and recommendations; and the annexes, which must include the SOW, STATA Do files for the analysis of DEVIDA Impact surveys, and a list of documents reviewed. The report must also include the data base files with complete technical description and STATA Do file. Before issuing the final report, the evaluation team will present the main findings and conclusions to USAID/Peru staff and implementing partners. Additional presentations to national or regional authorities shall be planned as required by either USAID /Peru or the implementing partner. U.S. Agency for International Development Av. La Encalada s/n, Santiago de Surco Lima, Perú