1 Does Assistance to Farmers Translate into Community Welfare Improvements? Non-Experimental Program Evaluation of USAID Assistance to Smallholder Farmers in Guatemala FINAL REPORT This report was produced for review by the United States Agency for International Development (USAID). It was prepared by Optimal Solutions Group, LLC, for USAID’s “Learning, Evaluation, and Analysis Project” (LEAP). Contract Number: AID-OAA-C-11-00169 2 United States Agency for International Development Learning, Evaluation, and Analysis Project (AID-OAA-C-11-00169) A Non-Experimental Program Evaluation of USAID Assistance to Smallholder Farmers in Guatemala FINAL REPORT Prepared for United States Agency for International Development Prepared by Optimal Solutions Group, LLC University of Maryland Research Park, M Square 5825 University Research Court, Suite 2800 College Park, MD 20740, USA www.optimalsolutionsgroup.com Evaluation Team Ms. Christabel Dadzie, Program Manager/ Evaluation Analyst Dr. Claudia Gonzalez Martinez, Senior Evaluation Specialist Dr. Peter Murrell, Senior Economist August 18, 2014 Disclaimer The views expressed in this publication are those of the authors and do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 Table of Contents ACKNOWLEDGMENTS .......................................................................................................................................... 8 ACRONYMS............................................................................................................................................................... 9 EXECUTIVE SUMMARY ...................................................................................................................................... 10 CONTEXT OF THIS EVALUATION................................................................................................................... 21 METHODOLOGY................................................................................................................................................... 22 DATA SELECTION AND MANAGEMENT ................................................................................................................................23 2006 and 2011 ENCOVIs....................................................................................................................................................23 2002 Population Census.....................................................................................................................................................27 FINDINGS: DIFFERENCES-IN-DIFFERENCES (DID) REGRESSION ANALYSES ...........................................................43 Per-capita income ................................................................................................................................................................. 45 Household income ................................................................................................................................................................. 49 Literacy of the Head of Household................................................................................................................................. 51 Percentage of School-Age (5-17) Females Attending School............................................................................53 Awareness of Birth Control Methods............................................................................................................................55 Children Aged 5 or Younger Receiving the Pentavalent Vaccine....................................................................57 Connection to an Electricity Distribution Grid ........................................................................................................59 Rate of Ownership of Living Quarters..........................................................................................................................61 CONCLUSIONS AND RECOMMENDATIONS ................................................................................................. 63 APPENDIX A - REFERENCES............................................................................................................................. 74 APPENDIX B – STATEMENTS OF WORK...................................................................................................... 75 APPENDIX C - USAID AGRICULTURAL INTERVENTIONS OVERVIEW................................................ 85 SUMMARY: USAID AGRICULTURAL INTERVENTIONS PRIOR TO 2010.......................................................................86 AGEXPORT...........................................................................................................................................................................86 ANACAFE “COMPETITIVE ENTERPRISES”.......................................................................................................................88 MERCY CORPS – WALMART “INNOVATION MARKET ALLIANCE FOR RURAL ENTREPRENEURS”(IMARE) PROJECT ..................................................................................................................................................................................91 FUNDACIÓN AGIL “QUALITY ASSURANCE AND SMALL BUSINESS DEVELOPMENT”...................................................92 APPENDIX D – LOCATION OF USAID AGRICULTURAL INTERVENTIONS ......................................... 93 PROGRAM IMPLEMENTATION AT THE DEPARTMENT LEVEL.........................................................................................94 PROGRAM IMPLEMENTATION AT THE MUNICIPALITY LEVEL........................................................................................95 APPENDIX E – OVERVIEW OF DATASETS ................................................................................................... 96 THE XI CENSO NACIONAL DE POBLACIÓN Y VI DE HABITACIÓN. .................................................................................96 HOUSEHOLD CENSUS (AT THE MUNICIPAL LEVEL)-“MI FAMILIA PROGRESA (MIFAPRO)”................................96 THE ENCUESTA NACIONAL DE CONDICIONES DE VIDA (ENCOVI), .............................................................................97 APPENDIX F – TREATMENT AND CONTROL MATCHED ANALYSIS SAMPLE................................... 98 APPENDIX G – DID REGRESSION RESULTS...............................................................................................101 4 APPENDIX H – CRITERIA FOR AN IDEAL IMPACT EVALUATION......................................................102 APPENDIX I – CHOICE OF METHODOLOGY OF THE CONSTRUCTION OF THE DATASET TO BE USED IN THE EVALUATION ANALYSIS.......................................................................................................104 5 Figures Figure 1. Treatment and matched municipalities household income, 2006 and 2011 ............ 31 Figure 2. Educational achievement in intervention municipalities, 2002, 2006 & 2011......... 35 Figure 3. Educational achievement in non-intervention municipalities, 2002, 2006 & 2011 36 Figure 4. Educational achievement in Guatemala, 2002, 2006 & 2011 .......................................... 37 Figure 5. Per-capita income in Guatemala, 2006–2011........................................................................ 45 Figure 6. Household income in Guatemala, 2006–2011....................................................................... 49 Figure 7. Literacy of the head of household, 2006–2011..................................................................... 51 Figure 8. Percentage of school-age females (5–17) attending school, 2006–2011..................... 53 Figure 9. Percentage of reproductive-age females (12–49) who have heard of birth control methods, 2006–2011......................................................................................................................................... 55 Figure 10. Percentage of children aged 5 and younger who have received the pentavalent vaccine, 2006–2011........................................................................................................................................... 57 Figure 11. Living quarters connected to an electricity distribution grid, 2006–2011.............. 59 Figure 12. Rate of ownership of living quarters, 2006–2011............................................................. 61 6 Tables TABLE 1. AGRICULTURAL PROJECTS CONSIDERED IN THE EVALUATION..................................... 23 TABLE 2. INTERVENTION-ASSISTED MUNICIPALITIES SURVEYED IN ENCOVIS’ DATASETS ... 24 TABLE 3. INTERVENTION MUNICIPALITIES.............................................................................................. 26 TABLE 4. SUMMARY CHARACTERISTICS: INCOME AND COUNT OF HOUSEHOLDS...................... 33 TABLE 5. SUMMARY CHARACTERISTICS AND EDUCATIONAL ACHIEVEMENT: HEAD OF HOUSEHOLD ......................................................................................................................................................... 34 TABLE 6. SUMMARY CHARACTERISTICS: HEALTH AND HEALTH INSURANCE OF HEAD OF HOUSEHOLD ......................................................................................................................................................... 39 TABLE 7. SUMMARY CHARACTERISTICS: CHILDREN AGES 5 YEARS AND YOUNGER ................. 40 TABLE 8. SUMMARY CHARACTERISTICS: FEMALE HEALTH AND CHILDREN OF SCHOOL AGE41 TABLE 9. SUMMARY CHARACTERISTICS: HOME CHARACTERISTICS............................................... 42 TABLE 10. DID ANALYSIS OF THE LOG OF PER-CAPITA INCOME....................................................... 48 TABLE 11. DID ANALYSIS OF THE LOG OF HOUSEHOLD INCOME...................................................... 50 TABLE 12. DID ANALYSIS OF THE LITERACY INDICATOR FOR THE HEAD OF HOUSEHOLD.... 52 TABLE 13. DID ANALYSIS OF THE PERCENTAGE OF SCHOOL-AGE FEMALES ACTUALLY ATTENDING SCHOOL ......................................................................................................................................... 54 TABLE 14. DID ANALYSIS OF PERCENTAGE OF REPRODUCTIVE-AGE FEMALES WHO HAVE HEARD OF BIRTH CONTROL METHODS...................................................................................................... 56 TABLE 15. DID ANALYSIS OF THE PERCENTAGE OF CHILDREN AGED 5 AND YOUNGER WHO HAVE RECEIVED THE PENTAVALENT VACCINE ....................................................................................... 58 TABLE 16. DID ANALYSIS OF THE HOME’S CONNECTION TO AN ELECTRICITY DISTRIBUTION GRID......................................................................................................................................................................... 60 TABLE 17. DID ANALYSIS OF OWNERSHIP OF THE LIVING QUARTERS/HOME............................ 62 TABLE 18. SUMMARY OF RESULTS FOR INTERVENTIONS (OF ALL TYPES) FOR ALL OUTCOMES ................................................................................................................................................................................... 66 TABLE 19. SUMMARY OF RESULTS FOR THE INTERVENTIONS BY AGEXPORT FOR ALL OUTCOMES ............................................................................................................................................................ 67 TABLE 20. SUMMARY OF RESULTS FOR THE INTERVENTIONS BY FUNDACIÓN AGIL FOR ALL OUTCOMES ............................................................................................................................................................ 68 TABLE 21. SUMMARY OF RESULTS FOR THE INTERVENTIONS BY ANACAFE FOR ALL OUTCOMES ............................................................................................................................................................ 69 7 TABLE 22. SUMMARY OF RESULTS FOR THE INTERVENTIONS BY MERCY CORPS FOR ALL OUTCOMES ............................................................................................................................................................ 70 TABLE 23. SUMMARY OF RESULTS FOR RELEVANT INTERVENTIONS WHEN ONLY ONE INTERVENTION TOOK PLACE, FOR ALL OUTCOMES .............................................................................. 71 TABLE 24. SUMMARY OF RESULTS RELEVANT WHEN TWO INTERVENTIONS TOOK PLACE, FOR ALL OUTCOMES .......................................................................................................................................... 72 TABLE 25. SUMMARY OF RESULTS RELEVANT WHEN THREE OR MORE INTERVENTIONS TOOK PLACE, FOR ALL OUTCOMES............................................................................................................... 73 8 ACKNOWLEDGMENTS The LEAP evaluation team is grateful for the guidance provided by USAID/LAC, in particular, Alice Brooks, economist, USAID LAC/RSD, who has provided our team guidance and support throughout this evaluation. Many thanks are also extended to the Economic Growth and Program Office team at the USAID/ Guatemala Mission who provided us with project documentation and responded to our many questions during the site visit. We would also like to thank the Guatemala national statistical institute, Instituto Nacional de Estadistica (INE), for providing us with the raw datasets needed for this evaluation. Finally, the evaluation team is profoundly grateful to Dr. Peter Murrell, Economist, University of Maryland and Senior Research Fellow at Optimal, who spent time reviewing and developing an updated version of this report that was better suited for policy review. We are also grateful to Dr. Mark Turner, LEAP Chief of Party and CEO of Optimal, for his senior review throughout this project. 9 ACRONYMS ANACAFE Asociación Nacional del Café (National Coffee Association) AGEXPORT Asociación Guatemalteca de Exportadores (Guatemalan Association of Exporters) Fundación AGIL Fundación de Apoyo a la Generación de Ingresos Locales (Foundation to Support Local Income Generation) DID Difference-in-differences ENCOVIs Encuestas de Condiciones de Vida (Surveys of Living Conditions) IMARE Innovation Market Alliance for Rural Entrepreneurs INE Instituto Nacional de Estadistica (National Institute of Statistics) LAC/RSD Latin America and the Caribbean/Regional Sustainable Development Bureau LEAP Learning, Evaluation, and Analysis Project LSMS Living standard measurement study SAE Small Area Estimation USAID U.S. Agency for International Development EXECUTIVE SUMMARY The United States Agency for International Development (USAID) Latin America and the Caribbean/Regional Sustainable Development (LAC/RSD) Bureau, contracted with Optimal Solutions Group, LLC (Optimal), through its Learning, Evaluation, and Analysis Project (LEAP) contract, to conduct a retrospective evaluation of the impact of USAID’s agricultural value chain programs initiated in Guatemala prior to 2010. Under this contract, Optimal conducted an evaluation feasibility study and subsequent evaluation study. This report details the outcome of the Guatemala Evaluation Study, which was undertaken following the determination that a non￾experimental impact evaluation was feasible based on information gathered during a site visit to Guatemala to assess program goals, objectives, implementation, and performance data and to collect available secondary datasets. This evaluation focuses on whether there is evidence that the Guatemalan agricultural value chain programs improved the welfare of individuals within the municipalities affected by the programs. It is therefore an impact evaluation,1 asking whether observed impacts were caused specifically by these programs by using counterfactual comparisons. This is a non-experimental (or non￾randomized-control) evaluation, because the counterfactuals were not generated by random assignment during program initiation but instead were constructed ex-post using statistical methods applied to available data sources. The difference between this evaluation and a performance evaluation should be emphasized, because a performance evaluation has already been completed for some of the programs that are the subject of this current evaluation.2 A performance evaluation usually asks how a program is being implemented, whether it is accompanied by positive welfare outcomes for beneficiaries, and how the program is perceived and valued. Those readers who find contrasting overall results from these existing evaluations and the one documented here should remember that impact and performance evaluations ask very different questions and use very different methodologies. It is a challenge to find empirical program data to assess the total welfare impact of agricultural programs, in large part because USAID guidance did not request that missions collect and report the most relevant types of information prior to 2010, such as household income changes for USAID programs. The most closely related standard indicators used at that time were the value of international and intra-regional exports of targeted commodities as a result of U.S. government assistance. Neither total household income nor farmer income from all sources—on farm and off farm—was tracked for the Guatemala agricultural activities. Even if such data had been available, comparable information for a control group of farmers was not collected to enable an impact evaluation on the income levels of individual farming households. 1 Impact evaluation is the systematic identification of the effects (positive or negative, intended or unintended) on individual households, institutions, and the environment caused by a given development activity, such as a program or project. (2004). In OED and Impact Evaluation―A Discussion Note, World Bank. http://www.worldbank.org/oed/docs/world_bank_oed_impact_evaluations.pdf, downloaded November 2011. 2 The two programs that had a performance evaluation were ANACAFE and AGEXPORT. See USAID, "USAID/Guatemala Final Performance Evaluations for Four Economic Growth Office Projects," http://pdf.usaid.gov/pdf_docs/pdacu509.pdf. 10 11 The importance of non-farm income for Latin American farmer households is well documented,3 and a study on welfare changes would be incomplete without data on all income sources. In the search to find a reasonable measure of total income changes though alternative data sources, the evaluation team turned to the Guatemalan government's living standards measurement study (LSMS) surveys, the Encuestas de Condiciones de Vida (ENCOVI's—Surveys of Living Conditions). While the ENCOVI datasets bring a richness of household welfare data to the analysis, it is important to keep in mind that they do not offer the ability to track the specific individuals who were the direct beneficiaries of the USAID programs. Therefore, this evaluation used the welfare of the households in the municipalities in which the USAID programs were implemented as a proxy for the welfare of the direct beneficiaries themselves. Consequently, the findings of this evaluation are best viewed as testing whether assisting producer groups by increasing their incomes through agricultural improvements has a measurable effect on income of their communities as a whole. It is possible that individual beneficiaries received significant welfare benefits, but these benefits remained local and did not multiply throughout the municipality. It is also possible that the benefits obtained did not endure and could not be measured at program closure. For these and other possible reasons, we may see a disconnect between individual vs municipal level welfare impacts. Due to these analytical constraints and data limitations, this evaluation does not claim to precisely estimate the effects on smallholder farmers’ livelihoods following participation in USAID agricultural value chain programs. Instead, this evaluation compares the evolution of the municipalities where the assisted groups resided with trends in those municipalities where no programs were implemented. Because the assisted municipalities were naturally not a random sample of those in the whole country—they were poorer and more rural, for example—the evaluation used an additional methodology that offered the possibility of obtaining a more precise estimate of the effects of the programs, but only in one subset of municipalities. Using six criteria, this methodology identified a subset of municipalities where assisted groups resided that could be matched with a comparable set of non-assisted municipalities. Then the analysis compared the welfare status of the two matched groups before and after program implementation. The two matched subsets of municipalities happened to be the poorest municipalities affected by the programs. Hence, this evaluation comprises two sets of comparisons of two groups of municipalities, thus involving four sets of municipalities. For brevity and clarity, the rest of this document uses the following terms as labels for these four sets: • “Intervention municipalities” refers to municipalities where producer groups were assisted with USAID agriculture activities. • “Non-intervention municipalities” refers to municipalities nationwide where producer groups were not assisted. 3 “The agricultural labor market and the rural nonfarm economy often account for 70 percent of rural incomes and employ 55 percent of the active rural labor force (based on representative household surveys that cover 85 percent of the Latin American population).” World Bank “The Agenda for Latin America and the Caribbean” http://go.worldbank.org/PUW8DFUUH0 12 • “Treatment municipalities” refers to the subset of intervention municipalities that were matched to a subset of non-intervention municipalities. • “Control municipalities” refers to the subset of non-intervention municipalities that were matched to the treatment municipalities. Key findings The key findings of this evaluation are as follows: 1. On measures of real per-capita income (or real household income), the treatment group of municipalities fared worse than the matched control group of municipalities, a result that is of some significance, because the treatment and control groups were some of poorest municipalities in Guatemala. On these same measures, there was no significant difference in trends between intervention and non-intervention municipalities. As noted previously, this should not necessarily be interpreted as a lack of impact of the interventions on individual producers. Rather, this might be interpreted as changes in the welfare of producers not being reflected in broader municipal level welfare. 2. In contrast, the treatment municipalities fared better than the control municipalities on both connections to an electricity grid and rate of ownership of living quarters. However, on these measures, there was no appreciable difference in trends between intervention and non-intervention municipalities. 3. The contrast between the results on income and those on electricity and home ownership indicate that programs can bring significant welfare benefits beyond income changes. 4. The contrast between the results on income and those on electricity and home ownership indicate some lack of correspondence between measured changes in real income obtained from survey data and measured changes in other indicators of well-being from the same surveys. For new and ongoing programs, expanding the list of indicators tracked for program beneficiaries would help better inform total welfare changes. The appropriate indicators would be country-specific, reflecting measures of household consumption of items (e.g., durables) that vary significantly across the country's households. 5. On average, children ages 5 years or younger in the intervention municipalities had lower vaccination rates than those in non-intervention areas at the beginning of these USAID programs. Although vaccination rates significantly increased in the country as a whole, the intervention areas were able to close the gap with the rest of Guatemala over the period of the interventions. 6. Program effects can vary significantly with the initial situation of beneficiaries, as indicated by differences in the results for the whole set of intervention municipalities and the results for the treatment group of municipalities. 13 Methodology This non-experimental impact evaluation assesses four USAID/Guatemala agricultural interventions that carried out programs on capacity building, technical assistance, financial incentives, and training. The four interventions all had the same general goal of improving smallholders’ farming practices and providing access to national, regional, and international markets to increase employment at the local level and to make the country more competitive on the international market. The selected programs were implemented by the Asociación Guatemalteca de Exportadores (Guatemalan Association of Exporters; AGEXPORT), Asociación Nacional del Café (National Coffee Association; ANACAFE), Fundación de Apoyo a la Generación de Ingresos Locales (Foundation to Support Local Income Generation; Fundación AGIL), and Mercy Corps. Because this evaluation began after the start of implementation of the projects being evaluated, the evaluators were not able to influence any decisions made by the implementers, or even gather precise information on how these decisions were made. These decisions—in particular the choice of localities in which the interventions occurred—affect the precision of the evaluation in ways that the evaluators are not able to assess. For example, implementers might have chosen the areas of Guatemala where success would naturally be most difficult, because these areas were the most in need. If this had been the case, then estimates of the program's effects would cast a much more pessimistic picture than would actually be warranted. But the information necessary to assess whether this was the case was not collected and therefore not available for this evaluation. Based on input from the Guatemala Instituto Nacional de Estadistica (National Institute of Statistics; INE) during Optimal’s initial site visit, the 2006 and 2011 ENCOVIs were selected as baseline and end line datasets, respectively. The ENCOVIs provide detailed information on living conditions, income, and consumption patterns. The evaluation team identified the municipal locations of assisted beneficiary groups (the intervention municipalities) from USAID/Guatemala mission files. The ENCOVI data-collection methodology draws random samples of households nationwide, using sampling methodologies developed by the Guatemalan government with assistance from the World Bank. Not all municipalities are sampled in each survey, meaning some USAID-assisted municipalities are not represented in any given survey. Only those municipalities for which data were available in both the 2006 and the 2011 ENCOVIs were used in the analysis. 14 Number of municipalities subject to program interventions 80 Number of municipalities subject to program interventions with data available from both 2006 and 2011 ENCOVI's 71 Number of municipalities subject to program interventions with data not available from ENCOVI's 9 Number of municipalities not subject to program interventions 251 Number of municipalities not subject to program interventions with data available from both 2006 and 2011 ENCOVI's 206 Number of municipalities not subject to program interventions with data not available from ENCOVI's 45 Total municipalities in Guatemala (2006) 331 In the case of those municipalities that received USAID assistance, this decision meant discarding 9 of the 80 municipalities receiving assistance, and in the case of those not receiving USAID assistance, 45 of the 251 municipalities were dropped. This selection itself might have led to some non-generalizability of the results of the analysis, as the municipalities that were not sampled in both years may not have been randomly omitted. The municipalities receiving assistance were not a random sample of Guatemalan municipalities; they were purposefully selected by USAID based on need and other factors. Intervention municipalities had mean per-capita incomes in 2006 at 60 percent of the mean level for the whole of Guatemala. In the 2006 ENCOVI, 60 percent of respondents in intervention municipalities were classified as living in rural parts of the municipalities, while 46 percent were so classified for non-intervention municipalities. The table below presents summary statistics highlighting the differences between the households in intervention and non-intervention municipalities that existed in 2006, the baseline year for this evaluation. 15 Indicators of household welfare in 2006, by intervention status Percentage of rural households in sample Mean number of household members Per-capita mean monthly income in 2006 currency units Literacy level of head of household (in percent) Households in intervention municipalities 60.5 5.4 764 65.2 Households in non￾intervention municipalities 41.8 4.7 1,388 75.2 Households in all Guatemalan municipalities 46.2 4.9 1,228 72.9 This places limits on the generalizability of the results of the evaluation to interventions in Guatemala as a whole, for two reasons. First, the interventions were in those places where previous development progress has been slower—perhaps those places where success is intrinsically more difficult, leading to more pessimism from an evaluation than would be warranted by events on the ground. Second, events between 2006 and 2011 might have affected intervention and non￾intervention municipalities in different ways (because these municipalities are inherently different), leading to bias in estimates of the effect of the interventions. A standard approach to counter such problems is to include in the analyses variables that capture differences between households in the intervention and non-intervention municipalities. Thus, this analysis includes the number of persons per household, the age of the head of household, whether a household has a female head, the number of children ages 5 to 17, the number of children younger than 5 years old, and the number of females of reproductive age. To the extent that these variables capture the main differences between households in intervention and non-intervention municipalities, the results comparing intervention and non-intervention municipalities provide valid estimates of the effect of interventions in those municipalities included in the two ENCOVIs. An additional approach to counter these problems, which can be used in combination with the first, is to implement a matching methodology, which constructs a subset of intervention municipalities (the treatment group) that is as closely matched as possible along six dimensions (e.g., income of urban households, income of rural households) to a subset of non-intervention municipalities (the control group). Data from the 2006 ENCOVI were used for this purpose. The matching methodology resulted in treatment and control groups comprising 28 intervention municipalities and 25 non￾intervention municipalities, respectively. The municipalities for which a match was identified happened to be those with the lowest income level among all the municipalities (see table below). As a result, the selected treatment group is not representative of the entire group of intervention municipalities. This means that results from comparing the effect of the interventions in treatment and control groups reflect the effect of an intervention on a municipality that is typical of Guatemala's poorest municipalities. These results 16 are not immediately generalizable to the whole of Guatemala: they do not reflect the effect of an intervention on a randomly chosen Guatemalan municipality. Nevertheless, the matching methodology has the serendipitous benefit of providing a set of results that is directly relevant to the most important targets of development assistance—the poorest regions and households, measured in terms of household income. (But, somewhat paradoxically, as the table below shows, not the poorest in terms of other indicators, such as girls in school or connections to the electricity grid). Indicators of household welfare in 2006, all intervention municipalities versus treatment municipalities Percentage of girls attending school Percentage of municipalities connected to electricity grid Per-capita mean monthly income in 2006 currency units Literacy level of head of household (in percent) Households in the 71 intervention municipalities for which ENCOVI data are available 52.7 60.5 764 65.2 Households in the subset of 28 treatment municipalities 60.5 74.7 421 54.4 The evaluation team then used the household-level ENCOVI data to estimate the effects of interventions. This was accomplished by using a difference-in-differences (DID) approach applied to developments in eight indicators of household welfare. In this approach, the estimates capture whether the change in an indicator from 2006 to 2011 was different in non-intervention municipalities than in intervention municipalities. An analogous analysis was performed comparing developments of the same indicators in treatment and control municipalities. The reader should be cautious in interpreting the results. For the estimates to have a causal interpretation, the two central assumptions discussed above must hold. First, the analysis should have included variables sufficient to capture the main differences between households in the intervention and non-intervention municipalities. Second, for the comparison of treatment and control groups, it is necessary that the matching methodology produced treatment and control groups that were similar along relevant dimensions (apart from the fact that interventions had occurred in only one of these groups). It is inherent in the nature of such statistical analyses that these two assumptions cannot be precisely tested and therefore the reader's judgment comes into play. If these assumptions hold, then the estimates below can be interpreted as indicating causal effects of interventions. If the assumptions do not hold, then the estimates should be interpreted as highlighting trends in the development of municipalities that did and did not receive interventions. Whichever interpretation is placed on the results—causal or correlation—it is important to remember that estimates comparing intervention to non-intervention municipalities apply to Guatemala as a whole, while estimates comparing treatment to control municipalities apply to the poorest set of municipalities that was assisted. 17 Findings Eight household and family welfare indicators were selected for the DID analyses. It should be noted that although these variables reflect phenomena that interventions might have affected, these variables may not have been specifically targeted by the interventions. These indicators are (1) per￾capita income, (2) household income, (3) literacy of the head of household, (4) school attendance of females ages 5 to 17, (5) awareness of birth control methods among women of reproductive age (12–49), (6) children ages 5 or younger who have received the pentavalent vaccine, (7) connection of the living quarters to an electricity distribution grid, and (8) ownership of living quarters. The table below summarizes the results of this evaluation at the very highest level of generality, focusing on the overall effects of interventions (independent of which particular organization conducted the intervention). The paragraphs following the table provide further details. Outcome analyzed Results comparing intervention to non-intervention regions (i.e., results relevant to Guatemala as a whole) Results comparing treatment to control regions (i.e., results relevant to the poorest regions in Guatemala) Per-capita income Statistically weak positive effect Statistically strong negative effect Household income Statistically weak negative effect Literacy indicator for head of household Statistically weak positive effect Percentage of school-age females attending school Percentage of reproductive￾age females who have heard of birth control methods Statistically strong negative effect Percent of children 5 or younger who received pentavalent vaccine Statistically strong positive effect Electricity distribution grid connectivity Statistically strong positive effect Ownership of the living quarters/home Statistically strong positive effect Note: Empty cells indicate no conclusion warranted by the statistical analysis. It is important to note that the conclusions differ according to which samples are analyzed, whether comparing intervention versus non-intervention municipalities or treatment and control municipalities. These differences are almost certainly a reflection of the fact that the matched treatment and control groups are a select set of municipalities—they are among the poorest municipalities in all of Guatemala. Per-Capita Income The entire country of Guatemala experienced a large decrease in per-capita income (in real terms) from 2006 to 2011. Intervention municipalities were poorer than those in non-intervention areas. For all municipalities considered together, interventions had a statistically weak positive effect on per-capita income. However, in the analysis of matched treatment and control municipalities, households affected by interventions fared worse than those not affected in terms of changes from 18 2006 to 2011. These results are particularly characteristic of the areas affected by the interventions of AGEXPORT and ANACAFE. Mercy Corps interventions evidenced positive (but very weak) effects in both the all-municipality and the treatment-control analyses. Household Income Not surprisingly, the results are very similar to those for per-capita incomes, such that no additional comments are necessary. Literacy of Head of Household On average, heads of household in the intervention municipalities had lower literacy levels than those in non-intervention areas. Across the country overall, literacy levels in 2011 increased with respect to 2006, an improvement that is slightly larger for households in the treatment group of municipalities compared to the matched control group. This result does not apply to those affected by the Mercy Corps intervention, where literacy decreased significantly relative to those not affected by these interventions. School-Age (5–17) Females Attending School On average, girls in the intervention municipalities had lower school-attendance rates than those in non-intervention areas. In Guatemala overall, female school-attendance rates significantly increased in 2011 as compared to 2006, an increase that is larger for girls in intervention areas but has very weak statistical significance. The only significant deviation from this general characterization is a decrease (with very weak statistical significance) in the percentage of females attending school for those households in areas affected by Fundación AGIL interventions. Awareness of Birth Control Methods On average, females of reproductive age in the intervention municipalities had lower awareness of birth control methods than those in non-intervention areas. This of course is a sign that the selection of intervention areas was not random. Knowledge of birth control methods greatly improved between 2006 and 2011 for the entire country, with that improvement being of the same order for the intervention areas. Households located in ANACAFE areas show a larger improvement in knowledge of birth control methods than other areas, but with weak statistical significance. Children Ages 5 or Younger Receiving the Pentavalent Vaccine On average, children ages 5 years or younger in the intervention municipalities had considerably lower vaccination rates in 2006 than those in non-intervention areas. Notably, vaccination rates greatly increased in Guatemala as a whole from 2006 to 2011, going from 32 percent to 93 percent. Moreover, this improvement is found to be significantly greater for intervention areas, which closed the gap with the rest of Guatemala. These results regarding the pentavalent vaccine are some of the strongest to emerge from this evaluation, with few indicators to the contrary. This large improvement holds for interventions by all implementing organizations except Fundación AGIL. 19 Connection to an Electricity Distribution Grid On average, homes in the intervention municipalities have lower electricity connection rates than those in non-intervention areas. The results indicate virtually no effect for intervention municipalities as a whole but a statistically strong positive effect for those households in the matched samples of municipalities. Hence, the interventions seem to have helped the poorest households for this particular outcome measure. The areas that received Fundación AGIL’s intervention increased their likelihood of being connected to electricity relative to other areas. However, for Mercy Corps, the results are opposite with interventions appearing to lower the likelihood of a household’s being connected to the power grid, relative to the rest of Guatemala. Rate of Ownership of Living Quarters On average, homes in the intervention municipalities had higher ownership rates than those in non￾intervention areas, a result that is neither weakened nor strengthened by the interventions in 2011 when examining the all-municipality sample. In contrast, for the matched sample, evidence suggests that the interventions have had a statistically strong positive effect, in particular the interventions of AGEXPORT. The overall lack of evidence of negative effects coupled with strong evidence of positive effects in the matched sample is important, because the matched sample comprises some of the very poorest municipalities in Guatemala. Results for the Four Implementing Organizations Lastly, some conclusions can be reached concerning the overall effect of interventions by each of the four implementing organizations. The results point in a positive direction most strongly for Mercy Corps; for that organization, the results are positive for both the all-household sample and the matched-household sample, raising confidence in the validity of this conclusion. For AGEXPORT and ANACAFE, there is evidence of a positive effect, but it is quite statistically weak. For Fundación AGIL, the composite picture does not point in any specific direction that would warrant a conclusion. Conclusions It is difficult to draw one overall conclusion from the statistical analysis conducted for this evaluation, which is not surprising. The results of the interventions vary along three dimensions— by the two different samples analyzed (all-intervention analysis vs. matched municipalities), by the type of welfare indicator analyzed, and by the organization implementing the intervention. The four implementing organizations focused on different types of programs; the matched municipalities are not representative of the country as a whole; and the welfare indicators each capture very different aspects of household well-being. Thus, it is possible to pronounce the interventions successful in one area (e.g., increasing the percentage of children receiving the pentavalent vaccine in Guatemala as a whole) and not successful in another (e.g., increasing per-capita incomes for the poorest households). As these differences are general characteristics of the results detailed in the main body of this report, readers are urged to examine those individual results very carefully. Nevertheless, one can cautiously make a rough overall judgment. First, the overall evidence shows that the interventions of all four organizations combined—viewed in the composite—do not point to either a uniformly positive effect or a uniformly negative effect. Second, stronger conclusions 20 might be made when looking at the effect of interventions on specific indicators separately. There is an indication of negative effects of the interventions on incomes, especially for the poorest households (those in the matched samples). There is an indication of a positive effect on access to electricity and ownership of homes, which is also strongest in those poorest households. Third, the implementing organization Mercy Corps seems to have effected more positive results than the other implementing organizations. Fourth, positive effects seem to arise from isolated interventions rather than when two or more interventions are occurring simultaneously in the same municipality. The major limitation placed on the methodology of this study is that the evaluation was not built into the project design from the very beginning. It is ex-post and relies on existing general-purpose datasets rather than specialized information that focused specifically on the direct effects of the interventions and used data collected while the interventions proceeded. Evaluations are likely to be less equivocal than this one and more capable of producing conclusions that can help future project activities if the demands of precise evaluation are taken into account in project design, the very first step taken when initiating a project. 21 CONTEXT OF THIS EVALUATION The United States Agency for International Development (USAID) Latin America and the Caribbean/Regional Sustainable Development (LAC/RSD) Bureau is conducting a retrospective evaluation of the impact of USAID’s agricultural value chain programs, established prior to 2010, in both Honduras and Guatemala.4 This document focuses on Guatemala. Optimal Solutions Group, LLC (Optimal), was contracted to undertake the study, which began with a site visit to Guatemala to assess program goals, objectives, implementation, and performance data and to collect available secondary datasets to perform an Evaluation Feasibility Study.5 The Learning, Evaluation, and Analysis Project (LEAP) team included Alice Brooks (Economist, USAID LAC/RSD), Christabel Dadzie (LEAP Project Manager/Research Associate, Optimal), and Claudia Gonzalez Martinez (Evaluation Specialist/Senior Research Associate, Optimal). Prior to Optimal’s involvement, USAID/Guatemala, through its partners, implemented four projects under Strategic Objective 2 of USAID/Guatemala’s Country Plan for 2004–2009. These projects were intended to assist smallholder farmers in Guatemala, focusing on farming practices and access to markets. Interventions consisted of a variety of activities, including capacity building, technical assistance, financial incentives, and training of beneficiaries. All four interventions shared the same general goal, but each had different specific objectives. Given the timing of Optimal's involvement, its evaluation is of necessity retrospective—that is, it relies on existing data collected by others and applies standard statistical techniques to compare those individuals or businesses subject to the interventions (the "treated") with those individuals or businesses who are comparable but were not subject to the interventions (the "controls"). Although such retrospective evaluations are extremely common in social science and medical research, they have their deficiencies, which must be acknowledged when considering what one can learn from the results presented below. Their deficiencies result precisely from the fact that they are retrospective and rely upon the use of existing datasets that might not contain sufficient information to precisely identify the causal impact of the interventions. These deficiencies would not arise were the evaluation processes designed in tandem with the interventions and implemented concurrently with the interventions. Thus, for example, with ex-post data, it might not be possible to exactly identify who was “treated” and who was not; or important variables could be missing from the dataset that relate to the effects of the interventions; or relevant performance data might be unavailable. In fact, many more problems than these can occur. Appendix H contains a summary of such problems for the interested reader, which focuses on the differences between an ideal evaluation situation and the one to which this study applies. In reading Appendix H, one should be continually aware that few evaluations 4 “Impact evaluation” is the systematic identification of the effects (positive or negative, intended or unintended) on individual households, institutions, and the environment caused by a given development activity, such as a program or project. (2004). In OED and Impact Evaluation―A Discussion Note, World Bank. http://www.worldbank.org/oed/docs/world_bank_oed_impact_evaluations.pdf, downloaded in November 2011. 5 Refer to Appendix B for the detailed statement of work from USAID. 22 actually attain that ideal and that much information can be obtained from less-than-ideal analyses so long as one is aware of their limitations. The limitations intrinsic in this study are made clear as the methodology is described below. With these caveats in mind, the results of Optimal's evaluation of the impact of USAID’s agricultural value chain interventions are reported below. A quick summary of those results is as follows: There is little evidence that the interventions considered as a whole have been either successful for or detrimental to recipients. Examining subcategories of interventions, among the implementing organizations, there is more evidence that Mercy Corps' interventions have been successful than the interventions of any of the three other organizations. For the three other organizations, one would have to conclude that their interventions have been neither successful nor detrimental. In terms of the size of interventions, a positive effect is more likely when a municipality is subject to only one intervention than to two or more. It should be emphasized, however, that these conclusions rest on rather weak statistical evidence. The conclusions have to be considered in the context of the difficulties of undertaking retrospective evaluations using already existing—and somewhat limited—datasets, discussed above, and the resultant cautions that are inserted into the detailed discussion of methods and results below. METHODOLOGY This impact evaluation study was carried out in two stages. During the first stage, a feasibility study was conducted. The second stage corresponded to the study's data analysis. The first stage began with a site visit to the Guatemala USAID Mission to collect general background information on the interventions being evaluated. USAID/Guatemala, through its partners, implemented four agricultural projects intended to assist smallholder farmers in Guatemala in improving their farming practices and to provide access to national, regional, and international markets. The ultimate goal of these interventions was to increase employment at the local level and to make the country more competitive on the international market. The four projects were initiated under Strategic Objective 2 of USAID/Guatemala’s Country Plan for 2004–2009—Economic Freedom: Open, Diversified, and Expanding Economies. The interventions consisted of a variety of activities, including capacity building, technical assistance, financial incentives, and training for the beneficiaries. Even though all four interventions shared the same general goal, each had different specific goals and objectives. The four interventions are listed in table 1, and further details can be found in Appendix A. 23 Table 1. Agricultural projects considered in the evaluation Intervention name Program implementer Duration of program Increased Rural Household Income and Food Security Asociación Guatemalteca de Exportadores (AGEXPORT, formerly AgExpront) 2004 to present Competitive Enterprises Asociación Nacional del Café (ANACAFE) 2006 to present Quality Assurance and Small Business Development Fundación de Apoyo a la Generación de Ingresos Locales (AGIL) 2006–2011 Innovation Market Alliance for Rural Entrepreneurs (IMARE) Mercy Corps 2007 to present In addition to obtaining background information on the four USAID programs, during this visit the LEAP team met with the Guatemala Instituto Nacional de Estadistica (National Institute of Statistics; INE), whose staff provided three potential datasets with which to conduct the evaluation of the interventions: the Encuestas de Condiciones de Vida (Surveys of Living Conditions; ENCOVIs) of 2006 and 2011, and the 2002 Population Census. These existing datasets constituted all the possible data that the LEAP team accessed that was relevant to its evaluation. The next section describes these datasets and how the information in them was combined to provide the observations used in the analysis. Before the evaluation team chose the method of construction of datasets for analysis, it considered alternatives based on data availabilities and precision of information in constructed datasets. Appendix I summarizes the reasons why the LEAP team chose its methodology from the set of alternatives available. Data Selection and Management 2006 and 2011 ENCOVIs The 2006 (baseline) and 2011 (end-line) ENCOVI datasets were used for the impact evaluation. The ENCOVIs provide detailed information on living conditions, income, and consumption patterns of those surveyed.6 Each of the two ENCOVIs is divided into a household-level dataset and an individual-level dataset. The 2006 ENCOVI has a nationally representative sample of 13,686 households that was collected between March and September 2006. The individual-level database contains information on the members of these households, a sample of 70,035 individuals, of whom 33,810 are males and 36,225 are females. The 2011 ENCOVI contains a nationally representative sample of 13,482 households comprising 66,523 individuals, of whom 32,578 are males and 33,945 are females. These two ENCOVIs were cleaned and merged (by municipality). Only those municipalities for which data were available in 2006 and 2011 were kept.7 Table 2 documents the coverage of the 6 Further details on the ENCOVI data sets are found in Appendix C. 7 This selection has the potential to introduce an initial bias in the analysis data, as those municipalities that were not sampled in both years may not have been randomly selected and are likely to be among the poorest or most difficult to access in Guatemala. 24 ENCOVIs in terms of the overlap between the coverage of the survey and the interventions that are being evaluated in this study. Data were available for analysis from the 2006 and 2011 ENCOVIs for 71 of the 80 municipalities that received USAID assistance; these data correspond to 3,504 and 3,528 and household observations for 2006 and 2011, respectively. A general power analysis indicates that for a population of 14,000,000,8 a sample of fewer than 2,000 observations would be sufficient to achieve a confidence level of 95 percent for a confidence interval of 3 percent. As a result, the preliminary sample sizes obtained from both ENCOVIs were judged large enough to accomplish the evaluation analysis. ENCOVIs in terms of the overlap between the coverage of the survey and the interventions that are Table 2. Intervention-assisted municipalities surveyed in ENCOVIs’ datasets Intervention￾assisted municipalities Data available from both ENCOVIs Number of municipalities covered by implementing partners Number of survey observations (HH level) Total AGEXPORT Fundación AGIL ANACAFE Mercy Corps 2006 ENCOVI 2011 ENCOVI Yes Yes 71 40 38 10 10 3,528 3,504 No1 9 4 3 2 2 48 31 No Yes 206 0 0 0 0 9,752 9685 No1 45 0 0 0 0 358 262 Total 334 44 41 12 12 13,686 13,482 (1) These refer to the municipalities for which data were available for one of the ENCOVIs but not for the other or for neither of the ENCOVIs. Then two different analysis datasets were constructed: a. The first dataset included all the observations in the entire ENCOVI and was used in the analysis to depict the situation of the intervention municipalities relative to the municipalities in the rest of Guatemala. For ease of reference, these are referred to below as the intervention and non-intervention municipalities. b. A smaller dataset, a subset of the first, was constructed. The objective in constructing this dataset was to find a sample of intervention municipalities that was matched as closely as possible to a sample of municipalities that were not subject to the intervention. These two groups of municipalities were identified using the criteria listed immediately below. This report henceforth refers to the two sets of municipalities, and the types of households within them, as the treatment and control municipalities. (Note that the treatment municipalities are a subset of the intervention municipalities, and the control municipalities are a subset of the non-intervention municipalities.) 8 This figure, an approximation of the Guatemala total population, overestimates the size of the population affected by the interventions; thus, the sample size necessary is also overestimated. 25 The creation of treatment and control groups is a crucial element of the methodology. The essence of the methodology is that a set of “treated” (with the intervention) municipalities is identified (from project documents) and then a set of “non-treated” or control municipalities is identified, where the latter were not subject to the intervention and were very similar to the treated municipalities.9 This matching process—the identification of the control group for the analysis— proceeded as follows: • The average household income levels and count of households were estimated for all municipalities. For municipalities that included rural and urban households, estimation was done separately for rural and urban households. • For each intervention municipality that had rural households, the potential control municipalities were those that had an average rural income that differed by less than 5 percent from the rural income of the intervention municipality. • For each intervention municipality that had urban households, the potential control municipalities were those with an average urban income that differed by less than 5 percent from the urban income of the intervention municipality. • For intervention municipalities that had rural households, potential control municipalities were where the number of rural households’ was less than 50 percent different from intervention municipalities. • For intervention municipalities that had urban households, potential control municipalities were where the number of rural households was less than 50 percent different from intervention municipalities. • The percentage of rural households within the municipalities was compared between treatment and candidate control municipalities; those with a simple difference of less than 30 percent were kept. • Finally, a synthetic distance measure was calculated to delete matches between treated and control municipalities that were not close matches on any of the three criteria above. The distance measure was found by adding the percentage income difference between the intervention and control municipalities, plus the percentage population difference, plus the difference in the percentage of rural households. Treatment and control municipalities with an overall "distance" of less than 70 percent were kept. This selection methodology produced treatment and control groups comprising 28 of the intervention municipalities and 25 of the non-intervention municipalities. There are fewer control municipalities than treatment municipalities because three of the control municipalities were each matched to two treatment municipalities.10 9 Because households were not identified and were not followed from ENCOVI to ENCOVI, the matching could not be done at the household level and had to be based on a geographic unit. 10 It is important to note that the selection of treatment and control groups for analysis via matching the characteristics of the municipalities to select those that were most similar has most likely produced a biased subset of the intervention municipalities. This is particularly likely if those who benefitted from the interventions were not selected randomly. To see this, consider the case where the target population of the intervention is individuals with the lowest income levels. To the extent that everyone with low income levels has benefitted, then no individuals with low income levels remain who can be used as a control group for the intervention. Because of these situations, whenever beneficiaries of a given intervention are selected following a predetermined 26 The matching process is far from perfect. Table 3, below, presents the Guatemalan municipalities with smallholder farmers that benefitted from the interventions. Rows highlighted in orange identify the treatment municipalities—that is, those intervention municipalities for which matching control municipalities were identified for the analysis. In addition, note that this table is sorted in increasing order of average household income. Because of the sorting, it can be easily observed that the municipalities for which matches were identified are those with the lowest income level among all the intervention municipalities. As a result, the selected treatment group is biased—that is, it is not representative of the entire group of intervention municipalities. Table 3. Intervention municipalities Department Municipality Per-capita mean monthly income Household mean monthly income Estimated count of HHs Overall Rural Urban Overall Rural Urban Overall Rural Urban Huehuetenango San Sebastián Coatán 154 154 755 755 2,472 2,472 Sololá San Antonio Palopó 129 129 889 889 6,495 Alta Verapaz San Pedro Carchá 175 175 1,040 1,040 19,291 19,291 Alta Verapaz San Cristóbal Verapaz 224 224 1,094 1,094 4,842 4,842 Huehuetenango Santa Eulalia 182 182 1,144 1,144 2,891 2,891 Sololá San Andrés Semetabaj 210 210 1,154 1,154 672 672 Quiché Chajul 183 201 160 1,196 1,455 923 6,204 3,180 3,024 Totonicapán Santa Lucía La Reforma 209 209 1,255 1,255 8,400 8,400 Chimaltenango Pochuta 226 226 1,266 1,266 1,990 1,990 Alta Verapaz Chisec 199 181 280 1,388 1,342 1,541 23,006 17,762 5,244 San Marcos Tejutla 217 217 1,446 1,446 4,896 4,896 Jalapa San Pedro Pinula 284 224 607 1,521 1,307 2,234 15,750 12,122 3,628 San Marcos Tacaná 276 276 1,567 1,567 25,719 25,719 San Marcos Ixchiguan 240 240 1,568 1,568 2,453 2,453 Huehuetenango Todos Santos Cuchumatán 293 293 1,609 1,609 2,880 2,880 Baja Verapaz Cubulco 296 296 1,625 1,625 8,871 8,871 Santa Rosa Santa Cruz Naranjo 327 327 1,745 1,745 1,776 1,776 Quiché Sacapulas 281 250 306 1,860 1,668 2,008 12,461 5,424 7,037 Quiché Santo Tomas Chichicastenango 267 267 2,168 2,168 2,763 2,763 Izabal Livingston 416 291 891 2,171 1,502 4,826 11,792 9,420 2,372 Chimaltenango San José Poaquil 352 169 590 2,177 1,183 3,176 5,732 2,872 2,860 San Marcos San Pablo 395 335 574 2,273 2,062 2,765 10,483 7,326 3,157 Huehuetenango Jacaltenango 449 479 412 2,292 2,787 1,827 7,080 3,432 3,648 Chimaltenango Tecpán Guatemala 465 373 772 2,400 1,980 3,653 17,540 13,139 4,401 Quiché Joyabaj 364 364 2,562 2,562 9,420 9,420 San Marcos Malacatán 470 470 2,604 2,604 7,660 7,660 Baja Verapaz San Jerónimo 614 690 536 2,683 3,401 2,094 3,142 1,415 1,727 El Progreso Morazán 676 490 1,203 2,690 2,020 4,350 3,542 2,523 1,019 Huehuetenango Chiantla 462 414 581 2,713 2,378 3,629 16,718 12,242 4,476 Quiché Uspantán 533 98 1,284 2,765 550 5,908 8,054 4,724 3,330 Jalapa Monjas 538 612 386 2,841 3,350 1,890 4,277 2,787 1,490 Sacatepéquez Santa María de Jesús 510 510 2,863 2,863 2,365 2,365 Chimaltenango Comalapa 557 322 657 2,922 1,514 3,622 11,378 3,782 7,596 Chimaltenango Patzicía 649 587 694 3,258 3,260 3,256 6,679 2,566 4,113 Chimaltenango Patzún 579 276 1,053 3,358 1,600 6,128 4,363 2,669 1,694 Suchitepéquez Santo Domingo Suchitepequez 675 675 3,473 3,473 4,554 4,554 Totonicapán Totonicapán 681 379 819 3,501 2,112 4,067 20,926 6,064 14,862 Chimaltenango Acatenango 669 446 1,120 3,508 2,229 6,515 3,576 2,509 1,067 Chimaltenango San Martín Jilotepeque 725 333 2,418 3,591 1,668 11,488 9,476 7,620 1,856 Alta Verapaz Tactic 965 675 1,562 3,599 3,065 4,259 11,739 6,492 5,247 Chiquimula Jocotán 577 156 2,429 3,617 1,025 12,755 7,928 6,176 1,752 policy, chances increase that matched control individuals can be identified only for those beneficiaries who least needed the intervention. 27 Department Municipality Per-capita mean monthly income Household mean monthly income Estimated count of HHs Overall Rural Urban Overall Rural Urban Overall Rural Urban Zacapa Usumatlán 647 647 3,628 3,628 1,711 1,711 Sololá Santa Lucía Utatlán 801 520 1,326 3,755 2,656 5,399 5,136 3,078 2,058 Quiché San Juan Cotzal 706 394 742 3,776 2,428 3,909 7,347 660 6,687 Jalapa San Carlos Alzatate 680 680 3,851 3,851 1,770 1,770 Jalapa Mataquescuintla 700 632 783 3,927 3,430 4,576 5,120 2,897 2,223 Jutiapa Jutiapa 750 471 1,671 4,175 2,746 8,084 22,056 16,155 5,901 Baja Verapaz Salamá 853 431 1,750 4,201 2,345 7,176 10,477 6,451 4,026 Baja Verapaz Rabinal 843 609 1,101 4,205 2,985 5,607 8,754 4,680 4,074 Chimaltenango Zaragoza 955 753 1,213 4,220 3,361 5,291 6,328 3,510 2,818 Chimaltenango Santa Cruz Balanyá 802 802 4,300 4,300 1,617 1,617 Sololá San Juan La Laguna 913 913 4,382 4,382 920 920 Retalhuleu El Asintal 822 647 1,078 4,448 3,443 5,976 6,772 4,084 2,688 Sololá San Pedro La Laguna 1,080 1,080 4,626 4,626 2,902 2,902 Jalapa Jalapa 882 431 1,492 4,692 2,562 6,951 18,730 9,640 9,090 Petén San Andrés 691 498 871 4,753 3,986 5,301 4,647 1,935 2,712 Chimaltenango El Tejar 978 480 1,672 4,896 2,841 6,888 4,656 2,292 2,364 Sacatepéquez Magdalena Milpas Altas 1,016 178 1,555 5,142 1,125 6,967 1,921 600 1,321 Zacapa Teculután 1,058 967 1,304 5,364 5,310 5,475 2,178 1,468 710 Huehuetenango San Pedro Necta 787 787 5,367 5,367 6,600 6,600 Zacapa Estanzuela 1,441 1,268 1,476 5,986 5,813 6,017 3,818 588 3,230 Sacatepéquez Sumpango 1,286 729 1,515 6,519 4,305 7,258 4,275 1,070 3,205 Petén Melchor de Mencos 1,446 650 2,394 6,657 3,429 9,576 4,378 2,079 2,299 Chimaltenango Chimaltenango 1,691 730 1,769 7,812 3,214 8,202 18,799 1,470 17,329 San Marcos San Marcos 1,686 1,686 8,597 8,597 4,510 4,510 Alta Verapaz Cobán 1,731 858 2,984 8,960 4,702 14,305 46,763 26,028 20,735 Sacatepéquez Antigua Guatemala 1,918 1,233 2,153 9,212 6,527 10,020 10,098 2,336 7,762 Sacatepéquez Jocotenango 2,116 1,597 2,320 10,033 7,968 10,788 6,746 1,806 4,940 Sacatepéquez San Bartolomé Milpas Altas 2,844 2,844 10,082 10,082 1,980 1,980 Petén San Benito 2,318 248 3,498 11,381 1,735 14,678 5,606 1,428 4,178 Guatemala Guatemala 3,548 3,548 11,659 11,659 17,528 17,528 Source: Guatemala ENCOVI 2006 2002 Population Census The evaluation used 2002 census data to obtain a depiction of some of the pre-2006 economic indicators for the intervention and non-intervention municipalities. The census provided information on levels of employment, education, and other household characteristics that could be used to estimate the general economic situation of the areas analyzed. There are three main 2002 census datasets: individual, household, and living quarters. These three datasets provided information on 11,237,196 individuals, grouped into 2,200,610 households and 1,243 institutions, shared living quarters, or group homes. The 2002 census provided information on more municipalities than those contained in the ENCOVIs. For comparison purposes, only data for those municipalities included in both the 2006 and 2011 ENCOVIs were utilized for the 2002 census analysis. Lastly, the municipalities other than Guatemala in the Guatemala department had a different numbering system than those used in the ENCOVIs. Hence, for matching purposes, two groups were utilized: the municipality of Guatemala and all other municipalities in the Guatemala department. This matchup explains the large difference between the census and the ENCOVIs data in the number of non-control-non-intervention municipalities. 28 Analytical Approach The data selection and management process resulted in two datasets, one of which is a subset of the other. The first dataset contained the complete data from the 2006 and 2011 ENCOVIs, with municipalities categorized as intervention or non-intervention. This dataset is referred to below as the all-household dataset. The second dataset, the subset, contained only individuals living in the municipalities identified as part of either the treatment group or the control group, who were chosen to be matched as well as possible. This subset is referred to below as the matched dataset. The statistical analysis was performed with households as the unit of observation. Conducting the analysis this way includes an important assumption: that all the households in non-intervention municipalities were not affected by the interventions and that all the households in intervention municipalities were affected by the interventions. Although the ENCOVI datasets brought a richness of household welfare data to the analysis, it is important to keep in mind that they did not offer the ability to track specific individuals who directly benefited from the USAID programs. Therefore, this evaluation used the welfare of the households in the municipalities in which the USAID programs were implemented as a proxy for the welfare of the direct beneficiaries themselves. Consequently, the findings of this evaluation are best viewed as testing whether assisting producer groups by increasing their incomes through agricultural improvements has a measurable effect on the income of their communities as a whole. Due to these data limitations, this evaluation does not claim to precisely estimate the effects on smallholder farmers’ livelihoods following participation in USAID agricultural value chain programs. Instead, this evaluation compares the evolution of the municipalities where the assisted groups resided with trends in those municipalities where no programs were implemented. A difference-in-differences (DID) approach was utilized to identify any potential welfare effects that could be associated with the four interventions that are being evaluated. The DID approach, as applied in the present study to the all-household dataset, finds the effects of intervention by implicitly comparing outcomes for four samples of households: A. Individuals in intervention municipalities in 2006—that is, before interventions in those municipalities B. Individuals in non-intervention municipalities in 2006 C. Individuals in intervention municipalities in 2011—that is, after interventions in those municipalities D. Individuals in non-intervention municipalities in 2011 The DID estimator effectively compares differences in outcomes between samples C and D but adjusts those differences by the differences in outcomes between samples B and A. The adjustment is made because the intervention and non-intervention groups might not be exactly matched before the intervention begins, and therefore one needs to subtract any pre-existing differences in outcomes from the final difference in outcomes. When the DID approach is applied to the matched sample, the four samples of households are 29 A. individuals in treatment municipalities in 2006—that is, before interventions in those municipalities; B. individuals in control municipalities in 2006; C. individuals in treatment municipalities in 2011—that is, after interventions in those municipalities; and D. individuals in control municipalities in 2011. The above description of the methodology makes clear why the special matched sample was constructed and why it was important for the sample to have well-matched treatment and control groups. In DID analysis, the changes in outcome variables for the control group between 2006 and 2011 (that is, between samples B and D) are used as proxies for all things impinging on the outcomes of the treated individuals, apart from the treatment itself (the intervention). To the extent that this matching is unsatisfactory—and data limitations mean that it is far from perfect—the estimate of the effect of the intervention will be inaccurate (but no more inaccurate than a method that does not use matching). To the extent that one can find other control variables to add to the analysis to supplement the DID approach, this will diminish the inaccuracies. In this evaluation, therefore, a set of demographic and other control variables is added to the analysis: the number of persons in the household, age of head of household, whether the head of household is female, the number of children ages 5 to 17, the number of children ages 5 and younger, and the number of females at reproductive age (12–49). Three different specifications (DID regression equations) were used in this study: 1. The simplest specification assumed an intervention occurred if the household was located in a municipality that benefited from at least one of the four USAID interventions. 2. The second specification differentiated between the four organizations implementing the interventions—ANACAFE, AGEXPORT, Fundación AGIL, and Mercy Corps. 3. The third analysis did not differentiate specifically between the four organizations but took note of how many interventions had occurred in the municipality, differentiating between municipalities that had one intervention, two interventions, and three or more interventions. In the results section below and in the tables accompanying the results, these equations are referred to as specifications 1, 2, or 3. 30 FINDINGS: Summary Statistics Before turning to the results from the DID analysis, this section describes summary statistics obtained from the analysis of the ENCOVI datasets and the 2002 census. The reader purely interested in the bottom line—the results from the DID analysis—can skip this section and turn immediately to the section that follows, without loss of continuity but with some loss of perspective. Figure 1 depicts the mean household income for both the selected treatment and its matched control municipalities in 2006 and 2011. As this figure shows, the mean household income values in 2006 are very close to each other, with the exception of five treatment municipalities. These differences occur because these municipalities have small urban populations with large average incomes, and their matched control municipalities are 100 percent rural. A full table with the treatment and control municipalities and the statistics used to build the analysis sample is found in Appendix F of this report. In addition, because the data utilized are not representative at the municipality level, big changes observed between 2006 and 2011 cannot be interpreted as changes in the welfare level of each of the municipalities. 31 Figure 1. Treatment and matched municipalities household income, 2006 and 2011 - 2,000 4,000 6,000 8,000 10,000 12,000 Treatment 2006 Control 2006 Treatment 2011 Control 2011 32 Tables 4 through 9, below, contain summary statistics for several outcome variables that may have been affected by the interventions. The sources of the data are made clear in footnotes to the tables. The summary statistics in these tables are presented separately for the treatment group, the control group, the intervention group, and the non-intervention group. Table 4 presents the average per-capita and household incomes for intervention and non￾intervention populations, overall and separated by inclusion in the analysis sample. The bottom row of each sub-table shows the summary data for Guatemala in each respective year. As mentioned above, this table shows that the treatment-group households had on average lower monthly incomes in 2006 than those subject to the intervention but not in the treatment group. The same finding holds for the control households. This situation is also observed in 2011 but at a lower magnitude. It is found that, in 2011, the treatment and control selected households appear to have increased their average monthly incomes. This finding is in contrast with the situation of the non-selected intervention and non-intervention households, which appear to be worse off in real terms than in 2006. Table 4 also shows that, despite the fact that matching at the municipality has some problems, the aggregated average household and per-capita income levels of the selected treatment and control groups are similar in 2006, which provides some validity to the procedure for constructing the treatment and control groups. In addition, table 4 shows large differences in terms of income and proportions of rural and urban populations between the intervention households that were not sorted into the smaller treatment and control samples and those that were sorted. This indicates that the smaller treatment-and￾control sample is not representative of the whole population, suggesting the necessity of caution when interpreting the results. Because the 2002 census did not provide income data, only general population characteristics are described for 2002 in table 4. Note that all income figures in this report have been adjusted for price changes and are therefore all in constant 2006 units of currency. Table 5 and figures 2–4 focus on educational achievement and literacy. Figure 2 shows that the educational achievement of the head of household in intervention municipalities decreases over the period considered, as the proportion of those with no formal education and with only some primary education increases from 2002 to 2011. In contrast, Figure 3 shows that the educational level of heads of household in non-intervention municipalities increases from 2002 to 2011. Figure 4, in turn, shows the overall evolution of the educational level of heads of household in Guatemala between 2002 and 2011. A sizeable decrease in the proportion of heads of household with no formal education and a smaller decrease in the proportion of those with some primary education are observed. The larger increases are found for those with completed primary education and those with some secondary education. Overall, this graph indicates an increase in the educational levels completed for heads of household in Guatemala over the period 2002–2011; however, this positive change is driven by the non-intervention municipalities. 33 Table 4. Summary characteristics: Income and count of households 2002 summary statistics at household level Intervention In either treatment or control? Number of municipalities Number of households Family size Yes Yes 28 176,188 5.7 No 43 516,761 4.9 All 71 692,949 5.1 No Yes 25 105,644 5.5 No 158 1,268,811 5.0 All 183 1,374,455 5.1 All All 254 2,067,404 5.1 Source: Guatemala Census 2002 2006 summary statistics at household level Intervention In either treatment or control? Number of municipalities Number of households in sample % of rural HHs in sample Mean number of household members Per-capita mean monthly income Household mean monthly income Estimated count of HHs Original data in group Original data in group Overall Rural Urban Overall Rural Urban Overall Rural Urban Yes Yes 28 995 79.2% 5.9 421 318 867 2,463 1,894 4,583 13,272 9,672 3,600 No 43 2,533 51.2% 5.2 959 463 1,586 4,956 2,883 7,266 18,745 9,872 8,874 All 71 71 3528 3,528 60.5% 5.4 764 397 1,446 4,124 2,557 6,551 17,274 9,806 7,468 No Yes 25 555 86.5% 5.4 405 346 798 2,192 1,902 3,466 8,689 6,503 2,186 No 181 9,197 39.0% 4.7 1,459 716 2,048 6,830 4,239 7,962 37,132 7,588 29,543 All 206 206 9752 9,752 41.8% 4.7 1,388 670 2,027 6,556 4,073 7,902 36,686 7,511 29,175 Guatemala All 277 277 13,280 13,280 46.2% 4.9 1,228 582 1,919 5,991 3,653 7,604 32,536 8,147 24,390 Source: Guatemala ENCOVI 2006 2011 summary statistics at household level Intervention In either treatment or control? Number of municipalities Number of households in sample % of rural HHs in sample Mean number of household members Per-capita mean monthly income Household mean monthly income Estimated count of HHs Original data in group Original data in group Overall Rural Urban Overall Rural Urban Overall Rural Urban Yes Yes 28 1,139 76.6% 5.6 432 323 832 2,418 1,899 4,027 17,081 10,411 6,671 No 43 2,365 51.6% 5.2 808 473 1,228 4,242 2,967 6,260 28,718 15,448 13,270 All 71 71 3504 3,504 60.1% 5.4 675 407 1,145 3,621 2,594 5,678 25,240 13,689 11,551 No Yes 25 657 77.7% 5.3 512 422 875 2,733 2,439 4,350 11,629 7,679 3,950 No 181 9,028 40.5% 4.7 1,145 491 1,677 5,382 2,782 6,742 35,875 8,657 27,218 All 206 206 9685 9,685 43.1% 4.7 1,096 482 1,653 5,200 2,753 6,658 34,970 8,572 26,398 Guatemala All 277 277 13,189 13,189 47.2% 4.9 985 458 1,552 4,820 2,709 6,464 33,435 9,983 23,451 Source: Guatemala ENCOVI 2011 34 Table 5. Summary characteristics and educational achievement: Head of household 2002 head of household characteristics and educational achievement Intervention In either treatment or control? Head of household Education level of head of household % female Literacy Average age None Some primary Primary Some secondary Secondary Some college College and higher Yes Yes 19.3% 52.2% 44 47.8% 34.7% 9.5% 4.0% 2.5% 1.1% 0.4% No 25.5% 73.6% 45 26.4% 26.1% 16.6% 13.2% 7.5% 7.8% 2.4% All 23.9% 68.2% 45 31.9% 28.3% 14.8% 10.9% 6.2% 6.1% 1.9% No Yes 19.2% 51.5% 44 48.5% 33.0% 10.5% 4.5% 2.4% 0.8% 0.3% No 22.6% 67.8% 45 32.2% 31.4% 15.9% 9.9% 5.5% 3.8% 1.3% All 22.3% 66.6% 45 33.5% 31.5% 15.5% 9.5% 5.3% 3.6% 1.2% All All 22.9% 67.1% 45 32.9% 30.4% 15.3% 9.9% 5.6% 4.4% 1.4% Source: Guatemala Census 2002 2006 head of household characteristics and educational achievement Intervention In either treatment or control? Head of household Education level of head of household % Female Literacy Average age None Some primary Primary Some secondary Secondary Some college College and higher Yes Yes 15.7% 54.4% 45 52.1% 31.5% 9.6% 3.4% 1.9% 0.8% 0.2% No 20.7% 70.7% 45 34.2% 29.8% 13.5% 9.9% 6.7% 3.8% 1.7% All 19.0% 65.2% 45 40.2% 30.4% 12.2% 7.7% 5.1% 2.8% 1.2% No Yes 24.5% 55.3% 45 51.3% 31.9% 10.7% 4.3% 1.2% 0.5% 0.0% No 23.6% 76.5% 45 26.6% 29.4% 15.4% 13.3% 6.4% 5.8% 2.8% All 23.6% 75.2% 45 28.0% 29.5% 15.1% 12.7% 6.1% 5.5% 2.7% All All 22.6% 72.9% 45 30.9% 29.7% 14.5% 11.6% 5.8% 4.9% 2.3% Source: ENCOVI 2006 2011 head of household characteristics and educational achievement Intervention In either treatment or control? Head of household Education level of head of household % Female Literacy Average age None Some primary Primary Some secondary Secondary Some college College and higher Yes Yes 17.8% 65.5% 43 37.7% 36.3% 12.4% 5.9% 4.5% 1.8% 0.1% No 20.0% 71.6% 44 30.1% 29.4% 15.0% 11.6% 7.3% 3.2% 1.3% All 19.2% 69.5% 44 32.7% 31.7% 14.1% 9.6% 6.4% 2.8% 0.9% No Yes 16.4% 59.6% 45 43.3% 31.7% 13.5% 6.8% 2.9% 1.0% 0.1% No 22.4% 77.2% 46 25.2% 28.9% 17.7% 14.2% 6.3% 4.9% 1.7% All 22.0% 76.0% 46 26.4% 29.1% 17.4% 13.6% 6.1% 4.6% 1.6% All All 21.3% 74.5% 45 27.9% 29.7% 16.6% 12.7% 6.1% 4.2% 1.4% Source: ENCOVI 20011 35 Figure 2. Educational achievement in intervention municipalities, 2002, 2006 & 2011 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 None Some Primary Primary Some Secondary Secondary Some College College and Higher Educational Level Achieved by Head of Household Intervention Municipalities 36 Figure 3. Educational achievement in non-intervention municipalities, 2002, 2006 & 2011 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 None Some Primary Primary Some Secondary Secondary Some College College and Higher Educational Level Achieved by Head of Household Non-Intervention Municipalities 37 Figure 4. Educational achievement in Guatemala, 2002, 2006 & 2011 Figure 4 depicts basic statistics for the head of the household in 2006 and 2011. It shows that the literacy level increased for all groups between 2006 and 2011, and this increase appears to be larger for the selected treatment households (11.1 percent for treatment households, compared to 4.3 percent for control households). In contrast, participation in civic organizations appears to be reduced in 2011 for all groups. The average incidence of illness and accidents also lessens from 2006 to 2011 (11.6 percent for the treatment households and 9.8 percent for their corresponding control households). It is also observed that a small proportion of households has health insurance; moreover, it is found that this number decreases to half its initial value by 2011 for the households located in the intervention municipalities. 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 2002 2006 2011 None Some Primary Primary Some Secondary Secondary Some College College and Higher Educational Level Achieved by Head of Household Guatemala 38 Table 7 presents health statistics for children ages 5 years and younger. The most striking finding in this table is the sharp increase in the proportion of children receiving the pentavalent vaccine. In addition, this table highlights some initial differences between the selected treatment and control households, which were selected to be as similar as possible in terms of income in 2006. This table shows a large initial difference in the proportion of children vaccinated for measles and polio and receiving the pentavalent vaccine between the selected treatment and control households in 2006. This finding may indicate some intrinsic differences between both groups that may not be observed and thus could not be used in selecting treatment and control groups. These intrinsic differences may potentially affect the validity of conclusions from the evaluation. Table 8 depicts indicators of female health for women of reproductive age (12–49 years old) and for children of school age (5–17 years old). Large improvements are observed in females’ reproductive health and knowledge between 2006 and 2011. As in Table 7, sizeable differences are observed between the treatment and the control households in 2006 regarding reproductive health, knowledge, and children’s school attendance. As with Table 7, these differences may indicate unobservable disparities between both groups that may affect the outcome of the evaluation. Table 9 presents the summary statistics for the home. As in tables 7 and 8, there are large differences in 2006 for some of the indicators shown, in particular, for the connection to a grid of water distribution and electricity. 39 Table 6. Summary characteristics: Health and health insurance of head of household 2006 summary statistics at household level for analysis sample and non-selected municipalities Intervention In either treatment or control? Head of household: Organization participation & health indicators, previous month Organization participation Illness or accident Interruption of activities Days of interruption Required hospital admittance Days in hospital Health insurance Yes Yes 33.7% 33.0% 60.0% 10.6 3.1% 8.3 0.6% No 33.4% 35.3% 54.9% 8.9 5.3% 5.2 2.5% All 33.5% 34.5% 56.5% 9.5 4.5% 5.9 1.8% No Yes 35.8% 37.9% 57.0% 7.7 4.1% 3.8 1.5% No 28.8% 34.7% 45.1% 8.6 6.1% 5.7 3.4% All 29.2% 34.9% 45.9% 8.5 5.9% 5.6 3.3% Guatemala All 30.2% 34.8% 48.3% 8.8 5.5% 5.7 3.0% Source: Guatemala ENCOVI 2006 2011 summary statistics at household level for analysis sample and non-selected municipalities Intervention In either treatment or control? Head of household: Organization participation & health indicators, previous month Organization participation Illness or accident Interruption of activities Days of interruption Required hospital admittance Days in hospital Health insurance Yes Yes 14.4% 21.4% 50.9% 10.0 5.5% 9.1 0.3% No 18.6% 29.7% 67.5% 8.6 1.9% 5.2 1.2% All 17.1% 26.9% 63.1% 8.9 2.7% 7.0 0.9% No Yes 14.9% 28.1% 63.7% 7.6 2.3% 19.4 1.5% No 16.9% 24.5% 46.6% 9.4 5.7% 6.4 1.7% All 16.7% 24.7% 47.9% 9.3 5.3% 7.0 1.7% Guatemala All 16.8% 25.3% 51.8% 9.1 4.5% 7.0 1.5% Source: Guatemala ENCOVI 2011 40 Table 7. Summary characteristics: Children ages 5 years and younger 2006 summary statistics at household level for analysis sample and non-selected municipalities Interventio n In either treatmen t or control? Children 5 years and younger: Vaccines and breast feeding Children 5 years and younger: Health last month # children 5 or younger in HH % vaccinated measles % vaccinated BCG % vaccinated pentavalent % vaccinated DPT % vaccinated Polio % breast fed at least 6 months (older than 1 year) % diarrhea % respiratory infection % other accident or illness Yes Yes 1.16 74.9% 95.9% 24.1% 87.2% 90.6% 94.0% 33.8% 53.6% 2.4% No 0.84 75.4% 94.6% 26.4% 83.2% 88.9% 91.0% 30.3% 57.2% 3.2% All 0.95 75.2% 95.0% 25.6% 84.6% 89.5% 92.1% 31.6% 55.9% 2.9% No Yes 0.93 81.9% 97.2% 30.8% 88.5% 94.3% 93.8% 33.3% 48.6% 2.9% No 0.72 76.7% 94.6% 34.7% 81.6% 88.7% 84.0% 27.5% 51.1% 3.6% All 0.74 77.0% 94.8% 34.4% 82.1% 89.1% 84.7% 27.9% 50.9% 3.5% Guatemala All 0.79 76.6% 94.9% 32.2% 82.7% 89.2% 86.7% 28.8% 52.2% 3.4% Source: Guatemala ENCOVI 2006 2011 summary statistics at household level for analysis sample and non-selected municipalities Interventio n In either treatmen t or control? Children 5 years and younger: Vaccines and breast feeding Children 5 years and younger: Health last month # children 5 or younger in HH % vaccinated measles % vaccinated BCG % vaccinated pentavalent % vaccinated DPT % vaccinated polio % breast fed at least 6 months (older than 1 year) % diarrhea % respiratory infection % other accident or illness Yes Yes 0.91 78.8% 97.9% 90.9% 89.7% 90.4% 92.7% 28.1% 42.8% 2.1% No 0.82 79.4% 96.5% 94.7% 89.4% 92.0% 89.6% 35.2% 52.1% 3.1% All 0.85 79.2% 97.0% 93.4% 89.5% 91.4% 90.7% 32.7% 48.8% 2.8% No Yes 0.83 76.4% 98.1% 92.5% 90.7% 93.9% 92.6% 32.7% 42.3% 1.2% No 0.66 77.8% 97.0% 93.5% 91.0% 91.2% 86.4% 28.4% 45.6% 3.2% All 0.67 77.7% 97.1% 93.4% 91.0% 91.4% 86.9% 28.7% 45.4% 3.1% Guatemala All 0.71 78.1% 97.0% 93.4% 90.6% 91.4% 88.0% 29.8% 46.3% 3.0% Source: Guatemala ENCOVI 2011 41 Table 8. Summary characteristics: Female health and children of school age 2006 summary statistics at household level for analysis sample and non-selected municipalities Intervention In either treatment or control? Female reproductive health (ages 12–49) Number of children of school age (5–17) Percentage of children of school age attending school # females of rep. age in HH % pap smear % mammography % have heard of birth control Total Female Total Female Yes Yes 1.56 2.6% 0.1% 3.8% 2.3 1.1 54.8% 52.7% No 1.51 5.1% 0.4% 7.3% 1.9 0.9 65.0% 65.0% All 1.53 4.3% 0.3% 6.1% 2.0 1.0 61.5% 60.5% No Yes 1.46 5.3% 0.4% 8.8% 2.3 1.1 60.0% 60.8% No 1.36 7.3% 0.7% 9.3% 1.6 0.8 66.7% 65.4% All 1.37 7.2% 0.6% 9.3% 1.6 0.8 66.2% 65.1% Guatemala All 1.40 6.5% 0.6% 8.5% 1.7 0.9 65.0% 63.9% Source: Guatemala ENCOVI 2006 2011 summary statistics at household level for analysis sample and non-selected municipalities Intervention In either treatment or control? Female reproductive health (ages 12–49) Number of children of school age (5–17) Percentage of children of school age attending school # females of rep. age in HH % pap smear % mammography % have heard of birth control Total Female Total Female Yes Yes 1.58 18.6% 1.2% 24.7% 2.0 1.0 62.3% 61.9% No 1.56 22.9% 2.0% 36.2% 1.8 0.9 67.6% 69.4% All 1.57 21.4% 1.7% 32.2% 1.9 0.9 65.7% 66.8% No Yes 1.51 17.0% 0.7% 28.0% 2.0 0.9 67.1% 66.0% No 1.40 27.0% 2.2% 42.7% 1.5 0.7 69.0% 69.2% All 1.40 26.3% 2.1% 41.6% 1.6 0.8 68.8% 68.9% Guatemala All 1.44 25.1% 2.0% 39.3% 1.6 0.8 68.0% 68.3% Source: Guatemala ENCOVI 2011 42 Table 9. Summary characteristics: Home characteristics 2006 summary statistics at household level for analysis sample and non-selected municipalities Intervention In either treatment or control? Home Characteristics % own living quarters % connected to water distribution network % connected to drainage network % connected to electricity distribution network % connected to phone network % connected to a water consumption meter % connected to an electricity consumption meter Average number of rooms Yes Yes 86.8% 66.4% 16.3% 60.5% 4.0% 11.9% 60.4% 1.9 No 83.6% 76.2% 37.2% 81.8% 17.1% 31.5% 81.2% 2.3 All 84.6% 73.0% 30.2% 74.7% 12.8% 25.0% 74.2% 2.2 No Yes 91.3% 74.7% 16.3% 68.7% 5.0% 8.1% 69.0% 2.0 No 69.8% 80.8% 52.2% 84.7% 24.1% 46.1% 83.2% 2.4 All 71.1% 80.5% 50.1% 83.8% 23.0% 43.9% 82.3% 2.4 Guatemala All 74.2% 78.7% 45.5% 81.7% 20.6% 39.5% 80.5% 2.4 Source: Guatemala ENCOVI 2006 2011 summary statistics at household level for analysis sample and non-selected municipalities Intervention In either treatment or control? Home Characteristics % own living quarters % connected to water distribution network % connected to drainage network % connected to electricity distribution network % connected to phone network % connected to a water consumption meter % connected to an electricity consumption meter Average number of rooms Yes Yes 84.4% 63.0% 16.1% 71.5% 4.9% 10.2% 68.7% 1.9 No 83.3% 70.7% 34.4% 71.8% 9.2% 26.9% 70.0% 2.1 All 83.7% 68.1% 28.2% 71.7% 7.8% 21.2% 69.5% 2.1 No Yes 82.7% 65.9% 27.7% 63.8% 2.3% 15.7% 60.3% 1.9 No 70.0% 77.5% 51.1% 82.4% 13.9% 43.5% 78.2% 2.2 All 70.9% 76.7% 49.5% 81.2% 13.1% 41.6% 77.0% 2.2 Guatemala All 74.0% 74.6% 44.3% 78.9% 11.8% 36.7% 75.2% 2.2 Source: Guatemala ENCOVI 2011 43 FINDINGS: Differences-in-Differences Regression Analyses Following the methodology introduced above, DID regression analyses were performed on the two alternative datasets, each constructed from the ENCOVI data: 1. All the municipalities in the ENCOVI data for which there were observations in 2006 and 2011 (71 intervention municipalities and 206 non-intervention municipalities) 2. The selected matched treatment and control households (28 treatment and 25 control municipalities) Of course, the treatment households in dataset 2 are a subset of the intervention households in dataset 1, and the control households in dataset 2 are a subset of the non-intervention households in dataset 1. The hope is that because of the way dataset 2 was constructed, it contains a set of intervention households with characteristics similar to those of its non-intervention households, thereby isolating the effects of interventions more accurately. The DID analyses utilizing dataset 1 compare outcomes of the households located in the municipalities that received the intervention relative to the rest of Guatemala. In turn, the analyses utilizing the matched dataset 2 compare the outcomes of the selected treatment households with those of their matched control households. Eight household and family welfare indicators were selected for the DID analyses. These variables may potentially reflect any effects that the interventions may have had on the well-being of households in the recipient municipalities: • Per-capita income • Household income • Literacy of the head of household • School attendance of females ages 5 to 17 • Awareness of birth control methods among women of reproductive age (12–49) • Children ages 5 or younger who have received the pentavalent vaccine • Connection of the living quarters to an electricity distribution grid • Rate of ownership of living quarters The DID analyses used linear regressions for the continuous dependent variables (such as income, percentage of girls attending school, percentage of females aware of birth control methods, and rates at which children have been vaccinated) and logistic regressions for the dichotomous dependent variables (such as literacy, connection to electricity, and ownership of living quarters). All the DID analyses included control variables, although they were small in number; therefore, the explanatory power (R2, or goodness-of-fit) of these regressions is low. The presentation of the results for each of the eight indicators is identical. First, a figure is presented that summarizes the lowest level of a DID analysis—that is, it conveys the simple means of the outcome variables for 2006 and for 2011 and for each of the two groups in each of the two datasets noted above. Then a table is presented that captures the DID regression results for each of 44 the three specifications described in the methodology section, each of which compares outcomes from a baseline of no interventions to outcomes from interventions of the following types: 1. Any type of intervention, irrespective of implementing organization and number of interventions 2. Interventions by each of the four implementing organizations separately 3. Interventions differentiated by the number being implemented simultaneously in a municipality In the descriptions of the results below and in the accompanying tables, these are referred to as specifications 1, 2, or 3. All the specifications in these analyses contain a control variable that is an interaction between the intervention variable(s) and the year 2011. The coefficient on this interaction variable indicates the change in the outcome variable from 2006 to 2011 for the intervention households relative to the change for non-intervention households—that is, the interaction variables are the key variables to examine in terms of the main focus of this evaluation. The statistical significance of the outcome obtained is indicated as follows: • Three asterisks (***) indicate statistical significance at the 1 percent level • Two asterisks (**) indicate statistical significance at the 5 percent level • One asterisk (*) indicates statistical significance at the 10 percent level Although there are inherent problems in the methodology of this evaluation—detailed above and due to the fact that it began after the interventions and therefore relied on existing data—it should be noted that four different DID analyses were conducted (the figure and the three regressions specifications). To the extent that the results are consistent between all four for some particular outcome variable, those results are more reliable. Further clarification on the nature of the tables and the figures are made in context, when the first set of results is introduced immediately below (those on per-capita income). A full set of results with additional analysis for other relevant welfare-related indicators is found in Appendix E. 45 Per-capita income Figure 5, below, summarizes statistics on overall per-capita income (in 2006 currency units) by whether municipalities received any of the four interventions considered and by whether they were included in the matched estimation sample. This is a crude visual DID analysis, so long as the appropriate comparisons are made. Hence, a description of this visual DID might help readers who are unfamiliar with regression DID to understand how it works in making the visual DID compare each 2006 figure to each corresponding 2011 figure. For example the "in sample, intervention" group (that is, the treatment group) rises slightly from 2006 to 2011. In contrast, the "in sample, non-intervention" group (that is, the control group) rises by a greater amount from 2006 to 2011. This is evidence that the interventions have not been successful, because those receiving the interventions have seen a smaller rise in income than those not receiving the interventions. For this analysis, however, the conclusions from the results vary between the different samples. For the "all, intervention" group, income falls slightly from 2006 to 2011. In contrast, the "in sample, non-intervention" group income falls by a much greater amount from 2006 to 2011. This is evidence that the interventions have been successful, because those receiving the interventions have seen a smaller fall in income than those not receiving the interventions. Figure 5. Per-capita income in Guatemala, 2006–2011 - 200 400 600 800 1,000 1,200 1,400 1,600 Matched sample All Matched sample All All Intervention Non-Intervention Guatemala Per-Capita Income 2006 2011 Figure 5 reveals one possible source of the differences in the results between the smaller sample and the larger sample: Those households included in the treatment versus the smaller control 46 sample are considerably poorer than those in the larger sample. Perhaps the interventions differentially affected households at different income levels. The graphical DID, such as the one in figure 5, has several deficiencies. First, the important results are found via "eyeballing." Second, these estimates do not control for some variables that are available for the analysis that capture family characteristics. Third, graphs do not yield levels of statistical significance. For these reasons, the analysis turns to the use of regressions. Table 10 presents the results of the analysis of per-capita income. Tables 11–17, which contain the results for the other seven outcome variables, have identical structures to that of table 10. Therefore that structure is described in detail here. Each table contains six regression analyses—one for each of the three specifications (capturing different details on the interventions) times one for each of the two different samples, the all￾household sample and the sample of matched municipalities. The coefficients in the tables of most direct interest here are those that capture the effect of the interventions and generally appear in the tables as "[intervention variable] * Year 2011." The coefficient on this variable in the table captures the size and sign of the effect of the intervention on the particular outcome, and the t-statistic captures the statistical significance of the result. Statistical significance is indicated as follows: • Three asterisks (***) indicate statistical significance at the 1 percent level • Two asterisks (**) indicate statistical significance at the 5 percent level • One asterisk (*) indicates statistical significance at the 10 percent level For specification 1, the "Intervention * Year 2011" coefficient captures the effect of the intervention at year 2011, which for the complete sample has a coefficient of "0.11" with a t-statistic of "1.51," indicating that intervention produces a positive effect. But this estimated coefficient is "−0.32" for the matched sample, with a t-statistic of "−2.21," which indicates a negative effect of the intervention that is statistically significant. One can note that the qualitative features of these results are exactly the same as those identified by analyzing figure 5. For specification 2, the method of interpreting the results in the tables is the same, except the effects are attributed not to interventions in general but rather to the individual organization implementing each intervention. These individual organization effects are captured in the results for the coefficients that appear as "[organization name] * Year 2011" in the table (for example, "agexport * Year 2011"). In table 10, the results for AGEXPORT, Fundación AGIL, and ANACAFE show the same patterns as the results for specification 1 for all interventions in general—that is, an indicated positive effect for the all-households sample, but a negative effect for the matched￾households sample. But here the results for Mercy Corps are more interesting, because the coefficients for the "mercycorp * Year 2011" variable are positive for both samples and statistically significant for the all-household sample. This result indicates—with all the caveats discussed above—that the Mercy Corps intervention had positive effects on per-capita income. No such conclusion can be made for the other three implementing organizations. 47 Specification 3 examines whether there are differences in the estimates of the effects of interventions according to how many of the interventions have reached a certain municipality. Thus, for example, the "2 interventions * Year 2011" coefficient indicates the effect of interventions on per-capita income if there are two interventions in the municipality. The results for specification 3 do not add much information to the results of specification 1—they contain the same variations between the results for the whole sample and those for the matched sample, an estimated positive effect in the first case and an estimated negative effect in the second case. The size of the coefficients displayed in the tables also provides information. Because the dependent variable is the log of per-capita income in table 10, these coefficients can be interpreted as indicating the percentage change in income resulting from a unit change in the variable characterizing the intervention. For example, take the 0.08 coefficient for the “mercycorp * Year 2011” variable found for specification 2 using the all-household sample. This number indicates that a Mercy Corps intervention resulted in an increase in per-capita income of 8 percent. This figure was especially important because, by examining the "mercycorp" coefficient of −0.12, one can tell that the households subject to the Mercy Corps intervention were initially 12 percent poorer than those not subject to the intervention. Thus, two-thirds of the income gap was closed by the Mercy Corps intervention. 48 Table 10. DID analysis of the log of per-capita income Log of per-capita income All ENCOVI data N = 26136 Matched analysis sample N = 3467 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant 5.99 47.28 *** 5.18 24.82 *** 1 Persons by household 0.06 4.34 *** 0.06 2.37 ** 1 Age of head of household 0.05 8.16 *** 0.03 3.35 *** 1 Age of head of household squared –0.00 –9.01 *** –0.00 –3.91 *** 1 Female head of household –0.03 –0.80 0.10 1.23 1 Number of children ages 5–17 –0.33 –15.90 *** –0.20 –6.23 *** 1 Number of children ages 5 and younger –0.32 –14.53 *** –0.26 –8.54 *** 1 Number of females of reproductive age (12–49) 0.06 4.41 *** 0.06 1.84 * 1 Intervention –0.52 –4.70 *** 0.12 0.86 1 Year 2011 –0.34 –7.49 *** 0.10 0.84 1 Intervention * year 2011 0.11 1.51 –0.32 –2.21 ** 2 Constant 5.97 47.06 *** 5.22 26.13 *** 2 Persons by household 0.06 4.49 *** 0.06 2.32 ** 2 Age of head of household 0.05 8.17 *** 0.03 3.38 *** 2 Age of head of household squared –0.00 –9.03 *** –0.00 –3.97 *** 2 Female head of household –0.03 –0.88 0.10 1.20 2 Number of children ages 5–17 –0.33 –16.09 *** –0.20 –6.21 *** 2 Number of children ages 5 and younger –0.32 –14.02 *** –0.26 –8.36 *** 2 Number of females of reproductive age (12–49) 0.06 4.47 *** 0.06 1.81 * 2 agexport –0.47 –3.68 *** 0.17 1.18 2 agexport * year 2011 0.07 0.78 –0.27 –1.90 * 2 fundacion agil –0.18 –1.42 –0.04 –0.30 2 fundacion agil * year 2011 0.10 1.00 –0.06 –0.38 2 anacafe –0.31 –1.76 * –0.10 –0.91 2 anacafe * year 2011 0.03 0.22 –0.33 –2.68 *** 2 mercycorp –0.12 –0.79 –0.18 –1.25 2 mercycorp* year 2011 0.08 0.87 0.11 0.80 2 Year 2011 –0.34 –7.57 *** 0.06 0.54 3 Constant 5.98 47.01 *** 5.18 24.85 *** 3 Persons by household 0.06 4.46 *** 0.06 2.34 ** 3 Age of head of household 0.05 8.15 *** 0.03 3.38 *** 3 Age of head of household squared –0.00 –9.03 *** –0.00 –3.96 *** 3 Female head of household –0.03 –0.81 0.10 1.21 3 Number of children ages 5–17 –0.33 –16.38 *** –0.20 –6.15 *** 3 Number of children ages < 5 –0.32 –14.37 *** –0.26 –8.55 *** 3 Number of females of reproductive age (12–49) 0.06 4.43 *** 0.06 1.86 * 3 1 intervention –0.45 –3.56 *** 0.16 1.01 3 1 intervention * year 2011 0.10 1.06 –0.37 –2.49 ** 3 2 interventions –0.60 –4.93 *** 0.07 0.41 3 2 interventions * year 2011 0.20 1.59 –0.24 –1.07 3 3 interventions –0.63 –2.00 ** –0.01 –0.04 3 3 interventions * year 2011 0.08 1.38 –0.30 –2.23 ** 3 Year 2011 –0.34 –7.49 *** 0.10 0.84 Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 49 Household income Having established the approach to analyzing the information in the figures and the DID-regression tables, the report now summarizes the conclusions drawn from the figures and tables without repeating details that pinpoint the source of each conclusion. The method of drawing the conclusions is exactly the same as for per-capita income, and the form of the figures and tables (and their underlying analyses) are exactly the same. The only thing that varies between the subsections is the outcome variable. Figure 6 provides the simplest DID analysis for overall household income (in 2006 currency units). A quick visual inspection confirms that the results are approximately the same as those for per￾capita income in figure 5. This result is not surprising given the similar nature of the variables. The same comment can be made about the similarity between the results for tables 10 and 11. Their analyses are specified in exactly the same way, and the outcome variables are highly related to each other. Therefore, the substantive results from the two tables are qualitatively identical. Figure 6. Household income in Guatemala, 2006–2011 - 1,000 2,000 3,000 4,000 5,000 6,000 7,000 Matched sample All Matched sample All All Intervention Non-Intervention Guatemala Household Income 2006 2011 50 Table 11. DID analysis of the log of household income Log of household income All ENCOVI data N=26136 Matched analysis sample N=3467 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant h 6.69 51.79 *** 5.94 28.93 *** 1 Persons by ousehold 0.28 18.21 *** 0.23 8.33 *** 1 Age of head of household 0.05 8.44 *** 0.04 3.45 *** 1 Age of head of household squared –0.00 –10.40 *** –0.00 –4.42 *** 1 Female head of household ages –0.13 –3.71 *** 0.02 0.27 1 Number of children 5–17 –0.37 –16.35 *** –0.21 –6.34 *** 1 Number of children ages d younger –0.40 –15.70 *** –0.31 –8.70 *** 1 Number of females reproductive age (12–49) 0.07 5.12 *** 0.07 1.82 * 1 5 an of Intervention –0.53 –4.36 *** 0.10 0.67 1 Year 2011 –0.32 –6.57 *** 0.10 0.70 1 Intervention * year 2011 0.10 1.27 –0.26 –1.56 2 Constant h 6.68 51.74 *** 6.00 30.49 *** 2 Persons by ousehold 0.28 18.57 *** 0.23 8.12 *** 2 Age of head of household 0.05 8.44 *** 0.03 3.44 *** 2 Age of head of household squared –0.00 –10.41 *** –0.00 –4.46 *** 2 Female head of household ages 5 an –0.13 –3.84 *** 0.02 0.26 2 Number of children 5–17 –0.37 16.29 *** –0.21 –6.33 *** 2 Number of children ages d younger –0.40 15.25 *** –0.30 –8.53 *** 2 Number of females of reproductive age (12–49) 0.07 5.19 *** 0.06 1.80 * 2 agexport –0.48 –3.10 *** 0.17 1.15 2 agexport * year 2011 0.05 0.46 –0.25 –1.62 2 fundacion agil –0.22 –1.59 –0.08 –0.60 2 fundacion agil * year 2011 0.12 1.12 0.01 0.07 2 anacafe –0.29 –1.45 –0.24 –1.94 * 2 anacafe * year 2011 0.02 0.18 –0.18 –1.35 2 mercycorp –0.13 –0.81 –0.18 –1.37 2 mercycorp * year 2011 0.11 1.22 0.12 0.83 2 Year 2011 –0.32 –6.72 *** 0.05 0.39 3 Constant h 6.68 51.69 *** 5.95 29.52 *** 3 Persons by ousehold 0.28 18.44 *** 0.23 8.24 *** 3 Age of head of household 0.05 8.42 *** 0.04 3.46 *** 3 Age of head of household squared –0.00 –10.39 *** –0.00 –4.47 *** 3 Female head of household ages –0.13 –3.76 *** 0.02 0.25 3 Number of children 5–17 –0.37 –16.58 *** –0.21 –6.23 *** 3 Number of children ages d younger –0.40 –15.56 *** –0.31 –8.7 *** 3 Number of females reproductive age (12–49) 0.07 5.16 *** 0.07 1.85 * 3 5 an of 1 intervention –0.46 –3.13 *** 0.16 0.98 3 1 intervention * year 2011 0.05 0.56 –0.34 –2.07 ** 3 2 interventions –0.65 –4.69 *** 0.03 0.17 3 2 interventions * year 2011 0.22 1.73 * –0.14 –0.54 3 3 interventions –0.65 –1.92 * –0.07 –0.53 3 3 interventions * year 2011 0.12 1.30 –0.21 –1.41 3 Year 2011 –0.32 –6.57 *** 0.10 0.69 Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 Log of household income All ENCOVI data N=26136 Matched analysis sample N=3467 51 Literacy of the Head of Household Figure 7, below, summarizes literacy rates in exactly the same way as the previous figures do for income. However, in this figure, literacy rates are available from the 2002 census and therefore are included. The measure of literacy is whether the heads of the households know how to read and write. Literacy levels increase from 2006 to 2011, but the increase seems to be greater for those exposed to the interventions than those not so exposed. This is a first indication that the results for literacy might reveal less ambiguous conclusions about the effect of the interventions. Table 12 shows a positive effect of intervention in specification 1 and a positive effect of either one or two interventions in specification 3 for both samples; however, the statistical significance is weak. In specification 3 for both samples, the effect of three interventions is negative, with statistical significance for the all-household sample. In specification 2 for both samples, AGEXPORT has a positive coefficient and Mercy Corps has a negative coefficient, indicating positive and negative effects for these two organizations, respectively. The results for both of these organizations are statistically significant in the smaller matched sample. Figure 7. Literacy of the head of household, 2006–2011 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% Matched sample All Matched sample All All Intervention Non-Intervention Guatemala Literacy of Head of Household 2006 2011 52 Table 12. DID analysis of the literacy indicator for the head of household Literacy All ENCOVI data N = 26314 Matched analysis sample N = 3492 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant 4.46 16.04 *** 3.63 6.99 *** 1 Persons by household 0.01 0.68 0.01 0.24 1 Age of head of household –0.07 –6.61 *** –0.08 –3.74 *** 1 Age of head of household squared 0.00 2.04 ** 0.00 1.01 1 Female head of household –0.79 –12.59 –0.93 –4.16 1 Number of children ages 5–17 –0.22 –7.59 *** –0.13 –1.98 ** 1 Number of children ages 5 and younger –0.27 –7.85 *** –0.23 –3.70 *** 1 Number of females of reproductive age (12–49) 0.13 4.37 *** 0.13 2.05 ** 1 Intervention –0.50 –2.99 *** –0.03 –0.15 1 Year 2011 0.03 0.41 0.05 0.30 1 Intervention * year 2011 0.10 0.68 0.34 1.56 2 Constant 4.44 16.03 *** 3.59 6.94 *** 2 Persons by household 0.01 0.69 0.01 0.24 2 Age of head of household –0.07 –6.63 *** –0.08 –3.74 *** 2 Age of head of household squared 0.00 2.02 ** 0.00 1.02 2 Female head of household –0.79 –12.58 –0.93 –4.11 2 Number of children ages 5–17 –0.22 –7.53 *** –0.13 –1.95 * 2 Number of children ages 5 and younger –0.27 –7.70 *** –0.23 –3.76 *** 2 Number of females of reproductive age (12–49) 0.13 4.43 *** 0.14 2.18 ** 2 agexport –0.57 –2.56 ** –0.25 –1.05 2 agexport * year 2011 0.13 0.69 0.54 2.64 *** 2 fundacion agil –0.04 –0.24 0.30 1.25 2 fundacion agil * year 2011 0.10 0.69 –0.18 –0.93 2 anacafe –0.26 –1.37 0.03 0.14 2 anacafe * year 2011 –0.06 –0.35 0.51 1.08 2 mercycorp –0.06 –0.31 0.42 1.34 2 mercycorp * year 2011 –0.25 –1.20 –0.56 –2.50 ** 2 Year 2011 0.05 0.69 0.09 0.58 3 Constant 4.45 16.09 *** 3.62 6.90 *** 3 Persons by household 0.02 0.73 0.01 0.27 3 Age of head of household –0.07 –6.61 *** –0.08 –3.68 *** 3 Age of head of household squared 0.00 1.99 ** 0.00 0.96 3 Female head of household –0.79 –12.63 –0.92 –4.12 3 Number of children ages 5–17 –0.22 –7.67 *** –0.13 –1.97 ** 3 Number of children ages 5 and younger –0.27 –7.74 *** –0.23 –3.54 *** 3 Number of females of reproductive age (12–49) 0.13 4.35 *** 0.13 2.03 ** 3 1 intervention –0.49 –2.38 ** –0.16 –0.65 3 1 intervention * year 2011 0.21 1.19 0.44 1.84 * 3 2 interventions –0.58 –2.33 ** –0.03 –0.11 3 2 interventions * year 2011 0.23 1.43 0.34 1.31 3 3 interventions –0.41 –1.43 0.66 2.06 ** 3 3 interventions * year 2011 –0.40 –2.22 ** –0.20 –0.68 3 Year 2011 0.03 0.42 0.05 0.30 Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 53 Percentage of School-Age (5–17) Females Attending School Figure 8 summarizes per-household average female school attendance. All samples show attendance increasing over time, an increase that appears to be larger for girls in intervention areas that were selected into the matched sample. This last outcome is encouraging, as this group has the lowest initial female school attendance rates per household. Table 13 depicts the outcome of the DID analysis regarding the percentage of female children of school age (5–17) within each household that is attending school. There are positive intervention effects, but these are not statistically significant. In terms of the individual implementing organizations, evidence suggests that Fundación AGIL's interventions have had a negative effect, while the interventions of the other three organizations have been positive, although again these are usually not statistically significant effects. Some of these effects are large in size, with, for example, ANACAFE's statistically significant result indicating that its intervention raised school attendance by 22 percent. For this outcome variable, the strongest results on numbers of interventions are for three interventions being positive. Figure 8. Percentage of school-age females (5–17) attending school, 2006–2011 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% Matched sample All Matched sample All All Intervention Non-Intervention Guatemala % Girls Attending School 2006 2011 54 Table 13. DID analysis of the percentage of school-age females actually attending school All ENCOVI data N = 13683 Matched analysis sample N = 1986 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant 0.34 4.48 *** 0.56 4.41 *** 1 Persons by household 0.02 3.77 *** 0.02 1.39 1 Age of head of household 0.02 5.11 *** 0.00 0.66 1 Age of head of household squared –0.00 –5.98 *** –0.00 –1.08 1 Female head of household 0.04 2.66 *** 0.07 1.94 * 1 Number of children ages 5–17 –0.00 –0.16 0.02 1.46 1 Number of children ages 5 and younger –0.11 –15.73 *** –0.09 –6.20 *** 1 Number of females of reproductive age (12–49) –0.01 –2.27 ** –0.03 –3.08 *** 1 Intervention –0.03 –1.29 –0.06 –1.15 1 Year 2011 0.03 2.18 ** 0.05 1.24 1 Intervention * year 2011 0.01 0.57 0.02 0.34 2 Constant 0.35 4.60 *** 0.54 4.15 *** 2 Persons by household 0.02 3.78 *** 0.02 1.50 2 Age of head of household 0.02 5.07 *** 0.00 0.69 2 Age of head of household squared –0.00 –5.94 *** –0.00 –1.08 2 Female head of household 0.04 2.64 *** 0.07 1.99 * 2 Number of children ages 5–17 –0.00 –0.12 0.02 1.48 2 Number of children ages 5 and younger –0.11 –15.85 *** –0.09 –6.52 *** 2 Number of females of reproductive age (12–49) –0.01 –2.26 ** –0.04 –3.05 *** 2 agexport –0.03 –1.22 –0.03 –0.50 2 agexport * year 2011 0.02 0.97 0.01 0.27 2 fundacion agil –0.01 –0.40 –0.00 –0.06 2 fundacion agil * year 2011 –0.05 –2.05 ** –0.07 –1.30 2 anacafe –0.06 –1.29 –0.10 –1.73 * 2 anacafe * year 2011 0.06 1.10 0.22 4.03 *** 2 mercycorp 0.03 0.59 –0.01 –0.09 2 mercycorp * year 2011 0.08 1.72 * 0.12 1.33 2 Year 2011 0.03 2.21 ** 0.06 1.59 3 Constant 0.35 4.60 *** 0.55 4.31 *** 3 Persons by household 0.02 3.83 *** 0.02 1.49 3 Age of head of household 0.02 5.07 *** 0.00 0.71 3 Age of head of household squared –0.00 –5.94 *** –0.00 –1.11 3 Female head of household 0.04 2.64 *** 0.07 1.98 * 3 Number of children ages 5–17 –0.00 –0.17 0.02 1.46 3 Number of children ages 5 and younger –0.11 –15.95 *** –0.09 –6.13 *** 3 Number of females of reproductive age (12–49) –0.01 –2.25 ** –0.04 –2.99 *** 3 1 intervention –0.01 –0.42 –0.07 –1.32 3 1 intervention * year 2011 0.00 0.05 0.05 0.88 3 2 interventions –0.06 –2.13 ** –0.03) –0.51 3 2 interventions * year 2011 –0.02 –0.44 –0.05) –0.90 3 3 interventions –0.03 –0.59 –0.02) –0.26 3 3 interventions * year 2011 0.10 1.62 0.08 0.54 3 Year 2011 0.03 2.18 ** 0.05 1.25 Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 55 Awareness of Birth Control Methods Figure 9 summarizes the percentage of females of reproductive age (12–49) in each household who are aware of birth control methods. There is a very large increase in the birth control awareness in 2011 with respect to 2006. However, in table 14, specification 1 indicates that, if anything, interventions have had a negative effect on overall awareness, although statistically these results are weak for the smaller matched sample. There are highly significant results indicating that Mercy Corps has had a positive effect on this outcome variable and that AGEXPORT has had a negative effect. (The results for the other two organizations are ambiguous.) Figure 9. Percentage of reproductive-age females (12–49) who have heard of birth control methods, 2006–2011 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% Matched sample All Matched sample All All Intervention Non-Intervention Guatemala % Awareness of Birth Control Methods 2006 2011 56 Table 14. DID analysis of percentage of reproductive-age females who have heard of birth control methods All ENCOVI Data N = 22646 Matched Analysis Sample N = 3047 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant 0.23 5.17 *** 0.20 2.71 *** 1 Persons by household –0.00 –1.28 0.00 0.66 1 Age of head of household –0.00 –0.93 –0.00 –0.85 1 Age of head of household squared –0.00 –0.66 0.00 0.01 1 Female head of household 0.28 23.75 *** 0.22 8.28 *** 1 Number of children ages 5–17 –0.01 –2.55 ** –0.01 –0.94 1 Number of children aged 5 and younger –0.01 –2.28 ** –0.01 –1.66 1 Number of females of reproductive age (12–49) –0.02 –5.05 *** –0.03 –2.90 *** 1 Intervention –0.01 –1.66 * –0.02 –1.62 1 Year 2011 0.33 21.45 *** 0.21 7.36 *** 1 Intervention * year 2011 –0.07 –3.30 *** –0.02 –0.49 2 Constant 0.23 5.19 *** 0.19 2.61 ** 2 Persons by household –0.00 –1.22 0.00 0.92 2 Age of head of household –0.00 –0.98 –0.00 –0.76 2 Age of head of household squared –0.00 –0.61 –0.00 –0.13 2 Female head of household 0.28 23.73 *** 0.22 8.46 *** 2 Number of children aged 5–17 –0.01 –2.56 ** –0.01 –1.07 2 Number of children ages 5 and younger –0.01 –2.29 ** –0.02 –1.97 * 2 Number of females of reproductive age (12–49) –0.02 –5.03 *** –0.03 –3.07 *** 2 agexport –0.00 –0.41 –0.00 –0.17 2 agexport * year 2011 –0.10 –4.08 *** –0.09 –2.27 ** 2 fundacion agil –0.01 –1.46 –0.02 –1.56 2 fundacion agil * year 2011 –0.01 –0.29 0.07 2.05 ** 2 anacafe –0.03 –3.32 *** –0.07 –4.15 *** 2 anacafe * year 2011 –0.02 –0.61 0.11 1.45 2 mercycorp 0.01 1.04 0.00 0.26 2 mercycorp * year 2011 0.03 0.89 0.03 0.53 2 Year 2011 0.33 22.03 *** 0.22 8.09 *** 3 Constant 0.23 5.17 *** 0.20 2.76 *** 3 Persons by household –0.00 –1.28 0.00 0.65 3 Age of head of household –0.00 –0.94 –0.00 –0.85 3 Age of head of household squared –0.00 –0.65 –0.00 –0.03 3 Female head of household 0.28 23.72 *** 0.22 8.28 *** 3 Number of children aged 5–17 –0.01 –2.54 ** –0.01 –0.93 3 Number of children ages 5 and younger –0.01 –2.28 ** –0.01 –1.67 3 Number of females of reproductive age (12–49) –0.02 –5.03 *** –0.03 –2.95 *** 3 1 intervention –0.00 –0.24 –0.02 –1.52 3 1 intervention * year 2011 –0.07 –2.66 *** –0.03 –0.63 3 2 interventions –0.02 –4.99 *** –0.03 –1.98 * 3 2 interventions * year 2011 –0.07 –2.35 ** 0.01 0.25 3 3 interventions –0.02 –2.36 ** –0.00 –0.18 3 3 interventions * year 2011 –0.06 –2.17 ** –0.06 –2.15 ** 3 Year 2011 0.33 21.45 *** 0.21 7.36 *** Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 57 Children Aged 5 and Younger Receiving the Pentavalent Vaccine Figure 10 summarizes the percentage of children aged 5 and younger in each household who have received the pentavalent vaccine. A huge increase in the prevalence of this vaccination is obvious. However, because the intervention households started at a low level, the fact that their 2011 levels roughly match those of the non-intervention households indicates an effect of intervention. This conclusion is backed up in regressions table 15. In specification 1, the intervention is statistically significant and positive for the all-household sample and positive for the matched sample. In this case, according to specification 3, the greater the number of interventions, the larger the effect; indeed, the three interventions are statistically significant for both samples. The effect of three interventions is numerically important also, raising vaccination rates by 12 percentage points in one sample's estimates and 16 percentage points in the other sample's estimates. For the individual organizations, the results are ambiguous for Fundación AGIL. But interventions by ANACAFE, Mercy Corps, and AGEXPORT show positive effects, and each organization's effect is statistically significant for one of the samples. Figure 10. Percentage of children aged 5 and younger who have received the pentavalent vaccine, 2006–2011 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Matched sample All Matched sample All All Intervention Non-Intervention Guatemala % Pentavalent Vaccine 2006 2011 58 Table 15. DID analysis of the percentage of children aged 5 and younger who have received the pentavalent vaccine % children younger than 5 vaccinated: Pentavalent All ENCOVI data N = 13210 Matched analysis sample N = 1943 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant 0.36 7.66 *** 0.31 2.91 *** 1 Persons by household 0.01 1.32 0.02 2.32 ** 1 Age of head of household 0.00 0.59 0.00 0.28 1 Age of head of household squared –0.00 –0.79 0.00 0.29 1 Female head of household 0.01 0.84 0.04 1.27 1 Number of children ages 5–17 –0.02 –5.15 *** –0.02 –3.26 *** 1 Number of children aged 5 and younger –0.02 –3.32 *** –0.02 –1.64 1 Number of females of reproductive age (12–49) 0.00 0.53 0.00 0.25 1 Intervention –0.08 –3.04 *** –0.08 –1.42 1 Year 2011 0.59 30.24 *** 0.60 12.71 *** 1 Intervention * year 2011 0.08 3.04 *** 0.06 0.95 2 Constant 0.36 7.63 *** 0.30 2.87 *** 2 Persons by household 0.01 1.31 0.02 2.10 ** 2 Age of head of household 0.00 0.55 –0.00 –0.28 2 Age of head of household squared –0.00 –0.75 0.00 0.30 2 Female head of household 0.01 0.77 0.04 1.21 2 Number of children ages 5–17 –0.02 –5.14 *** –0.02 –2.93 *** 2 Number of children ages 5 and younger –0.02 –3.32 *** –0.02 –1.76 * 2 Number of females of reproductive age (12–49) - 0.58 0.00 0.39 2 agexport –0.07 –2.50 ** –0.05 –0.83 2 agexport * year 2011 0.06 1.92 * 0.01 0.09 2 fundacion agil –0.00 –0.15 0.01 0.15 2 fundacion agil * year 2011 0.01 0.38 –0.01 –0.11 2 anacafe 0.01 0.29 –0.16 –2.62 ** 2 anacafe * year 2011 0.02 0.39 0.21 3.30 *** 2 mercycorp –0.08 –2.06 ** –0.07 –0.99 2 mercycorp * year 2011 0.07 1.88 * 0.09 1.10 2 Year 2011 0.59 31.12 *** 0.62 13.87 *** 3 Constant 0.36 7.63 *** 0.31 2.96 *** 3 Persons by household 0.01 1.29 0.02 2.25 ** 3 Age of head of household 0.00 0.59 –0.00 –0.31 3 Age of head of household squared –0.00 –0.79 0.00 0.35 3 Female head of household 0.01 0.81 0.04 1.08 3 Number of children ages 5–17 –0.02 –5.12 *** –0.02 –3.08 *** 3 Number of children aged 5 and younger –0.02 –3.28 *** –0.02 –1.57 3 Number of females of reproductive age (12–49) 0.00 0.54 0.00 0.19 3 1 intervention –0.09 –3.18 *** –0.11 –1.74 * 3 1 intervention * year 2011 0.09 3.23 *** 0.10 1.56 3 2 interventions –0.03 –0.60 –0.01 –0.11 3 2 interventions * year 2011 0.03 0.53 –0.04 –0.37 3 3 interventions –0.11 –2.82 *** –0.17 –2.68 *** 3 3 interventions * year 2011 0.12 3.26 *** 0.16 2.34 ** 3 Year 2011 0.59 30.23 *** 0.60 12.71 *** Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 59 Connection to an Electricity Distribution Grid Figure 11 summarizes the rates at which homes are connected to the electricity distribution grid. Interestingly, households in the treatment group have lower levels of connection in 2006 than do the control group, but during the intervention period, the percentage of the control group who are connected falls and that in the treatment group rises, eventually attaining a higher level than the control group. This observation is supported in the regressions reported in table 16. For the smaller matched sample, the effect of the intervention is large and statistically significant. For the individual organizations, the results are ambiguous for AGEXPORT and ANACAFE. The intervention is indicated to have a positive effect in the case of Fundación AGIL, but at weak levels of statistical significance. For Mercy Corps, the interventions have negative effects, and in the case of the larger sample, a statistically significant one. The effects of one intervention and of two interventions are positive, with one statistically significant result for one intervention. The effects of three interventions are negative but statistically insignificant. Figure 11. Living quarters connected to an electricity distribution grid, 2006–2011 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% Matched sample All Matched sample All All Intervention Non-Intervention Guatemala % Connected to Electricity 2006 2011 60 Table 16. DID analysis of the home’s connection to an electricity distribution grid Household connected to electricity distribution grid All ENCOVI data N=26466 Matched analysis sample N=3515 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant 0.68 2.69 *** 0.03 0.07 1 Persons by household 0.16 4.95 *** 0.17 3.63 *** 1 Age of head of household 0.04 4.39 *** 0.02 1.10 1 Age of head of household squared –0.00 –4.93 *** –0.00 –1.20 1 Female head of household 0.38 5.43 0.69 3.99 1 Number of children ages 5–17 –0.33 –9.19 *** –0.21 –4.00 *** 1 Number of children aged 5 and younger –0.52 –14.20 *** –0.54 –7.30 *** 1 Number of females of reproductive age (12–49) 0.17 4.90 *** 0.17 2.41 ** 1 Intervention –0.46 –2.02 ** –0.32 –0.81 1 Year 2011 –0.25 –2.57 ** –0.48 –1.60 1 Intervention * year 2011 –0.02 –0.09 0.85 2.25 ** 2 Constant 0.62 2.46 ** –0.22 –0.44 2 Persons by household 0.17 5.77 *** 0.16 3.21 *** 2 Age of head of household 0.04 4.39 *** 0.03 1.20 2 Age of head of household squared –0.00 –4.95 *** –0.00 –1.29 2 Female head of household 0.38 5.39 0.70 4.06 2 Number of children ages 5–17 –0.34 –10.80 *** –0.20 –3.34 *** 2 Number of children aged 5 and younger –0.52 –14.26 *** –0.55 –7.48 *** 2 Number of females of reproductive age (12–49) 0.18 4.95 *** 0.17 2.51 ** 2 agexport –0.50 –1.77 * –0.20 –0.41 2 agexport * year 2011 –0.09 –0.41 0.49 1.24 2 fundacion agil 0.23 0.71 0.36 0.74 2 fundacion agil * year 2011 0.53 1.88 * 0.35 0.83 2 anacafe –0.83 –2.41 ** –0.18 –0.20 2 anacafe * year 2011 –0.02 –0.05 0.26 0.34 2 mercycorp –0.29 –1.01 0.10 0.22 2 mercycorp * year 2011 –0.56 –1.96 ** –0.27 –0.46 2 Year 2011 –0.25 –2.59 *** –0.28 –0.96 3 Constant 0.62 2.40 ** –0.09 –0.19 3 Persons by household 0.16 5.50 *** 0.17 3.62 *** 3 Age of head of household 0.05 4.36 *** 0.03 1.33 3 Age of head of household squared –0.00 –4.88 *** –0.00 –1.39 3 Female head of household 0.38 5.43 0.71 4.03 3 Number of children ages 5–17 –0.34 –10.57 *** –0.20 –3.80 *** 3 Number of children aged 5 and younger –0.52 –14.30 *** –0.54 –6.89 *** 3 Number of females of reproductive age (12–49) 0.18 4.84 *** 0.15 2.20 ** 3 1 intervention –0.40 –1.42 –0.69 –1.69 * 3 1 intervention * year 2011 0.05 0.27 1.00 2.91 *** 3 2 interventions –0.15 –0.44 0.28 0.47 3 2 interventions * year 2011 0.23 0.73 0.45 0.74 3 3 interventions –1.06 –2.55 ** 0.26 0.69 3 3 interventions * year 2011 –0.41 –0.83 1.05 1.32 3 Year 2011 –0.25 –2.57 ** –0.48 –1.59 Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 61 Rate of Ownership of Living Quarters Figure 12, below, summarizes the rates at which homes are owned by their inhabitants in Guatemala. Ownership in the small-sample treatment group decreases from 2006 to 2011, but it decreases by more in the small-sample control group, indicating a positive effect of the intervention. This observation is supported in the regressions reported in table 17. For the smaller matched sample, the effect of the intervention is large and statistically significant, but for the larger sample, the effect of the intervention is negative, although statistically insignificant. For the individual organizations, the results are ambiguous for AGEXPORT and ANACAFE. The intervention is indicated to have a negative effect in the case of Fundación AGIL, with a high level of statistical significance in the case of the larger sample. For Mercy Corps, the interventions have positive effects, and in the case of the larger sample, a statistically significant one. The effects of one intervention and of two interventions are ambiguous. The effects of three interventions are negative but statistically insignificant. Figure 12. Rate of ownership of living quarters, 2006–2011 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Matched sample All Matched sample All All Intervention Non-Intervention Guatemala Ownership of Living Quarters 2006 2011 62 Table 17. DID analysis of ownership of the living quarters/home House ownership All ENCOVI data N=26469 Matched analysis sample N=3515 Specification Independent variable Coeff. T-stat. Stat. sign. Coeff. T-stat. Stat. sign. 1 Constant –2.37 –7.57 *** –1.47 –2.84 *** 1 Persons by household 0.22 7.80 *** 0.30 3.95 *** 1 Age of head of household 0.09 6.05 *** 0.13 5.61 *** 1 Age of head of household squared –0.00 –3.73 *** –0.00 –4.58 *** 1 Female head of household –0.21 –3.20 0.03 0.16 1 Number of children ages 5–17 –0.03 –1.18 –0.10 –1.64 1 Number of children aged 5 and younger –0.01 –0.16 0.15 –1.19 1 Number of females of reproductive age (12–49) –0.13 –3.95 *** 0.32 –3.82 *** 1 Intervention 0.77 3.73 *** 0.65 –1.64 1 Year 2011 –0.03 –0.34 0.87 –2.89 *** 1 Intervention * year 2011 –0.02 –0.12 0.74 2.03 ** 2 Constant –2.35 –7.54 *** –1.64 –3.52 *** 2 Persons by household 0.22 7.78 *** 0.30 3.98 *** 2 Age of head of household 0.09 6.04 *** 0.13 5.77 *** 2 Age of head of household squared –0.00 –3.71 *** –0.00 –4.65 *** 2 Female head of household –0.20 –3.18 0.02 0.10 2 Number of children ages 5–17 –0.03 –1.11 –0.10 –1.55 2 Number of children aged 5 and younger –0.01 –0.21 –0.16 –1.31 2 Number of females of reproductive age (12–49) –0.13 –4.01 *** –0.32 –3.86 *** 2 agexport 0.43 1.40 –0.77 –1.35 2 agexport * year 2011 0.09 0.43 0.82 2.08 ** 2 fundacion agil 0.70 3.87 *** 0.53 1.20 2 fundacion agil * year 2011 –0.38 –2.81 *** –0.31 –1.00 2 anacafe 0.52 1.54 –0.78 –1.20 2 anacafe * year 2011 –0.11 –0.39 0.27 0.24 2 mercycorp 0.05 0.19 –0.12 –0.28 2 mercycorp * year 2011 0.63 2.74 *** 0.23 0.47 2 Year 2011 –0.03 –0.36 –0.75 –2.78 *** 3 Constant –2.36 –7.56 *** –1.55 –3.03 *** 3 Persons by household 0.22 7.79 *** 0.30 4.03 *** 3 Age of head of household 0.09 6.07 *** 0.13 5.85 *** 3 Age of head of household squared –0.00 –3.73 *** –0.00 –4.71 *** 3 Female head of household –0.20 –3.16 0.04 0.19 3 Number of children ages 5–17 –0.0 –1.13 –0.10 –1.53 3 Number of children ages 5 and younger –0.01 –0.20 –0.14 –1.13 3 Number of females of reproductive age (12–49) –0.13 –3.96 *** –0.33 –3.69 *** 3 1 intervention 0.64 2.39 ** –0.85 –1.63 3 1 intervention * year 2011 –0.05 –0.25 0.73 1.75 * 3 2 interventions 1.02 4.45 *** –0.29 –0.87 3 2 interventions * year 2011 –0.21 –0.98 0.75 1.94 * 3 3 interventions 1.00 2.96 *** –0.43 –1.53 3 3 interventions * year 2011 0.33 1.35 0.47 1.19 3 Year 2011 –0.03 –0.34 –0.87 –2.88 *** Sources: Guatemala ENCOVI 2006 and Guatemala ENCOVI 2011 63 CONCLUSIONS AND RECOMMENDATIONS Difference-in-differences (DID) econometric analyses were performed utilizing the Encuestas de Condiciones de Vida (Surveys of Living Conditions; ENCOVI) data for three different regression specifications. Each of these three specifications was applied to eight different dependent variables, utilizing datasets on most municipalities in Guatemala and on the matched treatment and control groups of municipalities. The dependent variables utilized in the regression correspond to socioeconomic indicators of household welfare and thus are variables likely to be affected by such programs as the four interventions evaluated. Tables 10–17 present the comprehensive DID regression results for the sake of completeness. However, the most important conclusions would focus on specific results within those regressions—namely, those particular coefficients that convey whether the interventions have been successful. These coefficients reflect the effects of a specific type of intervention in 2011—for example, "Intervention * Year 2011" in specification 1, "agexport * Year 2011" and the three other similar coefficients in specification 2, and "1 intervention * year 2011" and the two other similar coefficients in specification 3. Tables 18–25 summarize the results of each of these coefficients as simply as possible. Each of those tables has the same structure. To understand that structure, focus on Table 18, which captures all the results from specification 1 on whether any type of intervention has had any effect. For each of the eight outcome variables, the table lists whether the effect of intervention is estimated to be positive or negative for the all-household data and the matched household data. For per-capita income, for example, the estimated effect is opposite in sign in the two samples, suggesting little consistency in the results between the two different datasets. The third and fifth columns of the tables give adjectives that summarize the statistical significance of the results: "strong," "weak," and "very poor." "Strong" indicates the coefficient is statistically significant at the 0.1 level. (This is the probability that the result could arise from chance if there were no effects.) "Weak" indicates statistical significance between the 0.2 and 0.1 levels. "Very poor" indicates statistical significance at levels weaker than 0.2. It is a matter of debate in the statistical literature as to which of these results should be taken into account when reaching overall conclusions. Some analysts would focus only on the "strong" results and disregard the rest. Some would take into account all the signs in columns in two and four of the tables, while attaching more importance to the strong results than the weak ones and attaching much less importance to the "very poor" ones. It is clear from table 18 that it is impossible to conclude that a typical intervention (independent of the implementing organization and number of interventions) has a consistent effect, either negative or positive. The signs of the effects vary too much between different outcome measures and between different samples analyzed. Some coefficients are statistically significant and positive, and others are statistically significant and negative. This is exactly the mixed picture that would be obtained if it were the case that the effect of the interventions varied between different types of social indicators and different samples. Effectively, the same story can be told concerning the results relevant to the interventions of Fundación de Apoyo a la Generación de Ingresos Locales (Foundation to Support Local Income Generation; Fundación AGIL) in table 20. For Asociación Guatemalteca de Exportadores (Guatemalan Association of Exporters; AGEXPORT) in table 19 and Asociación Nacional del Café (National Coffee Association; ANACAFE) in table 21, there is very weak composite evidence of a positive effect—but the emphasis here is the fact the evidence is truly very weak. For Mercy Corps, in table 22, one can be somewhat more optimistic; 12 of the 16 estimated coefficients have positive signs, and 3 of the 5 coefficients that have strong levels of statistical significance are positive. Moreover, this optimism would arise when perusing the results for the all-household sample and the matched-household sample, raising confidence in the generality of the results. Tables 23–25 focus on the outcomes when there are one, two, or three or more interventions in the municipality. Again, the results are not strong, but some indications can be gained by comparing the three tables. There is more evidence of positive results when only one intervention was conducted in a municipality than when two, three, or more interventions were conducted. Table 23 shows that 11 of the 16 estimated coefficients have positive signs, and 4 of the 7 coefficients that have strong levels of statistical significance are positive. Little in tables 24 and 25 justifies concluding that positive effects occur when two, three, or more interventions are conducted. Notably, program effects can vary significantly with the initial situation of beneficiaries, as indicated by differences in the results for the whole set of intervention municipalities and the results for the treatment group of municipalities. These differences are almost certainly a reflection of the fact that the matched treatment and control groups are a select set of municipalities—they are among the poorest municipalities in all of Guatemala. Turning to results on specific measures, on real per-capita income (or real household income), the treatment group of municipalities fared worse than the matched control group of municipalities, a result that is of some significance, because the treatment and control groups were some of poorest municipalities in Guatemala. On these same measures, there was no significant difference in trends between intervention and non-intervention municipalities. As noted previously, this should not necessarily be interpreted as a lack of impact of the interventions on individual producers. Rather, this might be interpreted as changes in the welfare of producers not being reflected in broader municipal level welfare. In contrast, the treatment municipalities fared better than the control municipalities on connections to an electricity grid and rate of ownership of living quarters. However, on these measures, there was no appreciable difference in trends between intervention and non￾intervention municipalities. The contrast between the results on income and those on electricity and home ownership indicate that programs can bring significant welfare benefits beyond income changes. On average, children ages 5 years and younger in the intervention municipalities had lower vaccination rates than those in non-intervention areas at the beginning of these United States Agency for International Development (USAID) programs. Although vaccination rates significantly 64 65 increased in the country as a whole, the intervention areas were able to close the gap with the rest of Guatemala over the period of the interventions. Literacy levels improved slightly more for households in the treatment group of municipalities compared to the matched control group. For those looking for a single take-away conclusion, it must be emphasized that it is difficult to draw one overall conclusion from the statistical analysis conducted for this evaluation. This difficulty is not surprising. The results of the interventions vary along three dimensions—by the two different samples analyzed (all-intervention analysis vs. matched municipalities), by the type of welfare indicator analyzed, and by the organization implementing the intervention. The four implementing organizations focused on different types of programs; the matched municipalities are not representative of the country as a whole; and the welfare indicators each capture very different aspects of household well-being. Thus, it is possible to pronounce the interventions a success in one area (e.g., increasing the percentage of children receiving the pentavalent vaccine in Guatemala as a whole) and not successful in another (e.g. increasing per-capita incomes for the poorest households). As this is a general characteristic of the results detailed in the main body of this report, readers are urged to examine those individual results very carefully. The major limitation placed on the methodology was that it was an ex-post one that had to rely on existing data sources. This meant, for example, that the implementation of the interventions was not designed so that there was balance—in the sense of similar characteristics—between the municipalities that received the intervention and the ones that did not. Moreover, no information was actually available on those households or businesses that were subject to the intervention and those that were not. Therefore, the crude proxy of being in an intervention municipality had to be used for all households and businesses in that municipality as an indicator of whether they had been subject to the intervention. Future implementations of projects and their evaluations should therefore aim to collect information on outcomes for those beneficiaries that are directly affected by interventions rather than having to rely on indirect effects, as in this evaluation. The most reliable way in which such an objective can be achieved is for the requirements of evaluation to be tightly integrated into the design and implementation of projects. Evaluations are likely to be less equivocal than this one and more capable of producing conclusions that can help in the design of future project activities if the demands of precise evaluation are taken into account from the very first step of project initiation. 66 Table 18. Summary of results for interventions (of all types) for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Weak ▬ Strong Log of household income + Very poor ▬ Weak Literacy indicator for the head of household + Very poor + Weak Percentage of school-age females actually attending school + Very poor + Very poor Percentage of reproductive-age females who have heard of birth control methods ▬ Strong ▬ Very poor Percentage of children ages 5 and younger who have received the pentavalent vaccine + Strong + Very poor Electricity distribution and grid connectivity ▬ Very poor + Strong Ownership of the living quarters/home ▬ Very poor + Strong Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 67 Table 19. Summary of results for the interventions by AGEXPORT for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Very poor ▬ Strong Log of household income + Very poor ▬ Very poor Literacy indicator for the head of household + Very poor + Strong Percentage of school-age females actually attending school + Very poor + Very poor Percentage of reproductive-age females who have heard of birth control methods ▬ Strong ▬ Strong Percentage of children ages 5 and younger who have received the pentavalent vaccine + Strong + Very poor Electricity distribution and grid connectivity ▬ Very poor + Strong Ownership of the living quarters/home + Very poor + Strong Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 68 Table 20. Summary of results for the interventions by Fundación AGIL for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Very poor ▬ Very poor Log of household income + Very poor + Very poor Literacy indicator for the head of household + Very poor ▬ Very poor Percentage of school-age females actually attending school ▬ Strong ▬ Weak Percentage of reproductive-age females who have heard of birth control methods ▬ Very poor + Strong Percentage of children ages 5 and younger who have received the pentavalent vaccine + Very poor ▬ Very poor Electricity distribution and grid connectivity + Strong + Very poor Ownership of the living quarters/home ▬ Strong ▬ Very poor Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 69 Table 21. Summary of results for the interventions by ANACAFE for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Very poor ▬ Strong Log of household income + Very poor ▬ Weak Literacy indicator for the head of household ▬ Very poor + Very poor Percentage of school-age females actually attending school + Very poor + Strong Percentage of reproductive-age females who have heard of birth control methods ▬ Very poor + Weak Percentage of children ages 5 and younger who have received the pentavalent vaccine + Very poor + Strong Electricity distribution and grid connectivity ▬ Very poor + Very poor Ownership of the living quarters/home ▬ Very poor + Very poor Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 70 Table 22. Summary of results for the interventions by Mercy Corps for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Very poor + Very poor Log of household income + Very poor + Very poor Literacy indicator for the head of household ▬ Very poor ▬ Strong Percentage of school-age females actually attending school + Strong + Weak Percentage of reproductive-age females who have heard of birth control methods + Very poor + Very poor Percentage of children ages 5 and younger who have received the pentavalent vaccine + Strong + Very poor Electricity distribution and grid connectivity ▬ Strong ▬ Very poor Ownership of the living quarters/home + Strong + Very poor Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 71 Table 23. Summary of results for relevant interventions when only one intervention took place, for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Very poor ▬ Strong Log of household income + Very poor ▬ Strong Literacy indicator for the head of household + Very poor + Strong Percentage of school-age females actually attending school + Very poor + Very poor Percentage of reproductive-age females who have heard of birth control methods ▬ Strong ▬ Very poor Percentage of children ages 5 and younger who have received the pentavalent vaccine + Strong + Weak Electricity distribution and grid connectivity + Very poor + Strong Ownership of the living quarters/home ▬ Very poor + Strong Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 72 Table 24. Summary of results relevant when two interventions took place, for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Weak ▬ Very poor Log of household income + Strong ▬ Very poor Literacy indicator for the head of household + Weak + Weak Percentage of school-age females actually attending school ▬ Very poor ▬ Very poor Percentage of reproductive-age females who have heard of birth control methods ▬ Strong + Very poor Percentage of children ages 5 and younger who have received the pentavalent vaccine + Very poor ▬ Very poor Electricity distribution and grid connectivity + Very poor + Very poor Ownership of the living quarters/home ▬ Very poor + Weak Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 73 Table 25. Summary of results relevant when three or more interventions took place, for all outcomes Outcome analyzed All household data Matched household data Direction of the effect Statistical strength of the effect Direction of the effect Statistical strength of the effect Log of per-capita income + Weak ▬ Strong Log of household income + Weak ▬ Weak Literacy indicator for the head of household ▬ Strong ▬ Very poor Percentage of school-age females actually attending school + Weak + Very poor Percentage of reproductive-age females who have heard of birth control methods ▬ Strong ▬ Strong Percentage of children ages 5 and younger who have received the pentavalent vaccine + Strong + Strong Electricity distribution and grid connectivity ▬ Very poor + Weak Ownership of the living quarters/home + Weak + Very poor Notes: 1. Under “statistical strength of the effect,” "strong" indicates the effect is statistically significant at the 0.1 level, "weak" indicates statistical significance between the 0.2 and 0.1 levels, and "very poor" indicates significance at levels weaker than 0.2. 2. The number of households in the all-household data varies from 13,022 (pentavalent) to 24,798 (literacy) and in the matched-household data from 1,943 (pentavalent) to 3,492 (literacy). 3. The results for the all-household data reflect, at a maximum, 71 intervention municipalities and 206 non-intervention municipalities. The results for the matched-household data reflect, at a maximum, 28 intervention municipalities and 25 non-intervention municipalities. 4. As a reminder: The all-household data cover all municipalities for which data are available, but intervention and non-intervention municipalities are not necessarily comparable; the matched￾household data contain intervention municipalities that are matched to non-intervention municipalities, but the matched data contain mostly poor municipalities. 74 Appendix A - REFERENCES 3ie. “Principles for Impact Evaluation.” Downloaded May 2012. www.ode.ausaid.gov.au/publications/pdf/3ieprinciplesforimpactevaluation.pdf. Gertler, Paul J.; Martinez, Sebastian; Premand, Patrick; Rawlings, Laura B.; Vermeersch, Christel M. J.. 2011. “Impact Evaluation in Practice.” World Bank. Downloaded May 2012. https://openknowledge.worldbank.org/handle/10986/2550 Guatemala Instituto Nacional de Estadística (INE ). “Encuesta Nacional de Condiciones de Vida (ENCOVI).” http://www.ine.gob.gt/np/encovi/index.htm. Guatemala Instituto Nacional de Estadística (INE ). “Mi Familia Progresa (MIFAPRO).” 2011. http://www.ine.gob.gt/np/mifapro/index.htm. Guatemala Instituto Nacional de Estadística (INE ). “XI Censo Nacional de Población y VI de Habitación.” 2003. http://www.ine.gob.gt/np/poblacion/etnias%20por%20departamento.xls. LIS Data Center. “National Survey of Living Conditions / Encuesta Nacional de Condiciones de Vida (ENCOVI).” 2006. http://www.lisdatacenter.org/our-data/lis-database/by-country/guatemala-2/. NASA Socioeconomic Data and Applications Center. “Small Area Estimates Poverty and Inequality Data Set.” Retrieved 2012. http://sedac.ciesin.columbia.edu/data/collection/povmap/methods/methods4. USAID/Guatemala. “Economic Freedom: Open, Diversified, and Expanding Economies.” 2004. World Bank. “OED and Impact Evaluation-A Discussion Note,” Downloaded November 2011. http://www.worldbank.org/ieg/docs/world_bank_oed_impact_evaluations.pdf. 75 Appendix B – Statements of Work USAID/LAC/RSD Agricultural Value Chain Impact Analyses Scope of Work (Guatemala & Honduras) Purpose: USAID has a long and extensive history of agricultural value chain programs in Central America, prior to the current USG Feed the Future (FTF) initiative. Focusing exclusively on USAID agriculture programs prior to 2010 in Guatemala and Honduras, LAC/RSD/BBEG intends to conduct an analysis of relative economic improvements between municipalities that participated in USAID’s agricultural interventions, and those municipalities that have not. Background Information: In April 2012, LAC/RSD/BBEG contracted Optimal Solutions Group, under USAID’s LEAP contract mechanism to undertake a feasibility study in Guatemala to assess whether the aforementioned impact evaluation was possible in Guatemala given the data available to the evaluators. Their findings, given data limitations, conclude a non-experimental correlation analysis is feasible, which would provide linkages between USAID agricultural programs and economic improvements at the municipal level within the statistical constraints of this archival investigation. LAC/RSD/BBEG plans to use the same LEAP mechanism to implement a similar feasibility study in Honduras, to see if a similar correlation analysis can be conducted in Honduras. In addition, this scope will also cover completion of the correlation analysis for Guatemala and Honduras depending upon the results of Honduras’ feasibility study (see Figure 1). For this historical research analysis it is necessary to rely on data already collected by the national statistical office through household surveys and census datasets, as too much time has passed since the activities were implemented to statistically attribute current conditions at the household level to historical USAID programs. However, statistical constraints in the availability or quality of these data may prove prohibitive to conducting an impact evaluation. A chief concern is the granularity of the data that is available from household surveys, since the data was only reported at a municipal level and USAID activities work at the community or association level. Another concern is the timing of the surveys themselves, so they can provide appropriate baseline and endline data points. Drawing from the lessons learned from the feasibility study undertaken in Guatemala, the following is a description of how the Honduras feasibility (stage 1) and full correlation analysis/analyses (stage 2) are envisioned. 76 Stage 1: A brief feasibility study will be undertaken in Honduras to determine whether national survey datasets can be used for an analysis of relative economic improvements between municipalities that participated in USAID’s agricultural interventions, and those municipalities that have not. o Sufficient Data Found: If the feasibility study indicates sufficient attribution is possible in Honduras, USAID plans to undertake an impact evaluation or correlation analysis in both Honduras and Guatemala. o Insufficient Data Found: If the feasibility study indicates sufficient attribution is not possible in Honduras, the larger impact assessment will not proceed in Honduras. Stage 2: If the feasibility studies indicate sufficient attribution is possible in Honduras, the larger impact assessment will proceed in both countries. If the feasibility study indicates sufficient attribution is not found in Honduras, the larger impact assessment will proceed in Guatemala only. Figure 1: Decision Tree for Agricultural Value Chain Feasibility and Correlation Analyses completed (3) Both Guatemala & Honduras corre￾lation analyses completed (2) Honduras Correlation Analysis Feasibility Study No further work on Honduras; Guate correlation analysis (1) Guatemala Correlation Analysis Feasibility Study No further analysis performed, lessons learned document. STAGE 1: Feasibility Study in USAID/Honduras A. Purpose: To assess the feasibility of conducting a full impact analysis or correlation analysis of USAID agricultural support interventions on household outcomes in Honduras prior to 2010. B. Audience and Intended Uses: The audience for this feasibility study will be the USAID LAC/RSD/BBEG Team and USAID/Honduras Mission. At a minimum, the feasibility study will reveal the degree of coverage of USAID programs at the municipal level. Beyond that, it is hoped that the feasibility study will conclude a non-experimental correlation analysis is possible as was the case in Guatemala. If so, Stage 2 will be concluding the correlation analyses as a multi￾country study. C. Analysis Design and Methodology: 1) Relying on USAID/Honduras Mission documents and staff insights, identify municipalities where within the country USAID agricultural value chain programs were active and over what time period prior to 2010. Estimate how many smallholder farmers in those municipalities participated in or received the benefits of the USAID agriculture programs. 2) The researcher(s) will need to acquire household survey datasets from the National Statistical Institute and mine the data. The researcher(s) will be responsible for providing the required computing equipment and statistical software to complete this study. To obtain data comparable to the ENCOVI data that were used in the Guatemala feasibility study, the researcher(s) might start with the Encuesta Permanente de Hogares de Propósitos Múltiples (EPHPM), which appears to provide household survey data, including income, for the years 1991 to 2010. In addition, the researcher(s) should look at EPHPM and/or other data that most closely matches that found in Guatemala to allow for comparisons between the Guatemala and Honduras focus countries. Ideally, the analysis would use data from 2006 and 2011. However, it is left to the researcher’s discretion to determine which data are best used given the timeframe for USAID/Honduras’ agricultural programs, in order to have adequate pre￾and post-intervention data. 3) Determine whether - and in how many municipalities, per USAID program - household level changes at the municipal level can be attributed, or correlated, to USAID program outcomes, using benchmarks of degrees of confidence indicated by the researcher(s). 4) Arrive at a recommendation of whether it is possible to conduct a full impact analysis or non-experimental correlation analysis of any or all of the individual USAID value chain program(s) in Honduras, given data limitations. This recommendation should take into account any alternate data sources identified during the course of the feasibility study. 77 78 The table below from the Guatemala feasibility study summarizes the comparisons described above. Feasibility Study Tabulations for Pre-2010 USAID Agriculture Programs in Guatemala Intervention Assisted Municipalities Data available from both ENCOVIs Number of Municipalities Covered by Implementing Partners Number of Survey Observations Total AGEXPORT Fundación AGIL ANACAFE Mercy Corp 2006 ENCOVI 2011 ENCOVI Yes Yes 71 40 38 10 10 3,528 3,504 No1 9 4 3 2 2 48 31 No Yes 191 0 0 0 0 8,995 8,923 No1 63 0 0 0 0 334 262 Total 334 44 41 12 12 12,905 12,720 ENCOVI = national household survey D. Deliverables: A written summary of findings, of no more than four pages excluding annexes, will document the feasibility of conducting a full impact analysis in Honduras. Information on the density of coverage of USAID programming at the municipal level is of particular interest. The researcher(s) should also include in the summary of findings any alternative suggestions on how to move forward with an impact or performance analysis that have not been considered to date. E. Period of Performance: The necessary field work would be carried out over a maximum of five days in Tegucigalpa, Honduras in close coordination with USAID/Honduras; travel must be approved by USAID/Honduras in order to ensure that appropriate parties will be available to meet with the researcher(s). 79 Delivery of the draft report for the Honduras feasibility study to LAC/RSD should be no later than 30 days after conclusion of the field work. F. Personnel: The implementing partner should propose a team including, at a minimum: • A project manager with background in monitoring and evaluation • A professional analyst with advanced skills in statistical analysis, including facility with STATA analytical software (or other statistical software deemed necessary to complete the above mentioned task) The researcher(s) will need Spanish and English language capabilities to understand the fields of the household survey or census datasets and to interact with USAID/Honduras mission staff and national statistical office staff. The written summary of findings will be presented in English. G. Administrative Support and Logistics: The research team is expected to provide its own administrative support and logistical support under this task order. For the anticipated field visit, limited administrative support will be provided by USAID/Honduras (i.e., country clearance). USAID/Honduras staff shall be invited to attend any and all meetings conducted during a field visit; the research team is not, however, responsible for their participation (e.g., through provision of transportation). As in the case of the Guatemala feasibility study, LAC/RSD will accompany the contractor to provide escort and help sift through program documentation in the USAID/Honduras mission. STAGE 2: Completion of the Correlation Analyses for Guatemala and (if feasible) Honduras A. Purpose: To conduct a full impact analysis or correlation analysis of USAID agricultural support interventions on household outcomes in Guatemala and possibly Honduras prior to 2010. B. Audience and Intended Uses: The audience for the correlation analysis/analyses will be the USAID LAC/RSD/BBEG Team and the USAID/Guatemala and USAID/Honduras Missions. C. Analysis Design and Methodology: The chosen methodology of analysis is the utilization of the household survey data to build a treatment and non-treatment comparison of municipality groups. The team would identify these groups by locating municipalities where smallholder farmers have been impacted by the interventions. The groups can then be analyzed for differences and similarities over 80 time. The advantage of this methodology is that the team would not need to match variables between two or three different sources of data nor impute income values, thereby reducing the amount of error and uncertainty. A third and earlier point-in-time could be utilized to validate the comparisons and describe area growth trajectories over time. In the case of Guatemala, this pre-baseline data can be obtained from the 2000 ENCOVI, if available, or the 2002 Population Census. It is recognized that this methodology cannot account for the effects of other interventions or factors that would have taken place during the same time as the interventions for this analysis and that may have affected the economic outcomes of both the treatment and the control groups. Once the control and treatment groups are determined, as recommended by the LEAP team in their Guatemala feasibility study, a non-experimental impact evaluation approach that uses household survey data across multiple years in the form of a cross-sectional analysis can be applied. In this case, household level data would be used as units of observation, and would compare the general economic situation of those located in areas where interventions occurred across time with those located in similar areas that did not receive the interventions. D. Deliverables: A written summary of findings, of no more than ten pages excluding annexes, will document the full impact analysis or correlation analyses in Guatemala and possibly Honduras. E. Period of Performance: No additional travel is anticipated for the conclusion of the impact analysis or correlation analyses for Guatemala and Honduras. Delivery of the draft report of the Stage 2 Analysis to LAC/RSD should be no later than 90 days after the Stage 2 of the analysis begins. F. Personnel: The implementing partner should propose a team including, at a minimum: • A project manager with background in monitoring and evaluation • A professional analyst with advanced skills in statistical analysis, including facility with STATA analytical software (or other statistical software deemed necessary to complete the above mentioned task) 81 USAID/LAC Agricultural Value Chain Impact Evaluation Scope of Work Background Information: USAID has a long and extensive history of agricultural value chain programs in Central America, prior to the current USG Feed the Future (FTF) initiative. USAID LAC/RSD/BBEG intends to conduct an impact evaluation of agriculture value chain programs on small farmers targeted under those programs prior to 2010. Specifically, LAC/RSD/BBEG plans to use the already procured LEAP mechanism to evaluate the economic impact of USAID agricultural interventions on small farmers at the household level in Guatemala (and possibly Honduras). It is necessary to rely on historical data already collected through household surveys and census datasets for this analysis, as too much time has passed since the activities were implemented to statistically attribute current conditions at the household level to historical USAID programs. However, statistical constraints may exist to performing an impact evaluation as envisioned. A chief concern is the granularity of the data that is available from household surveys, since the data was only reported at a municipal level and USAID activities work at the community or association level. Given this concern, the BBEG team is taking multi-faceted approach to the evaluation. (See the attached LAC Central American Value Chain Assessment Decision Tree) Note: This scope of work is only intended to cover the first feasibility study (Stage 1) to be undertaken in Guatemala City, Guatemala. Subsequent scopes of work will be drafted depending on the outcome of the first feasibility study. • Stage 1: A brief feasibility study will be undertaken in Guatemala to determine whether USAID activities reached enough farmers in enough municipalities to attribute changes in smallholder outcomes at the municipal level to USAID programs. Effectively, USAID wishes to know whether the number of households directly impacted in communities or associations targeted is of sufficient size that a random sample taken from a municipal-level dataset might be representative of the USAID beneficiary population. o Positive Attribution Found: If the feasibility study indicates sufficient attribution is possible in Guatemala, a similar feasibility study will be undertaken in Honduras to determine whether USAID activities reached enough farmers in enough municipalities to attribute changes in smallholder outcomes at the municipal level to USAID programs. o No Positive Attribution Found: If the feasibility study indicates sufficient attribution is not possible in Guatemala, the larger impact assessment will not proceed in either Guatemala or Honduras. Instead USAID Washington and USAID Guatemala in conjunction with Optimal Solutions Group, LLC will identify alternative impact evaluations that can be conducted. • Stage 2: If the feasibility studies indicate sufficient attribution is possible in Guatemala and Honduras, the larger impact assessment will proceed in both countries. If the feasibility study indicates sufficient attribution is found in Guatemala, but is not in Honduras, the larger impact assessment will proceed in Guatemala only. This will require identification of municipalities where USAID programs had the greatest concentration or saturation and comparison municipalities, and where USAID programs were not present, to serve as a post￾facto comparison population. The research team will then proceed with analysis including counterfactual outcomes relying heavily on national survey and census datasets. Stage 1: Feasibility Study in USAID/Guatemala Evaluation Rationale A. Purpose: To assess the feasibility of conducting a full impact or performance evaluation of USAID agricultural support interventions on household outcomes in Guatemala prior to 2010. B. Audience and Intended Uses: The audience for this feasibility study will be the USAID LAC/RSD/BBEG Team and USAID/Guatemala Mission. At a minimum, the feasibility study will reveal the degree of coverage (saturation, or “Swiss cheese” effect) of USAID programs at the municipal level. Having a better understanding of coverage levels of historic or completed agricultural programs will be useful as Feed the Future programs move forward with their current emphasis on focusing and concentrating efforts. Evaluation Design and Methodology: 1) Relying on USAID/Guatemala mission documents and staff insights, identify municipalities where USAID agricultural value chain programs were active and over what time period. Estimate how many smallholder farmers in those municipalities participated in the USAID programs. 2) Using national government survey and census data (or other rigorously collected and reliable datasets, as appropriate), estimate how many smallholder farmers in total were in each of these municipalities. The researcher(s) will need to acquire household survey datasets from the National Statistical Institute (http://www.ine.gob.gt/np/productos/index.htm) and mine the data through the appropriate statistical package. The researcher(s) will be responsible to provide the required computing equipment and statistical software to complete this study. 3) Determine whether - and in how many municipalities, per USAID program - household level changes among small famers at the municipal level can be attributed to USAID program outcomes, using benchmarks of degrees of confidence indicated by the researcher(s). 4) Arrive at a recommendation of whether it is possible to conduct a full impact evaluation of any individual USAID value chain program(s) in Guatemala, given data limitations. This recommendation should take into account any alternate data sources identified during the course of the feasibility study. 82 Deliverables: 83 A written summary of findings, of no more than four pages excluding annexes, will document the feasibility of conducting a full impact evaluation in Guatemala. Information on the density of coverage of USAID programming at the municipal level is of particular interest. The researcher(s) should also include in the summary of findings any alternative suggestions on how to move forward with an impact or performance evaluation that have not been considered to date. Period of Performance: This feasibility study should be completed no later than April 1, 2012. The necessary field work would be carried out over five days in Guatemala City, Guatemala in close coordination with USAID/Guatemala; travel must be approved by USAID/Guatemala in order to ensure that appropriate parties will be available to meet with the researcher(s). At least one day of the field work, if not more of that time, would be spent working with USAID/Guatemala staff to more precisely identify the geographic coverage of pre-2010 agricultural programs. Proposed Budget and Work Plan: The research team is required to submit a proposed budget and work plan outlining the funding, time, and personnel necessary to achieve the above-listed objectives and provide the required deliverables. It is recommended to include the following steps into its proposed work plan: • Obtain information regarding project design and relevant background information to establish parameters and determine key data that is needed. • Undertake necessary data collection. • Undertake field visit, including meetings with USAID/Guatemala and external parties. • Deliver final report. The technical work plan should address all anticipated tasks, and be no more than 3 typed pages in length. The budget proposal should include line-item allowances for adequate staff time to complete all anticipated tasks. An estimated, illustrative budget for this task follows: Line Item Estimated Cost Labor: Expert in monitoring and evaluation (45 days @ $XXX.XX/day) $XXX.XX Fringe Benefits (28.91%) $XXX.XX Labor Sub-Total $XXX.XX Overhead (44.03%) $XXX.XX 84 Total Labor Cost $XXX.XX Transportation (U.S. – Guatemala) $XXX.XX Local Transportation (10 days @ $XXX.XX/day) $XXX.XX Per Diem (10 days @ $XXX.XX/day) $XXX.XX Total Transportation Cost $$XXX.XX Subcontract/Material Handling (0.88%) $XXX.XX G&A (23.91%) $XXX.XX Total Cost $XXX.XX The budget proposal should be based on the appropriate daily rates of the monitoring and evaluation expert and/or statistician. The daily rate for may be more or less than the rate specified above, based on qualifications and in line with U.S. Government regulations. Personnel: The implementing partner should propose a team including, at a minimum: • An expert in monitoring and evaluation • A professional analyst with advanced skills in statistical analysis, including facility with STATA analytical software (or other statistical software deemed necessary to complete the above mentioned task) The researcher(s) will need Spanish and English language capabilities to understand the fields of the household survey datasets and to interact with USAID/Guatemala mission staff. The written summary of findings will be presented in English. Administrative Support and Logistics: The research team is expected to provide its own administrative support and logistical support under this task order. In the event that a field visit is deemed necessary, limited administrative support will be provided by USAID/Guatemala (i.e., country clearance). USAID/Guatemala staff should be invited to attend any and all meetings conducted during a field visit; the research team is not, however, responsible for their participation (e.g., through provision of transportation). 85 86 Appendix C - USAID Agricultural Interventions Overview This appendix details the four interventions that the feasibility study, and potential evaluation would focus on. The information gathered was compiled upon completion of review of program files, primarily contracts, quarterly and annual reports, as well as work plans of these programs, with the permission of USAID/ Guatemala Mission. No personal identifiable information was collected during this study. Summary: USAID Agricultural Interventions Prior to 2010 USAID/ Guatemala, through its partners, implemented four projects related to the agricultural sector intended to assist smallholder farmers in Guatemala in improving their farming practices, and provide access to national, regional, and international markets, with the ultimate goal of increasing employment at the local level, and making the country more competitive on the international market. The four projects were initiated under Strategic Objective 2 of USAID/Guatemala’s Country Plan for 2004-2009 – Economic Freedom: Open, Diversified and Expanding Economies. The interventions consisted of a variety of activities including capacity building, technical assistance, financial incentives, and trainings to the beneficiaries. Even though all four interventions shared the same general goal, each had different specific goals and objectives. Intervention Name Program Implementer Duration of Program Increased Rural Household Income and Food Security Asociación Guatemaltecade Exportadores (AGEXPORT, formerly AgExpront) 2004 to present Competitive Enterprises Asociación Nacional del Café (ANACAFE) 2006 to present Quality Assurance and Small Business Development Fundación de Apoyo a la Generación de Ingresos Locales (AGIL) 2006 to 2011 Innovation Market Alliance for Rural Entrepreneurs (IMARE) Mercy Corps - Walmart 2007 to Present AGEXPORT AGEXPORT signed cooperative alliance with USAID to implement the Increased Rural Household Income and Food Security project. The initial contract was between October 2004 and December 2005. However several contract extensions were approved and the project is still underway. For this study, the review ends at 2010 information. This intervention was initiated to enhance the Agro industry; forestry; and tourism sectors, working with alliances to identify buyers and partners of the buyers. The implementing partners also had to contribute funds to the project. The project’s overall goal was to strengthen Guatemala’s competitiveness at the general (policy) level, and in key industry “clusters” thereby increasing income levels and ensuring that productive employment opportunities are appropriately remunerated – especially in rural areas. 87 AGEXPORT worked with organized groups to improve the ability and skills of small scale entrepreneurs to take advantage of national, regional and global market opportunities; provide technical assistance needed to produce high quality products that comply with international standards or buyer requirements either directly or through a local organization that works with producers in the field; improve technologies; develop new business ideas. The geographic focus was in the Western Highlands Departments of Quiche, Totonicapan, and San Marcos. A total of 1,700 families, mainly indigenous (850 per year) received this assistance. General project indicators included: 1. Changes in export composition by selected business clusters (sustainable tourism, high￾value horticultural and specialty coffee exports, and certified forest products or key products 2. Increase in Growth competitiveness Index Score (World Economic Forum measure) The project was initially conducted in two phases: • Phase 1 o Investment for Peace – increase private sector participation o Assistance to the Zonapaz (Departments of Huehuetenango, Quiche, Chimaltenango, Alta Verapaz, Baja Verapaz, and Peten • Phase 2 o Improve Guatemala’s market share and competitiveness to take better advantage of CAFTA, generate employment and investment activities. Activities include  Business Development Unit  Economic Business Development Centers  Business Alliance Unit Progression of the project was as follows: • Pre 2000 – Increased Rural Household Income and Food Security Project (investment for peace and business development unit) 53 alliances were sponsored in the program between 2003-2004 with activities including monitor of sales, assistance increasing jobs, and provision of technical assistance and capacity building. • October 2004 – Strategic Objective 2 Implementer (expanded on 2000 award) • 2005 – 30 supply chain alliances linking producers to national/ international buyers; work with 10,000 Guatemala producers. In FY2005, AGEXPORT promoted the creation of 30 supply chain alliances that linked Guatemalan producers in the agribusiness, forestry, tourism, and handicraft sectors directly or indirectly with international buyers. The result was that 10,000 new Guatemalan producers were incorporated into the export process and more than $15 million were generated in new export sales via efforts from the supply chain 88 alliance program. Visits to international trade fairs and continued support for alliances established in 2004. • 2006 – Hurricane Stan economic development assistance; increase access to markets; introduce new business/ technology ideas (specific to Hurricane Stan affected areas). This was partly a Government program aimed at assisting 1700 families. As a result of the Hurricane Stan that hit Guatemala in 2005, the government requested USAID’s assistance to finance the creation of productive value chains in Solala and San Marcos municipalities, that were most affected by the hurricane, with 60 percent of corn crops ruined; farm fields severely eroded; and farm-to-market roads blocked by landslides or washed away. This project was initiated in 2006 and implemented in two phases: • One year cooperative agreement formed by USAID with SEGEPLAN, MINECO, AGEXPORT (main implementer) • USAID, AGEXPORT Directors/ program coordinators to follow up and expand on activities executed under Phase 1 and will be closely coordinated by MINECo and SEGEPLAN This amendment would support a continuation of AGEXPORT’s supply chain activities and an expansion of such activities to support the government’s effort to recover the rural economy in areas most affected. In 2010, AGEXPORT expanded the creation of business alliances between small/ medium scale producers with national, regional, and international buyers to establish long term relationships as a means of enhancing poverty reduction. This change introduced high level technical assistance in the form of market intelligence, production of environment-friendly business, export of products, and business administration. Other activities included training on: • Food safety • Good agriculture and manufacturing practice certificates • Market intelligence • Trade promotion • Access to Mexican markets • Training for PYMES, foreign trade school Beneficiary selection criteria were identified in the program files at each stage of the project, but will not be detailed in this report. ANACAFE “Competitive Enterprises” Guatemala is the world 5th largest coffee exporter, and the third largest of washed Arabica coffee beans. Since 2001 until about 2005, there was a decline in coffee yields (about 25 percent), in 40 percent of the export volume and nearly 50 percent in employment generated by the industry. In 89 2002, coffee represented 1.5 percent of the GDP, and the contribution of coffee to exports declined from 49 percent in the period 1985-1989 to 9 percent in 2005. In addition, coffee corresponded to 31 percent of the jobs created in 2002, with more than 90,000 producers, out of which 68 percent were small producers. The Asociación Nacional del Café (ANACAFE)/USAID project “Competitive Enterprises” responds to the urgency brought upon by the falling coffee prices during the early 2000’s, along with the increasing supply and changes in consumer tastes and preferences. The initial phase of this project began on April 12, 2006 and ended on September 30, 2009. The purpose of this agreement was “to create a greater number of more competitive and market oriented small and medium rural enterprises and increase coffee sector competitiveness through activities all along the values chain from production to the final cup.” ANACAFE’s response to the early 2000’s situation was reflected in its Competitiveness Plan in 2005. This plan contained an extensive evaluation of past and present state of Guatemala’s coffee industry. It outlined the long term strategy for a stable and sustainable coffee sector. The main objectives from the project were: • Improve quality of life of small producers in rural Guatemala • Increase economic revenues of small producers, especially those of the indigenous population • Strengthen small-scale coffee producers • Provide technical assistance and technological innovation to coffee producer groups The project focused on three pillars of ANACAFE Competitiveness: • Sustainable Development • Income Diversification, and • Marketing The Sustainable Development component recognizes the role of small and medium enterprises (SMEs) in the transition towards a modern, competitive coffee sector. About half of the efforts refer to increasing the SMEs participation. The SMEs program is based on a total quality approach; this program also supports technology developments and application with a focus on the environment and enhanced biodiversity. The Income Diversification component focuses on tourism and forestry, as it complements the coffee production. An example is coffee tours. The Marketing component corresponds to aggressive domestic and international activities with the goal of increasing awareness and consumption of Guatemalan coffee. 90 Some 48 organizations (20 additional ones that received assistance as a result of the extension of the project in 2009) of small producers can apply agricultural technology to their plantations so as to improve productivity and sustainability. There were 28 organizations participating in this project during its first year; the additional 20 were incorporated during its second year. During the first three years of the project, 12 agronomy professionals worked with these organizations (four were added with the extension) to prepare, train, and guide them. Each of these professionals tended to four or five organizations. The expected results of this project were as follows: 1. Sustainable Development a. Improved production practices b. Post-harvest quality c. Enterprise strengthening d. Support for commercialization e. Research and technology transfer 2. Income Diversification a. Coffee tourism b. Forestry c. Hydro-power generation 3. Marketing a. Domestic consumption b. Coffee and tourists c. Coffee and health d. Prime / extra prime coffee promotion There were many different activities that aim to achieve each of these different expected results; and each of these would be used in a different geographic area. The geographical coverage ANACAFE activities corresponded to the seven coffee-producing regions in Guatemala: REGION DEPARTMENT • Region I Coatepeque, Quetzaltemango and San Marcos • Region II Mazatenango, Solola, Retalhuleu, Suchitepequez • Region III Chimaltenango, Escuintla, El Progreso, Guatemala, Sacatepequez • Region IV Barberena, Santa Rosa, Jalapa, Jutiapa • Region V Huehuetenango, El Quiche • Region VI Coban, Alta y Baja Verapaz • Region VII Zacapa, Chiquimula, Izabal 91 Mercy Corps – Walmart “Innovation Market Alliance for Rural Entrepreneurs” (IMARE) Project Supermarkets have transformed the food industry. This phenomenon has changed the way farmers produce, package and market their products. However, upgrading production, packaging and safety standards to respond to these changes are only possible for wealthier farmers with sufficient capital and land. In Guatemala, where smallholder farmers are traditional suppliers of produce, they also face the following market barriers: lack of access to agricultural credit and crop insurance, traditional technology, poor infrastructure, inadequate post-harvest management, lack of market information, and inadequate knowledge of new global food safety standards, among others. Smallholder farmers usually produce low yields of horticultural products that are inconsistent in quality and have dangerously poor sanitary profiles. Mercy Corps, leading a USAID Global Development Alliance in conjunction with Walmart and Fundación AGIL signed an agreement to create business opportunities and improve the quality of life of several producer groups in Guatemala. The two main objectives of the project were to: 1. Increase access to more profitable markets on a sustainable basis 2. Increase productivity and quality of life through better farm management, post-harvest and processing techniques. This project originally had a timeline of 36 months, and was concentrated in municipalities located in the Departments of: • Alta Verapaz • Baja Verapaz • Chimaltenango • Sololá • San Marcos This program was developed to provide direct benefits to about 600 families and indirect benefits to about 3,600 members living in households of direct beneficiaries. The program made beneficiary producer groups aware of the possibility of selling directly to Walmart, and not depending on intermediaries, which sets a great comparative advantage for them. The selection of beneficiary groups by municipality, department and time period was also available in the program files, but will not be detailed in this review. 92 Fundación AGIL “Quality Assurance and Small Business Development” This project was awarded initially with a period of performance between June 2006 and September 2009, but an extension of the program was granted subsequently. The main objectives of this program were to improve the quality of life of an additional 12,000 people in 60 new small producer organizations belonging to Guatemala’s rural production sector. Program performance was to be measured by: • Changes in export composition by selected business clusters (sustainable tourism, high￾value horticulture, and specialty coffee exports, and certified forest products) or key products, • Growth Competitiveness Index score (World Economic Forum) This agreement was expected to create additional and more productive business opportunities for SME farms and non-farms in rural Guatemala, directly resulting in: • New SMEs; • New partnerships and business alliances involving SMEs; • Expansion of existing SMEs into new, high value markets; • Introduction of new technologies for SMEs; • Stronger and more sustainable small and medium producer organizations, and • New and sustainable technical and business services for SMEs. The selection of the producer groups, which ultimately defined the beneficiaries, was to be conducted in coordination with local mayors, the Ministry of Agriculture, local supermarkets, and groups neighboring those already in “Alianza Agroindustrial y Artesanal Rural” (ALIAR). By 2008, 86 groups of small scale producers had been assisted and trained and more than $9 million had been generated in new sales. The purpose of this modification is to expand the Quality Assurance and Small Business Development Program to 60 new producer groups, using improved mechanisms that Fundación AGIL was developing under the Quality Insurance and Development of Small Companies (ACDPE) program. In addition to the new groups, Fundación AGIL was to provide follow-up services to 30 groups in order to complete the Global GAP certification process in October 2008. During the expansion period, a minimum of 15 additional organizations would become certified, bringing the total number of organizations to 60 (30 old and 30 new) maintained under their follow-on program. The three main groups participating were: 1. The original 36 groups in Fundación AGIL’s ALIAR program, 2. New producer groups, selected with input from local mayors, 93 3. Groups affected by Hurricane Stan New groups were selected from groups working with Walmart Central America and other supermarket chains, as well as international importers. The majority of groups should be located in the immediate Highland regions around Guatemala City with a few from the South Coast area and drier regions optimal for producing tomatoes, bell peppers and onions. Other groups were to be incorporated from the neighboring ALIAR groups from the original Highland areas; and others selected with input from Municipal Technical Units and the Ministry of Agriculture. A number of areas were identified including the plantain producing areas in lowland San Marcos (Ocos, La Blanca and Nica); Highland San Marcos regions (San Antonio Sacatepequez, San Pedro Sacatepequez, San Marcos); Highland horticultural-producing regions in Quetzaltenango (Zunil and Almolonga), Solola (Santa Clara La Laguna, San Juan La Laguna, Santiago, and Panajachel), and Chimaltenango (Tecpan, Patzun, San Jose Poaquil, and Comalapa). Through the selection process above, a total of 70 new groups were identified with 50 groups selected for assistance. Follow-up and support was to be provided to 50 percent of the groups considered for monitoring, especially in the area of marketing. Two years after the program’s execution, participation levels were as follows: • Assistance continues to be provided to 30 of the 50 groups selected in the first phase; • Follow-up and support will be provided to a minimum of 30 of the 60 new groups of small and medium agricultural producers which were identified during 2010. Appendix D – Location of USAID Agricultural Interventions The maps below provide a graphical representation of the location of the four interventions. As discussed in the report, this information will be key to identifying comparison groups for the evaluation. Additional analyses of these maps reviewing whether there are overlaps in implementation will be conducted during the second phase of the project. 94 Program Implementation at the Department Level 95 Program Implementation at the Municipality Level 96 Appendix E – Overview of Datasets The XI Censo Nacional de Población y VI de Habitación.11 Also known as the 2002 Census, it is a descriptive dataset collected between November and December of 2002 by the Guatemala Instituto Nacional de Estadística (INE). The objective of the 2002 census was to produce appropriate and reliable information regarding the living conditions of the Guatemalan Population (“Censos Nacionales XI de Población y VI de Habitación”, 2003). The INE had the goal of disseminating the general characteristics of the populations, households and private dwellings, at the national, department and municipality levels. The results of the census were officially disclosed on February of 2003 (“Censos Nacionales XI de Población y VI de Habitación”, 2003). This double census had the following scope: (“Censos Nacionales XI de Poblacion y VI de Habitacion”, 2003): • General population characteristics such as gender, age groups, population by geographic area, ethnic groups, schooling levels, etc. • Households’ living situation, private dwellings, households by ownership type of the dwelling, by access to clean water, etc. • Household characteristics, average number of members, number of member affected by disabilities, by form of garbage disposal, etc. • Dwelling characteristics, number of families per dwelling, number of persons living in each dwelling, type of dwelling, materials used in exterior walls of dwelling, etc. Household Census (at the Municipal level) - “Mi Familia Progresa (MIFAPRO)”12 The Guatemala INE developed the municipal census (Households Data Directory “Mi Familia Progresa” – MIFAPRO) during the period 2008-2011.This data directory is a fundamental tool for the cash transfers program of the same name. The data directory had the objective the collection and processing of the information concerning families living in the most vulnerable Guatemalan municipalities. The census results have allowed the classification of the population according to their poverty levels and the visualization of the socioeconomic profiles of those recipients of the Government’s cash transfers. To date, the census has been collected in 301 of the 334 Guatemalan municipalities; which is equivalent to 1.58 million households at the national level. 11http://www.ine.gob.gt/Nesstar/Censo2002/survey0/index.htm 12http://www.ine.gob.gt/np/mifapro/index.htm 97 The Encuesta Nacional de Condiciones de Vida (ENCOVI)13, 14 This survey provides information and assesses the living conditions and a number of social indicators of the Guatemalan population, as well as changes in poverty levels and factors that determine these changes. The Guatemalan Government develops the Strategy of Poverty Reduction (Estrategia de Reducción de la Pobreza – ERP) with the objective to significantly reduce poverty, extreme poverty and income inequality. To achieve this objective, it is required to design and construct indicators that measure the situation of all social sectors and aid the development of programs to overcome poverty, extreme poverty and income inequality. The main objectives of the ENCOVIs are: • To know and assess the living conditions of the population and the factors that determine those conditions at the national and departmental level. • To provide information to design strategies for poverty reduction, social programs, and the modernization and decentralization of the State. • To generate information for the improvement of the mechanisms of targeting public expenditures, and the national budget and investments The ENCOVIs are a basic element for users in the public and private sectors, for the following and evaluation of social programs and for the formulation of strategies that lead to poverty reduction. So far, there have been three installments of these surveys: • 2000 ENCOVI. This was the first ENCOVI, and collected baseline data on poverty levels and general living conditions in Guatemala. This first ENCOVI included three survey instruments: households survey, community survey and prices survey. • 2006 ENCOVI. The main results of the 2006 ENCOVI show, among other things, poverty levels at the national level, its distribution by regions, and departments. This survey also provides information regarding poverty incidence and its determinants and other socioeconomic indicators related to living conditions. • 2011 ENCOVI. In addition to the same objectives of the 2006 ENCOVI, the main purpose of the 2011 ENCOVI was to provide information on changes of poverty levels and the determinants of these changes. 13 http://www.ine.gob.gt/np/encovi/index.htm 14 http://www.lisproject.org/techdoc/gt/gt06survey.pdf 98 Appendix F – Treatment and Control Matched Analysis Sample Intervention and Control Municipalities in 2006 Intervention Municipalities Control Municipalities Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Rural Urban Rural Urban Rural Urban Rural Urban Alta Verapaz Chisec 38 1,342 1,541 17,762 5,244 77.2% Quiché Nebaj 55 1,334 12,482 100.0% Alta Verapaz San Cristóbal Verapaz 9 1,094 4,842 100.0% Alta Verapaz San Miguel Tucurú 11 1,041 3,685 100.0% Baja Verapaz Cubulco 87 1,625 8,871 100.0% Alta Verapaz Fray Bartolomé de las Casas 25 1,621 6,702 100.0% Baja Verapaz San Jerónimo 31 3,401 2,094 1,415 1,727 45.0% Quetzaltenango Cantel 25 3,537 2,142 1,300 1,920 40.4% Chimaltenango Acatenango 24 2,229 6,515 2,509 1,067 70.2% Chiquimula San Jacinto 31 2,227 3,166 100.0% Chimaltenango Pochuta 10 1,266 1,990 100.0% Quetzaltenango Palestina de Los Altos 11 1,220 1,529 100.0% Chimaltenango San Martín Jilotepeque 50 1,668 11,488 7,620 1,856 80.4% Alta Verapaz Fray Bartolomé de las Casas 25 1,621 6,702 100.0% Chimaltenango Santa Cruz Balanyá 11 4,300 1,617 0.0% Sacatepéquez San Miguel Dueñas 13 4,176 923 0.0% Chimaltenango Tecpán Guatemala 88 1,980 3,653 13,139 4,401 74.9% Huehuetenango La Libertad 23 1,888 8,622 100.0% Chiquimula Jocotán 80 1,025 12,755 6,176 1,752 77.9% Quiché Zacualpa 11 1,032 6,314 100.0% Huehuetenango Chiantla 48 2,378 3,629 12,242 4,476 73.2% Quiché Ixcán 45 2,282 8,594 100.0% Huehuetenango San Pedro Necta 11 5,367 6,600 100.0% Petén Santa Ana 21 5,604 3,554 100.0% Huehuetenango Santa Eulalia 7 1,144 2,891 100.0% Baja Verapaz Granados 24 1,127 3,238 100.0% Izabal Livingston 114 1,502 4,826 9,420 2,372 79.9% Santa Rosa Casillas 19 1,548 6,338 100.0% Jalapa San Pedro Pinula 112 1,307 2,234 12,122 3,628 77.0% Quiché Nebaj 55 1,334 12,482 100.0% Quiche Joyabaj 30 2,562 9,420 100.0% Jutiapa Agua Blanca 37 2,664 7,458 100.0% Quiche Santo Tomas Chichicastenango 9 2,168 2,763 0.0% Quetzaltenango Cabricán 13 2,261 2,288 0.0% Sacatepequez Santa María de Jesús 45 2,863 2,365 0.0% Suchitepéquez Santo Tomás La Unión 11 2,784 2,684 0.0% San Marcos Ixchiguan 11 1,568 2,453 100.0% San Marcos Tajumulco 12 1,522 2,724 100.0% San Marcos Malacatán 35 2,604 7,660 100.0% Jutiapa Agua Blanca 37 2,664 7,458 100.0% San Marcos San Marcos 10 8,597 4,510 0.0% Escuintla Siquinalá 13 8,214 3,549 0.0% San Marcos Tejutla 12 1,446 4,896 100.0% Chimaltenango Yepocapa 32 1,449 4,977 100.0% Santa Rosa Santa Cruz Naranjo 12 1,745 1,776 100.0% Suchitepéquez Patulul 12 1,672 948 100.0% 99 Intervention and Control Municipalities in 2006 Intervention Municipalities Control Municipalities Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Rural Urban Rural Urban Rural Urban Rural Urban Solola San Antonio Palopó 27 889 6,495 100.0% Quiché San Andrés Sajcabajá 22 849 6,040 100.0% Solola San Pedro La Laguna 19 4,626 2,902 0.0% Sacatepéquez Santo Domingo Xenacoj 27 4,530 2,862 0.0% Suchitepequez Santo Domingo Suchitepequez 22 3,473 4,554 100.0% Huehuetenango Aguacatán 14 3,455 4,494 100.0% Totonicapan Santa Lucía La Reforma 10 1,255 8,400 100.0% Jalapa San Manuel Chaparrón 37 1,196 1,658 5,266 1,209 81.3% Zacapa Usumatlán 33 3,628 1,711 100.0% Jutiapa Pasaco 11 3,601 1,562 100.0% Note: The rows colored orange are those for which there was no rural population data for the intervention municipalities in 2006. Intervention and Control Municipalities in 2011 Intervention Municipalities Control Municipalities Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Rural Urban Rural Urban Rural Urban Rural Urban Alta Verapaz Chisec 37 1,692 11,248 100.0% Quiché Nebaj 21 1,373 5,119 100.0% Alta Verapaz San Cristóbal Verapaz 23 1,317 6,637 100.0% Alta Verapaz San Miguel Tucurú 23 1,600 6,485 100.0% Baja Verapaz Cubulco 83 1,271 7,516 100.0% Alta Verapaz Fray Bartolomé de las Casas 20 1,583 6,770 100.0% Baja Verapaz San Jerónimo 43 2,182 4,886 1,854 3,905 32.2% Quetzaltenango Cantel 37 2,490 3,680 5,368 5,406 49.8% Chimaltenango Acatenango 24 1,785 2,634 100.0% Chiquimula San Jacinto 33 975 4,131 2,433 2,010 54.8% Chimaltenango Pochuta 13 1,644 2,197 100.0% Quetzaltenango Palestina de Los Altos 11 1,680 2,871 100.0% Chimaltenango San Martín Jilotepeque 72 2,729 4,861 10,723 2,002 84.3% Alta Verapaz Fray Bartolomé de las Casas 20 1,583 6,770 100.0% Chimaltenango Santa Cruz Balanyá 14 1,760 2,352 0.0% Sacatepéquez San Miguel Dueñas 51 4,707 3,604 0.0% Chimaltenango Tecpán Guatemala 72 1,620 3,978 11,899 2,547 82.4% Huehuetenango La Libertad 12 960 4,800 100.0% Chiquimula Jocotán 60 1,229 6,661 100.0% Quiché Zacualpa 10 1,184 4,160 100.0% 100 Intervention and Control Municipalities in 2011 Intervention Municipalities Control Municipalities Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Department Municipality Number of HHs in Data Mean Household Income Estimated Number of HHs % of rural HHs Rural Urban Rural Urban Rural Urban Rural Urban Huehuetenango Chiantla 33 1,076 15,723 100.0% Quiché Ixcán 45 2,099 11,724 100.0% Huehuetenango San Pedro Necta 10 3,092 4,420 100.0% Petén Santa Ana 11 1,514 3,949 100.0% Huehuetenango Santa Eulalia 11 723 2,464 100.0% Baja Verapaz Granados 11 1,740 1,925 100.0% Izabal Livingston 139 4,000 1,901 14,916 3,599 80.6% Santa Rosa Casillas 24 1,379 1,889 836 2,379 26.0% Jalapa San Pedro Pinula 70 1,092 7,615 100.0% Quiché Nebaj 21 1,373 5,119 100.0% Quiche Joyabaj 62 2,032 5,638 13,630 4,864 73.7% Jutiapa Agua Blanca 11 1,418 660 100.0% Quiche Santo Tomas Chichicastenango 99 1,826 2,843 9,776 17,024 36.5% Quetzaltenango Cabricán 12 3,390 1,884 100.0% Sacatepequez Santa María de Jesús 42 1,711 3,219 0.0% Suchitepéquez Santo Tomás La Unión 23 3,177 5,637 1,971 2,324 45.9% San Marcos Ixchiguan 13 1,040 14,443 100.0% San Marcos Tajumulco 23 1,143 9,964 100.0% San Marcos Malacatán 48 1,611 15,365 100.0% Jutiapa Agua Blanca 11 1,418 660 100.0% San Marcos San Marcos 11 11,805 6,963 0.0% Escuintla Siquinalá 39 3,354 4,323 3,828 3,094 55.3% San Marcos Tejutla 12 2,346 6,324 100.0% Chimaltenango Yepocapa 41 1,810 4,503 4,235 2,912 59.3% Santa Rosa Santa Cruz Naranjo 10 1,718 3,680 100.0% Suchitepéquez Patulul 71 4,869 5,204 9,773 5,893 62.4% Solola San Antonio Palopó 29 1,118 2,661 645 3,318 16.3% Quiché San Andrés Sajcabajá 12 2,137 3,600 100.0% Solola San Pedro La Laguna 31 2,242 4,197 0.0% Sacatepéquez Santo Domingo Xenacoj 39 3,559 4,591 3,302 1,690 66.1% Suchitepequez Santo Domingo Suchitepequez 34 2,487 6,773 100.0% Huehuetenango Aguacatán 34 3,303 18,559 100.0% Totonicapan Santa Lucía La Reforma 22 774 2,728 100.0% Jalapa San Manuel Chaparrón 22 1,350 2,123 100.0% Zacapa Usumatlán 22 3,191 2,843 520 1,188 30.4% Jutiapa Pasaco 21 2,824 4,048 0.0% Note: The rows colored orange are those for which there was no rural population data for the intervention municipalities in 2006. 101 Appendix G – DID Regression Results Due to size, the full regression results are submitted in a separate Excel file. 102 Appendix H – Criteria for an Ideal Impact Evaluation In a case where an evaluation focuses on whether a 'treated' individual shows the effects of a 'treatment' (or intervention), the ideal study identifies two groups, the treated group and the control group. In the ideal study these two groups would have identical characteristics at the beginning of the intervention and apart from the treatment the two groups would have experienced similar situations during the intervention.15 An ex-post evaluation endeavors to use existing data sources to identify two such groups of individuals and then examines the differences in outcomes between the two groups, associating that outcome with the effect of the treatment. But because the identification of the two groups is ex-post, it is often deficient in some way. This appendix lists the possible sources of deficiency and comments on whether they were present in the Guatemala evaluation. Identification of beneficiaries and acquisition of relevant performance data. The ideal study identifies exactly who was treated in the intervention and who was not treated. It then makes precise measurements on the variables targeted by the intervention. Because the Guatemala evaluation relies on secondary, general-purpose survey data, it is not possible to identify the direct beneficiaries of the programs. In addition, because beneficiaries cannot be identified, it is not possible to collect data regarding the direct effect of the programs on their economic well-being. Confounding factors. In order to attribute differences in outcomes between a treatment group and a control group to the intervention itself, it is necessary that were no other factors present that would have differentially affected treated and control groups. It is not possible to identify confounding factors, such as non-USAID-sponsored programs that may have had an effect on smallholder farmers, consumption, and other metrics of well-being in Guatemala. Selection bias. Selection bias occurs if the treatment population is not randomly selected and the criteria determining selection are correlated with the outcomes of interest. To give a simple example, it is a real problem if more ambitious people are more likely to find out about the treatment and be treated, because ambition is obviously related to many outcomes, and what is important, data on ambition is not available. In the Guatemala study, participation in the interventions analyzed occurred on a voluntary basis, which introduces potential selection biases, because unobserved characteristics may be inherent to the nature of those who elected to participate that would also affect program outcomes and cannot be accounted for by the data available. Impact heterogeneity. This term refers to differences in impact of intervention by participant type and context. Because data were not collected about the actual treatment group, the LEAP team would not be able to gauge how differences in impact varied by participant type or context. For example, the final evaluation report of one of the interventions analyzed (Mercy Corps) indicates that the intervention produced better results for those participants who had received more 15See http://www.3ieimpact.org/doc/principles%20for%20impact%20evaluation.pdf, retrieved on 05/25/2012 for a fully discussion of these issues. 103 education and resources since the onset of the program. Because program participants cannot be exactly identified on an individual basis, the evaluation was not be able to control for heterogeneity across recipients. Spillover effects. It is possible that households/municipalities in the control areas may have been affected by spillovers from the interventions, thus diminishing the estimated effect of the intervention on the treated relative to the controls. For example, there is an overlap of interventions in 24 out of the 80 identified municipalities that received assistance from the USAID programs. These overlaps and the potential complementarities among these programs make it impossible to isolate the effects of each of them separately. Each of these possible problems has to be taken into account when evaluating the implications that are drawn in the evaluation that is summarized in this report. 104 Appendix I – Choice of methodology of the construction of the dataset to be used in the evaluation analysis For the impact evaluation analysis, two possible data methodologies were considered. The first methodology is based on the procedure being followed by the World Bank to construct a Guatemala Poverty Map utilizing data from both the 2011 ENCOVI and from the 2011 Guatemala “Mi Familia Progresa” household census data. This methodology is known as Small Area Estimation (SAE). This methodology uses an imputation method to assign levels of income at the municipal level that are not available from the ENCOVI data.16 This technique requires two sources of data as a minimum. One source of data is highly detailed survey data gathered utilizing a small sample; these data would not be representative for small subpopulations, such as municipalities. The second source of data provides less detailed information, but the sample size would be large enough to be representative for small subpopulations. An example of this type of data is census data. In the case of the Guatemala evaluation, the sources of comprehensive survey data are the ENCOVI 2006 and 2011 datasets. The 2011 “Mi Familia Progresa” household census and the 2002 population census can be used as the large non-comprehensive dataset required for building the end-line data of the interventions. A challenge to using this methodology is the lack of a large non￾comprehensive dataset contemporaneous to the beginning of the interventions for building the baseline dataset. The 2002 Guatemala population census could be used instead as an approximation, as could the 2011 Mi Familia Progresa census.17 However, neither choice is optimal. In terms of data work, the utilization of four different datasets (the ENCOVIs, 2002 population census, and 2011 household census) complicates the analysis considerably, as they all need to be manipulated such that the final data used from each set are consistent and comparable. In addition, because of this methodology’s large levels of imputation of variables, the resulting standard errors would be extremely large, which implies that the outcome from utilizing this methodology would not be as reliable purposes of this study as the second proposed methodology. The second and chosen methodology of analysis is the utilization of the ENCOVIs’ data to build a treatment and non-treatment comparison of municipality groups. The team identified these groups by selecting municipalities where smallholder farmers who were beneficiaries of the intervention were located. These groups were then analyzed for differences and similarities over time. The advantage of this methodology is that matching variables between two or three different sources of data or the imputation of income values is not necessary, thereby reducing the amount of error and uncertainty. A third and earlier point-in-time was utilized to describe area growth trajectories over time. This pre-baseline data was obtained from the 2002 Population Census. As mentioned previously, this methodology is deficient because it cannot account for the effects of other interventions or factors that may have taken place during the same time as the interventions for 16 The SAE technique “involves econometric or quantitative indirect estimation procedures that combine spatial precision (such as censuses) with substantive depth (such as surveys). They have been developed and implemented by World Bank Development Economics Research Group and colleagues, in collaboration with country teams for the implementation of Poverty Reduction Strategy Programmes.” (2012). Retrieved from http://sedac.ciesin.columbia.edu/povmap/methods_nat_sae.jsp. 17 The 2011 “Mi Familia Progresa” was collected between 2008 and 2011. 105 this analysis and that may have affected the economic outcomes of both the treatment and the control groups.