USAID FROM THE AMERICAN PEOPLE FINAL REPORT MIDTERM EVALUATION OF THE INTEGRATED DEVELOPMENT AND CONSERVATION IN THE BOLIVIAN AMAZON PROJECT Disclaimer The views expressed in this publication are those of the authors and do not necessarily reflect the views of the United States Agency for International Development (USAID). Acknowledgements We offer our sincere thanks to all who contributed to the completion of this Mid‐Term Evaluation of the Integrated Development Conservation in the Bolivian Amazon Project. In particular we wish to thank Mrs. Ximena Rodriguez, COR assigned and responsible for Monitoring and Evaluation in USAID ‐ Bolivia, who has provided prompt support throughout the evaluation process. We also wish to thank Mr. Eduardo Galindo, responsible for the technical support of the project for USAID. We thank the IBA project team for their support during the evaluation process and for the information and support provided in a collaborative and transparent way. In particular, we wish to thank Dr. Eduardo Sandoval, Rodrigo Daza, Ing. Fatima Vargas and Ing. Fernando Pizarro. Additionally, we express our thanks to Raúl Lobo and Ivette Carrasco for the support provided while conducting fieldwork. We make public our gratitude to the authorities of the communities and municipal governments, representatives of local organizations, groups of beneficiaries interviewed and those involved directly in project implementation from CADEFOR, WCS and FDTA Valles who have provided valuable information for the evaluation in response to our questions. Finally, thanks to the administrative staff of HIB Latinoamerica and to Equipos Mori, for their support and commitment in carrying out the work in the field, analysis of the information and preparation of this report.  We would especially like to thank Nelly Villarroel for her dedication and support. i Executive Summary In August 2012, USAID hired the consultancy firm, Holz Industrieberatung Latin America SRL (HIB) to carry out the midterm performance evaluation of the Integrated Development and Conservation in the Bolivian Amazon Project (IBA). The project is implemented by Chemonics International and a group of national and international partners under contract EPP‐I‐00‐06‐00013‐00 established between USAID and Chemonics. The project, which started in September 2009, has duration of five years and a budget of US $12,393.861. The project falls under the Sustainable Economic Growth and Environment (SEGE) strategy in the office of Sustainable Development and Environment (SDE) within USAID/Bolivia. This evaluation covers the first 30 months of the project, from September 30, 2009 until March 31, 2012. Project expenditure through March 31, 2012 was US $6,539,396. PAI is the third of a series of projects implemented by USAID in the natural resources sector in Bolivia, a continuation from BOLFOR I and II. The project has been implemented during the second government of Evo Morales, in a period of raised tension between the governments of Bolivia and the United States.   The deterioration in relations between the United States Government and the Government of Bolivia has had significant impact on the implementation of the project, which has been further affected by the conflicts generated around the sensitive themes of forests and land in Bolivia, and exacerbated by the conflicts between indigenous organizations and government. In May 2012 there was a change in project management with the appointment of a new Chief of Party. Since then, the project has worked to develop a revised approach to implementation, in both its implementation strategy and management system. In December 2012 project implementation was put on hold due to a lack of funds.   This has led to the conclusion of contracts between Chemonics and implementing partners and the laying off of project staff. As a result, the project is in a state of limbo as the USAID Mission in Bolivia awaits definition of funding levels. The purpose of the project is to promote the conservation and sustainable use of biodiversity for the well‐being of the Bolivian people, taking into account global climate change. The strategy of the project is to promote the development of integrated forest management activities, tourism and agro‐ecology in a framework of land management and improved governance of natural resources with the active participation of stakeholders. The project’s four objectives are to1 : 1. Strengthen local and municipal governance to improve biodiversity conservation efforts, environmental and territorial management and to increase resilience to global climate change; 2. Strengthen the capacity of civil society umbrella organizations to participate in the development and implementation of productive activities, policies, norms, regulations and technical tools that promote the sustainable use of Bolivia's biodiversity, goods and services;                                                              1 As stated in the SOW ii 3. Promote economic growth and increase incomes of local and indigenous communities and producer groups through sustainable use of natural resources, other land‐based production systems, and improved commercialization and market linkages for related goods and services; and, 4. Seek opportunities to scale‐up the IBA project’s geographic focus. The logic of intervention to achieve these objectives emphasizes institutional strengthening, especially the strengthening of the capacity of local governments and grassroots organizations of indigenous people, to strengthen local forest governance framework, generating mechanisms of social control and equitable income distribution, with the participation of local actors. In parallel, the project promotes the valuation of forests by generating benefits derived from sustainable use of forest products and services. From outside the forest, project actions are intended to reduce pressure on forests by supporting the development of agricultural practices that reduce pressure on forest resources by addressing the threats of deforestation and forest fires. The project objectives remain valid and are aligned with national policies for integrated forest and land management expressed in the Constitution of the Plurinational State (NCPE‐PN) and the Law of Mother Earth, which are considered as opportunities for the Mission of USAID/Bolivia. Originally the project aimed to work in three priority regions of the country (Pando, northern La Paz / Western Beni and Guarayos in Santa Cruz department), but was obliged to discontinue activities in Pando shortly after beginning implementation, as a result of internal conflicts in this region. This reduction in geographical scope of the project was not made official by USAID through a modification to the contract, or any official communication.  The scope of the project has been reduced from the level presented in the original proposal to work in 18 municipalities to a total of seven municipalities (Ixiamas and San Buenaventura in the north of La Paz, Rurrenabaque, Santos Reyes and Santa Rosa de Yacuma in Beni, Ascension de Guarayos and Urubichá in Santa Cruz). Of these seven municipalities, two (Ixiamas and San Buenaventura) have had internal problems within  the municipal administration which made it impossible to work with the municipal governments, although work was carried out within the municipalities. The evaluation seeks to answer the following four main questions (Refer to Terms of Reference prepared by USAID / Bolivia Annex 1) 1. Which planned results and outcomes have been achieved by the project to date, supporting the achievement of the SEGE strategic objective? 2. Is the IBA project’s integrated approach working?   3. To what extent are the indicators attributable to the IBA project? 4. Since the IBA project is essentially an institutional strengthening project, how effective has the contractor been in strengthening local institutions, and putting in place mechanisms to monitor and evaluate progress in institutional strengthening? iii The evaluation methodology was designed using these questions as a basis, combining quantitative and qualitative methods to gather information, including conducting a formal survey of a sample of 473 project beneficiaries, 73 semi‐structured interviews with key informants, six focus groups, a Scorecard of Community forestry organizations and associations and conducting an impact assessment of field practices. Additionally, the evaluation team met with project staff, implementing partners, USAID staff and key informants in the natural resources sector in the country. The evaluation team was supported by Equipos Mori to carry out the household survey. In practice, project support to institutional strengthening of municipal governments and indigenous organizations has been extremely limited. It is recognized that this is partially due to the politically sensitive nature of the issues addressed by the project in the Amazon region. The expulsion of USAID from Pando, the indigenous marches, internal divisions within the indigenous organizations (COPNAG, CIDOB, etc.) are symptomatic of the problems faced by the project. This apparently led to the verbal instruction from USAID for the project to “keep a low profile.”.   This instruction has not formally reinforced by USAID/Bolivia. The application of this instruction was interpreted in a way that has implied severely limiting project efforts devoted to activities to support institutional strengthening of indigenous organizations and municipal governments. As a result, evaluation team found that the project has made little progress in achieving these objectives   Indicators have not been defined at the project purpose level, making it difficult to assess project progress. However, deforestation and forest degradation, resulting in biodiversity loss, continue at alarming levels in the country and in particular in the Guarayos area covered by the project. Although there has been some improvement in income levels, indicators to monitor improvements in the livelihoods of project beneficiaries have not been developed. Similarly, indicators for institutional strengthening have not been developed, making it difficult to objectively assess the degree of progress in the achievement of the project’s specific objectives.   However, the analysis carried out by the evaluation team revealed that the ability of local governments regarding territorial administration remains weak and the situation of natural resource governance is still precarious.  Local capacity to deal with climate change is very limited (Objective 1). The project has strengthened the capacity of productive organizations more in technical issues than in administrative and management matters. However, the project has only contemplated support for indigenous peoples’ organizations and not the intercultural communities of colonist farmers who pose a greater threat to forests and biodiversity. Even so, little progress has been made in strengthening indigenous organizations due to the political and institutional crisis affecting the context in which the project is implemented (Objective 2). The evaluation has confirmed improvement in the income levels of communities (Objective 3). Support for ecotourism activities has significantly improved income levels of the operating companies and communities. Improvement in the income of the community forestry organizations and associations are due not only to the increase in volume of sales, but also from the increase in the price of timber experienced over the past two years. The impact of the agro‐ecology component on income has been minimal, although in Guarayos a small number of rice producers have improved yields and income. Rather than expanding the geographical scope of the project as originally proposed, project coverage has become more focused, reducing attention from 15 to 7 municipalities (Objective 4). iv With regard to progress made towards the achievement of SEGE indicators by the project, the evaluation team made the following findings:  Of the 6.738 beneficiaries reported the evaluation team confirmed records of 1,473 direct beneficiaries that could be located from project lists. No indirect beneficiaries were registered by the project. Other direct beneficiaries as stated in the PMP could not be validated by the evaluation team due to the lack of registers of place of residence. Differences in numbers reported and numbers verified are due to two factors: firstly, many direct beneficiaries cannot be objectively verified from the lists, and secondly, multiple registers of the same individuals as beneficiaries2 were encountered. This second factor was due to the erroneous interpretation of the indicator employed by Chemonics and approved by USAID/Bolivia (although no formal record of the approval was recorded either by Chemonics or USAID/Bolivia). This difference has been exacerbated by the inappropriate selection of project lists as the means of verification which did not permit objective verification of the indicator.    The household survey revealed that 29% of the 473 households registered an increase in income for the period 2010‐2012 (extrapolated to 281 households). However, an increase equal to or greater than 5% occurs in 19.7% of cases. This amounts to a total of 174 households from a universe of 970 households that could be verified. The nationwide rise in timber prices represents a significant factor in the increase. Although this definition of the indicator is accepted in the PMP, the evaluation team does not consider that this proportion of the increase should be attributed to the project.  The indicator for the increment in the amount of sales has been exceeded by the project.   However, the definition of the indicator in the PMP accepts incremental sales that the evaluation team does not consider to be attributable to the project.  The indicator for the area under improved management of natural resources has been exceeded according to the definition in the PMP. However, in the opinion of the evaluation team not all of the area under improved management is attributable to the project.  The area of biological significance reported to be under improved management does not correspond to the definition of the indicator in the PMP.  The project reported 1,844 registers of training of which 1,581 are individuals. Since it was not possible to locate the place of residence of people trained, it was not possible to verify progress reported for this indicator. This forms part of the indicator of number of beneficiaries and suffers from the same deficiency in the means of verification (lists of participants) which are not objectively verifiable.  The reported amount of greenhouse gases reduced or sequestered represents 13% of the target. The indicator does not take into account the balance of greenhouse gases and only                                                              2 In the field verification of beneficiaries 11% of the sample of beneficiaries could not be traced. v considers the volume sequestered and does not take into account all areas of project intervention.  The project does not report people trained in climate change. However the evaluation team was able to verify 193 people trained in this area. The underestimation is due to an error in interpretation of the indicator by the project Significant problems were found in the definition of project indicators, the means of verification and the form of interpretation of indicators. The definition of indicators proposed by Chemonics has been approved by USAID Bolivia. Similarly, the way indicators have been interpreted by Chemonics has also been approved by USAID Bolivia, as was admitted in a clarification meeting held as part of this evaluation. The project has not developed appropriate indicators for each level of intervention (for example, there are no indicators of territorial management, governance, or biodiversity conservation). SMART criteria have not been applied in the formulation of indicators and several indicators are dependent on other indicators.   It is considered that internal project monitoring and external monitoring by USAID should have detected problems in recording and interpreting the indicators at an early stage of project implementation. For this reason, the evaluation team considers that project monitoring has been deficient and failed to verify information generated. We recommend that monitoring visits by both USAID and Chemonics should be more frequent and thorough in their analysis, taking place at least every six months, visiting all project areas, especially communities where local people are quick to highlight limitations. The evaluation team recommends that the project should review the PMP so that indicators selected better reflect the level of achievement of project results. The revised PMP should redefine the indicators and the levels that can reasonably be achieved to the end of the project, establishing a reliable system of data gathering and analysis.  In defining indicators SMART criteria should be employed and a formal agreement signed between USAID and Chemonics agreeing to the interpretation of revised indicators. Chemonics should correct the deficiencies in found with those indicators that are not objectively verifiable and then recalculate the number of beneficiaries, management areas and other indicators affected. Project implementation has been seriously affected by structural problems that stem from project design. The evaluation team highlights a confusion of roles and responsibilities of contractors, payments made without taking into account the results achieved, separation of institutional functions leading to disjointed project actions, weak coordination at operational levels, ad hoc relationships between USAID with project staff, and conflict of interest between management positions and service providers. The evaluation team recommends that Chemonics should review the operational structure, establishing clear lines of command and responsibilities based on the achievement of project outcomes and results. The project should develop a communications strategy to promote ownership of the project by the local communities and the dissemination of information generated by the project. It is considered that the original scope of the project was overly ambitious, especially given that the strategic approach represents a new experience for USAID and Chemonics in Bolivia. Project efforts should focus on developing a functional operational approach and then extending this approach to other areas. vi Table of Contents EXECUTIVE SUMMARY ........................................................................................................................................ I 1 INTRODUCTION ......................................................................................................................................... 1 2 EVALUATION METHODOLOGY.................................................................................................................... 8 2.1 LIMITATIONS IN THE METHODOLOGY ...................................................................................................................12 3 ANALYSIS AND RESPONSES TO KEY QUESTIONS.........................................................................................12 3.1 WHICH PLANNED RESULTS AND OUTCOMES HAVE BEEN ACHIEVED BY THE PROJECT TO DATE, SUPPORTING  THE ACHIEVEMENT OF THE SEGE STRATEGIC OBJECTIVE? ..............................................................................................................................12 3.2 IS THE IBA PROJECT’S INTEGRATED APPROACH WORKING? ......................................................................................23 3.3 TO WHAT EXTENT ARE THE INDICATORS ATTRIBUTABLE TO THE IBA PROJECT?..............................................................24 3.4 SINCE THE IBA PROJECT IS ESSENTIALLY AN INSTITUTIONAL STRENGTHENING PROJECT, HOW EFFECTIVE HAS THE CONTRACTOR BEEN IN STRENGTHENING LOCAL INSTITUTIONS, AND PUTTING IN PLACE MECHANISMS TO MONITOR AND EVALUATE PROGRESS IN INSTITUTIONAL STRENGTHENING?....................................................................................................................................27 3.5 TO WHAT EXTENT HAS THE IBA PROJECT GOAL OF USING LOCAL COMMUNITIES TO MONITOR BIODIVERSITY BEEN EFFECTIVE? …………………………………………………………………………………………………………………………………………………………….  27 3.6 GIVEN BOLIVIA’S POSITION ON GCC IN DURBAN, CAN THE PROJECT WORK IN THESE TYPES OF ACTIVITIES?........................28 4 FINDINGS OF THE MID‐TERM EVALUATION ...............................................................................................29 4.1 PROJECT MANAGEMENT...................................................................................................................................29 4.2 INSTITUTIONAL STRENGTHENING ........................................................................................................................30 4.3 PROJECT COMPONENTS....................................................................................................................................31 5 CONCLUSIONS ..........................................................................................................................................33 6 RECOMMENDATIONS................................................................................................................................36 7 LESSONS LEARNED....................................................................................................................................38 7.1 INSTITUTIONAL STRENGTHENING ........................................................................................................................38 7.2 PARTICIPATORY APPROACH ...............................................................................................................................38 7.3 GENDER APPROACH.........................................................................................................................................38 7.4 THE INTEGRATED APPROACH..............................................................................................................................39 8 BEST PRACTICES FOUND............................................................................................................................40 8.1 FOREST MANAGEMENT ....................................................................................................................................40 8.2 TOURISM.......................................................................................................................................................41 8.3 AGRO‐ECOLOGY..............................................................................................................................................41 9 REFERENCES .............................................................................................................................................41    vii INDEX OF FIGURES ILLUSTRATION 1. MAP OF THE AREAS OF INTERVENTION OF (SOURCE: EVALUATION TEAM)................................................................6 INDEX OF GRAPHS GRAPH 1. AREA BURNED IN IBA PROJECT MUNICIPALITIES  (PERIOD 2000‐2010). SOURCE: CUELLAR (2012).....................................7 GRAPH 2.  AUTHORIZED AND ILLEGAL DEFORESTATION IN BOLIVIA FROM 1997 TO 2010 (SOURCE ABT 2012)...................................8 INDEX OF TABLES TABLE 1. INITIAL AND CURRENT GEOGRAPHIC COVERAGE (SOURCE: DEVELOPED FROM INFORMATION PROVIDED BY IBA PROJECT) ............5 TABLE 2. RESPONSIBILITIES OF EXECUTORS CONTRACTORS PAI (SOURCE: AUTHOR).........................................................................6 TABLE 3. PROGRAM FOR THE IMPLEMENTATION OF THE MIDTERM EVALUATION OF IBA. (SOURCE: AUTHOR).......................................8 TABLE 4. GENERAL SUMMARY OF THE METHODOLOGICAL DESIGN OF THE EVALUATION (SOURCE: AUTHOR).......................................10 TABLE 5. DISTRIBUTION OF THE SAMPLES IN THE SURVEYED COMMUNITIES (SOURCE: AUTHOR) .......................................................11 TABLE 6. RELATIONSHIP BETWEEN SEGE INDICATORS AND IBA PROJECT INDICATORS (SOURCE: MTE) .............................................13 TABLE 7. PROGRESS REPORTED BY THE IBA PROJECT COMPARED TO PROGRESS VERIFIED BY THE MIDTERM EVALUATION (MTE) FOR THE PROJECT INDICATORS RELATED TO SEGE (SOURCE: MTE)................................................................................................14 TABLE 8. FREQUENCY OF MULTIPLE ENTRIES OF BENEFICIARIES’ NAMES FOUND IN THE RECORDS OF IBA LISTS. (SOURCE: MTE).............15 TABLE 9. NUMBER OF BENEFICIARIES BY SOURCE OF DATA (SOURCE: MTE)..................................................................................16 TABLE 10. HOUSEHOLDS BENEFITING FROM INCREASED INCOME WITH EXTRAPOLATED DATA BY MUNICIPALITY. (SOURCE: MTE) ...........18 TABLE 11. PERCEPTION OF HOUSEHOLD HEADS REGARDING PROJECT TRAINING (SOURCE: MTE)......................................................21 TABLE 12. ATTRIBUTION OF INDICATORS TO PROJECT ACTIVITIES (SOURCE: MTE).........................................................................26 LIST OF ANNEXES3 ANNEX 1. EVALUATION STATEMENT OF WORK.............................................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 2. EVALUATION TEAM .................................................................................................. ERROR! BOOKMARK NOT DEFINED. ANNEX 3. DISCLOSURE OF CONFLICT OF INTEREST ........................................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 4. DETAILS OF METHODOLOGY ....................................................................................... ERROR! BOOKMARK NOT DEFINED. ANNEX 5. DEFINITION OF SAMPLE SIZE....................................................................................... ERROR! BOOKMARK NOT DEFINED. ANNEX 6. REPORT OF FIELD SUERVEY OPERATION.....................................................................................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 7. REPORT OF PILOT FIELD SURVEY ..................................................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 8. INTERVIEWS AND MEETINGS ...................................................................................... ERROR! BOOKMARK NOT DEFINED.                                                              3 The numbering of the pages of this document is correlative to the numbering of the Evaluation Report viii ANNEX 9. EVALUATION OF FOREST MANAGEMENT PRACTICES ........................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 10. SOURCE OF INCOME BY MUNICIPALITY .......................................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 11. STRUCTURE OF INCOME FROM NATURAL RESOURCES ....................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 12. A ORGANIZATIONAL STRUCTURE TO 2011 ..................................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 13. METHODOLOGICAL PLANNING MATRIX.......................................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 14. INSTRUMENTS FOR INFORMATION GATHERING ............................................................ERROR! BOOKMARK NOT DEFINED. ANNEX 15. SELECTED PHOTOS FROM THE EVALUATION MISSION.....................................................ERROR! BOOKMARK NOT DEFINED.       ix Abbreviations Abbreviation Meaning AAA Area of Annual Cut ABT Authority of Public and Social Control of Forests and Land ACHIMU Indigenous Association of Cocoa Producers from the Municipality of Urubichá ASL Local Social Group BOLFOR Bolivia Forestal BPC Bolivian productivity and Competitiveness CADEFOR Center for Amazonian Forestry Development CARE CARE International CEJIS Center for Legal Studies and Social Research CEPAD Center for Participation and Sustainable Human Development CFO Certificate of Timber Origin CFV Voluntary Forest Certification Council CIDOB Confederation of Indigenous People of Bolivia CIEA Interethnic Center of Indigenous People of Guarayos CIPCA Center for Research and Promotion of Peasant Affairs CIPTA Indigenous Council of the Tacana People COPNAG Center for the Organization of Native People of Guarayos COR Contracting Officer Representative CRTM T’simane and Mosetene Regional Council DQA Data Quality Assessment EPA Environmental Program Assessment ESAF Agroforestry Services Company FAN Foundation of Friends of Nature FJMP Foundation José Manuel Pando FTDA-Valles Foundation for the Development of Agricultural Technology of the Valleys GCC Global Climate Change GDA Global Development Alliance GM Municipal Government x Abbreviation Meaning GOB Government of Bolivia IAPOAF Informe Anual de Aprovechamiento Forestal/ Submit annual Logging Reports IBA Integrated Development and Conservation in the Bolivian Amazon Project IBIF Bolivian Institute for Forestry Research ILS International Land Systems Inc. INE Instituto Nacional de Estadistica/ National Institute Statistical INIAF National Institute of Agricultural and Forestry Research MECOVI Mejoramiento de las Encuestas de Hogares y la Medición de Condiciones de Vida/ Monitoring Poverty and Social Indicator MMNPT Mancommunity of Municipalities of Northern La Paz M&E Monitoring and Evaluation NCPE-PN New Political Constitution of the Plurinational State of Bolivia OFC Community Forestry Organization OM Indigenous Organization NGO Non-Governmental Organization PCIB Productivity and Competitiveness Initiative for Bolivia Program PDCP Community Productive Development Plan PDM Municipal Development Plan PES Payment for Environmental Services PGMF General Forest Management Plan PGTI Indigenous Territorial Management Plan PMOT Municipal Territorial Land Use Plan PMP Program Monitoring Plan PND National Development Plan POA Annual Operational Plan POAF Forestry Annual Operational Plan PUMA Protection and Sustainable Use of the Environment (NGO) RB Biosphere Reserve REDD Reduced Emissions from Deforestation and Forest Degradation xi Abbreviation Meaning RVSRN Wildlife Reserve of Blanco and Negro Rivers SEGE Sustainable Economic Growth and Environment SERFORCU Cururu Forest Service SID Integrated Development Services SMART Specific, Measurable, Achievable, Realistic and Timebound SOW Scope of Works TCO Indigenous Community Territory UFM Municipal Forestry Unit USAID United States Agency for International Development WWF World Wildlife Fund WCS Wildlife Conservation Society 1 1 Introduction This report presents the results of the midterm performance evaluation of the Integrated Development and Conservation in the Bolivian Amazon (IBA) project, funded by USAID. The report was prepared by consultants, Holz Industrieberatung Latinoamerica SRL (HIB) commissioned by USAID‐Bolivia. The project is implemented by Chemonics International and a group of national and international partners under contract EPP‐I‐00‐06‐00013‐00 between USAID and Chemonics. The five‐year project started in September 2009 and has budget of $ 12,393.861. This evaluation covers the first 30 months of the project, from September 30, 2009 to March 31, 2012. The project expenditure for the period stands at US$ 6,539,396 (53% of the budget). USAID has been supporting the forestry sector in Bolivia since 1992, through support to BOLFOR I and II. Support to for biodiversity conservation and land management was provided until 2009 at a regional level through the Landscape Conservation Program, which focused on integrated planning of indigenous territories and protected areas, implemented at the regional level through strategic partners in the (WCS in Bolivia). Since 2009 USAID has changed the focus of its support, directing efforts towards the integrated management of natural resources under its strategy of Sustainable Economic Growth and Environment (SEGE). USAID has supported Bolivian Productivity and Competitiveness (BPC) Project implemented by Chemonics which among other sectors has supported the development of forest product value chains. Over recent years, Bolivia's economic growth and infrastructure development are causing unprecedented loss of forest cover and biodiversity. The indigenous people, who inhabit forested areas, represent the poorest sector of Bolivian society, benefiting little from the commercial use of their forest resources. In response to this situation the IBA project has been developed with the purpose of promoting the conservation and sustainable use of biodiversity for the well‐being of the Bolivian people, taking into account global climate. The project has the following objectives: 1. Strengthen local and municipal governance to improve biodiversity conservation efforts, environmental and territorial management and increase resilience to global climate change; 2. Strengthen the capacity of civil society umbrella organizations to participate in the development and implementation of productive activities, policies, norms, regulations and technical tools that promote the sustainable use of Bolivia's biodiversity, goods and services; 3. Promote economic growth and increase incomes of local and indigenous communities and producer groups through sustainable use of natural resources, other land‐based production systems, and improved commercialization and market linkages for related goods and services; and, 4. Seek opportunities to scale‐up the IBA project’s geographic focus. The logic of intervention to achieve these objectives is based on the development of integrated forest management activities, tourism and agro‐ecology, in a strategic framework for land management and governance of natural resources, with the active participation of the local stakeholders. The project implementation strategy is based on institutional strengthening, especially in strengthening the capacity 2 of local governments and indigenous peoples’ organizations to strengthen the local forest governance framework, creating mechanisms for more effective natural resources management. It aims to create incentives for conservation by valuing forests through the generation benefits from the sustainable use of forest products and services. From outside the forest, project actions are intended to reduce pressure on forests by supporting the intensification of agricultural practices to reduce the threats of deforestation and forest fires while at the same time generating economic benefits. The SOW defines that the mission will “evaluate the pertinence and coherence of the project and its components for the achievement of the Mission’s SEGE strategic objective and two of its intermediary results. Specifically, the purposes of the evaluation are: To assess implementation progress and effectiveness of all project components within the context of the project design and contract; and to ascertain if the project is contributing significantly to the attainment of the SEGE strategic objective through the review of the status of the progress towards the achievement of all targets planned for measurement indicators of success for both the strategic objective and the project as identified in both of these Program/Project Management Plans (PMPs). In addition, to assess the degree to which local and international implementation partners have been incorporated into project implementation, in response to USAID’s priorities to strengthening local partners (USAID Forward initiative for local capacity building).” Although not explicit, the evaluation team identified several implicit assumptions and hypotheses in the documentation provided including:  The policy framework is favorable for the implementation of the project.  A favorable economic environment for the implementation of project activities prevails.  Institutional stability of the parent organizations and municipal governments  Local government and indigenous peoples’ organizations are willing to work with the project.  The trainees have the will and the necessary conditions to implement the skills acquired.  Effective measures are implemented to control and regulate illegal logging.  Prevailing weather conditions do not increase the natural risks to forests and biodiversity. Although there was a real prospect that some of the hypotheses and assumptions might occur, no alternative implementation strategy was proposed for the different scenarios. At present no other program of international cooperation in Bolivia is implementing projects oriented to the integrated management of forests and agricultural lands with a focus on local governance and territorial management. However, there are several international cooperation initiatives that support institutional strengthening such as the Dutch government with the ABT. The governments of the Netherlands and Sweden support the CONFOR program, implemented by PUMA in Pando, Northern La Paz and Guarayos (the same three areas of work originally proposed by IBA) to develop the timber value chain and non‐timber forest products. Institutional strengthening activities for territorial management in indigenous territories are supported by the Fundación Amigos de la Naturaleza (FAN) in the northern Amazon region of Bolivia financed by the Dutch and Danish governments and the EU. The FCBC 3 (Foundation for Conservation of the Dry Chiquitano Forest) implements projects financed by the EU to strengthen the municipal territorial management capacities in the Chiquitania region, in eastern Bolivia. The NGOs CIPCA (Center for Research and Promotion of Peasant Affairs) and CEJIS (Center for Legal Studies and Social Research) work to strengthen the management capacity of indigenous peoples in various territories in the Bolivian Lowlands. Last year the IADB developed a loan to support the development of tourism in Bolivia.  The NGO, CEPAD (Center fro Participation and Sustainable Human Development) has been developing actions in support of tourism in the Rurrenabaque and the Chiquitanía. FCBC (Foundation for the Conservation of the Dry Chiqutiano Forest), FAN (Foundation for the Friends of Nature), PRISA (Program for the Implementation of Agro‐ecological Systems) and CARE support work with non‐timber forest products, agroforestry and agro‐ecological production in Ixiamas and Guarayos as well as in other regions of the country. The Rainforest Alliance runs its initiative to conserve the Amazon forest and the WWF (World Wildlife Fund) runs projects to support the development of forest management and forest value chains through the Global Forest Trade Network. This report is directed primarily to USAID‐Bolivia and Chemonics International to provide information to make adjustments to the project. The report is also directed to the Government of Bolivia to contribute to the discussion about the appropriate policy and institutional framework for integrated forest and territorial management approaches. Specifically, the information generated by the assessment will:    Strengthen and reorient strategies and project activities during the following years of implementation in order to determine which outcomes and results have been achieved,    Determine if the project is working with appropriate stakeholders,    Analyze what results will be achieved over the next 30 months,    Determine how to ensure the sustainability of the components necessary to achieve the strategic goal of SEGE, and    Identify any corrective actions needed by the project. This report responds to key evaluation questions posed by USAID in the Statement of Work (SOW), consisting of: 1. Which planned results and outcomes have been achieved by the project to date, which support the achievement the SEGE strategic objective? What outcomes in terms of forest management and biodiversity have been achieved (or can be expected) from the agro‐ecological programs? What are the necessary adjustments for improving the project’s effectiveness taking into consideration USAID/Bolivia’s priorities, the political context, and current relationships and engagement with project stakeholders and beneficiaries? Which lessons learned and best practices identified could be applied during the remaining life of this project? 2. Is the IBA project’s integrated approach working? Do current components and activities synergistically promote the conservation and sustainable use of Bolivia's biodiversity under current country political context? Are some components more relevant or effective than others? Are current implementation mechanisms effective? What other mechanisms could be utilized? 4 3. To what extent are the indicators attributable to the IBA project? Do the performance indicators capture the results and the achievements of the project? Are there other outcome and output indicators to be considered for better management of the project? 4. Since the IBA project is essentially an institutional strengthening project, how effective has the contractor been in strengthening local institutions, and putting in place mechanisms to monitor and evaluate progress in institutional strengthening? Have technical assistance activities resulted in strengthened governance and administration? To what extent has the project effectively engaged with local partners in implementing the IBA project components? Has the contractor developed and implemented activities to strengthen local partners? Identify which local partners have been most effective in the implementation of the project and whether their capacities have been simultaneously strengthened. In addition the SOW includes two complementary sets of questions to be addressed by the evaluation:  To what extent has the IBA project’s goal of using local communities to monitor biodiversity been effective? What are the incentives for local communities to carry out these functions? Is the project contributing to increase the incomes of local and indigenous communities and producer groups through sustainable use of natural resources? Does the project contribute to strengthening women’s priorities and rights regarding the sustainable development and access to a clean environment?  Given Bolivia’s position on GCC in Durban, can the project work in these types of activities? How should the project address them? To answer the evaluation questions the evaluation team adopted an exploratory methodological approach that combines quantitative and qualitative methods to gather information regarding the performance and perceptions of beneficiaries and local actors. The evaluation applied five methodological tools: Household Survey, applying a formal two‐stage random sample (households and communities), an Evaluation Scorecard of OFCs and ASLs, semi‐structured interviews, focus groups, and field assessments of forest management and agro‐ecological practices to monitor biodiversity.   The evaluation team included a forester, an ecologist, a rural sociologist, a forest economist, a statistician and a team of interviewers. Annex 2 presents the evaluation team and a curriculum vitae of each team member. Mori Equipment Company was responsible for the gathering of field data for the formal household surveys (see Annex 2). In May 2012 there was a change in project management with the appointment of a new COP (Chief of Party). Since then the project has worked on developing a revised strategic approach to project implementation and management system. The changes have addressed many of the problems and challenges identified by this evaluation mission4 by implementing a simplified organizational structure of the project (see Annex 15) with clearly defined lines of authority and responsibilities, reducing the                                                              4 This was not subject to evaluation by the mission as these events did not take place within the period covered; however, this information was presented to the evaluation mission by PAI project staff. 5 number of subcontractors, establishing a system of open tenders to local implementing organizations and reviewing the PMP. However, it remains to be seen what impact these changes will have on the achievement of project indicators. Due to the suspension of project funding from September 2012, at the time when the Mid Term Evaluation began, the project was in the process of closing offices and withdrawing field staff. The initial proposal of the project included 18 municipalities in four Departments of the Bolivian Amazon Region (Pando, Beni, North of La Paz and Santa Cruz Guarayos), as presented in Table 1 and Illustration 1.   At the time of the evaluation PAI operated in seven municipalities in three departments, having withdrawn from Pando as a result of conflicts in that region. The explanation for this change in geographical coverage was due to strategic issues, and to avoid problems and sensibilities with the Government of Bolivia. This change in strategy was the result of an instruction from USAID, however, but there is no formal record of the instruction and there are no amendments to the original contract documenting this change in geographical orientation. In practice, the project has focused on areas providing continuity of support to forest management initiatives assisted by previous USAID projects. Table 1. Initial and current geographic coverage (source: developed from information provided by IBA project) Department Proposed Municipalities Current Municipalities and Communities La Paz Teoponte, Guanay, Ixiamas Ixiamas (Ixiamas, Macahua, Santa Fe), San Buenaventura (Villa Alcira, Buena Vista, Bella Altura, San Juan del Bala) Beni Rurrenabaque, Riberalta Rurrenabaque (Rurrenabaque, Puerto Yumani, Carmen Soledad), Santa Rosa de Yacuma y Santos Reyes Santa Cruz Ascensión de Guarayos, El Puente, San Javier, San José, Roboré, San Ignacio de Velasco Ascensión de Guarayos (Ascensión de Guarayos, San Pablo), Urubichá (Cururú, Yaguarú, Urubichá) Pando Filadelfia, Porvenir, Bella Flor, Puerto Rico, Bolpebra, Gonzalo Moreno, San Lorenzo 6 Illustration 1. Map of the areas of intervention of (source: evaluation team) The project consortium is composed of national and international organizations. Table 2 presents the main thematic areas of responsibility of implementing partners. Table 2. Responsibilities of Implementing Partners of PAI (Source: Author) Implementing Partner Responsibilities Chemonics International Project Management and Coordination CADEFOR, (FJMP) Forest Management ILS Land Use and Tenure Winrock Climate Change and CO2 emissions FDTA Valles Supervision Agro-ecology component ESAF and SID Implementation of Agro-ecology component WCS and IBIF Monitoring of Biodiversity SOLIMAR Ecotourism The current national policy context for natural resource management and regulation is framed by the National Development Plan (NDP2006), the new political constitution (NCPE 2009) and the Tiquipaya 7 agreements (2010) in relation to Global Climate Change. These establish the regulatory framework for "climate justice" and the Joint Mechanism for Adaptation and Mitigation of Climate Change through non‐market incentives and an integrated approach to natural resource management, with emphasis on strengthening local institutions territorial administration as presented in the Government of Bolivia’s proposal at COP 17 in Durban. The Bolivian economy continues to grow at over 5% per annum due largely to the global boom in commodity prices.  This has also led to increased pressure on land and forest resources. At the same time there was a significant increase in the price of timber as a result of more effective measures of control implemented by ABT.   Even so, Bolivia is experiencing the highest rate of environmental degradation in the region5 . This is due to the expansion of road infrastructure, high international commodity prices and increasing wildfires in major forest areas of the country. The annual rate of deforestation, according Cuellar et al. (2012) between 2005 and 2010 was 205.080 hectares, while the ABT6 indicated that deforestation between 2000 and 2010 was 263,000 and in 2010 was 212.274 hectares. Environmental degradation is stimulated by certain public policies and institutional practices: legal deforestation patents are very low, the fines for illegal clearing are often waivered or not applied, fuel subsidies encourage deforestation and mechanized farming and the agricultural sector are exempt from VAT. Deforestation rates for the municipalities of Ascension and El Puente in Guarayos are over 1.5% per year, among the 10 municipalities with highest rates of deforestation in the country (Cuellar et. al. 20127 ). Ixiamas represents one of the municipalities most affected by forest fires in Bolivia (Graph 1). Graph 1. Area burned in IBA project municipalities (period 2000‐2010). Source: Cuellar (2012)                                                              5 http://www.washingtonpost.com/world/amazon‐forest‐threat‐is‐greater‐outside‐brazil/2012/08/30/2d47c4b0‐ ee08‐11e1‐b624‐99dee49d8d67_story.html   6 ABT. 2011. Informe anual 2010 y balance de la década del Régimen Forestal. Santa Cruz, Bolivia. 7 Cuellar, S., Rodríguez, A., Arroyo, J., Espinoza, S. y Larrea, D. 2012. Mapa de deforestación de las tierras bajas y los yungas en Bolivia 2000‐2005‐2010. FAN. Santa Cruz, Bolivia. 0 100000 200000 300000 400000 superficie 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 en hectáreas Ascensión de Guarayos Urubicha Ixiamas 8 Graph 2.  Authorized and illegal deforestation in Bolivia from 1997 to 2010 (Source ABT 2012) 2 Evaluation Methodology The midterm performance evaluation of the IBA project was carried out between August and December 2012, presenting the final report at the end of February 2013, almost 3 ½ years into project implementation. Table 3 presents the program for the implementation of the evaluation. Table 3. Program for the implementation of the midterm evaluation of IBA. (Source: Author) Activity May‐ Jul Ago Sep Oct Nov Dec Jan Feb Preparation of the Proposal                Award and signing of contract              Revision of secondary information               Field work         Analysis and interpretation of results and report writing        The evaluation methodology consists of three elements: (1) preparation and review of secondary information, (2) the implementation of the fieldwork and (3) the interpretation of information gathered and preparation of the report. The preparatory phase included coordination meetings and consultation with project staff. We examined various project documents, including annual and quarterly reports and other project progress reports, field visits and specific consulting and monitoring reports, reports of ABT and other secondary information in preparation for field work and site visits.   0 50.000 100.000 150.000 200.000 250.000 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Ha/year Authorized Deforestation Illegal Deforestation 9 Quantitative and qualitative methods of an exploratory nature were designed to respond to the key questions posed by USAID‐Bolivia in the SOW (Annex 1). Quantitative approaches included: a household survey designed to measure the changes in family income and the perception of the technical assistance received by the beneficiaries, and an evaluation scorecard used to survey Community Forestry Organizations (OFCs) and Local Forest User Groups (ASLs) to assess progress in the management of forestry operations. The survey was applied directly to project beneficiaries at the household level, whereas the scorecard was directed to organization representatives. Both were supported by complementary interviews with local authorities and community leaders. Qualitative methods included semi‐structured interviews with key informants and focus groups with local opinion leaders. Both qualitative and quantitative evaluation approaches were employed to assess training and technical assistance, particularly with regard to the application of improved forest management practices. The methodologies and tools used are explained in greater detail in Annex 4. Further details of the data collection process for the formal survey are presented in Annex 6. The methods and evaluation tools used and their relationship with each of the six major groups of questions can be found in Annex 16. Annex 17 contains details of the methodological tools used, including the questionnaire forms, guide to the semi‐structured interviews, scorecard forms, guidelines for the focus groups and forest management practices assessment checks.   In preparation for the field work and institutional visits for the evaluation, the team examined various project documents and other sources of secondary information, including annual reports, quarterly reports, the PMP and field reports as well as specific consultancy reports. A number of meetings were held with project staff to consult and to plan logistical aspects of the field work.  Upon preparation of the first draft report, meetings were organized to present findings to USAID and later to project staff to obtain feedback and to include suggestions in this report. During the preparatory phase, the questions were refined for the various research tools, In particular for the formal survey questionnaire. The evaluation team met on several occasions with the USAID Evaluation Officer of the USAID Program Office to discuss methodological procedures and to exchange views on the general approach to evaluation in light of the new USAID Evaluation Policy. The evaluation study employed a comprehensive and rigorous methodological design. The study has virtue and strength due to the comprehensive qualitative (focus groups and interviews) and quantitative (surveys and scorecard) approach employed, allowing a full analysis of the situation of the project, and generating precise findings based on information from survey data supported by complementary information gathered in the field. Table 4 presents a summary of the five instruments used in the evaluation.    10 Table 4. General Summary of the methodological design of the evaluation (Source: Author) Method Target group Sampling frame Sample size Beneficiary Household surveys Project Beneficiaries Two stage random cluster sample 473 Beneficiaries Semi-structured interviews with key informants  Local leaders from Grassroots Organizations  Municipal Technical Staff  Municipal authorities  Staff from ABT  Protected Area Authorities  Project partners and staff Qualitative exhaustive 73 cases Focus Groups Qualitative 6 groups Environmental assessment of forest management and biodiversity Communities with forest management plans, agro-ecological production and ecotourism activities Qualitative and quantitative Field visits to four forest management plans and nine agro-ecological production areas. Specialized Scorecard for OFCs and ASLs Community Forestry Organizations Census 15 cases A pilot exercise was conducted to test the implementation of the household surveys and the Scorecard survey. To test the household survey 12 questionnaires were completed in two communities in the municipality of Urubichá. The information was used to adjust the evaluation instruments in order to improve effectiveness of the data gathering process. For the formal survey work, Equipos Mori trained two groups of interviewers for three days prior to commencement of field work. Training included issues such as the correct completion of the assessment instruments.   Seven interviewers were trained in Rurrenabaque and nine in Guarayos. Annex 5 details the formula used to calculate the sample using the average comparison method with respect to a notional sample value. Initially we applied a two‐stage random sampling procedure. However, the random nature of the sampling procedure was affected as it was not possible to locate the beneficiaries listed by the project. As part of the process definition of the sample size, a first list provided by the project with 6,180 entries did not record the place of residence of a large number registrants and a majority of beneficiaries were repeated in the lists. A second list with 3,033 entries provided by PAI was refined to 1,473 individual beneficiaries. The difference in number is due to repetitions of names. The purpose of the exercise was to define a list of beneficiaries that could be located with the same probability of selection in the household survey. In section 3.1.1 we present our findings related to number of beneficiaries. The original sample size of 540, proposed in the methodology, was based on universe of 20,000 beneficiaries. Upon consultation with the evaluation official of USAID Bolivia, this sample size was maintained even though the universe was reduced to 1,473 beneficiaries. Of a total of 919 attempts to interview (62% of the sample universe) it was possible to carry out a total of only 473 interviews with 11 individual beneficiaries. Even with the extension of the sampling procedure from seven to twelve communities, it was not possible to locate the 540 beneficiaries agreed in the proposal submitted to USAID. This was because some communities had a sample size very close to the universe of beneficiaries with very few options for replacement. Even after refining the lists, some 2% of names were duplicated due to different spelling. A further 9% of beneficiaries could not be located due to either having left the community or not being known by community members, making it impossible to complete the original planned sample. To reach the number of 473 beneficiaries interviewed, the evaluation team exhausted the list of randomly selected replacements. The sample covers 32% of the universe of beneficiaries, generating results that can be applied with a very high level of confidence. Table 5 presents the breakdown of the distribution of samples for each of the communities studied. Table 5. Distribution of the samples in the surveyed communities (source: Author) Municipality Community Planned sample Implemented sample Guarayos Ascensión de Guarayos 60 74 San Pablo 0 18 Ixiamas Ixiamas 130 94 Macahua 0 33 Rurrenabaque Rurrenabaque 53 35 Puerto Yumani 18 12 Carmen Soledad 0 15 San Buenaventura Buena Vista 20 12 Bella Altura 0 15 Urubichá Urubichá 130 64 Yaguarú 130 90 Cururú 0 11 TOTAL 540 473 On conclusion of the field work, it was necessary to conduct a review of all the information gathered, encode and then analyze the resulting database. Annex 6 presents a detailed description of the field survey operation. This annex provides details of the planning of survey methodology, the scope and characteristics of the survey, the preparation and characteristics of the interviewers, the implementation of a pilot test to refine details and ensure proper implementation and the calculation of sample size and distribution between the communities of the project intervention. The report also presents details of the problems encountered by the survey team to find individuals initially selected. Finally, the report explains the procedures for processing information. Chemonics‐IBA explained that many of the beneficiaries were people who received technical assistance or training in locations other than their place of residence. As the survey was conducted towards the end of the logging season, it is possible that many of these people have returned to their place of origin or were working elsewhere. Details of findings from field incidents are registered in paragraph 4.4 of Annex 6.   12 2.1 Limitations in the methodology Many beneficiaries have also participated in activities of other institutions and projects limiting the degree to which rigorous and precise measurement of results can be assumed to be directly attributable to the project alone. The sample of beneficiaries was drawn from a list of people who received training, technical assistance or attending meetings. The project records the names of people who receive technical assistance and the place or community where this takes place. Training is usually not carried out in the communities where people live and the place of residence of people trained is not registered. Since the survey could only interview people who could be located and whose place of residence was registered, those who were trained outside of their communities could not be taken into account. Time constraints affected the execution of the household surveys due to the closure of the project offices. Despite these limitations, the team did not spare efforts to ensure methodological rigor, and the quality of the information gathered, returning several times to communities to find selected beneficiaries. Selection of communities was carried out on a random basis and did not take into account background information regarding project interventions in certain communities. The sample included an important number of communities in which PAI no longer worked. Previous knowledge of this situation would have allowed data to be interpreted more appropriately. The uncertainty of information regarding place of residence of a significant proportion of beneficiaries makes it impossible to define the total size of the universe of recipients, which affects the calculation of the selection weighting factor. To overcome this deficiency we used a self‐weighting sampling procedure, assigning proportionately larger samples to locations with higher concentrations of beneficiaries. Since the sample represents 32% of the universe of traceable beneficiaries it is considered that the results of this evaluation are valid and can be reliably extrapolated, although it is not possible to obtain absolute indicators relating to the universe of beneficiaries. 3 Analysis and Responses to key questions 3.1 Which planned results and outcomes have been achieved by the project to date, which support the achievement the SEGE strategic objective? PAI contributes to the achievement of three indicators of the strategic objectives of SEGE. At level of SEGE intermediate results, the PAI contributes to the achievement of two of these results. IBA contributes to three indicators of intermediate result 2 and two indicators for the intermediate result 4. The following table presents SEGE indicators and their relationship with IBA indicators.    13 Table 6. Relationship between SEGE indicators and IBA project indicators (Source: MTE) Hierarchy of Intervention Description of the intervention SEGE Indicator   IBA Indicator SEGE Strategic Objective Increase sustainable livelihoods through promotion of natural resource‐based business and productive opportunities Number of households with increased income as a result of USG assistance. 3.6.1 Amount of Sales generated with USG assistance 3.6.2 Number of hectares under improved Natural Resources Management (NRM) as a result of USG assistance. 3.1.2 Intermediate Result 2 Promote conservation and sustainable use of biodiversity goods and services to increase incomes and promote economic growth. Number of hectares in areas of biological significances under improved management as a result of USG assistance,   1.1.1 Number of people receiving USG supported training in natural resource management and/or biodiversity conservation 1.1.3 Number of people benefiting directly from USAID biodiversity, natural resources management and climate change assistance 1.1.4 Intermediate Result 4 Strengthen Bolivia’s ability to respond to the challenges posed by climate change. Quantity of greenhouse gas emissions, measured in metric tons CO2 equivalent, reduced or sequestered as a result of USG assistance in natural resources management, agriculture and/or biodiversity sectors. 1.2.3 Number of people with increased adaptive capacity to cope with impacts of climate variability and change as a result of USG assistance 1.2.4 Below, we present a summary of results of the analysis of the degree of achievement of indicators    14 Table 7. Progress reported by the IBA project compared to progress verified by the Midterm Evaluation (MTE) for the project indicators related to SEGE (source: MTE) Indicator SEGE No. Indicator IBA Target Progress reported by IBA Progress verified by Midterm Evaluation. Comments Number of people benefiting directly from USAID biodiversity, natural resources management and climate change assistance 1.1.4 20,000 6,738 (34%) The evaluation mission was able to validate 1,473 (7,3%) Direct Beneficiaries The means of verification established in the PMP does not permit validation of the results reported. Multiple registers of individual beneficiaries reduces the number that can be verified. Number of households with increased income as a result of USG assistance. 3.6.1 300 (100%) 330 (110%) 175 (58%) during the period 2011- 2012 Different measurement methods applied make direct comparison between reported and verified levels of achievement difficult. Amount of Sales generated with USG assistance 3.6.2 500,000 (100%) US$ 792,825 (158%) US$ 586,691 (117%) Two contracts predate the project and cannot be attributed to project effort. Increased value of sales due to rise in timber price should also be taken into account in the description. Number of hectares under improved Natural Resources Management (NRM) as a result of USG assistance 3.1.2 300,000 (100%) 481,858 (153%) 481,858 ha (153%) The area reported is in accordance with the indicator description. The MTE considers that progress is not entirely attributable to the project. Number of hectares dedicated to improved management under ecotourism 3.1.3 50,000 (100%) 57,173 (114%) 57,173 ha (114%) The area reported is in accordance with the indicator description. Number of hectares in areas of biological significance under improved management as a result of USG assistance 1.1.1 350,000 (100%) 167,401 (47%) The areas reported do not correspond to the description of the indicator in the PMP. Mistaken interpretation of the indicator in the PMP. Number of people receiving USG supported training in natural resource management and/or biodiversity conservation 1.1.3 3500 (100%) 1,844 (53%) 1,581(45%) Not verifiable. See comment for 1.1.4. Quantity of greenhouse gas emissions, measured in metric tons CO2 equivalent, reduced or sequestered as a result of USG assistance in natural resources management, agriculture and/or biodiversity sectors. 1.2.3 777,000 (100%) 107,675 (13%) 13% The project does not take into account all of the areas in which it works to assess its impact on CO2 capture or emissions. CO2 emissions from project areas are not taken into account but do not constitute part of the indicator. Number of people with increased adaptive capacity to cope with impacts of climate variability and change as a result of USG assistance 1.2.4 800 (100%) 0 (0%) 193 people (24%) The indicator is not correctly interpreted by either USAID or IBA. 15 3.1.1 Number of people benefiting directly from USAID in support of biodiversity conservation, natural resource management and climate change8 3.1.1.1 Comments on the indicator The indicator result forms part of the first objective of the project related to the strengthening of the municipal and local governance framework, but the indicator bears little relation to the result “implementation of local land management plans”.   The original list of beneficiaries provided to the evaluation team by the project contained 6,180 registers.  Of these only 3,033 could be verified.  The remaining registers did not record the address of the beneficiary and could not be verified. Of the 3,033 verifiable registers, 1,473 responded to individual names. 38% (564 of 1,473)9 of the beneficiaries were repeated in the registers. In an extreme case, one person was recorded 20 times in the lists (details of frequency of multiple registers are presented in Table 8). The interpretation of the indicator used by the project is derived from the sum of the number of people registered in the attendance lists for training events and provision of technical assistance.   There are therefore two sources of error: the means of verification is not valid and there are multiple registrations of individual beneficiaries. The interpretation of the indicator employed by the project was acknowledged by USAID‐Bolivia and approved by the Monitoring and Evaluation specialist of SEGE. The approval of the DQA (Data Quality Assessment) carried out on the project by the USAID M&E officer responsible for the project effectively validated data reported for all indicators. The DQA was conducted by e‐mail in April 2012. Additionally, the M&E officer made numerous visits to the offices to review the project monitoring system. Table 8. Frequency of multiple entries of beneficiaries’ names found in the records of IBA lists. (Source: MTE).                                                              8 Indicator 1.1.4 9 See details in Annex 17, paragraphs 1 to 4. Frequency of Repetition 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 19 20 TOTAL Number of Beneficiaries 909 222 134 79 40 24 16 15 14 3 7 1 2 2 1 1 1 1 1 1,473 Number of Registers 909 444 402 316 200 144 112 120 126 30 77 12 26 28 15 16 17 19 20 3,033 16 3.1.1.2 Analysis of the achievement of the indicator10 The quantitative analysis refers to a count of respondents who report having been contacted by the project for training or technical assistance11. Quantitative analysis revealed that in March 2012, the project reports 6,738 direct beneficiaries that are best explained as "records of attendance of training, technical assistance and other events." The lists submitted by PAI include as beneficiaries people participating in planning meetings, coordination, meetings, presentations, and dissemination of results. The number of direct beneficiaries (6,738) for the project reported in the quarterly report up to 31st of March 2012 represents the sum of the number of people registered on the list of training events and technical assistance and other events organized by the project. The number of beneficiaries reported by the project did not include indirect beneficiaries as defined in the PMP. There is a maximum of 1,473 direct beneficiaries that can be identified and traced which could be corroborated during the evaluation. However, of these it was found that a further 9% were not resident in the communities and 2% were repeated names with different spellings. (see Annex 7 Household Survey report, Table 3 Register of field incidents). Table 9. Number of beneficiaries by source of data (source: MTE). Source of Data Number of beneficiaries 1 Number of beneficiaries reported to 31/03/1212 6,738 2 First list of beneficiaries provided by the project13 6,180 3 Second list of traceable beneficiaries provided by the project. 3,033 4 List of individual traceable beneficiaries possible to be verified by the MTE 1,473 The evaluation team considers that the error in interpretation was not deliberate and did not have the intention to deceive. Neither is there any suspicion of collusion between USAID and Chemonics. But there was a joint misinterpretation of an indicator which is, in any case, poorly formulated.                                                              10 Annexes 5 and 6 show a detail of the methodology used to calculate the sample of beneficiaries surveyed and a report of the execution of field surveys with a field event log. 11 In this case, any respondent who has indicated that they have received at least one training or technical assistance in the areas of agro‐ecology, tourism, forestry, biodiversity monitoring and forest management is counted as a beneficiary. 12 According to data presented by PAI later, the number only amounted to 3,340 beneficiaries.   13 The first list of beneficiaries sent by PAI consists of records of people who received technical assistance and training in natural resource management (6.180 records). To locate people in their place of residence at the time of the survey, we were asked to submit the list of project beneficiary records with the place of residence. This second list contained 3,033 records. 17 3.1.2 Number of households with increased income14 3.1.2.1 Comments on the indicator The evaluation team used a methodology following the guidelines established under the World Bank Program for the Improvement of Surveys and the Measurement of Living Conditions in Latin America and the Caribbean (MECOVI15) used in Bolivia by the National Statistics Institute (INE) to measure income rather than using the unconventional approach recommended by the World Bank to measure income levels. The method used by the project, as specified in the PMP, estimates income based on coefficients of employment and average income per person (wages) according to productive activities. These coefficients are applied to sales values. 3.1.2.2 Analysis of achievement of the indicator16 The method of calculating increase in income used by the evaluation team was based on direct estimation, in accordance with the methodology presented to USAID for this evaluation. By extrapolation to the universe there were a total of 970 households17 from the 1,473 beneficiaries due to various cases of more than one beneficiary living in the same household. Of these, 283 households (29% of 970 households) experienced an increase in income of which 174 households (20%) achieved an increase of income equal to or greater than 5% (income equal to or greater than Bs 43,759) taking as the base year 2010. The average income ex post, since the intervention of the project is Bs 43,525 (Bs 3,627/month). Ex ante household income in the base year was Bs 41,676, resulting in an average annual increase of 4.4%. The municipalities with the largest increase in average household incomes are Ixiamas (6.4%. increase from Bs 50,930 to Bs 54,166) and Guarayos (6.1% increase from Bs 46,729 to Bs 49,576).   These municipalities are important centers of timber production, with incomes positively affected by the rise in timber prices in the national market. Other regions have lower rates of growth in household income: Rurrenabaque (3.7% from Bs 34,110 to Bs 35,382), Urubichá (2.1% from Bs 36,490 to Bs 37,265) and San Buenaventura (1.0% from Bs 33,030 to Bs 33,344).                                                                 14 Indicator 3.6.1: Number of families with income increased as a result of the assistance of the U.S. government. 15 Monitoring Poverty and Social Indicator. Program for the Improvement of Surveys and the Measurement of Living Conditions in Latin America and the Caribbean (LSMS).   http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/EXTLACREGTOPPOVANA/0,,contentMDK:20886217~pagePK :34004173~piPK:34003707~theSitePK:841175,00.html 16 The stated income is based on the answers provided by the beneficiaries in Module 2, "Employment and Labor Income" on the report card survey. A copy of it is found in Annex 17. 17 387 households were surveyed, occupied by the 473 beneficiaries interviewed (in some cases more than one beneficiary per household). Extrapolation of this study universe results in a total of 970 homes. 18 Table 10. Households benefiting from increased income with extrapolated data by municipality. (Source: MTE) Municipality Number of households with no increase in income. Number of households with increase in income of less than 5% Number of households with increase in income of 5% or more Total number of households Guarayos 77 5 7 89 Urubichá 370 60 92 522 Ixiamas 113 26 61 200 San Buenaventura 22 6 2 30 Rurrenabaque 105 12 12 129 Total 687 109 174 970 3.1.3 Volume of sales generated18 3.1.3.1 Comments on the indicator It is considered that this indicator shows nominal type information by failing to consider the effect of the timber price increase in Bolivia19. The effect of increased sales should use a price deflator to reduce the effect of "inflation" in nominal sales. According to the interviews conducted in the field, there are few cases of Community Forestry Organizations (OFCs) or Local Social Groups (ASLs) maintaining long‐term contracts with the same buyer. The change in customer should not be reported as an incremental sale, but as the difference in income between the new contract and the previous one, using the same base price to measure the actual effect of the increase or volume to provide a comparable basis of measurement of project impact. 3.1.3.2 Analysis of achievement of the indicator The project reported incremental revenue of USD 725,185 of which the evaluation team encountered two contracts that need to be explained in greater detail by the project: 18% (US$ 133,019 of US$ 725,185) of the reported incremental sales for the community correspond to sales by Cururú to CIMAL, their buyer for several years, and 8% (USD 60,057 to USD 725,185) in incremental sales which correspond to sales from the Yaminahua‐Machineri TCO to the company IMAPA in Pando, their longstanding buyer. It is therefore considered that a total of 26% (18% + 8%) of the increase in volume of sales reported is not attributable to the project because sales were agreed upon before the project. Additionally, the indicator does not consider the increase in the price of wood which has an effect of increasing sales value not attributable to the project. The definition of the indicator in the PMP accepts the inclusion of these cases, although the evaluation team does not consider this attribution to reflect the impact of the project.                                                              18 Indicator 3.6.2 19 Since 2011, new control procedures in place for wood origin for the ABT have produced a fall in the supply of wood and it has affected "naturally" in the increase of prices in the local market. This increase is further supported by the growth of the construction sector that demand different wood products 19 3.1.4 Number of hectares under sustainable forest management plans20 3.1.4.1 Comments on the indicator The definition of the indicator is vague and permits attribution to the project of areas that, in the view of the evaluation team, are not necessarily attributable to the project. The description of the indicator signals that the area to be reported is based on the area under PGMFs, while the rationale of the indicator is based on the area under operational plans. This criterion, in the view of the evaluation team, is not considered correct, although it is common practice in forest management projects to attribute the entire area under the project management plan. Areas under management plans are at least 20 times larger than the areas of annual intervention the sum of which should, in the view of the evaluation team, represent the area actually attributable to the project. According to the PMP, the area under forest management approved by previous USAID projects (BOLFOR I y II), can be included if some support or technical assistance has been provided to the management of the area. The definition of the indicator allows the attribution of the entire area of such management plans to the project and even specifies these areas in the PMP. 3.1.4.2 Analysis of the achievement of the indicator The 459,254 hectares are reported as a result of the technical assistance provided to different community forest management plans by the project. The stated goal is 300,000 acres under management plans. According to the specifications of the PMP, the area reported by the project is correct. The indicator sums the entire area under forest management plans, when in most cases, the intervention has been in annual logging areas (POAF). From 15 interviews with OFCs, ASLs or Associations supported by the project, in Ixiamas, Ascension de Guarayos, Urubichá and Yaguarú, 60% (9 interviews) indicated that they believe that the support of the PAI had not been decisive for the approval of general management plans. Whereas 87% (13 individuals) considered the support had not been decisive for the approval of annual harvesting plans. Under this definition of the indicator, the project has exceeded the target set to be achieved in terms of the area under forest management plans by 153%. Although the PMP attributes the entire area to the project, in the opinion of the evaluation team is not considered to be correct for the reasons stated above. It is suggested that the definition of the indicator used by the project should properly reflect the impact of the project in achieving the targets set in the PMP. This position is confirmed by the members of the OFCs and ASLs who consider that the project has relatively little effect on the approval of management plans. 3.1.5 Hectares dedicated to the improvement of natural resources for ecotourism, as a result of USAID assistance21 3.1.5.1 Comments on the indicator Like other indicators the description of the indicator in the PMP can attribute all of the area under tourism management to the impact of the project.                                                              20 Corresponds to indicator 3.1.2: hectares under forest management plans timber and non‐timber as a result of USG assistance 21 Indicator 3.1.3. It is not an intermediary results indicator of SEGE. 20 3.1.5.2 Analysis of achievement of the indicator The project has established 15 agreements with tour operators to enter a municipal certification process called "green action", of which eight achieved certified status. Through this certification, tour operators, in partnership with local communities, have shown improvements in codes of conduct for tourists, protecting the natural resources of the areas visited. The project reported that certified areas under improved tourism management correspond to 57,173 hectares. It should be clarified that these areas were already under management before admission to the ecotourism project. However, the definition of PMP allows the project to report areas for promoting the adoption of improved practices in the areas22. 3.1.6 Number of hectares in areas of biological significance under improved management as a result of the support of USAID23 3.1.6.1 Observations The table entitled "Performance Indicator Values" presented for this indicator in the PMP includes areas in the municipalities of Ascension, Urubichá, Guarayos and the Wildlife Reserve Rios Blanco y Negro, totaling 300,000 hectares. It also includes as an indicator for conservation areas within PDCPs and PMOTs.   The total area under forest management plans do not qualify to be included under the definition of the indicator as defined in the PMP, since the function of these areas is primarily for production rather than for conservation. In addition, the evaluation team considers that this indicator bears little relation to the result of capacity building for territorial management.   3.1.6.2 Analysis of achievement of the indicator The project reports having supported 167,401 hectares, including areas under forest management in the RVSRByN. The reported area does not correspond to the definition of the indicator presented in the PMP. The total area over the life of the project is 350,000 hectares. The area corresponds to the indicator reported for areas under forest management and should be added to this indicator. Where the project has supported the implementation of municipal or community conservation areas, the project has not reported progress made under this indicator. Within the forest management area, the project could legitimately take into consideration areas destined for conservation. USAID and Chemonics have agreed on this interpretation of the indicator. 3.1.7 Number of people receiving training in natural resource management and / or conservation of biodiversity as a result of the support of USAID 3.1.7.1 Comments on the indicator24 This indicator is part of Result 1.1 "Municipal Territorial Management Plans under implementation in selected municipalities." The training provided does not correspond to the development and implementation of municipal territorial management plans. It was not possible to verify the number of people trained since no record of residence of the person is registered in the database. The lists do not constitute, therefore, an objective source of verification. Peoples’ addresses are not registered due to                                                              22 Refer to second set of questions to Chemonics, answer number 7 in Annex 20 23 First indicator 1 SEGE intermediate outcome indicator 1.1.1 corresponds to PAI. 24 The second intermediate result indicator corresponds to one of SEGE PAI indicator 1.1.3 21 the practice of organizing training in the urban centers and not necessarily in the communities of residence of the participants. This prohibits the location of beneficiaries and the verification of their participation in training courses 3.1.7.2 Analysis of the achievement of the indicator The definition of the indicator contains two elements: the quantity (number of people trained) and the quality of training (if the person has completed more than eight hours of instruction). The stated goal is to reach 3,500 people trained. The project reported training 1,844 registers of which 1,581 are individuals trained up to March 31, 2012. The household survey based on beneficiary lists provided an indication of the beneficiaries’ perception of the quality of training and technical assistance provided in the areas of forest management, biodiversity monitoring and tourism. Table 11 summarizes the respondents’ perceptions of the value of training provided. Table 11. Perception of household heads regarding project training (source: MTE) Area of Training % of the curriculum completed % of beneficiaries that consider the training useful % of people and frequency of use of the training provided A lot Little None Tourism 65% 100% 74% 21% 4% Silviculture 59% 100% 62% 24% 14% Monitoring of Biodiversity 55% 99% 54% 35% 11% Forest Management 59% 98% 52% 31% 16% Agro-ecology 62% 98% 46% 32% 21% Average 58% 99% 57% 29% 14% Extrapolated Total 1.280 729 371 179 The training had greatest impact in tourism, while the training was least valued for the agro‐ecology component and for biodiversity monitoring. Of the 473 direct beneficiaries interviewed, 404 (86%) report having received training from the project. Extrapolating the results, this implies that 1,280 of 1,473 individuals reported having attended at least one technical training activity. Upon cross checking the project lists, several cases of people who asserted that they had not received training had signed the lists of attendance of training events. In one community training was suspended due to massive engagement in illegal timber sales, providing a clear motive for declaring that no support had been provided by the project25. On average only 58% of the minimum training content was acknowledged to have been delivered to the respondents in the various topics. This percentage represents the extent to which the thematic content (or curriculum) offered by the project has been completed.  This finding suggests that training activities were dispersed and poorly integrated. While almost all respondents positively valued the content and application of the training, it was found that 57% of the beneficiaries used a lot of the training provided (729 of 1,280 individuals extrapolated to acknowledge receiving any training or technical assistance from the project), 29% (371 individuals of the 1280) acknowledged giving little use to training and 14% (179 of 1,280) made no use of training provided.                                                              25 For further details refer to annex 19, additional questions answer 1. 22 As for the quality of training, the interviews and the application of the Scorecard were identified that newly created OFCs and the ASLs value the technical support of the project, while those with several years of operation indicated that the training was "more of the same" given by other projects or previously by BOLFOR. On the subject of agro‐ecology, community members criticized the quality of training, indicating that the technicians did not dominate the subject matter of the training offered; the field visits were sporadic (three or four visits during the life of the project); the support was untimely and seed was provided too late to be of use; quantities of seed and plants were insufficient to make a difference and of poor quality (in Ixiamas).  Many people criticized the project practice of organizing training courses in the urban centres, indicating that, in general, expectations were not met (contrasting the finding from the household survey). By contrast, respondents of the producers association in Guarayos highlighted the value of the support offered as part of an inter‐institutional effort to coordinate actions. Farmers reported that rice yields had doubled using improved varieties and better practices. The number of individuals trained (1,581) represents 86% of the reported number (1,844) due to the interpretation of the indicator as number of registers listed. The lists do not record place of residence. To validate this number it is necessary to be able to identify and locate the person registered. To date, this definition has been accepted by USAID and an agreement between USAID and Chemonics recognizes that this definition of the indicator, effectively recognizing a person trained several times as more than one person being trained. The evaluation mission considers that this is a mistake and that none of the records of people trained can be verified because of the limitations mentioned in the registers. 3.1.8 Total emissions of greenhouse gases reduced or sequestered as a result of USAID support in the areas of natural resource management, agriculture and / or biodiversity26 3.1.8.1 Comments on the indicator The evaluation was able to identify some limitations in the method of calculation of CO2 sequestered and reported.  This is mainly due to the form of calculation which is directly proportional to the area under forest management. However, the level reported by the project is based on the USAID carbon calculator.  The evaluation team considers that this gives rise to errors in estimation of the amount of CO2 sequestered as this is proportional to the area under improved management that can be attributed to the project. It should be noted that the variables used to calculate the quantity of CO2 are not reported. It is considered that the interest of the project indicator is not only the amount of CO2 sequestered but rather the balance between CO2 sequestration and release. The indicator does not consider the release of CO2 as a result of forest fires, deforestation and logging. The evaluation team suggests using the CO2 balance as an indicator to measure the impact of the project in this regard. In addition, the project should take into account additional areas of project intervention at the landscape level in order to calculate the carbon balance, especially for changes in agricultural production activities and ecotourism.                                                              26 Corresponds to indicator 1.2.3 IBA 23 3.1.8.2 Analysis of range indicator The goal aims to achieve a reduction of 769,324 tons of CO2 sequestered over the life of the project. PAI reports 107,675 tons of CO2 reduction due to the results of improved forest management of 397,837 hectares (14% of the target). The contribution from areas under improved management by ecotourism and areas of PDCPs, as defined in the PMP indicator Project, are not taken into account. 3.1.9 Number of people with greater ability to cope with the impacts of climate variability and change with the support of USAID27 3.1.9.1 Comments on the indicator The project interprets the indicator as number of people implementing adaptation actions on climate change and not number of people with improved capacity. The indicator should be based on what is to be achieved in terms of behavioral change, just as the project has interpreted the indicator, rather than the description in the PMP.   However, as it stands, the project has misinterpreted the definition presented in the PMP. 3.1.9.2 Analysis of achievement of the indicator The target for this indicator is to reach 800 people trained to cope with the impact of climate change. The project does not report any people trained. The project database registers 626 people trained in the field of climate change, representing 475 individuals. However, this number is not objectively verifiable. From the sample of 473 people interviewed in the evaluation, 111 declared that they had received training in climate change.  Of these 58 indicated that they had received training from the project. Upon extrapolation to the universe sampled the evaluation team was able to verify 193 individuals trained in this area. With regard to perception of the quality of training, the survey results indicate that 43% reported making good use of the knowledge acquired. In addition it should be noted that further training was performed by WCS in the Pilon‐Lajas Biosphere Reserve and in the Municipality of Rurrenabaque that are not reported under this indicator. 3.2 Is the IBA project’s integrated approach working? There are various different understandings of the integrated approach to be adopted by the project.   The initial understanding was for the project to adopt a holistic approach to natural forest management which means taking into account the use of timber, non‐timber forest products and environmental services, with the aim of conserving biodiversity. However, it was recognized that this approach alone would be insufficient to address the threats to biodiversity, namely deforestation and forest fires, driven by additional external factors. It is considered that the decision to extend the understanding of integrated management to the landscape level by including work in agro‐ecology.                                                              27 Indicator 1.2.4 IBA 24 Despite adoption of the revised integrated approach, no discussion was generated by promoting the exchange of criteria between the different thematic areas of the project or between the institutions of the consortium. There are several factors that have limited the effective application of an integrated approach in practice:  The management structure has separated institutional functions with different institutions responsible for the implementation of different components.  The coordination mechanisms have been ineffective at both the local and central management levels.  The geographical dispersion of institutional bases (U.S., La Paz, Cochabamba and Santa Cruz) and the field sites.  The specialization of indigenous production systems induces the abandonment of integrated traditional management practices.  The failure to work in a structural way with key actors (especially colonist farmers) at the territorial level has limited the integrated view of territorial management.  The socio‐political context has limited the degree of integration of the initiatives promoted by the project with the local political and institutional frameworks. 3.3 To what extent are the indicators attributable to the IBA project? We have identified several problems with the indicators that represent a recurring theme in several aspects of the project. As seen in section 3.1 above, questions are raised regarding the attribution of the achievement of many of the indicators reported by the project (area under management, increased sales, amount of CO2 sequestered, overestimation of number of beneficiaries, number of households with increased income, etc.). This does not reflect a problem in the implementation of the project but rather reflects deficiencies in the design and selection of indicators defined in the PMP. Deficiencies found by the mission include:  Indicators are used which do not reflect the character of the result.  Inappropriate indicators have been selected to measure the degree of achievement of results and do not apply SMART criteria.  Some indicators are not directly attributable to the project even though the definition used in the PMP assigns attribution (value of sales and area under management).  In some cases the source of information presented by the PMP is not objectively verifiable (lists of beneficiaries).  Other indicators do not present the data that is of direct interest to the project (e.g. balance CO2 rather than total sequestered CO2).  There are no indicators for objectives 1 and 2 and for key results (e.g. institutional strengthening and local governance). 25  At the highest level, indicators are not used for the conservation of biodiversity (best represented in terms of reduced fires and deforestation).  There is no indicator for improvements in the livelihoods of local populations (this is not adequately reflected in measures of increased income).    26 Table 12. Attribution of indicators to project activities (Source: MTE) Indicator SEGE No. Indicator Comments Number of people benefitting directly and indirectly from USAID biodiversity and NRM assistance 1.1.4 The PMP uses the attendance lists as a means of verification but these are not objectively verifiable due to the failure to include the addresses of beneficiaries. These cannot therefore be considered to be objectively verifiable. Indirect beneficiaries have not been included. The incorrect guidance from the donor, USAID, of how to report the indicators induced the overestimation of this indicator. Number of households with increased income as a result of USG assistance 3.6.1 There has been a general increase in the wage level dictated by the government and, in addition, there has been an increase in the value of timber and other raw materials that have influenced the increase in income attributable to the project. However, the PMP descriptor indicator does not provide a correction factor of these factors. Amount of sales generated by firms receiving USG assistance 3.6.2 The increase in volume of sales is reported based on new contracts. The new contracts include buyer changes that do not reflect increases in timber volume sold. The indicator on the PMP supports new contracts of this nature but not attributed to the project. Number of hectares under sustainable forest management plans for timber and NTFPs as a result of USG assistance 3.1.2 The description of the indicator in the PMP for areas under sustainable management includes areas that were supported by BOLFOR I and II. The description also allows for the inclusion of the entire area covered by a management plan even if the project has not supported the development of the management plan. The MTE questions the direct attribution to the project permitted under the PMP. Number of hectares under improved natural resources management for ecotourism as a result of USG assistance 3.1.3 As mentioned in section 3.1.5 above, a number of ecotourism management areas were established prior to commencement of the project. But the measure description in the PMP validates the inclusion of these areas. The MTE considers that these are not directly attributable to the project. The certification is more an indicator of quality of management practices than quantity. This is not reflected in the indicator. Number of hectares in areas of biological significance under improved management as a result of USG assistance 1.1.1 As mentioned in section 5.1.6, there is an error in the interpretation of the indicator by the project. Areas that could be included under this have not been and areas have been included that are not in accordance with the indicator description in the PMP. Number of people receiving USG supported training in natural resources management and/or biodiversity 1.1.3 The means of verification from project lists of beneficiaries who have attended training events is not considered to be objectively verifiable. Since none of the people who have received training can be located, the MTE is unable to validate the number of people trained by the project. This is a failing in the PMP which was prepared by the project and approved by USAID Bolivia. Quantity of greenhouse gas emissions, measured in metric tons CO2 equivalent, reduced or sequestered as a result of USG assistance 1.2.3 The indicator described in PMP attributes CO2 sequestration of all areas under management to project actions. Some of the areas as mentioned under indicator 3.1.3 are not considered to be attributable to the project even though the PMP permits the inclusion of areas supported by other projects. The indicator does not take into account CO2 sequestered in other areas of project activity nor does it take into account the CO2 emissions of the other areas of influence. Number of people with increased adaptive capacity to cope with impacts of climate variability and change as a result of USG assistance 1.2.4 The project has interpreted the indicator as a change in practices whereas the description in the PMP is the number of people trained in climate change. The MTE considers that interpretation of the project to be more appropriate. The project has not reported progress even though some progress has been made by the project in relation to the indicator as described in the PMP. 27 3.4 Since the IBA project is essentially an institutional strengthening project, how effective has the contractor been in strengthening local institutions and putting in place mechanisms to monitor and evaluate progress in institutional strengthening? The project has focused its actions on economic production with reduced emphasis placed on institutional strengthening. Objective 1 is intended to focus mainly on strengthening the capacity of local governments for territorial management and development of local governance for conservation. On the other hand, Objective 2 of the project is to promote both capacity building of technical and organizational management and the sustainable use of natural resources of both indigenous and producer organizations. However the indicators used to measure the achievement of the results related to these objectives are more oriented to capacity building in natural resource management than institutional strengthening. It is considered that the wrong selection of indicators has distracted the attention away from institutional strengthening concentrating actions on natural resource management. The project has focused on a limited number of approaches to capacity development and has not taken full advantage of the wide range of tools and capabilities needed to strengthen local and national institutions. The project has not implemented a system of monitoring and evaluation of progress in institutional strengthening and, as found during the field visits and interviews, the project has not had a significant impact in this regard either with municipal governments or with the indigenous organizations. Of the seven municipalities supported by the project none of them are implementing territorial management plans or using these plans in the prioritization of projects in the PDMs or POAs.   The three indigenous organizations, COPNAG, CIPTA and CRTM have not developed their capacity to implement actions in terms of territorial management and control. There are, therefore, important challenges remaining for the institutional strengthening in territorial management. As noted earlier this has been complicated with the current socio‐political context in Bolivia surrounding the conflictive issues of forests, land and the relations of political power regarding these resources. The failure to implement an appropriate system of monitoring and evaluation of institutional strengthening project is not just a failure of the executor. The responsibility for the development and approval of the indicators used to establish the basis for the monitoring system is shared between USAID and Chemonics, as the monitoring and evaluation instruments, the PMP, has been adopted and used by both. It should also be noted that in addition to the process of institutional strengthening, according to the stated objectives of the project, USAID has had his own agenda to strengthen institutions considered of strategic importance, which has often failed to coincide with the needs of the project to meet targets. CADEFOR and FDTA Valles have benefited from this type of action in the framework of the project. The project has developed a plan to strengthen the organization which has not been followed up. 3.5 To what extent has the IBA project goal of using local communities to monitor biodiversity been effective? The data collection for monitoring wildlife has been overseen by Wildlife Conservation Society (WCS) using a system of records based on evidence of the presence of key species. In one way or another, this 28 represents a practice that the community members have been implementing as part of their daily routine in assessing hunting opportunities. It is, therefore, not considered a burden to gather this information. However, the data collection of the flora to evaluate the dynamic of forest response to management interventions is more complicated, but is possible with a little training and support. This work is supported in communities by IBIF, a project subcontractor. One of the complications of the work is the special needs of data analysis and interpretation of the results. Both WCS as IBIF have carried out the analysis and interpretation of information gathered by the communities. The idea is that monitoring information is analyzed and used to improve forest management decision making in each of the management units has yet to be put into practice. In Ixiamas, the ABT has conditioned authorizations of annual logging on the implementation of permanent sample plots (PSPs) to analyze forest dynamics. This has generated the main incentive for conducting monitoring of the flora in this area. There is no similar incentive in Guarayos nor long‐term initiatives to continue the monitoring system of wildlife. In communities that receive tourists interested in seeing wild fauna monitoring would make more sense to create an incentive for implementation. The monitoring should be used to control the intensity of exploitation of species and to control the definition and establishment of conservation areas in sites with high diversity and levels of endemism.   Such criteria should be used to protect water courses and other important ecological areas within forest management areas. However, so far, the application of monitoring has not yet been put to such practical uses and has been considered until now largely ineffective. 3.6 Given Bolivia’s position on GCC in Durban, can the project work in these types of activities? Since Cancun, the Bolivian government's position in discussions at the COP28 has become more proactive in establishing an alternative mechanism for REDD. The current proposal from the Bolivian government, first presented in Durban and later consolidated in Doha, has several elements in common with the original project approach, emphasizing territorial management and governance at a local level, and supporting institutional strengthening of local autonomous bodies: municipal governments and indigenous organizations, as well as small farmer and intercultural organizations responsible for local management of natural resources. It is then considered that there are good conditions for the project to enter into dialogue with local governments and social organizations, which can develop proposals which can be supported by the project, as part of the national climate change policy. There are then promising conditions to support initiatives within this framework, by working with municipal and other territorial based organization. In this way the project could support the strengthening of the institutional capacities for territorial management, focusing in particular on organizational strengthening and the development of specific management instruments to operate at a local level in the control, regulation, supervision and monitoring of territories. The four municipal governments, the three indigenous organizations and representatives of the intercultural federations all expressed interest in collaborating with the project, indicating that for them it was not an issue to work with finance from USAID.                                                              28 Conference of Parties 29 4 Findings of the Mid‐Term Evaluation 4.1 Project Management   The structural design of the project presents problems in understanding the logic of intervention. This is principally due to the disconnection between the results to be achieved and the strategic objectives of the project. Consequently, the project can fully meet the performance indicators, but may not necessarily have contributed to the achievement of project objectives.   This weakness is further compounded by the complete lack of indicators for strategic objectives. With regard to the organizational chart of the project, we found that the relationship between partners or contractors is characterized by informal relationships with parallel lines of command with confused responsibilities. The direct relationship between CADEFOR and USAID, in particular, has weakened the leadership position of the COP and the poor definition of roles and responsibilities and has made it unclear when certain organizations are acting as contractors, partners or beneficiaries. The payment system has not allowed implementers to demand results from partners and contractors in some cases. The implementers of the agro‐ecology component had little experience in the areas of work, resulting in the adoption of misguided implementation strategies. In particular, the out‐sourcing to a wide range of entities over which it has not been possible to apply normal conditions of control, has complicated project management and generated high administrative costs. The monitoring system of the IBA project has a very well structured database which connects the indicator lists reported with scanned images of attendance lists with the individual signatures of participants.   Additional information includes details of the training programs delivered, documents produced, and geo‐referenced field sites that allow ready access of available supporting information. However, failure to ensure consistency between the reports generated and the description of the PMP indicators and verification with supporting field data has meant that the monitoring system fails to fulfill the essential function of measuring actual progress of the project. The problems of interpretation and attribution of the indicators presented in the PMP approved by USAID have been approved by the USAID‐Bolivia office.  The evaluation team was informed that the person responsible for M&E of SEGE in the USAID‐Bolivia office approved the PMP and interpretation of performance indicators and then carried out a Data Quality Assessment, validating the information contained in quarterly progress reports without observation. The officer in question has not carried out a single field visit to project sites. The project M&E system did not identify key failings in project implementation and was unable to detect the main causes of dissatisfaction among beneficiaries. The failure to detect these shortcomings is indicative of the weakness of the monitoring system to fulfill its purpose. Furthermore, the M&E specialist of SEGE should have been able to detect in an opportune way the weaknesses and limited progress made by the project in achieving its indicators.   Those responsible for monitoring, both internally for the project and for USAID‐Bolivia, clearly did not pay due attention to the reading of the PMP, giving rise to inconsistencies in the description and interpretation of the indicator and the way in which data is collected. 30 4.2 Institutional Strengthening   Institutional strengthening is one of the main strategic areas of IBA, forming the basis of objectives 1 and 2 of the project. Despite the degree of importance that institutional strengthening and capacity building have for the project, indicators were not developed to measure progress in institutional strengthening. It is considered that the use of the number of people trained is not a suitable indicator of the effectiveness of training provided. Indicators actually used are more focused on natural resource management than on institutional strengthening. Consequently, the mistaken selection of indicators has diverted effort from the institutional strengthening to actions related to natural resource management. Although the project should support local governance and territorial management, none of the municipal governments have developed capacity in these areas as a result of project support.   The municipal governments of Urubichá and Rurrenabaque acknowledged receiving training on specific issues. The municipal governments of Rurrenabaque, San Buenaventura and Ixiamas had little knowledge of PMOTs developed with support from USAID Landscapes program. Similarly, the municipal government of Ascension of Guarayos had little knowledge of the PMOT developed with the support of the Departmental Government of Santa Cruz. Only in Rurrenabaque has the municipal government used the planning criteria developed in PMOT in municipal management. The Central Organization of Tacana Indigenous Peoples (CIPTA) and the Confederation of Native Peoples of Guarayos (COPNAG) reported in interviews that they had not received support from the project, although these are specifically mentioned as target beneficiary organizations. Only T'simane Mosetene Regional Council (CRTM) reported receiving support on specific issues. Federations of intercultural communities have not been taken into account by the project, although they expressed interest in working with IBA. Some individual members have, however, benefitted from project activities. Although the proposal declares that the project would develop a program of technical assistance based on the need for institutional strengthening of productive organizations, in practice the project has developed standard training programs for OFCs and ASLs. These organizations indicated that the training did not respond to the specific needs of different organizations. The project has not promoted horizontal training through exchanges between beneficiaries (except in the ecotourism component). The principal demand of the community forestry organizations has been to strengthen the management and business administration skills of the organization, especially those which are more established. The support in agro‐ecological production has been on a personal rather than an organizational level which has been costly and inefficient. The use of direct financial transfers through the FDA for institutional strengthening has mainly benefitted CADEFOR, and to a lesser degree FDTA Valles. However, support has not been conditioned on the implementation of the institutional strengthening plan for the sustainability of the institution. At the same time CADEFOR has provided services to a beneficiary OFC under an independent contract and within management areas supported by the project, which creates a potential conflict of interest. It is considered that direct transfer removes the incentive for the organization to be competitive and respond to demands and can lead to unfair competition with other local organizations. In the long term, this is not considered an appropriate mechanism for strengthening local institutions. Furthermore, from 31 the beginning, the project has included, in some cases, local institutions with little experience in the areas of action, especially in agro‐ecology. Where competitive tendering has been applied by the project, this has resulted in the granting of contracts to organizations with little experience in the project component of agro‐ecology. In addition, many of the activities that used international organizations at a high cost could have been carried out by local institutions. 4.3 Project Components The agro‐ecological component has not developed an adequate strategy aimed at reducing the rate of deforestation, has not worked with key stakeholders responsible for deforestation, and has not addressed the root causes that threaten the biodiversity, namely the expansion of commercial agriculture and livestock. The work of FDTA Valles and the implementing agencies of the agro‐ecology component has been oriented with a focus on production chains and has not paid due regard to the objective of biodiversity conservation. Staff changes and lack of clarity of goals have led to the promotion of agricultural activities with a focus on food security based on specific crops (rice, bananas, peanuts and sugar cane) rather than the development of alternative production systems to slash and burn agriculture or livestock production. The extensive range of coverage of the activities supported with technical assistance has meant that support to communities has been much diluted. Undoubtedly, the problems have been increased by logistical problems faced by the headquarters of FDTA Valles, ESAF and SID located 800 km from the work areas in Cochabamba. In interviews and focus groups in communities it was explained that technical assistance was delivered in two or three short visits to individual producers.  In northern La Paz in particular, the project has not worked with community organizations or coordinated actions with other entities. The evaluation team received many complaints from communities about technical assistance and quality of seed provided by the project. One of the positive aspects has been the coordination of support in northern La Paz with SERNAP to support cocoa producers in the development of agroforestry systems in neighboring communities to the Madidi National Park. In Guarayos actions led to the establishment of a coordinating committee to provide technical assistance to agricultural producers. Support for the non‐timber forest products, wild cocoa and Cusi oil, are promising initiatives in the Guarayos area. The project has focused support on tourism around Rurrenabaque, and extends to indigenous communities located in Madidi National Park and Pilon Lajas, Indigenous Territory and Biosphere Reserve in the municipalities of San Buenaventura and Rurrenabaque, and also includes the municipal protected area of Santa Rosa de Yacuma. Several informants highlighted the contribution of project support to the sector, attributed to the “Green Action Program" which has contributed to the progress of tourism in the region. Of the 25 tourism operators in the region, 20 are registered with the project, eight have received the Green Action Program certification as a result of 15 agreements of understanding with tourism companies.  70 people have been trained and 13 companies supported in the adoption of improved practices, including the employment of workers with signed contracts. The project reports 57,173 hectares under improved management plans for ecotourism conservation. Of the components implemented by the PAI, tourism has undoubtedly contributed most to the achievement of the project purpose in terms of its contribution to the local economy and biodiversity 32 conservation. Despite the success, it is clear that opportunities to replicate these experiences are limited by the specific characteristics required. There is also a question as to the attribution of the effects of the work of the project and if the increase in turnover of the tour operators is entirely due to project support or was a result of a growing trend in the growth of tourism in the region. As for the forest management component, through interviews with members of the OFCs and ASLs in Northern La Paz, it was observed that in most cases the same members take an active role in the execution of the forest inventory and logging work. However, in Guarayos, there are a number of OFCs where the members adopt an entirely passive role, waiting to receive dividends that are distributed at the end of the logging season, without participating in any management practices or decision‐making processes. Obviously, training provided to the first category of OFC has been helpful while participants in the second category have not made any use to the training received. During the interviews it was noted that the older, more established organizations perceived the need for more support in administration and business management and a need for advice regarding their interest in primary timber processing. It was noted that the training offered largely repeated the training received in BOLFOR and did not correspond to the needs of organizations. However the newly established organizations that have not developed prior experience in this area valued the technical training received from the project. From the analysis of the information provided by the ABT it was possible to verify that with respect to forest censuses, an activity in which many users have been trained, it was found that in both Ixiamas and Guarayos, high levels of error were reported in the estimation of the bole height as well as in measurement of diameters. Additionally, inspection in  the forest found excessive damage to remaining trees as a result of logging operations, suggesting that although training is valued it has not always been put into practice in the forest. Thus, despite the progress reported, there are several management practices that need to be improved upon. For example, the ABT in Guarayos mentioned that there is a need for training in species identification as most of the OFCs use different names for the same tree species during the census compared to the logging patio report presented when requesting Certificates of Origin (CFO). Another problem encountered, especially in Guarayos, is the failure to submit Annual Logging Reports (IAPOAF) to the ABT at the end of the harvest, which precludes them from activities the following year. Seven project beneficiary OFCs have up to six IAPOAFs pending. Another problem that has been identified is that OFCs are logging species that are not permitted. Most OFCs and ASLs indicate that they distribute income from forest management according to their internal regulations. In most cases this distribution is in the form of "rent" for partners. There are no investment plans to reinvest profit in productive or social development. Income is then primarily distributed among participating families. Organizations indicate that they have current financial statements, but that there is limited capacity of decision makers and organization members to interpret reports. There is limited accountability of the organization leaders to the members in the rendering of accounts and there are ubiquitous complaints of unaccounted expenditure. Forest management leaves no significant benefit in the primary production sector. The forest management activities generate mainly wages and rents, and do not generate a solid basis for social and economic development.  Few ASLs and OFCS have long‐term contracts with companies and / or buyers and there is a continuous 33 rotation of buyers with the associated danger of opportunism by buyers and rent‐seeking by ASL and OFCs. This results in reducing the possibility of establishing partnerships with potential for long term growth. 5 Conclusions PAI objectives remain valid and in line with national policies related to integrated forest management and territorial management expressed in the New Constitution and the Law of Mother Earth. Despite seemingly favorable conditions for the implementation of the project, the project has been unable to take advantage of this situation and project execution has been marked by weak implementation by Chemonics and difficult relations between USAID and the Bolivian Government. The implementer’s interpretation of USAID’s instruction to keep a “low profile” has resulted in the reduction of activities to strengthen the institutional capacities of indigenous organizations and municipal governments which formed central objectives in the implementation strategy. Although the conceptual design of the project is considered to be sound the design of the implementation strategy has serious structural problems which have affected the achievement of project objectives. The evaluation team identified limited progress in the achievement of nearly all of the objectives and results of the project. The evaluation team encountered limitations in the design of indicators and lack of indicators for key project outcomes.   For example, there is no indicator for biodiversity conservation (e.g. levels of deforestation) as a result of project activities. This is due in part to the logic used in the project implementation, which has not been designed to reduce threats to biodiversity or mitigate the drivers of these threats. Although project activities have led to an increase in incomes, it is not perceived that they have significantly improved overall living conditions in local populations The project design has not developed indicators for specific objectives affecting the implementation strategy. Consequently more effort was dedicated to meeting result level indicators and while ignoring the fulfillment of higher level objectives. The effect has been exacerbated because many of the results do not correspond to the specific objectives, especially for the first specific objective. In addition, indicators do not always represent progress in the achievement of the results that they are intended to represent. Productive organizations (mainly OFCs and ASLs) have been strengthened in their technical abilities, but issues related to the strengthening of administrative and financial management were not adequately addressed by the implementing partner. The adverse socio‐political situation at the national level did not allow the project to advance in the development of institutional capacities to deal with territorial management or the development of a local governance framework. As a result, no improvement is seen within the framework of local governance in these municipalities. Similarly, indigenous organizations which are divided and weak in their ability to perform their core functions in territorial management were not strengthened in their management and organizational skills. However, at a community level, management capacities through the development of local norms and regulations have been improved. Although the evaluation team found moderate improvements in income of beneficiary households, it is not considered that this constitutes a basis for sustained growth of local economies; and the increase in 34 income cannot be fully attributed to the project. OFCs and ASLs do not apply long‐term business strategies, and commercial agreements are agreed on opportunistic bases. The fourth objective of the project which proposes that the project should expand its area of geographical coverage to beyond the regions originally included in the project proposal is not considered appropriate under current circumstances.   It is considered to be more coherent to concentrate efforts and to find a workable operational approach to implementation. As a result of poor communication between the implementer and USAID on the definitions of indicators (which has been particularly weak in the reporting of the indicator of number of beneficiaries). More than half of the registers could not be verified due to failure to register home addresses of beneficiaries who could not then be located.  An additional number of individuals were counted on multiple occasions reflecting the effort dedicated to training and technical assistance rather than the number of individual beneficiaries. These two factors led to the overestimation of the reported progress, of the 6,738 (38% of the target of 20,000) reported beneficiaries 1,473 could be verified (7.3% of the target). The project reports progress of 38% (6,738 beneficiaries) of the total goal of 20,000 beneficiaries. The means of verification (project lists) do not track verification of reported indicators (this has been determined to be a significant weakness in the projects monitoring and evaluation system). The fact that this deficiency has not been identified earlier demonstrates the weakness of the internal monitoring of the project by Chemonics and external monitoring by USAID. Moderate salary increases were recorded in the project’s beneficiaries. The results indicate that the increases are consistent with the levels found by the evaluation team. However, although the definition of the indicator in the PMP allows this conclusion, the evaluation team considers that not all of the increase is attributable to the project. The project has exceeded the indicator related to increased value of trade, but considers that 26% of the reported value needs to be revised and compared against the definition of the indicator which does not incorporate a price deflator to neutralize the effect of price increases in timber as a result of market dynamics. In the opinion of the evaluation team the indicator definition is flawed and allows the project to report sales increases that are not necessarily attributable to the project. According to the project, the indicator of the area of natural resources under management has been exceeded. The evaluation team considers that not all the reported area should be attributed to the project as its activities were implemented mostly within annual harvesting areas (5% or less of the total area under management) and not within the entire area. However, the indicator description in the PMP allows this attribution. It is concluded that the indicator used by the project does not accurately reflect the project's impact in meeting the goals set in the PMP. The project reports that the indicator for improved ecotourism management has been achieved. It is considered that the indicator does not reflect the effort of the project which has been more qualitative in nature. A better definition of the indicator is needed to improve the measurement of change in the quality and quantity of the management area as a result of the project intervention. The Green Action Program certification scheme reflects quality of service, not area under improved management. The 35 indicator does not take into account the quality of services provided and focuses on the management area. The reported number of people receiving training in natural resource management and / or biodiversity conservation corresponds (again) to the number of registers rather than number of individuals. Therefore, the reported number is greater than the number of individuals effectively trained. Generally it is noted that training provided did not correspond to a clear strategy reflected in the quality of training and technical assistance, especially of the agro‐ecology component. The indicator bears no clear relation to the result. The way it is reported, it measures delivered efforts and not outcome of project activities. It is necessary to establish an impact indicator of training and not simply the number of trainees. Interviewees expressed a high level of satisfaction with the training and indicated a relatively high level of adoption, reflected in findings in the field for the forestry and ecotourism components. Training has rarely been integrated either within or between project components and the levels of completion of component contents have been low with the exception of ecotourism. Much of the training in forestry in particular has focused on technical aspects of management whereas beneficiaries demanded more support in administration and financial management. The reported amount of reduced emissions or sequestration of greenhouse gases corresponds to the reported total area under forest management. The indicator measures only the areas with carbon sequestration and not the carbon balance in the areas of intervention (which should include emissions). It would make more sense for the project to use a carbon balance measurement considering project objectives. There is modest progress in the fulfillment of the indicator. However, progress was not reported in all areas of project activity. The indicator should take into account the balance of CO2 emissions and not just reductions. There is no report on the number of people trained with USAID support to deal with the impacts of climate variability and change. The evaluation team verified that the project has trained 193 people in adaptation to climate change. The interpretation of the indicator used by the project was incorrect and underestimates the degree of achievement. The interpretation applied by the project is more appropriate but still understates the degree of progress which has not effectively been measured. The differences between the reported indicators and the values found by the evaluation team are therefore mainly due to the misinterpretation of the description of key indicators.   It is considered that project progress has also been limited by problems in the management system, which have been characterized by ad hoc relationships between USAID and subcontractors, confusing internal control lines, unclear roles and functions, an excessive number of contractors and subcontractors, weak coordination mechanisms and the use of funding (grants) mechanisms without considering the objectives and results to be achieved. Consequently partners / contractors have not responded to the project guidelines. These same factors have limited the degree of integration of the components of the project, creating islands of action. The original design of the project with wide geographic coverage has complicated its management, especially considering the political problems that have affected its implementation. It is considered appropriate to concentrate the geographic scope of the project. 36 The strategy of having local communities monitor biodiversity has been costly in terms of time and investment in resources for the communities. Additionally, the interpretation of the collected information requires external support from professionals, especially in the case of the monitoring of flora. Weaknesses are noted in the strategic focus of the project, especially in the agro‐ecological component which has not had a significant impact on biodiversity conservation. Inefficiencies were introduced in the implementation of this component by contracting implementing agents with limited previous experience or operational base in the area of work and / or limited knowledge of local realities. With the exception of Guarayos, where a local technical committee was formed to coordinate institutional activities, there was little coordination with local authorities and other development projects in the two zones. Main stakeholders responsible for forest degradation and deforestation were not taken into account in the project implementation. It is considered a correct decision not to continue the agro‐ ecology component in the way it was being implemented. The very promising results obtained by the ecotourism component should be noted which contributed very positively to the achievement of the project objectives. There is a high degree of compatibility of the project strategy with the proposal of the Bolivian Government presented in COP 17 in Durban and COP 18 in Doha. It is considered that the project is oriented in accordance with the Government’s proposals to promote integrated forest management and strengthening capabilities of local organizations for territorial management. 6 Recommendations29 To make the project more effective in its implementation and in the achievement of its objectives, Chemonics should redefine the strategic guidelines of the project, establishing a coherent hierarchy of objectives and results with appropriate indicators. The PMP should be reviewed, so that it better reflects the objectives and results to be achieved by the project in accordance with the revised approach. Similarly, the revised PMP must redefine the indicators and levels to be achieved by the end of the project. In particular, it is recommended that Chemonics design a program to promote practices that address the threats to biodiversity, while continuing with the strategy for the valuation of forest goods and services. Chemonics should perform an analysis of the drivers of deforestation by different stakeholders including in particular the agribusiness sector, analyzing the incentive structure, especially of those incentives defined by the public policy framework. Chemonics and USAID must redefine the geographic focus for the project and the different components, identifying areas where the project can achieve greater impact. Chemonics and USAID should define the description of the indicators applying SMART criteria and establish a formal agreement on the interpretation of the indicators in the PMP, reviewing and                                                              29 Key question Number 1 (iii).   What are the necessary adjustments for improving the project’s effectiveness taking into consideration USAID/Bolivia’s priorities, the political context, and current relationships and engagement with project stakeholders and beneficiaries? 37 considering the changes already made to the PMP in August 2012, in order to measure the impact of the project in achieving its goals. Chemonics must correct the problems identified with the indicators that are not objectively verifiable and redefine realistic levels of achievement of the indicators in terms of number of beneficiaries, area under improved management and other indicators affected by the agreed definitions and interpretations. USAID and Chemonics should prepare and sign a formal document that sets out clearly the agreements reached in this regard. It is not within the scope of this evaluation mission to determine whether the misinterpretation has been deliberate or an omission of project management. Since the interpretation of indicators has been approved by USAID there is joint responsibility between both Chemonics and USAID. If USAID considers that the misinterpretation was deliberate it is recommended to carry out a technical and financial audit of the project and of USAID own internal procedures. To improve monitoring of the project, Chemonics should review its internal monitoring system according to the requirements of the new PMP, ensuring that appropriate controls are incorporated into the system. Chemonics and USAID should conduct systematic monitoring visits every six months, verifying the level of achievement of the indicators reported in the field. Mechanisms that enable participation of the stakeholders and their feedback to project management should also be established through the creation of project advisory committees at the local level. To improve the effectiveness of the project in reaching its objectives, Chemonics should reorient the organization of operational teams to reduce administrative levels and promote local levels of coordination. Chemonics should develop an operational structure with lines of command and responsibilities clearly established based on the scope of the objectives and results of the project30. In the future, USAID should separate its objectives of strengthening strategic partners from those defined in the project objectives. For its part, Chemonics must establish clear contractual terms with the project executors in order to meet the foreseen results and achieve the project objectives. The project should take advantage of the current circumstances to initiate dialogue with the Bolivian government regarding the project and seek to establish formal agreements for project implementation.   This should form part of the process to establish a long‐term economic and environmental cooperation strategy between USAID and the Bolivian Government. In order to do this, a solid communication strategy must be developed. This strategy should allow the project to define the ways to establish relations with the different government levels and agencies (especially SERNAP, ABT and INIAF) and to seek the development of specific actions in coordination with other government agencies, supporting the development of instruments to promote sustainable development, and taking into account the option to support initiatives via the Joint Mechanism for Adaptation and Mitigation of Climate Change.                                                              30 The evaluation team recognizes the progress made in restructuring project organization from May 2012. 38 7 Lessons Learned31 7.1 Institutional Strengthening The use of financial resources for institutional strengthening has mainly benefited CADEFOR and FDTA Valleys, but has contributed little to achieve the objectives of the project. It is considered that the direct and unconditional transfer of resources is not an appropriate instrument to either strengthen the local organization or support the achievement of project objectives.  The evaluation team considers that a more appropriate strategy would be to put resources in the hands of entities that require the services and not to subsidize the service provider. Regarding the strengthening of local organizations: the need to develop an institutional strengthening strategy is evident. This involves performing an analysis of the needs of municipal  governments  and indigenous organizations for strengthening their territorial management capabilities, focusing especially on the development of systems of control developing local regulations according to revised legal framework defined in the NCPE and the Law of Mother Earth.   Steps should be taken to strengthen intercultural Federations in natural resource management in the areas addressed by the project. Similarly, to improve the effectiveness and impact of training, project executors should develop a training strategy based on a training needs analysis of the various stakeholders. This strategy should aim at addressing the different organizational levels to promote participatory management of natural resources, according to the identified needs, coordinating actions with training processes already underway in the areas covered by the project, facilitating horizontal exchange and training processes between producers, and defining impact indicators for the training process. 7.2 Participatory Approach Some progress has been made in the development of standards, bylaws and community development plans however, in general, ownership of these management tools is not assumed by the communities. None of the communities visited had knowledge of the community productive development plans (PDCPs) produced by the project. No evidence of "effective participation" in development, let alone training for implementation could be perceived during visits and interviews to communities. The technical assistance strategy focused on the vertical transfer of knowledge, rather than on building local networks of horizontal exchange of knowledge. The project did not use the local capacities of the OFCs, established after several years of training, to reduce reliance on external training and technical assistance. 7.3 Gender Approach The project aims at strengthening the rights and priorities of women, but there is no explicit gender approach in the implementation of the project. The gender ratio in the project target population is                                                              31 Key Question 1 (iv). What lessons learned and best practices identified can be applied during the remaining life of the project? 39 unbalanced but it could still be rated high, considering the nature of the project activity. The participation of women in the project activities reached the following levels: Rurrenabaque 31%, San Buenaventura 41%, Ixiamas 31%, Urubichá 32%, and Ascension 48%. The project should work more seriously to address gender issues to be able support to strengthening of rights and priorities of women. Women have an essential role in the management of biodiversity resources in collecting forest products such as food, medicinal plants, wild fruits, and firewood. Community forestry and non‐timber products are part of the knowledge of women, which a comprehensive forests and biodiversity management programs cannot ignore. However, there is no information showing that women were consulted regarding their priorities. It is prudent to do so to know their priorities in their areas, since women depend on healthy forests more than any other stakeholder group. The achievement of success in attaining the project strategic objectives requires a female face. Working with women contributes significantly to conservation and biodiversity management. 7.4 The integrated approach The project has a conception of integrality that is not reflected in practice either in the field or through the management structure. Each component is an island. Prioritization of work with indigenous peoples is not equivalent to the integral action of the project and its components. Forestry activities are implemented in certain locations, tourism in the others and agro‐ecological projects in other communities. This spatial dispersion of the components shows the conception of the project operators, who have been unable to conceive of how to implement the concept of integrated natural resource management in practice. On the contrary, the intervention of some components of PAI induces loss of integrality. In Guarayos some families are increasingly dependent on the sale of timber, leaving agroforestry production and supply of forest foods and buying staple foods and fruit from the Chapare. The integrated approach to land management has social, cultural and environmental dimensions. In the social dimension, actions should not focus on one group nor exclude others. In the cultural dimension, the project should consider ancestral and local knowledge of each group regarding the environmental goods and ecosystem services of forests: primary production, flora, fauna (biodiversity) and freshwater climate regulation functions, temperature, wind patterns, rainfall, and environmental services functions to agriculture and all human activities. To promote the adoption of a holistic approach in the areas where the project works, the project should work with all actors present in the area, focusing on the actors that exert greater pressure on the forest and biodiversity. The project should support or create local comprehensive management platforms to promote exchanges between different interest groups that exist in the territories prioritized by the project. The project should resume the direction of land management and local governance, strengthening local legitimate actors (community and producer organizations) if the local political power and social organizations do not represent viable operational partners.   The promotion of conservation and sustainable use of biodiversity requires a strong synergy with government policies and existing local projects. The current policy framework has a structure of 40 incentives for the utilization and unsustainable use of natural resources: environmental crimes are not punished, patents for forest use and clearing are very low, and there is no support for the valuation of forest products and ecosystem services by society. Notwithstanding this situation, the recent Law of Mother Earth establishes the Joint Mechanism for Adaptation and Mitigation of Climate Change, which seeks to discourage the destruction of forests and encourage their conservation. In this sense, the mechanism will establish policy instruments to support sustainable initiatives in forest conservation, establishing a register of all such initiatives at the national level, including state, community and private projects initiatives. The Project PMP was the document subjected to greatest analysis and breakdown during the midterm evaluation. It has been shown that this monitoring and evaluation tool has not been coherent in contributing to the efficient management of the project and that the selection of indicators has not effectively promoted the adoption of an integrated approach. 8 Best practices found The PMP of the project explains that the number of good practices applied was to be used as an indicator of the strengthening of the municipal governments’ capacity to apply a land management strategy. The project quantifies the best practices in terms of the number of management units supported. This is, in the case of forest management, the number of management plans. In tourism, this is the number of plans (operators) supported by the project and the number of community development plans developed. The practices considered include those which contribute to biodiversity conservation, to mitigation and adaptation to climate change, and to the development of value chains for products and services oriented to sustainable development, agriculture and agroforestry systems. The essence of best practices, as understood in the project framework, is practices that are derived from the support to the implementation of PMOTs. The best practices registered are considered to correspond more to objective 3, which relates to the development of sustainable productive activities and income generation from timber and non‐timber forest products, agriculture, agroforestry systems, eco‐tourism and others. The best practices related to objective 1 should then better reflect, not so much the natural resources management, but the capacity for environmental and territorial management, as clearly stated in the objective, which should then be reflected by the application of PMOTs in the POAs and PDMs. Normally a best practice is considered as “a method or technique which consistently shows better results in relation to those obtained with other options”. 8.1 Forest Management In the area of forest management the project has promoted the adoption of a number of practices that, as a whole, constitute the forest management plan. By means of providing technical assistance and training, the project has promoted a series of practices associated with reduced impact forest management, based on the implementation of the operational plans. The promotion of good forest management practices represents one of the areas where the project has been successful, building on 41 previous experience of the BOLFOR projects. As mentioned above, there is still scope to improve certain practices. 8.2 Tourism The support provided by the project to the development of tourism in the area of Rurrenabaque has generated a number of actions that can be pointed to as best practices, which together have contributed to the growth of the sector in the zone. These practices include the “Program of Green Action” certification of responsible practices for sustainable tourism and the production of tourism development plans. The component was able to show that, by means of the development of good practices, it has been possible to attract the interest and support of the local and departmental governments, demonstrating that the best way to change preconceptions of the authorities is by means of positive impacts. 8.3 Agro‐ecology Despite the limited impact of the agro‐ecological component, there are some practices of the component that can be pointed as best practices. These include: the establishment of the inter‐ institutional committee and the coordination of technical assistance and training of producers in the area of Guarayos. It can be expected that this will continue beyond the end of the project and has already expanded during the pause in the project activities. The establishment of the seed bank in Guarayos for the multiplication of improved varieties of agricultural crops and the promotion of biological control is another best practice that the component has developed. 9 References ABT. 2011. Informe Anual 2010 y Balance de la Década. Autoridad de Fiscalización y Control Social de Bosques y Tierra. Santa Cruz, Bolivia. ABT. 2012. Informe Anual 2011. Autoridad de Fiscalización y Control Social de Bosques y Tierra. Santa Cruz, Bolivia. Dauber, E., T. S. Fredericksen, M. Peña‐Claros, C. Leaño, J. C. Licona & F. Contreras. 2003. Tasas de incremento diamétrico, mortalidad y reclutamiento con base en las parcelas permanentes instaladas en diferentes regiones de Bolivia. Proyecto de Manejo Forestal Sostenible, BOLFOR, Santa Cruz, Bolivia. Cuellar, S., Rodríguez, A., Arroyo, J., Espinoza, S. & Larrea, D.M. 2012. Mapa de deforestación de las tierras bajas y los yungas de Bolivia 2000‐2005‐2010. Fundación Amigos de la Naturaleza. Santa Cruz de la Sierra, Bolivia. Pariona, W., T. van Rooij, T. Siles & E. Domic. 2011. Guía práctica para la identificación de Bosques de Alto Valor de Conservación en el norte de La Paz. RA y WCS, La Paz, Bolivia. Rodríguez, A. 2012. Cartografía multitemporal de quemas e incendios forestales en Bolivia: detección y validación post‐incendio. Ecología en Bolivia 47: 53‐71. 42 Uhl, C. & J.B. Kaufmann. 1990. Deforestation, Fire susceptibility and potential tree responses in the Eastern Amazon.  Ecology  71 (2): 437‐449.