USAID/PVC Matching Grant Evaluation Series: Building Constituencies for Protected Areas Matching Grant FAO-A-00-97-00061-00 between The Nature Conservancy and USAID/PVC April 2002 Conducted under USAID/Evaluation Indefinite Quantity Contract #AEP-I-00-00-00024-00, Task Order # 2 (with Management Systems International) Evaluation team members: Michelle Libby (TNC, in Jamaica) Titayanto Pieter (TNC, in Indonesia) Mark Renzi (MSI, Team Leader, in Jamaica and Indonesia) Kath Shurcliff (TNC, in Indonesia) Jill Tirnauer (MSI, in Jamaica) Evaluation of ìBuilding Constituencies for Protected Areasî Matching Grant FAO-A-00-97-00061-00 between The Nature Conservancy and USAID/PVC Table of Contents List of Acronyms........................................................................................................................................iii Evaluation Identification Table ................................................................................................................iv 1.0 Executive Summary..............................................................................................................................v 1.1 Overview .......................................................................................................................................... v 1.2 Key findings, conclusions and recommendations ........................................................................... vi 2.0 Evaluation Methodology and Team Composition..............................................................................1 3.0 Matching Grant Background...............................................................................................................1 3.1 Historical & technical context and partners ..................................................................................... 1 3.2 Project goal, objectives, and major hypotheses to be tested............................................................. 3 3.3 Detailed Implementation Plan .......................................................................................................... 4 4.0 Purpose of the Evaluation ....................................................................................................................4 5.0 Program Implementation.....................................................................................................................5 5.1 Success in achieving DIP ................................................................................................................. 5 5.1.1 Status of DIP completion and DIP accuracy...........................................................................5 5.1.2. Quality of DIP and degree of success in implementation......................................................6 5.1.3. Familiarity with DIP and design ............................................................................................7 5.1.4. Major Successes and Shortcomings in Implementation.........................................................8 5.1.5 Impact Results.........................................................................................................................8 5.2 Assessment of project model and hypotheses ................................................................................ 11 5.2.1 Project hypotheses articulated in CA....................................................................................11 5.2.2 Effectiveness of decentralization model used by TNC.........................................................13 5.2.3 In-country sub-awards ..........................................................................................................15 5.2.4 Effectiveness of TNC/JCDT self-assessment tools ..............................................................17 5.2.5 Replication and Scale-up of Approaches in Project Area or Elsewhere...............................18 5.3 Advocacy under the project............................................................................................................ 18 5.3.1 Advocacy activities and impact ............................................................................................18 5.3.2 Partner/PVO roles in advocacy.............................................................................................20 5.4 Implementation Lessons Learned................................................................................................... 21 5.4.1 Implementation tools: utilization and learning .....................................................................21 5.4.2 Key TNC/JCDT capacity constraints to implementation......................................................21 6.0 Partnership Questions ........................................................................................................................22 6.1 Analysis of Partnership Schemes ................................................................................................... 22 6.2 Measuring Institutional Capacity ................................................................................................... 25 6.3 Constraints to Partnership............................................................................................................... 26 6.4 Information Technology................................................................................................................. 27 6.5 Use of local networks and service organizations............................................................................ 27 7.0 Program Management........................................................................................................................27 7.1 Strategic Approach and Program Planning .................................................................................... 27 H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc ii 7.2 Country Initiatives.......................................................................................................................... 27 7.3 Conflict Management ..................................................................................................................... 28 7.4 Monitoring and Evaluation............................................................................................................. 28 7.5 Overall Management ...................................................................................................................... 28 7.6 Sustainability.................................................................................................................................. 31 7.6.1 Overall sustainability survey.................................................................................................31 7.6.2 Shift in role of JDCT.............................................................................................................32 7.6.3. Potential for financial sustainability of JCDT......................................................................33 7.7 Financial Management ................................................................................................................... 33 7.7.1. Effectiveness of financial management ...............................................................................33 7.7.2. Leveraging other donor funds..............................................................................................34 7.7.3 Cost effectiveness of technical approach..............................................................................35 7.7.4 Repercussions of ìmatchingî requirement on program........................................................35 7.8 PVOís Information Management ................................................................................................... 36 7.9 Logistics ......................................................................................................................................... 37 7.10 Project Supervision....................................................................................................................... 37 7.11 USAID Management.................................................................................................................... 37 8.0 Lessons Learned..................................................................................................................................38 ANNEXES: Annex A: Key Events Timelines for Jamaica and Indonesia Annex B: Detailed Implementation Plans for Jamaica and Indonesia Annex C: Evaluation Scope of Work Annex D: Advocacy Implementation Chain for Jamaica and Indonesia Annex E: Partnership Tables for Jamaica and Indonesia Annex F: Sustainability Analysis for Jamaica Annex G: List of Persons Interviewed Annex H: Documents Reviewed H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc iii LIST OF ACRONYMS APR Asia/Pacific Region (a regional office of TNC) AR Annual Report BJCMNP Blue and John Crow Mountains National Park CAT Community Assessment Tool CBO Community Based Organization CLI Conservation Leadership Initiative (the institutional strengthening arm of TNC/APR) CMP Coastal Marine Program (of TNC, in Denpasar [Bali], Indonesia) COO Community Outreach Officer (Jamaica) CPD Country Program Director CTO Cognizant Technical Officer CTW Conservation Training Week DIP Detailed Implementation Plan DO Development Options, Ltd. (Jamaica) EFJ Environmental Foundation of Jamaica FKLL Partnership of Lore Lindu Coalition GOI Government of Indonesia GOJ Government of Jamaica JCDT Jamaica Conservation and Development Trust (TNCís Jamaica Partner NGO) KNP Komodo National Park (Indonesia) LAC Local Area Committee (Jamaica) LACD Latin America and Caribbean Division LLNP Lore Lindu National Park (Indonesia) LNGO Local NGO MSI Management Systems International MG Matching Grant NGO Nongovernmental Organization NRCA Natural Resources Conservation Authority NRM II A USAID/Indonesia project, which included funds for TNC in LLNP PA Protected area PiP Parks in Peril Program PVO Private Voluntary Organization PVC Office of Private Voluntary Cooperation RFA Request for Assistance SCP Site Conservation Planning TNC The Nature Conservancy YPAN Yayasan Pusaka Almam Nusantara (TNCís Indonesia Partner NGO) USAID US Agency for International Development WO Worldwide Office (TNC/HQs) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc iv EVALUATION IDENTIFICATION TABLE PVO Name The Nature Conservancy Matching Grant Title Building Constituencies for Protected Areas Cooperative Agreement Number FAO-A-00-97-00061-00 Amount of Grant $1,351,998 of USAID funding and $1,351,998 of TNC funding Period of Grant September 30, 1997 ñ September 29, 2000 Cost/No Cost Extension No cost extension granted through the end of September 2001 Current Status of Matching Grant Completed USAID/PVC Grant Officer (s) Martin Hewitt, Brian Greenberg Technical Area of Grant Local NGO strengthening, community conservation, and natural resources management Date of Evaluation August 2001 to March 2002 (duration extended due to delays caused by ì9-11î attack) Countries of Program Activity Jamaica; Indonesia Countries of Programs Evaluation Jamaica; Indonesia Evaluation Team Members Michelle Libby, TNC (Jamaica) Titayanto Pieter (Indonesia) Mark Renzi, MSI (Jamaica and Indonesia) Kath Shurcliff , TNC (Indonesia) Jill Tirnauer, MSI (Jamaica) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc v Evaluation of ìBuilding Constituencies for Protected Areasî Matching Grant FAO-A-00-97-00061-00 between The Nature Conservancy and USAID/PVC 1.0 EXECUTIVE SUMMARY 1.1 Overview In September 1997 The Nature Conservancy (TNC) received a $1.35 million Cooperative Agreement (CA) from USAIDís Office of Private Voluntary Cooperation (PVC), committing to matching the amount in order to develop its ability to mobilize local constituents to support conservation. TNCís original proposal suggested working in five different countries, but after feedback from USAID settled on working in two: in the areas around Blue and John Crow Mountains National Park (BJCMNP) in Jamaica and Lore Lindu National Park (LLNP) and Komodo National Park (KNP) in Indonesia. The purpose of the CA was to increase the Conservancyís ability to strengthen local conservation organizations for long-term management of targeted parks and reserves through developing entrepreneurial income-generating strategies and energizing community-based organizations and coalitions around protected areas. To that end, TNC has hired an ecotourism coordinator, a community conservation specialist, and an Institutional Development expert for the Latin America and Caribbean Division (LACD). It has gained both positive and negative experiences in attempting to promote enterprise development through LNGOs and consulting organizations. However, very little documentary evidence exists that these hard-earned institutional lessons on the ground have been formally integrated into TNCís way of doing business. Moreover, obtaining this learning, through this project, has been a daunting challenge. From the outset there was startling little ìownershipî of the grant within TNC. The individuals who had designed the original proposal were not fully satisfied with the re-design that focused on only two countries. A foreshadowing of problems to come, the bulk of the proposal was prepared by individuals at the Worldwide Office (WO) who worked for units that had little real ìreachî to the field. The few field￾based individuals who were involved, shifted away from the project during implementation. The project experienced remarkable turnover: TNC changed Country Program Directors (CPDs) in both Jamaica and Indonesia; both partner local NGOs (LNGOs) in each country lost their Executive Directors (not to be replaced), and there was a constant flux of technical staff and their reporting arrangements. The activities were managed by the country programs (the leaders of which were mostly not involved in the design) with no central coordinating function between them. In this context ñ and with a delayed start ñ lack of ownership is not surprising. These dynamics appear to have engendered ambiguity among country staff towards their partners and schizophrenia between TNC/WO and the respective country programs on how best to manage the partnerships. TNC lists one of the main successes of the matching grant (MG) as being that it learned a great deal about how to engage in partnerships with local non-governmental organizations (LNGOs) ñ please see Section 8 for a summary of lessons learned. If those lessons can successfully be integrated into the larger institution, that should help TNC achieve its objective of having conservation impact at scale. Unfortunately, much of that learning occurred as a result of difficult lessons emerging from dramatic partnership failures under the project. The agreement with the Jamaican LNGO, Jamaica Conservation and Development Trust (JCDT), was terminated prior to the end of the grant because TNC no longer thought the partnership could be salvaged under the MG. After proceeding with fits and starts to grow a LNGO (called YPAN) in its image in Indonesia, TNC ended up coming to a similar conclusion in Indonesia as in Jamaica by folding up its partnership table and ultimately playing a leadership role in disbanding YPAN, which re￾merged back into TNC prior to MG termination. Neither the Jamaica nor the Indonesian projects were successful in implementing the projects as designed and articulated in their Detailed Implementation Plans (Indonesia completed only one-half of planned activities; Jamaica less than 2/3.) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc vi TNC was not successful during the project period in promoting economically- and ecologically￾sustainable enterprise development within communities living near the targeted protected areas. At BMJCNP, in Jamaica, TNCís LNGO partner, JCDT, began by focusing on community- and resource￾based, enterprises. But, through a gradual easing of standards in response to cumulative implementation setbacks, the MG winded up trying to support a number of individual entrepreneurs in businesses with no direct ties to conservation. Even with that adjustment, the project was unable to produce bankable business plans by the end of the project, much less viable businesses. The team working at LLNP in Indonesia was able to generate some resource-based enterprises, but, unfortunately, was not able to help them become feasible at an appropriate scale in the park. It is hoped that at least one enterprise (cacao) will have potential elsewhere in Indonesia. Likewise, the KNP project was unable to produce viable businesses by the end of the project period. However, there is hope that some of the enterprise models developed will prove viable in the future in KNP and elsewhere in Indonesia. To place these shortcomings in perspective, it should be noted that starting conservation-friendly businesses in protected area buffer zones is not an easy matter anywhere in the world. One of the chief challenges faced by the project is that neither TNC nor JCDT had a significant track record in this field. However, both the Indonesia and Jamaica programs did achieve very significant success in the field of policy reform and advocacy. The Jamaica, program was successful in mobilizing communities to advocate with park management, other donors, and other parts of government. TNCís partner, JCDT, continued to play a leadership role in a national association of environmental LNGOs concerned with protected area management. In LLNP, the project helped strengthen an informal coalition of NGOs working in the buffer zone. In KNP, TNC was crucial in helping the national government design innovative joint management approaches for the park, and working with local government to begin to restructure the political landscape and specific regulations in favor of conservation. Based on what the evaluation team witnessed in these two programs, TNC needs to focus on improving some key aspects of its management systems. TNC, justly renowned in the environmental community for its fundraising prowess, was unable to raise significant match for the two projects. This appears to be directly related to the general lack of ownership of the project in the field. To its credit, TNC used its general funds to cover the shortfall in Indonesia. Unfortunately, it did not make up the shortfall in Jamaica, and serious implementation failings resulted. It would appear that TNC needs to consider how to ensure that organizational commitments to the USG can be met with a decentralized fundraising system. TNCís monitoring systems were at best ineffective, and at worst misleading. Annual reports were consistently overly rosy, self-assessment tools were not effective, and it appears that ìbad newsî has a difficult time making its way through the system. The project was unable to assemble and disseminate lessons learned from either project site, despite articulating this as a major objective of the MG. TNC was unable to engineer significant information sharing and learning between the two country programs under a single project, although it was an important project objective. These inadequacies must be addressed if TNC is to become a learning organization and meet the ambitious objectives of its current restructuring. Finally, project continuity suffered from a large degree of staff turnover especially in senior management. 1.2 Key findings, conclusions and recommendations Please see Section 5.1.4, for a table summarizing key implementation successes and shortcomings. Section 8.0 presents lessons learned that can be extrapolated beyond the TNC experience. The following reproduces some key conclusions and recommendations from the body of this report. Conclusions: (a) TNC did not succeed in strengthening its LNGO partners in Indonesia or Jamaica through this grant ñ a central component of the project. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc vii (b) However, both country programs as well as LACD and APR staff report that TNC has learned valuable lessons in LNGO partnering from the experience that could inform future partnering ñ thus improving TNCís capacity. (c) The project has had very significant policy/advocacy successes in both countries at the community, local, and national levels. This may be the most significant on-the-ground impact of the project. (d) TNCís self-assessment tools do not appear to have served it well in the MG, and did not effectively review their partnership arrangements. (e) The Indonesia and Jamaica projects were not integrated. One of the chief reasons for this was the lack of designation of an individual, or team of persons, that would be held accountable for assuring integration of the overall program. At the same time, one individual must be assigned responsibility for success of his or her respective country component. (f) Ambiguous management authority, TNCís changing vision for its role in Indonesia, and the lack of clear communication by TNC to YPAN about how TNCís vision was changing and how those changes would affect YPANís role greatly impeded the success of the project. (g) TNC should be credited with living up to its match obligation in Indonesia, despite the financial strain it placed on the country program (as a result of ineffective fund raising.) Unfortunately, however, TNC did not live up to its match obligations in Jamaica, seriously compromising the project. (h) It would seem that there may be an excessive concern with presenting positive news within TNC. This is a natural tendency. However, if allowed to pervade the system, it is difficult for accurate information to reach the relevant decision makers and can compromise donor/TNC trust. Such a culture ñ somewhat natural in an organization constantly required to raise funds to achieve its mission ñ can be a major obstacle to becoming a learning organization. Recommendations: (a) The MG experience indicates that TNC needs to improve the integrity of its monitoring systems to prevent double counting or mixing activities from other USAID funded projects. They need to be developed and used by field managers, to monitor implementation success. TNC also needs to more accurately target impact measures to provide objective assessments of actual impact of activities. To promote honest evaluation of success of activities, these indicators should be jointly agreed between USAID and the USPVO, and then used and reported on throughout the duration of the project. (b) USAID/PVC should improve the quality of DIPs, or discontinue them as a requirement. It would be wiser to focus on impact via use of standard project management tools, such as the logical framework and review of annual work plans. (c) PVC could make greater use of evaluations and periodic PVC/PVO meetings to highlight implementation issues, as well as successes. These will provide the opportunity for addressing emerging problems collaboratively. Formal reports could be used as the basis for these face-to-face meetings. (d) TNC should seriously review its ability to conduct assessments of partnership arrangements, including its use of self- assessments, MOUs, conflict mediation, and external evaluations. TNC needs to improve the way in which ñ organization-wide ñ it plans for and integrates difficult lessons emerging from such assessments and evaluations. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc viii (e) If TNC could avoid being overly identified with a single LNGO, TNC could provide greater latitude for LNGOs to adopt advocacy tactics at variance with TNC while not jeopardizing TNCís status in a country by being associated with internal political issues. (f) Compiling and disseminating lessons should be done on a more systematic basis, based on clear hypothesis testing, with structured framework for including activities, assessment of quality of outputs, monitoring impacts, and a process for review of these results, and making in-course changes to strategies and activities, which preferably are formally approved by USAID/PVC. (g) TNC should more systematically explore how communities can be effectively mobilized for conservation in the project area, especially where there is the most near-term chance for success and appears to have the most favorable funding scenario ñ Komodo National Park and Indonesiaís Center for Marine Protected Areas. This would involve integrating a greater social science component into the programís already well-developed natural science analytic capability. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 1 Evaluation of ìBuilding Constituencies for Protected Areasî Matching Grant FAO-!-00-97-00061-00 between The Nature Conservancy and USAID/PVC 2.0 EVALUATION METHODOLOGY AND TEAM COMPOSITION The evaluation was performed by two three-person teams, comprised of individuals from Management Systems International (MSI) and The Nature Conservancy (TNC). The team leader for this evaluation was Mark Renzi, a Senior Associate with MSI. The other team member from MSI was Jill Tirnauer, Program Associate. Accompanying Jill and Mark to Jamaica as TNCís member was Michele Libby, a Community Conservation Specialist. Mark was joined in Indonesia by Titayanto Peiter, former MG manager for Indonesia and current Partnership Director for TNC/Indonesia, and Kath Shurcliff, Director of TNCís Conservation Leadership Initiative who served as TNCís team members. Fieldwork occurred August 2001 in Jamaica, and February 2002 in Indonesia. The evaluation began with a Team Planning Meeting in Washington, DC with TNC and USAID/PVC. During this meeting, the scope of work was reviewed, terms of the evaluation agreed upon, and areas for examination prioritized. Following this meeting, MSIís team members reviewed documents prior to departure to the field and interviewed TNC staff members associated with the Jamaica program. In Jamaica the team interviewed TNC and JCDT (Jamaica Conservation and Development Trust) staff, visited the field site and interviewed Local Area Committee members, interviewed relevant stakeholders, and reviewed project documents. Further interviews at TNC were completed after the teamís return from Jamaica. Documents reviewed included the proposal, DIP, annual reports, trip reports, financial records and meeting minutes. The trip to Indonesia was delayed by travel restrictions resulting from the 11 September attack on the USA. The team interviewed TNC staff working on Indonesia in the interim and finally made a trip to Bali and Flores, Indonesia in February 2002. Unfortunately, due to security concerns, the team was not able to visit the project site in Palu, Indonesia, related to work in Lore Lindu National Park. Therefore, the evaluation is not able to go into as much detail as would have been desired with respect to that site. A similar range of inquiry was completed for Indonesia as for Jamaica. A preliminary draft report was drafted by Mr. Renzi and submitted for comments to the TNC co-evaluators. The emerging report ñ on which there was agreement among TNC and MSI team members ñ was then submitted to USAID and TNC for comments in April 2002. Results were incorporated into a final draft, followed by a formal debriefing with USAID and TNC. 3.0 MATCHING GRANT BACKGROUND 3.1 Historical & technical context and partners On 30 September 1997, TNC and USAIDís Office of Private Voluntary Cooperation (PVC) entered into a three-year Cooperative Agreement for implementation of Building Constituencies for Protected Areas. Although the original proposal for the Matching Grant Program targeted five countries across Latin America, the Caribbean and Asia, the final Cooperative Agreement provided funding for just two selected by USAID ñ Jamaica and Indonesia. The mission of The Nature Conservancy is to preserve plants, animals and natural communities that represent the diversity of life on Earth by protecting lands and waters they need to survive. Incorporated in 1951, TNC has protected more than 8 million acres in 50 states and Canada. At the time of the grant application in 1996, TNC had approximately 825,000 members and an annual budget of $170,000,000. TNC is currently organized into 9 geographic regions, including one dedicated to Latin American and the H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 2 Caribbean (LACD) and one to Asia and the Pacific (APR).2 Although intended to have institution-wide impact, the MG was essentially managed in TNC as if it were two separate projects: one for Jamaica (managed by LACD), the other for Indonesia (managed by APR). Jamaica Project LACD feels it has strong relationships with over 50 NGOs in the region, including Jamaica Conservation and Development Trust (JCDT). The Jamaica program focused on developing the capacity of JCDT to design and implement conservation initiatives, develop the capacity of three community-based organizations, and increasing and diversifying flows of income to local residents, local organizations, and the park itself through resource-based sustainable enterprises, park concessions and other approaches ñ all to promote sustainable conservation of the Blue and John Crow Mountains National Park (BJCMNP). TNC had worked with JCDT since its establishment in 1987 and played a lead role in establishing Jamaicaís system of protected area management, including the gazetting of BJCMNP. Matching Grant (MG) funding supplemented an existing USAID funding through the Parks in Peril (PiP) program, also executed in partnership with JCDT. BJCMNP is a large park, serving as a vital watershed to the nationís capital. It encompasses nearly 7% of Jamaicaís land mass and is considered one of the top three biologically important areas in Latin America and the Caribbean (Dinnerstein, et al). BJCMNP was declared a national park in 1993. Two years later the Government of Jamaica delegated management authority of the national park to JCDT. The joint TNC/JCDT project worked with three community associations among the approximately 100,000 people living within the buffer zone. JCDT used project funds to work intensively with the targeted CBOs, develop an environmental education program, and to struggle (mostly unsuccessfully) with a series of evolving tactics for economic development. TNC worked in fits and starts to strengthen JCDT while trying to maintain a productive working relationship with it. A sub-grant agreement between TNC and JCDT was signed in April 1998, with funds distributed to JCDT July 1998. During the first year of the project, TNC hired a local coordinator for both the MG and PiP funds. Relations between JCDT and TNC were strained throughout much of the implementation period. Due to TNCís perception that JCDT was not performing adequately, and difficulty by TNC in obtaining matching funds, TNC decided in September 2001 to stop substantive MG funding for JCDT, although it allowed for close-out funding through the end of the calendar year. Few of the deliverables or objectives had been accomplished, the partnership was in shambles, and TNC fell short of its matching obligations under the MG, despite a no-cost extension through September 2001. Indonesia Project TNC started its operations in Indonesia in June 1991, entering into a Memorandum of Understanding (MOU) with the GOI to work in one of Indonesiaís premier forest areas, Lore Lindu National Park in Sulawesi. In 1993, TNC started its marine conservation program in Komodo National Park, a world heritage site of internationally important marine and terrestrial biodiversity, home to the legendary Komodo dragons. As an international organization, TNC was not permitted to fundraise within Indonesia. To circumvent this restriction TNC formed a local foundation in 1995, Yayasan Pusaka Alam Nusantara (YPAN). YPAN began as basically a shell to fundraise within Indonesia. All TNC activities in Indonesia were reported as done collaboratively with YPAN despite the fact that the latter had no staff of its own. The TNC Senior 2 Please note that TNC was undergoing a fundamental restructuring at the time of report writing that is expected, among other things, to likely eliminate current bureaucratically-defined geographic divisions and replace them with others. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 3 Advisor in Indonesia at the time was overseeing all activities for both TNC and YPAN. Some members of the Board of Advisors of TNC in Indonesia were also members of the Board of Directors of YPAN. YPAN continued to raise funds locally for conservation activities undertaken by TNC/YPAN. However, during the process to extend its MOU with the government of Indonesia in 1997, TNC was instructed by the GOI to discontinue using YPAN simply for local fund raising. To remain credible to the GOI as an Indonesian NGO eligible to raise funds, YPAN had to become a truly independent organization. In August 1997, a transition plan was developed. At about the same time the matching grant was awarded by PVC to help implement this transition plan. The goal of the matching grant was to enhance in-country capacity for protected area co-management and compatible enterprise development at national parks by helping to build effective NGOs, community￾based organizations (CBOs), and government agencies. The grant focused on developing YPAN as the leading national partner NGO to design and implement conservation projects, as well as capacity building with a wider range of local organizations. The goal was expected to be achieved by (1) strengthening YPAN as an advocate for participatory biodiversity conservation; (2) developing model conservation coalitions at Lore Lindu and Komodo National Parks to address park management and community development issues; (3) documenting lessons learned from enterprise approaches at the two sites to develop strategies for long-term sustainability and promote best practices to a broader constituency throughout the region; and (4) building TNCís own capacity in the region to deliver in-country institutional strengthening programs. To provide immediate support to YPAN, all TNC local staff were transferred to YPAN in early 1998. Recruitment for an Executive Director for YPAN was also initiated, although none was identified until June 1999. YPAN also took over field work in the two national parks, Lore Lindu and Komodo, where TNC had been operating. TNC took the role of supplying technical advisors at both sites. However, by June 2000 it was agreed that YPAN halt all activities in Komodo and by November 2000 YPAN was out of Lore Lindu as the YPAN Board finally decided to dissolve its organization and re-merge it back into TNC. During the remaining ten months of the grant TNC implemented the field projects directly in each of the field sites. 3.2 Project goal, objectives, and major hypotheses to be tested The MG had an overall project goal, as follows: To increase the Conservancyís ability to strengthen local conservation organizations for long￾term management of targeted parks and reserves through developing entrepreneurial income￾generating strategies and energizing community-based organizations and coalitions around protected areas. Underneath that overall goal, each country program had its own goals and objectives, as follows: Table 3.2.1: Project Hierarchy of objectives (Jamaica) Jamaica Goal To advance sustainable conservation and to promote rural socio-economic growth through local capacity building and enterprise development Jamaica Objective 1 Strengthen environmental advocacy and local economic development within buffering communities of the Blue and John Crown Mountains National Park (BJCMNP), contributing to long-term management of the park Jamaica Objective 2 Build the Jamaica Conservation and Development Trustís (JCDT) financial capacity and sustainability of resource flows for park management H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 4 Jamaica Objective 3 Enhance TNCís ability to support enterprise-based conservation Table 3.2.2: Project Hierarchy of objectives (Indonesia) Indonesia Goal Enhance in-country capacity for protected area co-management and compatible enterprise development at representative national parks through training NGOs, community-based organizations(CBOs), and government agencies Indonesia Objective 1 Develop YPAN as a strong national conservation organization, and develop its financial sustainability through generation of bi- and multi-lateral support, income-generating ventures, and private-sector funds in Indonesia Indonesia Objective 2 Strengthen site-based coalitions and community organizations Indonesia Objective 3 Document lessons learned from enterprise approaches at the two sites. Develop strategies for long-term sustainability, and promote best practices developed at the two sites to broader constituency in Indonesia and throughout the region Indonesia Objective 4 Develop institutional strengthening capacity within TNC's Indonesia program to provide training opportunities and other capacity building initiatives for YPAN, community-based organizations, and other conservation partners (including government parks authority) The underlying hypothesis was that conservation depends upon NGOs and CBOs increasing their financial self-sufficiency from in-country sources and increasing their political voice through coalition building. A second hypothesis was that economic development of communities within and around parks using locally derived resources would contribute to further conservation. Both country programs based their strategy on strengthening local NGOs ñ institutionally and financially ñ to be able to implement the program and make the changes sustainable. 3.3 Detailed Implementation Plan A Detailed Implementation Plan (DIP) was submitted by TNC to USAID/PVC in March 1998. The DIP found in Annex B was culled from the original DIP and corresponding timeline and modifications found in the annual reports submitted to USAID as part of reporting requirements (no DIP update was provided in the MG final report.) The tables presented for Jamaica and Indonesia were up updated during the process of this report preparation. 4.0 PURPOSE OF THE EVALUATION This evaluation is a requirement of the Cooperative Agreement between TNC and PVC. The intended audience for this evaluation is TNC, local implementing partners, PVC and other parties interested in conservation issues. Participation by TNC in the evaluation process and the final report issued should increase the knowledge of TNC and its local partners about the success and pitfalls of the methods used as well as the broader utility of using economic development as a means to conservation. The evaluation should be used also as a tool to guide future partnerships both with local partners in the area of conservation development and with USAID as a grant making entity. USAID/PVC will use the information in this report together with the evaluation reports of all of its other MGs to assess how well the MG program is meeting stated goals and objectives; identify patterns and emerging issues across all of the projects; determine technical needs of grantees; shape new Requests for Assistance (RFAs) and any follow-on proposals; and share lessons learned within the entire PVO community. PVC will also use the H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 5 information for its annual Results Report and in USAIDís annual report to Congress. Other interested parties may include other donors or PVOs working in environmental conservation, local economic development, or strengthening the capacity of local partners in the areas of advocacy and/or financial sustainability. 5.0 PROGRAM IMPLEMENTATION 5.1 Success in achieving DIP 5.1.1 Status of DIP completion and DIP accuracy Findings (a) Some activities and outputs listed in the Jamaica DIP were modified or dropped. (b) In some cases, the same activity in the Jamaica DIP was listed twice under the same objective (e.g., Co-Management Company.) (c) Activities originally aimed at promoting revenue generation activities for community-based organizations in the Jamaica DIP were instead directed at individuals. (d) Several activities listed in the Jamaica DIP and reported in the annual reports do not appear to have been financed by the MG.3 Conversely, many activities (apparently) financed by the Indonesia program were not included in the DIP, such as pelagic fisheries, mariculture, and seaweed production. There does not seem to be a written record between USAID and TNC of approval for these changes. However, it appears that the Indonesian activities enjoyed a high degree of success and may have had a high leveraging potential. (e) In some cases reports on progress are misleading. For example,  the Jamaica DIP mentions that ìA total of 58 persons were trained in business skills under the theme ëStarting a Viable Small/Micro EnterpriseíÖ 32 business plans were developed.î However, closer examination of the training results reveals that the statement was substantively incorrect. Of the 58 individuals who attended initial trainings, only 22 completed the training, with only 13 business ideas presented as a result of the UTECH training. Of these, 7 were chosen for further support and training by Development Alternatives (DO). Forty percent did not attend DO training;  the ìmoving of the goalpostsî with respect to PiP indicators (see Section 5.1.2) removed from the DIP indicators which would have presented implementation progress in a negative light;  in the case of Indonesia, the DIP stated that implementation delays were the result of political instability, when, in fact, field staff considered that things had settled down sufficiently at that time as to present no real block to implementation; and (f) In others, serious problems were not mentioned (see Section 7.8 for examples.) (g) Data contained in the Indonesia DIP completed by TNC (presented in Annex B) appear to be accurate, to hold TNC up to a high standard of accomplishment, and appear complete. 3 For example: ìdesign and implement a fee structure for park useî and ìclarify roles of relevant entitiesî (Objective 5); ìassist communities in identification and achievement of infrastructure development needsî (Objective 4); and ìhiring LAC Ecotourism Coordinatorî (Objective 1.) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 6 Conclusions a) DIP reporting is misleading. It overstates TNC accomplishments and understates implementation shortcomings with sufficient frequency as to call the credibility of the reports into question. This not only puts strains on donor/PVO relations but also delays the opportunity for addressing emerging problems collaboratively. b) The candor of the reporting on the final version of the Indonesian DIP, on the other hand, is laudatory. Earlier annual reports were somewhat less frank (see Section 7.8) Recommendations The MG experience indicates that TNC needs to improve the integrity of its monitoring systems to prevent double counting or mixing activities from other USAID funded projects. TNC also needs to more accurately target impact measures to provide objective assessments of actual impact of activities, preferably in consultation with USAID. These measures should then be reported on, and not altered without further consultation with USAID. 5.1.2. Quality of DIP and degree of success in implementation Findings (a) Feasibility studies, business plans and business training, as stated in the Jamaica DIP, were originally aimed at community based enterprises, but were ultimately applied to individuals (and reported as being successful) within the three target areas, when community-level interventions proved difficult. (b) Impact indicators were not developed specifically for this project for either Jamaica or Indonesia. Data were from the Parks in Peril Scorecard, an index developed under a related regional protected area grant, which had a significantly different focus. Indonesia had in place monitoring of conservation success, but these indicators were not linked to activities taken specifically under the MG. There were no good impact indicators for capacity-building activities. (c) Of the seven original consolidated indicators stated in the Jamaica DIP, five were reported on in Year 3 Annual Report. Two which were dropped ñ indicator 13 Diversified Funding for the Site, and indicator 16 Income Generated from Site Conservation ñ would have been important metrics for this project (see also Section 5.2.4). (d) One measurement of outputs for the Jamaica project was an annual formal assessment by TNC. There is no record of its completion. (e) TNC successfully completed less than 2/3 (16 of 27) activities that were listed in the Jamaica DIP. (f) One-half of the activities (30/60) were completed as expected in the Indonesia DIP. Lowest success was reported for Objective (3), relating to dissemination lessons learned, where 1/3 were completed. The most successful was Objective (2), relating to building TNC capacity to support partners, where over æ of the tasks were completed. (g) Often the indicators do not measure the objectives. For Jamaica, indicators are generally not stated. For Indonesia, the indicators generally measure an accomplishment of higher level than the objective statement would imply. (h) The completed DIP was not included in the final report for either the Jamaica or Indonesia programs. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 7 (i) Although USAID has emphasized impact measures worldwide since at least 1990, there is no evidence that USAID recognized the poor quality of the DIPs produced by TNC or suggested their improvement. Conclusions (a) There was a very significant mismatch in both country programs between each DIPís goals, its objectives, the indicators articulated in the DIP, the activities described in the periodic reports, and the actual activities on the ground. While we would expect considerable adjustment during implementation, lack of documentation for the rationale behind the changes led to overall confusion in terms of management, implementation, and monitoring. (b) In neither country program does the DIP effectively measure impact. (c) The Jamaica project was not successful, when considered against its modest DIP. This is more disappointing when we consider that DIP targets were only output ñ not impact ñ level. TNC had little success achieving its stated outputs in Jamaica. Several times it ìmoved the goalpostî in its favor by accepting lower levels of accomplishment or eliminated indicators that would have shown poor performance. (d) Likewise, the Indonesia project was unable to live up to promises made in its DIP. (e) It is not possible to determine the impact of the Jamaica or Indonesia programs via the DIP. Few indicators accurately targeted impact and systematic annual assessments of the project were not done by TNC. (f) The DIP was not a useful management tool for the Jamaica or Indonesia programs and did not provide an accurate picture of impact to USAID. Recommendations a) Partners must understand the difference between output and impact metrics in measuring success. Perhaps technical assistance should have been provided in the development of the DIP, or indicators agreed collaboratively with USAID b) Given that the DIP was intended to be the primary objective criterion for assessing project success, and a central vehicle for USAID/TNC communication on implementation, USAID should require a higher quality DIP. Or, perhaps it would be wiser to return to a more tested tool, such as the logical framework (see recommendation under Section 5.1.3.) 5.1.3. Familiarity with DIP and design Findings (a) Staff of TNC and JCDT met together in Jamaica to prepare the DIP in March 1998. At annual retreats, staff from TNC and JCDT discussed annual work plans and Jamaica and TNC staff were familiar with the DIP. (b) Indonesia field staff were generally not familiar with the DIP and did not use it as an evaluative management tool. Conclusions H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 8 a) Although there was agreement at the outset between JCDT, TNC and PVC as to the goals, objectives, activities and indicators stipulated in the DIP, it seems that it was not used as a management tool by either JCDT or TNC to measure progress of activities and deliverables or impact. b) Likewise the DIP was not a useful management tool in Indonesia. Recommendations USAID/PVC should improve the quality of DIPs, or discontinue them as a requirement. It would be wiser to focus on impact via use of standard project management tools, such as the logical framework and review of annual work plans. Perhaps PVC could make greater use of evaluations and periodic PVC/PVO meetings as well. 5.1.4. Major Successes and Shortcomings in Implementation Implementation Experience at a Glance Major Successes Major Shortcomings 1. Strengthening the capacity of the COOs in Jamaica to work with the LACs to articulate community needs and work together to find solutions. 1. The inability of TNC to establish a local Indonesia NGO (YPAN) and to work effectively with its main partner in Jamaica (JCDT), leading to termination of YPAN and early severance of the TNC/JCDT implementation agreement. 2. The Community Assessment Tool was institutionalized at TNC and has been implemented in areas throughout TNCís regional offices (LAC and APR) as the Site Conservation Planning process has been adapted. 2. The severe underperformance of the economic development component of the project in Jamaica.. 3. Pursuit of Co-Management Agreements in both countries among government, NGOs, and business. 3. TNC did not meet its match obligations in Jamaica, compromising implementation. 4. Close to the end of the project and post project, a number of models developed under the Indonesia project were under consideration for replication in other regions of the country by TNC and other donors. 4. No case studies were completed or a formal venue established to share lessons learned in either Jamaica or Indonesia. 5. Both countries learned from the experience that it is not advisable to work with only one local partner, that a diversity of approaches to partnerships (joint work, contracts, grants, etc.) is useful, and that proper attention must be paid to the partnership itself and each organizationís role in the partnership dynamic ñ not just through assessing the institutional capacity of the LNGO. 5. A business plan and/or sustainability plan for JCDT was not done. Income generating activities that were to assist with the financial sustainability of JCDT were not designed or implemented. JCDT was not able to undertake a major fundraising campaign as a result of this grant. 5.1.5 Impact Results Findings: H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 9 In the absence of useful impact measures in the DIP (see Section 5.1.2), the evaluation team assessed each objective stated in the DIP, as follows: TNC Overall (a) Objective (1) relates to building TNCís capacity to support partner organizations in enterprise-based approaches to conservation, through a combination of staff development, institutional partnerships, documentation and sharing of lessons learned. 1. On the positive side TNC has hired an ecotourism coordinator, a community conservation specialist, and an Institutional Development expert for the LACD. It has gained both positive and negative experiences in attempting to promote enterprise development through LNGOs and consulting organizations. However, it appears that only the community conservation specialist actually worked on the project. 2. But, few TNC staff appear to have been trained in institutional issues and few formal lessons learned appear to have been disseminated throughout TNC. (b) Objective (2) relates to building TNCís capacity to support partner organizations in developing site￾based constituencies and coalitions through a combination of staff development, institutional partnerships, documentation and sharing of lessons learned. 1. It would appear that the individual country programs have learned a great deal about how to work with LNGOs, constituencies, and coalitions in both countries. Many of the lessons were difficult and resulted in little on-the-ground impact. But, often mistakes are made on the path to learning. 2. Very little documentary evidence exists that these hard-earned institutional lessons on the ground have been formally integrated into TNCís way of doing business. (c) Objective (3) relates to extending the reach of TNCís assistance programs in organizational development and financial sustainability by providing tools to in-country NGO partners who can then share them with their own partners and community based groups, creating a multiplier effect. 1. The central aspect of this objective was to work in partnership with YPAN and JCDT. The project was ultimately not successful in strengthening either organization.. 2. However, TNC/Indonesia does appear to have been able to improve the capacity of its Forum partners in Lore Lindu to work with local CBOs. No multiplier effects were apparent by the end of the project. 3. The CAT and SCP approaches appear to have been successfully applied in the project and have the potential for successful application elsewhere. 4. Success has been impressive in the way in which TNC directly (in Indonesia with respect to KNP) and through its partner (in Jamaica) has helped governments develop the institutional and financial approaches that might eventually translate into sustainable protected area management. Although government was not clearly stated as a target of this objective, it remains an important accomplishment. Jamaica (d) Objective (1) relates to strengthening environmental advocacy and local economic development within buffering communities to BJCMNP. 1. There is ample evidence that TNC, through JCDT, made significant progress in supporting community advocacy development (see Section 5.3.1.). 2. However, little impact can be reported with respect to local economic development as no successful enterprises had been developed by the end of the project, and very few business plans completed under the project showed promise. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 10 (e) Objective (2) relates to building JCDTís financial capacity and sustainability of resource flows for park management. 1. Progress was made on the co-management agreement (which has the potential to provide funding for park management and JCDT itself); but 2. Little progress was made on the outputs that would have contributed to financial sustainability of JCDT (See Annex B) and TNCís premature termination of the agreement with JCDT left JCDT somewhat in the financial lurch; nevertheless 3. It is possible, however, that in the long-term the forced independence of JCDT from TNC may induce it to be more creative in obtaining other funding. (f) Objective (3) intends to enhance TNCís ability to support enterprise-based conservation. Apart from the organizational-wide items mentioned above, little progress was made in achieving this objective. Most of the assistance was contracted out to Jamaican organizations, with little impact. Indonesia (g) Objective (1) related to the establishment of YPAN as a strong national conservation organization. This must be considered a failure since YPAN no longer existed at the end of the MG. (h) Objective (2) related to developing site based coalitions as constituencies for conservation in the two parks, with the primary focus being Lore Lindu National Park. 1. An existing coalition ñ FKLL, Partnership of Lore Lindu Coalition ñ was maintained in LLNP and is continuing to function effectively. In addition, FKLLís NGO members appear to have benefited considerably from the training provided to them (although, without a site visit, this was difficult to verify). 2. TNC in KNP is utilizing a looser network of organizations to provide the same coalition function. While this was a lesser priority (than LLNP) in the project design, the individual members appear to be fulfilling an important role in helping aggregate community interests. (i) Objective (3) concerned documenting and disseminating lessons learned on community enterprises. 1. Very few lessons learned were documented during the project, and none were disseminated widely; 2. However, since the end of the project additional lessons learned have been documented and models developed under the project are being applied elsewhere in Indonesia on an ad hoc basis. For example, Spawning Aggregation Site monitoring techniques developed in Komodo are being applied in Belize, and in Bunaken, Papua, and Karimun Jawa, Indonesia; and in Kalimantan TNC will integrate lessons learned in cocoa production. (j) Objective (4) was to build TNCís capacity to support partner organizations in developing site based coalitions and constituencies. 1. Few activities were completed in direct pursuit of this objective under project funding, outside of the Forum described for LLNP. 2. However, the Site Conservation Planning Tool was adapted for application elsewhere and TNC reports that it has recruited a CPD with broader partnership experience, that TNC/Indonesia has recruited staff with greater capacity to assist NGOs than the previous staff, and that TNC/APR has recruited CLI staff with partnership experience and based in the region. 3. TNC has changed its approach to working with partners in Indonesia to include a broader range of partners and to use a more diverse range of partnership modes. 4. Moreover, as seen in discussion of objective (2), TNC has improved its ability to provide assistance to partners in technical areas; and 5. The Komodo program has defined partner organizations to include government and is successfully building a central government, local government, and USPVO coalition that is H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 11 poised to integrate greater input from the private sector, communities, and potentially LNGOs. Conclusions: (a) TNC has failed in working through its LNGO partners in both Indonesia and Jamaica ñ a central component of the project. (b) However, both country programs as well as LACD and APR staff report that TNC has learned valuable lessons in LNGO partnering from the experience that could inform future partnering ñ thus improving TNCís capacity. (c) Similarly ñ depending on what emerges from current restructuring ñ TNC has increased its staffing in areas relevant to building coalitions in the LACD and APR. (d) The project has had very significant policy/advocacy successes in both countries at the community, local, and national levels. This may be the most significant on-the-ground impact of the project. (e) In a related accomplishment, TNC made significant progress in developing coalitions in both countries. (c) The project was not successful in documenting and disseminating lessons learned in the project although TNC is beginning to apply various models developed in Indonesia in other places in the country on an ad hoc, but relevant, and significant basis. Recommendation Sharing of lessons should be done on a more systematic basis, based on clear hypothesis testing, with structured framework for including activities, assessment of quality of outputs, monitoring impacts, and a process for review of these results, and making in-course changes to strategies and activities, which preferably are formally approved by USAID/PVC. 5.2 Assessment of project model and hypotheses 5.2.1 Project hypotheses articulated in CA Findings (a) The project hypotheses articulated in the CA are as follows: The underlying hypothesis is that conservation of biological diversity ñ park reserve protection depends on conservation NGOís and CBOís increasing their financial self-sufficiency from in-country sources and on increasing their political voice through coalition building. The second hypothesis is that a grass roots economic development based on products derived from the protected area can build a grassroots constituency for conservation (Cooperative Agreement, page 8.) (b) The project was unable to significantly increase the financial self-sufficiency in Jamaica of its LNGO partner, JCDT, or of the LACs (the CBOs); (c) Its NGO partner in Indonesia, YPAN, was no longer functional by the end of the project. (d) The Jamaica program winded up promoting community conservation through targeting formation of a small number of individual enterprises which were not based on natural resources related to the H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 12 protected areas (for example, one of the highest potential enterprises was a hardware store). International experience suggests, that to promote conservation, such enterprises must be based on protected area (PA) resources. (e) The project was unable to increase significantly grass roots economic development based on products derived from the protected areas in either Jamaica or Indonesia during the project period (although the Komodo area may have potential for the post-project period); (f) However, the project was able to increase the political voices of the LACs and JCDT in Jamaica through coalition building and communities appear to appreciate JCDTís efforts to support LAC efforts to address broader community social issues. Similarly, the Lore Lindu National Park (LLNP) project appears to have helped develop the capacity of local NGOs, and their Forum, to operative effectively. In Komodo, the project has achieved dramatic success in improving the enabling environment, making progress in having local government recognize the Parkís authority over certain resources and in working toward a comprehensive co-management agreement. (g) No framework was established for testing the hypotheses under the project. This is an important shortcoming since the MG was meant to be a pilot effort to inform future TNC activities. Conclusions: (a) Due to significant design and implementation shortcomings in Jamaica and Indonesia, we are not able to test the hypothesis that more financially sustainable CBOs and NGOs will increase conservation or that greater economic development in the buffer zone might improve grassroots constituency for conservation. This is a serious project shortcoming. (b) It does appear, however, that JCDT efforts to support a broader range of community interests than pure conservation and enterprise development is helping improve community/protected area relations in Jamaica. Likewise, efforts in Komodo and Lore Lindu may well be contributing to positive conservation outcomes in the future. (c) Project monitoring systems are not structured to determine whether or what aspects of improved relations have led to better conservation outcomes. Recommendations: (a) TNC/WO should broaden the above development hypothesis to recognize the importance of addressing broader community-identified social interests ñ in addition to enterprise development ñ when attempting to build local constituencies for conservation. This appears to be recognized already at the local implementation level. (b) TNC should more systematically explore how communities can be effectively mobilized for conservation in the project area, especially where there is the most near-term chance for success and appears to have the most favorable funding scenario ñ Komodo National Park and Indonesiaís Center for Marine Protected Areas. This would involve integrating a greater social science component into the programís already well-developed natural science analytic capability. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 13 5.2.2 Effectiveness of decentralization model used by TNC Findings (a) The MG program did not have a single grants manager for the combined activities in Jamaica and Indonesia, crippling coordination and communication between the Country Programs, and throughout TNC overall. However, the Director of CLI/APR did pitch in to coordinate annual report preparation; Jamaica (b) Until the final year of implementation, TNC leadership of the Jamaica program was based at TNC/WO and executed via electronic communication and periodic visits to Kingston by technical staff from Virginia. The major point of interface for this individual was the Executive Director at JCDT; (c) TNC/WO oversight was supplemented by an in-country TNC Coordinator, housed at JCDT, with a counterpart JCDT MG coordinator. (d) An annual Country Team Review process served the function of oversight and peer review of project implementation in Jamaica. (e) Once the DIP was developed, TNC delegated most Jamaica program implementation decisions to its local partner, JCDT, and was reluctant to be involved in program decision-making, except for the annual plan development; (f) Although the initial Jamaica Country Program Director (CPD) based at TNC/WO appears to have been aware of serious implementation shortcomings, it does not appear that this information was widely shared with TNC/WO colleagues. TNC appears to have been reluctant to apply the pressure required to press for improved management by its local partner or to help JCDT address its own shortcomings. (g) Posting of a full-time resident CPD in Jamaica in the projectís third year (see Annex A) was an effort by TNC to improve implementation. Reports by the new Jamaica CPD of project implementation problems do not appear to have been adequately heeded by TNC/WO staff. (h) By the time sanctions were imposed on JCDT (September 2000) and very little had been accomplished in the economic development portion of the grant. Indonesia (i) The Indonesia program was initially managed by a TNC/APR Senior Advisor, who had a fairly ìhands-offî approach to the program, and permitted significant levels of direct access to the project by TNC/APR technical officers. A training coordinator was hired by TNC/Indonesia to address the training portion of the grant for YPAN and TNC. By the second year the new CPD managed communication to ensure clearer lines of communication and by December 1999 had hired a former USAID/Indonesia local staff person to serve as the manager of the PVO project. (j) Approx $80,000 in Komodo expenses were rejected by TNCís US-based Grants Specialist as unallowable expenses under government rules. There was considerable back and forth between TNC/I and the Grants Specialist in attempting to resolve the issue. The Grants Specialist subsequently provided training for TNC/I staff in USG contracting rules in mid-2001 and prepared a guidance chart to help field staff anticipate which expenses would be allowable under USG regulations. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 14 (k) The Indonesia field office sends monthly financial reports to Hawaii, where overall financial position is determined. Field staff normally did not receive those reports until two months after the end of the period.4 (l) Initially the YPAN board became accustomed to communicating directly with the APR office, rather than the CPD, regarding important TNC/YPAN issues. The level of YPAN funding was determined by APR, not the CPD. But, the CPD resided in-country (while the Director of APR lived in the USA), the CPD was more familiar with the program, and was accountable for the success of the partnership. In August 2000, the Director of APR delegated primary YPAN communication responsibilities to the Acting CPD, which made local management of the partnership more effective. (m) Some key APR technical advisors appeared to be in favor of sustaining the commitment to having YPAN serve as TNCís implementing arm whereas most of TNCís field staff appeared to be more united wanting to regain control over implementation. In some cases it appears to have been a classic ìthe home office doesnít understand field realitiesî vs. ìthe field has lost touch with the purpose of the grant and the meaning of TNCî. Tension was evident. Conclusions (a) Failure to provide a single project manager for the MG, or at least to provide some sort of team structure that promoted coordination between the Jamaica and Indonesia components, may be one of the main reasons for lack of true synergy between the two country programs in the project. (b) It would appear that a culture of focusing on positive news at TNC/WO, and lack of independent monitoring of program implementation, reduced the ability of top TNC management to appreciate the importance of implementation shortfalls in the Jamaica program. On the other hand, difficulties in the Indonesia program appear to have been widely known within APR. (c) Although the original Jamaica Country Program Manager at TNC reports expressing his dissatisfaction with JCDTís performance, it appears that TNC was reluctant to pressure JCDT to higher performance. Perhaps this was due to fear of souring the relationship. This appears to have contributed to an unhealthy relationship where TNC, as the funding PVO, was unable to exert needed influence to obtain the outputs from the LNGO needed to meet its obligations to the MG. (d) If TNC had engaged earlier in more openly critical dialogue with JCDT on implementation issues, and taken responsibility to ensure that JCDT and TNC jointly addressed the implementation targets articulated in the DIP (including providing technical assistance where necessary), many of the implementation problems in Jamaica might have been avoided. (e) During the first two years of the Indonesia project (first year of field implementation) local ownership of the project was limited. Field personnel received uncoordinated inputs from a range of players, including APR Human Resources, Grants Specialist, and Conservation Leadership (with CLI taking the de facto lead), the CPD, and cross-YPAN/TNC technical communication. (f) In mid-1999, the new Indonesia CPD observed that spending was slow and management confused. He took actions to improve internal management of the project through clearer reporting practices and hiring of a PVO manager to coordinate project activities. (g) Field staff and APR Grants management staff did not have a common understanding of what constituted allowable expenses under USAID funding. This led to a large amount of expenses being disallowed by APRO for Indonesia and charged to unrestricted funds. 4 Note: TNC has recently improved turn around time significantly, with monthly reports being returned to the field within three weeks of the end of the period. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 15 (h) During the project period, slow turnaround in the APR Financial office made it difficult for the Indonesia MG Manager to know the financial position of the project. Since then, the system has improved. (i) TNC was wise to further decentralize management in Indonesia by delegating responsibility for communicating with partners to the CPD and providing greater autonomy in management. (j) Differences in perspective between home and field office on the fundamental issue of whether or not TNC should maintain its relationship with YPAN may have slowed the termination of YPAN. Rather than being counterproductive, this ìcreative tensionî appears to have provided a constructive, if interpersonally challenging, berth which may have enhanced the quality of the ultimate decision. The field view prevailed, but only after certain core TNC issues were discussed. Recommendations (a) Future multi-country projects that are intended to achieve synergies should have a single overall manager or a team structure that promotes synergies. (b) CPDs should be delegated full authority to manage in-country partnerships. (c) To compliment the training in management of USG contracts provided to TNC/Indonesia staff (provided by TNCís Grants Specialist) it would be helpful if, as a standard practice, the Grants Manager would review proposed work plans for USG-funded projects. He/she could work with the project manager to ensure that all planned expenditures are consistent with USG regulations before costs are incurred. 5.2.3 In-country sub-awards Findings: Jamaica (a) Table E1, in Annex E, provides a listing of the sub-awards under the grant. (b) The primary sub-award in Jamaica was to JCDT, with sub-contracts let by TNC to Development Options, Ltd. (DO) and from JCDT to University Technology Jamaica; (c) TNC has been working with JCDT since 1989. (d) JCDT received a sub-award agreement from TNC for their work under the MG program in Jamaica, but no overarching MOU exists between the two organizations. (e) The sub-award does not provide incremental leverage for TNC to work with JCDT to collaboratively address implementation issues as they arise, apart from the annual plan process and the threat of cutting funding (f) By December 2000 relations between TNC and JCDT had deteriorated drastically due to miscommunication, perceptions by TNC of sub-standard performance by JCDT, and the decision by TNC to cut MG funding to JCDT. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 16 (g) Both TNC and JCDT expressed uncertainty over the intentions of the other partner and suggested an MOU covering fundamentals of their relationship would have helped avoid problems. (h) TNC appears to have applied an institutional assessment of JCDT prior to the MG and financial assessments during implementation. (i) The financial assessments appear to have been utilized to determine that JCDTís financial systems were capable of accounting properly for continued funding. (j) However, there is no evidence that the institutional assessment was used to highlight the kind of institutional problems that contributed to the ìbreak-upî between TNC and JCDT (such as low level of board participation, poor intra-organizational communication, and low implementation performance.) (k) Reluctance on the part of TNC to acknowledge sub-standard LNGO performance within TNC, and to address those shortcomings, appears to be partly a function of corporate culture, and partly a result of TNC ìhaving all its eggs in one basketî by focusing all its institutional partnership efforts on a single LNGO in Jamaica. Indonesia (l) The main sub-award in Indonesia was to YPAN, although various pacts were also made with government agencies to coordinate implementation. (m) The main TNC/YPAN MOU detailed YPANís role as implementing TNCís programs in LLNP and KMP and provided a sense that YPANís implementation role would expand as partnership activities grew over time (see Section 6.1). (n) As indicated in Section 6.1, TNC gradually decided that it would, in fact, like to maintain a significant implementation role. However, the MOU was never adjusted to reflect this reality and very little TNC/YPAN open discussion of this gradual change occurred. Conclusions (a) MOUs with host country partners can be effective to help manage mutual expectations between TNC and LNGOs only if they are actually followed. But they should also be treated as living documents which serve as a basis for discussion regarding partner roles and responsibilities, and are formally changed to meet any changes in those roles and responsibilities. (b) Financial monitoring systems appear to be a useful tool in TNC partner management; (c) TNC does not appear to utilize partner institutional assessment data to inform the way in which it works with its partners. Recommendations: (a) TNC should ensure that its own staff follow the provisions of a properly prepared and agreed upon MOU between TNC and its key in-country partners. (b) Each member of the partnership needs to remain vigilant to evolutions in attitude and behavior that are inconsistent with the terms and spirit of the MOUs. Such role/responsibility/vision migration is H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 17 natural. Where it occurs, however, the Partners must openly discuss the issues, using the MOU to ground the dialogue, and modify the MOU ñ or their behaviors ñ as appropriate. (c) TNC should facilitate more effective institutional assessments of its partners and utilize the data to manage its relationships with partners. This is particularly important where TNC maintains close ties with only one LNGO. (d) If TNC were to diversify its local relations by partnering with more than one LNGO, it might have greater success at accomplishing its conservation agenda. 5.2.4 Effectiveness of TNC/JCDT self-assessment tools Findings (a) TNC utilized the Parks in Peril ìScorecardî as virtually its only impact metric in Jamaica. (b) TNC/Indonesia reported high variability among different individuals using the PiP Scorecard and reported in the final report that the metric was no longer valid. (c) Part way through the Jamaica project TNC removed from the scorecard indicators reporting on overall PA revenues and diversification of funding sources. (d) The PiP Scorecard was not directly tailored to the needs of the project, but was an international template used by TNC to measure protected area conservation success. (e) At the height of the tension between YPAN and TNC in Indonesia, the Director of TNC/APR hired (using Packard Foundation funds) a former TNC staff person as a consultant to perform an assessment of the YPAN/TNC partnership to help inform future TNC management decisions. The consultant concluded that TNC/Indonesia management lacked the requisite skills and commitment to support institutional strengthening of YPAN, but viewed the partnership as relevant and salvageable, and gave a number of recommendations of how to recover the partnership. However, only a few of these recommendations were actually followed by TNC/APR management, and indeed took actions which led to the termination of YPAN as a separate entity. Conclusions: (a) Eliminating Scorecard indicators for Jamaica dealing with financial sustainability had the effect of ìmoving the goalpostsî for the project, and eliminating the need to report on the failure of a central aspect of the project: financial sustainability. (b) The PiP Scorecard was not the correct impact measure for the MG, it was not applied consistently, and was subject to manipulation by project implementers. (c) TNCís MOUs and self-assessment tools do not appear to have served it well in the MG. There were no formal reviews of the partnership arrangements by TNC along with their partners, and no use of conflict mediation provided for within the MOU. An external evaluation of the TNC-YPAN partnership did not present objective criteria, was not openly reported, and only some of its recommendations were acted upon. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 18 Recommendations (a) TNC should seriously review its ability to conduct assessments of partnership arrangements, including its use of self- assessments, MOUs, conflict mediation, and external evaluations. TNC needs to improve the way in which ñ organization-wide ñ it plans for and integrates difficult lessons emerging from such assessments and evaluations.5 (See also Section 7.8.) 5.2.5 Replication and Scale-up of Approaches in Project Area or Elsewhere Findings (a) The enterprise models attempted in Jamaica to promote conservation were not successful, nor were they replicated. (b) TNC reports that its enterprise models in LLNP were not at a sufficient scale to encourage significant conservation and were not replicated. It is possible that the research into cocoa production (implementation of which was halted in LLNP when the park manager declared cocoa should not be planted in the LLNP buffer zones) will be used elsewhere in TNC or USAID project areas. It is also possible that some approaches developed in Komodo (such as fish aggregation devices and biophysical monitoring techniques) will be applied elsewhere as well. But, as of the end of the project period, no successful replication of enterprise models developed under the project had occurred. (c) Developed by TNC, tested in Jamaica, the Community Assessment Tool (CAT), has become an important diagnostic tool to assess project areas, and is being used throughout TNCís regional programs. Conclusions (a) No successful replication of TNC field models has occurred, although some learning about what not to do may have been acquired by the Jamaica program. TNC reports potential for replication of Indonesia models in other parts of that country. (b) The CAT evolved considerably in Jamaica and has become a useful sociological analysis tool for TNC internationally. 5.3 Advocacy under the project 5.3.1 Advocacy activities and impact Findings (See Annex D): Jamaica Program (a) TNC relied on JCDT to conduct advocacy in Jamaica. (b) JCDT effectively managed an advocacy program at the national level that resulted in a Co￾Management Agreement for the BJCMNP and the establishment of a national network of LNGOs involved in PA management, JPAN (Jamaica Protected Area Network), which continues to provide a productive PA management policy forum. (c) At the beginning of the project community mistrust of BJCMNP management ran high. 5 It is important to note that this was not the case among the TNC staff that participated on this internal/ external evaluation team. Their objectivity and insights were invaluable in developing this report. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 19 (d) JCDTís community outreach program supported Local Area Committeesí (LACsí) efforts to advocate for community interests with the BMJCNP and other government officials. LACs have managed to secure resources from the BMJCNP and other government entities on behalf of their communities. JCDT has helped communities develop important bridges with other agencies, including UNICEF, U Tech., and the Canadian-funded Environmental Action Program. Community/park relations have improved. Indonesia Program (e) TNC established a Forum in LLNP, consisting of LNGOs working on issues related to LLNP resources inside and outside the park. The Forum serves as a discussion arena among GOI, civil society, and the community. (f) In Komodo, TNC focused on utilizing tripartite forums (GOI, TNC and Local Government) to shape policy reform and support implementation. The results have been very impressive, including:  Ratification of local ordinances that (1) recognize the authority of Parks officials over resource use regulation in the KNP, and (2) regulate the use of certain fishing equipment (air compressors) thought to enable over-utilization of the fish resource;  Establishment of a working group (including government, TNC, and local community and private sector interests) to develop the concept of joint-venture management of important park management functions;  Enforcement coordination among GOI law enforcement agencies;  Work with local authorities to control immigration;  Coordination of local government across district boundaries; and  Very significant progress towards establishing a joint management agreement for KNP. (g) Many analyses were completed in relation to implementation of the above initiatives in KNP. (h) YPAN staffís support of a political demonstration in Palu was viewed as being overly confrontational by TNC and put strains on the TNC/YPAN relationship as TNC feared that TNC would be associated with YPANís actions (see Section 6.1.) Conclusions: (a) The advocacy portion of the project has achieved very significant results at both national and local levels in Jamaica and in Indonesia.  In Jamaica, LACs now appear poised to provide a positive community foundation for further work in community conservation with respect to the BJCMNP. National and local level advocacy on behalf of PA management was an important accomplishment under the project.  In Komodo, TNC has been very successful working in a tripartite regime (GOI, Local Government, and TNC) that is poised to manage significant donor inputs into a potentially sustainable public/private paradigm  In Lore Lindu, the Forum is adjusting its capacity and expectations to a reality without TNC funding and appears committed to continuing to work with government. (b) Differences in advocacy strategies can create discord among partners if there is not a harmonizing in advocacy approaches or an agreement to pursue different strategies. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 20 5.3.2 Partner/PVO roles in advocacy Findings (See Annex D) (a) Although TNC played an important role in policy development in Jamaica prior to the grant, they were not active during the project period. (b) JCDT was responsible for virtually all Jamaica advocacy work under the grant (except for minor support in developing JPAN from TNC at the Conservation Training Workshop). (c) JCDTís Executive Director was experienced in policy dialogue in the Jamaican environmental community and its Project Coordinator played a lead role in establishing JPAN. (d) JCDTís Community Outreach Officers (COOs) were instrumental in facilitating LAC advocacy with local and national government offices in Jamaica. (i) TND relied on the Forum it established in LLNP to work on advocacy issues and implemented its own policy dialogue with KNP officials, local government, and diving operators in the Komodo area. (e) TNC worked with YPAN in its initial work with the Forum in LLNP and then picked up the initiative once YPAN departed the scene. (j) Where its local partner was active in advocacy programs (beyond working with the Forum), it led to conflict between TNC and YPAN. TNC exclusively employs non-confrontational approaches to influencing policy where as on two occasions YPAN was believed by TNC to have supported open demonstrations against the government (including one instance against TNC as well.) Conclusion (a) JCDT appears to have had adequate existing capacity to advance the advocacy agenda at project start￾up. The model of the USPVO (TNC) delegating responsibility for advocacy work to the LNGO (JCDT) appears to have been successful in this case as TNC selected a competent advocacy partner and its approach was consistent with TNCís approach. (b) YPANís confrontational advocacy campaign was not in harmony with TNCís collaborative approach. On the one hand, it so alienated TNC that it was one of the final straws in the decision to fold up YPAN as a separate entity. On the other hand, YPAN was well within its rights as an indigenous NGO to adopt strategies that varied from TNC. (c) Properly managed, there is no reason why TNC could not maintain its non-confrontational approach (befitting an international NGO) while retaining the flexibility of allowing partner LNGOs the freedom to act more aggressively for change. Recommendations (a) TNC, and other USPVOs, need to be certain that their approaches to policy reform and advocacy are in harmony with LNGO partners or serious political ramifications are possible. (b) Establishing an ìarms lengthî distance between LNGOs and TNC would provide greater latitude for LNGOs to adopt advocacy tactics at variance with TNC while not jeopardizing TNCís status in a country by being associated with internal political issues. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 21 5.4 Implementation Lessons Learned 5.4.1 Implementation tools: utilization and learning Findings (a) As indicated above, in Section 5.4.2, the Community Assessment tool was developed under this project and found to be very valuable to TNC internationally. See Section 5.1.5 for tools and models that were applied beyond the project area after the project period. (b) As indicated in Section 5.2.1, above, the Parks in Peril Scorecard was found to be a less than appropriate tool as it was applied in Jamaica. Recommendations (a) Although there can be efficiencies in applying standard tools to different projects, one must avoid installing them ìwhole clothî without first determining to what extent they need to be adapted to suit local circumstances (such as the PiP Scorecard). 5.4.2 Key TNC/JCDT capacity constraints to implementation Findings (a) JCDT was the first NGO to be delegated stewardship for a protected area in Jamaica. That area was BJCMNP. JCDT had prior experience (although with mixed success) working with communities in the park, was a national leader in protected area policy, and had sufficient numbers of staff to implement the project. (b) TNC had a history of working with JCDT, helped the GOJ establish its protected area network, and is respected as a world class conservation agency in Latin America and the Caribbean. (c) However, neither organization is recognized for its accomplishments in promoting economic development or community mobilization (which, of course, is one of the reasons TNC pursued this capacity building MG). (d) In almost all cases, staff appear to have had the appropriate technical backgrounds and training for the work they were assigned. In fact, some TNC junior staff performed well above the level that might be expected. (e) However, it did appear to the evaluators that some very key staff in both organizations, while technically qualified, lacked the communication, transparency, and team skills as well as management judgment required. (f) These shortcomings appeared to be at the source of the fundamental partnership problems that led implementation failures on the ground. (g) Individual shortcomings were exacerbated by what appears to have been a lack of accountability at TNC to report on project shortcomings, to hold project managers to work plans, or to ensure that project managers developed alternative plans to take into account implementation difficulties (see also Sections 5.1.1, 5.1.2, 5.2.2, and 5.2.4). H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 22 Conclusion The implementation failure was directly related to management and communication difficulties which led to the TNC/JCDT partnership failure. Both could have been more effectively managed if there were a stronger culture of accountability and joint commitment to success. Recommendations TNC needs to examine how to identify partnership and implementation problems as they arise and address them proactively. More effective monitoring systems, linked to project manager accountability, would be important elements of a solution. 6.0 PARTNERSHIP QUESTIONS 6.1 Analysis of Partnership Schemes The partnerships are portrayed graphically in Annex E. In the interest of brevity the following section presents a summary of lessons learned from working with the primary partner, YPAN in Indonesia. Findings (See Annex A for a timeline of key events) (a) ìYPANís history is unique. In response to an internal TNC directive requiring all country programs to fundraise in the communities where they operate, but faced with the constraint that TNC was not permitted by Indonesian law to fundraise directly, TNC created YPAN in 1995 as a means to attract and receive local funding. At that time YPAN had no staff, no infrastructure, and no separate sources of funding. YPAN in fact existed principally on paper only. Then in 1997, the Government of Indonesiaís Secretary Cabinet (ëSetKabí) directed that YPAN could not exist as a sub-entity of TNC. At that time TNC chose to formally separate itself from YPAN, while simultaneously pledging its resources to YPANís development.î6 (b) In pursuing both the USAID/Indonesia NRM II program and the MG program, TNC saw an opportunity to breathe life into YPAN by providing YPAN capacity building (through the MG, awarded September 1997) and funded field activities (NRM II in LLNP, and the Komodo program.) However, YPAN did not yet have a real identity separate from TNC, a sincere drive of its own, or significant field implementation experience.7 Moreover, the Executive Director (hired 1.5 years into the MG) was not experienced in conservation, nor was the Director of Development an accomplished fund raiser at the time of her transfer from TNC to YPAN. (c) At the time of the initial period of the grant, TNC was planning on phasing out its on-the-ground presence and appeared committed to establishing an indigenous NGO that would work towards TNCís vision. The MOU between TNC and YPAN states: ßII, B: Roles and Responsibilities. In the current projects listed in Appendix I, and for any future joint projects, YPAN and TNC shall agree to clear roles and responsibilities for each organization. Unless otherwise stated, these will include: 1. YPAN shall assume the primary responsibility for all onsite activities. TNC shall provide technical expertise and advice to each project and, in the absence of onsite capacity to implement specific activities, shall provide staff assistance until YPAN counterpart staff can be recruited or trained. 6 De Ridder, p. 1. 7 Although many YPAN staff had previously worked for TNC, and had implementation experience with TNC, YPAN, as an organization, had no previous track record, or even an Executive Director. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 23 (d) Accordingly, the top YPAN leadership anticipated that they would have flexibility to implement their vision with TNC support. The pursuit of the MG reinforced that notion and provided specific funding for the institutional strengthening of YPAN, and especially for the hiring of an Executive Director and Director of Development. (e) YPAN and TNC shared some members of their Governing Board and Board of Advisors, respectively. It was difficult for many to distinguish between the two organizations. (f) However, TNC gradually came to reassess its vision for Indonesia, naming a Country Program Director (replacing the title Senior Advisor) in 1999, and deciding that TNC would continue to have an implementation role in Indonesia. (g) It is not clear that the expanding numbers of TNC staff, beneath the level of Senior Advisor/CPD, universally shared the view of TNC phasing out. In fact, it appears that many resented having YPAN usurp their important implementation role. This included concern about how to ensure the quality of YPANís output and whether YPANís focus would sufficiently support conservation outcomes. This may partly be the result of poor communication on the rationale behind the shift (see Section 7.5.) (h) TNC, feeling the fiduciary obligation to spend USG funds correctly and to ensure quality control, required approval of YPANís work plans. YPAN appears to have perceived this as a threat to its autonomy. It also appears YPAN lacked the understanding to create work plans at the standard expected by TNC (see Section 7.5.) (i) Protracted struggle over work plan development in Palu (LLNP) led to a situation where local TNC and YPAN staff would not speak to each other, even though they worked in the same building and on the same project. Intervention by LLNP authorities, in April 2000, demanded that the two partners get their act together and begin implementation (implementation had been delayed for eight months during this process.) Soon after a sub-contract was signed between TNC and YPAN for LLNP. 9 (j) In Komodo, TNC gradually came to feel that YPAN and TNC had different emphases in a classic conservation dilemma: TNC believed in providing economic opportunities to communities in order to promote conservation whereas TNC perceived that YPAN was interested in economic empowerment of communities, even if it might have consequences adverse to conservation. For example, TNC reports that at one point YPAN staff began a project to assist communities to market fish caught inside the park. (k) In Komodo, TNC/CMP staff became so frustrated with being held accountable for TNC funds, while having handed over control to YPAN, that they demanded, and received, a ìSecondment Agreementî in October 1999. This made YPAN little more than a personnel agency for staff that had once worked for TNC, then were managed and employed by YPAN, and now would be employed by YPAN and managed by TNC. These changes led to substantial confusion among communities, staff and other stakeholders. (l) YPAN and TNC further agreed that YPAN would manage all community development, education and awareness efforts and that TNC would be responsible for scientific and enforcement roles. Over time, TNC began to feel that they were being viewed more as the ìbad copî to YPANís ìgood copî role since they were effectively removed from positive associations in community membersí consciousness. By June 2000, TNC replaced all YPAN staff with TNC staff and really ramped up the rate of staffing and implementation. However, there was a gap where no community development work was completed June 2000 to November 2000 during the transition. 9 No sub-agreement between TNC and YPAN was ever executed for the Komodo efforts. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 24 (m) YPAN was believed to have supported a demonstration against LLNP authorities (related to illegal logging) in May 2000. TNC reports that LLNP authorities implicated TNC in subverting its authority, due to its partnership with YPAN, which had provided support for planning the demonstrations. This was an anathema to TNC as it has a rigid policy of not interfering with local politics. Relations between TNC and YPAN were critically strained at this point. (n) Some staff in TNC had identified institutional capacity as a key constraint to YPAN effectiveness. TNC sponsored a series of workshops for YPAN to attempt to address this issue: Communications Strategy (Oct-99); Strategic Planning Workshop (Nov-99); Financial Sustainability Workshop (Nov￾99); Capacity Building Planning Workshop (Feb-00); YPAN Organizational Assessment (May-00); a consultantís analysis of problems in the partnership (confidential, May-00), and a consulting firmís analysis of options with the partnership (McKinsey & Company, Nov-00). (o) Finally, the head of TNC/APR came to Indonesia to resolve the issue in May 2000. Soon afterward, the Executive Directors of TNC and YPAN left their posts (June and July, respectively.) By November 2000, the Board of YPAN had formerly re-merged YPAN into TNC and YPAN staff returned to TNC, effectively terminating the experiment in partnership and local empowerment. (p) With the termination of a separate YPAN, TNC field staff in Komodo and LLNP report being able to make far greater progress. In LLNP, TNC staff broadened the number and range of NGOs with which they work, as well as the nature of their engagement. In some cases NGOs from the forum would observe a study, in others they would be contracted, in still others a sub-grant relationship was forged. TNC feels that working with a range of NGOs in this way is far preferable than ìputting all their eggs in one basketî with a single NGO, such as YPAN. In the case of Komodo, TNC worked directly with park officials, natural resource users, dive operators, and the same community groups with which YPAN had been working. TNC/ Komodo reported that it was far easier to get things done without working through YPAN as an intermediary. (q) YPAN was expected to implement a broad range of activities in two very diverse areas at a time when they had limited implementation capacity and limited support structure. Conclusions (a) It clear that TNC wanted to have its local fund-raising arm pass muster as a full-fledged indigenous NGO to meet GOI legal requirements for local fundraising for TNC activities. But, in retrospect, which is often 20:20, the idea was ill-conceived. YPAN had never implemented anything significant, it lacked the internal passion that is necessary for NGOs to succeed, and it lacked the technical and institutional capacity to prosper. It was asked to do too much too quickly, without sufficient institutional support and practical technical assistance. (b) YPAN staff do not appear to have fully appreciated what was required for them to become a truly viable NGO. On the one hand they appeared to want to have full autonomy for implementation, but on the other they seemed reluctant to accept the accountability that comes with being an independent NGO (such as completing work plans, timely delivery of services, and managing their relations with their primary donor, TNC.) This is typical of many ìstart-upî LNGOs. (c) TNC appears to have changed its vision for the role of TNC in Indonesia without fully communicating the implications of this shift to its implementation LNGO, YPAN, or modifying their existing MOU. Likewise, TNC has expressed concern over differences in conservation philosophies between TNC and YPAN, but it is not clear that any participatory processes were pursued to arrive at a way for the partners to work out these differences in a positive implementation paradigm, perhaps through revision of the MOU. Much effort appears to have been invested on (needed) institutional H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 25 strengthening of YPAN relatively late in the game. But, relatively little seems to have been invested in harmonizing visions and approaches. (d) TNC experienced fundamental internal conflicts that made establishment of YPAN complicated. On the one hand, at the outset of the project TNC wanted to support the establishment of an autonomous independent Indonesian NGO that would be respected locally. On the other hand, it wanted the NGO to have vision and values congruent with TNC. It also had certain deliverables that it entrusted to YPAN, and it wanted YPAN to grow and get stronger at the same time. Thus, it set up an almost unbearable tension between autonomy and conformity to TNC and producing deliverables while investing in improving itself. (e) TNC channeled all implementation efforts through single LNGOs in each of Jamaica and Indonesia. This produced excessive levels of vulnerability for TNC and dependency for the LNGOs. Recommendations (a) TNC should avoid starting LNGOs from scratch, unless it is in a very graduated manner. It should avoid requiring the LNGO to produce daunting deliverables while establishing itself, and should provide room for the LNGO to develop its own mission. (b) TNC should recognize that the actual partnership needs to be managed, almost as a separate entity. Successful partnerships require attention the same way as do important personal relationships. It is not enough to assess each institution separately. It is necessary to examine the actual bonds, incentives, tensions, and structural issues that help or hinder effective partnership. TNC should assign responsibility to an individual to make management of important partnerships his/her concern. In addition, TNCís Conservation Leadership Initiative (CLI) should supply adequate support to these Partnership Specialists to enable them to be effective. This support could include special interventions to facilitate addressing specific problems. (c) As part of the process described immediately above, TNC, and its partners, should collaboratively agree on the balance to be struck between targeting accomplishment of implementation objectives and focusing on institutional strengthening. TNC should educate donors ñ at the outset ñ on the need for patience in obtaining deliverables if the donor and TNC share the objective of empowering LNGOs. TNC and its partners should periodically assess how well the institutional strengthening/implementation balance is serving the objectives of the partnership and fine tune the balance, as appropriate. In any event, both TNC and its partner should be held accountable for what is agreed upon. (d) TNCís CLI should operationalize the conscious approach to Partnership suggested in Recommendations (b) and (c), immediately above. This should be communicated to TNC staff world￾wide via an updated version of TNCís Value Creation Model, or similar product. (e) This active focus on partnership management should provide an ìearly warning systemî on emerging partnership problems. 6.2 Measuring Institutional Capacity Findings (a) TNC conducted institutional assessments of JCDT in mid the1990s and again at the turn of the century. None of the findings reported to the evaluators indicated significant capacity constraints for implementing projects H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 26 (b) TNC conducted a formal institutional capacity assessment of YPAN in May 1999 during a workshop with YPAN staff. In March 2000, TNC developed a capacity building plan for YPAN. In June 2000, TNC and YPAN staff updated the institutional assessment, by splitting the analysis by headquarters, Komodo, and Lore Lindu. (c) Results were used to help focus the YPAN work plan with respect to institutional strengthening and to inform decision-making regarding the partnership. Some YPAN staff found the measurement process more useful than others. (d) However, the capacity building plan based on the institutional assessment was completed by TNC staff and not YPAN staff. There was only limited ownership of the plan by YPAN staff. (e) A series of TNC-funded technical capacity building reports targeted to YPAN in 2000 make no mention of the root causes of the deterioration of the YPAN/TNC relationship or of any role that TNC may be playing in that dynamic Conclusions (a) As a practical matter, TNC did little to support the organizational strengthening of YPAN, apart from shifting TNC staff and equipment into the YPAN shell. TNC did not increase its own staffís capacity building skills and did not provide much of the basic organizational skill transfer normally required in a new organization. (b) The fact that many of the analyses involved little participation by YPAN staff limited YPANís ownership of the product. (c) While TNC should be praised for conducting two institutional assessments of its main partner in Jamaica, JCDT, it is surprising that the assessments did not appear to point to the institutional constraints that appear to have limited implementation success under the MG. Recommendations: TNC should use more participatory processes in conducting institutional assessments and capacity building plans that will promote ownership by the local NGO. 6.3 Constraints to Partnership In both Jamaica and Indonesia, TNC faced a challenge to implement a partnership with a single LNGO:  In Jamaica, TNC appears to have had difficulty in getting JCDT to implement the project as TNC would have liked. It reports that it had limited leverage over its sole partner. At the same time, TNC did not seem to have identified, in fine enough grain, the implementation difficulties observed at JCDT until relatively late in the project. Once implementation problems were identified, TNC did not seem to have provided the kind of targeted institutional technical assistance that may have helped JCDT get out of the implementation bind. At the same time, it is possible that JCDT would not have recognized the need for change or appreciated TNCís intrusion. However, it is unclear this approach was ever attempted.  As described in Section 6.1, above, TNC and YPAN had very different expectations from the partnership. YPAN expected autonomy and something bordering on unconditional support from TNC. TNC expected to be able to control YPAN, almost like a contractor, after it had spun it off as an independent entity. It also expected strong loyalty to its own mission, strategies and values. The relationship became fatally complicated when TNC decided that it no longer wanted an Indonesian NGO to implement its program, but wanted to handle implementation itself. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 27 6.4 Information Technology Nothing was noteworthy about the use of information technology in the project. 6.5 Use of local networks and service organizations Findings (a) As mentioned in Section 5.3 above, JCDT was instrumental in establishing a national protected area network (JPAN) and used it effectively to promote policy reform in Jamaica. (b) TNC funded a workshop within a larger biodiversity forum (sponsored by Kehati Foundation) which produced an action plan to develop a collaboration mechanism for cross-Provincial management of natural resources in Indonesia (July 2001). (c) TNC and YPAN worked with the Forum in LLNP to facilitate community/park communication. 7.0 PROGRAM MANAGEMENT 7.1 Strategic Approach and Program Planning Findings (a) TNC is in the process of revising its strategy in Jamaica. It will now work with a number of LNGOs and will emphasize a different protected area. (b) TNC is now focusing intensively at ìproducing tangible, lasting results at scaleî as part of its transformation. The results of the MG are helping inform how TNC Indonesia and Jamaica will respond to this organizational initiative. (c) The failure of YPAN has forced TNC/Indonesia to re-think its fund raising strategy, how it presents itself in country, its partnering strategy, and its expansion strategy. (d) The successful adaptation of the Site-based Conservation Planning methodology to Indonesia has helped integrate community input into conservation planning. Conclusion: The project did not target strategic or program planning. Apart from the changes in focus noted above, and fine tuning its SCP, no significant changes in planning systems appear to have resulted from the project. 7.2 Country Initiatives Findings (a) There is no evidence from the Jamaica program that TNC shared lessons learned with other donors and USAID beyond annual reporting requirements. The Indonesia program has successfully communicated to other international organizations (such as WWF and Conservation International) many of the enterprise models developed in Komodo National Park and TNC reports that there is considerable interest in replicating some of them. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 28 (b) The Indonesia program did share lessons learned among its own programs in Indonesia, though mostly after the end of the grant (see Section 5.1.5) (c) Neither the Jamaica nor Indonesia programs provided impact data to local USAID missions. 7.3 Conflict Management (a) Not applicable to Jamaica. (b) Indonesia experienced considerable political and economic unrest during the project period. As a result, TNC has developed contingency plans for each project site as well as for the Jakarta team. Different levels of emergency can be declared for each site, with different contingency plans, based on the risk assessment. An evacuation site has been established in Bali. TNC also uses the US Embassyís security warning as reference. Evacuation routes are posted in each project site. Added security has been installed at each site. TNC has utilized US Embassy Security staff for advice on security arrangements. 7.4 Monitoring and Evaluation Findings (a) See Section 7.8 for a discussion on shortcomings in the annual report to USAID. (b) As mentioned in Sections 5.1.2 and 5.1.3, the DIP was not a useful implementation tool for this project. No other project monitoring systems were employed for the MG, although other monitoring systems were in place in Indonesia to monitor ecological health in Komodo (c) No evaluations were conducted of any of the field projects during the project.10 (d) In several situations consultants were hired to write the Annual Reports for Indonesia and for Jamaica. The drafts would normally then be sent to an individual working in the Development (fundraising) Section of TNC for final edit. Country staff report that occasionally some accuracy suffered from the edits. Conclusions (a) The monitoring systems were not adequate to monitor the progress of implementation, the status of the partnerships (see Section 6.1), or the impact of the matching grants in either country. Recommendations (a) TNC needs to utilize improved monitoring systems at the activity, impact, and partnership levels. (b) TNC should avoid having technical reports drafted by consultants, and reviewed by (fundraising) development staff. This approach leaves the report too vulnerable to error and distortion. Rather, the implementing officers should draft the report and senior technical staff should review it for accuracy prior to submission to donors. These reports should be based on objective impact indicators, preferably collaboratively developed with USAID. 7.5 Overall Management 10 Although the annual reports sometimes refer to the updates of the PiP scoring sheets as ìevaluationsî they are really just updates of monitoring data. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 29 Findings Overall (a) There was no single manager for the overall MG. Rather, the matching grant was essentially managed as two separate grants, with combined annual reporting, in separate sections. Apart from one pair of exchange visits, and sharing of lessons learned in Site Conservation Planning, there was little real synergy between the Jamaica and Indonesia projects. (b) No structure was developed to promote cross-country synergies, although this was a stated objective of the grant. (c) One group of individuals at TNC were involved in writing the proposal for the matching grant. Other individuals, from other TNC programs were responsible for implementing the project. The implementers generally felt little ownership for the project. Jamaica (d) TNC hired a project coordinator in Jamaica during Year 1 of the MG, yet her role was ill defined and her relationship to JCDT was unclear throughout her tenure. JCDT staff felt their direct access to TNC Washington had been severed and resented the change. TNC filled a similar post in Palu for efforts in LLNP. Although YPAN staff reviewed her TOR, and participated on the selection panel, YPAN placed such strict barriers to her participation that she was not able to work effectively and resigned in frustration after fewer than six months. (e) Staff report that neither the original TNC CPD/Jamaica nor the JCDT Executive Director utilized participatory management styles. This appears to have hindered flow of information about the project at both TNC LACD and JCDT (f) TNC had difficulty delivering technical assistance in Jamaica, as evidenced by uncompleted feasibility studies and business plans. (g) The transfer of TNCís US-based Country Program Director left a temporary vacuum at TNC, not only for project supervision, but also for raising funds for the matching grant. Lack of matching funds was the stated reason for not extending the sub-grant agreement with JCDT for the full period of the no-cost MG extension awarded by USAID to TNC. (h) The new Jamaica CPD exerted firm management control. However, given his low opinion of the delivery capability of JCDT, he did not consider raising funds to support JCDT and MG efforts a high priority. The match was not raised. Indonesia (i) TNC had four Country Program Directors/Acting CPDs, with a three-month gap covered by the APR Deputy Director, in the course of the project (Annex A). (j) Management authority was exceptionally ambiguous in Indonesia. In the field, YPAN thought it had autonomy to pursue the technical course it selected whereas TNC felt YPAN was obligated to act consistently with TNCís core values and vision. TNC also felt it needed to apply pressure to YPAN to ensure TNC deliverables under the contract. Certain APR individuals felt a strong ownership of the project and exerted considerable influence at first, but less so as time wore on (see Section 5.2.2.) At the outset, the Senior Advisor seems to have treated the project as a low priority, although it was treated with greater urgency with the change of CPDs in June 1999. Staff shifted from being TNC H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 30 staff, to YPAN staff, back to TNC staff, while performing essentially the same function for a range of managers. (k) The various changes in technical leadership of field activities (first TNC in charge, then YPAN in charge of most implementation in KNP and LLNP, then YPAN reporting to TNC in Komodo, then TNC taking over all implementation in Komodo and staff sent to LLNP, and finally YPAN removed entirely from the picture and TNC taking over complete control) were made in a ìtop-downî fashion and not communicated effectively to field staff of either TNC or YPAN. (l) The original MOU clearly expressed the roles of TNC and YPAN. However, the changes in roles described immediately above occurred without modifying the MOU (see Section 6.1.) This left a considerable amount of ambiguity. There was never an effort by TNC to sit down formally with YPAN to discuss their changing roles and formally revise the MOU, despite recommendations in the Partnership Action Plan, written by the PVO Manager and CLI/APR/DIR. (m) At the outset of the grant TNC planned to phase out its presence in Indonesia and YPAN was considered an important part of its exit strategy. Subsequent changes in field implementation roles, described immediately above, coincided with a growing feeling in TNC that the organization no longer intended to phase itself out, but rather to increase its presence. (n) Although technical field staff appear to have been able to function reasonably effectively, there was considerable confusion among field site managers, YPAN managers, the TNC PVO manager, the CCMP Manager, and community members over YPAN/TNC roles and responsibilities. (o) There was considerable overlap between the running of TNC and YPAN. During her first year of employment in YPAN the Director of Development also continued to perform work for TNC. Likewise, TNC staff initially performed many of the project management functions for YPAN, which lacked the skills to perform them. This appears to have led to a kind of dependency on the part of YPAN for these important functions which was complicated when TNC discontinued providing those services, but had not yet transferred those skills to YPAN staff. (p) YPAN staff , apart from YPANís Director of Administration, do not appear to have received training in relevant TNC procedures that affected planning, financing, and implementation. This led to confusion over the role and format of work plans, funding expectations and timing. Over time, this led to mistrust and resentment over what appeared to be an opaque process (due to lack of knowledge of actual procedures.) Conclusions (a) Lack of clear lines of authority and supervision, either within TNC headquarters or between JCDT and TNC, was a contributing factor to the failure of either party to reach objectives stated in the grant. (b) The consolidation of information and decision-making amongst two key individuals during the first two years of the project impaired the ability of TNC and JCDT to complete activities as originally intended. This was further exacerbated upon their departure (c) The Indonesia and Jamaica projects were not integrated. One of the chief reasons for this was the lack of designation of an individual, or team of persons, that would be held accountable for assuring integration of the overall program. At the same time, one individual must be assigned responsibility for success of his or her respective country component. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 31 (d) Ambiguous management authority, TNCís changing vision for its role in Indonesia, and the lack of clear communication by TNC to YPAN about how TNCís vision was changing and how those changes would affect YPANís role greatly impeded the success of the project. (e) The fact that responsibility for implementation activities at the field sites alternated between TNC and YPAN confused community members and slowed implementation. Recommendations (a) It may not be possible for all staff to be involved in the project design phase. However, clear lines of roles and responsibilities should be delineated from initial implementation and further refined during the course of the project. TNC has recognized this and at the end of 1999 implemented the ìNew Grants Standard Operating Procedures for LACDî which included a detailed listing of accountability and responsibilities. (b) There should be an individual or team accountable for managing the project and should there be a change in management, adequate means should be taken for insuring that the transfer is timely and smooth. (c) While it is important to delegate management responsibility for each country project to staff within the project (for multi-country projects), it is also important to have a mechanism to promote coordination, learning, and synergies. The respective teams can determine whether it is better to have a single person coordinating from the WO, or one of the country project managers, or the team of project managers. But, there also needs to be a single contact person for the activity. (d) TNC and its partners should, at the outset of their partnership, detail their respective visions, strengths, roles, and responsibilities, the mix of institutional strengthening and implementation, and elaborate how they will work together. A formal Partnership Review should be conducted periodically, normally on an annual basis. Any necessary revisions to partnership agreements would be adopted at that time. The Review would be most effective with the assistance of an outside facilitator. 7.6 Sustainability 7.6.1 Overall sustainability survey Since YPAN was remerged into TNC, the sustainability survey does not apply to that project. Annex F presents the results of the sustainability analysis for the benefits flowing to BJCMNP. Findings: (a) JCDT continues to have strong support from government, though its credibility is somewhat tarnished by perceived implementation shortcomings and the battle with TNC. (b) JCDT has gradually solidified its relations with communities, but still has plenty of room for improvement. (c) Although TNC reports that it plans to keep working in BJCMNP, it is definitely in at least a ìcooling offî period with JCDT and appears to be placing far greater focus on other areas in Jamaica. (d) JCDT still retains institutional capacity, particularly at the field, financial, and administrative levels. However, it will need to replace the Interim Director with a permanent Director and the organization H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 32 has been suffering morale lapses as a result of the lay-off of staff related to the termination of the TNC grants. (e) With the end of the TNC grants, donor financial flows will be reduced in the short- to near-term. However, it is possible that the forced separation will force JCDT to become more effective at fund raising. Certainly the Interim Director is giving it his best shot. Conclusions: (a) The likelihood of sustainability of JCDTís efforts in BJCMNP does not look optimistic as of the end of the MG. The main patron of the park, TNC, is moving its focus elsewhere, its staff is reduced and somewhat demoralized, and it is under intense pressure to raise funds to survive. (b) It does not appear that the MG has yielded a significant positive net benefit to the sustainability of JCDTís efforts in the park. 7.6.2 Shift in role of JDCT Findings (a) The failure of TNC to raise its promised match, coupled with TNCís perception of non-performance by JCDT led TNC, in September 2000, to terminate funding for JCDT under the MG. Payment of salaries to JCDT staff was permitted through December 2000. (b) JCDT has received support from the Environmental Foundation of Jamaica (EFJ) to continue business training for individuals located in the buffer zone. (c) JCDT will continue to manage the BJCMNP through income generated from the trust fund, park fees and grants. Finalization of roles and responsibilities of parties entered into the co-management agreement are underway. (d) Despite prior warnings by TNC, JCDT staff feel the grant was abruptly ended and was detrimental to the COOs relationship with community members in the LAC regions. (e) JCDT will continue to fund at least one COO to work with LACs and people living in the buffer zones. Conclusions (a) Removing support has forced JCDT to reexamine its relationship with TNC and forge a new path for continued commitment to conservation issues in the BJCMNP. (b) The termination of the grant did not alter JCDTís commitment to working with the communities within the buffer zone and in some ways strengthened its resolve to find alternative funding sources. (c) However, the experience does seem to have reduced ñ at least in the short term ñ the degree of involvement of TNC in the vital BJCMNP as TNC shifts its relative emphasis from BJCMNP towards other areas of Jamaica. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 33 7.6.3. Potential for financial sustainability of JCDT Findings (a) The roughly simultaneous loss of TNC funding to JCDT through the MG and PiP programs has shocked the organization, resulting in staff lay-off, reduction in activities and forcing a serious re￾think of its financial foundations. (b) TNC met with JCDT and its board members on the issue of financial sustainability and alternative revenue sources. However, a self-sufficiency plan was never produced through the MG. (It was funded with non-MG funds and presented in draft form to JCDT after the grant ended.) (c) In 2000, grant income represented 75% of total income to JCDT. This percentage has been fairly consistent over the previous five years. The self-sufficiency plan by Project Development Services, finalized in July 2001, estimates grants over the next five years will remain extremely important to the financial health of JCDT, representing 64-72% of total income. (d) JCDT will continue to manage the Jamaica National Park Trust Fund, an important, though diminishing, source of income needed to manage the BJCMNP. JCDT hopes to get increasing revenue from its annual Green Expo event as well. (e) JCDT recognizes the need to diversify funding sources in the long-term and is looking at alternative revenue generating activities, including large-scale ecotourism efforts. Conclusions (a) Although the exodus of TNC funding was traumatic to JCDT, they are very proactively examining how to cut services and raise funds to maintain their programs. (b) Grants will remain an important source of income for JCDT in the near future (c) The ecotourism and other economic opportunities envisioned by the matching grant appear to remain as viable options for JCDT, although the failure by the project to complete a study on this potential means that this remains conjecture. Recommendations (a) Since grants are important not only to the financial health of JCDT, but also to preserve the park and surrounding areas, JCDT should undergo training in project planning and design, grant writing and grant management. 7.7 Financial Management 7.7.1. Effectiveness of financial management Findings (a) There was no grant manager at TNC for the overall project (that is, integrating Indonesia and Jamaica). (b) There was a gap of eight months from when the grant was made to TNC until funds arrived at JCDT. Similarly the first formal planning meetings were delayed for two quarters while TNC delayed initiating implementation while awaiting USAID funding to arrive. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 34 (c) Due to staff turnover at TNC, there was a gap of approximately one year where there was no one responsible for fundraising for the Jamaica matching grant. (d) JCDT claims to not have understand until the end of year 2 that the sub-grant award and budget was for one year and any funds left unspent would not be carried over until the following year. (e) Even though JCDT staff received ample training in financial record keeping from TNC, they were not, at least in the beginning, able to keep records separate for PVO and PiP projects. (f) TNC did not meet its promised match in Jamaica. As a result many activities were either not attempted at all or were only completed in a truncated fashion. (g) Part way through the Indonesia program, management realized that the organization had not focused on obtaining the match required for the MG. TNC reports that it managed to obtain general TNC unrestricted funds of over $240,000 to meet the shortfall, but this is not the way that the match had been intended. (h) Despite its problems raising a match for the project, the Indonesia program managed to fulfill its matching obligations to the MG through the use of unrestricted funds and effectively delivered funds to support field implementation. This resulted in a funding ìholeî for the program ñ from TNCís internal financial management perspective ñ of over $240,000. (i) With the re-merging of YPAN into TNC, activities in Komodo were focused to support TNCís field implementation of the Komodo project. This involved a different suite of activities than those described in the DIP. The Grants Manager, based in the US, disallowed close to $80,000 of the field expenses charged to the project as he considered them to be unallowable under USG regulations. (j) As a result of the back-and-forth between USAID and TNC at the proposal stage, the project was reduced to working in two countries, instead of five. Both countries had difficulty spending the amounts allocated to them and had difficulty raising the required match. Many TNC staff reported that the grant amounts were ìtoo largeî. The project had sufficient excess funding to permit a no-cost extension. Conclusions a) Inadequate financial management ñ chiefly by TNC and JCDT, but to a lesser extent by USAID ñ made it difficult for JCDT to effectively complete its work on this project and contributed to a range of cascading implementation failures in Jamaica. b) TNC should be credited with living up to its match obligation in Indonesia, despite the financial strain it placed on the country program (as a result of ineffective fund raising.) Unfortunately, however, TNC did not live up to its match obligations in Jamaica, seriously compromising the project. c) Although it is always difficult to turn down potential grants, TNC should have considered whether the amount of the grant ñ once target countries were reduced to two ñ might have been too large, at least in the case of Indonesia. Often institutional strengthening grants do not require large sums of money. 7.7.2. Leveraging other donor funds Findings H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 35 a) There is no evidence that JCDT or TNC used this grant to leverage other donor funding in the BJCMNP project area. b) The very modest investments in the Komodo project in calendar 2001 arrived at a time when the project was strapped for cash. Funds provided under the MG appear to have been extremely useful in helping to leverage millions of dollars in GEF (Global Environmental Facility) funding.11 7.7.3 Cost effectiveness of technical approach Findings (a) In general, we noted little in our investigation of the MG that would lead us to question the cost￾effectiveness of the technical approach. (b) One area that stood out, however, were the salaries paid under the MG to key staff of YPAN in Indonesia: the YPAN Executive Director was paid $6,900/month and Development Director $3,105/month (both figures include benefits.) The Executive Director had no prior experience in conservation and the Director of Development was not an accomplished fund raiser. (c) As indicated in Section 7.7.1, above, JCDT was frequently starved for cash which greatly inhibited their ability to implement the agreed upon work plan. Conclusions (a) The salaries of YPAN Executive Director and Development Director would appear to be far more than what might be expected for remuneration for leaders of indigenous Indonesian NGOs ñ especially given the individualsí modest prior experience in conservation and development, respectively. While TNC was certainly interested in attracting top talent to YPAN, and may have wanted to place the leadership on a par with TNCís Country Program Director, this may be seen as excessive to many observers. It also presented enormous sustainability challenges to a new local NGO. (b) Overall, however, our review of technical aspects of the projects did not reveal significant problems in cost-effectiveness. 7.7.4 Repercussions of ìmatchingî requirement on program Findings a) Shortfall in ìmatchingî funds was a factor in the failure of the Jamaica project. While implementation was not affected by the inability to raise specific matching grant funds in Indonesia, TNC was forced to use unrestricted funds to meet a large matching shortfall (over $240,000) in order to ensure effective completion of the project. b) Turnover in the TNC/HQ Jamaica project manager position hampered LACDís ability to meet the ìmatchingî requirement of the grant. Likewise, the extreme level of turnover in the Country Program Manager position in Indonesia (five different CPDs/Acting CPDs during the MG) limited potential ownership of the program, and, hence, commitment to fund raising. 11 But, it must be recognized that most of the foundation for the GEF proposal came from other sources, such as the Packard Foundation (almost $1 million), the Japanese Government, and other USAID funding. MG expenditure during the year was originally $120,000, and then reduced by approximately 60% as a result of certain costs being reallocated by TNC (See Section 7.7.1.) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 36 c) Staff members at TNC felt that JCDT should have done more to assist TNC in raising funds for the project. The economic collapse of Indonesia during the project period made local fund raising virtually impossible for YPAN in Indonesia. d) Fundraising for the Jamaica and Indonesia projects was divided between the geographic groups and was largely left up to the Country Programs. There does not appear to have been a centralized oversight of this function with respect to the MG. e) JCDT claims it was unaware of the lack of ìmatching fundingî and was surprised to learn that it would have to cancel further training for businesses and additional staff and LAC training as a result of the shortfall. f) Early in the project, most of the expenditure of USG MG funds in Indonesia was targeted to Lore Lindu, while the match was expected to come from the relatively richer (in non-USG funds) Komodo project. This resulted in some resentment from the Komodo team and internal pressure to allocate funds to TNCís Komodo program to help implement the project there in 2001. g) Although there was a diversity of opinion, most TNC staff in Jamaica and Indonesia felt a 1:1 match was too high and would recommend that TNC not pursue similar matches under the MG program in the future. Conclusions (a) More effective partnership communication could have led to wider knowledge of the shortfall in matching funds. Greater fiscal awareness could have tempered JCDTís planning and implementation of activities, the cancellation of which caused disappointment at JCDT and among other planned beneficiaries of scheduled activities, in addition to leaving key implementation gaps. (b) The ìmatchingî requirement of the grant did not necessarily increase ownership of the Jamaica project on-site, within TNC/LACD or of the Indonesia project on-site or within APR. In fact, the general lack of ownership of the MG appears to have reduced commitment to raising specific matching funds. (c) It is unclear why TNC chose to live up to its matching obligations to the USG in one country program (Indonesia) by accessing unrestricted funds to meet the shortfall, but not use such funds in another (Jamaica), resulting in significant impact short falls there. Recommendations a) Mechanisms such as cash flow analysis incorporating matching requirements might help warn of shortfalls in a timely manner. b) Once TNC has made an obligation to raise matching funds, it should be an obligation ñ and responsibility ñ of TNC overall and not fully left up to the country or regional programs. TNC may want to pursue a more centralized approach to raising funds for mechanisms such as the MG program. c) USAID should consider the rationale behind such a high level of match as it would appear, at least from the TNC case, that it can result in considerable implementation distortions and does not necessarily increase commitment. USAID may wish to consider reducing match levels in future for some MG activities. 7.8 PVOís Information Management H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 37 Findings (a) Although there were formal information exchange structures, such as weekly meetings at JCDT, staff felt they were left out of strategic decisions regarding the PVO project. (b) There was an almost complete break down of communications between YPAN and TNC in the field where respective project managers literally would not speak with each other for significant periods of time. (c) Annual reports were the chief means of communication from TNC to USAID. However, the reports often did not include important information for USAID oversight purposes. For example, the October 1999 ñ September 2000 report (dated 13 November 2000) does not mention the imminent remerging of YPAN to TNC, despite the fact that the YPAN board agreed to this in September and it became official in November. Likewise, the Final Report makes no mention of the fact that there were demonstrations against KNP management in the street in Labuan Bajo, and a barricading of the Komodo field office protesting use of excessive force by enforcement staff. (d) In both Indonesia and Jamaica, case studies, a deliverable under this grant, were not completed nor were dissemination efforts within TNC systemized. Conclusions a) Information arrived and was processed in a piecemeal fashion by both TNC and JCDT, which hindered effective management of the MG project. b) It would seem that there may be an excessive concern with presenting positive news within TNC. This is a natural tendency. However, if allowed to pervade the system, it is difficult for accurate information to reach the relevant decision makers and can compromise donor/TNC trust. Such a culture ñ somewhat natural in an organization constantly required to raise funds to achieve its mission ñ can be a major obstacle to becoming a learning organization. Objective implementation and impact indicators need to be used by field and project managers. 7.9 Logistics No significant difficulties were experienced in logistics in the MG that negatively affected implementation. 7.10 Project Supervision See Section 7.5. 7.11 USAID Management Findings (a) USAID staff visited the Jamaica project, but not the Indonesia site. JCDT staff report that they benefited from the USAID visit. (b) TNC reports receiving very little direct feedback on the annual reports submitted to USAID. (c) With the exception of an intensive period of correspondence while TNC was seeking a no-cost extension, there was little direct USAID/PVC interaction with the Indonesian program. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 38 (d) The USAID officer took advantage of the period when the MG extension was under discussion to have very open discussion in September 2000 with managers of the Jamaica and Indonesian programs. All participants point to the value of the frank discussion as cathartic, informative and a productive team building event. Conclusion (a) Although determining the right balance of oversight, it appears the project would have benefited from greater USAID interaction. Both major points of interpersonal contact had a significant impact. (b) All sides report that face-to-face meetings appear to have greater impact than annual reporting documents ñ although those documents provide an important basis for discussion. Such meetings appear to cut through many of the reporting issues discussed elsewhere. Recommendation USAID should consider greater use of frank face-to-face meetings between PVO project managers and their USAID CTOs (USAID cognizant technical officers.) These could be held twice per year and focus on implementation challenges, reconsidering fundamental design issues, agreeing and reviewing results indicators, and lessons learned. 8.0 LESSONS LEARNED The lessons learned ñ targeted to the general MG audience ñ are reproduced here from the body of the report (lessons learned specific to TNC are presented in Section 1.2: Managing Partnerships (a) USPVOs, and their partners, should, at the outset of their partnership, detail their respective visions, strengths, roles, and responsibilities, the mix of institutional strengthening and implementation, and elaborate how they will work together. (b) USPVOs should recognize that the actual Partnership needs to be managed, almost as a separate entity. Successful partnerships require attention the same way as do important personal relationships. It is not enough to assess each institution separately. It is necessary to examine the actual bonds, incentives, tensions, and structural issues that help or hinder effective partnerships. (c) It is easy for a USPVO to assume that it is in harmony with its partner NGOs on vital issues ñ such as approaches to advocacy, values and approaches to implementation -- if there is not conscious effort to discuss them. What is more, the two organizations can, unknowingly, diverge during the course of implementation if there is not a conscious effort to reaffirm commonality, or discover diversity from time to time. (d) A formal Partnership Review should be conducted periodically, normally on an annual basis. Any necessary revisions to partnership agreements would be adopted at that time. The Review would be most effective with the assistance of an outside facilitator. (e) Exclusivity in working with one NGO partner for implementation can lead to dependency on the side of the local NGO and vulnerability by the USPVO. The problem of ìputting all oneís eggs in one basketî. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 39 (f) USPVOs should avoid starting NGOs from scratch, unless it is in a very graduated manner, that avoids requiring the NGO to produce daunting deliverables while establishing itself, and which provides room for the NGO to develop its own mission within manageable bounds. Managing Grants with Partners (g) USPVOs and their partners should collaboratively agree on the balance to be struck between targeting accomplishment of implementation objectives and focusing on institutional strengthening. (h) USPVOs and their partners should periodically assess how well that balance is serving the objectives of the partnership and fine tune the balance accordingly. (i) The USPVO and its partner should be held accountable for what is agreed upon ñ both in performance and in faithfulness to the spirit of the partnership. Managing USAID-USPVO Partnerships (j) A high percentage of match does not necessarily translate into a high degree of ownership by a USPVO. USAID should consider the rationale behind such a high level of match as it would appear, at least from the TNC case, that it can result in considerable implementation distortions. USAID may wish to consider reducing match levels in future MG activities. (k) USAID should consider greater use of frank face-to-face meetings between PVO project managers and their USAID CTOs (USAID cognizant technical officers.) These could be held twice per year and focus on implementation challenges, reconsidering fundamental design issues, agreeing and reviewing results indicators and lessons learned. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 1 Annexes Annex A: Key Events Timeline* Jamaica Time Line Sandra McKenzie Hired (7/98) Enterprise Development Officer Hired Andy Drumm Hired - LAC Ecotourism Coordinator (7/98) C. Tufano Visit (Financial Management Training) Eco Empress Fund Started Fundraiser of $90 K plus $154 K pledged Training in Basic Computer Skills and Park Mgt Long-term Site Financial Plan Developed M. Libby visit (CAT) NGO Financial Mgt Training in DR Avi Tourism Consultant Brett Jenks Hired 8/98 DR Team visited Jamaica JCDT PVC Project Director visited Indonesia Jamaica Team visited DR; Indonesia Team visited Jamaica 6 Staff Members of Chairman of Board of JCDT attended Conservation Training Week JCDT/TNC Staff Meeting CAT Implemented CAT Implemented GIS Training for JCDT and Park field staff Annual Evaluation Meeting held in July 99 Hire 2 COOs and 1 assistant Three Workhshops hed to explore business opportunities Tour Guide Training; Leadership Development Training, Environmental Education Program developed Project Awarded DIP Submitted Contract Signed with JCDT PVO Project Begins with JCDT (Funding Received) 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q Oct-97 Oct-98 ACTIVITIES MANAGEMENT Year One Year Two MILESTONES H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 2 Jamaica aTime Line Terry Williams Hired 2nd Fundraising Trip Community Workshops; TNC Staff Development Training; Project Development Training Business Training; Community Workshop Conservation Summer Camp Tour Guide Training JCDT ? Line Management Training; Fundraising Training; Other Training LAC Executive Development Training; Seminar for CCED Project Co Management Agreement Signed Development Options Hired JCDT Self Sufficiency Plan Submitted (Not paid for by project) UTECH Contract Signed Project Ends Year 4 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q Oct-99 Oct-00 Oct-01 Year Three H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 3 Time Line in Graphic Form* Indonesia MG Project Management Training Manager hired to fill role TNC Country Program Director Klockner (Sr. Advisor) Klockner leaves; Apr￾June gap Regional Support filled gap temporarily Ridder new CPD (Jun) TNC efforts to Resolve Issues Hitz-Sanchez/ Darsie Org. Assess. YPAN (May) YPAN YPAN formed as local TNC fund￾raising arm (May 1995) GOI informs TNC that YPAN must have separate identity TNC staff person transferred as YPAN Director of Development (paid under MG; Oct) YPAN/TNC overall MOU signed (Apr); Ex. Dir. YPAN hired (Jun) Komodo YPAN hires new Komodo Director, most TNC staff transfer to YPAN TNC/CMP serves only advisory role in implmementation Lore Lindu Palu office split, with most staff becoming YPAN Begin difficult task of TNC/YPAN work plan (Sep) Political & Economic Indonesian Financial Crisis begins, ongoing challenge Change of Government and serious political unrest (May) By end of 1998 political unrest subsides -- for rest of project period Political unrest in Central Sulawesi (Apr) MILE￾STONES MG begins (1- Oct) MG funds arrive at TNC (Apr) (Pre-Project) 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q Oct-Dec 97 Jan-Mar 98 Apr-Jun 98 Jul-Sep 98 Oct-Dec 98 Jan-Mar 99 Apr-Jun 99 Jul-Sep 99 Year One Year Two ACTIVITIES Institutional H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 4 Annex A MG Project Management MG Project Manager hired (Dec) TNC briefs USAID/PVC and requests extension (Nov) TNC Country Program Director Ridder leaves (June) Pieter interim CPD (in addition to MG mgr.) in Jul Dutton as CPD (Apr) TNC efforts to Resolve Issues Communic. Strat. (Oct); Strat. Plan. wksp (Nov); Fin. Sust. Wkshp (Nov) YPAN Mar-Sep Capacity Building Plan (Feb; Aug) Mc Leod Report on YPAN/TNC Partnership (May) APRO head comes to Indonesia to resolve impasse (Jun) YPAN Ex Dir YPAN resigns (Jul) YPAN Board decides to re￾merge YPAN and TNC (Nov) All YPAN staff absorbed by TNC (Jan) Komodo TNC/CMP gains oversight authority over YPAN staff (Oct) YPAN implementation slows as TNC expresses disatisfaction (Feb) YPAN staff leave Komodo field office; some transfer to LLNP; CD work ceases (Jun) Anti-TNC/Anti￾Park Demonstration (Aug) TNC hires Comm. Dev. Officer, CD work resumes Rapid implementation of KNP work plan w/out YPAN (Jan) Lore Lindu LLNP complains:YPAN/ TNC work plan still not finished; work not begun (Apr) YPAN/TNC LLNP agreement signed (May); Demonstration against Park on illegal logging (May) Work finally gets under way in LLNP (Jul); Former YPAN KNP staff arrive (Aug) All YPAN field staff transferred to TNC (Jan) Political & Economic Political unrest in Central Sulawesi (May) MILE￾STONES One-year no￾cost extension begins Project ends 29-Sep-01 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q Oct-Dec 99 Jan-Mar-00 Apr-Jun 00 Jul-Sep-00 Oct-Dec-00 Jan-Mar '01 Apr-Jun '01 Jul-Sep '01 Indonesia Time Line ACTIVITIES Institutional Year Three Year Four (no-cost extension) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 1 Annex B Detailed Implementation Plan DIP = Detailed Implementation Plan DIP/TL = Detailed Implementation Plan Time Line AR = Annual Report TNC Jamaica The overall goal of the Matching Grant Program is to deepen the Conservancyís ability to strengthen local conservation organizations for long￾term management of targeted parks and reserves through the development of entrepreneurial income-generating strategies and the evolution of new skills to energize community-based organizations around protected areas. The goal of the program in Jamaica is to advance sustainable conservation and to promote rural socio-economic growth through local capacity building and enterprise development. Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Objective 1: Build TNCís capacity to support partner organizations in enterprise-based approaches to conservation, through a combination of staff development, institutional partnerships, documentation and sharing of lessons learned Hire Ecotourism Coordinator Andy Drumm was hired as the Ecotourism Coordinator Yes Yes Hire Caribbean Training Director Paul Hardy hired as Caribbean Divisionís Institutional Development Manager. Paul Hardy was not hired as the Caribbean Training Director; he was hired as an Institutional Development Manager. Paul Hardy had no direct role in the PVO project. He worked with JCDT on the Parks in Peril project. No Exchange Visits Exchange visit with program in Dominican Republic; Exchange visit with YPAN in Indonesia Yes [Repeated below, so not counted here.] N/A Seminars Only one ìbrown bagî lunch seminar held Lack of attention to deliverable. [One brown bag was not considered sufficient by evaluators or TNC.] No Annual Retreat Annual Retreat for the Jamaica Team; Conservation Training Week Yes Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 2 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Coordination of existing functions & working groups (EcoEmpreseas, Ecotourism Working Group, CCED, etc.) Occasional informal meetings of working groups No Objective 2: Build TNCís capacity to support partner organizations in developing site ñbased constituencies and coalitions through a combination of staff development, institutional partnerships, documentation and sharing of lessons learned Direct TA Community Assessment Tool was developed and tested in Jamaica and a variant is used widely at TNC Yes Yes Case Studies Yes [None completed] No Exchange Visits Exchange visit with program in Dominican Republic; Exchange visit with YPAN in Indonesia Yes Yes Seminars [already discussed above] N/A Annual Retreat Annual Retreat for Jamaica Team; Conservation Training Week Yes [already discussed above] N/A Training and capacity building activities by partner organizations (workshops, TA, etc.) JAMAICA Objective 4: Strengthened environmental advocacy and local economic development within buffering communities, contributing to long-term management of the Blue and John Crow Mountains National Park; Strengthened capacity of the Jamaica Conservation and Development Trust (JCDT) to support this. Consolidation Indicators: 14) Broad based Management committee/ technical advisory committee 15) Community Involvement in sustainable resources use 16) Income generated from site conservation 18) Environmental Education programs contributing to local support and conservation of reserve In Year 3 AR changes for indicators from baseline to March 2000 were as follows: 14) Increase from 2 to 3 15) No change at 3 18) Change from 2 to 4 7) No change at 3 10) Change from 2 to 3 Indicator 16 (Income generated from site conservation) was dropped from Year 2 AR. See Objective 5 As noted in Year 2 AR, Consolidated Indicator 7 (Site zoning and buffer zone management) and Consolidated Indicator 10 (Monitoring plan development and implementation) from the PiP scorecard were added. N/A H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 3 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Hire Key Additional Staff  JCDT Hires Enterprise Development Officer (DIP/TL) Never happened. No  JCDT Hires 2 Community Development Officers (CDOs) (DIP/TL) Lynette Wilkes and Dawn White were hired with project funds. Yes Use of Local Consultants Local consultants were hired for the design and/or facilitation of a number of trainings and workshops including business development, professional development, tour guide and LAC development training. Yes Training and TA in community mobilization, coalition building, business planning and management Three exchanges in FY 2 ñ Belize, DR, and YPAN Indonesia. Implementation of the Community Assessment Tool (CAT) in FY 2. Environmental Education ñ working with schools in the buffer zone Three workshops held to assist community members better utilize their resources and to unearth and explore business ideas. (Rpt FY 2) Implemented environmental awareness campaign through LAC meetings. Schools Environment Program was implemented through a collaborative effort between JCDT, JET and an Environmental Camp (paid for by PVO). Yes  Training of Trainers (CDOs and Park Staff) Individuals Trained Three staff development workshops were held Yes Staff training (only) under this project was incorporated in the i i í ll i i l No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 4 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? (DIP/TL) Park staff trained in project development, team building, time management and business skills development organizationís overall training plan, which was finalized in Year 3. Three workshops were held, however, conflict management was not covered. That training was scheduled to take place in June 2000, but postponed to September at the trainerís request, the placed on hold due to funding reduction for the project. [TOT is not the same as staff trainin g.]  Community Mobilization Workshops and Training (DIP/TL) Individuals Trained LAC members were trained in a series of Organizational Development Workshops. Topics included leadership, decision making, problem solving, record keeping, meeting facilitation and communication. These trainings improved their skills to better work members of their respective communities and provided an opportunity for LAC members to share experiences and network. Yes Yes  Assist LACs/CBOs in site-based capacity building (Year 3 AR) Training Plans and Materials Training plans, curricula and material developed in each workshop Three ëlive-iní workshops were held and a five session series of workshops were held in the community An average of 15 persons attended the workshops. The park-wide community environmental education program developed in year 2 was implemented. JCDT conducted an environmental summer camp targeting teenagers 12-18. The COOS facilitated involvement of Park schools in the Schools for Environment Program Yes Community Conservation Education Program was implemented by COOs during LAC meetings. Environmental topics were explored and threats identified, including deforestation and waste management. Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 5 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Follow-up assessment to measure results of training and environmental education was built in as part of the Case Study. Case Study was never completed.  Create Co-Management Company (CMC) to serve as the vehicle for stakeholder participation in park planning and management (DIP/TL) Instead of seminars, meetings held between JCDT, Forestry Department and Natural Resources Conservation Authority (NRCA) A co-management agreement was signed between JCDT, the Forestry Department and NRCA Yes Activities did not evolve as anticipated as discussions with the NCRA and the Forestry Department led to a decision to establish a co-management arrangement between the three entities before involving other stakeholders. The process was very time consuming, requiring pushing back the stakeholder seminars. While the seminars are still necessary, the process was placed on hold due to funding reductions. Yes, but not as envisio n-ed by propos al or Origin al DIP  JCDTís avi-tourism consultant works with LACs/CBOs to develop business opportunities in park planning and management (DIP/TL) Brett Jenks from RARE Centre for Tropical Studies conducted an initial assessment of avitourism (Year 1 AR). Removed from Year 3 AR No  Assist communities in identification and achievement of infrastructural development needs (DIP/TL) Infrastructure needs were identified. Near Hagley Gap, road improvement work was carried out on several sections with community members contributing labor and cash. Several linkages were created included UNICEF, the Social Development Commission (SDC), Environment Action Programme (ENACT), Forestry Department, the British High Commission and the Japanese Government Yes Yes  Train LACs/CBOs in business skills (feasibility No. of Feasibility studies A total of 58 persons were trained in business skills under the theme ìStarting a Data were not Business training sessions were prolonged well beyond the No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 6 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? studies, business planning, access to capital) based on outcome of needs assessment (DIP/TL) No. of individuals trained No. of Business plans No. of enterprises functioning Establishment of a community fund Viable Small/Micro Enterpriseî 32 business plans were developed. No enterprise has been established to date. correct. 22 people complete d training, 13 business ideas were presente d as a result of the UTECH training. Of these, 7 were chosen for further support and training by DO. 40% did not attend DO training. DO asked to terminat e the contract. timeline, causing reporting to be pushed back (See Case Study accomplishment under Assist LBOs/CBOS in site-based community capacity building). Delays mainly from scheduling problems and inclement weather leading to postponed sessions. The remaining activity is the establishment of the enterprises. This was pending identification of the funding sources needed. However an arrangement has been made between TNC and Development Options, a local entity that manages a micro-fund to work with JCDT and the trainees to establish a maximum of six businesses by the end of December. Indvls. were trained instead of LAC/ CBOs. The results reporte d were greater than verifie d.  Assist with feasibility studies and business plans No. of feasibility studies Not completed Paige McLeod visited JCDT in March 1999 and her trip report No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 7 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? for ecotourism, craft and other enterprises by CBOs and local groups, explore capital sources and market outlets (DIP/TL) notes the shift from community based enterprises to individual entrepreneur training. TNC/W would review business feasibility. Initial assess ments were comple ted by staff at TNC but no studies or plans were produc ed.  Prepare case studies of community development activities (DIP/TL) No. of case studies Not completed Yes This activity was delayed due to delays in training implementation, then placed on hold pending receipt of project funds. No  Scholarships for NGO participation in TNC Conservation Training Week and other training (DIP/TL) No. of scholarships In 1999, 6 staff members and Chairman of Board attended Conservation Training Week in Miami. Not sure of enrollment for 2000. Yes Yes Objective 5: Building JCDTís financial capacity and sustainability of resource flows for park management operations Consolidation Indicators: 11) NGO self-sufficiency plan 12) PiP site long-term financial plan 13) Diversified funding sources for the site In Year 3 changes for indicators from baseline to March 2000 were as follows: 11) No change at 3 12) No change at 2 It was reported in Year 2 AR Indicator 13 (Diversified funding sources for the site) was dropped because Indicator 12 (PiP site long-term financial plan) incorporated Indicators 13 and 16 (Income generated from site conservation). No Establishment of a Co￾Management Company See Objective 4 [described above] N/A Establishment of a concession structure H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 8 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met?  Design and implement fee structure for park uses; Clarify roles of relevant entities (JCDT/Co￾Management Company) (DIP/TL) Establishment of concession structure Pilot fee implementation program continued, but not finalized. Fees are collected from all visitors to the Holywell and Portland Gap recreational sites in the BJCMNP Roles of relevant entities, which formed the Co￾Management Company were not defined by the end of the project. Yes This activity was attributed to PIP funds by the Project Coordinator, Sandra Mackenzie. A pilot program was instituted at Holywell and Portland Gap Yes. A concess ion structur e was piloted but it is not clear that it was a result of PVO funding .  Conduct feasibility studies on potential ecotourism opportunities (note overlap with Objective 1) (DIP/TL) No. of feasibility studies Eco-tourism diagnostic was conducted at Holywell There is no evidence that the anything more than a trip report was completed. No  Develop business plans and marketing plans for feasible activities (DIP/TL) Not funded with PVO money, a Self-Sufficiency Plan for JCDT was completed in July 2001. JCDT business plan is currently being prepared by JCDT with its own funding. The plan incorporates financial planning for both the JCDT and the BJCMNP The original plan was to develop an eco-tourism management plan, a business plan for the Park, along with a financial plan funded in part by the PVC project. However, the change in funding led to a series of decisions that resulted in only a self sufficiency plan (and that funded outside MG funding). A business plan incorporating income generating activities for the park is being prepared. It was reported in Year 2 AR a Long-Term Site Financial Plan and Sustainability Plan had been developed. No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 9 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Training and launching of major fundraising campaign  Hire Enterprise Development Officer (DIP/TL) [already counted above] N/A  Develop JCDTís ability to undertake major fundraising campaign for BJCMNP, including case statement, background materials and training (DIP/TL) Posters and flyers were developed Visits were made to the IDB in Washington and the Jamaican Embassy in Washington. Funding was secured from the Environmental Foundation of Jamaica Training of at least 3 staff members in donor research and proposal writing was accomplished in part under the project development training. Yes Further training in proposal writing was scheduled for September 2000. Again reduction in funds led to training being cancelled. No The team was unable to verify comple ted trainin g in donor researc h & propos al writing . Objective 6: Building TNCís Capacity to Support Enterprise Approaches to Conservation To be assessed as part of the annual reporting process; measurement/ criteria to be developed at first annual retreat (See Objective 1)  Hire LAC Ecotourism Coordinator with Caribbean Focus (DIP/TL) Andy Drumm was hired as LAC Ecoutourism Coordinator [already counted above] N/A  Establish EcoEmpreseas Working Group held first meeting (Year 1 AR) There is no evidence that a team No. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 10 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? team (Patricia Leon) (DIP/TL) continued to meet.  Develop Caribbean Regionís ability to facilitate major donor trips (DIP/TL) First trip held with TNCís former President (John Sawhill) raised a $90K match; a pledge of $142K Second ìreconnaissanceî trip for Connecticut Chapter VIPs A trip planned for the second year of the project was cancelled by TNC. No  TA/Overnight Visits by TNC staff to Jamaica (DIP/TL) Visits by 11 home office staff, including Institutional Development Director and Protected Area Specialist New Jamaica Country Program hired with experience in community conservation Yes Yes  Hire Caribbean Training Director (DIP/TL) Paul Hardy hired as Caribbean Divisionís Institutional Development Manager. [already counted above] N/A  Identify TNC staff training needs and appropriate sources of training; set and meet annual staff development objectives (DIP/TL) Each staff involved in PVC-Match included in training objectives as part of annual objectives Yes  Integrate investigation/discussion of Matching Grant activities with LACD ìCommunity Conservation Working Groupî formed under Parks in Peril ìBalancing Themesî component (DIP/TL) Meeting held on integration of the PVC grant and Parks in Peril Project Project progress captured in the PiP Consolidation Scorecard Yes  Create working group to share lessons Indonesia/Jamaica and organization wide (DIP/TL) No formal working group was formed, although the following miscellaneous activities were supportive: Brown bag lunch sessions held at TCN home office, including presentation by Ecotourism No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 11 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Specialist and JCDT Executive Director Exchange of project team members between Indonesia and Jamaica Presentation and meetings during Conservation Training Week Michele Libby is now Community Conservation Specialist for both the Caribbean and Asia/Pacific regions  Monitor and evaluate program regularly; document results in special studies as appropriate (DIP/TL) Monitoring systems were weak and no evaluation of program appears to have been completed. No special studies were completed. Formal documentation of annual, quarterly and monthly meetings. No  Develop ìcommon languageî for evaluation of enterprise-based approaches to conservation across regions (Indonesia/Jamaica) (DIP/TL) Not accomplished. More structured opportunities for sharing of lessons learned were not planned beyond the international exchanges and annual reports. No  Publicize program activities (DIP/TL) Not accomplished, although the following internal communication was completed.: Project updates/highlights in July/August TNC Magazine Sharing of project at Conservation Training Week events Project was to receive greater coverage which was delayed due to delayed project progress. In Year 2 AR it was reported that a video would be produced highlighting work in Jamaica. It was not done because of lack of funding. No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 12 TNC Indonesia The overall goal of the Matching Grant Program is to deepen the Conservancyís ability to strengthen local conservation organizations for long￾term management of targeted parks and reserves through the development of entrepreneurial income-generating strategies and the evolution of new skills to energize community-based organizations around protected areas. The goal of the program in Indonesia is to advance sustainable conservation and to promote rural socio-economic growth through local capacity building and enterprise development. Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? INDONESIA Objective 1: Develop YPAN as a strong national conservation organization, and develop its financial sustainability through generation of bi- and multi-lateral support, income-generating ventures, and private-sector funds in Indonesia. N/A 1.1 YPAN is a strong national conservation organization 1.1.1 Conduct needs assessment (staff development) Staff development done over time based on needs assessment Training needs assessment done. No systematic overall training plan in place. Only selected staff participated in training. Real benefit to support of partners not reassessed Needs assessment is merely based on individual preference of staff and not a structured plan based on a long term organizational strategy No 1.1.2 Assemble / develop training materials Training materials assembled. Relevant materials translated/adapted to Bahasa Indonesia Materials assembled for stakeholders analysis, project management, sustainable financing, awareness and education. Documentation and information sheets done on for habitat survey, bird survey, and mapping. Translation done on select materials. Publication for wider audience pending funding from other sources Materials for institutional development available from TNC but mostly need translation for Indonesia context. Technical materials (monitoring, survey techniques etc) are generated locally by consultants and staff Yes 1.1.3 Train staff in coalition building, business planning and management, fundraising (Honolulu, Jakarta) Staff trained and capable to implement learned skills Coalition building conducted in Palu, fundraising, marketing and communications, and work planning and budget training conducted in Jakarta Exec. Dir,. Dir. of Development and Palu Program Manager attended training in the Training conducted locally to allow YPAN staff to use their own planning and budget data for practical outcomes. Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 13 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? US. Local fundraising initiated, but too short to show impact 1.1.4 Skills assessment post-training New or improved skills acquired through training Tools developed to assess benefit of training. Data collected to determine further needs for training for individuals. Assessment is not done rigorously. Some individuals do not complete post training assessment. No 1.1.5 Analyze evaluations and produce recommendations Training plan regularly updated to meet recurrent needs of organization Not applied to trainings attended by individuals. Was done only for training participated by several people, like CTWís Training report is not usually required; assessment was done irregularly, training programs not clear, Training Officer does not perform role. No 1.1.6 Conduct communication needs assessment (including marketing brand name) Communication priorities identified; communication plan in place Communication needs assessment was done by Exec. Dir.; plan developed to market the organization; communication strategy developed but not finalized Recruitment of communication specialist was slow, no structured efforts done to support Exec. Dir. on this initiative Yes 1.1.7 Develop national communication strategy Communication strategy in place with rooms for modification to fit organizational strategy Communication strategy developed. Hired YPAN communication specialist, training by TNC in marketing communications Yes 1.1.8 Implement national communication strategy Communication strategy implemented through multiple communication and marketing initiatives Communication strategy developed but not fully implemented (less than 20%). Organizational strategy was never finalized. Communication strategy can not be further developed in a vacuum. No 1.1.9 Conduct institutional needs assessment Baseline established for institutional development Assessment done with TNC generated ìInstitutional Self Assessmentî tool. Assessment done on overall YPAN, and for respective operational sites, i.e. Jakarta, Palu, Komodo. - Yes 1.1.10 Implement recommendations Required positions developed and filled; program priorities restructured as needed Dir. Of Development and Palu Program Manager participated in Environmental Leadership training conducted by Smithsonian Institute. Communications specialist hired, other candidates identified for GIS S pecialist, Conservation S pecialist, Recruitment in Palu was slow because Program Manager hesitant to recruit immediately given pending changes. Recruitment in Jakarta was not even initiated despite identified needs. No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 14 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Program Officer. 1.1.11 Develop personnel manual Personnel manual in place for implementation in all sites finish manual based on consultant's feedback YPAN HR specialist completed manual. Still to be translated to Bahasa Indonesia for all staff Moved to lower priorities given the bigger changes due to issues in partnership Yes 1.1.12 Establish organizational structure Organizational structure fit to implement strategic plan finish chart; Recommended organizational structure completed. Final chart pending finalization of strategic plan. Strategic plan was not finalized. Yes 1.1.13 Provide technical assistance and skills transfer in management, fundraising, communications and board development Needs for TA met T.A. provided in management, fundraising, communications and board development. Director of Admin and a finance staff attended training on USAID grant management in Manila. TA on Board Development was least utilized. - Yes 1.1.14 Annual retreat to analyze lessons learned, planning, update training needs Periodical review of plan and modification in place as needed Retreat conducted in June 1999 and August 2000 in combination with joint team meetings and team building training; review of lessons learned re: partnership followed by restructuring analysis conducted by McKinsey Co. Lessons learned on institutional development not explored well. Retreat ended up focused on issues re partnership; review of plan and update of training needs not done effectively Yes 1.2 YPAN is financially independent and self-sustaining 1.2.1 Hire strong Executive Director and fundraiser Executive Director in place to lead self sustaining efforts Executive Director was hired May 1999 but resigned end of July 2000 over differences in approaches on partnership. ED initiated marketing initiatives for YPAN brand name. Yes 1.2.2 MOU and Subcontract agreements signed with TNC Partnership responsibilities MOU completed and signed. Negotiate additional Process was protracted due to partnership problems. Second sub￾Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 15 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? defined Subcontracts as needed; New extended sub-agreement completed. Second sub￾agreement pending outcome of McKinsey findings; second sub-agreement was not finalized following decision to reunify agreement pending outcome of McKinsey findings. MOU was not fully implemented. Plan in place to review MOU and modify to reflect changing situation, but never completed. 1.2.3 Board Development Package Board trained Materials for Board Development identified and collated. Materials distributed to Board members but no formal sessions arranged to go through the materials nor collect feedback for further improvement Cultural hesitation to train Board members. ED of YPAN prefers to let the Board members obtain materials and read on their own pace and time. No 1.2.4 Board development workshops (orientation) Board trained Arrange exchange visits between Boards; individual board members visit sites, but no group visit was done. Board assumed leadership with partnership issues and management; Site visit and Board workshops planned for year 4 but never realized. Board members can not agree on a schedule for a group visit to sites. No 1.2.5 Develop strategic plan Strategic plan in place and implemented with rooms for at least annual review First draft of Strategic Plan completed with assistance of professional facilitator. Finalization pending new Exec. Director (eventually never finalized) Protracted process; no ownership; no clear vision for organization; strategy does not match actual strengths to grow No 1.2.6 Develop annual work plan Annual Work plan accepted as an internal process Completed for NRM funded activities in Palu with assistance of PVC Project Manager. Komodo site work plan completed. No organizational wide work plan il N t Work plan was never required previously by TNC; lack of capacity to prepare viable work plan not acknowledged, hence no efforts in place to systematically train staff and senior managers to prepare work plans that feed into a strategy. No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 16 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? in place. No process to institute preparation of work plan, although workshops conducted in Palu by the PVC Project Manager to get site staff involved in preparation of work plan. 1.2.7 Hold workshop on fundraising Senior managers and select staff trained Completed by a senior TNC fundraiser. Only Jakarta staff trained. Dir. of Development participated in other training on the subject. TNC resources not fully utilized. No clear assignment who to get involved in workshop; Yes 1.2.8 Develop fundraising strategy Strategy in place with rooms for periodical review Completed annual private sector fundraising plan. A more comprehensive strategy pending organizational strategy plan. Reliance on TNC led fund raising; training provided to Dir. of Development not structured well to allow gradual application Yes 1.2.9 Implement fundraising strategy submit grants and solicit corporate donors based on strategic plans Grants submissions and corporate cultivation initiated; Plans on hold during internal reorganization and recruitment of a new Executive Director Yes 1.2.10 TNC provides technical assistance in grant-writing provide TA as needed to be provided pending request. Training scheduled for one in country workshop but never implemented No Objective 2. Strengthen site-based coalitions and community organizations as the conservation constituency for two national parks: Lore Lindu (primarily) and Komodo (secondarily) 2.1 Strengthen site-based coalitions and community organizations 2.1.1 Explore applicability of conservation coalition for Komodo Coalition established when appropriate Community consultations and conservation awareness marketing conducted. An informal coalition established in KNP including youth cadres, NGOs, park management, business and community leaders Loose coalition recognized as more effective than a formal one; stakeholders varies; involvement wider than just the National Park management but to include local government, police and the navy. Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 17 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? 2.1.2 Conduct site-based scoping exercises in-house workshop in-house workshop completed for Lore Lindu and Morowali - Yes 2.1.3 Prepare coalition-building strategic plan Coalition supported Developed training plan as first step toward strategic plan. Phase over leadership to outside TNC and YPAN. Group leadership not ready for strategic plan. Coalition members in Lore Lindu prefer an informal structure to maintain independence of individual members N/A 2.1.4 Implementation of plan Consistent implementation with rooms for periodical review Not done. Group leadership not ready for strategic plan N/A 2.1.5 Conduct community consultations/workshops Regular consultations with community in and around the Park On going; Community consultations completed. One workshop on conservation awareness completed. Community workshops conducted in five villages to prepare community maps as basis for conservation agreements Quarterly workshops were planned but could not be organized due to security concerns Yes 2.1.6 Stakeholder analysis for Komodo Stakeholders mapped Stakeholders analysis completed in November 2000; priority target groups identified; intervention methods identified Yes 2.1.7 Institutional relationships analysis for Komodo Stakeholders mapped Stakeholders analysis completed in November 2000; choose to maintain loose coalition as stakeholders and interest too fragmented; differences in approach of previous community development work need to be resolved Yes 2.1.8 Analyze legal basis for community participatory management Existing legal basis identified to provide rooms for community involvement in management Analysis completed for Lore Lindu and being used as bargaining/ monitoring tool; Structure of Collaborative management being explored for both Lore Lindu and Komodo Yes 2.1.9 Phase-over coalition leadership Coalition led by non Negotiate additional Subcontracts as Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 18 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? from TNC to YPAN TNC members needed; Phase-over from TNC completed. Phase over from YPAN to other members of coalition initiated and completed. Secretariat is established outside TNC/YPAN office 2.2 Increase community capacity to participate in park management 2.2.1 Assess institutional capacity of CBOs/ NGOs Stakeholders mapped for support and collaboration A database of site based CBOs and conservation NGOs developed, but no institutional capacity assessment done. Some potential partners identified have applied Institutional Development Framework to assess own capacities. TNC generated tool made available but no follow up to complete assessment (did not get to do it) No 2.2.2 Assemble / develop community training materials Training materials available for community use with rooms for periodical review and update Not done (did not get to do it) No 2.2.3 Conduct training workshops on participatory planning, community mobilization, coalition building, business planning and participatory management Community trained and can contribute better in park management Community workshops conducted in five villages to prepare community maps as basis for conservation agreements. Framework for collaborative management still being finalized Quarterly workshops were planned but could not be organized due to security concerns in Lore Lindu. Yes 2.2.4 Provide technical assistance to communities in participatory planning, community mobilization, coalition building, business planning and participatory management TA available as needed to support community involvement in park management TA provided through the coalition members. Direct involvement with communities creates polarized tensions with other group supporting illegal occupation of a Park area. Framework for collaborative management, which would enable coordinated TA, not in place yet. No 2.2.5 Establish partnership forum at Lore Lindu Forum established to support park management Forum established with 14 members. Six members directly involved in works in the park. Membership is open but Park Management stays on the side line. - Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 19 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? Members of forum refuse formalized relationship with the Park to maintain independence. 2.2.6 Forum integrates community into management Community input into Park management Not completed Framework for collaborative management not yet in place; role of community in management not clarified Yes Objective 3: Document lessons learned from enterprise approaches at the two sites. Develop strategies for long-term sustainability, and promote best practices developed at the two sites to a broader constituency in Indonesia and throughout the region. 3.1 Document and promote "best practices" for conservation enterprise throughout Indonesia and the region. 3.1.1 In collaboration with LACD, analyze role of compatible economic development in conservation projects and determine best practices in conservation enterprise CE guidelines and principles, reports, additional exchange visits Guidelines developed for LLNP. One Jamaica  Indonesia visit; and one Indonesia  Jamaica visit. Work rescheduled as responsibilities were shifted from YPAN to TNC. Guideline development delayed due to civil unrest. No 3.1.2 Produce article/case studies on sustainability strategies for community￾based conservation, best practices in enterprise-based conservation Case studies available for local dissemination Completed by consultant in English. But not disseminated. Translation to Bahasa Indonesia pending available funding Late implementation, do not have time for dissemination No 3.1.3 Produce article/case studies on involvement of community-based coalitions in conservation Case studies available for local dissemination Not done. Staff can not write case studies. Consultant brought in by NRM to write case studies/ stories do not have time to tutor staff to prepare own case studies. A writing workshop was scheduled but not carried out Staff can not write case studies. No 3.1.4 Translate case studies, articles, publications, etc. into Bahasa Indonesia Case studies available for local dissemination Not done Staff can not write case studies. No 3.1.5 Disseminate case studies, articles, publications on lessons learned through workshops, meetings, and exchange visits to government agencies, NGOs and CBOs Lessons learned disseminated One workshop scheduled in Palu but canceled over safety and security concerns; no other dissemination done. CE is still in experimental stage in Komodo. Approach in CE in Lore Lindu being adopted in TNC East Kalimantan new site. No 3.2 Develop long-term sustainability strategy for conservation enterprises H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 20 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? 3.2.1 TNC collaborates with WWF and CI in investigating Debt-for-Nature swaps to secure long-term funding for conservation DNS and other mechanism utilized for conservation financing On going; continue participation in DNS committee. YPAN drops out from the Committee, the Indonesian Foundation of Biodiversity (Kehati) joins in. Briefings provided to the parliament and several minister, information and TA provided for Ministry of Environment; European Union and USAID provide additional funding to establish a Secretariat but the DNS mechanism is still not applied. Strong oposition voiced by local NGOs for concerns over the fund being used to bail out bad debtors. Yes 3.2.2 TNC collaborates with the Indonesian Navy on the abatement of destructive fishing practices in Indonesia Threat abatement measures applied outside the Park Areas Continue collaboration. The Distreict Government of Manggarai issue an unprecedented District regulation to ban use of compressor and transfer the right to issue fishing permit in the area adjacent to the Park to Park Management. Issuance of regulation allows the Navy and joint patrol to expand surveillance and enforcement efforts. The collaboration still limited to the Komodo national park. Yes 3.2.3 Develop community and coalition building strategies and tools Tools and strategies in place for adoption and adaptation in other sites Two models developed so far, the formalized institution in Lore Lindu, and interest based loose coalition in Komodo. But no write up has been done on both approaches. No 3.3 Expand conservation enterprise activities. 3.3.1 Hire business consultants Conservation enterprises based on sound business practices As needed, a consultant hired to assess eight enterprises with impacts of significant scale. Consultant recommends support to coffee and Follow up works on cocoa delayed because Park Management does not support initiative over concerns of Yes H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 21 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? cocoa in Lore Lindu and support to organic farming of rice. encouraging encroachment. 3.3.2 Organize enterprise-based community cooperatives Community take leadership of development of conservation compatible enterprises None established in Lore Lindu; two fishers groups established in Komodo to work on pelagic fishery; two groups established to work on seaweed cultivation. Cooperatives exist in Lore Lindu but communities have bad experience with these organization. Yes 3.3.3 Conduct feasibility studies with community involvement Community can assess other opportunities A feasibility study for tree bark cloth was done in Lore Lindu; market link established. Community members involved in enterprises are too small to create economic and conservation impacts. Further work to expand enterprises delayed over safety and security concerns and later because the Park Management does not support initiative. - No 3.3.4 Prepare participatory business plans for most feasible enterprises Business plan available; community knows how prepare one Not done in Lore Lindu. Done by consultant/staff in Komodo Community have not prepared to participate in this exercise No 3.3.5 Establish a community enterprise fund Fund established; community can expand enterprises Not done in Lore Lindu; workshop initiated to establish fund for wider conservation works in Central Sulawesi; initial revolving fund created by the fishers in Komodo; plan to support fund to allow fisher group to expand business; plan to provide fund to start up fish grow up facilities Enterprises assessed not in scale to create economic and conservation impacts; effectiveness of fund for expansion unknown No 3.3.6 Build incentive frameworks to encourage local financial services to fund environmentally related enterprises Financing available for community to expand enterprises Not done No 3.3.7 Community cooperatives implement business plans Business plan implemented with rooms for review Not done in Lore Lindu; partially implemented by fishers groups in Komodo; previous plan to expand Community cooperatives is still in infant stage No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 22 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? and update market by bringing in buyers discontinued but fishers already recognized potential market and follow up on their own initiatives. 3.3.8 Monitor effects of enterprise on environment Impacts measured and enterprises approach and techniques modified accordingly Not done systematically. Broad assessments done in Lore Lindu revealed small positive conservation impacts of enterprises due to their small scales. Enterprises in Komodo show positive impact to protection of reef due to decrease pressures and destructive practices as fishers relocate effort to seaweed farming and pelagic fishery No plan in place for systematic monitoring of impacts. But, working monitoring broad ecological impacts in Komodo. Also, Komodo has not yet done EIAs but will do so for several enterprise projects as part of GEF work. No significant monitoring has been done in LLNP. No 3.3.9 Develop linkages with business community through CCE (Corporate Council for the Environment), other means, for consultations and potential joint ventures Wider support to conservation compatible enterprises Not done Not properly plan No Objective 4: Build TNCís capacity to support partner organizations in developing site ñbased constituencies and coalitions through a combination of staff development, institutional partnerships, documentation and sharing of lessons learned 4.1 TNC capable of providing technical assistance to YPAN, CBOs, and PKA on organizational management, project cycle management, and sustainable financing. 4.1.1 Hire training coordinator Training Coordinator in place and provide support to partner organizations Training Coordinator hired October 1997; position upgraded to PVO Project Manager in December 1999 to provide broader support to partner organizations. In July 2000 the PVO Manager is assigned with additional task to be the Interim Country Director. Recognition that capacity building is more than training, and also the need to better manage the project. Since being assigned as Interim Country Director, cannot carry out the role of Training Coordinator Yes 4.1.2 Assess current staff capacity for institutional strengthening Assigned qualified staff to work with partners Inventory of a range of skills of individuals plus indication of staff willingness to assist partners but no agreement to take on the additional responsibilities. Only two Lore Lindu- based staff built support to Staff perceive this as additional workloads and perception that this will dilute works on threat abatement. No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 23 Project Objectives/Activity Indicator/End of Project Target Accomplishment Data Verified Explanation for Variance Target Met? partners into their performance goals. 4.1.3 Develop/assemble training materials and resources Training materials assembled. Relevant materials translated/adapted to Bahasa Indonesia Materials assembled for stakeholders analysis, project management, sustainable financing, awareness and education. Documentation and information sheets done on for habitat survey, bird survey, and mapping. Translation done on select materials. Publication for wider audience pending funding from other sources Materials for institutional development available from TNC but mostly need translation for Indonesia context. Technical materials (monitoring, survey techniques etc) are generated locally by consultants and staff Yes 4.1.4 Train staff (exchange visits, seminars, workshops) Improved technical and organizational skills based on a need assessment Training needs assessment done. No systematic overall training plan in place. Only selected staff participated in training. Real benefit to support of partners not reassessed Needs assessment is based merely on individual preference of staff and not a structured plan to support partners No 4.1.5 Set up data base of training resources Database of trainers, institutions and relevant training resources completed with rooms for regular update Only a simple list of organizations with training expertise and resource persons with specific skills e.g. facilitation, conflict resolution, community organizing etc. A database structure was developed but database not completed No follow up done to collect feedback information to build the database. No 4.1.6 Establish/participate in Indonesia network of organizations supporting community enterprise approaches Network utilized to share lessons learned in community enterprise approach Contacts established with organizations applying similar approach (ecotourism, butterfly farming, medicinal herbs). Participate in government initiated working group to share appropriate technology(s) to support community based enterprise. No such network in place, none established. No H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 24 Annex C: Evaluation Scope of Work * Evaluation of ìBuilding Constituencies for Protected Areasî Matching Grant FAO-!-00-97-00061-00 between The Nature Conservancy and USAID/PVC REPORT FORMAT and SCOPE OF WORK This evaluation will be used to assess project performance as well as to feed into PVCís yearly results reporting process. In order to get more consistent information across all Matching Grants (MG) programs a standard evaluation template has been developed. Through input from The Nature Conservancy (TNC) and the USAID/PVC staff, this template was tailored to suit the particular needs of the TNC MG. This document presents the evaluation report format, and the evaluation questions that the team is expected to address. ELEMENTS IN THE REPORT I. EXECUTIVE SUMMARY 1-3 page summary of essential aspects of the evaluation. II. PROGRAM IDENTIFICATION Include the following: PVO name Cooperative agreement number Date of the evaluation Country programs evaluated III. EVALUATION METHODOLDOGY AND TEAM COMPOSITION Provide a brief description of the evaluation methodologies/instruments employed and document data sources (using annexes, as appropriate.) Describe how confidence levels in findings were derived. Where appropriate, the evaluators will utilize the Agencyís microenterprise indicators to assess the status of microenterprise interventions. Include a copy of the SOW as an annex. Name evaluation team members; describe their backgrounds and their roles in the evaluation. IV. PROGRAM BACKGROUND  Briefly provide basic information on the program that will be evaluated, including: - History of the program - Current implementation status - Local Partners  Summarize project goal and objectives, and include a copy of the Program Planning Matrix as an annex. V. PURPOSE OF THE EVALUATION Outline the information needs of the evaluation audience (PVC, the PVO and local partners), and how each partner will use this information. Some questions to guide the discussion follow:  who wants the evaluation information?  what do they want to know? H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 25  for what will the information be used? to inform TNC and the partner of the best practices for future community conservation work. TNC has many partners and there are similar cases , particularly internationally, where the local population lives in close proximity to the natural protected area. For example: The final (or mid-term) evaluation fulfills the requirements of the USAID/BHR/PVC Matching Grant (MG) Program. The MG program will use the information to: assess how well the MG is meeting its objectives; determine patterns and emerging issues across all MG funded programs; determine technical support needs for grantees to shape new RFAs and to review any follow-on proposals; and develop internal and external documents to demonstrate the effectiveness of the MG program and to share lessons learned with the entire PVO community. PVC will use information outlined in the SOW template in its annual Results Report and in USAID's annual report to Congress. VI. THE EVALUATION QUESTIONS: The evaluation team will assess the following program and institutional questions, providing evidence, criteria for judgment, and citing data sources. Technical/program opinions and observations are also valued elements of the evaluation --- but should be clearly stated as the evaluatorís estimate, opinion or forecast Any observations on relatively lower data quality or constrains to its interpretation should be clearly stated since data from these evaluations are used for USAID reporting purposes and are subject to audits. Findings will be separated from Conclusions/Recommendations in the text of the report. All conclusions and recommendations will be directly supported by findings. The evaluators will assess both headquarters and the country-level programs. While all questions listed in this document are considered worthy of investigation, they are not all of equal importance to USAID and TNC, as staff from those institutions indicated in a team planning meeting (TPM) for this evaluation, held 15 August 2001. Each of the evaluation areas described below is noted as being of ìhighî, ìmediumî, or ìlowî importance to USAID/TNC. Relative evaluation team effort, and depth of analysis, is meant to correspond to this prioritization. The evaluators should indicate any cases where any resultant constraints to data collection and analysis restrict the level of confidence. A. Program Implementation: [EMPHASIS FOR THIS EVALUATION --- High] 1. Assess progress towards each major objective [subtotal -- High] NOTE: TNC is contractually accountable for achieving what is specified in the DIP. Accordingly, compliance with performance standards will be limited to assessment of the completion of items specified in the DIP. However, to appreciate whether the project was truly able to accomplish its objectives will require consideration of impact metrics not included in the DIP (as approved by USAID/PVC) since the DIP does not provide impact-level indicators of achievement of project objectives. Accordingly, the evaluation team and the respective Country Directors will agree on approaches to determine whether or not project objectives were met in each country.  Based on the original Detailed Implementation Plan (DIP), and amendments prepared by TNC for the project, determine whether activities leading to accomplishment of the program objectives have been met, were partially met, or were unattained. This is the single most important element the evaluation must document. In addition to the H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 26 discussion of project results in the text of the evaluation, this information should also be put into matrix format. List each objective, and key activities as specified in the DIP (include in the analysis all items included in the DIP matrix as well as the text of the DIP). In so doing, also: Identify major successes and constrains to achieving objectives as well as unanticipated effects. As part of this discussion comment on the PVOís and its local partnersí capacity to perform program monitoring and evaluation. Note any constraints that prevented the PVO from measuring achievement of program objectives. If the program does not have ìbaselineî and end-of-project data from which judgments can be made about the achievement of project objectives this should be noted. Identify if the project had a detailed implementation plan describe field staff familiarity with the project design, implementation plan, and monitoring and evaluation plan and data.  Assess effectiveness of models, approaches or assumption that underlie the project. In particular consider the following: The decentralization model utilized by TNC in both countriesí programs and the advantages and complications that this approach expressed in each program; The process of sub-awards to Partners in-country and how ongoing institutional assessment of Partnersí capacity was, or was not, utilized to inform partnership decisions; and How successful were self-assessment approaches utilized by TNC in each of the countries?  Have key approaches been scaled-up in the project area or replicated elsewhere in country or in other countries?  Consider whether the articulated hypotheses were successfully tested in this pilot activity: The underlying hypothesis is that conservation of biological diversity ñ park reserve protection depends on conservation NGOís and CBOís increasing their financial self￾sufficiency from in-country sources and on increasing their political voice through coalition building. The second hypothesis is that a grass roots economic development based on products derived from the protected area can build a grassroots constituency for conservation (Cooperative Agreement, page 8.)  Has the PVO engaged in program or policy advocacy? What was the focus of the advocacy and its impact? What are the benefits of being directly involved with the program as opposed to ëhiding behind the partnerí? Detriments? Compare the two country programsí approach to advocacy and related impacts. Has TNC used the project data for advocacy with the public sector or with other PVOs in-country or with non-partner NGOs?  Discuss what the PVO and local partners have ìlearnedî implementing this project. Identify if these ìlessons learnedî have been applied elsewhere (other projects or countries.) How has the Community Assessment Tool changed and developed within TNC as it evolved into the Stakeholder Analysis?  What were the key implementation capacity challenges faced by TNC and its Partners? How did TNC address them along the way? H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 27 2. Assess the status of partnership(s) with NGOs, community based organizations or local level government. [High] NOTE: Determination of which organizations are considered ìPartnersî for this evaluation will be made through agreement between the respective Country Directors and the evaluation team.  Include a chart that: Categorizes local level partners. Are the partners NGOs, affiliates of the PVO, private or commercial groups, cooperatives, community-based organizations, regional or local governments or intermediate service organizations? Identify the type of mechanism employed with each partner, i.e. MOU, sub-grant, contract. Outline the role, responsibilities and decision-making responsibility of the partners. Identify the fiscal autonomy and amount of grant funds directly managed in past year.  Assess the process that the PVO used to build and maintain local partnerships. Does the PVO have a partnership policy and approach to assess potential partners? Discuss the effectiveness of TNCís institutional assessment of partners and the TNC Institutional Assessment Tool, where used. Has this project improved/expanded use of the tool throughout TNC? Did the PVO do a formal assessment of local partner capacity and develop plans to build their capacity?  Document change in local partner capacity. What were the major constraints to effective partnerships? Discuss recommendations for resolution. The sub-recipient monitoring of financial systems seems effective, but a deeper analysis of the Institutional Development field is appropriate. Paul Hardy is the Caribbean counterpart. Has the project increased the local partnersí access to information technology? How?  Assess the local level partnersí satisfaction with the partnership.  Assess the PVOís and its local partnersí involvement in local networks or with intermediate service organizations.  Building community constituency can be expected to vary according the local partnerís prior interactions with the communities. Is there a way to determine a time frame for initial conflict resolution and building of trust?  What activities were most useful in garnering community participation and interest?  What methods were deemed least effective?  If negative feelings towards the Park or local partner were expressed, what were the recurring themes? 3. Networks [low] H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 28  What effect did participation in networks or service organizations have on the operational or technical capacity of the local partner? What would make it more effective? Cite the major implementation lessons learned and recommendations B. Program Management [Medium] The objective of the MG is to build PVO headquarters and field organizational and technical capacity. This section of the evaluation should assess change in TNCís operational management capacity as a result of PVC grant. It would be appropriate to discuss the structural change process currently underway within the Conservancy. 1. Strategic Approach and Program Planning [low -- 30%] Have changes occurred in TNCís headquarters capacity to:  Manage the planning process --- program renewal, strategy integration, project design;  Address over-arching program issues of replicability, scale-up, sustainability; and  Use performance data to forecast emerging trends and develop strategic plans? 2. Country level Initiatives [low] Identify and assess (if relevant), PVO contributions in the following areas: TNC cooperation and coordination with the USAID mission and other development partner programs including natl./local government agencies; Has TNC consistently shared lessons learned with other PVOs in-country or with non-partner NGOs? Does TNC provide the local USAID Mission with results data? If the country or program area has a history of violent conflict, other man￾made/natural disasters, or food insecurity, describe: (a) TNC activities in conflict prevention, mitigation, resolution or post-conflict transition (b) TNC's contingency plan to ensure the safety of program staff and program continuity. 3. Monitoring and Evaluation [moderate] Has the project implemented a process and put into place a sustainable system to monitor project performance and collect results (effects or impact) data? Is TNC collecting data from the DIP? Is it using the data? Is Partnersí capacity to manage for results using data being improved? Indicate the level at which data is collected (output vs. results) and how data informs decision￾making. Has TNC used the MG to develop a sustainable capacity at headquarters and in the field offices to monitor project performance and measure effects and impact? What were the biggest constraints to improving project monitoring and evaluation and what are the recommendations for PVC and the PVO? H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 29 How could TNCís system of monitoring for impacts have been improved? How effective were TNC's Consolidation scorecard and Institutional assessment tools? Did they adequately serve as early warning systems for monitoring TNCís and its Partner's needs, problems, and performance in order to inform adaptive management decisions? 4. Overall Management Assess overall project management. How could the program design have been improved in terms of oversight, inter-country coordination, cross-learning, and donor relations? What were implications for management of the grant? 5. Assess progress towards sustainability [Medium]  Does the project have a system for addressing financial or operational sustainability? Do relevant aspects of the project have business plans?  Describe the program elements, financial or operational, that are intended to be sustained (objectives); the means for judging if the sustainability objectives have been achieved (indicators); and sustainability achievements and prospects for post-grant sustainability. Describe the positive and negative effects of the projectís managerial shift since the beginning of this fiscal year. (Sub-recipient funding ended and TNC directly contracts Development Options.)  Identify if the project has any cost-recovery mechanisms, i.e., local level financing or approaches to generate resources to support project operations. Describe the achievements of these mechanisms and provide an estimate of the magnitude of the system, for example, if possible, provide a ratio of costs recovered to operational expenses. If possible with limited effort, identify potential funding sources for community conservation work by JCDT. Currently, funds raised as a result of community mobilization go directly into the community. How can the local partner integrate and sustain its education and community work? 6. FINANCIAL MANAGEMENT [LOW]  Are adequate financial monitoring systems in place?  Has the program leveraged additional resources (beyond the match)?  How cost-effective is the technical approach?  What were the challenges related to the matching requirements? What implications did these have on implementation? Strategically, have ìlossesî absorbed by TNC affected TNCís ability to and willingness to pursue additional grants with similar matching requirements? [high] 7. Information Management [low]  Comment on the utility and timeliness of PVOs required reports.  Has the PVO developed, disseminated and used ìlessons learnedî from the project? 8. Logistics [low] Comment on the adequacy and timeliness of PVOs material inputs. 9. Supervision/HRD [medium] Assess if there were sufficient staff with the appropriate technical and management skills to oversee program activity at both headquarters and in the field program. Discus delays in project H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 30 hiring, the rate of staff turnover, and the impact of these dynamics on project implementation. The project is a representative example of a common theme and road block in past TNC grant management. Responsibilities are spread over several individuals, which doesnít facilitate cohesion and can be confusing or frustrating to the Partner. Discuss the possible benefits of a single grant manager focusing on capacity building 10. USAID Management [medium] Comment on USAID's oversight and backstopping of this cooperative agreement as well as the procurement process leading to the award. 2. Cite the major management lessons learned and recommendations. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 31 Annex D: Advocacy Implementation Chain* 1. Jamaica Program Responsible Organization (Type) 12 Advocacy Capacity Improvement Activities13 Organization(s) targeted for improvement Advocacy policy Targets and (institution targeted) Actual Advocacy Events Policy changes realized Comments National Level JCDT (LNGO) N/A N/A Fully implementing NGO co-management of protected areas (NEPA, Forestry) Enabling JCDT to generate fees at BMJKNP (Forestry, Treasury, NEPA) Greater Govít. participation in protected area management (NEPA, Forestry) Many meetings, legal analysis, and ongoing dialogue Development of co￾management agreement for BMJKNP with Forestry, NEPA, and JCDT Collection/ utilization of fees at BMJKNP (but still needs to be formalized) Successful, if slow. Agreement needs more detail, and may benefit from longer duration. Needs greater formal LAC role in mgt. JCDT (LNGO) Leadership in establishment of JPAN: Jamaican Protected Area Network Other LNGOs involved in P.A. management Establishment of a network of LNGOs to share experiences, lobby government, pursue donor funding, and integrate P.A. management (LNGOs in P.A. management) Establishment of JPAN, subsequent meetings, communications with GOJ Establishing a unified advocacy and feedback forum for park management between civil society and government Very successful 12 Describe in parentheses as USPVO, LNGO, Govít., business, etc. 13 List as separate rows advocacy activities that an organization does directly by itself (such as a USPVO advocating directly with government or establishing policy forums), and tactics where an organization strengthens another organization to perform the advocacy (such as when a USPVO strengthens LNGO capacity to conduct advocacy.) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 32 Responsible Organization (Type) 12 Advocacy Capacity Improvement Activities13 Organization(s) targeted for improvement Advocacy policy Targets and (institution targeted) Actual Advocacy Events Policy changes realized Comments TNC (USPVO) Networking and training at International Training week JCDT Other LNGOs NGO/Govít protected area management protocols (NEPA, Forestry) Jamaican Protected Area Network advocacy forum (LNGOs in PA management) None Ongoing LGNO (and GOJ) PA management policy forum Mostly operating in background, but helpful role. Local Level JCDT (LNGO)  Strengthening of LACs  Linking LACs to formal government structures LACs 1, 2, and 3 Improved park management (BMJKNP, NEPA) Improved civic advocacy for local econ/social development concerns (Various GOJ ministries and field officers) Monthly LAC meetings with GOJ participation Greater participation in P.A. management A number of government responses to civic initiatives, including access to government services, input into Community Development Committees, local education reform, etc. Very successful. Important linkages formed with national government. LAC role in P.A. management needs to be formalized. JCDT (LNGO) N/A N/A Direct advocacy on behalf of LAC interests, chiefly for improved access to govít. services. (Various GOJ ministries and field officers) Numerous discussions and meetings by JCDT staff on behalf of LACs Numerous local initiatives received greater attention as a result of these efforts Taking advantage of JCDTís location in capital, proved valuable asset to community. 2. Indonesia Program H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 33 Responsible Organization (Type) Advocacy Capacity Improvement Activities Organization(s) targeted for improvement Advocacy Targets (policies) and institution targeted Actual Advocacy Events Policy changes realized Comments National Level Forum in LLNP Assistance to establish Forum Training of NGO members in organizational issues Coalition itself to serve as an effective communication forum regarding park affairs To expand community involvement in park management. No specific policies. Rather it was to improve communication among stakeholders Periodic meetings of Coalition Brief local government and local legislature Researchers must go through Forum to work in park and must brief Forum on results of analysis Forum is vehement about maintaining its independence YPAN in LLNP None LLNP To improve enforcement of ogging rules. Demonstration in Palu to protest inactivity of LLNP staff in stopping illegal logging and to express public support for ranger enforcement functions YPAN reports that LLNP officials appreciated the demonstration as it provided support to rangers. TNC felt the demonstration was confrontational Led to major conflict between TNC and YPAN over autonomy and appropriate advocacy approaches. TNC in Komodo None KNP Involvement of more takeholders in park management Functioning working groups; Analyses; assistance drafting Drafting of Joint￾Management Strategy for KMP Establishment of Tourism Concession New precedents set for PA mgt. Much interest in replication. TNC in Komodo None District Governments Better coordination among ocal government players cross district boundaries round KNP Facilitating meetings Establishment of Coordination Venue across district boundaries new precedent for district￾level coordination TNC in Komodo None Navy, Army, KNP, Coast Guard, Police Enforcement coordinated among law enforcement agencies inside and outside KNP Facilitation of meetings; training of park rangers; provision of patrol boats Joint Navy, Army, Coast Guard and KNP enforcement patrols H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 34 Responsible Organization (Type) Advocacy Capacity Improvement Activities Organization(s) targeted for improvement Advocacy Targets (policies) and institution targeted Actual Advocacy Events Policy changes realized Comments TNC in Komodo Functioning working groups; Analyses; assistance drafting Working with KNP, Local Govít and business on Mgt. Plan Local government also endorsed Park Management Plan. Local government now engaged in two other parks similarly in Indonesia Local Level TNC in Komodo Sub-district government Village government Control level of human population increase in park Facilitated meetings and discussions @ village level Village Heads agreed to restrict immigration to communities located within KNP TNC in Komodo None Local Government and fishers Improved fishing practices Ongoing negotiation, research, assistance drafting ordinance Giving Park authority to issue fishing permit outside park boundary Local ordinance passed on fishing practices (compressors) New precedent as Local Government recognizes authority of Park for fish resource outside park boundary H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 35 Annex E: Partnership Table, by Country Program Visited* The following table summarize the relationships between the various Partners in the project, as they relate to the USPVO. Arrows indicate sources of funding emanating from the MG. Dotted lines indicate funding from other sources. The following key indicates shape corresponds to which kind of organization. Table E 1: Key to Partnership Tables Figure E 2: TNC Jamaica Partnership Diagram During Project TNCís Partnership arrangements are presented graphically above, in Figure E2, and in tabular form below, in Table E1. TNC has an institutional culture which has traditionally favored maintaining long￾term relationships with a single NGO partner. As can be seen in Figure E2 this arrangement meant that TNC has no direct relationship with CBOs or with the key GOJ organizations involved in Park management. This practice delegates considerable latitude JCDT to make local political and implementation decisions under the grant, but also provides little leverage over local developments. Coupled with a lack of an overall MOU with JCDT or mechanisms to play a fuller role in implementation decisions, TNCís Jamaica Country Director felt he the only leverage he had to manage what he perceived as an unresponsive LNGO that was not achieving adequate results was to cut off funding. USPVO Sub-sub grantee Affiliate/ Independent CBO Primary Grantee Key to Symbols Implementation Partners that donít get MG funds Funding/TA TNC JCDT U-Tech LACs 1-3 D.O. TNC Jamaica Program Partnership Diagram During Project Forestry NEPA USPVO PA Mgt. & Comm. Conservation ME contractors CBOs H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 36 Figure E 3: TNC Jamaica Post-Project Partnership Strategy Based on experience under the grant the Jamaica Country Director has decided to diversify TNCís local partnership portfolio in both the policy and implementation arenas. As can be seen in Figure E3, TNC plans to have direct relationships ñ including training, technical assistance, and policy dialogue ñ with key government agencies involved in park management (Forestry and NEPA) and with other LNGOs. TNC plans to focus future efforts in a different area, Cockpit, and utilize other LNGOs in seeking to establish park management arrangements. Figure E 4: TNC Indonesia Partnership Strategy 1999-2000 TNC JCDT LACs 1-3 TNC Jamaica Program Post-project Partnership Strategy Diagram Forestry NEPA USPVO PA Mgt. & Comm. Conservation CBOs EFJ ? Other (Cockpit area) LNGOs Other CBOs in Cockpit area ? TNC/ Jakarta YPAN Jakarta NR User groups TNC Indonesia Program Partnership Diagram 1999-2000 LLNP Officials USPVO CBOs Local YPAN LLNP Govít Field Office TNC/ LLNP Field Office YPAN KNP Field Office TNC/ KNP Field Office KNP Officials National Park Officials TNC/ CMP LLNP Forum NGOs NGOs NR User groups Lore Lindu Activities Komodo Activities GOI Navy P.A Mgt. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 37 Figure E5: TNC Indonesia Program Partnership Diagram 2001 Table E 1: Jamaica Partnership Table Level Organizations Organization Type14 Agreement type15 Role/Responsibility16 Funding level, source, and autonomy17 Quality and outcome of Partnership18 Primary Local Partners19 JCDT LNGO Standard sub￾award agreement. No fundamental MOU longer￾term and more fundamental roles/ responsibilities in Jamaica. Primary (virtually sole) partner for TNC in Jamaica since 1989. Main implementation agent for grant. Sub-award of approximately $380,000. TNC/JCDT agree on work plan consistent with DIP, with little considerable autonomy in implementation. Due to dissatisfaction on the part of TNC, sub￾award was terminated prematurely. Future of partnership, which began in 1989, is uncertain. 14 For example, LNGOs, affiliates of the PVO, private or commercial groups, cooperatives, community￾based organizations, regional or local governments or intermediate service organizations 15 For example, MOU, sub-grant, contract 16 For example, service delivery, aggregation of interests, target of institutional strengthening, or others. 17 USD equivalent of funds provided through the grant and degree of autonomy that organization has over expenditure. Indicate source of funds (USPVO, affiliate, LNGO, or sub-awardee, or other support organization) for funds trickling down from MG. 18 Was Partnership successful? Is it still ongoing? Growing? Has it terminated as planned? What are organizationsí (USPVO and other organizationsí) perception of the quality of the Partnership? TNC/ Jakarta YPAN Jakarta NR User groups TNC Indonesia Program Partnership Diagram 2001 LLNP Officials USPVO CBOs Local YPAN LLNP Govít Field Office TNC/ LLNP Field Office YPAN KNP Field Office TNC/ KNP Field Office KNP Officials National Park Officials TNC/ CMP LLNP Forum NGOs NGOs NR User groups Lore Lindu Activities Komodo Activities GOI Navy P.A. Mgt. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 38 Level Organizations Organization Type14 Agreement type15 Role/Responsibility16 Funding level, source, and autonomy17 Quality and outcome of Partnership18 Development Options (D.O.) Contracting firm Contract. Hired to assess ME support work to date by JCDT and U-Tech as well as to provide TA to LAC entrepreneurs leading to business plans. $15,000 from TNC, to be matched equally by D.O. funds. Based on SOW. Ongoing Partners that are ìsubsî to primary partners20 University of Technology Jamaica (U￾Tech) University consulting division. Contract Provide TA/Training to LAC entrepreneurs leading to business plan development. Based on SOW. Terminated Implementation partners that do not receive funds Dept. of Forestry GOJ Eventually a Co￾management Agreement for BMJKNP was developed. Forestry is responsible for all forestry resources in BMJKNP. None. Partnership was primarily with JCDT. It continues to take a long time to finalize a Co￾Management Agreement that permits cost recovery for JCDT for Forestry cabins in the BMHJKNP. National Environmental Protection Agency (NEPA) GOJ Eventually a Co￾management Agreement for BMJKNP was developed. NEPA is responsible for PA management and delegates that role to JCDT under the Co-Management Agreement None No major problems. Table E 2: Indonesia Partnership Table Level Organizations Organization Type Agreement type Role/Responsibility21 Funding level, source, and autonomy Quality and outcome of Partnership Affiliates/Other Independent Partners YPAN NGO created by TNC ñ originally as a local fund raising vehicle ñ and Memorandum of Understanding and Sub￾Agreement for LLNP (No YPAN was to be the field implementer of TNCís programs in KNP and LLNP Legally autonomous, but created by TNC to have same mission as Partnership failed (see Section 6.1) 19 Receiving funds directly from the USPVO. 20 Organizations that receive funds that are re-granted from TNC, through and intermediate primary partner. 21 For example, service delivery, aggregation of interests, target of institutional strengthening, or others. H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 39 Level Organizations Organization Type Agreement type Role/Responsibility21 Funding level, source, and autonomy Quality and outcome of Partnership later to implement TNCís program Sub￾Agreement for KNP) TNC (and as its fundraiser.) Virtually 100% financially dependent on TNC. Primary Local Partners Various members of the LLNP Forum. NGO Varied: informal mentoring to grants and contracts To implement various trainings and research projects of the program. Also to provide a place for communities, NGOs and GOI to talk about issues related to LLNP. Independent NGOs. TNC no longer providing significant resources. Previously was mutually beneficial. A tourism operator. Private business Forging a joint venture to manage certain aspects of KNP JV partner with TNC in managing aspects of KNP. TBD Not yet finalized. Partners that are ìsubsî to primary partners Implementation partners that do not receive funds Various groups in KNP, such as of fisherman and villages Varies from informal networks to villages Informal agreements To jointly implement enterprise initiatives. No direct funding, although inputs are provided by TNC. Ongoing. National Parks/KNP/LLNP GOI MOU @ national level. Working on Mgt. Agreement @ KNP. GOI is steward of national parks, TNC helps manage resource, provides TA, raises funds. TBD, (likely to be approx. $5million in GEF grant. Ongoing District Government Local Government Signatory to KNP Mgt. Plan Responsible for resources outside protected area. TNC needs their support to succeed. None. Ongoing H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 40 Annex F: Sustainability Analysis* JAMAICA PROGRAM Focus is on sustainability of benefits to BMJCNP Table F 1: Jamaica Sustainability Table Item Supporting factors Inhibiting factors Conclusion Political National Government Support JCDT (with TNC) was pioneer in NGO management of PAs Strong support by NEPA JPAN provides more leverage with government Declining funds available from GOJ Perception by GOJ that JCDT is less capable lately Basically strong government ties are an asset to sustainability. However, recent lapses in performance threaten government support if performance is not soon upgraded. Community support Project work in addressing community social issues has improved community relations Education program will likely improve support over medium to long term Lack of success in economic benefits front my increase community skepticism JCDT has improved community relations at JCDT, but still has considerable room for improvement. Institutional Country Director located in Kingston Decision to manage program locally Very high technical high caliber of Country Director and level of local respect Independent office Challenge of establishing new programs in new areas Need to determine future of role in BMJCNP Plenty of challenges for the CPD, but TNC is much stronger with a competent in-country presence. However, this may be a moot point if TNC focuses on other PAs in Jamaica. TNC Partnerships Partnership with JCDT is currently in very difficult straights TNC is diversifying partnerships Decision by TNC to forge its own relationships with government and other NGOs is sound. TNC perceives new partners as easy to work with. Difficulties with JCDT make progress in BMJCNP difficult. Although TNC plans to work with other LNGOs, it would appear that this would chiefly be outside of BMJCNP. Local Partner (JCDT) Management Interim Exec. Dir. appears to be very strong and capable Improved, broader, vision for JCDT Improved communication in JCDT Purge of top management of JCDT Very low morale at JCDT Loss of personnel. JCDT is in a difficult position. They will be challenged to regain their momentum. But, they remain the key player in BMJCNP. Financial Local Partner Viability Green Expo as source of funding Dire financial circumstances have forced JCDT to focus on financial sustainability Lack of business plan under project Loss of EU and MG projects The lack of TNC funding appears to be forcing JCDT to reexamine what it needs to do to become more financially sustainable. A moderate likelihood of success. Fortunately, some funding will continue to be available through the trust fund, park fees and some grants. Not applicable to Indonesia (since YPAN no longer exists). H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 41 Annex G: List of Persons Interviewed Name Organization Azis Abdullah, Chair, Harapan Keluarga Fishers Group, Labuhan Bajo Abubakar TNC (Indonesia) Azis Anggu Conservation Cadres (Indonesia) Dan Cambell TNC, Arlington, VA (Jamaica) Albert Daley Environmental Foundation of Jamaica (EFJ) Rili Djohani TNC (Indonesia) Ian M. Dutton Indonesia Country Director, TNC Damayanti Eliyanti Donna Fray JCDT (Jamaica) Gina Green VP & Director, TNC, Caribbean Division Brian Greenberg USAID/BHR/PVC Paul Hardy TNC, Arlington, VA (Jamaica) Harsono TNC (Indonesia) Paul Hartman TNC (Indonesia) Hirmen Sofyanto TNC (Indonesia) Madeline Holly TNC, Arlington, VA (Jamaica) Michelle Libby TNC/VA Community Conservation Specialist (Jamaica and Indonesia) Peter Maus TNC, Indonesia Betsy McGean TNC, World Wide Office Siti Mariam TNC (Indonesia) Peter Mous TNC (Indonesia) Marcia Mundle JCDT (Jamaica) Andreas Mulyadi TNC (Indonesia) Duncan Neville TNC, Indonesia Novianto Bambang Wawandono Komodo National Park Pamy TNC (Indonesia) Jos Pet, Deputy Director TNC (Indonesia) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 42 Poerwanto TNC (Indonesia) Residents of Kukusan Village Indonesia LAC members of Jamaica LAC members of Jamaica Sadiq, Partnership Forum of Lore Lindu National Park David Smith Former Executive Director, JSCT (Jamaica) Marcia Smith JCDT (Jamaica) Robert M. Stephens Board Member; Acting E.D., JCDT (Jamaica) J. Subijanto TNC (Indonesia) Sudaryanto TNC (Indonesia) Gatot Wibisono TNC (Indonesia) Agung Wibowo TNC; formerly YPAN Darwina Wijayanto Formerly of YPAN Lynette Wilkes, COO JCDT (Jamaica) Catherine Winata TNC (Indonesia) Terrance Williams TNC Jamaica County Director Darwina Wijayanti Formerly of YPAN Carey Yaeger USAID/Indonesia H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 43 Annex H: Documents Reviewed De Ridder, Kim: ìTNC and YPAN: Partnership and Capacity Buildingî, October 1999. Dubkowski, Keith, et al. ìFact Sheets on Management of Nine Coastal and Marine Protected Areas, an Internship Reportî August 2001. Environment North and Associated Consultants ìKomodo National Park Tourism Strategyî, July 2001. Erdman, Mark V. and Pet, Jos S. ìKrismon & DFP: some observations on the effects of the Asian Financial Crisis on destructive fishing practices in Indonesiaî SPC Life Reef Fish Information Bulletin, # 5 (April 1999) Fox, Helen E., ìEnhancing Reef Recovery in Komodo National Park, Indonesia: A proposal for Coral Reef Rehabilitation at Ecologically Significant Sitesî, undated. Fox, H. E., et al. ìIncreased Coral Cover in Komodo National Park, Indonesia: Monitoring for Management Relevanceî (undated.) Mac Leod, Paige: ìTNC Indonesia-YPAN Partnership Diagnosticî April 21-May 6, 2000. Memoranda of Understanding between TNC and YPAN: general MOU as well as the MOU for LLNP Mckinsey & Company: ìWorking Together for the Future: TNC/YPAN Board and Management Workshopî Jakarta, November 4, 2000. Mous, Peter et al. ìCyanide fishing on Indonesian coral reefs for the live fish market ñ What is the problem?î SPC Life Reef Fish Information Bulletin #3 (May 2000). Monitor Company, ìTourism-concession for Komodo National Park Preliminary Diving￾strategyî, September 1999. Mous, Peter and A. Halim and Jos S. Pet, ìHarvest Characteristics of Gango, a Method to Capture Fingerling Groupers from Mangrove Areas in West Flores, Indonesia: a report for the TNC/YPAN Komodo marine conservation projectî (June 1999). Mous, Peter and Maya Gorrez, ìStakeholder Involvement in the Site Conservation Planning Process for Komodo National Park ñ Work plan and Budgetî Mous, Peter and Jos S. Pet ìStatus of the coral reefs in and around Komodo National Park 1996- 1998: a monitoring reportî (undated) Neville, Duncan, ìLessons learned from Compatible Enterprise Development initiatives at Lore Lindu National Parkî (November 2001) Pet, Jos S. and Rili H. Djohani, ìCombating destructive fishing practices in Komodo National Park: Ban the hookah compressor!î SPC Life Reef Fish Information Bulletin, #4, (April 1998) H:\INCOMING\06-26-MSI-final-TNC-evaluation\PVC TNC Final Evaluation Report.doc 44 Pet, Jos S. and Andreas Muljadi, ìTNC Pohnpei Training Workshop: Grouper Spawning Aggregation Sites (SPAGS) Conservation and Monitoringî (March 2001) Pet, Jos S. and Lida Pet-Soede ìA note on cyanide fishing in Indonesiaî SPC Life Reef Fish Information Bulletin, #5 (April 1999) Pet, Jos S. ìDestructive fishing methods in and around Komodo National Parkî SPC Life Reef Fish Information Bulletin, #2 (May 1997). Pet-Soede, C. et al ìImpact of Indonesian coral reef fisheries on fish community structure and the resultant catch compositionî, Elsevier Fisheries Research 51 (2000) pp 35-51. Pet-Soede, C. et al ìEconomic Issues Related to Blast Fishing on Indonesia Coral Reefsî Pesisir & Lautan, Volume 3, No. 2 (2000) Pet-Soede, C. et al ìBlasting Away: the Economics of Blast Fishing on Indonesian Coral Reefsî (undated) Pet-Soede, Lida and Mark Erdmann ìAn overview and comparison of destructive fishing practices in Indonesiaî SPC Live Reef Fish Information Bulletin, #4 (April 1998) Ruitenbeeek John and Cartier ìKomodo National Park, Indonesia: Economic Issues, Analyses and Prescriptionsî Draft Report, 2001 Sadovy, Yvonne and Jos Pet: ìWild collection of juveniles for grouper mariculture: just another capture fishery?î SPC Life Reef Fish Information Bulletin #4 (April 1999). The Nature Conservancy, promotional material on the Coastal and Marine Conservation Center The Nature Conservancy, ìStrategy and Action Plan for the TNC Komodo Fish Culture Projectî, July 2001. The Nature Conservancy, ìWorkshop on Sustainable Marine Tourism in Komodo National Parkî (March 2001) The Nature Conservancy, various functional project reports. Benjamin Kahn (Apex Environmental), ìKomodo National Park Cetacean Surveys, Monitoring Reportî October 2000. Kahn, Benjamin (Apex Environmental), et al ìKomodo National Park Cetacean Surveys: A Rapid Ecological Assessment of Cetacean Diversity, Abundance & Distributionî August 2000. The Nature Conservancy, ìKFO Beatsî, monthly newsletters from Komodo Field Office. ìTwenty-five year Master Plan for Management, 2000-2025, Komodo National Parkî Williams, Terrence and Titayanto Pieter (TNC), Project Annual Reports and Final Report for the project period.