1 March, 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Gary Morgan (Team Leader), Darmawan & Am Azbas Taurusman Evaluation of the USAID-MMAF Marine Resources Program (MRP), Indonesia Page | 1 Table of Contents ACRONYMS AND ABBREVIATIONS...................................................................................................4 EXECUTIVE SUMMARY.........................................................................................................................6 1. INTRODUCTION........................................................................................................................11 1.1 Background ...........................................................................................................................11 1.2 Importance of Marine Fisheries and Biodiversity Conservation in Indonesia ......................12 Policy Alternatives ........................................................................................................................12 Production, exports and food security.........................................................................................13 Management..................................................................................................................................14 Indonesia’s Vision and Strategy for marine resources development............................................15 1.3 USAID Assistance .................................................................................................................16 1.4 Underlying Development Assumptions................................................................................17 2. DESCRIPTION OF THE MARINE RESOURCES PROGRAM (MRP)..........................................18 3. EVALUATION PURPOSE & QUESTIONS ..................................................................................19 4. EVALUATION METHODOLOGY...............................................................................................21 4.1 Findings: Data Collection and Analysis.................................................................................22 Document and Literature Review ................................................................................................22 In-depth Key Informant Interviews (KIIs) .....................................................................................22 Data analysis..................................................................................................................................23 4.2 Conclusions and Recommendations.....................................................................................24 4.3 Limitations.............................................................................................................................24 5. Findings, Conclusions and Recommendations..............................................................................24 5.1 Over-arching Recommendation............................................................................................24 5.2 Level 1 Evaluation .................................................................................................................26 5.2.1 Key Question 1.................................................................................................................26 5.2.2 Key Question 2.................................................................................................................35 5.2.3 Key Question 3.................................................................................................................40 5.3 Level 2 Evaluation .................................................................................................................42 5.3.1 Key Question 4.................................................................................................................42 5.4 Level 3 Evaluation .................................................................................................................44 5.4.1 Key Question 5.................................................................................................................45 6. PLANNING FOR LONG TERM OUTCOMES............................................................................48 SUMMARY OF CONCLUSIONS & RECOMMENDATIONS .............................................................51 Page | 2 REFERENCES........................................................................................................................................56 ANNEXES.............................................................................................................................................57 ANNEX 1: STATEMENT OF WORK..............................................................................................57 ANNEX 2: EVALUATION DESIGN AND METHODS...................................................................72 (a) Questions posed at each level of Evaluation: ...................................................................72 (b) Overall Approach to KIIs:.................................................................................................74 ANNEX 3: DATA COLLECTION TOOLS .....................................................................................81 ANNEX 4: LIST OF INFORMATION SOURCES............................................................................84 ANNEX 5: SUMMARY OF EVALUATION DESIGN AND METHODS .........................................88 ANNEX 6: MATRIX OF PROJECT AND MRP COMPONENT PERFORMANCE ........................89 ANNEX 7: RESULTS FRAMEWORKS FOR IMACS AND MPAG................................................140 ANNEX 8: CONFLICTS OF INTEREST & GOVERNANCE.........................................................142 Page | 3 Evaluation of the USAID-MMAF Marine Resources Program (MRP), Indonesia March 2013 Page | 4 ACRONYMS AND ABBREVIATIONS BAPPENAS Badan Perencanaan Pembangunan Nasional - National Development Planning Agency BAPPEDA Badan Perencanaan Pembangunan Daerah - Local Development Planning Agency BPSDM Badan Pengembangan Sumber Daya Manusia – Human Resources Development Agency CAP Conservation Action Planning CCA Climate Change Adaptation CCR Coastal Community Resilience and Climate Change Adaptation component of IMACS CI Conservation International COP Chief of Party CTC Coral Triangle Center CTSP USAID Coral Triangle Support Partnership CTI-CFF Coral Triangle Initiative for Coral Reefs, Fisheries and Food Security CTSP-I Indonesia component of USAID Coral Triangle Support Partnership DKP Dinas Kelautan dan Perikanan - Marine and Fishery Agency DOJ US Department of Justice EAFM Ecosystem Approach to Fisheries Management E-KKP3K Panduan Evaluasi Efektivitas Pengelolaan Kawasan Konservasi Perairan, Pesisir dan Pulau-pulau Kecil – MMAF’s Management Effectiveness Indicator for MPAs FAO Food and Agriculture Organization of the United Nations GOI Government of Indonesia Ha Hectares HT Hypothesis Testing IBRC Indonesian Biodiversity Research Center ID Institutional Development component of IMACS IMACS Indonesia Marine and Climate Support project IUU Fishing Illegal, Unregulated and Unreported fishing KIIs Key Informant Interviews KKJI Direktorat Konservasi Kawasan dan Jenis Ikan - Directorate of Conservation for Area and Fish Species, MMAF KKP Kementerian Kelautan dan Perikanan Republik Indonesia (Ministry of Marine Affairs and Fisheries) KP3K Kelautan Pesisir dan Pulau-pulau Kecil - Directorate General of Marine, Coasts, and Small Islands, MMAF MEY Maximum Economic Yield MIS Management Information systems MMAF Ministry of Marine Affairs and Fisheries MOU Memorandum of Understanding MPA Marine Protected Area MPAG USAID Marine Protected Area Governance Project Page | 5 MRP USAID Marine Resources Program MSY Maximum Sustainable Yield NCC National Coordination Committee of CTI-CFF NGO Non-governmental Organization NOAA US National Oceanic and Atmospheric Administration NP National Park P4KKP Penilaian, Penetapan, Perancangan dan Pengelolaan Kawasan Konservasi Perairan - Team for Assessment, Designation, Design, and Management of Marine Protected Areas for Savu Sea National Park PMP Performance Management Plan PUSKITA Pusat Analisis Kerjasama Internasional dan Antar Lembaga - Center for Analysis of International Cooperation and Institutions, MMAF PUSLAT Pusat Pelatihan Kelautan dan Perikanan - Training Agency for Marine and Fisheries SFM Sustainable Fisheries Management component of IMACS SOW Scope of Work TNC The Nature Conservancy TNC-IMP The Nature Conservancy - Indonesia Marine Program UP University Partnerships UCSC University of California, Santa Cruz UCLA University of California, Los Angeles URI-CRC University of Rhode Island, Coastal Resources Center USAID United States Agency for International Development WCS Wildlife Conservation Society WWF World Wildlife Fund (within US)/Worldwide Fund for Nature (international) WWF-I Worldwide Fund for Nature – Indonesia WPP Wilayah Pengelolaan Perikanan (Fisheries Management Area) Page | 6 EXECUTIVE SUMMARY An evaluation of the USAID Indonesia’s Marine Resources Program was undertaken during the period January-March 2013 by a team of three independent contractors assisted by three USAID Washington staff. EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of the performance evaluation was to provide USAID/Indonesia (USAID) and the Ministry of Marine Affairs and Fisheries (MMAF) with an independent review of progress to date of assistance provided by USAID to MMAF under the Marine Resources Program (MRP) and to guide the development of innovations and adaptation of USAID’s assistance in a dynamic environment. The evaluation will be used in the short-term by USAID and MMAF to modify ongoing assistance, and in the longer-term to plan future assistance strategy as part of USAID Indonesia’s Country Strategy 2014-2019.. The MRP evaluation was conducted at three inter-related levels:  Level 1 – individual projects (outputs achieved)  Level 2 – MRP component (results achieved) and  Level 3 – overall MRP (strategic approaches) PROJECT BACKGROUND MRP is designed to support Government of Indonesia’s commitment to the CTI-CFF to manage and conserve critical marine resources. Of the five CTI-CFF goals, USAID priorities are aligned to support the goals of (a) an ecosystem approach to fisheries management, (b) marine protected areas (c) climate change adaptation. USAID authorized the Marine Resources Program (MRP) at an estimated funding level of $31,900,000 to support a five-year program of assistance (2010–2014). The funding for the MRP comes primarily from USAID biodiversity funding (approximately 75 %) and as such the activities of IMACS and MPAG are focused on reducing key threats to marine biodiversity in Indonesia and to achieving sustainable commercial fisheries resources. In addition, a significant percentage (approximately 25%) of funding comes from USAID climate change adaptation funds. Climate Change funds and biodiversity funds are often used when and where an activity can achieve both biodiversity and CC objectives, with many such cases occurring as part of the MRP’s small grants program. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation team adopted methods to capture sufficient and accurate information for conducting a performance and developmental evaluation at three levels, viz. Individual project level (outputs achieved), MRP component level (results achieved) and overall MRP level (strategic approaches). Page | 7 The team’s methodology was highly participatory, collaborative and open, and engaged USAID staff, MMAF staff, local Government agencies, MRP partners and stakeholders many times throughout the process. A project work plan, incorporating the detailed evaluation design and methods, was prepared at the beginning of the evaluation project and this guided the work of the team. MRP ACHIEVEMENTS TO DATE In the three years of operation of the MRP, there have been significant achievements at the project level, as documented in MPAG and IMACS Annual and quarterly reports. Overall, most project activities are either on schedule or ahead of schedule with the MPAG component having achieved more than 70% of its overall planned results up to 2012, when measured against work plans and the MPAG PMP. Likewise, IMACS has achieved over 70% of its planned results as at the end of 2012 with activities such as training being ahead of schedule when measured against Performance Indicators. Some of these project-level achievements (e.g. I-CATCH, MPA design) have been incorporated into MMAF processes and planning but assisting MMAF in developing an overall strategic framework for fisheries management and biodiversity conservation has not proceeded as quickly. These issues are further explored throughout this evaluation. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The over-arching recommendation for the evaluation is that during the period to 2014, IMACS and MPAG should concentrate on assisting MMAF, at both national and regional level, to develop a clear strategic, integrated action plan to implement its current vision and strategy for sustainable use and conservation of marine biodiversity and fisheries resources. This would provide a firm foundation for any future USAID assistance. This will require IMACS and, to a lesser extent, MPAG establishing more effective and sustainable working relationships with all levels (National, Provincial and District) of the Ministry of Marine Affairs and Fisheries (MMAF) and other relevant Government agencies. Within this context, the principal findings, conclusions and recommendations of the evaluation are as follows: Page | 8 Principal Findings and Conclusions1 Key Recommendations Level 1 Evaluation 1. IMACS and MPAG project activities have either achieved or on track to achieve planned results with both MPAG and IMACS having achieved more than 70% of their overall planned results to date. However, the effectiveness of the activities in building institutional capacity (e.g. in addressing IUU fishing) or achieving sustainable fisheries, concrete biodiversity outcomes or increased resilience to climate change is questionable 2. Policy actions, institutional relationships and partnering arrangements pursued by IMACS and MPAG in implementing project activities have generally been effective in achieving project outputs against targets. However, the way in which training needs are identified by IMACS needs attention so that training is better integrated with Government processes. Also, follow-up assessment of training effectiveness is insufficient. 3. IMACS has successfully developed a tool (I￾CATCH) to assess community vulnerability to climate changes and this tool has now been adopted by MMAF. However, the planned activities of I-CATCH are behind schedule for some components. 4. IMACS project-level activities are generally not well connected to Government agency processes at National (MMAF), Provincial or District (DKP) levels. MPAG activities are better connected to Government processes 5. IMACS’s and MPAG’s professional and administrative capacity is generally good. However, there is a skills gap in fisheries strategic planning and management and in implementing institutional change. 6. The small grant program is not well aligned to overall MRP or project-level objectives although it is now being administered efficiently. 7. IMACS has not applied a high level of corporate management skills in its dealings with MMAF.  IMACS should move to a secondment mode of technical assistance and institutional capacity building.  IMACS should take a greater role in developing and adopting a consistent approach for all MRP training based on the model that MPAG uses. This approach should ensure that training is (a) assessed regularly for its effectiveness in contributing to sustainable improvements in institutional capacity (b) aligned with broader MRP and MMAF strategic goals, (c) consistent with existing Government training processes and (d) undertaken co-operatively with MMAF  IMACS and MPAG should utilize the expertise in the IBRC to both inform their own project activities and to encourage better linkages between the IBRC and MMAF.  Budget needs to be allocated to funding the implementation of priority actions resulted from I￾CATCH exercises and IMACS needs to speed up the development of the outreach/communication program.  IMACS Regional Offices should be given greater autonomy to work with Provincial and District DKPs in integrating IMACs project activities into local Government planning and administrative processes. This autonomy, however, should be within the overall IMACS management, budgetary and strategic framework.  Consideration needs to be given to developing and implementing multi-year agreements with training implementation partners such as NOAA  IMACS should engage a senior-level person with fisheries strategic planning and management and institutional change experience to work with senior counterpart staff in MMAF..  Stop the further issue of Small Grant RFAs as well as the commitment of further Small Grant funds and use the unspent funds to implement activities more clearly focused on MRP Objectives.  IMACS and also MPAG should review their corporate governance procedures and institute changes that demonstrate best practice and provides the ‘paper-trail’ that is necessary for guiding Government action, particularly on outcomes of high-level meetings.. Level 2 Evaluation 8. The MRP is generally achieving satisfactory results against targets in (a) Strengthening coastal community resilience and climate change adaptation and (b) Creating and effectively managing Marine Protected Areas. However, results for (a) Strengthening MMAF through institutional development, (b) Improving sustainable fisheries management and (c) Improving capacity to reduce Illegal, Unreported and Unregulated (IUU) Fishing are less  The integration between the sustainable fisheries component of IMACS and MPAG marine conservation areas should be strengthened, particularly as it relates to EAFM and climate change adaptation  IMACS and MPAG need to better aligned their activities with the planning and budgeting cycle within local government system, especially when considering the timing when project results should be proposed/incorporated into formal proposition to DKPs.  IMACS needs to speed up development of the 1 Note that some recommendations shown in the body of the text have been consolidated for this summary Page | 9 Principal Findings and Conclusions1 Key Recommendations satisfactory. outreach/communication program with clear key messages, target groups, and specific means/conduit for each group and message. It is important to obtain support and encouragement not only from direct beneficiaries of targeted activities, but also from the general public. Level 3 Evaluation 9. The MRP is not achieving its overall strategic objectives, partly because those strategic objectives are not clearly articulated and partly because the MRP strategy is ineffective in producing sustainable improvements to ecosystems and institutions 10. There is little or no strategic integration between MRP activities although some limited operational integration occurs.  Reporting by IMACS and MPAG against the KRAs of the MRP Results Framework should be requested by USAID. This will allow assessment of how IMACS and MPAG activities are contributing to higher level MRP objectives and key result areas.  IMACS should be required by USAID to adopt better institutional assessment and capacity building strategies that are consistent with those adopted by MPAG. It is particularly important that such a consistent approach is adopted for training activities.  Implement other Level 1 and Level 2 evaluation recommendations, particularly those related to improving the effectiveness of service delivery and the institutionalization of MRP component activities.  IMACS should collaborate with the Coral Triangle Initiative of Coral Reefs, Fisheries and Food Security (CTI-CFF) through Indonesia’s CTI National Coordinating Committee in MMAF and actively participate in its working groups 11. Recommendations related to Planning for long term outcomes  USAID-I should initiate activities in preparation for the implementation of the Host Country System (HCS), or foreign grant management system  In any future assistance, measurement of MRP broad performance indicators related to better marine biodiversity outcomes and achievement of sustainable fisheries, in addition to the USAID standard indicators, should be incorporated into project designs PLANNING FOR LONG-TERM OUTCOMES Overall, the evaluation recommended that the period to September 2014 should be used by IMACS and, to a lesser extent, MPAG to establish closer relationships with national and local Government institutions so as to lay the foundations for effective assistance, under the USAID 2014-19 Country Strategy, in improving marine resources management in Indonesia. The areas where that closer relationship is needed are:  Supporting and guiding the Government’s vision for marine resources management and development so that a shared, long-term vision and strategy results.  Better integrating IMACS’s (and to a lesser extent, MPAG’s) activities into MMAF and local Government planning and budgetary processes  Adjusting project activities, such as training, in ways that better link to already existing Government procedures Page | 10 This report makes recommendations as to the mechanisms that can be used to achieve this better linkage and integration with MMAF and local Government, the most important of which are:  Having key technical project staff seconded to MMAF (and in appropriate circumstances, to local Government) to work alongside counterparts.  Better integrating IMACS and MPAG activities so that a consistent approach to delivering MRP outputs is achieved that are aligned to existing Government processes. This would assist the MRP partners in better supporting MMAF.  Development and reporting against a Results Framework for the MRP that clearly demonstrates the high level linkages to MMAF’s vision and strategy. Future assistance will also need to engage a broader range of Government and non￾Government stakeholders in implementing a national vision and strategy for marine resources management. Also, in assisting MMAF and the Government more broadly in developing and gaining support for such a vision, greater and more effective engagement with the public and NGOs through public relations and media activities will be needed to highlight the benefits of sustainable marine resources management. Working in this way both from a bottom-up as well as a top-down perspective with stakeholders is more likely to be effective in achieving a shared national vision. Any longer term assistance should also address the issue of the current lack of any measurement, assessment or monitoring of achievements towards meeting overall MRP strategic objectives of biodiversity conservation and sustainable fisheries management as an integrated process. Page | 11 1. INTRODUCTION 1.1 Background The Marine Resources Program (MRP) is the marine portfolio of the USAID mission in Indonesia. The program was designed to support the Government of Indonesia (GOI) in its commitment to promote sustainable fisheries, marine conservation and marine protected areas, as reflected in its recognition of the Coral Triangle Initiative (CTI). The MRP, which was an outcome of USAID-Indonesia’s analysis for its 2009-2014 strategy (including Tropical Forestry and Biodiversity Assessment, FAA 118/119), therefore recognizes the importance of the protection of marine biodiversity in contributing to Indonesia’s economy and social welfare and was therefore designed to support and strengthen the capacity of the MMAF in improving the management of marine resources and biodiversity. The MRP includes five components which support the Objectives of the MRP (see Section 1.3) and five awards that are detailed below. The five components of MRP are: 1. Strengthen MMAF through institutional development (ID); 2. Improve sustainable fisheries management (SFM); 3. Strengthen coastal community resilience and climate change adaptation (CCR); 4. Create and effectively manage Marine Protected Areas (MPA); and, 5. Improve capacity to reduce Illegal, Unreported and Unregulated Fishing (IUU). These components support the Coral Triangle Initiative (CTI) Indonesian draft National Plan of Action (NPOA), which defines Indonesia’s plans to preserve marine biodiversity. They are also key elements in the MMAF Strategic Plan 2010-2014 (RENSTRA), that was completed in August 2012 and which focuses on competitiveness, value adding and community welfare2 . The MRP is implemented through 5 sources:  A contract, the Indonesia Marine and Climate Support (IMACS) project led by Chemonics International with subcontractors of PNCI, Coastal Resources Center (CRC) of the University of Rhode Island, and RARE3 . This began in December 2010. It is due for completion in September 2014. IMACS is designed to provide primary support for three MRP components (ID, SFM and CCR) while secondarily supporting the other components and providing program integration and coordination for MRP to USAID and MMAF  A NGO Consortium led by WWF-US with implementing partners of WWF-I, CI, TNC, CTC and WCS, was the first assistance mechanism to start in December 2009 2 The vision and objectives of RENSTRA have changed during its lifetime 3 The RARE subcontract was subsequently terminated with IMACS taking over direct responsibility for communications activities Page | 12 with a focus on MPA system development. The work of the consortium in Indonesia was originally funded as a component under the Regional Coral Triangle Support Partnership as CTSP-Indonesia for 2010 and 2011, and then shifted to a direct USAID/Indonesia bilateral cooperative agreement called the Marine Protected Area Governance (MPAG) project for 2012-2014.  An interagency agreement with the U.S. National Oceanic and Atmospheric Administration (NOAA) was started at the same time as the NGO MPA assistance mechanism to provide NOAA training expertise. Initially NOAA worked with MMAF and the NGOs on MPA training (2010-2012), then also with MMAF and IMACS (2012-2013) on other training for sustainable fisheries data collection (on￾board observers), ecosystem approach to fisheries management (EAFM), observer program and port state measures (IUU fishing).  An interagency agreement with the U.S. Department of Justice, International Criminal Investigative Training Assistance Program (ICITAP) was implemented during 2010-2011 to provide technical assistance to build the capacity of Indonesian law enforcement organizations to engage and prosecute environmental criminal activities to protect forest and marine ecosystems. A small portion of this project provided limited training to law enforcement personnel dealing with IUU fishing.  A university partnership (UP) program was added to the MRP in 2010 through USAID higher educational funding using the USAID Annual Program Statement (APS) for “Supporting Universities to Partner Across the Pacific.” The first partnership was with the University of California of Los Angeles (UCLA) and Udayana University (UNUD), Diponegoro University (UNDIP) and Papua University (UNIPA) to establish an Indonesia Biodiversity Research Center to advance research and studies in marine biological resource conservation. The second partnership in 2011 was with the University of California of Santa Cruz (UCSC) and Hasanuddin University (UNHAS), Diponegoro University (UNDIP) and the Eijkman Institute to establish an Indonesia Marine Biotechnology Partnership. A 2 year extension of the UCLA UP was funded through Biodiversity funding. These partnerships added $1.2 million to overall USAID marine sector funding, plus another 30% of this amount in cost share contributions from the universities. 1.2 Importance of Marine Fisheries and Biodiversity Conservation in Indonesia Policy Alternatives In managing its natural marine resources, Indonesia has a range of policy choices it can make, each resulting in sustainable production but with different impacts on profitability of the sector and different levels of risk to fish resources and to marine biodiversity. The current situation (see Management section below) is that fishing capacity in Indonesia is not effectively managed and, as a result, the fishing industry is, as a whole, operating at a point Page | 13 where there is near-to-zero overall industry profit (or economic rent) being generated. This point is also the point where risks to the sustainability of fish stocks are significant since it is also a point where overall fish biomass is low. If fishing capacity were better managed, there would be significant economic and conservation benefits. For example, if fisheries were managed at a point of Maximum Economic Yield, MEY (i.e. the point where economic rent, or industry profitability is maximized), this is also the point where fish biomass would be much higher than at present and therefore the risks to the sustainability of fish stocks would be significantly reduced from what they are at the present. The following Figure 1 demonstrates the alternative policies: Figure 1: The relationship between the value of catch (the same as volume of the catch since price is assumed constant), the costs of taking that catch, the resulting net industry profit, or economic rent and fishing effort (e.g. number of vessels). Indonesia’s fisheries are currently operating at the point indicated whereas significant economic and conservation benefits would accrue if a policy of operating at the point of MEY was adopted and achieved. The potential economic gains achievable by moving from the current position to a point of MEY have been estimated at $25 billion per year (see Management, below) Production, exports and food security Marine fisheries and resources are of vital importance to Indonesia, an Archipelagic state with in excess of 14,000 islands, and are also important within a regional and international context. Indonesia is the 3rd largest fish producer in the world, ranking behind only China and Peru and, according to the Food and Agriculture Organization (FAO 2012) the sector produced 6.7 million tons of fish in 2011. Current position Page | 14 Fisheries production amounts to 13.8% of the country’s Agricultural GDP (World Fish Centre, 2011) with fish providing 53% of total animal protein to local communities, one of the highest dependency rates in the world. Women are highly involved in the fisheries sector, often in key local community roles of small business management, trading and processing. Some of the poorest, marginalized, and indigenous groups – who often lack secure land tenure – rely either directly or indirectly on fishing and marine resources for their livelihoods. In addition to its importance in food security, nutrition, and livelihoods, the sector also supports a large export industry with Indonesian fish exports amounting to $US3.5 billion in 2011 (MMAF 2012, FAO 2012). Of these exports, approximately 35% go to the United States, accounting for 16% of total US fish exports. The Indonesian fisheries sector is therefore not only of importance to Indonesia but also to the USA, a major trading partner. Management Effective management of fisheries in Indonesia faces many challenges (De Young, 2006)4 , resulting in the sector not contributing to the national economy to its full potential. In a comprehensive analysis of the growth potential of Indonesia’s economy to 2030 (McKinsey Global Institute, 2012), the fisheries sector was identified as one of the major potential growth areas for the country. The report estimated that $25 billion per annum could be added to GDP by 2030 if fishing capacity was better managed5 but also noted that the sector currently was characterized by low productivity6 and there were a large number of barriers to achieving this growth potential. These conclusions regarding the potential for the fisheries sector to dramatically improve its economic performance through better, sustainable, management is consistent with experiences elsewhere when moving from unregulated to sustainably managed fisheries. Despite the barriers to achieving sustainable, profitable fisheries, Indonesia has made some progress in developing policies and actions to manage fish stocks although much remains to be done. Effective Monitoring, Control and Surveillance (MCS) capabilities are a major challenge with Indonesia having only a limited capacity to conduct surveillance and law enforcement (APFIC, 2007). Indonesia is meeting its commitments to the International Convention on Biological Diversity (CBD) to establish marine protected areas (MPA) and currently has 15.7 million 4 The evaluation team saw evidence of overfishing, removal of undersized/ immature fish, and many fish in very poor condition/quality. Fishers and boat owners relate having to go further out to find fish, taking longer trips. The economic return on fisheries is low and many fish are being wasted due to poor handling conditions, or converted into low value products such as shredded fish or fertilizer. This results in a significant food safety issue. Also, endangered species were found at landing docks, including sharks. 5 This is the equivalent of adding approximately $1.75/lb to the value of the current catch or $29,000 per annum to the profitability of the average fishing vessel 6 Indonesian fisheries sector productivity was estimated, for example, to be about 30% of that in neighbouring Malaysia Page | 15 Ha of MPAs with plans to increase this to 20 million Ha by 2020. The Government has strategies to achieve this 2020 target and therefore the target appears to be feasible. At present, the Government is establishing tools and methods for effective management of the declared MPAs. Importantly, while the Government’s views on MPAs are that they should be part of marine resource management and not established simply as prohibited areas, this is not always the case in practice. MPAs are managed within a single Directorate General and, because of the ‘silo’ structure of MMAF, they are therefore in most cases, not integrated into broader social, economic and fisheries management activities within MMAF. Indonesia’s Vision and Strategy for marine resources development Indonesia has a clear vision and associated objectives, targets and policy direction for fisheries (MMAF, 2011 and updates) although the vision changes regularly in response, apparently, to political influences. Visions are often related to increased production (e.g. the 2011 vision is that Indonesia will become the largest producer of marine and fisheries products in the world by 2015) or increased competitiveness. There are a range of possible policy objectives for marine resources exploitation and Indonesia has, as a sovereign nation, the right to choose the vision and policy objectives that are suitable to its circumstances and that are consistent with its international obligations. The MRP has been designed to help MMAF to develop and communicate a clearer strategic vision and, therefore, as part of the work of the evaluation team, the extent to which this has occurred was investigated. The implications of the current, and alternate, visions and policy objectives for Indonesia’s do not, to the evaluation team, appear to be well recognized and more importantly, the current vision of focusing on production (either on increasing production or achieving Maximum Sustainable Yield) are high-risk strategies from a resource sustainability point of view. To achieve a more conservative, lower-risk policy objective such as MEY (that would also result in significant economic benefits to the sector and the country) requires data, broad Government policy commitment, effective monitoring, control and surveillance activities and most importantly a level of management, particularly of fishing capacity, which currently does not exist within MMAF. While effective fisheries management is difficult in any country (and particularly in a country as large and as diverse as Indonesia), the lack of these basic building blocks for effective fisheries management has repercussions, They include potential economic benefits not being realized and heightened risks to the sustainability of fish stocks and marine biodiversity. Page | 16 MRP’s strategy in assisting MMAF to achieve and develop its vision and strategy is based on a combination of improving MMAF’s management capacity as well as providing technical assistance. However, in addition to improved management capacity, there are other factors that are critical to achieving sustainable fisheries, including effective MPA management. An overall strategy for better integration of the various aspects of marine resources management (e.g. MPAs, climate change adaptation, sustainable use of fisheries resources etc) needs to be adopted and shared by all partners so that management capacity is strengthened in the right areas. MMAF have developed a vision but no strategy for integrating MMAF functions to achieve that vision is evident. In addition, the MMAF’s vision does not guide the activities of all MRP participants, particularly IMACS. Healthy and biodiverse ecosystems are critical to maintaining fisheries productivity, as well as providing opportunities for the growing coastal tourism trade. Geographically, Indonesia is located within the Coral Triangle Region, recognized as a global center of marine biodiversity. Indonesia’s president, Susilo Bambang Yudhoyono, was the initiator of the Coral Triangle Initiative on Coral Reefs, Fisheries and Food Security (CTI-CFF) which recognizes the importance of sustainable management and conservation of these vital resources. However, there is much to be done in improving management of fisheries and marine biodiversity in Indonesia before that initiative achieves its objectives. USAID’s assistance in seeking to achieve sustainable marine fisheries and improved marine biodiversity protection in Indonesia are therefore addressing a sector that is of vital importance to Indonesia, as well as the United States from biodiversity conservation, economic, food security and trade perspective. However, given the number of issues facing fisheries and marine biodiversity management in Indonesia, USAID’s assistance is also necessarily strategic in complementing the work of MMAF and others. 1.3 USAID Assistance USAID’s Marine Resources Program (MRP) is designed to support USAID Indonesia’s Country Program Strategy 2009 – 2014, Objective 2: “Sustainable Management of Natural Resources”, Intermediate Result 2 “Improved Management of Marine Ecosystems”. The MRP has two objectives7 : 1. Restore and enhance ecosystem productivity, biodiversity and resilience for food and economic security; and 2. Increase the resilience of natural ecosystems and coastal communities to adapt to climate change and reduce risks from disasters The Ministry of Marine Affairs and Fisheries (MMAF) is a strong partner of USAID. MRP is designed to support Government of Indonesian commitment to the CTI-CFF to manage and conserve critical resources. Of the five CTI-CFF goals, USAID priorities are aligned to 7 The wording of the objectives of MRP differs slightly in various documents. These objectives have been taken from the fully executed IMACS contract (AID-EPP-I-00-06-00013 / AID-497-TO-11-00003), December 2010 Page | 17 support the goals of (a) an ecosystem approach to fisheries management, (b) marine protected areas (c) climate change adaptation. In late 2008 and early 2009, USAID conducted analyses to guide its 2009-14 strategy. Principally, these were (1) the Tropical Forestry and Biodiversity Assessment to identify principal threats and priority actions to be taken for forestry and marine biodiversity and (2) a study on Enhancing Government Effectiveness with a particular focus on an analysis of the needs of MMAF to address key issues related to its responsibility for managing fisheries and coastal areas8 . These studies formed the foundations of the subsequent development of the MRP. The funding for the MRP comes primarily from USAID biodiversity funding (approximately 75 %) and as such the activities of IMACS and MPAG are focused on reducing key threats to marine biodiversity in Indonesia. In addition, a significant percentage (approximately 25%) of funding comes from USAID climate change adaptation funds and is used to increase the resilience of natural ecosystems and coastal communities to adapt to climate change (CC) and reduce risks from disasters. In relation to this, CC funds and biodiversity funds are often used when and where an activity can achieve both biodiversity and CC objectives, with many such cases occurring as part of the small grants program. 1.4 Underlying Development Assumptions In 2005, the USA and other donors committed to the Paris Declaration for Aid Effectiveness and in 2008 to the Accra Agenda for Action to strengthen country ownership over development and build more effective and inclusive partnerships. Included in the Accra Agenda were: strengthening partner countries’ national development strategies and associated operational frameworks (e.g., planning, budget, and performance assessment frameworks), and increasing alignment of aid with partner countries’ priorities, systems and procedures and helping to strengthen their capacities. The MRP conforms to these commitments and is based on the premise that better management of Indonesia’s marine resources and marine ecosystems will result in long term sustainability of production from those resources (or at least a reduction in the risk of major fish stock declines) and greater protection to the marine ecosystems whose processes underpin that production. This appears to the Evaluation Team, based on experience in other countries, to be a reasonable assumption. Also, in addressing risks to future production and to fisher’s welfare, climate change impacts and adaptive capacity of fishers and communities to those impacts were identified as significant issues to be addressed. 8 This was based on USAID’s commissioned report Enhancing Government Effectiveness (EGE) in Indonesia: A study of the Ministry of Marine Affairs and Fisheries (Feb. 2009). Priorities identified in that report included improved policy formulation based on effective data management, decentralized operations, integrated planning, a public outreach campaign to engage the support of the Indonesian people, and improved extension services. Page | 18 To achieve better management, the central development hypothesis is that if the management capacity of the government (national MMAF and local) is strengthened, this will lead to sustainable fisheries management and effective management of MPAs. MRP’s overall strategy is to strengthen the capacity of MMAF and provide technical support for key activities that support marine resources and coastal communities. This strategy has been interpreted in different ways by MRP participants with IMACS taking a narrow focus and concentrating on training to achieve capacity building while MPAG takes a broader approach and includes training and institutional improvement. However it is achieved, the implied assumption in MRP’s strategy and activities is that strengthening government institutional management capacity will alone result in sustainable fisheries management and effective management of MPAs. This, to the Evaluation Team, is a problematic assumption. In addition to improved management capacity, there are other factors that are critical to achieving sustainable fisheries, including effective MPA management. First, as noted in Section 1.2, an overall vision and clear objectives for better integration of the various aspects of marine resources management needs to be adopted and shared by all partners so that management capacity is strengthened in the right areas. Secondly, the enabling environment must be strengthened9 . Barriers to achieving sustainable management of fisheries include the difficulty of managing fishing capacity and access to fisheries, co-management policy/regulations, secure access rights to marine resources, the effectiveness of the decentralization process and a lack of reliable data on the sector etc. The MRP assistance and any future activities therefore need to be seen within this broader development context and the inherent assumptions. 2. DESCRIPTION OF THE MARINE RESOURCES PROGRAM (MRP) The MRP is the overall name of the marine portfolio of the USAID mission in Indonesia. The program, which was designed to support the Government of Indonesia (GOI) in its commitment to improve the management of marine resources and biodiversity, includes five components which support the Objectives of the MRP (see Section 1.3) and five awards that are detailed below. The five components of MRP are: 1. Strengthen MMAF through institutional development (ID); 2. Improve sustainable fisheries management (SFM); 3. Strengthen coastal community resilience and climate change adaptation (CCR); 4. Create and effectively manage Marine Protected Areas (MPA); and, 5. Improve capacity to reduce Illegal, Unreported and Unregulated Fishing (IUU). 9 The enabling environment for transitioning to sustainable fisheries management in Indonesia is weakly developed and McKinsey Global Institute (2012) identified that there were more barriers to growth of the fisheries sector in Indonesia than existed in other sectors Page | 19 These components support the Coral Triangle Initiative (CTI) Indonesian draft National Plan of Action (NPOA), which defines Indonesia’s plans to preserve marine biodiversity. They are also key elements in the MMAF Strategic Plan 2010-2014 (RENSTRA), that was completed in August 2012 and which focuses on competitiveness, value adding and community welfare10 . USAID authorized the Marine Resources Program (MRP) at an estimated funding level of $31,900,000 to support a five-year program of assistance (2010–2014). Implementation arrangements are provided in Section 1. As an initial activity of MRP, USAID supported the CTSP (a coalition of NGOs) to conduct an analysis - in partnership with Indonesian NGO, universities and government officials - of the priority geographic areas for conserving Indonesian’s marine biodiversity and this was followed by the other components shown above. The structure of MRP is as follows: 3. EVALUATION PURPOSE & QUESTIONS The purpose of the performance evaluation is to provide USAID/Indonesia (USAID) and the Ministry of Marine Affairs and Fisheries (MMAF) with an independent review of progress to date of assistance provided by USAID to MMAF under the Marine Resources Program (MRP) and to guide the development of innovations and adaptation of USAID’s assistance in a dynamic environment. 10 The vision and objectives of RENSTRA have changed during its lifetime Page | 20 The evaluation will be used in the short-term by USAID and MMAF to modify ongoing assistance, and in the longer-term to plan future assistance strategy as part of USAID Indonesia’s Country Strategy 2014-2019 and MMAF’s Strategic Plan. To achieve the Evaluation purpose, and in accordance with the Scope of Work (SOW), the evaluation team adopted methods to capture sufficient and accurate information for conducting a performance and developmental evaluation at three levels; individual project level (outputs achieved), MRP component level (results achieved) and overall MRP level (strategic approaches). At each level, Guidance for developing questions to inform the Evaluation followed those outlined in the Scope of Work (SOW). However, additional questions were framed to address issues that the Evaluation Mission identified as important. The MRP evaluation was conducted at three levels:  Level 1 – individual projects (outputs achieved)  Level 2 – MRP component (results achieved) and  Level 3 – overall MRP (strategic approaches) The evaluation of individual projects was focused on the progress of the two main implementing mechanisms that USAID used to provide technical assistance under the MRP – the IMACS and the MPAG projects. NOAA provided training to MMAF working cooperatively with IMACS and with MPAG for sustainable fisheries and MPA management, respectively. These joint activities are covered under the IMACS and MPAG components of this evaluation. The University Partnerships (UP) has recently been evaluated separately11 and, as a result the UP activities were not a focus of this evaluation. However, the UP evaluation report was used as an input and as background for the team’s work. The ICITAP component was addressed through discussions with relevant MMAF recipient agencies but was not further followed up given the length of time that had elapsed since training. The Key Questions therefore posed during the Evaluation were: 1. What has been the effectiveness of the IMACS and MPAG projects’ activities in achieving planned results in accordance with Work Plans and Performance Management Plans, including the extent to which projects have produced concrete biodiversity conservation outcomes and activities that comply with USAID’s biodiversity earmark criteria (Level 1 evaluation), 2. What is the capacity of the IMACS and MPAG project teams to implement activities in an efficient and timely manner, including utilization of grants, subcontracts, and other resources? (Level 1 evaluation), 3. What were the important policy actions, institutional relationships, and partnering arrangements that facilitated project success? (Level 1 evaluation), 4. Is the MRP achieving satisfactory results against targets in the following component areas: (a) Strengthening MMAF through institutional development, (b) Improving 11 Evaluation of the University Partnerships Program: Phase One – Partnerships #1 and #2, International Business & Technical Consultants and JBS International, USAID/Indonesia, November 13, 2012. Page | 21 sustainable fisheries management, (c) Strengthening coastal community resilience and climate change adaptation, (d) Creating and effectively managing Marine Protected Areas, and (e) Improving capacity to reduce Illegal, Unreported and Unregulated Fishing? (Level 2 Evaluation) 5. Is the MRP achieving its overall objective of sustaining the long-term welfare of coastal Indonesians by reducing the destruction of marine resources and preparing them for climate change impacts and is the MRP strategy of strengthening the capacity of MMAF and providing targeted technical assistance, producing sustainable improvements to ecosystems and institutions? (level 3 evaluation) In addressing these 5 primary questions, the SOW (see Annex 1) also suggested a number of contributing, or subsidiary questions that could be used to fully answer the primary questions. All of these subsidiary questions were used to guide Key Informant Interviews (KIIs) discussions and to focus the KIIs on achieving complete answers to the five key questions. Details of the approach to and the questions posed at each Level of evaluation are provided in Annex 2. Reporting against these questions for each MRP component is shown in Annex 6. 4. EVALUATION METHODOLOGY In accordance with USAID’s Evaluation Policy (January 2011), this evaluation adopts terminology that is consistent with that policy and, so far as practical, is evidence-based. The details of the methodology are found both in this section and also in the relevant Annexes. The MRP has not previously been subject to any previous Evaluation although one operational component, the Universities Partnership (UP) activities has been evaluated12 in late 2012. This Evaluation Report was used to inform the current work. The major components of the MRP of IMACS and MPAG have Project Performance Plans (PMP) and current and previous annual Work Plans and Reports in place and these were used as source documents for the current Evaluation. In accordance with the SOW, the evaluation team adopted methods to capture sufficient and accurate information for conducting a performance and developmental evaluation at three levels, viz. Individual project level (outputs achieved), MRP component level (results achieved) and overall MRP level (strategic approaches). The team’s methodology was highly participatory, collaborative and open, and engaged USAID staff, MMAF staff, local Government agencies, MRP partners and stakeholders many times throughout the process. The team was gratified by the quality and quantity of engagement of both USAID and their implementing partners throughout the process. 12 Evaluation of the University Partnerships Program: Phase One – Partnerships #1 and #2, International Business & Technical Consultants and JBS International, USAID/Indonesia, November 13, 2012. Page | 22 The evaluation design and methodology used, and which is documented in the approved Work Plan (February 2013) was, so far as possible, evidenced-based although primary data￾collection activities were limited by the length of the evaluation mission. This, for example, precluded any independent data collection or activities such as focus groups. The evaluation therefore adopted a three-pronged approach of:  Documentation and literature review  One-on-one and group key Informant Interviews (KIIs) in both Jakarta and in project sites at various locations in Indonesia (see Annex 2 – Travelling and Meeting Schedule)  Hypothesis-Testing (HT) approaches through follow-up interviews. Details of the data collection tools and processes used are included in Annex 3. Gender-specific data was collected, analyzed and reported wherever possible although the availability of such data was very limited. 4.1 Findings: Data Collection and Analysis Document and Literature Review The evaluation team examined a wide range of documents supplied by USAID (in excess of 600 documents) as well as documents identified and sourced by the Team as important as the evaluation mission proceeded. A listing of these documents is included in Annex 4. Especially relevant to the Team’s work were the various work plans, PMPs, annual and quarterly reports and other technical documents produced by IMACS and MPAG since early 2010. In addition, MMAF documents over this period were examined to determine the extent of alignment with the activities of the MRP program. However, the team was cautious in not relying too heavily on dated secondary written sources when more recent evidence-based primary information was available. In-depth Key Informant Interviews (KIIs) KIIs were conducted both in Jakarta and at project sites (South East Sulawesi Province, West Nusa Tenggara Province and East Nusa Tenggara Province). In order to ensure maximum advantage from meetings, the team contacted potential interviewees prior to the interview through e- mail or text or phone. IMACS’s regional offices and MPAG provided logistical support and assisted in arranging interviews when necessary. Since the team includes Indonesian professionals, there was generally no requirement for translation at interviews. The targeted KII participants include past and present principal stakeholders who have or had key roles in the MRP. In addition, direct beneficiaries were interviewed at project sites as well as in Jakarta in addition to indirect stakeholders, such as other donors. Page | 23 The evaluation team conducted semi-structured key informant interviews (KIIs) with all relevant MRP participants, including all key project partners. This included KIIs at project sites in South East Sulawesi Province (Kendari, Wakatobi, Muna, and Bau-Bau), West Nusa Tenggara (East Lombok, Central Lombok and North Lombok), and East Nusa Tenggara (Kupang). There was insufficient time to visit all regional project sites and therefore site visits were prioritized on the basis of (a) the scale of the activities being undertaken and (b) the integrated activities being undertaken by IMACS and MPAG at some sites (South East Sulawesi and West Nusa Tenggara Province). KIIs also included telephone/conference call interviews with NOAA staff based in both Washington and Hawaii. In addition, MRP participants from URI-CRC and UCLA were interviewed while they were in Jakarta. The interview instruments used were the development of key questions, specific to the target group but consistent with the broad questions outlined in section 3 (above). These questions were posed to senior managers and administrators as well as to the full range of stakeholders. In addition, follow-up interviews were undertaken as necessary to cross-check the accuracy of the details, particularly where different views of events, procedures or activities were held by different parties. Responses were collated, summarized and analyzed (see Section 5 and Annex 2 for the analytical methodology) and used to develop conclusions and recommendations. Data analysis Analysis of the data from background documents concentrated on the PMP, work plans, and quarterly and annual reports of the IMACS and MPAG components of the MRP. At the Level 1 evaluation stage, these documents were used to assess achievements by mapping actual outputs against planned outputs. In addition, outputs were mapped against the 5 key evaluation questions to determine the extent to which progress in each project component was being achieved. Matrices were then prepared summarizing these results and are included in Annex 6. The analysis of the KII information was undertaken by collating responses and mapping them according to broad themes related to the 5 key evaluation questions which covered all three levels of the evaluation. This approach allowed the quantitative as well as qualitative analysis of responses (as numbers responding) and allowed the evaluation team to identify trends between and within respondent groups as well as disparities. These analyses were then used to develop verifiable conclusions from the findings. Details of the methodology for data analysis are provided in Annex 2 Page | 24 4.2 Conclusions and Recommendations Using the Findings that were developed through the data collection and analytical processes outlined in Section 4.1 (above), preliminary conclusions (or hypotheses) were developed which were consistent with, and explained, the findings. These hypotheses were then tested by re-interviewing relevant MRP stakeholders that were selected based only on the interviewee’s relevance to the hypotheses posed. Interview methodology, data collection protocols and analysis was the same as those for the initial KIIs. However, specific interview tools were developed (as a series of focused questions) to address the hypotheses developed as part of this method. The hypotheses and the details of the focused questions are included in Annex 2 Once validated (or amended) through this process, the hypotheses became the Conclusions of the Evaluation which were then used to develop Recommendations that were based on those Conclusions. 4.3 Limitations The primary limitation to the evaluation methodology was one of time and the geographic extent of the MRP’s activities. As a result, there was insufficient time to undertake activities such as (a) establishing focus groups or other group activities as a means of data collection (b) implementing formal surveys of the broader MRP recipient beneficiaries (c) comparing sustainable fisheries and biodiversity outcomes in areas that have been and have not been the subject of MRP activities (i.e. controlled experiments) and (d) visiting all project sites, such as those in Papua or Anambas and (e) investigating the details of individual small grants processed by the project. However, the Evaluation team did not believe that these limitations significantly influenced the results of the Evaluation since the data collection techniques utilized covered a wide range project participants and beneficiaries. A summary of the evaluation design and methodology is shown in Annex 5. 5. Findings, Conclusions and Recommendations A logical progression from verifiable Findings supporting Conclusions that lead to Recommendations was followed as part of the Evaluation procedures with this process being described in Section 4. This section is arranged so as to preserve this logical structure and therefore Findings, consequent Conclusions and Recommendations are presented together. 5.1 Over-arching Recommendation Before presenting the findings, conclusions and recommendations of the evaluation at various levels of the MRP, it is worthwhile articulating the views of the evaluation team as to Page | 25 an overall strategy for the MRP, given both the current lack of clarity of such a strategy (see below) and the impending conclusion of both major MRP components, IMACS in September 2014 and MPAG also in 2014. The necessity for this over-arching foundation and strategy became clear during the progress of the team’s work and, in particular, was a result of the dual focus of the evaluation of (a) addressing both past and planned activities for the MRP and (b) addressing longer term strategic interventions that might contribute to USAID-Indonesia’s Country Strategy for 2014-19. OVER-ARCHING RECOMMENDATION It is recommended that during the period to 2014, IMACS and MPAG should concentrate on assisting MMAF, at both national and regional level, to develop a clear strategic, integrated action plan to implement its current vision and strategy for sustainable use and conservation of marine fisheries and resources. This would provide a firm foundation for any future USAID assistance. This will require IMACS and, to a lesser extent, MPAG establishing more effective and sustainable working relationship with all levels (National, Provincial and District) of the Ministry of Marine Affairs and Fisheries (MMAF) and other relevant Government agencies. Currently, the MMAF have a clear vision for marine resources development in Indonesia and have a strategy for achieving this vision. However, the vision has been subject to change and the various activities that are needed to achieve the vision are not well integrated within MMAF13 . It is therefore difficult for IMACS in particular to align its vision with that of MMAF. However, assisting, sharing and guiding MMAF’s vision and providing the strategic support for implementing that vision is critical in building MMAF capacity and trust between MMAF and IMACS. Concentrating only on institutional strengthening will not by itself result in the achievement of the MRP strategic goals of sustainable fisheries, improved marine biodiversity conservation and improved resilience of coastal communities to climate change. Without an implementation plan for its vision and an improved enabling environment, any institutional strengthening of MMAF will lack direction and operational capability to achieve the MRP/MMAF strategic goals. If no progress is made in aligning IMAC’s, and to a lesser extent, MPAG’s activities to support, guide and contribute to MMAF’s vision and to improving the enabling environment, many of these institutional strengthening activities are unlikely to be sustained, with the exception of MPA development and management. There will therefore be no broad platform upon which future assistance to MMAF can be based. 13 This is a result of the various Directorate-Generals within MMAF operating in a ‘silo’ mode as noted in the 2009 MMAF Governance Effectiveness Report (USAID, 2009) Page | 26 5.2 Level 1 Evaluation 5.2.1 Key Question 1 What has been the effectiveness of the IMACS and MPAG projects’ activities in achieving planned results in accordance with Work Plans and Performance Management Plans, including the extent to which projects have produced concrete biodiversity conservation outcomes and activities that comply with USAID’s biodiversity earmark criteria CONCLUSION 1 IMACS and MPAG project activities have either achieved or on track to achieve planned results as measured by Performance Indicators. However, the effectiveness of the activities in achieving sustainable fisheries, concrete biodiversity outcomes or increased resilience to climate change is questionable14 . This will be further explored under sec. 5.3.1 (Key Question 4) and 5.4.1 (Key Question 5) and is elaborated further for the MPAG component under Conclusion 2. KEY FINDINGS SUPPORTING THE CONCLUSION  Comparisons between PMP documents and annual and quarterly reports of IMACS and MPAG showed that most project activities were on track and, where they were not on track, remedial action was being taken to correct the problem. Detailed analysis of the degree of achievement of IMACS and MPAG activities, by project component are shown in Annex 6-2. CONCLUSION 2 MPAG has achieved more than 70% of its overall planned results up to 2012, when measured against work plans and the MPAG PMP. This includes 70% of component 1 (establishment of a sustainable MPA national system, 50% of component 2 (advance MPA management effectiveness in selected national district MPAs and MPA networks) and 100% of component 3 (host country system assessment and design). Management effectiveness programs are in place that incorporate USAID standard indicators as well as project-specific indicators. However, there are no indicators or supporting programs in place to measure biodiversity outcomes of that management. Therefore the extent to which concrete biodiversity outcomes are being achieved by project activities is not able to be assessed. Like the I-CATCH initiative (see below), some of MPAG’s activities have been adopted and integrated into MMAF processes. For example, MPAG has successfully establishing a 14 While the MPAG component meets most of USAID’s Biodiversity funding requirements (referred to as the Biodiversity Code), including MPA site selection, identification of threats etc, the method of analysis of threats varied between sites as a result of different approaches by NGOs Page | 27 management tool to evaluate the effectiveness of MPA management (E-KKP3K) and this tool has been officially adopted by MMAF. FINDINGS SUPPORTING THE CONCLUSION Annex 6 provides the performance details against project-level performance indicators for both the MPAG and IMACS activities. The information in Annex 6 is based on data derived from the PMP, Work Plans and quarterly and annual reports of the component project. The detailed reporting in Annex 6 also shows that achievements against performance indicators varied between sites and also between the sub-components. KIIs undertaken at both project sites in Kupang, Wakatobi and Lombok and in Jakarta with MPAG staff, NGO participants, beneficiaries, MMAF staff and local Government staff confirmed the analysis undertaken of the project documents. Integration of some aspects of MPAG’s activities into Government policies and processes was confirmed by reference to a Decree by the Director-General of KP3K, no. 44, 2012. CONCLUSION 3 IMACS project-level activities are generally not strategically well connected to Government agency processes at National (MMAF), Provincial or District (DKP) levels, thereby limiting their effectiveness and sustainability. This results in project activities not contributing to the desired capacity building of the MMAF as an organization or its staff. The same conclusion can be drawn for the strategy, operations and outcomes at Provincial and District levels through the DKPs. MPAG, with single focus on marine protected areas issues has had much success in developing a coordinated plan and activities with the appropriate technical directorate in MMAF (Directorate of Area and Species Conservation) including a specific human resources development plan. This is a significant issue for the IMACS project activities of Institutional development and sustainable fisheries management but less so for the climate change resilience project activities. KEY FINDINGS SUPPORTING THE CONCLUSIONS The evaluation team identified a number of findings that contributed to this conclusion. At the national level (not in any order of priority), these are:  Documentation review and KIIs with IMACS, MPAG and MMAF staff show a lack of synchronization between the IMACS planning processes (which operates on an Oct￾Sep year) and the MMAF processes, which operate on a Jan-Dec year. This results in difficulties in integrating IMACS activities into MMAF planning and budgetary processes. For example, IMACS training activities are not synchronized with MMAF processes, resulting in budget pressures for MMAF operating units in providing staff, financing and facilities for IMACS training activities. Also, the priority action plans Page | 28 from I-CATCH are not submitted at the local level in time so they can be incorporated into local Government plans and budget processes. USAID-I is aware of this issue and has directed IMACS and MPAG to develop a 15 months plan, instead of the usual 12 month plan, to address the problem. However, these efforts are still unable to fully adjust the timing of project’s activities to coincide with MMAF and Provincial/Districts/Municipalities planning cycle and implementing actions.  IMACS has successfully developed a tool (I-CATCH) to assess vulnerability to climate changes at the community level and this tool has now been adopted by MMAF as the program Desa Pesisir Tangguh (see also Conclusion 3).  IMACS work plans and annual and quarterly reports as well as KIIs with MMAF staff at Director and Director-General level show that, apart from formal training, technical-level interaction between MMAF staff and IMACS staff is minimal. For example, the Directorate of Fisheries Resources states that it has little interaction with the sustainable fisheries component of IMACS despite the obvious potential synergies. KIIs with MMAF staff of Director level supported this view with all (100%) suggesting that the top-down co-ordination arrangements within MMAF (under PUSKITA) contributed to this minimal interaction. Overall, this is a particular issue when MMAF staff (as well as Provincial and District DKP staff) are not involved in aligning Government priorities with IMACS activities.  MPAG work plans and annual and quarterly reports as well as KIIs with MMAF staff showed that MPAG has a much greater technical interaction with MMAF, particularly with the Directorate of Area and Species Conservation (KKJI) and the Human Resource Development Agency (BPSDM). This different relationship when compared with IMACS appears to be a result of the different counterpart arrangements between MMAF and MPAG. In particular, the history of previous arrangements through CTI-CFF, that included a good connection and communication between KKJI and members of the NGO consortium, has allowed MPAG to build on those already-established relationships.  As part of this better technical interaction, MPAG has facilitated and actively participated in the development of a working group (Team 11) which was established to develop a specific human resources development plan for marine protected areas management. IMACS has not yet developed such a plan to identify and assess capacity building needs to guide training and other activities in either EAFM or CCA within related technical directorates.  Communication mechanisms between IMACS and MMAF are ineffective for a number of reasons. IMACS relies on communications with high level (DGs and above) staff within MMAF by emails, semi-annual formal ‘co-ordination meetings’ and working through the MMAF International Co-ordination Unit (PUSKITA). However, information rarely filters down to technical staff and units within MMAF and direct personal contact between MMAF and IMACS staff is rare (only 1 of 8 MMAF staff of Director level and above, or 12% reported direct personal contact with IMACS staff). Direct contact between MMAF staff and MPAG staff is however, better developed (see previous two points) Page | 29  In late 2010, IMACS, USAID-I and MMAF looked at the idea of secondment of IMACS staff to MMAF and an office was made available within PUSKITA. However, while key IMACS and KKP counterparts were identified it appears as if no long term secondment of staff to operational Directorates in KKP was made although IMACS staff did work within Directorates of KKP for short periods. The reasons why this did not become a more permanent arrangement are not entirely clear but certainly a lack of clear engagement protocols played a major role.  MMAF capacity building activities conducted by IMACS are confined almost exclusively to formal training courses designed to upgrade individual staff skills. However, as noted in section 1.4, staff training is not sufficient by itself to result in strengthening of MMAF’s institutional capacity. In addition, (a) the way in which training priorities have been developed has varied during the course of the MRP with initial arrangements involving collaboration with MMAF but current arrangements being more driven by priorities identified by IMACS (KIIs with MMAF staff and training providers) (b) most training materials and training delivery was in English whereas not all of the participants, who are Indonesian, were comfortable in using the language. Using an interpreter during and throughout the courses was discouraged because many things were lost in translation and the process was time consuming (c) the effectiveness of the training courses in achieving sustainable upgrading of organizational capacity is reduced because of MMAF structural and policy issues15 such as staff rotation and (d) no follow-up assessment of training effectiveness is undertaken, with IMACS relying on the Government’s capacity assessment tool (MMAF internal management systems) to assess changes in institutional capacity (IMACS PMP). However, a KII with the Planning Bureau in the Secretariat General of the MMAF showed that the Government assessment tool is not designed to, and does not, measure changes in institutional capacity.  MMAF capacity building activities conducted by MPAG are better integrated with Government processes in that they are based on a competency-based training system that conforms to Indonesian Government requirements. Training priorities are identified through the competency-based training system in collaboration with the Directorate of Conservation for Area and Fish Species and the Human Resources Development Agency. Nevertheless, there is no evaluation mechanism to measure training effectiveness. Of importance in reaching this conclusion is the assumption (see also Section 1.4) that individual staff training is a necessary and sufficient activity to result in a strengthened MMAF capacity for sustainable fisheries management and biodiversity conservation. As elaborated in Section 1.4, this assumption is not supportable or warranted since other factors need to be considered in strengthening MMAF capacity, particularly the development of a strategic action plan to support the vision for marine resources management and protection, and the 15 Many of which were identified in the report Enhancing Government Effectiveness (EGE) in Indonesia: A study of the Ministry of Marine Affairs and Fisheries (Feb. 2009) Page | 30 development of the enabling environment to support operational reform. Staff training activities are therefore necessary but not sufficient. The conclusion is also a result of (a) a lack of an integrated approach to the development of training plans – each component of MRP develops and pursues its own training plans, thereby re-enforcing rather than helping to break down the MMAF ‘silo’ culture. Some operational co-ordination of training planning takes place but not strategic co-ordination (b) a lack of synchronization between the IMACS training planning processes (which operate on an Oct-Sep year) and the MMAF processes, which operate on a Jan-Dec year. Extending the planning period to 15 months (instead of 12) has still not been able to adjust the timing of project’s activities (not only training) so that they coincide with the MMAF and Provincial/Districts/Municipalities planning cycle and implementing actions (c) training not being linked to MMAF Human Resources Development Plans - training should be conducted through the Agency of Human Resources Development, but the needs and standard curricula should comes from appropriate Directorate Generals (d) the continuing MMAF policy of staff rotation which results in the loss of trained staff to other parts of the MMAF Comparisons between the recommendations made by the 2009 study16 on improving Government efficiency and effectiveness and the current activities and structure showed little or no change and that the recommendations had not been taken up. Information on current arrangements was gathered from the current MMAF organizational chart as well as from KIIs with 16 senior MMAF officials of Secretary General, Director-General, Director, and Deputy Director rank. A number of project-level issues were also evident at the provincial and district level that demonstrated the lack of process integration. Among these (again, not in order priority) are:  Lack of understanding at local level. Based on KIIs with local Government staff and managers in areas where IMACS and/or MPAG were active, all (100%) reported that they either unclear or totally ‘confused’ about what IMACS and MPAG were doing. It should be noted that in MPAG case, recognition of MPAG activities was confused with the NGOs who are receiving the funding and actively running the program (TNC/WWF for Wakatobi, WCS for Gili Matra and TNC for the Savu Sea). None of the interviewees reported any knowledge of the overall IMACS or MPAG structure or strategic goals and all stated that nobody had ever explained the programs to them. This is despite IMACS having conducted socialization workshop at the provincial level. 3 of 5 DKP Managers interviewed (60%) compared the IMACS engagement process unfavorably with previous experiences with donor agencies, such as Asian Development Bank (CO-Fish project). The other 2 of 5 DKP managers had no previous experience with other donor agencies and therefore had no opinion. Local Government staff and managers were, however, generally aware of the MPAG NGO implementing partners’ 16 USAID (Feb. 2009) Enhancing Government Effectiveness (EGE) in Indonesia: A study of the Ministry of Marine Affairs and Fisheries Page | 31 activities although did not link those activities to USAID and have little knowledge of MPAG as a program. Their knowledge came directly from the NGOs.  IMACS activities are not integrated into budgets or planning processes of DKPs although MPAG have made more progress in this regard. For example, MPAG activities in the Savu Sea to integrate the local government and USAID fiscal budgetary planning are proceeding well. A project coordination meeting is normally held in June or July for the forthcoming year program with the local government agencies proposing the budget for activities from January to March. These are then adapted by MPAG to support and integrate the related local government project at the Savu Sea. An evaluation meeting is then held in December for the project implementing year to assess progress. In the Savu Sea site, this integration process is facilitated by the P4KKP (province institution established by NTT Governor to support the establishment of the Marine National Park of Savu Sea). In East Lombok, IMACS has established an effective data collection and fisheries assessment process for tuna species by establishing a ‘Fisheries Data Collection Commission’ which includes, among others, representatives of DKP, a University and a private sector company. This initiative however is at risk and will probably cease when the IMACS project ends in September 2014 because there has been no assistance provided to the DKP on how to incorporate provisions for continuing the activity into its 2014 (and beyond) planning and budget process which it is currently developing. This lack of attention to the sustainability of the activity is of particular concern because this initiative is meant to be a ‘model’ or demonstration site for a process that is envisaged to be extended throughout the entire Fisheries Management Area (WPP 573) and later, all over Indonesia. The extent to which the activity can be used as a more general ‘model’ also remains unclear since it is confined to just the fishery for tuna species.  Although unconfirmed, it is understood that other local Government institutions such as BAPPEDA (local government planning agencies) also have had no contact with, or knowledge of, IMACS or MPAG activities. IMACS achieved important political support from provincial and district level leader by the signing of Co-operation Agreements between IMACS and one Provincial (South East Sulawesi Province) and 5 Districts/Municipalities in the province (examined by the Evaluation Team). Part of the agreement requires the establishment of ‘Monitoring and Evaluation Committees’. However, all districts/municipalities Fisheries Offices stated they have not been given any further direction or instructions and were unsure what they were supposed to be monitoring and evaluating. The provincial Fisheries Office had, however, established the required committee. A possible way of improving interaction at the provincial and District level (and supporting the decentralization activities of MMAF) is to give greater independence and authority to IMAC’s project field offices in addressing local issues and interacting with local Government officials within the confines of a clear management and budget responsibility framework. KIIs with IMACS regional office staff all reported that they receive instructions from IMACS in Jakarta as to activities and are limited in being able to make independent operational or budgetary decisions. Also, there is currently no quarterly or yearly planning and budgeting at Page | 32 regional level of IMACS to provide clear directions as to activities and which could support increased autonomy. Questioning regarding day-to-day activities confirmed that the regional offices operated mostly as an administrative centre for Jakarta-based staff, arranging visits and meetings, providing transportation, accommodation etc. Any substantive engagement with local Government only occurred under direct request and supervision from Jakarta CONCLUSION 4 IMACS has successfully developed a tool (I-CATCH) to assess vulnerability to climate changes at the community level and this tool has now been adopted by MMAF as the program Desa Pesisir Tangguh. However, the planned activities of I-CATCH are behind schedule for some components i.e. number of people trained, number of community plans developed, number of adopted communities, number of local government management plans developed etc This delay has resulted in (a) Small grants program not being coordinated as planned with I￾CATCH activities and (b) I-CATCH not being aligned with the district planning cycle which in turn has led to minimal adoption because of the dependence on local Government initiatives for funding. FINDINGS SUPPORTING THE CONCLUSION Annex 6 provides the performance details against project-level performance indicators for the Climate Change Adaptation project of IMACS with that information being based on data derived from the PMP, Work Plans and quarterly and annual reports of the component project. In addition, KIIs with MMAF staff and IMACS staff supported the findings of Annex 6. The reason why I-CATCH is behind schedule is due to a number of factors, including:  The change of IMACS and MMAF management which lead to change of engagement policy and created constriction in communication and coordination with technical directorates  Termination of communication sub-contractor (RARE) left IMACS with no outreach activities. Although RARE were not necessarily tasked to specifically undertake I-CATCH communications activities, IMACS, without its own communication staff or the sub￾contracted institution to do the work, had no capability to launch its awareness or outreach activities to ‘sell’ the idea of Climate Change Adaptation issues and prepare target communities, and local governments alike. IMACS has since recruited its own communications specialist to address this, and other, communications and outreach issues. CONCLUSION 5 IUU-related training was carried out and met target numbers of trainees. However, there is no evidence that training activities related to IUU fishing (observer training, port state measures, improving capacity of law enforcement organizations) has resulted in (a) improved data for control of IUU fishing (b) improvement in inter-agency MCS cooperation or (c) additional legal instruments to support Community Enforcement Networks Page | 33 FINDINGS SUPPORTING THE CONCLUSION Annex 6 provides details of the analysis of project-level reporting documents and includes data as reported on IUU training activities. These data were the primary data source for establishing the conclusion although KIIs with the Directorate General of Marine and Fisheries Resources Surveillance also confirmed the project reporting. However, KII with the Director General of Fisheries Resources resulted in him stating that he requested that observer program training participants have a High School qualification. This request was made on the basis of (a) the vessel owner would be more willing to pay for High School graduates because of their lower salary point (b) the functions to be performed were more appropriate to High School graduates and were more ‘blue-collar jobs and (c) there were previous experiences with behavioral issues when over-qualified staff worked on board vessels. However, the Director of Training under BPSDM apparently insisted the participants be undergraduates, and IMACS and NOAA, who work more closely with BPSDM, followed. The result of this has been ineffective training where the vessel owners are reluctant to use the trainees and the trainees do not like their job. Follow up assessment of training effectiveness would, if undertaken, have identified this problem (see also Section 5.2.1 Conclusions and Recommendations. In addition, there have been no new mechanisms or changes to the data collection system being developed by the Directorate of Surveillance, the coordination mechanisms between related institutions and agencies related to MCS cooperation remain the same and there have been no new regulation or any legal instruments instituted for supporting the Community Enforcement Networks. In addition to the formal IUU training activities coordinated by IMACS, MPAG also supported local authorities (e.g. in Wakatobi) in establishing co-management approaches to address IUU fishing. RECOMMENDATIONS ADDRESSING THE ABOVE 5 CONCLUSIONS The key issues identified for which actions are required are (a) the relationship between project activities and impacts – in general, both IMACS and MPAG activities are on schedule but are not resulting in effective biodiversity, sustainable fisheries or climate change adaption outcomes (b) the connection between project level activities and Government processes - MPAG activities are well connected to Government processes whereas IMACS activities are not as well integrated. As a result, the Evaluation Team recommends the following:  Project activities should be considered within the broader context of MMAF institutional capacity development. Therefore, subject to agreement with MMAF regarding the provisions of adequate facilities, MRP partners should move to a secondment mode of technical assistance and institutional capacity building. This would require (a) MRP partners, USAID-I and MMAF jointly identifying key and appropriate counterparts in the MMAF (probably at Page | 34 Director-General or Director level) for IMACS and MPAG national and regional activities17 (b) IMACS and MMAF establishing engagement protocols as to how IMACS and MMAF staff would interact and work together. These engagement protocols may include arrangements for a limited number of key IMACS’s staff to have access to counterpart MMAF staff but does not necessarily mean permanent relocation of IMACS staff to MMAF. The aim is effective counterpart interaction rather than a rigid secondment arrangement. This requires (a) appropriate counterpart staff and Directorates being identified and (b) a flexible arrangement that is agreeable to all parties. Learning from the previous attempt at secondment, it is important that IMACS seconded staff have full access to all appropriate operational Directorates and Secretariat General Divisions and not be restricted to a single point of contact within MMAF.  IMACS should be undertaking better institutional assessment and capacity building strategies that are consistent with those adopted by MPAG to ensure that component activities address broader MRP objectives in a consistent way. This should include developing and adopting a consistent approach for all MRP training that includes assisting related Directorate Generals in (a) standardization of competencies to positions (these would be related to the Key Personal Indicators process being developed within MMAF) (b) mapping current competencies of staff (c) undertaking a training needs analysis (d) developing standardized curricula (e) developing an agreed training plan. The process should also be coordinated with MMAF’s planning and budgetary cycle  Regular assessments of the effectiveness of training implemented by IMACS and MPAG should be undertaken and used to both support MMAF institutional development and to design better project supporting activities, including training  Budget should be allocated to funding the implementation of priority actions that resulted from I-CATCH exercises. IMACS has not budgeted any funding for the implementation of any activity related to the Climate Change Adaptation (CCA) Plan, which is the end result of I-CATCH exercise. Since there has been (a) ineffective integration with local Government (DKPs) planning processes and (b) only a small number of the grants made under the Small Grant Program being related to the CCA, the implementation of the CCA Plan is unlikely to be funded under present arrangements. 17 Unlike IMACS, MPAG have no regional offices despite there being a need for strengthening and working with local institutions including district and province level Government and particular the regional office of MMAF for MPA management (BKKPN). This can be achieved either by integrating MPAG and IMCS activities through the IMACS regional offices or by MPAG having a seconded MPA expert where needed. This is a particular need in the BKKPN Kupang to support the limited capability of staff there. Page | 35  IMACS Regional Offices should develop, at the very least, a quarterly plan of activities and budget, to enable enough flexibility to be given to them to adapt to “local needs” and assist in developing IMAC’s provincial and District strategies and work plans. This autonomy should be within the context of (a) integrating IMACS activities with DKP activities and planning and (b) assisting in communicating overall visions and strategies at the provincial and district level. 5.2.2 Key Question 2 What is the capacity of the IMACS and MPAG project teams to implement activities in an efficient and timely manner, including utilization of grants, subcontracts, and other resources? CONCLUSION 6 IMACS’s and MPAG’s professional and administrative capacity is generally good although some issues have arisen in the past, such as a high level of IMACS staff turnover which has impacted IMACS ability to efficiently implement IMACS’s activities. However, a significant skills gap is apparent in that there are no IMACS staff with (a) practical senior-level Government experience in fisheries management (strategic and management planning, operational strategies etc) - as distinct from fisheries stock assessment/research (b) experience in implementing Government institutional change. These skills will be important in assisting MMAF to implement its vision for sustainable fisheries and marine conservation. Note however that IMACS has very recently hired a professional, Dr. Purwanto who may fill this skills gap. CONCLUSION 7 The small grant program is not well aligned to overall MRP or project-level objectives although it is now being administered efficiently CONCLUSION 8 IMACS has not applied a high level of corporate management skills in its dealings with MMAF, and, as a result, these should be improved so that they provide an example to MMAF of good corporate management and governance practices KEY FINDINGS SUPPORTING THE CONCLUSIONS Skills & stability of MPAG and IMACS Based on KIIs with IMACS and MPAG staff as well as examination of staff CVs, both IMACS and MPAG has appropriately qualified professional staff to undertake technical aspects of the MRP. However, for a program with a major focus on institutional capacity building of the MMAF, there is a surprising lack of key staff with senior experience in institutional capacity building or in working in a functioning fisheries management agency outside of Indonesia who could serve as a catalyst for the introduction of contemporary fisheries management Page | 36 practices. This is a major skills gap and has probably been a factor in the lack of integration of activities with Government priorities and processes. A significant issue influencing the efficiency of implementation of project activities has been the very different paths of stability of IMACS and MPAG. MPAG had its origin in the CTSP which started in 2009 as part of CTI-CFF regional project activities. When it was established as an Indonesian project in 2011, MPAG retained essentially the same key staff from CTSP￾Indonesia. As result, it continues a good working and stable relationship with MMAF and other key stakeholders, including USAID Indonesia. MPAG also has had opportunities to establish essential networking collaboration with regional activities on Marine Protected Area Management issues through the CTI-CFF Working Group on MPA. By contrast, the IMACS component began in 2010 and, since that time has had two Chiefs of Party and a temporary replacement and other significant staff turnover. As a result of this instability, IMACS have not developed the same consistent relationships with MMAF and other key stakeholders. However, both organizations now appear to be more stable and are building the necessary internal and external long term relationships. KIIs with the major technical partners (NOAA) report that they now have consistent and excellent working relationships with IMACS and USAID. Administrative capacity of IMACS While administrative capacity is generally good, there have been past issues. For example, RFAs for small grants (RFA01) were advertised and received prior to any staff or processes being in place to administer small grants. This has since been rectified and there are no current areas of significant administrative weakness identified as part of this evaluation. Small grants program In addition to past administration issues, examination of small grant applications and KIIs with IMACS small grants administrator and USAID show that the small grants are not generally aligning with the overall objectives of MRP and are not adding substantial value, especially when their administrative burden both for IMACS and USAID is taken into account. But as noted above, the governmental approval process takes considerably more time than anticipated. Weak understanding of climate change and adaptation responses has also lead to adaptation plans of questionable usefulness while the capacity of local NGOs and other organizations to deal with USAID financial requirements is weak. USAID-I has a process for approving various steps in the Small Grant process, including RFA advertising, grant approval and monitoring. However, this function is undertaken by one person (not as a full time function) and the evaluation team has some concerns that if the number of small grants increases, this may create an administrative bottleneck. From KIIs with DKP officials, all 5 (100%) reported that they did not know the criteria for Small Grants and were not involved in the identification of priorities prior to RFAs being Page | 37 advertised. One respondent additionally reported that they provided, on request, a generic letter of support for a number of applications following their submission. The process for aligning small grant applications with overall MRP objectives, while usually clear in the advertising stage appears more subjective during the assessment phase. In addition, the assessment of applications was carried out by IMACS staff without any representatives from KKP or any DKP, although KKP were represented in an earlier evaluation committee. In examining the details of a number of Small Grants and undertaking KIIs with a number of recipients, the Evaluation Team is particularly concerned about the impacts and sustainability of some activities. These range from (a) concerns about whether activities can continue in the absence of aid funds to (b) concerns about the effectiveness of outcomes in contributing to MRP objectives, even if successfully completed. Of the $2.0 million allocated to the Small Grants program, about $1.6 million has been committed under a total 14 RFAs. Of the grants initially approved under RFA01, 7 recipients received grants totaling approximately US$ 206.000. Of this 45% ($ 93.5k) has been earmarked for a private company to do data collection on tuna fisheries, 32% ($65k) for work on alternative livelihood activities by 4 local recipients, 11% ($22k) for desalination technology, and 12% ($24k) for strengthening local customary regulations. During the period of the evaluation, additional grants were approved. Some examples of Small Grants and their relationship, or lack of it, to MRP and/or individual project objectives are as follows: No Small Grant Title Objective Review/Findings 1 Ecotourism development and environmental management in Matahora village (Wakatobi). Value IDR 249,239,000 To create a village-level eco-tourism program in Matahora village (Wakatobi) that will catalyze the economic potential of the area while preserving its environment. The village is included in a pilot project on Climate Change Adaptation (I-CATCH). The grant proposal was developed before the villagers finalized their climate change adaptation plan (priority action plan). As such, the grant project was being developed regardless of the I-CATCH activities. 2 Distillatory units using boat exhaust gas to produce fresh water on board for fishers in Kelurahan Bone-Bone, to help off-set climate change problems related to drought. Value IDR 216,190,000,- To desalinate sea water into fresh water using a distillatory method by utilizing boat exhaust gas, for fishers in Kelurahan Bone-Bone. The village is included in a pilot project on Climate Change Adaptation (I-CATCH). It is not clear if this project was part of prioritized actions planned by the village. The evaluation team has concerns about the sustainability of this project beyond the grant period because the plants need to be customized to each vessel, making costs to fishermen prohibitive. In addition, small boat desalination plants are commercially available at prices significantly less than the probable costs of the plants being developed. Impacts on the sustainability of fisheries resources of facilitating longer fishing trips also have not been analyzed. Page | 38 3 Producing fish powder, fish fodder, and fish oil using waste fish in Lombok Timur District, NTB. Value Rp. 134.994.000,- To increase income of coastal communities through producing fish powder, fish fodder, and fish oil using wasted fish in Tanjung Luar and Labuhan Lombok. Straightforward economic activity not necessarily related to any Climate Change Adaptation (I￾CATCH) or Sustainable Fisheries Management activities by IMACS 4 The Management of Bumbang Bay by Stipulation of Local Rule (Awig-awig) to support Strategic Plan of DKP Central Lombok 2013 and Increasing Local Community Based Wisdom at Bumbang Bay, Pujut, Central Lombok area IDR. 237,747,500- To manage Conservation Area of Bumbang Bay and increasing community wisdom in Bumbang Bay, Central Lombok area. The place is outside task 2 and task 3 activities. 5 Fishing and Living￾Enhancement of Indonesian Handline Tuna Fishery: An Integrated Community￾Based Approach to Sustainability. Value IDR 935,000,000.00 To involve local fishing companies in fisheries data collection, To increase availability of accurate data on the status of fish stocks. To improve collaboration between private sector and government agencies, To improve management of fish stocks and fisheries. To improve the lives of fishermen by ensuring a stable and productive fishery. Directly support task 2 program on Sustainable Fisheries Management. 6 Coastal Community Empowerment through Fish Drying Activity and Rehabilitation of Mangrove Forest in Rumba-Rumba Village, South Konawe District, Kolono Sub-district. Value Rp. 134.900.000,- To provide alternative source of income through fish drying activities and mangrove forest rehabilitation To reduce the threats to and rehabilitate marine ecosystem biodiversity This is not an ‘alternative livelihoods’ project that provides an alternative to income generation activities that might be lost because of climate change since it still relies on the same supply of fish for drying 7 Development of alternative Income for fishermen and women through utilization and processing of snail and clamshells into handicrafts with economic value. Value Rp. 128.000.000,- To increase sustainable alternative sources of income for small fishermen families and women in Wangkolabu village To strengthen fishermen community resilience in responding to the impacts of climate change. Grant proposal was developed before the I￾CATCH outputs being written. It was due to the existence of a local NGO prior to IMACS in the village, hence communication and collaboration was developed beforehand. Page | 39 Corporate processes Based on an examination of IMACS and MPAG documentation and KIIs with IMACS and MPAG staff, the evaluation team has also concluded that IMACS’s and MPAG’s corporate process capacity is generally well developed. However, an opportunity is being missed to extend that good corporate governance to MMAF and other Government partners to re￾enforce capacity building activities by demonstrating best practice. For example, there are no agendas prepared, no minutes taken and no follow-up actions for the important ‘co￾ordination meetings’ with MMAF. Only meeting ‘notes’ (in English) are prepared by IMACS following these meetings. Government Ministry processes are document-driven with documentation in Bahasa Indonesia, and not English, and providing assistance to MMAF in preparing meeting documentation would, in the evaluation team’s view, greatly assist the interactions between IMACS and MMAF and hence the effectiveness of capacity building. Given the practices within MMAF for rotating staff, directors and director generals, formal paper/document trails are the only reference that can be used to provide continuous development of the program within MMAF and DKPs. One area where IMACS corporate processes can be improved is in the devolution of responsibility (see also Conclusion 2 above). The evaluation team’s view, confirmed by KIIs with both regional and Jakarta-based staff, was that decision–making was strongly hierarchical. A greater downward devolution of responsibility would result in better service delivery of task activities, particularly in regional areas. This issue was not a feature of MPAG since most of site based staff are NGO’s staff. RECOMMENDATIONS ADDRESSING THE ABOVE 3 CONCLUSIONS  IMACS seek approval to engage a person (possibly at DCOP level) with practical senior-level governmental experience in institutional change and fisheries management (as distinct from fisheries stock assessment or biology), in a credible fisheries management agency, to work with senior counterpart staff in MMAF on management strategies and objectives, management planning and operational strategies. This, together with other recommended secondments (see recommendation no. 1), would provide the strategic link between MMAF/IMACS activities in achieving sustainable fisheries objectives and the work of technical experts embedded within the operational units of MMAF, including linkages between fisheries management and MPA development. As noted above, IMACS have already begun addressing this issue.  Stop the further issue of Small Grant RFAs as well as the commitment of further Small Grant funds and use the unspent funds to implement activities more clearly focused on MRP Objectives. Page | 40 Some of the potential activities identified by the Evaluation Team that are directly related to MRP Objectives that should be considered for funding are (a) commission a study on assessing the sustainability (including integration with Government planning) and the methodology of replication of existing grants to other areas and/or communities (b) directly support activities which are being developed from vulnerability assessments/priority actions plan as part of I-CATCH. The suggested “new” authority of the IMACS Regional Offices could be used to identify and determine which activities/actions plan should be directly supported/funded or arranged via third party sub-contracts. This would better target funds to identified priority actions on climate change adaptation (c) commissioning a study to analyze the best way to document and integrate research being done by Universities in Indonesia on fisheries, marine protected areas and coastal climate change issues into Government (particularly MMAF) processes. This would enable policies in these areas to be developed using the full scientific knowledge of the issues that exists in Indonesia. There is currently no process for institutionalizing and incorporating this knowledge into Government decision making.  IMACS and MPAG should review their corporate governance procedures and institute changes, such as proper documentation of high-level meetings with MMAF, that demonstrates best practice and provides the ‘paper-trail’ that is necessary for guiding Government action on outcomes of meetings. 5.2.3 Key Question 3 What were the important policy actions, institutional relationships, and partnering arrangements that facilitated project success? CONCLUSION 9 Policy actions, institutional relationships and partnering arrangements pursued by MRP contributors in implementing project activities have generally been effective in achieving project outputs against targets. However, IMACS and MPAG have not paid sufficient attention to the potential of the IBRC in providing the scientific support for MMAF policies and for MPAG and IMACS project activities. As noted earlier (Conclusion 1), although project activities have been successfully achieved, these achievements have not led to the achievement of overall program strategic goals. CONCLUSION 10 IMACS and MPAG have worked with NOAA and URI-CRC to deliver high-quality training activities. However, the way in which training needs are identified by IMACS needs attention so that training is better integrated with Government processes. Follow-up assessment of training effectiveness by IMACS and MPAG also needs to be expanded KEY FINDINGS SUPPORTING THE CONCLUSIONS Support for implementation of project activities by MPAG and IMACS occur under several arrangements including (a) University Partnerships between Indonesian Universities and the Page | 41 University of California of Los Angeles (UCLA) to establish an Indonesia Biodiversity Research Center (IBRC) and with the University of California of Santa Cruz (UCSC) to establish an Indonesia Marine Biotechnology Partnership and (b) sub-contracting arrangements between IMACS and URI-CRC and between MPAG and NOAA (to deliver MPA training) and IMACS and NOAA (to deliver EAFM and IUU training). The University Partnerships program has been evaluated separately in 2012 with one of the key findings being that more attention should be paid to strengthening partnerships with GOI Ministries. The Evaluation Team sees merit in engaging the IBRC and other Universities in research focused on key marine biodiversity priorities of MMAF to support the strategic planning functions of MMAF. IMACS and MPAG could, and should, play a role in developing that partnership as part of its more general support to MMAF in assisting with the development of a strategic action plan for implementation of MMAF’s vision for sustainable fisheries and biodiversity conservation. KIIs with IMACS, MPAG, NOAA, UCLA and URI-CRC have shown that current service delivery, particularly of training activities, through the existing partnership arrangements is an effective means of achieving project activities against targets. For IMACS, the subcontracting arrangements with NOAA and URI-CRC are working well with current relationships being described by these two institutions as ‘very effective’ and ‘excellent’. This is quite different to the situation earlier in the IMACS contract when high staff turnover and policy and procedural changes made for a very disruptive environment for the partnering institutions. These institutions need to plan activities well ahead to ensure staff availability and therefore a stable and long-term relationship is vital to their ability to deliver services effectively and efficiently. For MPAG, the ongoing collaboration with NGOs and CTI-CFF partners has provided a strong and stable partnership that result in those partnering arrangements being able to deliver project activities in a very effective manner. MPAG have good relations and are well integrated into Government (MMAF) processes. At the local Government level, relationships between local Government agencies and MPAG (and their NGO partners) vary between sites and, importantly, the success of these relationships appear to impact on MPAG’s project-level success. For example, in the Savu Sea MPA site, where there is strong support and good cooperation with local Government, 90% of target activities have been met (see Annex 6). This can be compared with the Berau MPA site, which is implemented by the same NGO, but where a lack of local Government support has resulted in only 10% of planned activities being achieved (Annex 6). The extent to which activities are supported by local Government and integrated with their processes therefore appear to influence the success of those project activities. This is an important lesson for all MRP stakeholders. IMACS are making progress in developing relationships with local Government under their new COP although much remains to be done. IMACS has successfully established a formal Page | 42 Memorandum of Understanding (MOU) and Memorandum of Agreement (MOA) with the Governor of South East Sulawesi Province and every Mayor and Regent in the area where IMACS works. This formal relation between the project and the local government can be used to bring “buy-in”, conceptual adoption and focused activities in the future. However, it is important for the success of the MOU/MOAs that this initiative is followed up by better, and continuing, co-ordination between IMACS and local Government partners. The growing relationship between all MRP partners, USAID and the Government has resulted in a request from MMAF to support its effort to develop the new 5 (five) years strategic planning (2014 – 2019) and shows that MMAF respects USAID assistance. However, the recent change of the Secretary General may bring a change in preferences, priorities and emphasis of future MMAF fisheries policy which need to be taken into account. From MPAG reporting documentation (see Annex 6), as well as from KIIs with NOAA, training effectiveness studies have only been conducted for MPA training although no training effectiveness studies having been undertaken for IMACS administered training. Because the internal policies of MMAF (such as staff rotation) have the potential to dilute the impact of training, effectiveness surveys should be extended to all areas of training provided by the projects. RECOMMENDATIONS ADDRESSING THE ABOVE 2 CONCLUSIONS  Existing partnering arrangements have been effective in delivering high quality training to support both IMACS and MPAG project activities and should be continued. However, consideration needs to be given by USAID-I, IMACS and MPAG to (a) developing and implementing multi-year agreements with training implementation partners such as NOAA to provide greater planning certainty and ability for partnering institutions (b) expanding training effectiveness activities to ensure that partnering activities are contributing to overall IMACS and MPAG goals, particularly MMAF institutional strengthening.  IMACS and MPAG should utilize the expertise in the IBRC to both inform their own project activities and to encourage better linkages between the IBRC and MMAF. 5.3 Level 2 Evaluation 5.3.1 Key Question 4 Is the MRP achieving satisfactory results against targets in the following component areas: (a) Strengthening MMAF through institutional development, (b) Improving sustainable fisheries management, (c) Strengthening coastal community resilience and climate change adaptation, (d) Creating and effectively managing Marine Protected Areas, and (e) Improving capacity to reduce Illegal, Unreported and Unregulated Fishing? CONCLUSION 11 Page | 43 The MRP is generally achieving satisfactory results against targets in (a) Strengthening coastal community resilience and climate change adaptation and (b) Creating and effectively managing Marine Protected Areas. However, results for (a) Strengthening MMAF through institutional development, (b) Improving sustainable fisheries management and (c) Improving capacity to reduce Illegal, Unreported and Unregulated (IUU) Fishing are less satisfactory. However, the targets (PIs) of both IMACS and MPAG are not explicitly linked to the Results Framework (see Level 3 evaluation below) and therefore the effectiveness of activities in achieving higher level strategic objectives is questionable. See also Level 1 Evaluation (above) and Level 3 Evaluation (below). The lack of integration between Task 2 and Task 3 within IMACS and between IMACS and MPAG in marine conservation areas has the potential to have unintended consequences that could create a disjointed development plan for villages, districts and provinces. In addition, the activities are not being aligned with or included in local government (provincials and districts) planning cycles. KEY FINDINGS SUPPORTING THE CONCLUSION Comparisons between IMACS and MPAG annual work plans, PMPs and quarterly and annual reports show the various components are meeting targets (see Annex 6 for a detailed analysis). For IMACS Task 1 (Institutional Strengthening), the IMACS second year annual report (December 2012) shows that 4 of the 8 performance targets (50%) are on, or ahead of schedule. Remedial action has been proposed to address those that are behind schedule. For Task 2 (Sustainable Fisheries), 5 of the 7 performance targets (71%) are on or ahead of schedule while activities related to improving capacity to reduce capacity IUU fishing was reported to be behind schedule. Sustainability of the interventions to strengthen MMAF through the provision of training is also of concern. IMACS should pay particular attention to ensure that its training plan supports the overall strategic goals of MMAF and is better aligned with the needs and the human resources development plan of each Directorate-General within MMAF. This is not such an issue with MPAG who have spent considerable time developing a system for aligning, integrating and standardizing training with the Directorate of Conservation Areas and Species. However, these training activities are being pursued by MPAG in isolation and without reference to the broader MRP strategic goals. RECOMMENDATIONS  IMACS should take a greater role in developing and adopting a consistent approach18 for all MRP training that (a) is aligned with broader MRP and MMAF strategic goals and (b) is undertaken co-operatively with MMAF. This consistent approach should include assessing the impact of training 18 Possibly based on the MPAG approach Page | 44  IMACS and MPAG needs to better aligned their activities with the planning and budgeting cycle within local government system, especially when considering the timing when project results should be proposed/incorporated into formal proposition to DKPs.  The integration between the sustainable fisheries component of IMACS and MPAG marine conservation area in MPAG should be strengthened, particularly as it relates to EAFM and climate change adaptation  IMACS needs to speed up development of the outreach/communication program with clear key messages, target groups, and specific means/conduit for each group and message. It is important to obtain support and encouragement not only from direct beneficiaries of targeted activities, but also from the general public.  Budget needs to be allocated to funding the implementation of priority actions resulted from I-CATCH exercises. As the group responsible for integration of MRP activities, IMACS should be taking a greater role in ensuring a consistent approach to MRP training activities, using the model that MPAG have already developed that is integrated with Government procedures. Such a consistent approach to training activities might include the following activities:  standardization of competencies to positions – related to Key Personal Indicators process being developed within MMAF  map the current competencies of the staff within each D-G,  undertake a training needs analysis  Develop standardized curriculum (cf. MPAG Specific Competence Standards)  Develop an agreed training plan IMACS should also take a greater role in coordinating other activities outlined in the recommendations above. Communications and public relations activities have been identified (see Section 6) as being important in the longer term in assisting MMAF and the GOI to develop broad Government and public support for a shared vision for marine resources management. These activities can and should be started during the current MRP. Such social marketing activities were originally part of the MRP and were subcontracted to a group, RARE Conservation. However, that contract was terminated and IMACS have recently recruited a communications specialist to take over that function. These activities provide an excellent opportunity to work with MMAF in developing a strategic action plan for implementing their vision for sustainable fisheries and marine biodiversity protection and therefore should be pursued as a priority. 5.4 Level 3 Evaluation In evaluating the overall progress of the MRP in achieving its strategic objectives, a key question is does USAID have the right mix and balance of program components to achieve the strategic goals of the program? The IMACS Results Framework (Annex 7) provides the Page | 45 linkages between overall strategic objectives of USAID and IMACS activities, although the MPAG Results Framework (Annex 7) does not provide that framework. So far as the Evaluation team can ascertain, no reporting of the KRAs in the IMACS or MPAG Results Framework has been completed and therefore the contribution of MRP activities in meeting the higher level KRAs, IRs and overall Strategic Objectives has not to date been measured. The Level 3 Evaluation was therefore carried out in that context. 5.4.1 Key Question 5 Is the MRP achieving its overall objective of sustaining the long-term welfare of coastal Indonesians by reducing the destruction of marine resources and preparing them for climate change impacts and is the MRP strategy of strengthening the capacity of MMAF and providing targeted technical assistance, producing sustainable improvements to ecosystems and institutions? CONCLUSION 12 The MRP is not achieving its overall strategic objectives, partly because those strategic objectives are not clearly articulated and partly because the MRP strategy is ineffective in producing sustainable improvements to ecosystems and institutions The overall objectives of the MRP (see section 1.3) are: 1. Restore and enhance ecosystem productivity, biodiversity and resilience for food and economic security; and 2. Increase the resilience of natural ecosystems and coastal communities to adapt to climate change and reduce risks from disasters The 5 components of MRP are designed to achieve these strategic objectives with a key development assumption being that MRP training activities and technical assistance alone will result in institutional strengthening of MMAF and hence achievement of the two strategic objectives. This has been considered in Section 1.4 and elsewhere. Training activities and technical assistance are necessary but not sufficient to result in institutional capacity building and therefore the underlying strategy of the MRP is flawed. This has been exacerbated in the implementation of the MRP by the lack of effective integration by IMACS both between MRP component activities (see below) and between IMACS and MMAF planning and operational processes. The end result of these two factors is that while various project activities are being carried out, they are not contributing to overall strategic objectives. CONCLUSION 13 There is little or no strategic integration between MRP activities although some limited operational integration occurs. Page | 46 The MRP is undertaken essentially as a portfolio of activities rather than as an integrated program with each component being managed and conducted as a stand-alone activity. Evidence for this comes from a number of sources, including KIIs with IMACS, MPAG and MMAF staff, records of MRP co-ordination meetings and project component work plans and annual and quarterly reports. For example, there is a lack of an integrated approach to the development of training plans – each component of MRP develops and pursues its own training plans although some limited operational co-ordination takes place during quarterly ‘co-ordination meetings’ between IMACS and MPAG. Even within IMACS, there is little or no integration in the strategic approach or activities between sustainable fisheries management task and climate change adaptation. Although IMACS has responsibility under the MRP for integration of program activities across all MRP components, they are somewhat limited in what can be done in this respect because they have no authority to achieve integration through contractual or other arrangements. This puts IMACS in a difficult position with regards meeting their integration responsibilities. IMACS and MPAG have their own separate contractual arrangements with USAID, with the MPAG contract not requiring reporting through IMACS. In addition, since MPAG activities are just one of a portfolio of activities for the larger NGO partners, the NGOs have little incentive to submit to an MRP-led strategic direction. However, these difficulties can be overcome (and have been in other projects) by developing effective working relationships with other MRP components, particularly MPAG. IMACS have not done this. In summary, IMACS should be encouraged by USAID to improve its effectiveness in achieving MRP operational integration, particularly since such integration is also critical in assisting MMAF realize its vision for sustainable fisheries and marine biodiversity protection. USAID can further assist in this process by making it clear to IMACS and MPAG that operational integration of MRP activities through IMACS is expected. The technical linkages within MRP between sustainable fisheries activities, establishing Marine Protected Areas (which include fishery reserves and which protect critical habitats) and climate change adaptation (improved fishery management is critical for adaptation and resilience) are important and should be both emphasized and communicated to MMAF by IMACS. This emphasis on the inter-connectedness of activities would greatly assist both MMAF’s understanding of the MRP and IMAC’s activities in assisting MMAF to develop a strategic action plan to support their vision for sustainable fisheries and biodiversity conservation. At the present time, the inter-connectedness of MRP activities is certainly not being clearly articulated. The evaluation team found that there was no apparent involvement from IMACS staff in Indonesia’s National Coordination Committee (NCC) of CTI-CFF. IMACS has not Page | 47 participated or become a member in any NCC’s working groups nor attend meetings being conducted by NCC on sustainable fisheries, climate change adaptation and marine protected areas. This lack of involvement not only isolates IMACS from the major political supported initiative in Indonesia (and MMAF) but also deprives IMACS of a chance to take a role as the major player in strengthening MMAF capacity to integrate various aspects on sustainable fisheries management and climate change adaptation. By actively participating in the National Coordination Committee on CTI-CFF, especially in EAFM and CCA working groups, IMACS could not only facilitate the process but could also show leadership toward the development of the necessary framework for EAFM and CCA. It could also show leadership in finding ways to integrate sustainable fisheries management with marine protected area management under climate change adaptation circumstance. As such, MMAF roles with USAID-I support, in such regional initiative could be exemplary and provide leadership for other member countries of CTI-CFF. The evaluation team is not aware of any reporting against the KRAs of the Results Framework for the overall Marine Resources Program that would define and measure progress overall MRP strategic objectives. The absence of this is, in the evaluation team’s view, a critical omission and should be addressed by USAID. KEY FINDINGS SUPPORTING THE CONCLUSION There has been no reporting against the Results framework that defines either the MRP strategic objectives or the linkages with objectives, intermediate results and KRAs for the various components of the MRP. The Evaluation Team has therefore examined performance against KRAs and concluded that, of the 22 Key Result Areas (KRAs) contained within the IMACS Results Framework, only 2 (KRA 310: Increased awareness and knowledge of Climate Change risks and adaption practices and KRA 340: mechanisms for Climate Change Resilience and Adaptation increased) have made significant progress while one other (KRA 250: Capacity for fisheries modeling and social accounting improved) has made some progress. None of the KRAs related to institutional strengthening of the MMAF or program integration have made any measurable progress. The MPAG Results Framework is not explicitly linked to the MRP or USAID-I strategic objectives and therefore how the activities and results of the MPAG component should contribute to the overall objectives of MRP is not articulated. As a result, a definitive analysis of MPAG’s effectiveness in contributing to the MRP is not possible although, as noted above, MPAG’s activities clearly are not integrated with other MRP components. RECOMMENDATIONS Page | 48  Reporting by IMACS and MPAG against the KRAs of the MRP Results Framework should be requested by USAID. This will allow assessment of how IMACS and MPAG activities are contributing to higher level MRP objectives and key result areas.  As recommended under Level 1 evaluation (see above), IMACS should be required by USAID to adopt better institutional assessment and capacity building strategies that are consistent with those adopted by MPAG. It is particularly important that such a consistent approach is adopted for training activities.  Implement other Level 1 and Level 2 evaluation recommendations, particularly those related to improving the effectiveness of service delivery and the institutionalization of MRP component activities. Such initiatives would result in a greater level of achievement of MRP objectives.  IMACS should collaborate with the Coral Triangle Initiative of Coral Reefs, Fisheries and Food Security (CTI-CFF) through Indonesia’s CTI National Coordinating Committee in MMAF and actively participate in its working groups 6. PLANNING FOR LONG TERM OUTCOMES In accordance with the over-arching recommendation of this evaluation, the period to September 2014 should, in the evaluation team’s opinion, be used to establish closer relationships with national and local Government institutions so as to lay the foundations for effective assistance, under the USAID 2014-19 Country Strategy, in improving marine resources management in Indonesia. The areas where that closer relationship is needed are:  Supporting and guiding the implementation of the Government’s vision for marine resources management and development so that a shared, long-term vision and strategy results19. Such a vision should be based on sustainable resource use (including the necessary biodiversity conservation outcomes to support that sustainable use) to which the Government has already committed.  Better integrating IMACS (and to a lesser extent, MPAG) project partners activities into MMAF and local Government planning and budgetary processes. A study on Host Country System (HCS), or foreign grant management system, has been completed by MPAG, covering the cycle of planning, budgeting, disbursement, program implementation, financial reporting and auditing. Based on this study, a mechanism to channel foreign grants to local government and NGOs has been designed and disseminated internally within MMAF (with support and facilitation from the Ministry of Finance). This system could be implemented for better integrating IMACS, MPAG and other project partners’ activities into MMAF and local Government planning and budgetary processes. USAID-I, as the initiator, should take the initiative to start implementing the system to test its effectiveness and feasibility in accommodating various interests from both donor institutions as well from the Government of Indonesia itself. 19 Ideally, a vision and strategy that is, so far as practicable and possible, independent of short term political influences. This can be achieved by, for example, embedding agreed fisheries management plans in Law or Ministerial Decree rather than as policy statements from Directorate-Generals Page | 49  Adjusting project activities, such as training, in ways that better link to already existing Government procedures This report makes recommendations as to the mechanisms that can be used to achieve this better linkage and integration with MMAF and local Government, the most important of which are:  Having senior technical project staff seconded to MMAF (and in appropriate circumstances, to local Government) to work alongside technical counterparts. The current arrangement of having project implementation teams located in offices remote from the Government agencies they are meant to support is ineffective and does not support the close working and mentoring role that is needed. Original IMACS work plans were linked to MMAF objectives and consideration should be given to reviving this approach in the 2014 IMACS Work Plan.  Better integrating MRP component activities so that (a) a consistent approach to technical activities such as EAFM, training activities etc is achieved rather than the current inconsistent and sometimes contradictory approach from individual MRP participants (b) the relationship between marine biodiversity protection (MPAs), coastal climate change adaptation and sustainable use of resources is clear and broadly communicated. The current situation is that these activities are being carried out, and being seen, as separate activities with no common objective. Better integration of MRP activities would also facilitate IMAC’s work in assisting MMAF to develop a strategic action plan to guide its MMAF’s vision. Such an action plan could address both integration within MMAF of sustainable use (fisheries) and conservation, and better national–local integration of policy and implementation.  Development of a reporting mechanism for IMACS and MPAG against the MRP Results Framework that would clearly demonstrate the high level linkages to MMAF’s vision and strategy. This would also assist in communicating why, and how MRP activities are linked and integrated with Government visions and strategies MMAF has approached USAID to request assistance with its 2014-19 Strategic Planning process and attending to the above issues would allow USAID to be effective in providing that assistance. However, attending the above issues would also shape the way in which longer term assistance is delivered. For example, in the spirit of assisting MMAF in developing a strategic action plan to implement its vision for marine resources management that is shared with USAID, the role of program implementers should be focused on facilitation and advice to MMAF and not on developing the document. This would ensure that MMAF takes ‘ownership’ of what is an important policy document. Page | 50 This does not, of course, preclude USAID’s expert policy and operational input into the document; in fact it makes such expert advice more important. One, of many, areas where USAID could assist MMAF is in addressing the unresolved issue of the details of devolution of responsibilities to local Government. Responsibilities of each tier of Government for marine resources management need to be clear and preferably established in law and this is an area where expert, impartial US advice could assist. Future assistance will also need to engage a broader range of Government and non￾Government stakeholders in implementing a national vision and strategy for marine resources management. For example, the climate change adaptation activities need to engage the National Climate Change Commission, economic incentives for sustainable fisheries management policies needs the involvement of the Ministry of Finance etc. Also, in assisting MMAF and the Government more broadly in developing and gaining support for such a vision, greater and more effective engagement with the public and NGOs through public relations and media activities will be needed to highlight the benefits of sustainable marine resources management. Working in this way both from a bottom-up as well as a top-down perspective with stakeholders is more likely to be effective in achieving a shared national vision, making the implementation of that vision easier, more sustainable and less subject to change through political expediency. Finally, in any longer term assistance, the evaluation team were often confronted with situations where project activities were being carried out successfully but there was no measurement, assessment or monitoring of achievements towards meeting overall MRP strategic objectives. For example, excellent work is being done by the Government of Indonesia, assisted by MPAG, in establishing Marine Protected Area and in setting up management plans for those MPAs. However, the impact of those MPAs and their management arrangements on achieving better biological diversity outcomes is not being measured20. The same observation applies to sustainable fisheries. Therefore, in any future assistance, it is recommended that the issue of measuring these broad performance indicators, in addition to the USAID standard indicators, needs consideration. 20 By, for example, regular census of coral cover, fish abundance, species diversity etc Page | 51 SUMMARY OF CONCLUSIONS & RECOMMENDATIONS Over-Arching Recommendation: It is recommended that during the period to 2014, IMACS and MPAG should concentrate on assisting MMAF, at both national and regional level, to develop a clear strategic, integrated action plan to implement its current vision and strategy for sustainable use and conservation of marine fisheries and resources. This would provide a firm foundation for any future USAID assistance. This will require IMACS and, to a lesser extent, MPAG establishing more effective and sustainable working relationship with all levels (National, Provincial and District) of the Ministry of Marine Affairs and Fisheries (MMAF) and other relevant Government agencies. Key Questions Principal Findings and Conclusions Key Recommendations Level 1 Evaluation Question 1: What has been the effectiveness of the IMACS and MPAG projects’ activities in achieving planned results in accordance with Work Plans and Performance Management Plans, including the extent to which projects have produced concrete biodiversity conservation outcomes and activities that comply with USAID’s biodiversity earmark criteria? 1. IMACS and MPAG project activities have either achieved or on track to achieve planned results as measured by Performance Indicators. However, the effectiveness of the activities in achieving sustainable fisheries, concrete biodiversity outcomes or increased resilience to climate change is questionable  Project activities should be considered within the broader context of MMAF institutional capacity development. Therefore, subject to agreement with MMAF regarding the provisions of adequate facilities, MRP partners should move to a secondment mode of technical assistance and institutional capacity building. This would require (a) MRP partners, USAID-I and MMAF jointly identifying key and appropriate counterparts in the MMAF (probably at Director-General or Director level) for IMACS and MPAG national and regional activities21 (b) IMACS and MMAF 2. MPAG has achieved more than 70% of its overall planned results up to 2012, when measured against work plans and the MPAG PMP. This includes 70% of component 1 (establishment of a sustainable MPA national system, 50% of component 2 (advance MPA 21 Unlike IMACS, MPAG have no regional offices despite there being a need for strengthening and working with local institutions including district and province level Government and particular the regional office of MMAF for MPA management (BKKPN). This can be achieved either by integrating MPAG and IMCS activities through the IMACS regional offices or by MPAG having a seconded MPA expert where needed. This is a particular need in the BKKPN Kupang to support the limited capability of staff there. Page | 52 Key Questions Principal Findings and Conclusions Key Recommendations management effectiveness in selected national district MPAs and MPA networks) and 100% of component 3 (host country system assessment and design). Management effectiveness programs are in place that incorporate USAID standard indicators as well as project-specific indicators. However, there are no indicators or supporting programs in place to measure biodiversity outcomes of that management. Therefore the extent to which concrete biodiversity outcomes are being achieved by project activities is not able to be assessed. establishing engagement protocols as to how IMACS and MMAF staff would interact and work together. These engagement protocols may include arrangements for a limited number of key IMACS’s staff to have access to counterpart MMAF staff but does not necessarily mean permanent relocation of IMACS staff to MMAF..  IMACS should be undertaking better institutional assessment and capacity building strategies that are consistent with those adopted by MPAG to ensure that component activities address broader MRP objectives in a consistent way. This should include developing and adopting a consistent approach for all MRP training that includes assisting related Directorate Generals in (a) standardization of competencies to positions (these would be related to the Key Personal Indicators process being developed within MMAF) (b) mapping current competencies of staff (c) undertaking a training needs analysis (d) developing standardized curricula (e) developing an agreed training plan. The process should also be coordinated with MMAF’s planning and budgetary cycle  Regular assessments of the effectiveness of training implemented by IMACS and MPAG should be undertaken and used to both support MMAF institutional development and to design better project supporting activities, including training  Budget should be allocated to funding the implementation of priority actions that resulted from I-CATCH exercises. 3. IMACS project-level activities are generally not well connected to Government agency processes at National (MMAF), Provincial or District (DKP) levels. MPAG activities are better connected to Government processes 4. IMACS has successfully developed a tool (I-CATCH) to assess vulnerability to climate changes at the community level and this tool has now been adopted by MMAF as the program Desa Pesisir Tangguh. However, the planned activities of I-CATCH are behind schedule for some components i.e. number of people trained, number of community plans developed, number of adopted communities, number of local government management plans developed etc 5. IUU-related training was carried out and met target numbers of trainees. However, there is no evidence that training activities related to IUU fishing (observer training, port state measures, improving capacity of law enforcement organizations) has resulted in (a) improved data for control of IUU fishing (b) improvement in inter-agency MCS cooperation or (c) additional legal instruments to support Community Enforcement Networks Question 2: What is the capacity of the IMACS and MPAG project teams to implement activities 6. IMACS’s and MPAG’s professional and administrative capacity is generally good although some issues have  IMACS seek approval to engage a person (possibly at DCOP Page | 53 Key Questions Principal Findings and Conclusions Key Recommendations in an efficient and timely manner, including utilization of grants, subcontracts, and other resources? arisen in the past, such as a high level of IMACS staff turnover which has impacted IMACS ability to efficiently implement IMACS’s activities. However, a significant skills gap is apparent in that there are no IMACS staff with (a) practical senior-level Government experience in fisheries management (strategic and management planning, operational strategies etc) - as distinct from fisheries stock assessment/research (b) experience in implementing Government institutional change. These skills will be important in assisting MMAF to develop a strategic action plan for implanting its vision for sustainable fisheries and marine conservation. level) with practical senior-level governmental experience in institutional change and fisheries management (as distinct from fisheries stock assessment or biology), in a credible fisheries management agency, to work with senior counterpart staff in MMAF on management strategies and objectives, management planning and operational strategies.  Stop the further issue of Small Grant RFAs as well as the commitment of further Small Grant funds and use the unspent funds to implement activities more clearly focused on MRP Objectives.  IMACS and MPAG should review their corporate governance procedures and institute changes, such as proper documentation of high-level meetings with MMAF, that demonstrate best practice and provides the ‘paper-trail’ that is necessary for guiding Government action on outcomes of meetings.  IMACS and MPAG need to better aligned their activities with the planning and budgeting cycle within local government system, especially when considering the timing when project results should be proposed/incorporated into formal proposition to DKPs. 7. The small grant program is not well aligned to overall MRP or project-level objectives although it is now being administered efficiently. 8. IMACS has not applied a high level of corporate management skills in its dealings with MMAF, and, as a result, these should be improved so that they provide an example to MMAF of good corporate management and governance practices Question 3: What were the important policy actions, institutional relationships, and partnering arrangements that facilitated project success? 9. Policy actions, institutional relationships and partnering arrangements pursued by MRP contributors in implementing project activities have generally been effective in achieving project outputs against targets. However, IMACS and MPAG have not paid sufficient attention to the potential of the IBRC in providing the scientific support for MMAF policies and for MPAG and IMACS project activities.  Existing partnering arrangements have been effective in delivering high quality training to support both IMACS and MPAG project activities and should be continued. However, consideration needs to be given by USAID-I, IMACS and MPAG to (a) developing and implementing multi-year agreements with training implementation partners such as NOAA to provide greater planning certainty and ability for partnering institutions (b) expanding training effectiveness activities to ensure that partnering activities are contributing to overall IMACS and MPAG goals, particularly MMAF institutional strengthening.  IMACS and MPAG should utilize the expertise in the IBRC to both inform their own project activities and to encourage better linkages between the IBRC and MMAF. 10. IMACS and MPAG have worked with NOAA and URI￾CRC to deliver high-quality training activities. However, the way in which training needs are identified by IMACS needs attention so that training is better integrated with Government processes. Follow-up assessment of training effectiveness by IMACS and MPAG also needs to be expanded Page | 54 Key Questions Principal Findings and Conclusions Key Recommendations Level 2 Evaluation Question 4: Is the MRP achieving satisfactory results against targets in the following component areas: (a) Strengthening MMAF through institutional development, (b) Improving sustainable fisheries management, (c) Strengthening coastal community resilience and climate change adaptation, (d) Creating and effectively managing Marine Protected Areas, and (e) Improving capacity to reduce Illegal, Unreported and Unregulated Fishing? 11. The MRP is generally achieving satisfactory results against targets in (a) Strengthening coastal community resilience and climate change adaptation and (b) Creating and effectively managing Marine Protected Areas. However, results for (a) Strengthening MMAF through institutional development, (b) Improving sustainable fisheries management and (c) Improving capacity to reduce Illegal, Unreported and Unregulated (IUU) Fishing are less satisfactory  IMACS should take a greater role in developing and adopting a consistent approach22 for all MRP training that (a) is aligned with broader MRP and MMAF strategic goals and (b) is undertaken co-operatively with MMAF. This consistent approach should include assessing the impact of training  IMACS and MPAG needs to better aligned their activities with the planning and budgeting cycle within local government system, especially when considering the timing when project results should be proposed/incorporated into formal proposition to DKPs.  The integration between the sustainable fisheries component of IMACS and MPAG marine conservation area in MPAG should be strengthened, particularly as it relates to EAFM and climate change adaptation  IMACS needs to speed up development of the outreach/communication program with clear key messages, target groups, and specific means/conduit for each group and message. It is important to obtain support and encouragement not only from direct beneficiaries of targeted activities, but also from the general public. Level 3 Evaluation Question 5: Is the MRP achieving its overall objective of sustaining the long-term welfare of coastal Indonesians by reducing the destruction of marine resources and preparing them for climate change impacts and is the MRP strategy of strengthening the capacity of MMAF and providing targeted technical assistance, producing sustainable 12. The MRP is not achieving its overall strategic objectives, partly because those strategic objectives are not clearly articulated and partly because the MRP strategy is ineffective in producing sustainable improvements to ecosystems and institutions  Reporting by IMACS and MPAG against the KRAs of the MRP Results Framework should be requested by USAID. This will allow assessment of how IMACS and MPAG activities are contributing to higher level MRP objectives and key result areas.  As recommended under Level 1 evaluation (see above), IMACS should be required by USAID to adopt better institutional assessment and capacity building strategies that are consistent with those adopted by MPAG. It is 13. There is little or no strategic integration between MRP activities although some limited operational integration occurs. 22 Possibly based on the MPAG approach Page | 55 Key Questions Principal Findings and Conclusions Key Recommendations improvements to ecosystems and institutions? particularly important that such a consistent approach is adopted for training activities.  Implement other Level 1 and Level 2 evaluation recommendations, particularly those related to integration of MRP component activities, improving the effectiveness of service delivery and the institutionalization of MRP component activities. Such initiatives would result in a greater level of achievement of MRP objectives.  IMACS should collaborate with the Coral Triangle Initiative of Coral Reefs, Fisheries and Food Security (CTI-CFF) through Indonesia’s CTI National Coordinating Committee in MMAF and actively participate in its working groups 14. Recommendations related to Planning for long term outcomes  USAID-I should initiate activities in preparation for the implementation of the Host Country System (HCS), or foreign grant management system that has been completed by MPAG, covering the cycle of planning, budgeting, disbursement, program implementation, financial reporting and auditing. These activities should be designed to test the HCS’s effectiveness and feasibility in accommodating various interests from both donor institutions as well from the Government of Indonesia itself. This system could be implemented for better integrating IMACS, MPAG and other project partners’ activities into MMAF and local Government planning and budgetary processes.  In any future assistance, measurement of MRP broad performance indicators related to better marine biodiversity outcomes and achievement of sustainable fisheries, in addition to the USAID standard indicators, should be incorporated into project designs Page | 56 REFERENCES APFIC (2007) Managing Fishing Capacity and IUU Fishing in the Asian Region, Asia Pacific Fisheries Commission/FAO Report RAB 2007/18, 46pp De Young, C. ed. (2006) Review of the state of world marine capture fisheries management: Indian Ocean. FAO Fisheries Technical Paper. No. 488. Rome, FAO, 458p. FAO (2012) The State of World Fisheries and Aquaculture, Fisheries & Aquaculture Department, FAO Rome, 207pp McKinsey Global Institute (2012) The Archipelago Economy: Unleashing Indonesia’s Potential. McKinsey and Company, September 2012, 116pp MMAF (2011) Indonesian Fisheries Book, 2011, Ministry of Marine Affairs and Fisheries/Japan International Cooperation Agency, May 2011, 90pp MMAF (2012) Ministerial Decree, no. 15, August 2012 on MMAF Strategic Planning for 2010-14, Ministry of Marine Affairs and Fisheries, Jakarta MPAG (2013) MPAG News, edition 3, 2013, Marine protected Areas Governance Program, Jakarta, 12pp World Fish Centre (2011) Aquaculture, Fisheries, Poverty and Food Security, WFC Working Paper 2011-65, Penang, 61p. Page | 57 ANNEXES ANNEX 1: STATEMENT OF WORK I. Objective: The objective of the performance evaluation is to provide USAID/Indonesia (USAID) and the Ministry of Marine Affairs and Fisheries (MMAF) an independent review of progress to date of assistance provided by USAID to MMAF under the Marine Resources Program (MRP) and to guide the development of innovations and adaptation of USAID’s assistance in a dynamic environment. The evaluation will be used in the short￾term by USAID and MMAF to modify ongoing assistance, and in the longer-term to plan future assistance strategy. II. Background: In September 2009 following analyses of the needs of MMAF to address key issues related to its responsibility for managing fisheries and coastal areas, USAID authorized the Marine Resources Program (MRP) at a estimated funding level of $31,900,000 to support a five-year program of assistance (2010–2014). The MRP has five components: 6. Strengthen MMAF through institutional development (ID); 7. Improve sustainable fisheries management (SFM); 8. Strengthen coastal community resilience and climate change adaptation (CCR); 9. Create and effectively manage Marine Protected Areas (MPA); and, 10. Improve capacity to reduce Illegal, Unreported and Unregulated Fishing (IUU). These components were chosen to support GOI priorities under the Coral Triangle Initiative (CTI) National Plan of Action (NPOA), which defines Indonesia’s plans to preserve marine biodiversity. They are also key elements in the MMAF Strategic Plan 2010-2014 (RENSTRA), that was completed in February 2010 and focuses on increasing fisheries production and improving the public welfare of coastal communities. Subsequent to authorization of the MRP, USAID designed and contracted for assistance from five sources.  A NGO Consortium led by WWF-US with implementing partners of WWF-Indonesia, Conservation International (CI), The Nature Conservancy (TNC), and Wildlife Conservation Society (WCS) was the first assistance mechanism to start in December 2009 with a focus on MPA system development. The work of the consortium in Indonesia was originally funded as a component under the Regional Coral Triangle Support Partnership as CTSP-Indonesia for 2010 and 2011, and then shifted to a direct USAID/Indonesia bilateral cooperative agreement called the Marine Protected Area Governance (MPAG) project for 2012-2014. 1. An interagency agreement with the U.S. National Oceanic and Atmospheric Administration (NOAA) was started at the same time as the NGO MPA assistance mechanism to provide NOAA training expertise. Initially NOAA worked with MMAF and the NGOs on MPA training (2010- 2012), then also with MMAF and IMACS (2012-2013) on other training for sustainable fisheries data collection (on-board observers), ecosystem approach to fisheries management (EAFM), and port state measures (IUU fishing). 2. A contract called the Indonesia Marine and Climate Support (IMACS) project led by Chemonics International with subcontractors of PNCI, Coastal Resources Center (CRC) of the University of Rhode Island, and RARE is the largest assistance mechanism and started in December 2010. IMACS is designed to provide primary support for three MRP components (MMAF institutional development, sustainable fisheries management, and coastal community resilience) while secondarily Page | 58 supporting the other components and providing program integration and coordination for MRP to USAID and MMAF. 3. An interagency agreement with the U.S. Department of Justice, International Criminal Investigative Training Assistance Program (ICITAP) was implemented during 2010-2011 to provide technical assistance to build the capacity of Indonesian law enforcement organizations to engage and prosecute environmental criminal activities to protect forest and marine ecosystems. A small portion of this project provided limited training to law enforcement personnel dealing with IUU fishing. 4. A university partnership (UP) program was added to the MRP in 2010 through USAID educational funding using the USAID Annual Program Statement (APS) for “Supporting Universities to Partner Across the Pacific.” The first partnership was with the University of California of Los Angeles (UCLA) and Indonesian Universities of Udayana, Diponegoro, and Papua to establish an Indonesia Biodiversity Research Center to advance research and studies in marine biological resource conservation. The second partnership in 2011 was with the University of California of Santa Cruz (UCSC) and Indonesian Universities of Hasanuden and Diponegoro and the Eijkman Institute to establish an Indonesia Marine Biotechnology Partnership. These partnerships added $1.2 million to overall USAID marine sector funding, plus another 30% of this amount in cost share contributions from the universities. The structure of MRP is shown in the following diagram. III. Evaluation Objective Page | 59 The objective of the performance evaluation is to provide USAID/Indonesia (USAID) and the Ministry of Marine Affairs and Fisheries (MMAF) an independent review of progress to date of assistance provided by USAID to MMAF under the Marine Resources Program (MRP), and to guide the development of innovations and adaptation of USAID assistance in a dynamic environment. The evaluation will be conducted at three levels: individual project (outputs achieved), MRP component (results achieved), and overall MRP (strategic approaches). These levels are not separate evaluations, but a suggested way of disaggregating to understand a complex, dynamic environment. The findings and recommendations of the evaluation will be used in the short-term by USAID and MMAF to modify ongoing assistance (improving outputs and results), and in the longer-term to plan future assistance strategy (guiding strategy). At each level the evaluators will identify which elements are most effective, which elements are not, as well as the systems of change driving outcomes. The team will focus on evaluating performance and results achieved versus the stated objectives and goals, identifying which elements have the greatest effect, and recommending which aspects need to be considered for continuation under possible future USAID funding. USAID views this as a “developmental evaluation”23 as defined by Michael Quinn Patton. The evaluation will provide timely feedback for development, generate learning, and support action in the development process. See pages 23-26 in the reference below for a summary table of Patton’s approach. The evaluation team shall provide USAID and MMAF with: a) Analysis of the MRP’s performance: whether the projects, the components and the overall MRP are achieving the planned objectives and having desired effects; b) Direct and indirect benefits: identify achievements to date can be attributed to the MRP activities and determine if the planned results are the only results that are being produced; c) Identification of the factors that are contributing to the achievement of outputs, results and effects; d) Constraints, if any, to achievement of outputs, results and effects, including any recommendations for addressing the constraints; e) Policies, practices, and partnerships which are vital to the success of the MRP; or lack/absence of such that are hindering the success of projects, components, or overall MRP program. f) Gender approaches used and the results disaggregated by gender. g) Sustainability path of the interventions; h) Summary of the findings and conclusions; i) Recommendation for the remaining years of MRP and for a follow-on marine program. IV. Statement of Work The evaluators shall analyze the performance of MRP at three levels: Level 1 - Individual project (outputs achieved); Level 2 - MRP component (results achieved); and Level 3 - Overall MRP (strategic approaches). Level 1 - Individual Projects Evaluation The evaluation of individual projects will focus on the progress of the two main implementing mechanisms that USAID used to provide technical assistance under the MRP – the IMACS and the CTSP-I/MPAG 23 Michael Quinn Patton, “Developmental Evaluation: Applying Complexity to Enhance Innovation and Use,” The Guilford Press, New York, 2011. Page | 60 projects. As described below, NOAA provided training to MMAF working cooperatively with IMACS and with CTSP-I/MPAG for sustainable fisheries and MPA management, respectively. ICITAP provided one training course in cooperation with CTSP-I/MPAG for conservation enforcement. These joint activities will be covered under the IMACS and CTSP-I/MPAG evaluations. The University Partnerships (UP) have been evaluated separately24 and the UP evaluation report should be used as an input to this evaluation. This level 1 evaluation will include but not be limited to: a) effectiveness of the projects’ activities in achieving planned results in accordance with Work Plans and Performance Management Plans, including the extent to which projects have produced concrete biodiversity conservation outcomes and activities comply with USAID’s biodiversity earmark criteria; b) capacity of project teams to implement activities in an efficient and timely manner (including utilization of grants, subcontracts, and other resources); and, c) important policy actions, institutional relationships, and partnering arrangements that facilitated project success. The evaluation will focus on these three primary questions. However, under the primary questions different contributing questions are suggested below for IMACS and CTSP-I/MPAG, since the two projects have different scopes of work. The following guide questions cover questions of interest to USAID and MMAF, but these questions should not constrain the evaluators from pursuing other relevant issues that might be identified within a developmental evaluation approach. A. IMACS USAID’s Indonesia Marine and Climate Support (IMACS) project, implemented by Chemonics International Inc., is designed to strengthen MMAF for sustainable management of Indonesia’s marine and coastal ecosystems. IMACS strengthens the management capacity of the MMAF, enhances the MMAF’s engagement with local communities and the private sector through open and transparent governance, and provides technical support for key activities that support marine resources and communities. IMACS has four major components: 1. Institutional Development (ID) of the MMAF 2. Sustainable Fisheries Management (SFM) 3. Coastal Community Resilience (CCR) and Climate Change Adaptation 4. Program Integration, Coordination and Administrative Support The evaluation of the implementation progress of IMACS should address: 1. Effectiveness of the IMACS project’s activities in achieving planned results and producing concrete biodiversity conservation outcomes. a. What progress is being made toward institutional development of the MMAF, including: strengthening policies and regulations; improving technical training; improving training design and implementation; strengthening MMAF communications; and responding to MMAF priority needs? b. What progress is being made toward sustainable fisheries management (SFM or EAFM), including: raising awareness and support for SFM across MMAF; improving data systems to 24 Evaluation of the University Partnerships Program: Phase One – Partnerships #1 and #2, International Business & Technical Consultants and JBS International, USAID/Indonesia, November 13, 2012. Page | 61 support SFM decision making; demonstrating SFM activities; increasing understanding of SFM; and, expanding partnerships for SFM? c. What progress is being made toward coastal community resilience (CCR) for climate change adaption and disaster risk reduction, including: strengthening policies and regulation; raising awareness and support for CCR across MMAF; improving data systems to support CCR; demonstrating CCR activities; increasing understanding of CCR; and, expanding partnerships for CCR? d. Which elements of the program are achieving the best results? Which elements are not? Which aspects of project design need to be adjusted? e. How well is this program achieving its objectives? What factors contribute to the achievement of its objectives? Are there challenges and/or hindrances? What are these and how are they addressed? 2. Capacity of IMACS project team to implement activities in an efficient and timely manner. a. Has IMACS’ assistance been organized and managed in an efficient manner to plan and implement activities and to achieve planned results? b. What is the level of program engagement of the MMAF at various levels of government? Is it a relevant engagement? c. Is there an effective system for monitoring and reporting of performance that meets the needs of both USAID and MMAF? d. What is the possibility that the interventions made under this program will become sustainable? 3. Important policy actions, institutional relationships, and partnering arrangements. a. How effective is IMACS in collaborating with the other MRP projects? What key initiatives would benefit from stronger collaboration? b. How effective is IMACS’ integration and coordination of the MRP? Is the implementation of all the MRP components being well coordinated? Is the implementation of the MRP coordinated with related activities being carried out by MMAF? Is there coordinated work planning and implementation? c. How effective is IMACS’ partnership with the MMAF? d. How effective is IMACS and MMAF’s engagement with local communities and the private sector? B. CTSP-I/MPAG The former Coral Triangle Support Partnership–Indonesia Program (CTSP-I in years 1 and 2) and now the Marine Protected Areas Governance (MPAG in years 3 - 5) projects are focused on improving marine protected area (MPA) governance through strengthened data analysis, policy development, and capacity building for improved MPA management to yield the optimal fisheries and biodiversity conservation benefits. The MPAG project continues the MPA management support USAID/Indonesia has previously provided under the Coral Triangle Support Partnership-Indonesia Program (CTSP-I) that was awarded and managed through USAID/RDMA (Bangkok). The MPAG project is implemented nationally and in eight priority field sites by a consortium of NGOs led by WWF-US and including WWF-I, CI, TNC, WCS and Coral Triangle Center (CTC). MPAG provides technical assistance, training, and other assistance to for the MMAF in order to strengthen MPA management through the development of a national MPA system linked with priority MPA field sites in both national and local MPAs. The objectives of the MPAG project in providing support to the Ministry of Marine Affairs and Fisheries (MMAF) include achieving the following: 1. Designing a sustainably financed national MPA system that expands MPA coverage to achieve the target of 20 million hectares of MPA coverage by 2020 and includes representation of the nation’s highest priority biodiversity and fisheries resources; Page | 62 2. Advancing MPA effectiveness in Priority Geographies through direct field action and linking these high priority activities to national programs and to long-term sustainable sources of finance to expand and improve MPA capacity, coverage, and effectiveness. 3. Supporting the preparation of Host Country System (HCS) implementation through strengthening government capacity, and developing the HCS design for MPA management The evaluation of the implementation progress of CTSP-I/MPAG should address: 1. Effectiveness of the project’s activities in achieving planned results and producing concrete biodiversity conservation outcomes. a. What progress is being made toward establishment of a sustainable MPA National System, including: review and formulation of MPA policies and regulations; building national and local capacity for sustainable MPA management; developing a decision support system; integrating MPAs into an ecosystem approach to fisheries management and into marine spatial planning; and developing MPA sustainable financing? b. What progress is being made toward increasing management effectiveness in the priority geographies, including: the establishing new MPAs; strengthening the management of MPAs; and strengthening of MPA Networks? c. What progress is being made toward assessment and design of a Host Country System for MPAs grants, including: assessing MMAF capability to implement HCS; training of MMAF personnel in relation to HCS design and implementation; and, designing a HCS for MPAG in Indonesia? d. Is the project achieving planned results, according to its Performance Management Plan (PMP) approved by USAID? 2. Capacity of the MPAG project team to implement activities in an efficient and timely manner. a. Has the NGO consortium been organized and managed in an efficient manner to plan and implement activities and to achieve planned results? b. Has the NGO consortium reached enough stakeholders to achieve significant results that can influence the government and communities and take a lead or set a trend for other institutions or groups to follow? c. What is the level of project engagement of the MMAF at various levels of government? Is it a relevant and effective engagement? d. Is there an effective system for monitoring and reporting of performance that meets the needs of USAID and MMAF? e. What is the possibility that the interventions made under this project will become sustainable? 3. Important policy actions, institutional relationships, and partnering arrangements. a. How effective has CTSP-I/MPAG been in collaborating with the other MRP projects? What key initiatives would benefit from stronger collaboration? b. How effective is the CTSP-I/MPAG partnership with the MMAF? c. Is there sufficient and significant integration between this project and other donors outside of USAID? C. NOAA USAID/Indonesia provides funds to NOAA through the regional agreement established by the USAID Regional Development Mission for Asia (RDMA) based in Bangkok. Utilizing these funds, plus its own matching contributions, NOAA provides training to MMAF in technical areas that include: 1. Marine Protected Areas Management - with CTSP-I/MPAG 2. Ecosystems Approach to Fisheries Management (EAFM) – with IMACS 3. Port State Measures (PSM) – with IMACS Page | 63 4. On-board Observers Program – with IMACS D. ICITAP The U.S. Department of Justice ICITAP Indonesian Program implemented USAID-funded activities to build the capacity of the Indonesian law enforcement agencies to address environmental crimes that threatened forest and marine ecosystems. ICITAP project was active for over two years, during which time it implemented with CTSP-I/MPAG one IUU training workshop for a Coastal Community Network (CCN) in Berau, East Kalimantan. The intent of the training was to develop the investigative and forensic capabilities of the Indonesian National Police (INP), the Directorate General of Monitoring and Surveillance (PSDKP) of MMAF, and other relevant enforcement authorities. The three main objectives of the ICITAP project were: multi-agency coordination support and training; training the MMAF Patrol Unit and the INP Marine Police Unit; and, training Coastal Community Networks (CCN). E. University Partnerships (UP) The university partnerships were added into the MRP by USAID in order to establish better linkages and strengthen Indonesian universities to support applied research and to provide technical assistance to MMAF and local governments for improved marine resources management. The two partnerships with UCLA (Indonesia Biodiversity Research Center - IBRC) and UCSC (Biotechnology Research Partnership) have been focused on biological diversity conservation. However, in phase 2 funding for IBRC starting in 2012 USAID expanding IBRC to work in sustainable fisheries management with private industry partnership. IMACS is also developing university partnerships with the State Universities in SulTra and NTB to more broadly support local government via its subcontract with the Coastal Resources Center of the University of Rhode Island. The evaluation of the implementation progress of University Partnership Marine Program for IBRC was completed by an independent team in November 2012 and will be provided to the team by USAID. Level 2 - MRP Component Results The evaluation of MRP component results will review the combined results of the individual projects under each of the five MRP components. 1. Strengthen MMAF through institutional development (ID) Based on the December 2008 assessment of MMAF, Enhancing Government Effectiveness (EGE), USAID provided technical assistance and training focused on improving MMAF’s weaknesses identified in the EGE assessment, and on strengthening overall operations and management. Priorities included improved policy formulation based on effective data management, decentralized operations, integrated planning, a public outreach campaign to engage the support of the Indonesian people, and improved extension services. The evaluation of the implementation results of the MRP ID component should address: a. Technical training to MMAF staff for the execution of their responsibilities, including courses covering: fishing licensing/permitting procedures; fishing extension services focused on sustainable fishing practices; monitoring/control/surveillance of fishing; and port state measures. b. Assistance for legislative and regulatory framework to ensure it is sufficient for the MMAF to execute its mandate to protect marine resources and fisheries, and provide technical assistance to develop new or revised policies and regulations. Page | 64 c. Support to improve the execution of the MMAF national five-year strategy. This involves strengthening coordination between the central and district offices and preparing districts to become more accountable, especially in data gathering and processing. Is MRP achieving results? Its initial illustrative five-year targets are:  24 policies, laws, agreements, procedures or regulations promoting sustainable natural resources management conservation or climate change are endorsed by the GoI.  10 priority nationwide plans of operation approved jointly by DGs in MMAF.  1,430 MMAF staff and fisheries/climate change support or service personnel trained in courses pertaining to their operational responsibilities.  7 new training modules adapted into curriculum and used by Indonesian agencies and organizations. 2. Improve sustainable fisheries management (SFM) Under this component, USAID provides support to MMAF in its efforts to oversee a locally implemented, nationwide program in ecosystems approach to fisheries management (EAFM). EAFM is a set of principles for managing marine resources that considers broader environmental and ecosystems issues in fisheries. This holistic approach supports the sustainable utilization of marine resources focusing not only on managing target production species, but also the habitat and associated food chain that supports them. Activities range from establishing new policies and procedures at the central and district levels, to working with local fishers to utilize equipment with lower environmental impact. This component also involves work with the private sector all along the fisheries value chain from production to market and promotes demand for marine products that have been sustainably produced. The evaluation of the implementation results of the MRP SFM component should address: a. Promotion of sustainable fishing practices (e.g. use of circle hooks, hook and line fishing to reduce destructive practices (e.g. trawling, purse seine, long line) that damage habitats, threaten non-target species, and work with the private sector to adopt integrated best fisheries management practices in critical fisheries; b. Improvement of central and district level policies or regulations to support ecosystems-based fisheries management, including fishing licensing and permitting, zoning, and allowable gear policies; c. Improvement of GOI monitoring and data collection of fish catch to include information on fishing locations that can help forecast available stocks in each management area for planning annual fishing permits in each area; d. Development of livelihood programs that offer alternatives during fishing moratorium periods to reduce the rate of destructive fishing practices and lessen environmental impact. Activities may include aquaculture and mariculture fish breeding and rearing practices that take advantage of organic or cost-saving processes. Is MRP achieving results? Its initial illustrative five-year targets are:  5 million ha areas with improved fisheries management  3 new policies/ regulations that support sustainable fishing practices implemented  5 public private partnerships to promote sustainable fisheries management 2. Strengthen coastal community resilience and climate change adaptation (CCR). Page | 65 Coastal Community Resilience is the capacity of a community at risk from coastal hazards to adapt to and influence the course of environmental, social, and economic change. Climate change adaptations are actions taken to help communities and ecosystems moderate, cope with, or take advantage of actual or expected changes in climate conditions. Elements that contribute to community resilience to climate change impact and disaster risks include governance, economy, coastal resource management, land use, risk awareness, warning and evacuation, emergency response, and adaptation or recovery. Beyond sustainable management of ecosystems addressed in the other components, this component strives to reduce exposure along shorelines and infrastructure, engage families in diversified livelihoods, strengthen services to protect human health and safety during disasters, and improve water resource and land use in coastal areas. The evaluation of the implementation results of the MRP CCR component should address: a. Increased awareness of vulnerable coastal population of potential disaster and climate change impacts and risk reduction practices in select communities. Activities may include technical assistance to establish climate change action plans, to create public outreach units in local government offices, and to execute public awareness campaigns. b. Increased resilience of coastal communities to the economic impacts of climate change and disasters through alternative livelihoods such as seaweed cultivation, aquaculture, and mariculture. These livelihood activities diversify income opportunities and strengthen food security, addressing opportunities in production, value-added processing, and marketing. c. Assistance to local governments to develop policies, budgets and procedures that protect the coast from degradation for disaster preparedness and climate change adaptation. Is MRP achieving results? Its initial illustrative five-year targets for CCR were:  10,000 communities and stakeholders with increased capacity to adapt to the impacts of climate variability  65 communities adopt strategies to enhance marine resources, community security, supplement economic livelihoods, or diversify income opportunities.  10 local governments establish plans and budgets for climate change adaptation, disaster risk reduction, and community resilience 3. Create and effectively manage Marine Protected Areas (MPA). Under this component, USAID is providing technical assistance and training to support identification, legalization, and effective management of a network of scientifically determined high biodiversity sites with high value fish spawning grounds. As a result of this component, the most valuable marine biological resource areas will be preserved for sustainable use. The evaluation of the implementation results of the MRP MPA component should address: a. Assistance to the GOI for increasing number of high priority marine protected areas that are areas of highest marine biodiversity or highest value fish spawning and aggregation sites and for improving research and science inputs for decision making. b. Promotion of the effective management of MPAs by: engaging MMAF, local governments, universities, and communities; defining roles and responsibilities of each stakeholder; developing multi-stakeholder management agreements; and providing training. d. Assistance to local governments for better allocating adequate resources and moving toward sustainable financing of MPAs. Is MRP achieving results? Its initial illustrative five-year targets for MPA are:  At least 2.5 million ha of newly designed MPAs in prioritized geographical areas and 5 million ha of existing MPAs with effective and sustainable management Page | 66  A sustainably financed national MPA system designed and established to support the effective and sustainable management of MPA  850 people receive USG-supported training in natural resource management and/or biodiversity conservation 4. Improve capacity to reduce Illegal, Unreported and Unregulated Fishing (IUU) Reducing illegal, unreported, and unregulated fishing requires strengthened monitoring of fishing practices and effective enforcement of laws governing fishing practices. Regulations are meaningless without enforcement. Under this component, the law enforcement chain is being strengthened to more effectively detect, investigate, and prosecute crimes. Improved communication and coordination among ministries is critical. The evaluation of the implementation results of the MRP IUU component should address: a. Improvement of fisheries database for MMAF to better understand the need for control of fishing level of effort and the relative importance of illegal, unregulated, and unreported fishing. b. Improvement in policy, regulatory, and enforcement cooperation among MMAF officials and Indonesian law enforcement agencies, and among national and local government entities to address marine environmental crimes and stronger enforcement of laws. c. Training of coastal Community Enforcement Networks on environmental laws, observation, documentation, species identification, and reporting of suspicious activities related to marine environmental crimes. Is MRP achieving results? Its initial illustrative five-year targets for IUU are:  Improve data collection systems for management of capture fisheries nationally in 3 fisheries management area and locally in 2 provinces and 10 districts  5 coastal community enforcement networks trained and operating effectively Level 3 - Overall MRP Strategic Approaches The objective of the Marine Resources Program (MRP) is to sustain the long-term welfare of coastal Indonesians by reducing the destruction of marine resources and preparing them for climate change impacts. To achieve this, MRP’s strategy is to strengthen the capacity of MMAF and provide technical support for key activities that support marine resources and coastal communities. This strategy involves a partnership with the GOI, through MMAF, which reflects its financial commitment toward sustainable marine resource management. The implementation of all five components described above is expected to make modest but nonetheless significant progress to:  Achieve good management effectiveness in 7.5 million ha of new and existing high biodiversity marine protected areas.  Establish improved fisheries management covering 5 million ha in three fisheries management areas.  Improve coastal community resilience in accordance with climate change vulnerability assessments in 100 villages within 10 districts and 2 provinces.  Improve the performance of MMAF across all components of the Marine Resources Program as measured by survey and performance audit. The evaluation of MRP strategic approaches will look for broad sustainable effects on ecosystems and institutions from all MRP activities. Page | 67 Illustrative issues and questions for the team to pursue include:  Has the leadership of MMAF embraced MRP? Has MRP helped MMAF to develop and communicate a clearer strategic vision?  Which elements of the program are the most successful? Which elements are not and why? Which aspects of the overall program design need to be adjusted?  How well is this program achieving its objectives? What factors contribute most to the achievement of its objectives? Are there challenges and/or hindrances? What are these and how are these addressed?  Are the existing programs, which comprise the MRP sufficient for the MRP to achieve its objectives? Are there other approaches, projects or institutions that could be included to help achieve the objectives further? V. Methodologies The evaluators should conduct a performance evaluation following the Statement of Work and USAID evaluation policy.25 The team should propose an evaluation design and describe the methodology to be used in its Work Plan (First Deliverable). As discussed above in Section III, USAID views this as a developmental evaluation. Accordingly, the methodology should use systems thinking to capture complex systems dynamics, be flexible in measures and tracking mechanisms, and pay attention to unexpected consequences. The methodology will be presented in the Work Plan, and discussed with and approved by USAID once the team has arrived in Indonesia. The evaluation team is expected to review relevant documents, contracts, and reports from each component and project, as well as existing monitoring and evaluation data collected by the project implementers. The team will have full access to data collected by IMACS and MPAG. They will interview USAID staff, partners, and stakeholders of each project. They will also interview MMAF staff. The evaluation team will travel to selected field sites, identify and interview communities and organizations receiving support from USAID through the MRP ensuring geographic representation. During the site visits, the team will observe on-the ground activities. Travel site selection (to be approved by USAID as part of the Work Plan) should be determined by what is most cost effective and practical to get knowledge and information about the projects without sacrificing the quality and validity of data. The following is a list of possible resources for this evaluation: 1. MMAF 2. Other GOI institutions at the national and local levels 3. Subcontractors 4. Grantees 5. Private sector representatives 6. Community members 7. MPAG, IMACS 8. USAID VI. Geographic Areas 25 For USAID evaluation policy see http://www.usaid.gov/evaluation Page | 68 The final locations that the evaluation team will visit should be proposed in the Work Plan and shall be decided by USAID from among the MRP’s current or recent intervention areas. VII. Deliverables 1. Work plan for the evaluation including design (evaluation framework) and time-frame a. Work Plan written report, including evaluation design, methodology, key questions, data collection plan including travel of team, and the schedule and calendar for the evaluation b. PowerPoint presentation of the Work Plan for discussion with USAID and stakeholders at the beginning of the evaluation. 2. PowerPoint presentation of findings to USAID and MMAF 3. Draft Evaluation Report a. Executive Summary with key findings, conclusions and recommendations. b. Table of Contents and Acronym List c. Introduction covering purpose, audience, and description of the scope undertaken and the methodology used d. Current performance of the MRP including how well the program is meeting its objectives and targets and issues faced e. Analysis (technical and administrative) of the IMACS and CTSP-I/MPAG projects comprising the MRP and the achievements of each of these projects, including any shortcomings or areas for enhancement f. Analysis of MRP components and the factors contributing to the achievement of results and any shortcomings g. Policy, partnership, or other overarching issues and their effect on the program h. Summary of the findings and conclusions i. Recommendations on the present performance and any follow-on programs, including geographic focus to improve results and to facilitate management coordination with other programs and on sustainability and exit strategy j. Annexes: Statement of Work, methodology, schedules, meetings, interview lists, bibliographical references and tables in concise readable formats. Note: Per USAID policy, all data should be disaggregated by gender as appropriate. 4. Final report The final report will have the same format and materials as the draft report and will incorporate USAID comments and suggestions. Submission will include 2 hard copies and electronic versions in both MS Word and PDF formats. Total page limit of the main report, excluding the Table of Contents, Acronyms List, Tables and Charts, and Annexes, is 30 pages. The report will be submitted to the USAID COR. In addition, all quantitative data collected by the evaluation team shall be submitted to the COR in an easily readable electronic format, organized and documented for use by those not familiar with MRP. The report and accompanying documentation will be owned by USAID and may not be used without USAID’s permission. VIII.Resources, Procedure, and Schedule A. Data Sources The evaluation team will review the following documents: 1. Program Contracts or Agreements 2. Annual Work Plans 3. Annual and Quarterly Reports Page | 69 4. Key deliverables and other outputs 5. Results Frameworks 6. Performance Management Plans 7. Geographic Priorities for Marine Biodiversity Conservation, MMAF, 2012 8. FA 118/119 Forest and Marine Biodiversity Assessment 9. Other key documents USAID will give a briefing to the team at the start of the evaluation’s work in Indonesia. They will be introduced to the key leaders of each program/project who will then arrange the schedule of visits to selected MMAF offices and project sites. Meetings and interviews with priority government partners, stakeholders, community groups, private sector partners, and experts/consultants will be set during the first few days of the evaluation period. B. Duration and Timing of the Evaluation The evaluation of the MRP will be completed in 45 workdays. An illustrative schedule is below. Task Duration Review background documents and prepare draft work plan 5 days Travel to Indonesia 2 days Work Planning. Team planning meeting with USAID and others. Work plan rand time frame reviewed by USAID agreed on. Schedule and itinerary finalized 4 days Meetings with MMAF representatives, and key officers and staff of MRP 5 days Field visits, meetings, interviews, and other data gathering activities 11 days Preparation of draft report 5 days Debrief meetings. Presentation of key preliminary findings with USAID, MMAF, other key stakeholders 2 days Depart Indonesia 2 days Continue to draft report and receive comments from USAID 4 days Final Report preparation 5 days Total 45 days C. Logistical Support The evaluation team members are responsible for arranging their international travel and U.S. logistical support. In Indonesia, the IMACS project will provide the team limited administrative support, travel logistics assistance, and help ensure timely access to key individuals to be interviewed. IX. Team Composition The evaluation team is to be comprised of experts/evaluators who are independent of the project implementers and the implementers’ staff. Written disclosures of conflicts of interest will be required of all core team members. The core team, which is responsible for preparation of reports and presentations, and submissions to USAID, will consist of one expatriate and two national staff. USAID and MMAF staff shall participate as resource persons for the team. They will be funded separately. The evaluation team should have technical expertise in evaluation design and implementation and in marine, coastal and fisheries resource management. Page | 70 A. Team Leader, Evaluation Specialist with Knowledge of Marine and Coastal Resources (U.S. national) The Team Leader should have a strong background and experience in evaluation design and implementation and a good knowledge of USAID evaluation policy. The Team Leader should have managerial and operational experience with complex, national and sub-national activities involving coordination of multiple program partner institutions; subject matter expertise in coastal-marine and fisheries resource issues, environmental policy and institutional development, sustainable management challenges, policy and market constraints, and effective approaches; and strong task output leadership and communications skills. Familiarity with the political, social and cultural context of Indonesia and Bahasa language is a strong plus. A U.S. national is preferred. The Team Leader will be responsible for management of the team and for the timely preparation and submission of all deliverables. The Team Leader will focus on preparation of the Level 3 evaluation findings, preparation of ID components of the Level 2, preparation of the Level 1 evaluation of IMACS institutional development component, as well as synthesis of all findings and recommendations of the team. The Team Leader will also ensure the efficient operation of the team and good relationships with USAID and MMAF. B. Sustainable Fisheries Specialist (Indonesian National) The Sustainable Fisheries Specialist should have strong sustainable fisheries expertise, especially related to practices to reduce overfishing and destructive fishing practices. The Specialist should also have an understanding of climate change and activities for strengthening coastal communities to improve their economic and social resiliency The Specialist should have strong written and oral communications skills in English. Familiarity with the Indonesian political, social, and cultural context and of Bahasa Indonesia is a strong plus. The Sustainable Fisheries Specialist will be responsible for preparation of Level 2 SFM, CCR, and IUU components evaluation and leading the Level 1 evaluation of the IMACS projects. C. Biodiversity Conservation Specialist (Indonesian National) The Biodiversity Conservation Specialist should have strong expertise and experience in marine protected area management, institutional development and strengthening complex institutions in developing countries. The Specialist must demonstrate exceptional program leadership and management skills as well as written and oral communications skills in English. Familiarity with the Indonesian political, social, and cultural context and good communications skills in Bahasa Indonesia are a strong plus. The Biodiversity Conservation Specialist will be responsible for preparation of Level 2 MPA component evaluation and leading the Level 1 evaluation of the CTSP-I/MPAG project. D. USAID Resource Person (U.S. National) The USAID Resource Person will be a USAID employee or contractor who has broad knowledge of the marine sector, extensive USAID project management experience, and thorough understanding of USAID regulations and procedures for project design and implementation, performance management, and evaluation. Familiarity with the Indonesian political, social, and cultural context is a strong plus. The USAID Resource Person will provide advice and guidance to the core team on USAID procedures for monitoring and evaluation and on relevant USAID technical experience. The USAID resource person will provide the team with access to relevant USAID project and program information, and analysis of institutional and management issues. Page | 71 E. MMAF Resource Person (Indonesian National) MMAF Resource Person will be an MMAF employee who has broad knowledge of the marine sector, extensive MMF project management experience, and thorough understanding of MMAF regulations and procedures for project design and implementation, performance management, and evaluation. The MMAF Resource Person must demonstrate strong expertise and experience in institutional development and strengthening complex institutions. The Resource Person should have good program leadership and management skills as well as written and oral communications skills in English. Familiarity with the Indonesian political, social, and cultural context and good communications skills in Bahasa Indonesia are a strong plus. The MMAF Resource Person will provide advice and guidance to the core team on MMAF and Government of Indonesia procedures for monitoring and evaluation on relevant MMAF technical experience. The MMAF Resource Person will provide the team, with access to relevant MMAF project and program information, and with analysis of local government and NGO institutional relationships. Page | 72 ANNEX 2: EVALUATION DESIGN AND METHODS (a) Questions posed at each level of Evaluation: Level 1 Evaluation For the Level 1 evaluation (project/activity level), the primary questions posed for the IMACS and CTSP-I/MPAG components were: a) effectiveness of the projects’ activities in achieving planned results in accordance with Work Plans and Performance Management Plans, including the extent to which projects have produced concrete biodiversity conservation outcomes and activities comply with USAID’s biodiversity earmark criteria; b) capacity of project teams to implement activities in an efficient and timely manner (including utilization of grants, subcontracts, and other resources); and, c) important policy actions, institutional relationships, and partnering arrangements that facilitated project success Level 1 evaluation assessed the following projects/activities: IMACS 1. Activities related to Institutional Development of the Ministry of Marine Affairs and Fisheries 2. Activities related to Sustainable Fisheries Management (SFM) 3. Activities related to Coastal Community Resilience (CCR) and Climate Change Adaptation 4. Activities related to Program Integration, Coordination and Administrative Support MPAG 1. Activities related to Designing sustainably financed nation MPA system 2. Activities related to Advancing MPA effectiveness in Priority Geographies 3. Activities related to Supporting the preparation of Host Country System (HCS) implementation NOAA 1. Activities related to providing training on MPA Management, EAFM, PSM and On-Board Observers Program ICITAP 1. Activities related to Multi-agency coordination support and training 2. Activities related to Training the MMAF Patrol Unit and the INP Marine Policy Unit 3. Activities related to Training Coastal Community Networks (CCN) Page | 73 The ICITAP component was addressed through discussions with relevant MMAF recipient agencies but was not further followed up given the length of time that had elapsed since training. However, the outcomes of this component were addressed during discussions with other MRP participants and partners. University Partnership (UP) activities 1. UCLA – Activities related to the Indonesia Biodiversity Research Center 2. UCSC – Activities related to Biotechnology Research Partnership 3. Coastal Resources Center URI activities – Univ in Sultra and NTB The University Partnerships (UP) has recently been evaluated separately and, as a result the UP activities were not a focus of this evaluation. However, the UP evaluation report was used as an input and as background for the team’s work and was, again, addressed during discussions with other MRP participants and partners. Level 2 Evaluation As part of the Level 2 evaluation (MRP Component level), the Evaluation Team reviewed the combined results of the individual projects under each of the five MRP components and addressed whether MRP is achieving satisfactory outcomes in the following areas: Strengthening MMAF through institutional development (ID) a. Technical training to MMAF staff b. Assistance for legislative and regulatory framework development and implementation c. Supporting the development of the MMAF national five-year strategy. Improving sustainable fisheries management (SFM) d. Promotion of sustainable fishing practices e. Improvement of central and district level policies or regulations to support ecosystems￾based fisheries management f. Improvement of GOI monitoring and data collection of fish catch g. Development of livelihood programs that offer alternatives during fishing moratorium. Strengthening coastal community resilience and climate change adaptation (CCR) h. Increased awareness of vulnerable coastal population of potential disaster and climate change impacts and risk reduction practices in select communities. i. Increased resilience of coastal communities to the economic impacts of climate change and disasters j. Assistance to local governments to develop policies, budgets and procedures that protect the coast from climate change-related impacts. Creating and effectively managing Marine Protected Areas (MPA). k. Assistance to the GOI for increasing number of high priority marine protected areas l. Promotion of the effective management of MPAs Page | 74 m. Assistance to local governments for better allocating adequate resources and moving toward sustainable financing of MPAs. Improving capacity to reduce Illegal, Unreported and Unregulated Fishing (IUU) n. Improvement of fisheries database for MMAF to better understand the need for control of fishing level of effort and IUU fishing. o. Improvement in policy, regulatory, and enforcement cooperation among MMAF officials and Indonesian law enforcement agencies and local government p. Training of coastal Community Enforcement Networks on environmental laws Level 3 Evaluation As part of the Level 3 evaluation (overall MRP Strategic Approaches), the Evaluation Team sought information on broad sustainable effects on ecosystems and institutions from all MRP activities and posed the following questions: a. Has the leadership of MMAF embraced MRP? Has MRP helped MMAF to develop and communicate a clearer strategic vision? b. Which elements of the program are the most successful and which were not and why? c. How well is the MRP achieving its strategic objectives? d. Are the existing programs, which comprise the MRP sufficient for the MRP to achieve its objectives? (b) Overall Approach to KIIs: A structured approach to interviews with Key Informants, subsequent analysis and follow-up interviews is to be used. This structured approach has the following components:  The prior development of questions for initial KIIs that will guide the interview. These questions include BOTH the general questions posed by the SOW (and listed in Section 3 of this Work Plan) and also interviewee-specific questions as detailed below  The posing of the question to Key Informants. The approach taken will be a ‘snowball’ questioning technique (as opposed to a tightly structured approach) where interviewees will be free to comment on issues that are both directly and indirectly related to the question posed. Although more time consuming, the use of this method should bring out issues that may have escaped the development of the questions.  Summarizing the responses, for each interview, in a matrix where responses are mapped against question themes.  Analysis of all responses and the development of major ‘hypotheses’, or preliminary conclusions on the issues posed by the SOW. These will be developed during the period 16-20 February (see Annex A)  The testing of those hypotheses by the subsequent re-interviewing of Key Informants with KIs being selected according to their relevance to the hypothesis in question. This process will occur during the period February 21-25 (see Annex A)  Verification, or rejection, of the developed hypotheses on the basis of the Hypothesis Testing (HT) tool, including interviews and the subsequent analysis. Page | 75 General Questions – posed to all KIs These general questions are posed by the SOW and are included in Section 3 of this work plan (above}. The questions also address all 3 levels of the Evaluation (i.e. project level, component level and program level). At the project level, the primary questions posed for the IMACS and CTSP-I/MPAG components are:  What, in your opinion, has been the effectiveness of the projects’ activities in achieving planned results in accordance with Work Plans and Performance Management Plans, including the extent to which projects have produced concrete biodiversity conservation outcomes and activities comply with USAID’s biodiversity earmark criteria;  Is the capacity of the project teams sufficient to implement activities in an efficient and timely manner (including utilization of grants, subcontracts, and other resources); and,  What are the important policy actions, institutional relationships, and partnering arrangements that facilitated project success At the MRP Component Level, the Evaluation Team will review the combined results of the individual projects under each of the five MRP components and pose the following questions to those KIs who are involved in each component: Component 1: Strengthening MMAF through institutional development (ID) Is the component achieving satisfactory outcomes in the following areas?  Technical training to MMAF staff  Assistance for legislative and regulatory framework development and implementation  Supporting the development of the MMAF national five-year strategy? Component 2: Improving sustainable fisheries management (SFM) Is the component achieving satisfactory outcomes in the following areas?  Promotion of sustainable fishing practices  Improvement of central and district level policies or regulations to support ecosystems￾based fisheries management  Improvement of GOI monitoring and data collection of fish catch  Development of livelihood programs that offer alternatives during fishing moratorium. Component 3: Strengthening coastal community resilience and climate change adaptation (CCR) Is the component achieving satisfactory outcomes in the following areas?  Increased awareness of vulnerable coastal population of potential disaster and climate change impacts and risk reduction practices in select communities.  Increased resilience of coastal communities to the economic impacts of climate change and disasters  Assistance to local governments to develop policies, budgets and procedures that protect the coast from climate change-related impacts. Page | 76 Component 4: Creating and effectively managing Marine Protected Areas (MPA). Is the component achieving satisfactory outcomes in the following areas?  Assistance to the GOI for increasing number of high priority marine protected areas  Promotion of the effective management of MPAs  Assistance to local governments for better allocating adequate resources and moving toward sustainable financing of MPAs. Component 5: Improving capacity to reduce Illegal, Unreported and Unregulated Fishing (IUU) Is the component achieving satisfactory outcomes in the following areas?  Improvement of fisheries database for MMAF to better understand the need for control of fishing level of effort and IUU fishing.  Improvement in policy, regulatory, and enforcement cooperation among MMAF officials and Indonesian law enforcement agencies and local government  Training of coastal Community Enforcement Networks on environmental laws At the level of evaluation of the overall MRP Strategic Approaches, the Evaluation Team will seek information on broad sustainable effects on ecosystems and institutions from all MRP activities and pose the following questions:  Has the leadership of MMAF embraced MRP? Has MRP helped MMAF to develop and communicate a clearer strategic vision?  Which elements of the program are the most successful and which were not and why?  How well is the MRP achieving its strategic objectives?  Are the existing programs, which comprise the MRP sufficient for the MRP to achieve its objectives? Specific, KI-focused questions 1. Questions for UCLA The focus of the evaluation is on the Mare Resources Program, how well it is achieving its objectives, problems encountered and, importantly, what lessons can be learned, or what changes can be made to direct USAID’s future assistance in this area. Responses to the following would be of interest to us:  What is the process for identifying UCLA’s priority assistance areas to MRP? Discussion with IMACS or MPAG or MMAF? Direct discussions with partner Universities? Other?  Related to the previous question, how is liaison with MMAF handled? Through University partners, direct, through IMACS/MPAG or other?  Are you aware of the overall objectives of the MRP and its component activities delivered through IMACS, MPAG and DoJ? In your opinion, how well does UCLA integrate with other activities to address the objectives? Page | 77  From your perspective, is the integration of activities under the UP with those of other UP partners effective?  What do you see as some of the ‘success stories’ for UCLA’s activities in contributing to the MRP objectives?  What problems have been encountered in delivering UCLA’s programs in an effective way? What can be improved?  Are there activities not currently being undertaken by UCLA that, if undertaken, could enhance the achievement of the MRP objectives?  Other general comments on the UCLA activities, particularly as they relate to achieving MRP objectives Questions for IMACS Major areas for questions:  MRP Progress, achievements, impediments: Institutional development, sustainable fisheries, coastal resilience, IUU  Technical approach – achieving biodiversity conservation outcomes  Partnerships and integration role; responsiveness to MMAF  Small grants contribution to larger objectives and strategy  Gender considerations  Future actions Specific questions: From your perspective, what is the objective of the MRP? Objective of the IMACS program? Progress: 1. How effective have IMACS project activities been in achieving planned results and producing concrete biodiversity conservation outcomes? What are major program successes and which activities may have sustained results or impacts? a. What progress is being made toward institutional development of the MMAF? i. What are the impediments and challenges? ii. How can we create a shared vision for sustainable use and conservation within MMAF? iii. The theory of change for this component appears to be: If the institutional capacity of MMAF is strengthened, then this will lead to improved marine resource conservation and sustainable fisheries management. Are there any demonstrations that this development hypothesis is effective, especially given that the decentralization process is still evolving? Are other actions needed? iv. For Task 1 (Institutional Development of the MMAF), how is performance measured? For example, the PI for Task 1 in the PMP is Percentage Improvements in performance of priority management systems in each DG of MMAF. How is this being measured? Are there baselines? Page | 78 b. What progress is being made toward sustainable fisheries management (SFM or EAFM)? i. How well does MMAF embrace EAFM, SFM, and MPAs? ii. What are major achievements? What factors contribute to those successes? What are major hindrances and challenges? What are major areas for improvement? iii. How is fisheries management being improved at the field level? How measured? c. What progress is being made to reduce destructive fishing and IUU fishing? Destructive fishing was part of the original program objective. d. The workplan mentions that a threats analysis was conducted to guide activity design. May we have a copy of that document? The 118/119 analysis and the workplan identify coral reefs and mangroves as the most threatened biodiversity areas; how is the program working in these areas? Are there any indicators that monitor reduction in threats? 2. What progress is being made toward coastal community resilience (CCR) for climate change adaption and disaster risk reduction? What are the challenges and/or hindrances? How are changes in coastal community resilience to climate change being measured? 3. What is the role of IMACS in integration for the MRP? What are the objectives of integration at the demonstration sites? i. How well is the MRP partnership working and what is the effectiveness of integration (a) within the components of IMAC and (b) between IMACS and other components of the MRP? ii. Are programs capturing synergies toward commonly-defined goals? In particular, and as an example, are MPAG activities and MPAs being considered as a component of sustainable fisheries management or as a stand-alone activity? iii. How were the demonstration sites selected? iv. Are there impediments to integration? How can integration be improved to enhance MRP outcomes? 4. Are there any staffing or staff capacity issues within IMACS that are impacting its ability to achieve program objectives? 5. Are there important policy actions, institutional relationships, and partnering arrangements that are facilitating or hindering project success? Partnerships and Integration: 6. How strong is IMACs partnership with the MMAF in terms of joint actions, dedicated resources and commitment to program objectives? What is the level of program engagement of the MMAF at various levels of government? 7. USAID has asked IMACS to be responsive to the requests of MMAF: how as this impacted progress toward program objectives (a hindrance or opportunity)? Small grants and gender: Page | 79 8. What is the objective of the small grants program? What is the strategy for the selection of grants and how are they designed to contribute to overall program objectives, including biodiversity conservations? Could they be used to strengthen curriculum development in universities or to build local NGO capacity? Could they be aligned with implementation of adaptation plans? 9. How are gender issues incorporated into program design and activities? MRP and Future Planning: 10. The MRP has 2 Objectives: (a) Restore and enhance ecosystem productivity, biodiversity and resilience for food and economic security; and (b) Increase the resilience of natural ecosystems and coastal communities to adapt to climate change and reduce risks from disasters. How is IMACS contributing to the overall MRP objectives? Are there activities not currently being undertaken by IMACS that, if undertaken, could enhance the achievement of the MRP objectives? 11. Looking forward to future planning, what are critical policies, approaches, capacity building or other actions that are needed to conserve marine ecosystems and sustain fisheries in Indonesia? How will the various components of IMACS activities be sustained following the completion of the project? Do you believe that the MMAF has the capacity to fully implement EAFM? Are there any ‘legacy’ activities being put in place by IMACS? Questions for MPAG Component Questions: What are the most significant achievements of the MRP and MPAG towards achieving sustainable marine resource management? How has MMAF institutionalized the technical advice and trainings? And do you seen an effect on improving MPA management, sustainable fisheries, and the data streams into MPA management? Limitations? What were the major successes and limitations towards achieving the goal of 20 mil Ha of MPAs? Achieving effective management within them? Are there key leverage points to overcome those limitations? Are there unique challenges and successes for establishing and managing the MPA networks? Integration Questions: Do you feel that MMAF is committed at all levels to taking the steps necessary to conserve and sustainably use Indonesia’s marine resources? How can we achieve the coordination/Integration program of MRP among ministries, (national agencies, e.g. MOE) and among DG/Directorate in the MMAF? Page | 80 How has the Decision Support System and Information Technology and Communications cross￾cutting support facilitated greater integration across MMAF DGs? Esp R&D, KKIJ (Small Islands Dev), Capture? How can we achieve better vertical integration between KKP, DKP, and local communities in planning and implementation? Relative focus on national level capacity development vs. local level implementation? Right balance? How is well is the partnerships/integration working? How do you view the role of IMACS in facilitating integration? How can it be improved? How has sustainable fisheries management & marine spatial planning been incorporated into MPAG focus sites? What were key drivers or impediments to “by seamlessly melding the (NGO) partners’ combined efforts into a sustainable, system-wide project approach”? How does the changing context of direct NGO involvement in the field affect MPAG and NGO partner relations/actions? How can other donors and projects be leveraged and/or aligned to increased the impact of MPAG? How has gender been integrated into MPAG and the MRP overall? Areas of weakness/concern? Areas of strength? Overall: Are there any gaps in the program for achieving the objectives of MPAG and objectives of MRP? What is the continuity between the CTSP-I and MPAG’s engagement in CTI and meeting CTI goals? What are the major threats to the sustainability of your program investments? How do we achieve the sustainability program, especially to sustainable funding for the program? What are the key areas that we should invest in for the future to enhance marine resource management and marine stewardship? Page | 81 ANNEX 3: DATA COLLECTION TOOLS The evaluation design and methodology used was, so far as possible, evidenced-based by the collation and analysis of primary data, although activities were be limited by the length of the evaluation mission, which precludes any independent data collection or activities such as focus groups. The evaluation was therefore be undertaken by adopting a three-pronged approach of:  Documentation and literature review  One-on-one key Informant Interviews (KIIs) in both Jakarta and in project sites at various locations in Indonesia (see Annex B – Travelling and Meeting Schedule)  Hypothesis-Testing (HT) approaches through follow-up interviews Document and Literature Review The evaluation team examined the wide range of documents already supplied by USAID (in excess of 600 documents) as well as documents identified by the Team as important as the Evaluation Mission proceeded. In-depth One-on-One Key Informant Interviews (KIIs) The team conducted semi-structured key informant interviews (KIIs) with all relevant MRP participants, including all key project partners. This included KIIs in at 4 project sites (Kandari, Wakatobi, Mataran (Lombok) and Kupang. There was insufficient time to visit all regional project sites and therefore the above three were prioritized because of (a) the integrated activities being undertaken by IMACS and MPAG at the first 2 sites and (b) the importance of Bali as the center for a number of MPAG NGO partners as well as being the location of an MPA network. The interview instruments included the development of key questions, specific to the target group but consistent with the broad questions outlined in section 3 (above). These were posed to senior managers and administrators as well as to the full range of stakeholders. In addition, follow-up interviews were conducted to cross-check the accuracy of the details, particularly where different views of events, procedures or activities are held by different parties. In addition, the evaluators used the interviewee’s knowledge to understand the wide-ranging effects of each project and component, such as the degree to which USAID-funded partnerships have influenced changes in behavior or changes in processes. Responses were collated and summarized and provided the essential background information for making recommendations to increase or improve U.S. government support to the MRD program. Remote Discussions To capture information through the KII process, the interviews were not be limited to Indonesian stakeholders but included telephone/conference call interviews with MRP contributors based in the USA. Page | 82 Proposed Methodology Profile of Key Informants The targeted KII participants included past and present principal stakeholders who have or had key roles in the MRP. In addition, direct beneficiaries were interviewed at project sites as well as in Jakarta in addition to indirect stakeholders, such as other donors. Data Collection Interviewers took structured notes of respondents’ answers and recorded any insights and observations that may be pertinent to the evaluation. Notes and information collected during each interview were collated and are presented in this draft final report as a consensus of responses. Analysis The analysis of the KII information generally involved the construction of a matrix to capture the respondents’ answers vis-à-vis the key evaluation questions and other project objectives (e.g. Annex 6). The approach will allowed the evaluation team to identify trends, disparities or significant feedback among the different respondent groups and will help shape principal findings and recommendations. Hypothesis-Testing (HT) approaches through follow-up interviews Following the initial KIIs, the team analyzed the results with the objective of developing a series of hypotheses and/or alternatives related to the key questions outlined in Section 3. These hypotheses were in the form of scenarios, developed by the evaluation team based on the information that had been gathered, to describe how the MRP might meet its short and long term (strategic) objectives. These hypotheses was then be tested by re-interviewing relevant MRP stakeholders that are selected following the analysis of results. This selection was be based only on the interviewee’s relevance to the hypotheses posed. Interview methodology, data collection protocols and analysis was the same as those for the initial KIIs. However, specific interview tools were developed (as a series of focused questions) to specifically address the hypotheses developed as part of this method. Key Evaluation Considerations Emphasis on Confidentiality All interviewees were informed that their responses are confidential and would not be shared with other MRP stakeholders, employers, or others and responses will be reported without attribution. Cultural Sensitivity Page | 83 Members of the team are Indonesian nationals or have worked in Indonesia and were well aware of existing cultural and geographical issues and differences. The design and implementation of appropriate interviews and meetings were therefore sensitive to cultural and regional issues and differences. Page | 84 ANNEX 4: LIST OF INFORMATION SOURCES Personnel and Institution Name Institution Directorate Job Title 1 Celly Catharina USAID Environment Office Marine Program Specialist 2 Brian Crawford Univ. Rhode Island (URI) Coastal Resources Center Director 3 Richard Mounsey IMACS Chief of Party 4 Yuni Kusmiati IMACS Task 1 Leader 5 Peter Mous and Team IMACS Task 2 Leader 6 Kun Praseno and Team IMACS Task 3 Leader 7 Bob Fagen IMACS Task 4 Leader 8 Edwin Shri Bimo IMACS Communication Manager 9 Pahala Nainggolan MPAG Chief of Party 10 Handoko and Rony MPAG Specialist 11 Janna Shackeroff NOAA Coral Reef Conservation Program International Coordinator 12 Russell Brainard NOAA Pacific Islands Fisheries Science Center Chief – Coral Reef Ecosystem Division 13 Megan Moews & Tomoko Acoba NOAA Coral Triangle EAFM Technical Coordinator 14 Gellwyn Jusuf MMAF – Sec.Gen. Secretary General 15 Anang Nugroho MMAF – Sec.Gen. Center of Analysis for International Cooperation and Institution (PUSKITA) Director 16 Nilanto Perbowo MMAF – Sec.Gen. Planning Bureau Director 17 Waluyo Susanto MMAF – Sec.Gen. Planning Bureau Head Division of General Planning 18 Rudy Alex MMAF – Sec.Gen Planning Bureau Head Division of Monitoring & Evaluation 19 Ishartini MMAF – Sec.Gen Planning Bureau Head Division of Program Planning 20 Isaac MMAF – Sec.Gen. Planning Bureau Head Division of Budget Planning 21 Fuad Hilmawan MMAF – Sec.Gen. Legal and Organization Bureau Head Division of Cross Sectoral Legislation and Marine Law Development 22 Tony Ruchimat MMAF – Directorate Directorate of Marine Conservation and Fish Species (KKJI) Director 23 Sofyan Hasan MMAF – DG. Marine, Coastal & Small Islands Directorate of Coastal and Marine Head Section 24 Agus Apun Budiman MMAF – DG. Capture Fisheries Directorate of Fisheries Resources Director 25 Hary Christijanto MMAF – DG. Capture Fisheries Directorate of Fisheries Resources Head Division of Territorial Sea and Archipelagic Waters 26 Sere Alina Tampubolon MMAF – DG. Marine & Fisheries Resources Surveillance Directorate of Fisheries Resources Surveillance Director 27 Slamet MMAF – DG. Marine & Fisheries Resources Surveillance Directorate of Fisheries Resources Surveillance Head Division of Fishing Surveillance in East Area Page | 85 Name Institution Directorate Job Title 28 Turman MMAF – DG. Marine & Fisheries Resources Surveillance Directorate of Fisheries Resources Surveillance Head Division of Fishing Surveillance in West Area 29 Narmoko Prasmadji MMAF – Fish Quarantine and Inspection Agency Director General 30 Nazori Djazuli MMAF – Fish Quarantine and Inspection Agency Directorate of Quality and Safety Certification Director 31 Budi Sugianti MMAF – Fish Quarantine and Inspection Agency Directorate of Fish Quarantine System Director 32 Ari MMAF – Human Resources Development Agency Center of Marine & Fisheries Training Head Division of Marine and Fisheries Training 33 Priyantini Dewi MMAF – Human Resources Development Agency Center of Marine & Fisheries Training Head Division of Training Implementation 34 Ahmadi MMAF – Human Resources Development Agency Center of Marine & Fisheries Training ???? 35 Duto Nugroho MMAF – Marine & Fisheries Research and Development Agency Research Center for Fisheries Management and Fish Resources Conservation Head of Research Center 36 Tiene Gunawan and Team Conservation International MPAG/CI Team Head of Marine Program 37 Gondan Putirenosari and Team The Nature Conservancy MPAG/TNC Team 38 Sudarsono and Team WWF Indonesia MPAG/WWF￾Indonesia Team 39 WCS MPAG/WCS Team 40 Marthen and Team Coral Triangle Center MPAG/CTC Team 41 Paul Barber UCLA 42 Stacey Tighe ADB – COREMAP III 43 Abdul Karim Naser IMACS SE Sulawesi Regional Office Regional Director 44 Wa Ode Diana IMACS SE Sulawesi Regional Office Field Coordinator 45 Alfian IMACS SE Sulawesi Regional Office Grants Assistant 46 Rahmi Sofiarini IMACS West Nusa Tenggara Office Regional Director 47 Zia Ul-Haq IMACS West Nusa Tenggara Office Fisheries 48 Asni Yurika IMACS West Nusa Tenggara Office Grants Assistant 49 Juhrin and team IMACS West Nusa Tenggara Office Enumerators 50 Laode Ridwan Bolu SE Sulawesi Province Dinas KKP Provinsi Head of Aquaculture Division 51 Lely Fajarina SE Sulawesi Province Dinas KKP Provinsi Staff Aquaculture Division 52 Ishaq SE Sulawesi Province Dinas KKP Provinsi Staff 53 Sri Agustiati Muna District Dinas KKP District Head of Coast and Marine Division 54 St. Nurwasyrah Muna District Dinas KKP District Staff 55 Asrul Arumi Buton Municipality Dinas KKP Municipal Head of Coastal Management Section 56 Kudrat Priadi Buton Municipality Dinas KKP Municipal 57 Oktawinus Wakatobi District Dinas KKP District Head of Business Development Division Page | 86 Name Institution Directorate Job Title 58 Irmanto Lantele Wakatobi District Dinas KKP District 59 Kamarudin Thamzibar LEPASAMA Director 60 Mundzir LEPASAMA 61 Mitzan YASCITA Project Coordinator Small Grant 62 Sumardin YASCITA Admin-Finance 63 Jefferson Tasik YASCITA Project Coordinator I-CATCH 64 Oktovianus YASCITA Staff 65 Rahmatia Unidayan – Bau-Bau LPPM Head of LPPM 66 Baride Unidayan – Bau-Bau LPPM Technical Staff 67 Sumitro Unidayan – Bau-Bau LPPM Admin-Finance 68 M. Ridwan East Lombok District KKP District Head of Conservation Section 69 Hermanto North Lombok District KKP District Head of Agriculture, Estate, Forestry, Marine and Fisheries Division 70 Wahyu Darmawan & Sudirman North Lombok District KKP District Secretary & Head Section of Fishery 71 Yessayamau UPT Conservation/Balai Konservasi Perairan Nasional –BKPB Kupang Director 72 Zulkarnaen Lubis UPT Conservation Head of Gili Area 73 Tasrif MPAG/WCS Leader 74 Abdul Magrib Gafar Univ. Gunung Rinjani LP3M Head 75 Johan Maligan Univ. Gunung Rinjani LP3M 76 Jotam Ninef P4KKP Savu Sea MNP Team Leader 77 Ferdi Kapitan P4KKP Savu Sea MNP Co Team leader 78 Joni Rohi P4KKP Savu Sea MNP Team Members 79 Ayup Mecco University concorsium for sustainable fisheries in NTT (Uniconsufish) Coordinator 80 Rusidi Uniconsufish Vice coordinator 81 Sulastri Rasyid DKP NTT province Aquaculture & small island management division Head of division 82 Isac Anguar Mase DKP NTT province Section of Conservasion and reahbilitation of coastal resources Head of section 83 Alexander Tanody TNC Kupang office Project Leader 84 Yusuf Fajariyanto TNC Kupang office Conservation coordinator 85 Miqdon Abolla NTT province Agency of Environment Management Rehabilitation & conservation division Head of division 86 Rofi Al Hanif BKKPN Kupang Conservation planning Head of section 87 Rhemondus Nggajo BKKPN Kupang Conservation Development & Monev Head of section Page | 87 List of Document categories consulted No Title/Heading Author 1 Annual Report FY 2011 and FY 2012 IMACS 2 Workplan 2011, 2012 and 2013 IMACS 3 PMP January 2013 IMACS 4 Selected Technical Reports Task – 1 IMACS 5 Selected Technical Reports Task – 2 IMACS 6 Selected Technical Reports Task – 3 IMACS 7 Selected Technical Reports Task – 4 IMACS 8 Annual Report FY 2012 MPAG 9 Workplan year 1 and year 2 MPAG 10 PMP May 2012 MPAG 11 EGE Indonesia MMAF Assessment USAID - Indonesia 12 USAID Country Strategy 2009 - 2014 USAID - Indonesia 13 Greening Indonesia’s Blue Economy Kieran Kelleher 14 Report on Fisheries Management in Indonesia John G. Sutinen 15 Evaluation Policy 2011: Learning from Experience USAID 16 Quarterly report Oct – Dec 2012 MPAG 17 Cooperative agreement No. AID-497-LA-12-00001 Indonesia MPAG Program USAID - Indonesia 18 Biodiversity Conservation: a Guide for USAID staff & Partners USAID Page | 88 ANNEX 5: SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis conducted Data Sources Type and size of sample Limitations What has been the effectiveness of the IMACS and MPAG projects’ activities in achieving planned results? (Level 1 evaluation) a. Document review b. Key Informant Interviews (KII) c. Hypothesis testing PMPs, Work Plans, Quarterly & Annual reports, responses to KIIs Approx 25 relevant documents plus approx. 30 KIIs Not all project sites could be visited What is the capacity of the IMACS and MPAG project teams to implement activities in an efficient and timely manner? (Level 1 evaluation) a. Document review b. Key Informant Interviews (KII) c. Hypothesis testing Staff resumes, responses to KIIs from grant recipients etc 5 resumes, 11 KIIs Nil What were the important policy actions, relationships & partnering arrangements that facilitated project success? (Level 1 evaluation) a. Document review b. Key Informant Interviews (KII) c. Hypothesis testing PMPs, Work Plans, Quarterly & Annual reports, responses to KIIs 9 documents, 23 KIIs Many partners were located in the USA, necessitating telephone interviews Is the MRP achieving results against targets in the areas of: (a) Institutional development (b) Improving sustainable fisheries management, (c) Climate change adaptation, (d) Marine Protected Areas, and (e) Reducing IUU Fishing? (Level 2 Evaluation) a. Document review b. Key Informant Interviews (KII) c. Hypothesis testing Responses to KIIs, Quarterly & Annual reports, Results Frameworks and KRAs 26 KIIs, 10 reports, 2 Results Framework Is the MRP achieving its overall objective of sustaining the long-term welfare of coastal Indonesians by reducing the destruction of marine resources and preparing them for climate change impacts? Is the MRP strategy of capacity building & targeted technical assistance producing sustainable improvements to ecosystems and institutions? (level 3 evaluation) a. Document review b. Key Informant Interviews (KII) c. Hypothesis testing Responses to KIIs, Results Frameworks and KRAs 2 Results Frameworks, 29 KIIs No Results Framework for MRP. No assessment undertaken of overall MRP outcomes (e.g. biodiversity outcomes, improvements in sustainable management etc Page | 89 ANNEX 6: MATRIX OF PROJECT AND MRP COMPONENT PERFORMANCE 6-1. Matrices for Evaluation Level 1 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes 110 Review the legislative and regulatory framework to ensure it is sufficient for the MMAF to execute its mandate to protect marine resources and fisheries, and provide technical assistance to develop new or revised policies and regulations as necessary. 1.1 Number of policies, laws, agreements or regulations promoting sustainable natural resource use and fisheries conservation that are implemented. 9 0 12 111 Assess the international obligations, threats, policy priorities and representation of MMAF in forums concerning marine resources and fisheries management. 5 3 4 - Not necessary to assess MMAF obligation or representation in international forum. - Assessment conducted through STTA provides good baseline regulatory understanding specifically on tuna fisheries - Should use Sutinen’s report - All claimed achievements were not in IMACS identified policies’ list. - Most of the achievements (12) 112 Review the national aspects of the MMAF mandate for marine resources and fisheries management and identify key gaps and concerns in the existing legal framework. Page | 90 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes 113 Organize expert roundtable and working group discussions on priority policy issues for MMAF with a view to formulating policy options as input to legal instruments. were not regulatory identified in the expected results. List of achievement: a) West Lombok Coastal and Small Island Law (PERDA) b) Blue Economy Policy (paper) c) Business Sea Leases recommendation (paper) d) Ocean Policy e) MOU with provinces (2) f) MOA with municipalities/dist ricts (6?) 114 Assist MMAF with background research on policy issues and drafting of national regulations and other policy instruments. Page | 91 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes 120 Provide technical assistance to improve the efficiency of organizational operations and develop a shared vision among the Directorate Generals for sustainable resource use, adoption of an Ecosystem approach to fisheries management, and the acceptance of MPA as a fisheries management tool. 1.2 Number of joint plans or operations between Directorate Generals. 10 0 1 121 Assess the baseline knowledge, attitudes and practices (KAP) of selected MMAF working units and develop an action plan to strengthen operations. 1 1 1 - Unclear what was the achievement (1) - Not use EGE 2009 results as base to develop activities plan - Incorrect lead agency in MMAF to work on this issue (PUSKITA), cause it doesn’t have needed authority nor mandate to deal with it. - It should directly work with each Secretary Directorate General - Therefore this activity was not effectively conducted. - The project team will not have enough capacity to reach the 122 Organize forums with working units for discussion, training and the monitoring of improvements in operational efficiency and changes to MMAF business functions. 123 Assist MMAF with Page | 92 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes preparing operational guidelines for corporate functions and study materials on subject matters relevant to sustainable resource use, EAFM and MPA. objective. 130 Provide technical training to MMAF staff for the execution of their responsibilities and strengthen the institutional capacity for staff training. Courses may include fishing licensing/permittin g procedures, sustainable fishing practices, 1.3 Number of MMAF personnel trained in courses pertaining to their operational responsibilities. 900 0 49 3 131 Assess existing training activities, materials and needs of MMAF target groups and prepare a comprehensive training plan. 3 3 3 - IMACS has to facilitate formalization of ach Directorate Generals/Directorates essential need for human resources and its capacity - BPSDM only mandated to conduct the training but not setting up the human resources needs in each DGs - Since the training was developed for Indonesia, hence the 1.5 (C) Improvement of performance in each DG of MMAF. 25 % 0 % 132 Develop training curricula and materials / modules suitable to a hierarchy of needs for different target groups (introductory, Page | 93 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes monitoring, control and surveillance, and port state measures. practitioner, trainer) curricula and material should be in Indonesia - Presentation should always be conveyed in Indonesia. - Curricula needs to be standardized to avoid became just an ad hoc kind of trainings. 133 Organize national and provincial & district technical training on selected topics in collaboration with MMAF and MRP partners. 140 Provide technical support to improve execution of the MMAF national five year strategy at the local level. This involves 1.6 Number of nationwide priorities executed at the provincial or district level. 5 0 1 141 Organize workshops in collaboration with MMAF to discuss the 5-Year Strategic Plan, local problems, and IMACS and MRP 2 2 2 - Data collection mechanism in demo site only with private sector (ANOVA) and was not involved local government. - No preparation to Page | 94 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes strengthening coordination between the central and district offices and preparing districts to become more accountable, especially in data gathering, processing and managing. activities in supporting plan implementation. strengthen, improve or prepare local government to handle the demonstration site once IMACS done. - Lead agency shouldn’t be PUSKITA. It should be with Directorate of Fisheries Resources and PUSDATIN for data collection and management, planning bureau for 5 -year strategic plan. - Needs to more closely facilitate the development of 5 year strategic plan. Placing an expert/advisor in the planning bureau will be more effective - Data collection was not covered capacity of fishing. - No involvement from 142 Assessment of capacity for data collection and handling at district level and develop an action plan for strengthening data management systems, especially in pilot areas. 143 Conduct structured on -the -job training and technical support for working units at central and district level on aspects of data collection, quality control, handling, storage / retrieval. Page | 95 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes Provincial and Local governments’ DKP. 150 Provide technical support to improve MMAF’s public outreach capacity with the objective of enhancing community participation in the management of Indonesia’s marine resources. 1.7 Number of local or national public awareness campaigns supporting the mandates of MMAF and EAFM. 12 0 4 151 Assess MMAF’s public communications plans, target audiences, outreach activities, campaign designs and media, and prepare recommendations to improve its effectiveness. 1 3 1 - Unclear product - Unclear target groups - Unclear key messages 1.4 (C) - Unclear means/conduit Increase in public opinion of MMAF services. 25 % 0 % 152 Assist MMAF with the design, production and dissemination of information media for building awareness and participation of target audiences. Page | 96 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes 160 Provide technical support to increase fisheries extension services provided to communities and the private sector for sustainable fisheries and marine conservation. This includes, but is not limited to support to develop MMAF capacity to advise fishers on technical issues such as environmentally￾friendly fishing techniques, harvested fish quality, and port state measures. 1.8 Number of community groups and private companies that receive technical guidance from MMAF in ecosystem protection or sustainable fisheries management. 100 0 13 161 Assess the baseline knowledge, attitudes and practices (KAP) of fisheries extension personnel and develop an action plan to strengthen their capabilities. NA NA NA - Unclear program 162 Review current training program, performance monitoring mechanisms and continuous-learning support facilities for fisheries extension services. 163 Assist MMAF to develop the capacity needed to advise fishers and private sector on relevant technical and legal matters Page | 97 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes affecting fish harvesting, processing and marketing. 170 Provide options to increase investments in fisheries, coastal management and MPA. Strengthen MMAF’s ability to support procedures related to attaining certification of sustainable fishing practices. Provide technical assistance to develop new or revised policies and regulations as X Policies and regulations that allocate finances into improved fisheries management and marine resource conservation. (This indicator has been combined with 1.1 above) -- -- 171 Assist MMAF to develop its prospective plan of investment needs, opportunities and policies; and strengthen capacity for investor relationship management and problem-solving. NA NA NA - Unclear program 172 Assess institutional options for and constraints to certification of sustainable fishing practices through MMAF and/or other organizations. Page | 98 TASK - 1 No . Activity Indicator Target EOP Achieved 2011 2012 Work plan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated project success Comments/findings/notes necessary. 173 Organize forums for stakeholder information dissemination, training and problem solving on adoption and certification of sustainable fishing practices. (S) – Refers to a Standard indicator that applies to all similar USAID projects (C) – Refers to additional indicators prescribed for IMACS. Legend of the scale Effectiveness Scale Capacity Scale Importance Scale 1 = Not Effective 1 = Incapable 1 = Not important 2 2 2 3 = Effective 3 = Capable 3 = Important 4 4 4 5 = Highly/Very Effective 5 = Highly Capable 5 = Very Important Page | 99 TASK -2 No . Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangement s Comments/findings/ notes 210 Promote sustainable fishing practices to reduce overfishing and destructive practices (e.g. trawling) that damage habitats, threaten non￾target species, and jeopardize future stocks, indiscriminent to maturity. 2.1 Number of hectares under improved fisheries management. 5 M 0 1.9 M 211 Develop draft regulations, procedures, and systems under Fisheries Law Number 45 of 2009 to allow for control of effort, and test them in the field. 2 4 3 - The selection of tuna fisheries as model for both management of specific fisheries and also the practice of good public private partnership was excellent. However, it is too insignificant to represent the overall fisheries management area (WPP 573). - Since it was specific for tuna, it can be difficult to replicate the model for other fisheries or common commercial fisheries, even in that same WPP - Minimal involvement of local government (and national) in conducting the exercise and lack of buy-in of the model of data collection in 212 Develop TURFs-and￾reserves as tools for implementing Ecosystem Approach to Fisheries Management in Indonesian waters under Fisheries Law No. 45 of 2009. Page | 100 TASK -2 No . Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangement s Comments/findings/ notes their system, will affect replication and sustainability of the model. 2.7 Number of persons in law enforcement trained in enforcement of fisheries regulations 150 0 27 213 214 Make recommendations to MMAF and DKPs for strengthening compliance with fisheries regulations: clarification of roles and responsibilities in enforcement, identify which types of enforcement are suitable for which types of fishery. Strengthen governmental financing of fisheries management. 2 4 2 - - Increased numbers of persons who got training is not automatically improved their performance. It needs follow-ups activities and evaluation to determine the outcomes - Training program should be standardized, developed and presented in Bahasa Indonesia - Recommendation on which types of enforcement methods suitable for which types of Page | 101 TASK -2 No . Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangement s Comments/findings/ notes fisheries should correlate with results from EGE 2009 on “unit clustering” within MMAF - Most of the activities were meetings and produced reports or “white paper”. It needs to have a much better endorsement to develop actual buy￾in or continuation of the ideas. Page | 102 TASK -2 No . Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangement s Comments/findings/ notes 220 Improve GOI monitoring and data collection of fish catch to include information on fishing locations that can help forecast available stocks in each management area for planning annual fishing permits and quotas in each area. 2.2 (S) Number of persons in local government and communities trained in SFM practices. 120 0 311 221 222 Conceptual design of a fisheries management information system Technical implementation of a fisheries management information system 2 4 3 - Unsuitable indicator for the activity. Although monitoring and data collection in a fisheries management system is indeed part of good SFM practices, - Needs to develop a clear and direct roadmap toward the adoption or strengthened current Data Management System and Data Collection Mechanism 2.3 Number of GOI person-hours dedicated to on-deck fishing observation. 265 0 46 223 Select indicators for ecosystem approach to fisheries management and include selected indicators in procedures, practices, 1 3 2 - Unclear connectivity between work plan strategies with the indicator Page | 103 TASK -2 No . Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangement s Comments/findings/ notes and SOPs of the fisheries management information system 230 Assist the private sector to adopt sustainable, integrated best fisheries management practices in critical fisheries by providing technical assistance to large fishing companies, demonstrating to the MMAF that sustainable approaches benefit both the environment and the business. This may include technical assistance in obtaining certifications for eco-labelling seafood products. 2.4 Number of public-private partnerships formed supporting sustainable fisheries mgmt or adopt best practices 6 0 2 231 Develop capacity of MMAF and fisheries services to develop management responsibility in fisheries communities (fishers plus traders). Support public-private / on-the-ground partner initiatives on SFM by helping government agencies provide supportive regulatory framework. Supporting activities comprise entire supply chain for better traceability of fish. 3 5 3 - Good exercise and model for specific fisheries (tuna/blue swimming crab) - May not easy to replicate for other types of fisheries - Could not representing much broader areas (WPP) - Needs to develop more models which represent more varied fisheries in the same WPP 232 Page | 104 TASK -2 No . Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangement s Comments/findings/ notes 240 Improve central and district level policies and regulations to support ecosystems-based fisheries management, including the use of fisheries reserves, managed access through licensing and permitting, zoning, and allowable gear policies. 2.5 Number of policies or regulations approved that support EAFM Practices. 3 0 1 241 Develop draft procedures and systems (under Fisheries Law Number 45 of 2009) for management planning and management implementation by fishery within WPPs, including licensing-by￾fishery, TURF-reserve systems, and marine protected area networks. 2 3 3 - EAFM or EBFM is not yet adopted concept within MMAF, much less in provincial and districts level - Activities still limited to meetings and develop papers - Needs to produce a guidance on “best possible” way to develop WPP Management Plan with existing imperfect condition in Indonesia 242 Improve legal and procedural basis under Fisheries Law Number 45 of 2009 for management by fishery type within WPPs, based on accurate information. Page | 105 TASK -2 No . Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangement s Comments/findings/ notes 250 Improve central level capacity to utilize modern fisheries modelling approaches to ensure a more precautionary approach to management for sustainability, moving away from maximum economic yield to maximum sustainable yield. 2.6 Number of fisheries managed on a more sustainable basis. 46 0 1 251 Develop set of standard methodologies with direct links to requirements of fishery management plans. Adapt state-of-the-art insights to data-poor situations. 2 5 3 - It was a good models, but were too small in number and too specific (tuna/blue swimming crab) - It is still too early to see the output of the experiment 252 Promote training and education for junior scientists as well as district, provincial and national technical officers on fisheries analysis methods and stock assessments. Legend of the scale Effectiveness Scale Capacity Scale Importance Scale 1 = Not Effective 1 = Incapable 1 = Not important 2 2 2 3 = Effective 3 = Capable 3 = Important 4 4 4 5 = Highly/Very Effective 5 = Highly Capable 5 = Very Important Page | 106 TASK - 3 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/finding s/notes 310 Increase vulnerable coastal populations’ awareness of potential disaster and climate change impacts and risk reduction practices in select communities. Activities may include technical assistance for climate change public outreach by local government offices and to execute public awareness campaigns; zoning to accommodate sea level rise; and protection of critical habitats. 3.1 (S) Number of people reached with increased capacity to adapt to climate variability and change as a result of USG assistance. 5,000 0 713 311 Organize forums to develop comprehensive training program for local government personnel and communities on CC risk assessment and adaptation strategies with MMAF, universities, Sea Partnership/Mitra Bahari (MB), NGO, among others. 2 3 3 - Target was not reached 3.4 (S) Number of people trained in climate change issues and resilience. 5,000 0 738 312 Organize training courses and workshops for local government personnel on conducting CC vulnerability assessments and adaptation planning in collaboration with universities, MB, MMAF and MRP partners. 2 3 3 - Target was not reached - Training curriculum needs to incorporate fisheries and integrated coastal zone management 3.2 Number of people reached through 10,000 0 4.400 313 Development and execution of community 2 3 3 - Needs to have clear key Page | 107 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/finding s/notes climate change and disaster risk reduction, public awareness campaign that may incorporate community task force group. awareness campaigns related to local marine conservation and climate change concerns. massages, target groups and channel of communication 320 Prepare communities to take specific actions to mitigate the impacts of climate change and reduce the risks of disasters through assisting communities to develop community action plans. 3.7 Number of community based action plans formulated through local multi stakeholder groups 50 0 8 321 Assist local governments in pilot areas to undertake CC vulnerability assessments and zoning with reference to USAID guidebook. 3 3 3 - 322 Assist with the development of community action plans to reduce the risks and mitigate the effects of natural disasters. - - - - 330 Increase the resilience of coastal communities to the economic impacts of climate change and disasters by 3.5 (S) Number of communities taking steps with local governments to plan for and implement resilience activities for climate change adaptation 50 0 12 331 Provide technical assistance to community organizations in the formulation of small grant proposals and implementation of 1 2 3 - Needs to integrate with fisheries management, conservation, biodiversity and coastal Page | 108 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/finding s/notes increasing household resilience, promoting sustainable fisheries and adopting appropriate supplemental livelihoods. and disasters activities to build sustainable fisheries and livelihoods. management 3.8 Number of communities or households that adopt strategies to enhance community or household security, supplement economic livelihoods, or diversity income opportunities or sources for food. 50 0 30 332 Assist MMAF to improve its capacity for conducting scientific and economic analysis related to maintenance of blue carbon sinks as part of climate change adaptation strategies. 2 3 3 - Unclear results 340 Assist local governments to develop policies, budgets and procedures that protect the coast from degradation, for disaster preparedness and climate change 3.3 (S) Number of laws, policies, agreements, or resolutions addressing climate change proposed adopted, or implemented. 7 0 2 341 Conduct policy review to determine areas for mainstreaming CC adaptation and resilience activities into national and district planning processes. 2 3 3 - Limited to “agreement” between communities with local government, not regulation or policy 3.6 (S) Number of local governments trained in climate change adaptation, 10 0 28 342 Provide technical assistance to local governments on 2 3 3 - No follow ups activity or target for participants after the training. Page | 109 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/finding s/notes adaptation. disaster risk reduction and community resilience. development of appropriate plans, policies and programs for disaster preparedness and CC adaptation. - Not standardized training curricula 3.9 Number of local government:  risk & vulnerability assessments,  disaster risk management or emergency response plans,  coastal management regulations,  land use management strategies, or early warning systems established, or  local government budget utilized for adaptation or disaster planning activities. 10 0 4 343 Assist with formulation of small grant proposals and implementation of activities for disaster preparedness and protection from coastal degradation. 1 2 3 - Small grants program was not worked to support this activity - Local government budget allocated to only small number of target villages - Needs to better aligned with planning cycle within local government system Page | 110 TASK – 4 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/findings /notes 410 Establish a process for coordinating and integrating among IMACS, CTSP, NOAA and DOJ activities; and with the MMAF. 4.1 Number of activities coordinated among partners 22 8 16 411 Development of a communications plan for IMACS and MRP partners that coordinates outreach activities. 1 2 3 - Limited to NOAA - Isolated from CTI-CFF activities and champions - Limited effective communication with technical Directorates in MMAF and Local governments - No communication strategy developed - Unclear key messages 412 Assessment of MRP communications/ out-reach activities with recommendations and on-going support to Page | 111 TASK – 4 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/findings /notes development of common media channels and content for target audiences. 413 Design, production and distribution of information media for building awareness and participation for IMACS activities. 414 Organize meetings and other forums to plan, coordinate and monitor the implementation of activities among MRP partners and with MMAF and Menko Kesra. Page | 112 TASK – 4 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/findings /notes 420 Design and implement an effective capacity building and training program that ensures that the training activities of NOAA, DOJ, CTSP and IMACS have sustained impacts. Identify the best use of training workshops to institutionalize the knowledge, within MMAF, universities, NGOs, or other institutions. 4.2 Number of training modules adapted into curriculum or used by Indonesian agencies or organizations 3 0 2 421 Organize meetings with NOAA, CTSP, DOJ and other stakeholders for coordinated planning, design and implementation of training workshops and mechanisms to institutionalize knowledge. 1 2 3 - IMACS doesn’t have long term expert in developing training curricula - Curriculum was designed without undergoing training standardization process - Most material and presentation was conducted in English instead of Bahasa Indonesia - No follow-ups activities or targeted products for participants post training. Page | 113 TASK – 4 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/findings /notes - Should focus more on producing trainers rather than “mass” 422 Design and implement systems for MRP data compilation, analysis and reporting on a quarterly basis. 423 Prepare and implement a monitoring and evaluation framework for assessing and reporting on MRP accomplishments. 430 Work with partners to establish at least two model sites that demonstrate transformational 4.3 Number of effective demonstration sites established 2 2 2 431 Prepare site criteria and preliminary assessments of potential pilot areas to select 2 4 3 - Too little in number and too specific to be able to represent the overall fisheries Page | 114 TASK – 4 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/findings /notes development. preferred locations in collaboration with MMAF. management area - Difficult to replicate to other kind of fisheries - Lack of buy-in from national and local government - Lack of effective communication to transfer conceptual knowledge of the demonstration to local governments - 432 Provide technical assistance to local governments and MMAF in defining needs and designing actions Page | 115 TASK – 4 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/findings /notes plans needed for marine & fisheries sector-based sustainable development in pilot areas. 440 Provide administrative support and coordination to ensure consolidated reporting and unified public communications. 4.4 Number of consolidated progress reports submitted on a timely basis. 15 3 5 441 Organize common systems, parameters and templates for compiling administrative reports from MRP projects. 2 4 3 - No report on integration between sustainable fisheries management and climate change adaptation - No report on how all of those activities addresses KRA of MRP - Needs better connectivity between activities and overarching objectives of Page | 116 TASK – 4 No. Activity Indicator Target EOP Achieved 2011 2012 Workplan Strategies Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements Comments/findings /notes MRP 442 Prepare progress and special reports, and public outreach media, with common look and feel for MRP. 443 Organize program coordination and performance review meetings with USAID, MMAF and the MRP projects. Legend of the scale Effectiveness Scale Capacity Scale Importance Scale 1 = Not Effective 1 = Incapable 1 = Not important 2 2 2 3 = Effective 3 = Capable 3 = Important 4 4 4 5 = Highly/Very Effective 5 = Highly Capable 5 = Very Important Page | 117 Annex 6-1. Level 1 Assessment of MPAG program. The result of activities on reports is evaluated against target on work plans. Scale 1 – 10 scale (10 – 100 %) for target achievement by 2012´, highlight colour is on target. CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success Establishment of Sustainable MPA National System Sustainable MPA National System is established Institutional Improvement - Secondment to KKJI - Vertical Integration￾Regulation - Integration between MPAs and sustainable fisheries - Spatial Planning MPA policies, regulations, guidelines developed Target 3 indicators by Y1 (Number of Ministerial Decrees by y1=1, y2=2), (Number of DG KP3K Decrees by y1=1, y2=1) (Number BPSDM) Decrees by y1=2) (Number of MMAF guidelines by y2=1) MPAs integrated into an Ecosystem Approach to Fisheries Management and Marine Spatial Planning. (Number of MPAs testing EAFM techniques). (EAFM training module 3 (30%) Only one (DG KP3K on EKKP3K) of 3 target was achieved 5 (50%) Three of 7 target indicators was achieved. For institutional improvement, all the NGOs consortium members & COP team take responsible as leader for different DG or directorate general in MMAF, e.g. CI for BPSDM & spatial planning directorate, WWF for SDI, WCS & TNC for KKJ, etc. The capacity team to achieve the output was different as presented at the target results. For the project part they also have been collaborating with KKJI, Legal Bureau of MMAF, District DKP, SDI, IPB, District DKP, IMACS, NOAA, DoJ, Local 7 (70%) Following the SOW, there are four focus evaluation on this part: 1) MPAG has been collaborated well with NOAA for capacity building & not well collaborated yet with IMACS. 2) MPAG has been effective work with the MMAF at the national, regional (BKKPN), local (province & district) as well as site levels. MPAG program has been successful to establish a management tool to measure the effectiveness level of MPAs management (or E-KKP3K) and then Page | 118 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success improved) (Launch EAFM communication tools) (Engage potential private sectors to implement program based on EAFM pilot testing result) (Number of joint activities between MPAG and IMACS established and implemented) (Technical recommendation for synchronization of MPAs and spatial plans produced); (Socialization of agreement between Director of KKJI and Director of Spatial Plan supported) (Technical support on integration of MPAs into spatial plan at site level accomplished) Universities. Actually they have been engaged high level on MMAF, e.g. Secretary general, technical DG, director or biro leader. However, according to them to develop a ministerial decree is complicated process and it needs a long bureaucracy and involving cross directorates, include Legal Bureau of MMAF. So that, need time more than target on the work plan. adopted by GOI by a DG KP3K-MMAF decree. A bottom up process to establish a new MPA (Savu Sea Marine National Park) with area of 3.5 million Ha (10 districts) seems on track to achieve. So, this experience will be an important lessons learned by GOI (MMAF) for establishing a new MPA by such processes. 3) There is significant integration MPAG project with other funding (donors) in implementing the program, especially in the sites. USAID (MPAG) is only partly budged supporting of the NGO consortium. Another donors e.g. JICA, FAO, BMU etc. 4) Most of the MPA sites on the low level management. Thus, it cannot generate income yet from the Page | 119 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success activities. 5) Dynamic conditions in the MMAF & local government (e.g. general elections) potentially hindered the program success. Capacity Development National and local capacity for sustainable MPA management strengthened. Target 6 indicators by Y1 (Number of people receiving MPA related trainings by y1 = 260, y2=290, y3=310) (Number of curricula and modules based on minimum core competencies developed, y1=3) (Number of training information systems developed, y1=1) 7 (70%) Four of six target indicator was achieved. Number of MPA-related trainings delivered above the target = 16. For capacity development program is coordinated by CI and collaborated with BPSDM, NOAA, IMACS, Walton Family Foundation Page | 120 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success (Number of studies on functional positions for conservation, y1=1) (Number of MPA-related trainings delivered, y1=10, y2=10, y3=5) (Number of training modules on EAFM indicators, y1=1) Management Decision Support - Decision Support System (DSS) - Scientific Inputs DSS developed, with scientific input and used by decision makers/stakeholders. Target 4 indicators by Y1 (Number of SOPs developed, y1=2) (Number of data sets, scientific input, entered into the system, y1=10, y2=10, y3=5) (Number of website visitors, y1=500, y2=2000, y3=20,000) (Number of learning and information centers on EAFM, y1=1) 8 (80%) Three of four target indicator s was achieved. Number of SOPs developed =10 & data sets = 22, above target. For this project part is coordinated by TNC, collaborated with KKJI, PUSDATIN, P4KSI, NGOs, MOF, Other Government Agencies Sustainable Financing MPA sustainable finance mechanism developed Page | 121 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success A Trust Fund for MPA sustainable financing mechanism developed Target 3 indicators by Y1 (Number of SOPs, y1=2) (Number of background papers on a Trust Fund, y1=1) (Number of PKS between BP MIGAS and MMAF for managing MPA, y2=1) 10 (100%) All target indicators for Y1 was achieved. This program coordinated by WWF, collaborated with KKJI, MOF, National Development Planning Agency (BAPPENAS), Ministry of Environment, Ministry of Finance, NGOs, including Kehati. Advance MPA Management Effectiveness in selected National, District MPAs and MPA networks Management effectiveness advanced in selected national, district MPAs and MPA Networks. Support the establishment of MPAs. The new MPAs established: MPA establishment progressing towards 2020 goal, (Area of biological significance and/or natural resources under improved natural resource management as a result of USG assistance, y1= 8,027,397 Page | 122 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success - West Kei Kecil ha) - Establishment of new MPA sites (West Kei Kecil and Berau); and -Improved management effectiveness (Anambas, Nusa Penida, Raja Ampat, Savu Sea, Wakatobi, and GiliMatra). West Kei Kecil. Target 3 indicators by Y1 Number of: (y1=1, y2=1 Bupati Decree on proposed MPA and management unit); (y3=1 Ministerial Decree on enactment of West Kei Kecil MPA; (1 management and zoning plan completed by y2 & approved by y3); (y3=2 management protocols/SOPs developed) ( y1=1, y2=1 MPA-related trainings delivered) & (‘red’ by y1, ‘25% ‘yellow’ by y2, 50% ‘yellow’ by y3 7 (70%) Two of three target indicators was achieved for West Kei Kecil MPA. For West Kei Kecil MPA, WWF has collaborated with West Kei Kecil District Government, BPSDM. Page | 123 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success - Berau on level of management) Berau Target 2 indicators by Y1 Number of: (y3=1 ministerial Decree on enactment of Berau MPA); (y2=1 management plan and zoning plan completed in 2 conservation areas); (y1=1, y2=1 ecological and socioeconomic survey implemented & disseminated in 2 conservation areas); & (‘yellow’ by y1, ‘25% ‘green’ by y2, 50% ‘green’ by y3 on level of management) 1 (10%) No target indicator was achieved for Berau MPA. In Berau MPA, TNC collaborated with Berau District Government, local community groups and KKJI. The project output indicated that it didn’t work well to get the result. Support the operations and strengthen management of MPAs - Savu Sea MPA management and operations supported Savu Sea Target 4 indicators by Y1 Number of: 9 (90%) Two of four target Page | 124 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success - Nusa Penida (y3=1 Ministerial Decree on enactment of Savu Sea MNP; (y2= 1 management plan and zoning plan completed & approved); (y1= 35 meeting notes from public consultations); (y1=2, y2=2 protocols on patrol and monitoring); (y1=2, y2=1 MPA-related training delivered) & (‘yellow’ by y1, ‘25% ‘green’ by y2, 50% ‘green’ by y3 on level of management) Nusa Penida Target 3 indicators by Y1 Number of: (y3=1 Ministerial Decrees on enactment of Nusa Penida MPA); (y1=1 Bupati Decree on Management Unit); (y2=1 management plan and zoning plan completed & approved); (y2=2, y3=2 management protocol/SOPs developed; indicators was achieved & two 50% on target for Savu Sea. Additionally, public consultations above the target (71) 5 (50%) One of three target indicators was fully achieved and one 50%, for Nusa Penida MPA. Bupati Decree on the Management Unit was not realize yet In Savu Sea MPA, TNC strongly collaborated with Team P4KKP, BKKPN Kupang, District Governments, local NGOs, University. TNC has been working at the governor & bupati level on the NTT, e.g. P4KKP was established by governor decree. In Nusa Penida MPA, CTC has been collaborating with Klungkung District Government, AHAWISRI, KKJI, & Tour Operators. Page | 125 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success - Anambas - Wakatobi y2=1 sustainable financing concept); (y1=1, y2=1, y3=1 ecological and socioeconomic survey), & ‘(yellow’ by y1, ‘25% ‘green’ by y2, 50% ‘green’ by y3 on level of management) Anambas Target 3 indicators by Y1 Number of: (y3=1 Minister Decree on enactment of Anambas MRP); (1 management plan and zoning completed by y2 & approved by y3); (y1=1 Marine Rapid Assessment Program (MRAP) report; (y1=1, y2=1, y3=1 MPA training delivered; (by2=2, y3=2 management protocols/SOPs) & (‘yellow’ by y1, ‘25% ‘green’ by y2, 50% ‘green’ by y3 on level of management) 7 (70%) two of three target indicators was fully achieved for Anambas MPA. In Anambas MPA, CI collaborated with Provincial and District DKP, Oil Companies and BP Migas, KKJI, Loka Office at Pekanbaru, & Directorate of Small Islands management. The CI capacity seems that has enough to manage the program. In Wakatobi NP, TNC & WWF play the role as project coordinator and collaboration with Page | 126 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success - Gili Matra Wakatobi Target 4 indicators by Y1 Number of: (y1=1 concept of co￾management developed); (y1= 2, y2=2 stakeholders forum meetings on co￾management held), ( y1=1 inputs on ecotourism business plan developed) & (25% ‘blue’ by y1, ‘50% ‘blue’ by y2, 75% ‘blue’ by y3 on level of management) Gili Matra Target 4 indicators by Y1 Number of: (1 management plan and zoning plan completed by y1 & approved by y2); (y1=1 ecological and socioeconomic survey implemented & disseminated); (y2=1, y3=1 management protocols/SOPs developed); 2 (20%) Only one of 4 target indicators was achieved for Wakatobi NP. WNP use MOF criteria for MPA management effectiveness, not E￾KKP3K 8 (80%) Three of four target indicators was fully achieved for Gili Matra MPA. District DKP, MOF, KKJI, IMACS, and Packard Foundation. However, they didn’t work yet with IMACS. In this site, WCS as project coordinator collaboration with District DKP, KKJI, Ecotrust & IMACS. The achievement indicated they have collaborated well. Page | 127 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success (y1=2, y2=2, y3=1 MPA training delivered, (y2=1 co-management mechanism concept developed) & (25% ‘green’ by y1, ‘50% ‘green’ by y2, 750% ‘green’ by y3 on level of management) Support the establishment and strengthening of MPA Networks - Bird’s Head Seascape (BHS) MPA Network MPA Network management effectiveness strengthened BHS MPA network Target 5 indicators by Y1 (Number of lesson learned documents produced on MPA Network development by y1=1) (Number of technical support trainings on operation of UPTD Raja Ampat by y1=1, y2=1) (Number of progress reports on development of RZWP3K Raja Ampat by y1=1, y2=1) (Number of MPA trainings delivered by 6 (60%) Three of five target indicators was fully achieved for BHS MPA network MPA. Training MPA more than target (5 trainings) In this site CI as coordinator for the project management, collaborated with West Papua Province Government, Raja Ampat District Government, KKPN Raja Ampat, BPSDM, UNIPA, NOAA, BAPPEDA, Walton Family Foundation, & Starling Resources. It seems that the team Page | 128 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success - Bali MPA Network y1=1, y2=1, y3=1) (Number of MOUs between BPSDM, NOAA, UNIPA, and CI on capacity development, y1=1) Bali MPA Network Target 3 indicators by Y1 (Number of Bali MPA Network concept notes (blueprints) endorsed by district & MOF, by y1= 1) (Number of Agreements from districts and MOF on the concept of Bali MPA Network by y1=1) (Number of MPA-related trainings delivered by y1=1, y2=1, y3=1) 3 (30%) Only one of three target indicators was fully achieved for Bali MPA network. Training MPA more than target (2) work well enough to get the results. In this site CI work with Bali Province Government, District Governments, MOF, BPSDM & local NGOs. A low target achieved indicated that project team was not well collaborated. Host Country System (HCS) Assessment and Design Feasibility of Host Country System (HCS) concept assessed and system designed. Target 3 indicators by Y1 10 (10%) All target indicators for Y1 was achieved. HCS consultant work well with MMAF, Ministry of Finance, BAPPENAS facilitated by MPAG team Page | 129 CTSP-I/MPAG Component: Activities (planned) Results & Intermediate results (target indicator) Effectiveness of the projects’ activities in achieving planned results Capacity of project teams to implement activities Important policy actions, relationships, & partnering arrangements that facilitated/hindered project success Assess MMAF capability to implement HCS MMAF capacity to implement HCS assessed. (Number of Indonesian HCS assessments, y1=1) MMAF staff trained in HCS design and implementation. (Number of trainings on design of HCS delivered, y1=1) HCS developed for MMAF: (Number of recommendations and HCS developed, y1=1) Note: Level of MPA management based on E-KKP3K: Red (MPA initiated), Yellow (MPA established), Green (MPA managed minimally), Blue (MPA managed optimally), Gold (MPA managed independently) ANNEX 6-2. Matrices for Evaluation Level 2 IMACS Page | 130 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing Degree to which Technical training to MMAF staff has been effective 2. Each training has different results Providing assistance for legislative and regulatory framework development and implementation 3 It is an ongoing process, many ad hoc policy paper request Supporting the development of the MMAF national five-year strategy 3 It is an ongoing process. It start a secondment staff promotion of sustainable fishing practices 2 it doesn’t show results yet improvement of central and district level policies or regulations to support ecosystems-based fisheries management 2 insignificant involvement of the national and local level government in setting up the demonstration sites improvement of GOI monitoring and data collection of fish catch 2 it covers only tuna fisheries and specific buyers, not enough representation of other fisheries in the area Development of livelihood program that offer alternatives during fishing moratorium periods to reduce rate of destructive fishing practices and lessen environmental impact 1 no such program exist in fisheries sites Increased awareness of vulnerable coastal population of potential disaster 4 through trainings the Page | 131 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing and climate change impacts and risk reduction practices in select communities. level of awareness increased Increased resilience of coastal communities to the economic impacts of climate change and disasters 2 uncertainty due to lack of follow up assessment on the outcomes of the trainings Assistance to local governments to develop policies, budgets and procedures that protect the coast from climate change-related impacts. 2 only limited commitment given by local government due to incomplete process of the program (not yet finalized prioritized action plan, etc) Assistance to the GOI for increasing number of high priority marine protected areas 3 effectively support national government in developing tools and measurement for effectively manage the areas Promotion of the effective management of MPAs 5 successfully established management effectiveness measurement as MMAF regulation Assistance to local governments for better allocating adequate resources 2 still work in Page | 132 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing and moving toward sustainable financing of MPAs. progress and not yet shown any progress at local level Improvement of fisheries database for MMAF to better understand the need for control of fishing level of effort and IUU fishing 1 no evidence that data collection was getting any better or improved Improvement in policy, regulatory, and enforcement cooperation among MMAF officials and Indonesian law enforcement agencies and local government 1 nothing can be traced back to shown impact of the program Training of coastal Community Enforcement Networks on environmental laws 2 increased awareness for the participants, but not give any outcomes in establishing better laws Legend Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing 1 = Not Effective 1 = Not Effective 1 = No Increase 1 = Not Effective 1 = No Improvement 2 2 2 2 2 = 3 = Effective 3 = Effective 3 = Increased 3 = Effective 3 = improve 4 4 4 4 4 = 5 = Very Effective 5 = Very Effective 5 = Highly Increased 5 = Very Effective 5 = Highly improved Page | 133 Page | 134 MPAG MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing Degree to which Technical training to MMAF staff has been effective 4 MPAG has developed three training modules namely basics MPA management, MPA Management Planning, training information system, & conducted MPA related field training. Number of people receiving MPA related training more than target (260). However, training module on Sustainable Fisheries Management in MPAs (EAFM indicator) is not finished yet by 2012 as well as monitoring as evaluation system to evaluate the effectiveness of training delivered is not designing yet. Providing assistance for legislative and regulatory framework development and implementation 3 MPAG has been supporting the MMAF for legislative aspect, namely: Ministerial Decree on MPA Networks, an MPA Management Effectiveness Guideline, a Ministerial Decree Page | 135 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing on Collaborative Management, a Ministerial Decree on Standard for Specific Competency (SKK), and secondment program on KKJI, BPSDM and SDI. However, by 2012 only one, DG KP3K was decreed on evaluation tool of MPAs effective management (E-KKP3K) Supporting the development of the MMAF national five￾year strategy 1 No MPAG activities directly deal to this issue promotion of sustainable fishing practices 4 MPAG supported the development of EAFM website (eafm-indonesia.net) and will be integrated into SDI-MMAF website to promote sustainable fisheries principle. This integration is to ensure sustainability and government ownership of the website. MPAG engaged a fishing company, Sea Delight, to implement EAFM principles in its fishing operation. However, the indicator of joint activities between MPAG and IMACS was not established and implemented Page | 136 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing yet ) improvement of central and district level policies or regulations to support ecosystems-based fisheries management 3 An ecosystem approach for fisheries management (EAFM) indicators have been finalized included pilot testing in some Fisheries Management Areas (FMAs). EAFM is still in concept, not be regulation and implementation yet as the result of MPAG support. improvement of GOI monitoring and data collection of fish catch Development of livelihood program that offer alternatives during fishing moratorium periods to reduce rate of destructive fishing practices and lessen environmental impact Increased awareness of vulnerable coastal population of potential disaster and climate change impacts and risk reduction practices in select Page | 137 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing communities. Increased resilience of coastal communities to the economic impacts of climate change and disasters Assistance to local governments to develop policies, budgets and procedures that protect the coast from climate change￾related impacts. Assistance to the GOI for increasing number of high priority marine protected areas 3 Indonesian government has target to achieve 20 million ha of MPA coverage by 2020. MPAG support to establish 2 new MPAs as 450,000 ha: West Kei Kecil (150,000 ha) & Berau (305,000 ha). USAID target 2.5 million ha newly designed MPAs Promotion of the effective management of MPAs 3 MPAG has been supporting the effective management of 8 million ha MPAs on 9 high priority sites. By 2012 indicated that there were Page | 138 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing different achievement related to sites (10 – 90%) or average 50 %. Assistance to local governments for better allocating adequate resources and moving toward sustainable financing of MPAs. 4 The working group of Trust Fund finalized two SOPs (institutional design and work mechanism) and background paper related to establishment of Trust Fund for marine conservation (90 % achieved by 2012). A model of sustainable financing strategy for effective MPA management has been establishing at Nusa Penida MPA and BHS MPA network with target achieved by 2012 is 50 & 60 %, respectively. Improvement of fisheries database for MMAF to better understand the need for control of fishing level of effort and IUU fishing 2 MPAG has activities of baseline data study, it will contribute to improve data of fish community, ecosystem & habitat, especially in the MPA selected sites, then Page | 139 MRP Level 2 Assessment Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing possibility use for IIU fishing management Improvement in policy, regulatory, and enforcement cooperation among MMAF officials and Indonesian law enforcement agencies and local government 2 MPAG supporting the local authority in establishing a management tool to protect biodiversity as well as fish resources and habitat from IUU fishing activities, e.g. in Wakatobi by co￾management approach Training of coastal Community Enforcement Networks on environmental laws Legend Institutional Development of MMAF Sustainable Fisheries Management Coastal Community Resilience (CCR) and Climate Change Adaptation Creating & managing MPAS. Improving capacity to reduce IUU fishing 1 = Not Effective 1 = Not Effective 1 = No Increase 1 = Not Effective 1 = No Improvement 2 2 2 2 2 = 3 = Effective 3 = Effective 3 = Increased 3 = Effective 3 = improve 4 4 4 4 4 = 5 = Very Effective 5 = Very Effective 5 = Highly Increased 5 = Very Effective 5 = Highly improved Page | 140 ANNEX 7: RESULTS FRAMEWORKS FOR IMACS AND MPAG Page | 141 MPAG Result Framework MMAF’s Strategic Plan 2010-2014 includes the following targets: 1. Establishment of MPAs spanning 2 million ha, starting from a baseline of700,000 ha in 2010; and 2. Effective management of MPAs spanning 4.5 million ha, starting from a baseline of 900,000ha in 2009. MPAG’s Results Framework is designed to support MMAF in achieving these targets). MPAG’s Objective: To improve MPA governance to yield optimal fisheries and biodiversity conservation benefits. MPAG’s Purpose: To strengthen MPA management through the development of a national MPA system linked with priority MPA field sites. R.2 Management effectiveness advanced in selected national and district MPAs and MPA networks Intermediate Results IR2.1 MPA establishment progressing towards 2020 goal. IR2.2 MPA management and operations supported. IR2.3 MPA network established and strengthened. R3. Feasibility of Host Country System (HCS) concept assessed and system designed Intermediate Results IR3.1 MMAF capacity to implement HCS assessed. IR3.2 MMAF staff trained in HCS design and implementation. IR3.3 HCS developed for MMAF. Potential funding mechanism for MMAF Source of funding for MPAs R.1. A sustainable National MPA System is established Intermediate Results IR1.1 MPA policies, regulations and guidelines developed. IR1.2 National and local capacity for sustainable MPA management strengthened. IR1.3 Decision support system developed. IR1.4 MPAs integrated into an ecosystem approach to fisheries management and marine spatial planning. IR1.5 MPA sustainable finance mechanism developed. Page | 142 ANNEX 8: CONFLICTS OF INTEREST & GOVERNANCE This evaluation of the USAID/Indonesia Marine Resources Program (MRP) was carried out over a period of 45 working days within the period January – April 2013. The report addresses the successes, shortcomings and the lessons learnt to date from the Program and provide recommendations for future USAID/I assistance in this sector. The three non-USAID members of the Evaluation Team were contracted by one of the main implementing partners of the MRP. The independence of these Team members was preserved by (a) having no reporting chain through the implementing partner, either at the local level or at the US Headquarters level and (b) being physically located in separate offices to those of the implementing partners project office. . Statements for each Team Member follow: Page | 143 Name Dr. Gary Morgan Title Consultant Organization Morgan Consulting Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID-MMAF Marine Resources Program (MRP), Indonesia I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. X Page | 144 Name Dr. Darmawan Title Fisheries Specialist Organization Independent Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID-MMAF Marine Resources Program (MRP), Indonesia I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. Page | 145 Name Dr. Am Azbas Taurusman Title Biodiversity Specialist Organization Independent Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID-MMAF Marine Resources Program (MRP), Indonesia I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation.