MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT (CCAP) MIDTERM EVALUATION January 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by an Evaluation Team from ECODIT and Verde Azul. Prepared for USAID/Mozambique under REPLACE IDIQ Prime Award Number AID-OAA-I-14- 00016, Task Order Number AID-656-TO-16-00004 This analysis is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of ECODIT and do not necessarily reflect the views of USAID or the United States Government (USG). PREPARED BY The Evaluation Team 1300 Wilson Blvd., Ste. 920 Arlington, VA 22209 USA Tel: +1 703 841 1883 Fax: +1 703 841 1885 Web: www.ecodit.com Cover photo: Area of natural and replanted mangroves (upper right) at Icidua, a neighborhood of Quelimane Municipality situated on land only a few meters above sea level, and therefore highly vulnerable to flooding from high tides and storm surges associated with tropical cyclones and from rising sea level. Photo credit: B. Byers/ECODIT MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT (CCAP) MIDTERM EVALUATION January 2017 REPLACE IDIQ Prime Award Number AID-OAA-I-14-00016, Task Order Number AID-656-TO-16-00004 DISCLAIMER The views expressed in this publication are those of the Evaluation Team and do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Acknowledgements...................................................................................................................................... iii Abbreviations and Acronyms....................................................................................................................... iv Executive Summary.......................................................................................................................................1 1. Evaluation Objectives and Questions..................................................................................................6 1.1 Evaluation Objectives....................................................................................................................6 1.2 Evaluation Questions....................................................................................................................7 2. Project Background.............................................................................................................................8 3. Evaluation Methodology and Limitations...........................................................................................9 3.1 Evaluation Approach.....................................................................................................................9 3.2 Methodology.................................................................................................................................9 3.3 Limitations...................................................................................................................................10 4. Results, Conclusions, and Recommendations...................................................................................12 4.1 Results.........................................................................................................................................12 4.2 Conclusions.................................................................................................................................28 4.3 Recommendations......................................................................................................................31 Annexes.......................................................................................................................................................35 Annex A: References Cited and Documents Reviewed...........................................................................35 Annex B: Evaluation Statement of Work ................................................................................................39 Annex C: Biographical Sketches of Evaluation Team..............................................................................58 Annex D: Final Evaluation Design and Work Plan...................................................................................60 Annex E: Key Informants and Other Persons Contacted ........................................................................80 Annex F: Evaluation Questions and Sub-Questions Guide/Script...........................................................83 Annex G: Lessons Learned in CCAP Mangrove Restoration....................................................................93 ACKNOWLEDGEMENTS More than 50 people, representing beneficiaries, implementers, managers, and other stakeholders of the Mozambique Coastal City Adaptation Project (CCAP), willingly and graciously made time to talk to us, and freely shared their experiences, knowledge, opinions, and insights about the successes and challenges of this important project. The Evaluation Team would like to express our deep appreciation to all of them. We especially wish to thank CCAP staff in Maputo: Olanda Bata, the Chief of Party, and Casimiro Antonio, Deputy Chief of Party, who facilitated our communication with their field staff in Pemba and Quelimane and with project stakeholders in those cities; and Gilberto Muai, the CCAP Monitoring and Evaluation (M&E) Specialist, who worked with us to understand the details of the project’s M&E system. We also wish to thank Sádia Chicalia, Maria “Loly” Cox, Angela Hogg, and Marcia Matenja of USAID/Mozambique, who provided guidance and support as we prepared our Evaluation Design and Work Plan, and facilitated communication with CCAP staff. These USAID/Mozambique staff, including Daniel Evans, provided constructive comments on an earlier draft of this report. Senior staff of USAID/Mozambique, including the Mission Director and Deputy Mission Director, attended our presentation of preliminary findings, and their thoughtful comments and suggestions were also useful and appreciated. We received much information and heard many viewpoints from the people we met, and if in any way we have misunderstood them or misrepresented their views, the fault is ours. ABBREVIATIONS AND ACRONYMS ARCC African and Latin American Resilience to Climate Change ACCRA Africa Climate Change Resilience Alliance AJAQ Association of the Young Friends of the City of Quelimane (Associação dos Jovens Amigos da Cidade de Quelimane) ANAMA Association of the Inhabitants and Friends of Madal (Associação dos Naturais e Amigos da Madal) APS Annual Program Statement CCA Climate Change Adaptation CCAP Coastal City Adaptation Project CCIS Climate Change Integration Support CCRD Climate Change Resilient Development CMCP Municipal Council of the City of Pemba (Conselho Municipal da Cidade de Pemba) CMCQ Municipal Council of the City of Quelimane (Conselho Municipal da Cidade de Quelimane) DEPTADER Provincial Directorate of Lands, Environment and Rural Direção Provincial de Terra, Ambiente e Desenvolvimento Rural DRR Disaster Risk Reduction ETOA Environmental Threats and Opportunities Assessment FRELIMO Mozambique Liberation Front (Frente de Libertação de Moçambique) GOM Government of Mozambique INGC National Institute for Disaster Management (Instituto Nacional de Gestão de Calamidades) IPCC Intergovernmental Panel on Climate Change LGSAT Local Government Self-Assessment Tool LOP Life of Project M&E Monitoring and Evaluation MAE Ministry of State Administration (Ministério da Administração Estatal) MITADER Ministry of Lands, Environment and Rural Development (Ministerio da Terra, Ambiente e Desenvolvimento Rural) MOU Memorandum of Understanding NGO Non-governmental Organization NRM Natural Resources Management PLA Local Adaptation Plan (Plano Local de Adaptação as Mudanças Climáticas) RENAMO Mozambican National Resistance (Resistência Nacional Moçambicana) SOW Statement of Work UEM Eduardo Mondlane University (Universidade Eduardo Mondlane) UNILURIO Lurio University (Universidade Lurio) UNISDR United Nations International Strategy for Disaster Reduction USAID United States Agency for International Development USFS United States Forest Service WRI World Resources Institute WWF World Wildlife Fund MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 1 EXECUTIVE SUMMARY EVALUATION OBJECTIVES AND QUESTIONS This midterm evaluation of the Mozambique Coastal City Adaptation Project (CCAP) was designed to determine how CCAP is performing, and provide an objective and independent view of progress toward the project’s expected results and objectives. The results and conclusions of the evaluation lead logically to six recommendations for improving CCAP implementation in its final years, and also for planning future programs with related climate change adaptation goals. The evaluation was designed to answer the seven general evaluation questions posed by USAID/Mozambique in the Evaluation Statement of Work. The evaluation team reviewed existing documents and gathered primary information from key informants and communities working with CCAP to answer the evaluation questions, and to evaluate the project’s progress toward its three objectives and its goal of increasing “climate resilience in selected Mozambican coastal cities.” PROJECT BACKGROUND About 60% of Mozambique’s population lives in coastal cities, and those cities account for a disproportionate share of national economic activity. Many coastal cities and communities are poorly prepared for extreme climate events such as floods and tropical storms that are a normal part of climate variability, and such events will likely become more frequent and extreme as the climate changes. USAID selected the municipalities of Pemba and Quelimane to develop, test, and implement a series of activities that aim to improve the provision of climate-resilient urban services by municipalities and increase adoption of climate resilience measures by communities. The project’s goal is to increase climate resilience in selected Mozambican coastal cities. The project’s “theory of change” and Results Framework proposed that this goal can be reached if: 1) municipalities have the capacity to incorporate climate change adaptation into their planning processes and provide more climate-resilient municipal services, and 2) communities in the municipalities adopt and implement more climate-resilient measures. EVALUATION METHODS AND LIMITATIONS Our approach to this midterm evaluation of CCAP had four elements. It used a participatory, transparent, “friendly” approach that recognizes potential sensitivities in any evaluation process. We also employed a mix of information-gathering methods to “triangulate” findings and control for biases to the extent possible. We had the expectation of finding both successes and unmet challenges – that is, we took the position that both positive and negative results, both of which can contribute to adaptive learning, would be expected, and that negative findings are as important and useful as positive ones for the goal of adaptive learning. Finally, we wanted the evaluation to be an objective, independent, unbiased process, conducted with complete professional integrity. We employed various methods to gather the information that provides evidence to answer each of the seven general evaluation questions posed by USAID-Mozambique. We used both qualitative and quantitative information from existing secondary data sources, such as project quarterly reports, and from primary sources such as key informant interviews, meetings, and site visits in Pemba and Quelimane. Each of the seven evaluation questions is complicated and multi-faceted, because each relates to many different project activities, accomplishments, and challenges. Because of this complexity, we developed sub-questions for each of the seven questions and organized them into interview “scripts” or guides, which were used to structure our interviews with key informants of various kinds (e.g., CCAP MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 2 staff, municipal staff, partners from universities or NGOs, staff from national government agencies working at the national or local level, and communities). Site visits provided a first-hand view of some of the successes and challenges of the program and a chance to meet with community members. The visits – to the most vulnerable areas and communities in both Pemba (Paquitequete) and Quelimane (Icidua and Mirazane) – underscored the lack of basic services such as water supply and sanitation and highlighted the challenges of trying to create long-term resilience to climate change or even to natural climate risks in the short-term when basic development needs are so great. Any evaluation has limitations and uncertainties. One limitation of our methodology is that it depended in part on data from CCAP’s M&E system and reporting, as is typical for all evaluations. Our review of project M&E data and project design documents provided some of the evidence of progress that we needed to answer the evaluation questions, as we indicated it would in our Evaluation Design. Another limitation of our methodology relates to the participatory way in which we conducted the evaluation. Our information was based, to a significant degree, on the opinions of project staff, partners, and stakeholders. Because each type of stakeholder has interests in, and sensitivities with regard to, the project, some biases are inevitable in a participatory evaluation. Another limitation of this midterm evaluation is that some CCAP activities are much farther along than others. For example, activities under Objective 3, Increase local awareness of economic risk-management tools, such as insurance plans and contingency funds, for at-risk urban infrastructure and livelihoods, were intended to start later in the project, and were then modified in April 2016 – and so we found little or no information about them and cannot evaluate this component of the project. A final limitation of this evaluation stems from the fact that it is a midterm evaluation in a short, five year project with the ambitious goal of increasing climate resilience in selected Mozambican coastal cities. Climate resilience is complex and can be difficult to define and measure. CCAP implementation began in January, 2014, and it is probably too early to assess the project’s impact at the level of the project goal. RESULTS Seven general evaluation questions formed the organizing structure for the evaluation, and the Evaluation Team used that structure in reporting results from both the review of existing and secondary information and from primary information gathered from interviews with key informants, meetings, and site visits. According to both existing documents and interviews with key informants, CCAP has been partially successful in assisting Pemba and Quelimane municipalities to incorporate climate change adaptation into their planning processes. In general the biggest successes were seen as developing and promoting certain “tools” (i.e., processes, procedures, methodologies) for incorporating climate change adaptation into planning in both municipalities, including Local Adaptation Plans (PLAs), support for SIGIU – the Integrated Urban Information Management System, a “Climate Change Adaptation and Resilience” short course, and vulnerability maps, especially as linked to the land cadaster systems of the municipalities. There are big differences in CCAP’s successes and challenges in Pemba and Quelimane, according to CCAP field staff and project partners. In Pemba, where the municipal government is of the same political party as the district government, “the municipality opens all the doors, and it’s easy to work,” said a CCAP staff member. In Quelimane, different political parties control the district government and the municipality, creating communication problems and roadblocks to implementation. In terms of CCAP’s success in helping municipal stakeholders (communities, civil societies, NGOs, and universities) to actually implement adaptation measures, our key informants expressed the view that CCAP has been successful in contacting and engaging stakeholders since the beginning of project implementation. Engagement is certainly a necessary, but not sufficient, step toward actual implementation, which is complex and has many steps and requirements beyond “engagement.” Technical aspects of the engagement must be correct for effective implementation, for example, such as MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 3 in the case of mangrove restoration activities or climate resilient housing design. A review of project M&E data for the four indicators related to this project objective showed mixed progress towards the targets set by the project itself in its 2015 M&E Plan (refer to Table 4.1 for a list of CCAP indicators). Regarding progress towards CCAP’s overall goal, we evaluated the extent to which CCAP increased the climate resilience of the most vulnerable populations in Pemba and Quelimane. The “theory of change” underlying the Results Framework for the project argues that if the capacity of relevant institutions – municipalities, primarily – is improved, then that capacity will eventually enable, or translate to, on-the￾ground actions that will reduce climate vulnerability and increase resilience. Our key informants, and existing CCAP M&E data show that that institutions are being engaged and their capacity improved. CCAP field staff told us they always make efforts to involve women and young people in project activities and provided examples that show how they have done so. To determine the extent to which CCAP has incorporated gender and youth into activities, we reviewed existing M&E indicator data from quarterly reports, as well as other relevant existing documents. Quarterly reports show that 28% of project activities have engaged young people, exceeding the life of project target of 20%. The Evaluation Team was told that although the CCAP M&E system is capturing data disaggregated by gender, those disaggregated data are only provided upon Mission request for annual reports, so we are unable to evaluate the extent to which project activities have involved both women and men. Our review of CCAP M&E information showed that the project has exceeded or nearly reached targets for six of its 12 indicators by the end of FY2016, and is behind its own proposed timeline on the other six. The project seems to be quite strong in developing and providing municipalities with tools for planning for, and responding to, climate variability and change and in training people to use the tools. Based on the M&E data, it also seems to be successful in engaging communities, including young people, in certain activities. The project has faced some significant challenges and obstacles in implementation, some of which have led to delays in a number of project activities. The within-project challenges have been one cause of the mixed progress toward some of the indicator targets in the M&E Plan. In general, project staff feel that they have done a good job in adapting to or overcoming many of the challenges they have faced. Project staff and partners anticipate further challenges and obstacles over the remaining years of the project, including the deteriorating economic situation in Mozambique, and renewed armed conflict between RENAMO and FRELIMO supporters in some provinces, which is creating insecurity and accelerating rural to urban migration, further stressing municipal planning and services. CONCLUSIONS AND RECOMMENDATIONS Six conclusions emerged from the Evaluation Team’s synthesis and interpretation of the data and information collected. Those conclusions in turn led logically to six recommendations. 1) CCAP has been generally successful in assisting municipalities to incorporate climate variability and change into their planning processes. This conclusion is supported by evidence both from our review of existing information and our primary information gathering through interviews with key informants and meetings with communities. We therefore recommend that CCAP maintain and expand support for the most successful tools. Continuing support is needed in Pemba and Quelimane to solidify and institutionalize the use of the most important tools, and these same tools should be replicated in Nacala. Some of the successful tools for integrating climate change adaptation into municipal planning processes could form the foundation of some components of future projects. 2) Better coordination and collaboration between municipalities and national-level institutions, facilitated by CCAP, would strengthen and help sustain some aspects of the project. The main tools so far developed under CCAP will have a bigger impact if they are replicated in other coastal cities, but municipalities do not have the mandate or funding for scaling up. National government institutions need MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 4 to be involved, but so far it is not clear which institution(s) will eventually take over those tools and be responsible for scaling them to a national level. Therefore, we recommend that CCAP renews or initiates activities to improve the coordination with, and balance the engagement of, municipalities and the relevant national institutions on climate change adaptation issues. CCAP should develop a strategy for handoff of the key tools to an institution that can and will take responsibility for replicating them on a national scale. Future projects should be designed to effectively balance efforts at the national and the local levels. 3) The CCAP M&E system needs to be tightened and strengthened. In general, the Evaluation Team found that CCAP was tracking the significant progress it is making in its quarterly reports. In some cases, however, we found it difficult to track and map activities to annual work plans and to M&E Plan indicators. We therefore recommend that CCAP review and revise CCAP’s M&E system and reporting practices. Clearly correlating or “mapping” activity reporting to the project’s Results Framework and work plan should be an organizing principle for future progress reporting, and project staff told us that the Year 4 work plan does this. We recommend that the project retrospectively analyze and report on gender-disaggregated indicator results whenever possible. We also recommend that for existing and future CCAP activities, and in future projects, the M&E and reporting system be linked with the project results framework and work plans in an explicit way, to improve both adaptive management of the project during its life, and to strengthen the communication of results from project activities. 4) The mangrove restoration activity in Quelimane has serious design flaws that should be corrected. The Evaluation Team learned during fieldwork for this evaluation that a rapid reassessment of the mangrove restoration activities in Quelimane had recently been completed by mangrove experts from Eduardo Mondlane University (Bandeira and Macamo 2016). This study is a step toward bringing CCAP mangrove restoration in line with science-based best practices being used around the world. We now understand that CCAP is in the process of awarding grants to NGOs in Quelimane to begin to implement the recommendations of the recent assessment, and we recommend that CCAP continue to adapt and modify the mangrove restoration component of the project. The Evaluation Team also recommends that the CCAP mangrove restoration work become a case study in the USAID-funded SWAMP Project’s “Mangrove Restoration Best Practices Manual for East Africa,” now being developed. We also recommend that an assessment of the status of the mangrove hydrological monitoring program in Quelimane be conducted immediately, and all available data collected to date be provided to the US Forest Service team which designed the monitoring program for analysis. A more detailed evaluation of the mangrove restoration component of CCAP is provided in Annex G. In the future, any mangrove regeneration work to be included in a coastal project as a climate change adaptation measure should be designed from the beginning with expert technical advice and follow scientifically-established best practices. 5) The Evaluation Team found that the proposed CCAP Social and Behavior Change Communications (SBCC) Strategy is very complicated, technical, and difficult to understand. We reviewed the recently finalized action plan for implementing the strategy. It reinforces the impression that the SBCC strategy focuses almost exclusively on trying to reduce “awareness and knowledge” barriers to the adoption of desired behaviors through a variety of communications activities, when in fact the main barriers to the desired behaviors do not seem to be awareness and knowledge factors, but things like economic costs and lack of enforcement of existing regulations.The Evaluation Team recommends that the SBCC Strategy and its action plan be simplified and streamlined to focus on lowering the barriers to a few of the most important climate-adaptive behaviors through activities that address the most important factors creating those barriers, not only lack of awareness and knowledge. All relevant factors motivating or acting as barriers to desired behaviors should be identified and targeted for interventions, including economic factors and enforcement of laws and regulations. Future climate change adaptation projects would benefit from the use of a behavior-change framework in their design from the beginning. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 5 6) The relative emphasis placed on short-term climate disaster risk reduction (DRR) versus long-term climate change adaptation (CCA), and both of these relative to basic development interventions, would benefit from further analysis and thought to make the balance more explicit and clear. The development-DRR-CCA “nexus” provides some major philosophical and practical challenges to a project like CCAP. The evaluation results lead us to questions such as:  How can the project balance disaster risk reduction (DRR) and climate change adaptation (CCA)?  How can the project build resilience through DRR and CCA when basic development needs are so great in the most vulnerable communities? We recommend that CCAP review its efforts to balance long-term climate change adaptation (CCA) and short-term disaster relief reduction (DRR) efforts, and improve integration with development interventions, and to rebalance project activities if needed during the remaining years of the project. Practical actions to improve balance and integration could start by emphasizing the CCAP planning tools that point from short-term DRR toward long-term CCA. For example, municipal vulnerability maps linked with municipal cadastral systems are a tool to steer building and development away from vulnerable zones so that risks from climate-related extreme events are not worsened in the future. CCAP could help build the capacity of municipalities to enforce the restrictions on building in vulnerable zones, and steer municipal services away from those zones, with the knowledge that people will have to be relocated from some of them in the future. Part of achieving an effective balance between long-term climate resilience and short-term disaster preparedness involves deliberately not supporting activities that would provide incentives for communities to settle or expand in areas of high vulnerability to future climate risks. Future projects should be designed carefully to integrate climate change adaptation, disaster risk reduction, and development, and take a long-term perspective that favors interventions that will be sustainable in the face of climate change projections. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 6 1. EVALUATION OBJECTIVES AND QUESTIONS 1.1 Evaluation Objectives This midterm performance evaluation had the following objectives:  Determine how CCAP is performing relative to its three integrated objectives;  Provide an objective view of progress towards the project’s expected results;  Identify any possible gaps in project performance that could hinder success;  Help USAID/Mozambique and its implementing partner for CCAP to determine what changes may be necessary to solidify progress during the remaining project period; and  Provide lessons learned to inform climate change adaptation efforts in other coastal cities in Mozambique and elsewhere. Evaluations that objectively review the performance of a project are an important tool for adaptive learning. Effective evaluations with this objective require the cooperation and participation of the designers, funders, and implementers of the project being evaluated, but each of these groups is invested in, and has sensitivities regarding, project performance. Therefore, transparent evaluation methods and trust among all participants in the evaluation were essential elements of our approach to the evaluation. We viewed the CCAP Midterm Evaluation as a learning-oriented exercise, and conducted it in a participatory and “friendly” manner. USAID/Mozambique invited the Evaluation Team to present recommendations of two kinds based on our findings: 1) practical, achievable recommendations for mid-course adjustments to adaptively-manage and improve CCAP during the remaining life of the project, and 2) ideas that could inform the design for follow-on projects with related themes. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 7 1.2 Evaluation Questions The evaluation described here was designed to answer the seven general evaluation questions (Exhibit 1) posed by USAID-Mozambique in the Evaluation Statement of Work (Annex B). We compared the CCAP Results Framework and list of indicators given in the M&E Plan (USAID CCAP, 2015) with the evaluation questions. Although there is not an explicit, one-to-one correspondence, the Evaluation Team believes that answering the seven general evaluation questions also will provide sufficient evidence to evaluate CCAP progress to toward its three objectives. EXHIBIT 1: EVALUATION QUESTIONS 1a. To what extent has CCAP been successful in assisting Pemba and Quelimane municipalities to incorporate climate change adaptation into their planning processes? 1b. To what extent has CCAP been successful in helping the relevant stakeholders of municipalities to implement adaptation measures? (communities, civil societies, NGOs, and universities)? 2a. To what extent has CCAP increased climate resilience of the most vulnerable populations of Pemba and Quelimane municipalities, including those living in the most vulnerable areas? 2b. To what extent has CCAP incorporated gender considerations and youth into implementation of its activities? 3a. To what extent is CCAP prepared to achieve the project’s objectives over the next 2.5 years? 3b. What are some challenges or obstacles (related to staffing, finances, etc.), reported by project personnel and what is the project’s capacity to respond to those challenges? 3c. What are some major implementation obstacles or challenges and opportunities (reported by the municipalities and other stakeholders) anticipated over the next 2.5 years of implementation? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 8 2. Project Background CCAP’s overall project goal is to increase “climate resilience in selected Mozambican coastal cities.” About 60% of Mozambique’s population lives in coastal cities, and those cities account for a disproportionate share of national economic activity. Many of them are poorly prepared for extreme climate events such as floods and tropical storms that are a normal part of climate variability, and such events are predicted to become more frequent and extreme as climate changes. Underlying the vulnerability to extreme climate events and to climate change is a fundamental lack of basic services like water supply, electricity, sanitation, and solid waste management in many cities. Many options exist for reducing the vulnerability, and increasing the resilience, of Mozambique’s coastal cities to the emergencies and disasters that climate events can cause in unprepared cities. CCAP’s M&E Plan lists as goal-level Indicator #2 “Number of stakeholders with increased capacity to adapt to the impacts of climate variability and change as a result of USG assistance [emphasis added]. This is a GCC required indicator 4.8.2-26 at the outcome level. (USAID CCAP, 2015, M&E Plan, p. 15). According to CCAP, USAID “retired” this indicator at the end of FY2016, and is recommending its replacement with new indicator EG.11-5: “Number of people supported by the USG to adapt to the effects of climate change.” We note that the meaning is similar except that the word “climate variability” is no longer used. The other two goal-level (outcome-level indicators in the CCAP M&E Plan are Standard Foreign Assistance Indicators in the State Department F-framework (US Department of State, 2016), and refer only to “climate change” (i.e., they also do not use the word “climate variability”). The Results Framework for the project proposes three underlying objectives proposed for working toward the project goal: Objective 1: Improve the provision of climate-resilient urban services by municipalities; Objective 2: Increase adoption of climate resilience measures by communities, civic, and community organizations, including civil society, nongovernmental, and faith-based organizations; and Objective 3: Increase local awareness of economic risk-management tools, such as insurance plans and contingency funds, for at-risk urban infrastructure and livelihoods. Each objective is further broken down into intermediate results (IR), under which corresponding activities are grouped. “Specific intended results include: 1. Increased understanding of urban adaptation issues by municipal authorities and increased application of adaptation-relevant management options; 2. Decreased vulnerability to climate change for the population of select coastal cities; 3. Increased local capacity for managing resources to adapt to climate change; and 4. Synthesis and dissemination of lessons learned regarding coastal adaptation in urban settings, which can be applied by other coastal cities and future USAID urban adaptation efforts.” (USAID-CCAP, 2015, M&E Plan, p. 1) The FY2015 Annual Report, which covers Oct. 1, 2014 – Sept. 30, 2015, for the first time in project documents describes three CCAP “action themes”: creating tools, engaging communities, and scaling up – which it says will organize activities designed to achieve the project’s three objectives. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 9 3. EVALUATION METHODOLOGY AND LIMITATIONS 3.1 Evaluation Approach Our perspective on this midterm evaluation of CCAP is that it should:  Use a participatory, transparent, “friendly” approach that recognizes potential sensitivities in any evaluation process;  Be an independent, unbiased process, conducted with complete professional integrity;  Use a mix of information-gathering methods to “triangulate” findings; and  Have the expectation of finding both successes and unmet challenges – that is, take the position that both positive and negative results that can contribute to adaptive learning are expected, and negative findings are as important and useful as positive ones for the goal of adaptive learning. 3.2 Methodology The methodology we used was straightforward, technically sound, and appropriate to the Mozambican context. We employed various methods to gather the information that provides evidence to answer each of the seven general evaluation questions posed by USAID-Mozambique. Our methodology used information both from existing secondary data sources, such as quarterly reports and project M&E data, and from primary information collected by the Evaluation Team during meetings, interviews, and site visits in Mozambique. We gathered both qualitative and quantitative information. Our information￾gathering process was designed to allow for cross-checking (“triangulating”) and validating results to the maximum extent possible. Our main method of gathering primary information was through semi-structured interviews with key informants, representing the institutions, partners, and stakeholders working with CCAP. We developed an interview guide, or “script,” for use in these key informant interviews (see Annex F: Evaluation Questions and Sub-Questions Guide/Script). The interview guide was tailored to the type of key informant we were interviewing (e.g., project staff, municipal staff, community representatives, university faculty). USAID/Mozambique provided us with a list of institutions currently engaged with the CCAP, we obtained information for points-of-contact in these institutions from the project, and we scheduled meetings to interview representatives of those institutions. Each of the seven general evaluation questions posed in the SOW for this midterm evaluation is complicated and multi-faceted, because each relates to many different project activities, accomplishments, and challenges. Evaluation Question1a, for example, deals with municipal planning capacity and what are called “tools” for climate change adaptation planning, and there are at least five such tools or processes that the project has developed and supported to enhance municipal planning capacity. Because of the complexity of the evaluation questions, we needed sub-questions for each in order to provide evidence to answer them. We developed these sub-questions and organized them in what we called interview guides, which we used to structure our interviews with key informants of various kinds (e.g., CCAP staff, municipal staff, partners from universities or NGOs, staff from national government agencies working at the national or local level, and communities). The interview or focal discussion group guides had a combination of semi-quantitative, semi-structured, and open ended MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 10 questions. A complete list of the sub-questions we used in our key informant interviews, organized under the seven general evaluation questions posed for the evaluation, is given in Annex F: Evaluation Questions and Sub-Questions Guide/Script. Notes from the key informant interviews were analyzed, providing both qualitative and quantitative information for answering the evaluation questions. Most interviews lasted approximately an hour. During the interviews, the guides were not used rigidly, but as a flexible tool to gather the information we needed to answer the evaluation questions. Depending on the interviewee’s experience with, and knowledge of, the project, the exact wording or order of questions was sometimes adapted in order to obtain their opinions, but the Evaluation Team never suggested responses to the key informants or guided them toward expected or anticipated opinions. The interview and discussion guides worked well with the different types of key informants, and elicited useful information for the evaluation. The Evaluation Team interviewed more than 50 individual “key informants” to gather firsthand information and opinions for the evaluation. We interviewed seven people from three institutions in Maputo, 23 individuals from seven institutions in Pemba, and 13 individuals from eight institutions in Quelimane. The team also interviewed CCAP project staff in Maputo, Pemba, and Quelimane; Chemonics and USAID staff in Washington, D.C.; and U.S. Forest Service staff in South Carolina. Email communication was used with some key stakeholders to follow up on topics of special interest and increase our understanding of certain issues. A full list of these persons and institutions, with contact information, can be found in Annex E: Persons Contacted/Interviewed. The site visits provided a first-hand view of some of the successes and challenges of the program, as well as a chance to meet with community members. The visits – to the most vulnerable areas and communities in both Pemba (Paquitequete) and Quelimane (Icidua and Mirazane) – underscored the lack of basic services such as water supply and sanitation, and highlighted the challenges of trying to create long-term resilience to climate change or even to natural climate risks in the short-term when basic development needs are so great. 3.3 Limitations Any evaluation has certain limitations and uncertainties. One limitation of our methodology is that it depended to a significant degree on data from CCAP’s M&E system and reporting to provide the evidence of progress that we needed to answer the evaluation questions. In reviewing quarterly and annual reports from the project, we found some inconsistencies in reporting, and sometimes had difficulty understanding and interpreting reported indicator data. Another limitation of our evaluation methodology relates to the participatory way in which we conducted the evaluation. Our information was based, to a significant degree, on the opinions of project staff, partners, and stakeholders. Because each type of stakeholder has interests in, and sensitivities with regard to, the project, some biases are inevitable in a participatory evaluation. However, these people are the people who know more about the project than anyone else, and their information and opinions are all important and valuable. Our evaluation methodology avoids and mitigates such biases whenever possible through comparison of opinions from different key informant groups, which we call “triangulation”. Our job, as an Evaluation Team, is to listen to all the perspectives and try to synthesize and “triangulate” them. We did not assume that anyone’s individual view is complete or completely accurate and unbiased. No one person interviewed had a complete overview of the project; some had been involved in only some of the project’s components or activities. Although we interviewed more than 50 people, the small sample size for quantitative or semi￾quantitative questions precludes statistical analysis of differences, for example, between responses from the same types of key informants from Pemba and Quelimane. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 11 Another limitation of this midterm evaluation is that some CCAP activities are much farther along than others. For example, activities under Objective 3, “Increase local awareness of economic risk-management tools, such as insurance plans and contingency funds, for at-risk urban infrastructure and livelihoods,” were intended to start later in the project, and were then modified in April 2016 – so we found little or no information about them, and cannot evaluate this component of the project. On the other hand, the mangrove restoration (for example) has been underway for quite a while, and we can provide a fairly robust evaluation of that component. Another possible uncertainty stems from the fact that this is a midterm evaluation in a short (five year) project with the ambitious goal of increasing “climate resilience in selected Mozambican coastal cities.” Climate resilience is complex, and difficult to define and measure. CCAP implementation began in January, 2014, and it is probably too early to assess the project’s impact at the level of the project goal. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 12 4. RESULTS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 Results 4.1.1 GENERAL OBSERVATIONS ON THE CCAP MONITORING & EVALUATION SYSTEM A review and analysis of existing, secondary information was an important component of this evaluation, and contributed to our answers to the seven evaluation questions. The Evaluation Team reviewed and analyzed CCAP documents provided by USAID and project staff as well as other relevant documents and reports we obtained from other sources. We compiled cumulative indicator data from CCAP quarterly reports, including the first three quarters and an annual report from Fiscal Year (FY) 2014, four quarters and an annual report from FY2015, and four quarterly reports from FY2016. Annual targets over the LOP were obtained from Annex A of the 2015 M&E Plan, and these allowed us to evaluate the project’s progress according to its own proposed schedule. Life of Project (LOP) targets are those proposed by the project itself in the 2015 M&E Plan, unless modified in the 2016 CCAP Modification (USAID/Mozambique, 2016). Table 4.1 shows cumulative results for each of CCAP’s current 12 indicators compared to expected results at the end of FY2016 and at the end of the project (note that the project originally listed 13 indicators, but dropped Indicator #6 in FY2016). TABLE 4.1. CCAP CUMULATIVE RESULTS AND LIFE-OF-PROJECT TARGETS Indicators by Goal, Objective, and IR Current Cumulative (FY2014-FY2016) End of FY2016 Targets (from 2015 M&E Plan) and Cumulative % LOP Targets (from 2016 Modification) and Cumulative % Project Goal: Climate Resilience in Selected Mozambican Coastal Cities Increased 1. Numerical score on UNISDR's Local Government Self-Assessment Tool (LGSAT) (Impact) Pemba +0.23 Queli. +0.06 None established None established 2. Number of stakeholders with increased capacity to adapt to the impacts of climate variability and change as a result of USG assistance (Outcome, GCC required indicator 4.8.2-26)* [GCC EG11.1-1 and GCC EG11.3-1] 980 3,000 (33%) 6,000 (16%) 3. Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance (Outcome, F Indicator 4.8.2-28)* [GCC EG11.2-1 and GCC EG11.2-2] 34 70 (49%) 100 (34%) 4. Number of institutions with improved capacity to assess/address climate change risks issues as a result of USG assistance 26 18 (144%) 20 (130%) MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 13 (Outcome, F Indicator 4.8.2-14) [GCC EG11-3] Objective 1: Improve the provision of climate-resilient urban services by municipalities 5. Number of CCA or DRR tools, technologies and methodologies developed, tested and/or adopted as result of USG assistance (Outcome) [GCC EG11.1-3] 25 9 (277%) 20 (125%) 6. Amount of investment mobilized from private and public sources for CCA or DRR as a result of USG assistance (Outcome, F Indicator 4.8.2-10) GCC EG11-4] ** Indicator dropped from M&E in 2016 Modification Dropped in 2016 Modification and omitted in FY2016 Q4 quarterly report ----- ----- IR 1.1 Municipal capacity to apply urban adaptation measures through science and analysis increased 7. Number of person hours of training completed in climate change as a result of USG assistance (Output, F Indicator 4.8.2- 29) 27,142 6,550 (414%) 20,000 (136%) 8. Number of proposals submitted for CCA or DRR projects as result of USG assistance (Output) 5 8 (63%) 10 (50%) IR 1.2 Application of management, soft engineering, and hard engineering climate adaptation measures by municipal authorities through effective citizen engagement increased 9. Area (hectares) impacted by at least one CCA or DRR intervention implemented with citizen input per year (Outcome) 1,106 1.050 (105%) 3,000 (37%) Objective 2: Increase adoption of climate resilience measures by communities, civic and community organizations including civil society, NGOs and faith-based organizations 10. Number of people with increased knowledge of climate change impacts and adaptation strategies as a result of USG assistance (Outcome) [GCC EG11.3-2] 584 500 (117%) 600 (97%) IR 2.1 Citizen knowledge of local climate change vulnerabilities and adaptive options increased 11. Number of person-contact hours of information disseminated about climate change vulnerabilities and adaptive options (Output) 1,530,576 2,050,000 (75%) 3,000,000 (51%) IR 2.2 Community organizations’ ability to implement a local set of risk mitigating measures improved 12. Proportion of CCA or DRR interventions implemented with community contributions (Outcome) 100% 24% (417%) 50% (200%) IR 2.4 Contributions of women, men, boys, and girls to climate change adaptation more equitable 13. Proportion of individuals engaged in CCAP activities who are youth (Output) 28% 43% (65%) 20% (140%) MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 14 Table 4.1 shows that by the end of FY2016, half of CCAP’s 12 current indicators exceeded the targets set for them by the project itself, according the schedule given in Annex A of the 2015 CCAP M&E Plan, and that five of 12 have already exceeded their proposed LOP targets. However, it also shows that by the end of FY2016, six of the 12 project indicators are behind schedule to achieve the targets proposed by CCAP itself in the M&E Plan. These mixed results indicate that progress towards the intended results has not been consistent. This information can help to generate hypotheses about why some of the intended results have been slower and harder to move toward than others, and such hypotheses can suggest adaptive changes in the project’s approach, investment, or management that may improve performance going forward. We will discuss these issues later in the sections on conclusions and recommendations. Some activities described in quarterly reports could easily be mapped to their respective indicators. For example, the FY2016 Q2 quarterly report (p. 4) noted that a Memorandum of Understanding (MOU) was signed with the Municipality of Nacala to enhance its resilience to climate change. That activity contributes directly to Indicator #3, Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance. In some cases, however, after reviewing the quarterly reports we found it difficult to match activities to Objectives, Intermediate Results, and their reported indicators in the M&E Plan. For example, Quarterly Report FY2016 Q4 states that for Indicator #12, Proportion of CCA or DRR interventions implemented with community contributions, 100% of the interventions implemented had a community contribution. With only 50% “community contributions” projected as the Life-of-Project (LOP) target, the cumulative contribution by communities was then calculated as 200%, dramatically exceeding the target. This indicator is intended to measure progress toward Intermediate Objective 2.2, Community organizations’ ability to implement a local set of risk mitigating measures improved. The Indicator Reference Sheet for this indicator given in the M&E Plan states that it measures “CCA or DRR interventions implemented in which local communities provide labor or materials for implementing or sustaining projects.” The quarterly report narrative did not explicitly say which activities were included in the calculation, making it difficult to understand what was being done and being counted. Quarterly and annual reports include a section dedicated to reporting indicator progress, and we observed some inconsistencies in this progress reporting in the existing project documentation. For example, the FY2015 Q4 report states for Indicator #9, Area (hectares) impacted by at least one CCA or DRR intervention implemented with citizen input per year, that 7 hectares were replanted with mangrove seedlings, out of total area of 22 hectares set aside by the municipality for this activity. The following quarter, FY2016 Q1, the area given for Indicator #9 changed from 7 hectares to 1,101 hectares. The report provides no explanation for how this increase was calculated nor does it describe the relationship to the 22 hectares set aside by the municipality. It is unclear if the previously reported 7 hectares were included in the 1,101 hectares. We noted also that probably what is meant by “impacted” in the description of this indicator is “improved,” whereas the word “impacted” has the connotation of “negatively affected.” In addition, the phrase “per year” does not make sense in the description; this indicator an area, not a rate. The CCAP M&E Specialist explained that the initially-reported 7 hectares was measured using a different methodology from the 1,101 hectares. The 7 hectares are the area first planted with mangrove seedlings, while the 1,101 hectares are the result of the geospatial mapping exercise used to produce the vulnerability map for Quelimane, and may encompass the entire area mapped. The Evaluation Team did not review financial reports and budgets of CCAP, which was outside the scope of the evaluation. This limited our understanding of how resources were allocated for the various activities. For example, when we visited Quelimane, the mangrove restoration activities were highly visible, and the emphasis on mangrove restoration that emerged in our interviews with key informants seemed to suggest that it was a relatively large component of the work there. However, we learned from the CCAP office that the mangrove restoration activity in Quelimane accounted for only about MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 15 10% of the project’s budget for activities there, a much smaller share than we would have judged based on the visibility and emphasis on this component on the ground. Our review found that more clarity is needed in reporting activities that are counted by more than one indicator. While this is normal for some indicators to be counted in more than one category, it is unclear from the progress reporting which activities were “double counted.” For example, vulnerability maps were counted as contributing to both Indicators #5 and #9, but quarterly reports do not provide a narrative explaining this double counting. This underscores the need for analysis and explanation of how data are collected for indicator reporting, decide whether to keep or delete comma and how activities are mapped to the objectives and intermediate results in the project’s Results Framework. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 16 4.1.2 RESULTS BY EVALUATION QUESTION As explained in Section 3, Evaluation Methodology and Limitations, seven evaluation questions formed the organizing structure for the evaluation, and we use that structure below in reporting results from both our review of existing and secondary information, and primary information gathering from interviews with key informants, meetings, and site visits. Evaluation Question 1a. To what extent has CCAP been successful in assisting Pemba and Quelimane municipalities to incorporate climate change adaptation into their planning processes? Review and analysis of existing, secondary information This evaluation question is most relevant to evaluating progress towards the Results Framework’s Objective 1: Improve the provision of climate-resilient urban services by municipalities, its two component IRs, and four current indicators (see Table 4.1, CCAP Indicators 1-4). As shown in Table 4.1 above, progress toward the targets for these four relevant indicators appears mixed. For CCAP Indicator 1, “Numerical score on UNISDR's Local Government Self-Assessment Tool (LGSAT),” no target was set; for CCAP Indicators 2 and 3 are at a low percentage of LOP targets proposed by CCAP itself; and for CCAP Indicator 4 the LOP target has already been surpassed Interviews with key informants We asked Evaluation Question 1a directly to our key informants, and 13 felt comfortable answering it because, as municipality or CCAP staff, they felt they had a good understanding CCAP’s assistance to these municipalities obtained the results shown in Table 4.2. The 13 listed in the quantitative scoring table, are the only ones interviewed who felt comfortable answering the question, and felt they had a good overview of all the CCAP assistance towards the Municipality. Meaning, that it mainly includes the CCAP staff and the Municipality staff. TABLE 4.2. OPINIONS ABOUT EVALUATION QUESTION 1A Answers Quelimane Pemba CCAP field staff Extremely successful 1 1 Very successful 1 1 Moderately successful 3 1 4 Somewhat successful 1 Not very successful One of CCAP’s objectives involves supporting municipalities with a variety of tools for incorporating climate adaptation in their planning processes. The project has involved various stakeholders developing and using these tools. One CCAP field staff member saw the biggest successes as the fact that “The municipality didn’t have instruments [tools, methods], now they do. They didn’t speak about climate change, now it is part of the discussion.” In our interviews with all 47 key informants in Mozambique, we asked what they saw as the biggest successes of CCAP so far, and they typically mentioned the planning tools. Content analysis of the interview notes led to the list below, in which the “tools” are shown in order of the number of times (given in parentheses) they were mentioned by key informants. We summarize some of the views about each tool below: 1) PLAs – Local Adaptation Plans (7) 2) Support for SIGIU – the Integrated Urban Information Management System (6) 3) Climate Change Adaptation and Resilience short course (4) MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 17 4) Support for SIGIC – the Integrated Disaster Information Management System (4) 5) Support to Local Disaster Response Committees (4) 6) Design of resilient houses (4) 7) Vulnerability Maps, especially as linked to the land cadaster system of the municipality (3) Local Adaptation Plans were the planning “tool” most often mentioned as a success. The PLAs were developed using a new approach that was adapted to cities rather than rural areas, which was based on experiences from Durban, South Africa, a city that has implemented urban climate change adaptation measures CCAP supported the adaptation of the Durban methodology for developing PLAs so it fit with Mozambican district-level planning processes. According to our key informants, a strength of the PLA is that it is developed with community participation and involvement of various actors and stakeholders, and identifies risk areas and needs in the neighborhood. PLAs have been developed and adopted by both Pemba and Quelimane Municipalities and incorporated into their planning. One challenge for municipalities is that because developing a PLA is a participatory process, it is very time-consuming. Good facilitators with knowledge of local languages are needed. Another challenge is that to be sustainable, the municipality has to take ownership of the process and of the PLA that is developed. Finally, municipalities need to raise the funds to implement the prioritized adaptation measures. The Integrated Urban Information Management System, SIGIU, was often noted by municipal staff as a useful and successful system. One key informant from UEM expressed the view that SIGIU is a “Useful information gathering tool, but it is not a climate change tool. No doubt it is important, but the SIGIU has to be adapted to collect information on climate change. Universities have to be involved to analyze the data on climate, because municipalities don’t have the capacity.” This informant suggested that information on extreme weather events (e.g., rainfall, wind speed, tide levels, storm surges) and their effects on urban populations and infrastructure could be collected through the SIGIU. The five-day short course on “Climate Change Adaptation and Resilience” that was developed by UEM faculty members with CCAP support was often mentioned. Many key informants told us that it succeeded in raising the level of awareness and knowledge about climate change among municipal staff and assembly members, local university staff, and NGOs in Pemba, Quelimane, and Nacala. A needs assessment informed the development of the course, and UEM developed different versions or levels of the course for different target audiences. The course included field visits with a practical component. Course materials will be available for free online. The climate vulnerability maps that were developed with CCAP support were another commonly￾mentioned success. They were especially useful, some of our key informants said, when linked to, and overlain on, the land cadaster system of the municipality. In Pemba, the municipality is not allocating land in Paquitequete, the most vulnerable of the four vulnerability zones. The municipality developed a waiver for building permits that must be signed before building anything, showing that the permittee understands the risks in that zone, the types of construction that are permitted, and the building standards that should be followed in any construction in that area. Some municipal staff stated that there was a need to effectively monitor the construction being built and check if people are following the rules and standards. In general, they said, a municipal zoning approach should be applied and enforced, and no building permits granted whether because of a lack of capacity to provide basic services or because of high vulnerability to extreme climate events and climate change. Municipal staff in Pemba said that the use of this tool is sustainable, and will continue after CCAP ends. Quelimane is also using the vulnerability maps linked to the land cadaster in issuing land-use and building permits, according to one key informant. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 18 There are big differences in the success and challenges in the two municipalities, according to CCAP field staff and others. In Pemba, where the municipal government is of the same political party as the district government, the municipality opens all the doors, and it’s easy to work. In Quelimane, different political parties control the district government and the municipality, creating communication problems and roadblocks to implementation. In Pemba, for example, key informants from INGC and DEPTADER told us that coordination between the municipality and these national agencies was successful. A representative of INGC said “CCAP is a project more linked with the municipality, however they integrated INGC and MITADER. CCAP is doing a hard job for us, taking information to the communities. The project has equipped three Local Disaster Response Committees in the city and provided refresher training to five committees. The training was developed by INGC, the municipality, and CCAP.” In Quelimane, key informants from these same national institutions mentioned lack of engagement of CCAP with them. DEPTADER would like to be more involved with CCAP, decide whether to keep or delete comma and to have access to reports and documentation from the project. CCAP should link with national climate change initiatives under MITADER, they suggested. INGC representatives said that their relationship with the municipality is not the best, and implied that it may be because of politics, and they also stated that coordination with CCAP is not good. They had been invited to a one-week training on some of the tools CCAP is supporting, but otherwise did not know much about those tools, such as the SIGIC and SIGIU. Evaluation Question 1b. To what extent has CCAP been successful in helping the relevant stakeholders of municipalities to implement adaptation measures? (communities, civil societies, NGOs, and universities)? Review and analysis of existing, secondary information We reviewed quarterly reports and indicator data to determine the extent the CCAP has been successful in assisting stakeholders implement adaptation measures. This evaluation question is most relevant to evaluating progress towards the Results Framework’s Objective 2: Increase adoption of climate resilience measures by communities, civic and community organizations including civil society, NGOs and faith￾based organizations, its three component IRs, and four indicators (see Table 4.1, CCAP Indicators 10-13). As shown in Table 4.1 above, we again see mixed progress toward the targets for the four indicators: two have already exceeded the LOP targets, one has almost reached the target, and one is about halfway to the LOP target. Interviews with key informants In general, our key informants expressed the view that CCAP has been successful in contacting and engaging stakeholders since the beginning of project implementation. Many contacts have been made and expectations created among the various stakeholders, they said, and CCAP has involved communities and community-based associations, NGOs, and universities in various activities. Engaging communities, civil society organizations, and universities in municipal planning processes is a first and necessary step toward implementing adaptation measures–which this evaluation question asks about–but planning is not by itself implementation, of course. Some examples of how CCAP has helped municipal stakeholders implement adaptation measures include:  Refresher training for Local Disaster Response Committees. These committees were already formed, but CCAP and INGC provided training to refresh their knowledge, and distributed emergency kits.  Resilient housing designs: The model comes from the community, and local artisans have been involved and trained to construct using techniques that are more resistant to extreme climate events. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 19  In Quelimane, CCAP worked with ANAMA, the Associação dos Naturais e Amigos de Madal, a local association, was involved in mangrove restoration; and AJAQ, the Associação dos Jovens Amigos de Quelimane, to clean drains along the main street with new equipment bought by CCAP.  Design of latrines: CCAP supported the design of an “improved latrine,” and the design is now being used in another project that is constructing 600 latrines. Evaluation Question 2a. To what extent has CCAP increased climate resilience of the most vulnerable populations of Pemba and Quelimane municipalities, including those living in the most vulnerable areas? Review and analysis of existing, secondary information This evaluation question is directly relevant to evaluating progress towards the CCAP goal, Climate Resilience in Selected Mozambican Coastal Cities Increased. The M&E Plan has four indicators at the goal level (see Table 4.1, CCAP Indicators 1-4). According to the CCAP M&E Plan, Indicator #1, Numerical score on UNISDR's Local Government Self￾Assessment Tool (LGSAT), is the only impact-level indicator for the project. The Evaluation Team reviewed all documents related to the LGSAT provided by USAID/Mozambique. The M&E Plan says that the baseline score for this indicator at the beginning of the project will be set by holding an exercise score the LGSAT Tool with each municipality, and that “Once baseline scores are set, [the project] should also agree on [the] target with each municipality.” The baseline assessments were conducted in November 2014 in Pemba and Quelimane, and in June 2016 in Nacala. The baseline score for Pemba was 1.81/5, and the baseline for Quelimane was 1.91/5 (USAID-CCAP, 2016, LGSAT Summary, Aug. 2016). These baseline numbers were not updated in the M&E Plan, which was revised in February 2015, and no Indicator #1 targets were proposed. The LGSAT Tool exercise was re-done in each municipality in June 2016; Pemba’s score increased slightly to 2.04, and Quelimane’s to 1.97. It does not appear to the Evaluation Team that a statistical analysis of the scores using this tool would be possible, but the issue is not discussed in project reporting. The changes in Indicator #1 scores are very small in any case, Suggestions to improve support for implementation of climate adaptation measures by stakeholders:  Strengthen the climate change awareness campaign, it is not very far-reaching, and not in local languages.  In Icidua, Quelimane, the community was most concerned about learning farming techniques, making stronger blocks for house construction, learning to build better latrine, and improving their domestic water supply.  In Quelimane, many institutions, such as INGC and DEPTADER, mentioned lack of engagement of CCAP with them.  In Carioca, Pemba, the neighborhood was most concerned to develop a behavior change campaign for solid waste management, and to learn about standards for stronger house construction. More community involvement is needed.  In Paquetiquete, Pemba, an urgent need is to stop coastal erosion that leads to flooding during storm surges. Families that built their houses in the drainage channel need to relocate, and the channel needs to be dredged and deepened.  The municipalities and INGC should conduct simulations of an emergency situation, where the disaster relief plans are tested.  Areas for boats and canoes to dock should be identified and constructed, because unregulated docking is damaging and eroding shorelines.  Coasts should be protected in some areas with gabions or other structures. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 20 and those small changes hover around scores in the range of 2.0 out of a possible 5.0, indicating that “Achievements have been made but are incomplete, and while improvements are planned, the commitment and capacities are limited.” Therefore, the small changes in this indicator do not provide convincing evidence that the project had much of an impact on climate resilience in Pemba and Quelimanebetween November 2014 and June 2016. Our review showed that the project has already exceeded its target for Indicator #4, Number of institutions with improved capacity to assess/address climate change risks issues as a result of USG assistance, with 23 institutions with improved capacity out of a target of 20 institutions (130% of target) reported at the end of FY2016. However, Indicator #2, Number of stakeholders with increased capacity to adapt to the impacts of climate variability and change as a result of USG assistance, is probably the most directly relevant indicator for climate resilience of the “most vulnerable populations” in the “most vulnerable areas,” and for that indicator the project seems to be far from on pace to reach the annual and LOP targets proposed by CCAP for this indicator. Annex A of the 2015 M&E Plan projected the following schedule for achieving annual targets: FY2014 = 0; FY2015 = 1000; FY2016 = 2000; FY2017 = 1500; FY2018 = 550; and as shown in Table 4.1 above, the cumulative LOP target was raised from 5,050 to 6,000 persons in the 2016 Modification (USAID/Mozambique, 2016). By the end of FY2016, however, cumulative numbers stood at 980 persons, about a third of the original target projected at this stage of the project, and an even smaller percentage given the increased LOP target from the 2016 Modification. The slow pace of reaching projected targets may be the result of issues with the original project design and its assumptions, or due to challenges that have arisen since implementation began. Documentation and analysis of why these targets are off pace may be useful for making adaptive adjustments in the project at this stage. Likewise for Indicator #3, Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance, the project now counts 34 such agreements, out of an LOP target of 100 – better, but still off pace to achieve the goal. The “theory of change” underlying the Results Framework for the project argues that if the capacity of relevant institutions is improved – municipalities, universities, and NGOs primarily – then that capacity will eventually enable, or translate to, on-the-ground actions that will reduce climate vulnerability and increase resilience. Indicator #4 at the Project Goal level suggests that institutions are being engaged and their capacity improved. Objective 1 and its two IRs aim to build municipal capacity for climate adaptation, and M&E data provide some evidence of progress, with targets for two indicators already surpassed (see Table 4.1, CCAP Indicators 5 and 7), and two behind schedule (CCAP Indicators 8 and 9). This evaluation question is also directly relevant for evaluating progress toward CCAP’s Objective 2, Increase adoption of climate resilience measures by communities, civic and community organizations including civil society, NGOs and faith-based organizations. According to the CCAP Deputy Chief of Party, an integral part of the project’s approach to increasing the adoption and implementation of climate adaptation measures is a social and behavior change communications strategy. The first mention of a formal behavior change strategy is found in the M&E Plan (USAID-CCAP, 2015, p. 4), and it is described as an approach to achieve Intermediate Result 2.1: “We will integrate behavior change communication strategies into CCAP activities to reach these target groups and we will demonstrate increased knowledge through pre- and post-intervention testing (e.g. through surveys, focus groups, written tests, etc.” Quarterly Report FY2015 Q4 and the FY2015 Annual Report provide the first mention of the phrase “social and behavior change communications,” and gave it the abbreviation “SBCC.” The FY2015 Annual Report stated that implementation of the SBCC strategy had been delayed, but a Social and Behavior Change Communications Strategy report was released in December 2015 (FY2016 Q1) and revised in July 2016 (FY2016 Q4). This phrase and abbreviation were not found in any of the four quarterly reports of FY2016, suggesting a further delay in MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 21 implementation. An SBCC “Action Plan” was finalized by CCAP in September 2016, after this evaluation had started. According to that report, the Social and Behavior Change Communications strategy aims to support CCAP Objective 2 via Indicator #10, Number of people with increased knowledge of climate change impacts and adaptation strategies, with a life-of-project target of 500 people, and measure progress toward Intermediate Result 2.1 through Indicator #11, Number of person-contact hours of information disseminated about climate change vulnerabilities and adaptive options, with a target of 3 million person-contact hours. Table 4.1 above shows that the target for Indicator #10 (Objective 2) has almost been reached and for Indicator #11 (Intermediate Result 2.1) is about half achieved. According to the report, the SBCC strategy “will use a newly-developed Climate-Smart Cities branded social and behavior change communications (SBCC) platform, based on the Socio-Ecological Model for Change.” No citation is given for the “Climate-Smart Cities” platform, although our research shows the phrase to be a trademark of the U.S. NGO Trust for Public Land (Climate-Smart CitiesTM ). The Socio￾Ecological Model for Change is a framework adapted from a 2000 book (McKee, et al., 2000) by the USAID-funded C-Change Project, implemented from 2007-2012 by FHI 360. The SBCC strategy was developed by in-house consultant in close collaboration with CCAP Team in Maputo and in the sub-offices in Pemba and Quelimane. Household data from a FY2015 rapid assessment conducted in Icidua, Quelimane, and Paquitequete, Pemba, informed the design of the CCAP SBCC strategy. The institutional, gender, and youth assessments carried out in the beginning of the project found out that the awareness and knowledge of climate change and resilience among the residents in both Pemba and Quelimane were very limited and the topics were new for most of the people. Those assessments also found that some existing environmental practices were increasing climate vulnerability, such as cutting mangroves and mismanagement of existing protection infrastructure such as drainage channels. These findings motivated the development of the SBCC strategy, with the aim of increasing citizen awareness and knowledge of climate change vulnerability and motivating behavior change to improve resilience. The strategy proposes to affect six types of behaviors involving household infrastructure (house construction, latrines, and water-catchment systems), sanitation (disposal of solid waste), and, in Quelimane, protecting and realizing economic benefits from mangrove restoration (called in the report “green infrastructure”). The strategy proposes to affect behaviors involving household infrastructure (house construction, latrines, and water-catchment systems), sanitation (disposal of solid waste), and, in Quelimane, realizing economic benefits from mangrove restoration. The analysis of “barriers” to these desired climate-resilient behaviors, presented in Table 3 of the report, identifies several categories of factors that can motivate or impede behavior change, including lack of awareness and knowledge, options, skills, and social norms, as well as economic factors like cost, and lack of laws and regulations, and the enforcement thereof. Interviews with key informants This evaluation question is closely linked with the previous one, Evaluation Question1b, because the causal logic of the project design assumes that if stakeholders in municipalities implement adaptation measures, then the resilience to climate variability and change of even the most vulnerable populations in those municipalities will be increased. According to this logic, some of the successful adaptation measures discussed above, such as the refresher course for Local Disaster Response Committees and provision of emergency response kits, have increased the resilience of residents of the most vulnerable areas. Two main activities, however, are seen by our key informants as the most significant for increasing climate resilience of vulnerable populations: mangrove restoration and climate resilient housing. Mangrove restoration organized by CCAP began in Quelimane in April 2015, decide whether to delete or retain comma and so has been in progress for about one and a half years. Approximately 13 hectares of the intertidal zone near the neighborhoods of Icidua and Mirazane have been planted with seedlings of MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 22 the most common species in the zone, Avicennia marina. It is estimated that approximately 5,700 hectares of mangroves exist in the vicinity of Quelimane (Bandeira, 2016, personal communication). Google Earth views and measurements of this area show that approximately one-half of the original mangroves have been cut, cleared, or degraded – an area of at least 2,500 hectares. Given these facts, it is difficult to imagine that the restoration of 13 hectares of mangroves near these extremely vulnerable communities will make a significant contribution to restoring the loss of ecosystem benefits from the degradation of an area around 200 times as large. However, a hydrological monitoring program designed by the U.S. Forest Service International Programs and implemented by the UEM School of Marine and Coastal Sciences could eventually provide evidence to evaluate the effectiveness of restoring even such a small area of mangroves, such as in protecting local wells from salinization. This possibility reinforces the importance of the hydrological monitoring program, but there are indications that this monitoring has encountered problems (C. Trettin, 2016, personal communication). More details about CCAP’s mangrove restoration efforts are provided in Annex G. The design and promotion of houses that are more resilient to cyclones and other extreme climate events is now underway. Designs were developed with input from community members and builders, and ground-breaking for construction of model houses took place in Pemba and Quelimane last month. The rate of adoption of these climate resilient designs will of course depend mainly on economic factors, but if a significant proportion of the houses in the most vulnerable neighborhoods of the two municipalities were eventually constructed according to resilience standards, the overall climate resilience in these most vulnerable areas would increase. Other CCAP activities, such as cleaning drainage ditches in Pemba, may in the short term have reduced the risk from flooding, but even this short-term benefit requires that the municipality and communities organize periodic re-cleaning, and institute mechanisms for managing solid wastes that clog drainage systems if not properly disposed of. In areas such as Paquitequete in Pemba, and Icidua and Mirazane in Quelimane, which are extremely vulnerable because of their exposure to climate change-related risks such as sea level rise, short term actions will probably not increase long-term climate resilience very much. Evaluation Question 2b. To what extent has CCAP incorporated gender considerations and youth into implementation of its activities? Review and analysis of existing, secondary information To determine the extent the CCAP has incorporated gender and youth into activities, we reviewed existing M&E indicator data from quarterly reports, as well as other relevant existing documents. CCAP conducted a Gender and Youth Stakeholder Analysis early in the project (USAID-CCAP 2014), and from it developed a set of recommendations. The analysis assesses differences between men, women, and youth in terms of political, social, economic, and cultural factors that influence their respective access to services and land rights, and the ability of the project to engage them in project activities. Annual work plans call for most activities undertaken by CCAP to include both men and women, and young people as well as adults. Gender and youth are considered a cross-cutting measure supporting Objective 2. Indicator #13 under IR 2.4 measures the “proportion of individuals engaged in CCAP activities who are youth.” Gender is not mentioned in this indicator. Quarterly reports up to FY2016 Q4 show that 28% of project activities have engaged young people, exceeding the life-of-project target of 20% (see Table 4.1). CCAP’s Second Annual Work Plan, dated April 2015 (USAID CCAP, 2015), stated that “CCAP updated the results framework since the version presented in the first work plan, primarily by streamlining Objective 1 and creating a specific IR for gender and youth.” However, Indicator #13, worded such that it considers only youth, not gender, was first being tracked in the second 2015 quarterly report (USAID MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 23 CCAP, 2015), and no revisions of this indicator, or additions of a new indicator tracking gender￾disaggregated engagement, appeared in the quarterly reports. The Second Annual Work Plan stated also that “we think that there is a need for specific interventions (e.g., capacity building, training, outreach) that have gender and youth at the core.” However, searches for keywords “women” or “gender” in all quarterly reports from FY2015 and FY2016 find very few mentions of specific activities with a gender dimension; the main one mentioned is the participation of women’s groups in meetings about more resilient housing. Based on our review of existing reports, the CCAP M&E system does not appear to be tracking any project activities or engagement disaggregated by gender, so we were unable to provide a clear answer to the gender aspect of Evaluation Question 2b from existing, secondary information. Interviews with key informants CCAP staff told us that the project has first sought to create an enabling environment for gender mainstreaming by providing information to both men and women equally. They also stated that they believe project activities to date have been “gender neutral,” and have not increased gender inequality. CCAP field staff felt that CCAP incorporated gender considerations in implementing its activities very well. They stated that they encourage women to participate in all their activities. When asked to list three CCAP activities that effectively incorporated gender considerations in implementation, they responded that in Quelimane, mangrove planting effectively involved women and youth, and in Pemba, activities in Xtique – a cleaning campaign, creating gardens, and participation in the awareness campaign – also did this. CCAP field staff told the evaluation team that men sometimes do not respect women’s viewpoints in group meetings. For example, when they held meetings about resilient housing with women and me together, women said they had experience repairing houses after flood and storm damage, and knew what improvements in design and construction were needed, but some men said that was stupid. CCAP staff then decided to hold separate meetings with women and men. Although this may be a way of initially getting unfiltered opinions, in the long term perspective gender mainstreaming would favor meetings with both men and women, and a facilitation process to understand and resolve the differences in perspectives. It could be, for example, that men are responsible for getting the materials for house construction and doing the construction, and that certain design ideas favored by women would be difficult to construct. In general, most of the simple adaptations measures used in households, such as placing sandbags to block water from coming into the house during flooding, are done by women. CCAP field staff felt that the project has incorporated young people in the implementation of activities moderately well. One stated that “We have to involve more young people in our activities.” One approach to do so, they explained, was through signing MOUs with local universities about enabling students to help the municipalities in various kinds of CCAP-related activities. CCAP field staff gave the following examples of how they had worked with youth:  In Pemba, they work with an existing group of young people from Paquitequete, in an awareness campaign, cleaning drainage ditches, solid waste management, gardening at schools, and other activities.  In Quelimane students from Edwardo Mondlane University School of Marine and Coastal Sciences are involved in mangrove monitoring activities.  In both municipalities university students were used to collect data for the Social and Economic Assessments used in developing the SIGIU. In August 2016 USAID/Mozambique released a USAID Global Development Alliance Annual Program Statement (APS) to support community level engagement and activities that should focus on women and youth as possible audiences. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 24 Evaluation Question 3a. To what extent is CCAP prepared to achieve the project’s objectives over the next 2.5 years? Review and analysis of existing, secondary information Our review of existing information suggests that CCAP will only partially achieve its objectives during the remaining life of the project. As noted above, CCAP has exceeded or nearly reached targets (proposed in the schedule given in Annex A of its 2015 M&E Plan) for six of its 12 indicators by the end of FY2016 and is behind its proposed schedule on the other half. The project seems to be quite strong in developing or providing municipalities with tools for planning for and responding to climate variability and change and in training people to use them. Based on the M&E data, it also seems to be successful in engaging communities, including young people, in certain activities. The vulnerability maps are a fundamental tool for incorporating climate change into municipal planning and decision-making processes. Those maps led to counting over 1,000 hectares being positively affected CCAP interventions under CCAP Indicator #9 (see Table 4.1). Although the maps can guide and influence the location of the result (IR 1.2, Application of management, soft engineering, and hard engineering climate adaptation measures by municipal authorities through effective citizen engagement increased) that CCAP Indicator #9 is supposed to measure , the maps by themselves do not represent the “application” of these measures, so counting them for this indicator seems to overstate their value to the desired result. According to the M&E Plan, to be counted as part of an area attributed to Indicator #9, actions must be “implemented interventions.” Interviews with key informants CCAP field staff judged that the project is prepared and on target, to a significant degree, to achieve the project’s objectives over the next 2.5 years. They estimated that about 70% of targets will be achieved. This opinion points to a strong internal feeling of confidence and success within the project itself. At the same time, project staff noted to many challenges and obstacles (see Evaluation Question 3b), but their opinion seemed to be that these challenges were being, or could be, overcome in the next 2.5 years to a large extent. Evaluation Question 3b. What are some challenges/ obstacles (related to staffing, finances, etc.), reported by project personnel and what is the project’s capacity to respond to those challenges? Review and analysis of existing, secondary information Quarterly reports and M&E data were reviewed to determine challenges and obstacles related to CCAP staffing, finances, etc., and the capacity of staff to respond to the challenges. Challenges reported in quarterly reports led to significant delays in a number of project activities. The within-project challenges have been one cause of the mixed progress toward indicator targets shown in Table 4.1 above. These challenges seem to be typical in the Mozambican context – for example, challenges of recruiting skilled staff, negotiating subcontracts, and managing grants under contract. Interviews with key informants USAID/Mozambique staff managing the CCAP project told the Evaluation Team that the modification that was just finalized in August was, in part at least, in response to concerns about the design of the project from the USAID side, and that some of USAID’s assumptions at the time of the original design were overly ambitious. The original CCAP design was based on a “very aggressive” application of USAID Forward principles, including working with local institutions and building their capacity, and leveraging private resources for climate change adaptation. The CCAP design assumed that there was greater local MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 25 capacity and interest than proved to be the case, so the project has involved a lot of “adaptive learning,” according to these USAID staff. Since the project started, the Chief of Party has also changed, and changing leadership always is both an opportunity and a challenge – although this change was part of the project design from the beginning. CCAP field staff listed the three biggest challenges/obstacles to achieving the project’s objectives as:  It is difficult for the municipalities to understand that CCAP could not provide operational support, but only support the development and training in new “tools” that might be useful to the municipality as a way of getting their “buy in” on the CCAP project;  Knowledge about climate change among the municipal staff is at a low level, and educational levels among those staff are low in general; and  Demand for and pressure on the municipality to provide basic services is stronger and a higher priority among citizens than to get ready for climate change or even to prepare for natural disasters. One CCAP staff member said “a big limitation [is], there are no basic services, so it is hard to convince communities to adopt climate resilience measures.” CCAP staff told us that coordination with the municipality has been a challenge. Addressing the differences between Pemba and Quelimane in government engagement and implementation of CCAP activities due to the party politics of those two municipalities has been especially challenging. Evaluation Question 3c. What are some major implementation obstacles/challenges and opportunities (reported by the municipalities and other stakeholders) anticipated over the next 2.5 years of implementation? Review and analysis of existing, secondary information Documents and other existing information relevant to answering this question included news stories about Mozambique’s current political and economic situation, climate-related news, donor reports, and NGO and scientific reports. Challenges include:  Economic situation: Mozambique is currently entering a nationwide economic depression, which could spiral into an unstable situation. Inflation is rising sharply, commodity prices and international trade are trending unfavorably, and the metical has depreciated by nearly 50 percent since the beginning of 2016 (World Bank, 2016) Interested multilateral development banks, led by the IMF and the World Bank, are coordinating with the Government of Mozambique to develop an approach to stabilize the economy (IMF, 2016). The approach will result in significant policy changes, which could affect the structure and priorities of the ministries, and in turn affect budgets and priorities in the provinces, districts, municipalities, and communities. These may affect municipal allocations and environmental policies, including those related to adaptation to climate change.  Regional drought: The recent drought, associated with an El Niño event, have accelerated rural-to-urban migration, including to coastal cities like Pemba and Quelimane, where migrants aspire to better living conditions and employment opportunities.  La Niña projected to increase flooding and affect food security: A new report Food Security Outlook: Mozambique 2016-2017 (FEWS NET, 2016) projects a widening food security crisis and increased stresses to prices, livestock, and food stocks across Mozambique beginning with the 2016-2017 rainy season. A strong La Niña event is forecast, which generally brings above-average rainfall to Southern Africa. Cyclones also are more likely during La Niña periods.  Renewed rural warfare: A power struggle between RENAMO and FRELIMO that has led to armed attacks in certain provinces. Both parties agreed to meet to negotiate a peaceful MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 26 resolution of the situation, but the negotiations are taking longer than expected. There is increasing insecurity in some provinces and armed attacks are regularly reported by the media. Combined with drought, this insecurity has led to increased rural to urban migration, putting additional pressure on municipalities, including Pemba and Quelimane. Interviews with key informants We asked key informants from municipalities to list what they thought were the “three biggest challenges facing this city in preparing for climate change.” Content analysis of our interview notes led to this list of proposed challenges: 1) Coordination between CCAP and institutional partners such as MITADER, DEPTADER. INGC, and universities; 2) Capacity of municipalities to use planning and adaptation tools; 3) Capacity of municipalities to sustain activities after end of CCAP; 4) Lack of flexibility of CCAP to adapt to the municipalities’ plans; and 5) Limited computers and software. One indication that coordination between CCAP and its institutional partners could be improved comes from our interview with MITADER staff working on climate change in Maputo. They told us “We have little information about the CCAP project. We were invited to Quelimane and Pemba in the beginning of the project, where some information regarding the project was also shared. Our perception is that CCAP is a community based project aimed to mangrove restoration and raising awareness about community adaptation measures.” A DEPTADER staff member said “Because we are responsible for environmental management, DEPTADER should be more involved in the CCAP. We’ve even designed a proposal to partner with CCAP but we haven’t gotten a response. We would also benefit from the reports and documentation from the project.” An INGC staff member in the field held the opinion that changes in CCAP staff had led to a breakdown in efficient communication with the project. The evaluation team is aware that CCAP, whose primary institutional partners are municipalities, needs to carefully balance its relationship with national ministries and agencies, given Mozambique’s centralized political system. Although the municipalities are quasi-independent institutions, they depend on GOM funds from the Ministry of Finance, and legally are coordinated by the Ministry of State Administration. Climate change funds and coordination are the responsibility of MITADER’s, Department of Climate Change. The municipalities have formed an “Associacao dos Municipios de Moçambique” to promote their common interests at the national level, and universities, with their research and consulting capabilities, are important actors. All of these institutions have a role to play in sharing experiences and promoting tools for climate change adaptation. Specific challenges or obstacles mentioned by municipal staff or assemblymen in both Pemba and Quelimane include:  Lack of computers, software, internet connections, and lack of municipal funds to pay for these, and lack of staff with the technical knowledge to maintain them.  Lack of capacity to use and maintain computer and internet-based tools among municipal technical staff.  Lack of information from CCAP about costs of activities and equipment; key informants said they doubt they will be able to financially sustain the tools and activities supported by CCAP, but don’t really know because they don’t know the costs. Pemba is already looking for new partners to finance the continuation of some tools and activities started under CCAP. This is a positive indication that they find the tools and activities to be valuable, but suggests that they are MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 27 not yet planning to incorporate them directly in municipal operations and fund them through the municipal budget. Many key informants suggested that closer relationships between the municipalities and local universities could help address some of the challenges, such as:  Create a course focused on some of the tools at a local university, or integrate them in an existing course;  Place interns from local universities in the municipalities to address some urgent issues or problems; and  Develop a formal relationship between municipalities and local universities, such as through an MOU, to analyze data, provide technical advice, etcetera. Staff at Eduardo Mondlane University in Maputo, who developed the short course on “Climate Change Adaptation and Resilience,” gave the opinion that “The use of the tools are not sustainable without involving universities, because municipal technicians don’t have the capacity to analyze the data they are collecting. Local universities have faculty and students who can collaborate with the municipality to understand the data and create informed decisions.” This may be a partially self-serving view, but it may also have some merit. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 28 4.2 Conclusions In this section we have interpreted and summarized the results presented in Section 4.1 above. Six main conclusions emerge: 1) CCAP has been generally successful in assisting municipalities to incorporate climate variability and change into their planning processes. The Evaluation Team found that CCAP has been generally successful in assisting Pemba and Quelimane to incorporate several excellent tools for adaptation to climate variability and change into municipal planning and emergency response processes. This conclusion is supported by evidence both from our review of existing information and our primary information gathering through interviews with key informants and meetings with communities in those cities. However, it is too soon to tell whether the project’s results have functionally and measurably improved long-term climate resilience in those municipalities. CCAP’s success in developing adaptation tools is also indicated by requests from other municipalities to participate in project activities, reflected, for example, in the MOU signed with Nacala, and in the inclusion of municipal staff from Mocimboa do Praia in trainings. 2) Better coordination and collaboration between municipalities and national-level institutions would strengthen some aspects of the project. The main tools so far developed under CCAP will have a bigger impact if they are scalable. In testing tools for incorporating climate change adaptation into planning processes in Pemba, Quelimane, and Nacala, the relevant national institutions (e.g., MITADER, DEPTADER, INGC, Ministry of Finance, Ministry of State Administration) need to be involved so they can endorse and promote them in other cities and/or districts. This is especially important for the Climate Change Adaptation and Resilience short course, vulnerability mapping, and the PLAs. So far it is not clear which institution(s) will eventually take over those tools and be responsible for scaling them to a national scale. Besides consolidating its successes in Pemba and Quelimane, in its final years CCAP should develop a strategy for handoff or “appropriation” of the key tools to an institution or institutions that can and will take responsibility for replicating them on a national scale. 3) The CCAP M&E system needs to be tightened and strengthened. In general, the Evaluation Team found that CCAP was tracking the significant progress it is making in its quarterly reports. In some cases, however, we found it difficult to track and map activities to annual work plans and to M&E Plan indicators. For example, the FY2016 Q2 quarterly report lists several activities, including signing of an MOU in Nacala, an environmental compliance training in municipalities, a resilient housing event, a local adaptation plan submitted for approval, an assessment of green infrastructure, and an assessment of SIGIU. The descriptions of those activities lack a clear and explicit connection to their respective Intermediate Results and Objectives in the project Results Framework. The activity descriptions also do not indicate which indicators they support. 4) The mangrove restoration activity in Quelimane has serious design flaws that should be corrected. CCAP apparently followed the recommendation, made in the last USAID Mozambique Environmental Threats and Opportunities Assessment (USAID, 2013), to support mangrove restoration where appropriate. The project conducted a rapid assessment of mangrove areas in Quelimane in October 2014 (Garrido and Carimo, 2014). In line with science-based best practices for mangrove regeneration that have been developed from worldwide experience (Lewis, 2009), the initial assessment recommended removal of dykes and restoration of natural tidal flows to allow natural regeneration of mangroves. That recommendation was not followed by CCAP, however, and planting of mangrove seedlings began, using an untested method of digging channels, or furrows, in the mud, which further disrupted the natural hydrology of the area. A mangrove hydrological monitoring system was designed MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 29 and installed with the help of experts from the US Forest Service in 2015, funded by USAID. It appears that the monitoring program is having problems, and its status is unclear (Trettin, 2016; personal communication). A recent assessment by mangrove scientists from Eduardo Mondlane University (Bandeira and Macamo, 2016) suggest need for changes in mangrove restoration approaches and techniques. CCAP is in the process of awarding grants to local NGOs in Quelimane to carry out some of the recommendations from this assessment We provide a more detailed evaluation of the mangrove restoration component of the project in Annex G: Lessons Learned in CCAP Mangrove Restoration. 5) The proposed Social and Behavior Change Communications strategy is complex and unlikely to produce the expected results as now conceived. We found that the CCAP Social and Behavior Change Communications Strategy (USAID CCAP, 2016) is very complicated, technical, and difficult to understand. We think it will be difficult to use it as a practical guide for designing and implementing effective behavior-change activities and interventions. The “Socio-Ecological Model for Change” (McKee, et al., 2000) upon which it is based is complex, abstract, and academic, and it is difficult to see how a project manager or on-the-ground practitioner could easily adapt it to find actionable guidance. CCAP finalized an action plan for implementing the SBCC strategy while this evaluation was underway, which the evaluation team has now reviewed. We found that it reinforces the impression that the SBCC strategy is focusing almost exclusively on trying to reduce “awareness and knowledge” barriers to the adoption of desired behaviors through a variety of communications activities, when in fact the main barriers to the desired behaviors do not seem to be awareness and knowledge factors, but things like economic costs and lack of enforcement of existing regulations. 6) The relative emphasis placed on short-term climate disaster risk reduction (DRR) versus long-term climate change adaptation (CCA), and both of these relative to basic development interventions, would benefit from further analysis and thought to make the balance more explicit and clear. The relative balance of effort and investment in DRR and CCA in the project is not completely clear and explicit, and may not be deliberate. The nexus of development-DRR-CCA objectives provides some major philosophical and practical challenges to a project like CCAP. Our evaluation results lead us to questions such as:  How can the project balance disaster risk reduction (DRR) and climate change adaptation (CCA)?  How can the project build resilience through DRR and CCA when basic development needs are so great in the most vulnerable communities? The Intergovernmental Panel on Climate Change (IPCC) defines vulnerability to climate change as a function of exposure, sensitivity, and adaptive capacity. Vulnerability to the normal risks from extreme climate events could be seen as a similar function. DRR measures aim to reduce risks from weather￾related emergencies. CCA measures aim to reduce vulnerability and increase resilience to climate change in the long term. Both DRR and CCA share fundamental elements, and in most cases DRR and CCA measures are mutually beneficial, but there are counterexamples, and DRR measures need to be evaluated for their long-term consequences for CCA. CCAP used the IPCC definition of vulnerability as the framework for vulnerability mapping, stating that “The vulnerability map was calculated as a function of Exposure, Sensitivity and Adaptation Capacity.” The frequency and intensity of extreme weather events such as tropical cyclones are predicted to increase with climate change, thereby increasing “exposure” to climate-related natural hazards. Adaptation to current and potentially greater future climate exposure requires actions either to reduce sensitivity, or to strengthen adaptive capacity. Some CCAP interventions are designed to reduce sensitivity (e.g., mangrove restoration, construction of more durable houses, zoning to prevent or MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 30 regulate building in most vulnerable zones, cleaning drainage ditches, improving water supply) and others to increase adaptive capacity (e.g., SIGIC, local disaster response committees). In terms of the relationship between climate change adaptation and development, the World Resources Institute (WRI) study “Weathering the Storm: Options for Framing Adaptation and Development” (McGray, et al. 2007) says that “Any effective development and planning process will need to take climate adaptation into account and, conversely, adaptation efforts themselves will often require development interventions to succeed. … While climate impacts are increasingly observed, the debate over managing adaptation has progressed very slowly. This in part is due to confusion about the relationship between adaptation and development—a definitional problem that has hindered not only project design but also the allocation of funding for adaptation efforts.” In the Foreword to that book, WRI President Jonathan Lash wrote: “Unfortunately, the merging of the development and adaptation agendas has a down side. Most existing mechanisms for funding adaptation to climate change have been designed to distinguish carefully between “normal” development activities and the “additional” activities needed to adapt to climate change. Such mechanisms do not fit well with a world that calls for integrated approaches to these problems … The framework of adaptation approaches proposed in this report contributes to breaking down the “either/or” thinking that has constrained adaptation funding, by providing a practical alternative for thinking about when and how adaptation and development intersect.” Both USAID/Mozambique and CCAP project staff seem to recognize the tension between development and climate change adaptation, and the significant challenges it reflects. A senior staff member at USAID/Mozambique told the Evaluation Team that it has been a challenge for them to “find the sweet spot where you can help people a little bit [in meeting basic needs], but which also helps with [climate] adaptation.” Another of the Mission’s senior staff said “It is impossible to address the needs of the poorest [Mozambicans in a project like this]. The set of CCAP activities is well-targeted. To think that we are going to make a huge difference in the poorest of the poor … it’s impossible.” One of our key informants from the CCAP staff told us: “The project has a goal of improving resilience, and it’s not going to be in five years that we will be able to achieve the goal. When CCAP started, both municipalities had problems, they didn´t have a clear strategy for climate change adaptation, and they had no one with the capacity to develop one. The municipalities are our partners, and have to work together with us; we can’t do everything. However, the municipality isn’t ready to absorb these tools, and they are still dealing with supplying basic services.” The “lessons learned” from WRI’s review of more than 130 case studies of projects at the interface of development and adaptation may provide some useful ideas for the CCAP project in dealing with these challenges in its final years. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 31 4.3 Recommendations Our Statement of Work for this evaluation stated that “Based on the above evaluation questions the Evaluation Report should provide targeted evidence as generated from the evaluation to make actionable recommendations for improving CCAP implementation in its final years and for planning future programs.” USAID/Mozambique leaders and staff invited the Evaluation Team to present recommendations of two kinds: 1) practical, achievable recommendations for CCAP during the remaining life of the project; and 2) “big picture, visionary thinking toward future designs.” Below are our key recommendations, both for potential action within the remaining life of the project and for potential future projects with related objectives. 1. Maintain and expand support for the most successful tools The Evaluation Team strongly recommends that CCAP maintain its support for the most successful tools, discussed above, that improve the capacity of municipalities to incorporate climate change adaptation into their planning processes. Continuing support is needed in Pemba and Quelimane to solidify and institutionalize the use of the most important tools, and these same tools should be replicated in Nacala. Some of the successful tools for integrating climate change adaptation into municipal planning processes could form the foundation of some components of future projects. 2. Strengthen linkages and coordination between municipalities and national-level agencies on climate change adaptation issues The Evaluation Team recommends that CCAP renew or initiate activities to improve the coordination with, and balance the engagement of, municipal and national institutions. The project should reassess how it engages with national government agencies. There are three main coordinating institutions: the Ministry of Science and Technology, MITADER, and MAE, the Ministry of State Administration, which apparently compete for funding for climate change oriented programs. USAID/Mozambique and CCAP should carefully consider any decision to fund a Climate Change Center of Knowledge under one institution only (e.g., UEM, INGC), because doing so may weaken initiatives to create a single such center. The Evaluation Team recommends that CCAP coordinate any efforts to establish a Center of Knowledge on Climate Change with MITADER’s Climate Change Department, which we believe is the most appropriate institutional home for such a center, and involve other relevant agencies such as INGC and partners such as UEM. CCAP should sponsor and organize an annual workshop to evaluate the tools and lessons learned from the project and develop new strategies, but it should be coordinated by an institution such as UEM or the Climate Change Department of MITADER. This workshop would be a forum for presenting CCAP to the national-level government institutions, to ensure that the project’s objectives are aligned with national policy and with other projects. Future projects should be designed with mechanisms that ensure an effective balance of efforts at the national and the local levels. 3. Review and revise CCAP’s M&E system and reporting practices Clearly correlating or “mapping” activity reporting to the project’s Results Framework and work plan should be an organizing principle for future progress reporting. Project staff told us that the Year 4 work plan does this, although we did not have a chance to review that work plan. The Evaluation Team recommends that all future CCAP quarterly reports follow the Objective/IR/Activity structure of the annual work plan. Currently, activities are not listed in a manner that clearly connects to IRs and Objectives. This change could facilitate a better understanding of, and clearer communication about, the MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 32 connections between CCAP activities and the project’s Intermediate Results, Objectives, and Goal, with USAID, the CCAP team, and project partners. We recommend that the project retrospectively analyze and report on gender-disaggregated indicator results whenever possible. We also recommend that for future projects, the M&E and reporting system be explicitly linked with the project results framework and work plans, to improve both adaptive management of the project during its life, and to strengthen the communication of results from project activities. 4. Reassess and redesign the mangrove restoration component The Evaluation Team learned during fieldwork for this evaluation that a rapid reassessment of the mangrove restoration activities in Quelimane had recently been completed by mangrove experts from Edwardo Mondlane University (Bandeira and Macamo 2016). This study is a step toward bringing CCAP mangrove restoration in line with science-based best practices being used around the world. We now understand that CCAP is in the process of awarding grants to NGOs in Quelimane to begin to implement the recommendations of the recent assessment. The CCAP Technical Brief on the mangrove restoration activity should be completely revised, and photos of mangroves being planted in furrows be removed. We recommend that the CCAP mangrove restoration work become a case study in the SWAMP Project’s “Mangrove Restoration Best Practices Manual for East Africa,” now being developed (SWAMP is the Sustainable Wetlands Adaptation and Mitigation Program, a collaborative effort by the Center for International Forestry Research (CIFOR), the US Forest Service (USFS) and Oregon State University, which receives support from USAID). We also recommend that an assessment of the status of the mangrove hydrological monitoring program in Quelimane be conducted immediately, and all available data collected to date be provided to Dr. Carl Trettin of the US Forest Service for analysis. We provide a more detailed evaluation of the mangrove restoration component of the project and further recommendations in Annex G. In the future, any mangrove regeneration work to be included in a coastal project as a climate change adaptation measure should be designed from the beginning with expert technical advice, and follow scientifically-based and internationally-accepted best practices. 5. Review and streamline the behavior change and communications strategy The main emphasis of the SBCC Strategy seems to be communication activities and intervention. In general, such interventions are most effective, at least as a first step, when the main barriers to behavior change are lack of awareness and knowledge. Although awareness and knowledge are often a barrier to the DRR and CCA behaviors that CCAP seeks to promote in vulnerable communities, some of the major barriers to the desired behaviors CCAP go beyond awareness and knowledge. Table 3 of the SBCC Strategy shows that economic factors (“high costs”) are also barriers to five of the seven desired behaviors. For such behaviors, communication-based strategies will not be effective without other types of interventions, including significant economic incentives, but neither the SBCC Strategy nor the recently-finalized SBCC “Action Plan” directly targets activities to address the economic barriers to behavior change. Communication-based behavior change strategies have proven to be more effective in some sectors than others. In the health sector, for example, where simple sanitation practices such as hand-washing can provide large health benefits at low cost to participants, communication-based strategies may work well. In order to change behaviors relevant to conservation and natural resources management, however – such as protecting mangroves as “green infrastructure” for climate change adaptation – increasing awareness and knowledge alone rarely motivates sufficient behavior change (Byers, 1996; Byers 2000). All relevant factors motivating or acting as barriers to desired behaviors must be identified and prioritized for a behavior change campaign to be effective, and none given only secondary attention as part of the general social “enabling environment.” The Evaluation Team recommends that the SBCC MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 33 Strategy, and its “Action Plan” as currently proposed, be reviewed, and simplified and streamlined to focus on lowering the barriers to a few of the most important climate-adaptive behaviors through actionable activities that address the most important factors creating those barriers. We also recommend that the project target desired behavior changes in stakeholder groups other than communities and households, such as municipal staff and decision-makers, and not consider them “secondary audiences” for behavior change. Motivating municipal staff to enforce regulations related to construction in highly-vulnerable areas would be one example of a climate-adaptive behavior among this group of actors. We recommend that future climate change adaptation projects use a behavior-change framework in their design from the beginning. 6. Balance long-term climate change adaptation (CCA) and short-term disaster relief reduction (DRR) efforts, and improve integration with development interventions. Part of achieving an effective balance between long-term climate resilience and short-term disaster relief preparedness involves deliberately not supporting activities that would provide incentives for communities to settle or expand in areas of high vulnerability to future climate risks. Some lessons learned from evaluations of USAID-funded integrated conservation and development projects in the 1990s may have relevance for current efforts to better integrate climate change adaptation and development. One lesson is that without careful planning, such projects may have unintended, counterproductive consequences. In an assessment of a USAID-funded integrated conservation and development project in Madagascar, for example, project activities in villages on the borders of protected areas, such as funding schools, clinics, or agricultural improvements, in some cases attracted more people to settle in the area, increasing, rather than decreasing, pressure on the protected area (Grimm and Byers, 1994). A parallel with integrated climate change adaptation and development could be that if activities done in the name of climate change adaptation – such as providing assistance with home construction in areas highly vulnerable to climate change risks – provide short-term incentives for people to settle or stay in those vulnerable areas, the long term goal of climate resilience may actually be hindered, even though in the short term the natural climate hazard risks may be decreased. Some of the communities in highly vulnerable areas with which CCAP is working, such as Icidua and Mirazane (Quelimane Municipality) and Paquitequete (Pemba Municipality), may provide examples. These communities are only a few meters above current sea level, with some houses located much lower (one meter or less). Although detailed data on sea level in Mozambique do not exist, the trends observed seem to be consistent with global and regional trends (INGC, 2009). The Intergovernmental Panel on Climate Change projects that sea levels on the coast of Mozambique will rise between 18 and 59 centimeters (approx. 0.5-2 feet) by the 2090s compared to 1980-1999 sea levels (USAID, 2012). These estimates are based on projections of ocean warming only, and do not include more drastic changes such as melting of polar ice sheets and ice caps. In its 2009 report on climate change and disaster risk, INGC also included a “high sea level rise” scenario, which includes melting polar ice; this scenarios projects a potential sea level rise of up to one meter by 2060, and up to five meters by 2100 (INGC, 2009). INGC also points out that a projected increase in frequency and intensity of tropical cyclones due to global warming will interact with and exaggerate the effects of sea level rise. In addition, Quelimane, like many coastal cities in Mozambique is located in a delta area, an area of land subsidence, further exaggerating the effects of sea level rise. Communities like Paquitequete, Mirazane, and Icidua were identified on the Vulnerability Maps produced with CCAP support as zones of high vulnerability for climate change, with unavoidable exposure to rising sea levels being one of the criteria for that vulnerability score. Both municipalities are now trying to deter people from building in those zones through their zoning and permitting processes. The head of MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 34 the municipal health department in Quelimane told us that in his view, “people should not be living in places like Mirazane.” An INGC staff member told us that relocation of communities in those areas to less vulnerable zones was a high priority from the point of view of disaster prevention, and should be promoted if funding is available. In these areas, which have the potential to become uninhabitable in a generation, any activities that provide incentives for people to settle or stay, and not seek less vulnerable areas, could be seen as counterproductive. If the Municipality of Quelimane decided to repair the currently damaged bridge between Icidua and Mirazane, for example, that would increase access to Mirazane and potentially reduce the risk to the community of Mirazane during a tropical cyclone, giving them a more robust evacuation route, or giving disaster responders better access. However, it could also encourage the current population to stay in a highly vulnerable area that is becoming more and more vulnerable there, and maybe entice more people to move there. In the long term this action, which seemed to promise a short-term reduction in risk, would in the long-term make more people more vulnerable to climate hazards. The Evaluation Team recommends that CCAP carefully consider such potential trade-offs between development, DRR, and CCA activities in the remaining years of the project. We recommend that future projects should be designed carefully to integrate climate change adaptation, disaster risk reduction, and development, and take a long-term perspective that favors interventions that will be sustainable in the face of climate change projections. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 35 ANNEXES Annex A: References Cited and Documents Reviewed Bandeira, S. and C. Macamo. 2016. Avaliação Rápida do Programa de Restauro e de Práticas de Protecção da Floresta de Mangal e, Avaliação de Oportunidades para Meios de Subsistência (Livelihoods) para as Comunidades na Cidade Costeria de Quelimane. Universidade Eduardo Mondlane. Maputo, Setembro 2016. Bandeira, Salomao. 2016. Personal communication by email on 21/24/2016. BLID Consulting Services. 2014. CCAP Stakeholder Analysis in Pemba and Quelimane, June 2014. Byers, B.A. 1996. Understanding and Influencing Behaviors in Conservation and Natural Resources Management. African Biodiversity Series, No. 4, Biodiversity Support Program: A USAID-funded consortium of World Wildlife Fund, The Nature Conservancy, and World Resources Institute. Washington, D.C. Byers, B.A. 2000. Understanding and Influencing Behaviors: A Guide. Biodiversity Support Program. Washington, D.C. FEWS NET. 2016. FEWS NET Food Security Outlook: Mozambique 2016/17, June 2016. Available at: http://www.fews.net/southern-africa/mozambique/food-security-outlook/july-2016. Garrido, Sergio, and Culsumo Carimo. 2014. Avaliação Rápida do Estado do Ecossistema de Mangal no Bairro de Icidua, Possíveis Acções de Intervenção e Diagnóstico de Actores. CCAP report. Grimm, Curt, and Bruce Byers. 1994. NGOs and the Integration of Conservation and Development in Madagascar: An Assessment for the USAID SAVEM Project. Bulletin of the Development Anthropology Network, Vol. 12, Nos. 1 & 2, pp. 30-38. http://www.brucebyersconsulting.com/wp￾content/uploads/2013/07/USAID-Madagascar-ICDPs-Evaluation-1994.pdf IFCR (International Federation of Red Cross and Red Crescent Societies). 2013. A Guide to Mainstreaming Disaster Risk Reduction and Climate Change Adaptation. Accessed November 6, 2015. http://www.ifrc.org/PageFiles/40786/DRR%20and%20CCA%20Mainstreaming%20Guide_final_26%20Mar _low%20res.pdf INGC (National Institute for Disaster Management). 2009. Study on the Impact of Climate Change on Disaster Risk in Mozambique: Synthesis Report. IMF. 2014. Mozambique Rising: Building a New Tomorrow: Infrastructure and Public Investment, Page 28, July 2014. Available at: www.imf.org/external/pubs/ft/dp/2014/afr1404.pdf IMF. 2016. "IMF Staff Concludes Visit", September 2016. Available at: http://www.imf.org/en/News/Articles/2016/09/29/PR16436-Mozambique-IMF-Staff-Concludes-Visit. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 36 Lewis, Roy R. III. 2009. Methods and Criteria for Successful Mangrove Forest Restoration. Ch. 28 in: Gerardo M. E. Perillo, Eric Wolanski, Donald R. Cahoon, Mark M. Brinson, editors, Coastal Wetlands: An Integrated Ecosystem Approach. Elsevier, 2009, p. 787. ISBN: 978-0-444-53103-2. Lewis, Roy R. III. 2010. Mangrove Field of Dreams: If We Build It, Will They Come? Wetland Science and Practice. 27 (1). March 2010. http://www.mangroverestoration.com/pdfs/Lewis%202010%20Mangrove%20Field%20of%20Dreams%20c omplete%20with%20cover.pdf McKee, Neill, Erma Manoncourt, Chin Saik Yoon, and Rachel Carnegie, eds. 2000. Involving People, Evolving Behaviour. Southbound, Penang, Malaysia and UNICEF. Município Da Cidade De Pemba. 2016. Plano Local De Adaptação Às Mudanças Climáticas, March 2016. Município Da Cidade De Quelimane. 2016. Plano Local De Adaptação Às Mudanças Climáticas 2016 - 2026, February 2016. Primavera, et al. 2012. https://www.zsl.org/sites/default/files/media/2014- 05/Manual%20on%20Community-Based%20Mangrove%20Rehabilitation%20(cover).pdf Stringer, Christina, and Carl Trettin. 2015. Assessing the Effectiveness of Mangrove Restoration to Mediate Flooding on the Quelimane Site, USAID Coastal City Adaptation Project, Mozambique: Hydrology Monitoring Program Establishment Report. US Forest Service International Programs Office. Trettin, Carl. 2016. Personal communications by email and telephone discussing status of mangrove monitoring in Quelimane, availability and quality of data, etc.: 10/28/2016; 11/9/2016; 11/28/2016; 11/30/2016; 12/5/2016. USAID. 2012. Climate Change Adaptation Plan. https://www.usaid.gov/sites/default/files/documents/1865/Agency%20Climate%20Change%20Adaptation% 20Plan%202012.pdf USAID. 2012. Mozambique Climate Vulnerability Profile. https://www.climatelinks.org/sites/default/files/asset/document/mozambique_climate_vulnerability_profile _jan2013.pdf USAID. 2012. Climate Change Adaptation in Mozambique. https://www.climatelinks.org/sites/default/files/asset/document/mozambique_adaptation_fact_sheet_feb2 012.pdf USAID. 2013. Mozambique Environmental Threats and Opportunities Assessment. http://pdf.usaid.gov/pdf_docs/pnaea332.pdf USAID CCAP. 2015. Disaster Resilience Scorecard: Pemba Baseline Assessment (PPT), February 2015. USAID CCAP. 2015. Disaster Resilience Scorecard: Quelimane Baseline Assessment (PPT), February 2015. USAID CCAP. 2014. First Annual Work Plan: Phase I: Initial Work Plan Framework, March 2014. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 37 USAID CCAP. 2014. FY2014 Q1 Quarterly Report, May 2014. USAID CCAP. 2014. FY2014 Q2 Quarterly Report, September 2014. USAID CCAP. 2014. FY2014 Q4 Quarterly Report, October 2014. USAID CCAP. 2014. Gender and Youth Stakeholder Analysis, August 2014. USAID CCAP. 2014. Gender and Youth Stakeholder Analysis Recommendation Report, November 2014. USAID CCAP. 2015. FY2015 Annual Report,, October 2015. USAID CCAP. 2015. 2nd Annual Work Plan, April 2015. USAID CCAP. 2015. FY2015 Q1 Quarterly Report, January 2015. USAID CCAP. 2015. FY2015 Q2 Quarterly Report, March 2015. USAID CCAP. 2015. FY2015 Q3 Quarterly Report, July 2015. USAID CCAP. 2015. FY2015 Q4 Quarterly Report, October 2015. USAID CCAP. 2015. Monitoring and Evaluation Plan, February 2015. USAID CCAP. 2015. Summary of Institutional Assessment of Pemba and Quelimane Municipalities and the Gender and Youth Stakeholder Analysis, October 2015 USAID CCAP. 2016. FY2016 Q1 Quarterly Report. January 2016. USAID CCAP. 2016. FY2016 Q2 Quarterly Report. April 2016. USAID CCAP. 2016. FY2016 Q3 Quarterly Report. July 2016. USAID CCAP. 2016. FY2016 Q4 Quarterly Report. October 2016. USAID CCAP. 2016. Overview of Information Services 3-2-1, 2016. USAID CCAP. 2016. Social and Behavior Change Communications Strategy. (December 2015, revised August 2016. USAID CCAP. 2016. Disaster Resilience Scorecard: Nacala Baseline Assessment (Summary PPT), June 2016. USAID CCAP. 2016. Disaster Resilience Scorecard: Pemba Midpoint Assessment (Summary PPT), June 2016. USAID CCAP. 2016. Disaster Resilience Scorecard: Quelimane Midpoint Assessment, June 2016. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 38 USAID CCAP. 2016. Disaster Resilience Scorecard: Quelimane Midpoint Assessment (Summary PPT), June 2016. USAID CCAP. 2016. Local Government Self-Assessment Tool (LGSAT) – Disaster Resilience Scorecard: Nacala Baseline Assessment, June 2016. USAID CCAP. 2016. Local Government Self-Assessment Tool (LGSAT) – Disaster Resilience Scorecard: Pemba Midpoint Assessment, June 2016. USAID CCAP. 2016. Local Government Self-Assessment Tool (LGSAT) – Summary Report, August 2016. USAID CCAP. 2016. Technical Brief: A Course on Climate Change Adaptation and Disaster Risk Reduction, 2016. USAID CCAP. 2016. Technical Brief: Local Adaptation Plan to Climate Change, 2016. USAID CCAP. 2016. Technical Brief: Integrated System for Disaster Information Management (SIGIC), 2016. USAID CCAP. 2016. Technical Brief: Integrated System of Urban Information Management (SIGIU), 2016. USAID CCAP. 2016. Technical Brief: Resilient Housing to the Effects of Climate Change, 2016. USAID CCAP. 2016. Technical Brief: Resilient Housing to the Effects of Climate Change, 2016. USAID/Mozambique. 2016. CCAP Modification – Revised SOW 07.11.2016. CCAP modification document provided by USAID/Mozambique. US Department of State. 2016. Standard Foreign Assistance Indicators. https://www.state.gov/f/indicators/ VEDOR Consulting (Mozambique). 2014. Institutional Assessment of the Pemba Quelimane and Municipalities, July 2014. World Bank. 2016. Mozambique Economic Update: Navigating Low Prices: Actualidade Economica de Mocambique: Navegando Precos Baixos, March 2016. Available at: http://documents.worldbank.org/curated/en/698961467989521291/Mozambique-economic-update￾navigating-low-prices MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 39 Annex B: Evaluation Statement of Work SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK C.1 BACKGROUND The United States Agency for International Development (USAID) is the U.S. Government’s (USG’s) principal implementing organization for international development assistance. The objective of the performance evaluation is to assess project implementation and adequacy of strategies towards achievement of the contract results and objectives. ACTIVITY TO BE EVALUATED Activity/Project Name Coastal City Adaptation Project (CCAP) Implementer Chemonics International, Inc. Contract # AID-656-C-14-00001 Total Estimated Ceiling of the Evaluated $14,904,208.00 Project/Activity(TEC) Life of Project/Activity December 2013- December 2018 The Coastal City Adaptation Project (CCAP) works with municipal governments to increase understanding of urban adaptation issues and increase the application of management options for urban adaptation. CCAP also engages with academia and an array of civic organizations to increase climate awareness and the technical expertise of future urban planners and municipal authorities, and to facilitate local adaptive measures. C.2 MISSION OBJECTIVES The title of the program to be implemented under this Task Order is “The Coastal City Adaptation Project (CCAP) – Mid-Term Performance Evaluation”. This performance evaluation comes at the second year of a five year implementation schedule. CCAP is implemented under USAID/Mozambique Country Development Cooperation Strategy (CDCS) Development Objective 2 (DO2): Resilient, Broad-based Economic Growth Accelerated. The best approach to achieve resilient, broad-based growth and private sector investment in Mozambique is to focus on four priorities: agriculture, economic policy, biodiversity, and climate change. Given Mozambique’s unique biodiversity and vulnerability to climate change, MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 40 economic growth would be accelerated by strengthening conservation through sustainable tourism and protecting the major economic centers along the coast. Because Mozambique is a country faced with chronic poverty, it has populations that are affected by climate almost every year through droughts, severe flooding, and reduced crop yields from extreme temperatures. These recurring crises impede achievement of USAID’s development assistance objectives, especially in the agricultural sector. In response to this extreme poverty and vulnerability, USAID is incorporating resiliency into all three intermediate results (IRs). Under IR 2.1, USAID will promote drought tolerant crops; conservation agriculture; business partnerships; and diversified smallholder production systems. IR 2.2 will support renewable energy and secure land tenure. IR 2.3 will develop natural resource management and biodiversity conservation linked to income and employment. Activities under IR 2.3 also support climate resilience in select urban coastal cities to protect economic assets (i.e., agro-processing; trade infrastructure; tourism), livelihoods, and people; and develop an enabling environment to reinforce resiliency as a central theme in urban planning. USAID’s Climate Change and Development Strategy 2012 – 2016 prioritizes three strategic objectives. Of these, the second objective - Increase resilience of people, places, and livelihoods through investments in adaptation – and the third objective – Strengthen development outcomes by integrating climate change in Agency programming, learning, policy dialogues and operations – guide Mission efforts to increase climate resilience in a country that is highly vulnerable to adverse climate change. Development Hypothesis for IR 2.3: Better management of natural resources (IR 2.3) will occur if conservation efforts are linked to income generation (sub-IR 2.3.1) and if Mozambique’s vulnerable coastal cities are made more resilient to climate change events (sub-IR 2.3.2). Mitigating the impact of more frequent storms in major population centers along Mozambique’s lengthy coast will help safeguard the economy and lives – particularly those living in extreme poverty – from destruction related to climate change. CCAP has three integrated objectives that support the overall project goal and USAID/Mozambique’s CDCS Intermediate Result 2.3.2. CCAP Objective 1: Provision of climate￾resilient urban services by municipalities improved; Objective 2: adoption of climate resilience measures by communities, civic and community organization, including civil society, nongovernmental and faith-based organization increased; and Objective 3: Capacity to potentially implement economic risk-management tools, such as insurance plans and MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 41 contingency funds, for at-risk urban infrastructure and livelihoods increased. Each objective is further broken down into intermediate results (IR), under which corresponding activities are grouped. C.3 SCOPE OF WORK Implementation overview CCAP initially focused on identifying partners, establishing relationships and determining shared priorities. During the first year of implementation CCAP designed most of the initial interventions and defined the technical approaches for activities. During the second year of implementation the project began many of these activities. The project has focused on three principle areas to achieve the project’s objectives and IRs: 1) Creating tools for climate change adaptation and resilience. CCAP has worked closely with the municipalities and other private and public organizations to develop a number of tools to help prepare Pemba and Quelimane for climate change. Examples of these tools are: a. Local adaptation plans to help Pemba and Quelimane integrate climate change into local planning efforts. b. Climate change vulnerability maps that are used to help determine where to build businesses and houses. c. The Integrated Urban Information Management System (SIGIU), which provides cities with unprecedented capabilities for collecting and processing data to inform decisions. d. The Local Government Self-Assessment Tool (LGSAT), a UN-developed tool to help cities measure their progress toward climate adaptation and resiliency. 2) Engaging communities in the field. Community leaders questioned if this project would be like the others that come and go and never work at the field-level with them. CCAP designed a mangrove restoration activity in Quelimane along the Bons Sinais River, to show the impact of climate change and what communities can do about it. Through this activity, CCAP builds trust with local community members by working shoulder to shoulder in the field. This activity has progressed beyond our expectations with the municipality setting aside over 22 hectares as municipal conservation areas; two communities engaged in the production of 55,000 mangrove seedlings in 2015 and committed to produce 120,000 in 2016; at least half a dozen other organizations involved in different aspects of the activity, including the U.S. Forest Service that will be leading the training of UEM faculty and students on mangrove monitoring. The project is also in the process of building ‘climate MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 42 smart houses’ in Pemba and Quelimane to help communities be better prepared for extreme weather and climate events. 3) Scaling up. The project aims to ‘scale up’ tools or solutions that can be more broadly adopted or applied to benefit other coastal cities in Mozambique. The city of Nacala, where another USAID-funded project—Climate Resilient Infrastructure Services (CRIS)—had worked on resilient infrastructure planning, requested CCAP support to help them continue their efforts to build a more climate resilient city. The project conducted an assessment and will be implementing a range of activities that have proven successful in Pemba and Quelimane, among them: LGSAT, SIGIU, local adaptation plans and the development of vulnerability maps for integration with their cadaster. The most ambitious scaling up of CCAP activities, however, is the evolution of the early warning system initially designed for the cities of Pemba and Quelimane, into the national-level Integrated Disaster Information Management System (SIGIC), led and managed by Mozambique’s National Disasters Management Institute (INGC). INGC officially launched SIGIC at a public event in early October 2015 and will test the system during their annual national simulation exercises, which is scheduled for the middle of October 2016. Project goals and development hypothesis Mozambique’s coastal cities serve as economic hubs and primary drivers of the country’s development. These coastal cities house much of the country’s key infrastructure and productive workforce, which are vital to sustaining the strong economic growth levels Mozambique has enjoyed over the past few years. But they are also vulnerable to sea level rise and projected changes in extreme events. Due to their exposure to climate change and a generalized lack of access to resources for adaptation, the Government of the Republic of Mozambique (GRM), National Institute for Disaster Management (INGC) and international development agencies have identified Mozambique’s coastal cities as among the most vulnerable in Africa. Extreme climate events such as cyclones and tropical storms already impose large costs on Mozambican cities. Climate change will worsen the toll by causing sea levels to rise, inundating unprotected low-lying areas. Climate change is also likely to increase the frequency and severity of high-rainfall storms and the most intense cyclones, leading to more destructive floods and damaging or destroying coastal ecosystems and livelihoods. These recurring events will have serious impacts on urban infrastructure and the health of local populations and biodiversity. The most comprehensive analysis to date on the economic costs of climate change in Mozambican cities found that average losses due to climate hazards (for example, inland flooding, coastal flooding, wind damage, and epidemics) will increase substantially by 2030. In the city of Quelimane where CCAP is implemented, annual losses due to climate change are MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 43 projected to rise from approximately US$8 million in 2010 to over US$45 million by 2030, representing a 4 to 5 percent loss of Gross Domestic Product (GDP). Data reflecting the economic costs of climate change for other coastal cities reveal similar results, making climate change adaptation a priority in order to minimize this economic loss. Improving the resilience of major urban centers in the face of climate change is a critical priority for protecting health and livelihoods and safeguarding hard-won development gains. The GRM is aware of the country’s vulnerability to climate change and has begun to take action. At the Rio+20 summits in June 2012, Mozambican President Armando Guebuza revealed Mozambique’s Roadmap to a Green Economy, which was lauded by many as a bold step toward ensuring sustainable economic development. The roadmap covers many sectors including city planning and presents a number of concrete mitigation and adaptation measures. A wide array of evidence persuasively demonstrates that proactive investments in adaptation can cost-effectively avert a significant portion of the projected costs of climate change while yielding substantial co-benefits. To facilitate this process in vulnerable Mozambican coastal communities, CCAP works with municipal governments to increase understanding of urban adaptation issues and increase the application of management options for urban adaptation. CCAP also engages with academia and an array of civic organizations to increase climate awareness and the technical expertise of future urban planners and municipal authorities, and to facilitate local adaptive measures. Specific intended results include: 1.Increased understanding of urban adaptation issues by municipal authorities and increased application of adaptation-relevant management options; 2.Decreased vulnerability to climate change for the population of select coastal cities; 3.Increased local capacity for managing resources to adapt to climate change; and 4.Synthesized and disseminated lessons learned regarding coastal adaptation in urban settings, which can be applied by other coastal cities and future USAID urban adaptation efforts. Under the primary Award, Chemonics is responsible for implementing several grants to NGOs and municipal advisors, and plays a supportive role in facilitating all elements of the program, including working closely with INGC. C.3.1 Target Areas and Groups Municipalities of Pemba (Cabo Delgado Province) and Quelimane (Zambezia Province), Mozambique. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 44 C.3.2 Critical Assumptions The critical assumptions are: 1) that political and civil stability will generally prevail; and 2) that no major natural disasters will occur. C.3.3 Existing Data • Baseline survey: CCAP undertook a baseline survey in 2014 including the Local Government Self-Assessment (LGSAT). • CCAP annual work plans and quarterly and annual reports • Monitoring data: CCAP collects regular monitoring data and report it quarterly to USAID. A Monitoring and Evaluation Plan (with a list of the indicators tracked and annual targets) and the most recent annual report will be provided to the evaluator to review. • CCAP Institutional Analysis for Pemba and Quelimane Municipalities • CCAP Gender and Youth Analysis • Site visit reports • Partner and USAID conducted data quality assessments (DQAs) C.3.4 Scope of Evaluation 1. Audience The primary audience of the evaluation report will be the USAID/Mozambique Mission, specifically the Agriculture Environment and Business team, the Mission Management team, the Program Office, and the Financial Office. The implementing partners will be another key audience. 2. Intended Uses USAID/Mozambique will use the report to facilitate discussions internally and with the implementing partners about whether changes to the implementation plan are recommended for the duration of the activity. The findings will also be used to inform future project design. It will also be used to enhance in-house organizational learning and will provide important information about integrated programs to stakeholders, including the GRM and implementing partners. 3. Evaluation Questions The evaluation will seek to address the following questions: A. To what extent has CCAP been successful in: MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 45 a. assisting Pemba and Quelimane municipalities incorporate climate change adaptation into their planning processes? and; b. helping the relevant stakeholders of municipalities implement adaptation measures? (Communities, civil societies, NGOs, and universities)? B. To what extent has CCAP: a. increased climate resiliency of the most vulnerable populations of Pemba and Quelimane municipalities, including those living in the most vulnerable areas? b. incorporated gender considerations and youth into implementation of its activities? C. To what extent is CCAP prepared to achieve the project’s objectives over the next 2.5 years? a. What are some challenges/obstacles (related to staffing, finances, etc.), reported by project personnel and what is the project’s capacity to respond to those challenges? b. What are some major implementation obstacles/challenges and opportunities (reported by the municipalities and other stakeholders) anticipated over the next 2.5 years of implementation? 4. Evaluation Technical Requirements a. Evaluation Scope This Evaluation will cover CCAP activities in Pemba and Quelimane cities. b. Evaluation Design and Methodology The evaluation data collection and analysis is expected to apply a mixed method approach, utilizing both qualitative and quantitative methods. Data should also be collected using primary and secondary sources. Bidders are asked to use their best judgment in proposing appropriate quantitative and qualitative methods that can be applied in answering the evaluation questions. However, the following illustrative examples are provided to stimulate thinking and guide the decision-making process in the final design of the evaluation methods by the external evaluator. c. Data collection methods Quantitative Data Collection – Data collected using quantitative methods may include surveys of respondents using structured questionnaires. This may include a mini-survey that is administered face-to-face. For example, as part of the CCAP activities, municipality officials MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 46 have participated in workshops and surveys conducted in both Quelimane and Pemba. These could potentially be respondents to surveys that could help inform question 1 and 2 above. Qualitative Data Collection – Qualitative data collection and analytical methods provide opportunities for capturing insights, perceptions, and opinions about the project, from those who have directly or indirectly experienced or participated in the activities of CCAP. For example, there are a variety of qualitative methods that the evaluation team can apply in collecting primary data to answer the key evaluation questions. These may include, but not limited to in-depth interviews with key informants, and focus group discussions (FGD). In-depth Interviews – Using a structured questionnaire comprising primarily of open ended questions, the assessment team could conduct in-depth interviews with stakeholders who will serve as key informants. These may include appropriate respondents from the Municipality of Pemba and Quelimane, respondents from the Ministry of Land, Environment and Rural Development (MITADER) who participated in CCAP activities or are knowledgeable about the project, respondents from other institutions such as Universidad Unilurio (Unilurio) or Universidad Eduardo Mondlane (UEM/Quelimane), etc. These interviews will be expected to provide insights into the programmatic, technical, management, and operations of CCAP over the last two years. Depending on the respondents, and the design of the data collection instrument, these interviews are also likely to yield evidence of results that were achieved by CCAP in financial management, operational transparency and accountability, and human capital strengthening. Focus Group Discussions –The evaluation team may also conduct a series of FGDs with groups of respondents that have directly participated in the CCAP activities either as an implementer, or recipient of the expected benefits derived from the CCAP activities. Each FGD may be comprised of 8-10 respondents. For example, individual FGDS may include discussions with key municipal counterparts, university partners, local subcontractors and grantees, non-governmental organizations and civil society organizations (NGOs/CSOs) and community members. If feasible, FGDs should also be conducted with respondents at the district and provincial levels. Data collected through FGDs will contribute to answering questions on the results and challenges experienced by CCAP during its initial two years of implementation. Review of Documents – Secondary data will be collected by the evaluation team, through the review of documents that were produced on the project by Chemonics, and its partners, throughout the implementation period of CCAP. These documents may include work plans, quarterly and annual reports, or documents related to technical aspects of implementation. These documents will be reviewed prior to primary data collection and may provide the team with a historical narrative of CCAP implementation, while contributing background information in the development of primary data collection tools by the evaluation. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 47 CCAP Monitoring Data – Data collected by Chemonics as part of the project monitoring and evaluation system will also serve as a source of secondary data collection by the evaluation team. Through these data, the evaluation team may be able to conduct secondary analysis of key output indicators that may contribute to the analysis of primary data that will be collected by the team. While these methods are being suggested, it is expected that the evaluation team will lead the effort in deciding on the most appropriate data collection and analytic method that best answer the evaluation questions. Evaluation Questions Expanded by CCAP Action Themes Evaluation Creating Tools for Climate Engaging Data Sources & Collection Methods Questions Change Communities Scaling Up Adaptation and in the Field Resilience 1a. To what How and to what How and to How and to what Secondary/Existing Data extent has CCAP extent did the what extent extent have the • Review baseline LGSAT surveys, climate been successful in tools developed have lessons and change vulnerability maps, and municipal assisting Pemba and/or communities successes of adaptation plans. and Quelimane transferred by been engaged CCAP in Pemba • Review CCAP M&E data, quarterly reports, municipalities CCAP (e.g. by the and Quelimane etc. to determine number and extent of inputs incorporate LGSAT, SIGIU, municipalities been used in (e.g., training, workshops, technical climate change climate change in planning planning assistance) to teach/transfer tools to build adaptation into vulnerability processes to processes in other planning capacity their planning maps) lead to the incorporate coastal cities Primary/Data to be gathered directly by processes? incorporation of climate change and/or in national Evaluation Team climate change adaptation planning • Interview CCAP Project staff, as well as KIs adaptation measures processes? from municipal governments, about use and measures into /actions? benefits of tools for planning, using structured municipal questionnaire and open-ended questions planning processes? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 48 Evaluation Questions Expanded by CCAP Action Themes Evaluation Creating Tools for Climate Engaging Data Sources & Collection Methods Questions Change Communities Scaling Up Adaptation and in the Field Resilience 1b. To what How and to what How and to How and to what Secondary/Existing Data extent has CCAP extent have the what extent has extent have the • Review CCAP M&E data, quarterly reports, been successful in tools developed CCAP engaged lessons and etc. to determine number and extent of inputs helping the by CCAP communities in successes of (e.g., training, workshops, technical relevant motivated or implementing CCAP in Pemba assistance) to various stakeholder groups to stakeholders of helped municipal adaptation and Quelimane in implement adaptation measures municipalities stakeholders measures? municipal Primary/Data to be gathered directly by implement (e.g. stakeholders Evaluation Team adaptation communities, implementing • Interview KIs representing stakeholder measures? NGOs, adaptation groups about CCAP assistance in (communities, universities) measures been implementing adaptation measures using civil societies, implement transferred to or structured questionnaire and open-ended NGOs, and adaptation used by other questions universities)? measures? coastal cities? • Conduct site visits to observe implementation of adaptation measures Evaluation Questions Expanded by CCAP Action Themes Evaluation Creating Tools for Climate Engaging Data Sources & Collection Methods Questions Change Communities Scaling Up Adaptation and in the Field Resilience • Hold focus group discussions (FGDs) about project implementation with stakeholder groups 2a. To what How and to what How and to How and to what Secondary/Existing Data extent has CCAP extent has actual what extent extent have the • Metrics for “climate resiliency” defined increased climate resiliency of the have the most lessons and through review of documents about resiliency of the most vulnerable vulnerable successes of international “best practices” (e.g., UNISDR MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 49 most vulnerable populations to populations in CCAP in Pemba LGSAT scores, IPCC 2012 adaptation best populations of climate the most and Quelimane in practices, USAID ARCC Project analyses) Pemba and variability and vulnerable identifying and • PMP/M&E data reviewed to determine Quelimane change been areas been engaging the most number and extent of “climate resilient” municipalities, affected by identified and vulnerable adaptation practices/measures achieved by including those CCAP? engaged in populations been CCAP living in the most adaptation used in other Primary/Data to be gathered directly by vulnerable areas? activities? coastal cities Evaluation Team and/or at a • Conduct site visits to the most vulnerable national scale? areas of Pemba and Quelimane to observe Evaluation Questions Expanded by CCAP Action Themes Evaluation Creating Tools for Climate Engaging Data Sources & Collection Methods Questions Change Communities Scaling Up Adaptation and in the Field Resilience climate resilient practices/measures achieved by CCAP • Interview KIs from municipal government and all stakeholder groups about their opinions on improved resilience using structured questionnaire and open-ended questions 2b) To what How and to what How and to How and to what Secondary/Existing Data extent has CCAP extent did the what extent extent have the • PMP/M&E data reviewed to determine incorporated tools developed were gender lessons and number and extent of male/female and youth gender and/or and youth successes of incorporation into implementation activities considerations transferred by considerations CCAP in Pemba Primary/Data to be gathered directly by and youth into CCAP (e.g. incorporated and Quelimane Evaluation Team implementation of LGSAT, SIGIU, into with • Questions on structured questionnaire will its activities? climate change community incorporation of ask about gender balance and youth vulnerability engagement gender and youth participation in CCAP implementation maps) processes and considerations activities Evaluation Questions Expanded by CCAP Action Themes Evaluation Creating Tools Data Sources & Collection Methods MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 50 Questions for Climate Engaging Change Communities Scaling Up Adaptation and in the Field Resilience incorporate activities? been scaled up? gender and youth considerations? 3. To what extent To what extent To what extent To what extent Secondary/Existing Data is CCAP will CCAP will CCAP will CCAP • PMP/M&E data reviewed to determine prepared to achieve its achieve its achieve its progress toward targets for all indicators achieve the objective (and objective (and objective (and • All project progress reports reviewed project’s targets) for targets) for targets) for objectives over creating tools? engaging scaling up? Primary/Data to be gathered directly by the next 2.5 communities? Evaluation Team years? • Project staff interviewed • KIs from municipal government and all stakeholder groups asked for opinions on project progress toward objectives 3a) What are What are some What are some What are some Secondary/Existing Data some challenges/ challenges/ challenges/ challenges/ • PMP/M&E data reviewed to determine obstacles (related obstacles to obstacles to obstacles to progress toward targets for all indicators Evaluation Questions Expanded by CCAP Action Themes Evaluation Creating Tools for Climate Engaging Data Sources & Collection Methods Questions Change Communities Scaling Up Adaptation and in the Field Resilience to staffing, meeting the meeting the meeting the • All project progress reports reviewed finances, etc.), objective and objective and objective and reported by targets for targets for targets for scaling Primary/Data to be gathered directly by project personnel creating tools for engaging up the lessons and Evaluation Team and what is the climate change communities successes in • Project staff interviewed about challenges project’s capacity adaptation and reported by Pemba and using structured questionnaire and open￾to respond to resilience project Quelimane to ended questions those challenges? reported by personnel? other coastal project cities or the personnel? national level reported by MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 51 project personnel? 3b) What are What are some What are some What are some Primary/Data to be gathered directly by some major challenges/ challenges/ challenges/ Evaluation Team implementation obstacles to obstacles to obstacles to • KIs from municipal government and all obstacles/challen meeting the meeting the scaling up the stakeholder groups asked for opinions on Evaluation Questions Expanded by CCAP Action Themes Evaluation Creating Tools for Climate Engaging Data Sources & Collection Methods Questions Change Communities Scaling Up Adaptation and in the Field Resilience ges and objective and objective and lessons and obstacles, challenges, and opportunities opportunities targets for using targets for successes in anticipated over the next 2.5 years of CCAP (reported by the tools for engaging Pemba and using structured questionnaire and open￾municipalities improving communities in Quelimane to ended questions and other climate change climate change other coastal stakeholders) adaptation and adaptation cities or the anticipated over resilience reported by national level the next 2.5 years reported by municipalities reported by of municipalities and municipal municipalities implementation? and municipal stakeholders? and municipal stakeholders? stakeholders? 5. Recommendations Based on the above evaluation questions the Evaluation Report should provide targeted evidence as generated from the evaluation to make actionable recommendations for improving CCAP implementation in its final years and for planning future programs. C.4 IMPLEMENTATION AND MANAGEMENT PLAN The contractor shall provide contract management necessary to fulfill all the requirements of this Task Order. C.5 PERFORMANCE MONITORING MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 52 The contractor’s performance shall be evaluated based on the completion of specific tasks as outlined in the Task Order, adherence to the work plan, and reports submitted to the Contracting Officer Representative (COR). END OF SECTION C SECTION F – DELIVERIES OR PERFORMANCE F.1 PERIOD OF PERFORMANCE The estimated period of performance for this task order is 22 weeks, August 16, 2016 to January 17, 2017. It is anticipated that the final deliverable will be submitted by week 17. However, five (5) weeks have been added to allow for additional performance time in the event of unforeseen delays. F.2 DELIVERABLES Timeframe Deliverables Week 1-4 Evaluation Work Plan and Design: Within two weeks of the award of the contract, a draft work plan and evaluation design shall be completed and presented by the lead evaluator to the Contracting Officer’s Representative (COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. The evaluation design (which will become an annex to the Evaluation report) will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID offices and relevant stakeholders are asked to take up to 5 business days to review and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 53 days. In-briefing: Within one day of arrival in Mozambique, the evaluation team will have an in-briefing with the USAID/AEB/Environment team for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. Week 5-8 Mid-term Briefing and Interim Meetings: The evaluation team is expected to hold a mid-term briefing with USAID/AEB/Environment on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation COR/manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. Final Exit Briefing and Presentation: The evaluation team is expected to hold a final exit briefing and presentation with the larger USAID Mission prior to leaving the country to discuss the status of data collection and preliminary findings. This presentation will be scheduled as agreed upon during the in-briefing. USAID feedback obtained in the final briefing should be addressed in the final report. Week 9-17 Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section F.6. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, Mission Program Office will have 15 business days in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again the Mission Program Office will review and send comments on this final draft report within 5 business days of its submission. Final Evaluation Report: The evaluation team will be asked to take no more than 5 business days to respond/incorporate the final comments from USAID Mozambique. The evaluation team leader will then submit the final report to the COR. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. Final Evaluation Report Requirements: MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 54 All of the evaluation findings, conclusions, and recommendations shall be documented in the Final Report. All written deliverables shall also be submitted electronically to the COR. Bound/color printed deliverables may also be required, as directed by the COR. The evaluation final report should be less than 30 pages and it should include an executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described in this link: http://usaidlearninglab.org/library/how-note-preparing-evaluation-reports The report should be formatted according to the evaluation report described in this link: http://usaidlearninglab.org/library/evaluation-report-template The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). F.3 TECHNICAL DIRECTION AND DESIGNATION OF RESPONSIBLE USAID OFFICIALS Adam Walsh Contracting Officer U.S. Agency for International Development (USAID)/Mozambique Rua 1231, no. 41 JAT Complex, Maputo Mozambique Telephone: +258-21-352181 Email: awalsh@usaid.gov The Contracting Officer Representative (COR) will be designated at the time of award. F.4 PLACE OF PERFORMANCE The place of performance under this Task Order is Maputo, Pemba and Quelimane, Mozambique, as specified in the Statement of Work. F.5 AUTHORIZED WORK DAY / WEEK MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 55 No overtime or premium pay is authorized under this Task Order. F.6 REPORTS AND DELIVERABLES OR OUTPUTS In addition to the requirements set forth for submission of reports in Section H, and in accordance with AIDAR clause 752.242-70, the contractor shall submit reports, deliverables or outputs as further described (referenced in Sections F.2). All reports and other deliverables shall be in English language. The contractor must adhere to USAID’s requirements for reporting as stipulated in ADS 203. (www.usaid.gov/policy/ads/200/203.pdf ) As a minimum, the Draft and Final Reports must include the following elements: • Executive Summary • Scope and Methodology used • Limitations • Important findings • Conclusions • Recommendations • Lessons Learned • Annexes with all data collection instruments and actual sources of information To ensure the quality of the evaluation, the reports must answer all questions raised in the SOW and the SOW must be included in an annex in the Final Report. Recommendations must be stated in an actionable way and responsibility for implementation must be assigned. F.7 AIDAR 752.7005 SUBMISSION REQUIREMENTS FOR DEVELOPMENT EXPERIENCE DOCUMENTS (JAN 2004) (AAPD 04-06) (a) Contract Reports and Information/Intellectual Products. (1) The Contractor shall submit to USAID’s Development Experience Clearinghouse (DEC) copies of reports and information products which describe, communicate or organize program/project development assistance activities, methods, technologies, management, research, results and experience as outlined in the Agency's ADS Chapter 540. Information may be obtained from the Contracting Officer Representative (COR). These reports include: assessments, evaluations, studies, development experience documents, technical reports and annual reports. The Contractor shall also submit to copies of information products including training materials, publications, databases, computer software programs, videos and other MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 56 intellectual deliverable materials required under the Contract Schedule. Time-sensitive materials such as newsletters, brochures, bulletins or periodic reports covering periods of less than a year are not to be submitted. (2) Upon contract completion, the Contractor shall submit to DEC an index of all reports and information/intellectual products referenced in paragraph (a)(1) of this clause. (b) Submission requirements. (1) Distribution. (i) At the same time submission is made to the COR, the Contractor shall submit, one copy each, of contract reports and information/intellectual products (referenced in paragraph (a)(1) of this clause) in either electronic(preferred) or paper form to one of the following: (A) Via E-mail: docsubmit@dec.cdie.org; (B) Via U.S. Postal Service: Development Experience Clearinghouse, 8403 Colesville Road, Suite 210, Silver Spring, MD 20910, USA; (C) Via Fax: (301) 588-7787; or (D) Online: http://www.dec.org/index.cfm?fuseaction=docSubmit.home (ii) The Contractor shall submit the reports index referenced in paragraph (a)(2) of this clause and any reports referenced in paragraph (a)(1) of this clause that have not been previously submitted to DEC, within 30 days after completion of the contract to one of the address cited in paragraph (b)(1)(i)of this clause. (2) Format. (i) Descriptive information is required for all Contractor products submitted. The title page of all reports and information products shall include the contract number(s), Contractor name(s), name of the USAID contracting officer representative, the publication or issuance date of the document, document title, author name(s), and strategic objective or activity title and associated number. In addition, all materials submitted in accordance with this clause shall have attached on a separate coversheet the name, organization, address, telephone number, fax number, and Internet address of the submitting party. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 57 (ii) The report in paper form shall be prepared using non-glossy paper (preferably recycled and white or off-white using black ink. Elaborate art work, multicolor printing and expensive bindings are not to be used. Whenever possible, pages shall be printed on both sides. (iii) The electronic document submitted shall consist of only one electronic file which comprises the complete and final equivalent of the paper copy. (iv) Acceptable software formats for electronic documents include WordPerfect, Microsoft Word, and Portable Document Format (PDF). Submission in PDF is encouraged. (v) The electronic document submission shall include the following descriptive information: (A) Name and version of the application software used to create the file, e.g., MSWord6.0 or Acrobat Version 5.0. (B) The format for any graphic and/or image file submitted, e.g., TIFF-compatible. (C) Any other necessary information, e.g. special backup or data compression routines, software used for storing/retrieving submitted data or program installation instructions. END OF SECTION F MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 58 Annex C: Biographical Sketches of Evaluation Team Dr. Bruce Byers, Team Leader Bruce Byers is an ecologist and natural resources management specialist with more than 30 years of experience working in more than 40 countries in Africa, Asia, and Latin America. He has led many multi-disciplinary and international teams for major evaluations, assessments, and strategic planning exercises, on topics including biodiversity conservation, forestry, climate change adaptation and mitigation, ecosystem services, and environmental communication, outreach, and behavior change. In 2007 and 2008, he led a comprehensive final evaluation of the USAID Global Conservation Program. Dr. Byers has worked in Mozambique several times. In 2000 he served on an evaluation of the WWF Southern Africa Regional Program, which included Mozambique, and in 2002 and again in 2012 he led teams conducting Environmental Threats and Opportunities Assessments (ETOAs) for USAID/Mozambique. His strong written and oral communication skills are reflected in numerous publications and presentations, which synthesize complex information and clearly communicate findings to diverse target audiences and stakeholders. Mr. Michael Cote, Climate Change Adaptation Specialist Michael Cote has more than 10 years of experience providing technical support on climate change adaptation in more than a dozen countries. He currently supports the USAID/GCC Office’s Climate Change Integration Support (CCIS) project, which implements the Agency Adaptation Plan and Executive Order (EO) 13677, “Climate Resilient International Development.” He formerly managed parts of the Climate Change Resilient Development (CCRD) project, and supported the CRIS pilot projects in Africa, Dominican Republic, Peru, and Vietnam. He was Technical Lead of the High Mountain Adaptation Partnership in Nepal and Peru and was Director of Communications for the overall project. Ms. Ariane Dinis, Data Analyst Ariane Dinis is a researcher and project manager specializing in socio-economic projects with Verde Azul in Maputo, Mozambique. She holds Bachelor’s and Master’s degrees from the NOVA School of Business and Economics, Lisbon, Portugal. Ms. Dinis has five years of experience in data analysis, economic modelling, and statistical assessments, and has participated in numerous data collection projects, academic studies, and analysis and modeling projects in Mozambique and in Portugal. Recently she served as Deputy Director of a project of “Re-Registration of Beneficiaries of INAS (National Institute Social Action)” in Mozambique, where she was responsible for overseeing field work, database management, and quality control. She is a native speaker of Portuguese and is fluent in English and Spanish. Mr. Rui Mirira, Survey Administrator/Interviewer Rui Mirira is a forest engineer employed by Verde Azul. He holds a Master’s Degree in Rural Development, and has more than 12 years’ experience in natural resources management. Mr. Mirira has managed several environmental impact assessments, designed environmental management plans, conducted ethnobotanical studies, and led renewable energy studies in Mozambique. He has taught natural resources management and environmental impact assessment courses at the Pedagogical MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 59 University, Institute of Gaza, and São Thomas University. He speaks Portuguese, English, and several local Mozambican languages. Dr. Kemal Vaz, Evaluation Advisor Kemal Vaz holds a Ph.D. from the University of Virginia, and has more than 20 years of professional experience in land use planning, natural resources management, and agriculture. Dr. Vaz is Managing Director of Verde Azul, a Mozambican consulting firm with more than 65 permanent staff and a portfolio of projects of approximately $4 million USD. He is a specialist in designing and implementing environmental audits, environmental impact assessments, and environmental management plans. Dr. Vaz has been a Lecturer at Eduardo Mondlane University, Faculty of Agronomy and Forestry. He was a founding member and first president of the Mozambican Association of Environmental Impact Assessment (AMAIA). MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 60 Annex D: Final Evaluation Design and Work Plan Revise to show final schedule as implemented, and add a note that the schedule has been revised U.S. Agency for International Development Request for Task Order Proposals (RFTOP) Number SOL-656-16-000005 The Coastal City Adaptation Project (CCAP) Midterm Performance Evaluation Final Evaluation Design and Work Plan Prepared by: Major Subcontractor (Local): MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 61 September 29, 2016 Arlington, VA Table of Contents 1. Evaluation Background and Approach ............................................................................1 2. Evaluation Questions.......................................................................................................3 3. Gathering Information to Answer Evaluation Questions.................................................4 3.1 Information from meetings, interviews, and site visits .............................................4 3.2 Review and analysis of secondary/existing information contained in documents and reports ........................................................................................8 4. Work Schedule ..............................................................................................................10 Annexes Annex A: Evaluation Questions and Sub-Questions Guide/Script ...................................12 Annex B: Detailed Work Schedule – Information-Gathering Phase in Mozambique ......17 MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 62 1 Evaluation Background and Approach The ECODIT Consortium will carry out the Midterm Performance Evaluation of the Coastal City Adaptation Project (CCAP) in a participatory, unbiased, efficient, and cost-effective manner. We propose a process that is straightforward, technically sound, and appropriate to the Mozambican context. We understand the objectives and requirements of the evaluation given in the RFTOP, and understand USAID/Mozambique’s intended uses of the evaluation findings. The Evaluation Design described here will achieve the following objectives:  Determine how CCAP is performing relative to its three integrated objectives;  Provide an objective view of progress towards the project’s expected results;  Identify any possible gaps in project performance that could hinder success;  Help USAID/Mozambique and its implementing partner for CCAP to determine what changes may be necessary to solidify progress during the remaining project period; and  Provide lessons learned to inform climate change adaptation efforts in other coastal cities in Mozambique and elsewhere. Evaluations that objectively review the performance of a project are an important tool for adaptive learning. Effective evaluations with this objective require the cooperation and participation of the designers, funders, and implementers of the project being evaluated, but each of these groups is invested in, and has sensitivities regarding, project performance. Therefore, transparent evaluation methods and trust among all participants in the evaluation are essential elements of this Evaluation Design. We see the CCAP Midterm Evaluation as a learning-oriented exercise, and plan to conduct it in a participatory and “friendly” manner. We here present a Final Evaluation Design and Work Plan that meets the requirements of the Scope of Work (SOW), responds to the evaluation questions posed by USAID/Mozambique, and takes into account such factors as the availability of existing information and available resources (financial, time, and human). In this Final Evaluation Design we have incorporated the suggestions and addressed the comments from USAID/Mozambique on our Draft Evaluation Design and Work Plan. The institutional “landscape” related to climate change adaptation, planning, and development in Mozambique is complex, and Mozambican experience to understand and navigate it is required. Responsiveness of and access to key informants has a personal dimension in Mozambique. Working with Verde Azul, we will bring to bear an understanding of the institutional and economic context of development and climate change in the country. Through Verde Azul, we will have access to a responsive network of contacts in Mozambican government agencies and non-governmental organizations (NGOs), and will be able to efficiently arrange meetings and interviews with representatives of CCAP partners and stakeholders who will serve as our key informants and primary sources of first-hand information. We do not anticipate significant risks that would prevent us from completing the evaluation and providing a high-quality Evaluation Report on time. However, any evaluation has certain limitations and uncertainties. One such limitation relates to the participatory way in which we expect to conduct the evaluation. We will, to a significant degree, depend on the opinions of MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 63 project staff, partners, and stakeholders to assess CCAP’s progress toward its objectives, so some biases are inevitable. Our evaluation methodology avoids and mitigates such biases whenever possible (such as through “triangulation” of opinions from different key informant groups). We plan to fully explain the limitations of the data and interpretations of results in the Evaluation Report. Another possible uncertainty stems from the fact that this is a midterm evaluation. CCAP implementation began in January, 2014, and it may be too early to assess the project’s impact at the level of the project goal, “Climate resilience in selected Mozambican coastal cities increased,” given the complex nature of climate change adaptation activities that the project is implementing. Nonetheless, we are confident that the conclusions of this midterm evaluation will be accurate, useful, and actionable. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 64 2 Evaluation Questions The Final Evaluation Design described here is intended to fully answer the seven general evaluation questions posed by USAID-Mozambique in the RFTOP (Exhibit 1). We have compared the CCAP Results Framework and list of indicators given in the M&E Plan with the evaluation questions, and we believe that despite the lack of explicit, one-to-one correspondence, answering the seven general evaluation questions will provide evidence to evaluate CCAP progress to toward its three objectives. Exhibit 1: Evaluation Questions 1a. To what extent has CCAP been successful in assisting Pemba and Quelimane municipalities to incorporate climate change adaptation into their planning processes? 1b. To what extent has CCAP been successful in helping the relevant stakeholders of municipalities to implement adaptation measures? (communities, civil societies, NGOs, and universities)? 2a. To what extent has CCAP increased climate resiliency of the most vulnerable populations of Pemba and Quelimane municipalities, including those living in the most vulnerable areas? 2b. To what extent has CCAP incorporated gender considerations and youth into implementation of its activities? 3a. To what extent is CCAP prepared to achieve the project’s objectives over the next 2.5 years? 3b. What are some challenges/ obstacles (related to staffing, finances, etc.), reported by project personnel and what is the project’s capacity to respond to those challenges? 3c. What are some major implementation obstacles/challenges and opportunities (reported by the municipalities and other stakeholders) anticipated over the next 2.5 years of implementation? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 65 3 Gathering Information to Answer Evaluation Questions The Evaluation Team will use various methods to gather the information that will provide evidence regarding each of the seven general evaluation questions posed by USAID￾Mozambique. Our methodology will draw information both from existing secondary data sources, such as quarterly reports and project M&E data, and from primary information collected by the Evaluation Team during meetings, interviews, and site visits in Mozambique. We will gather both qualitative and quantitative information, which will allow for cross-checking and validating results to the maximum extent possible. 3.1 Information from meetings, interviews, and site visits Our main method of gathering primary information will be structured interviews with key informants representing the institutions, partners, and stakeholders working with CCAP. We have developed an interview guide, or “script,” for use in these key informant interviews (see Annex A: Evaluation Questions and Sub-Questions Guide/Script). The interview guide will be adapted and tailored based on the type of key informant we are interviewing (e.g., project staff, municipal staff, community representatives). USAID/Mozambique provided us with a list of institutions currently engaged with the CCAP (Exhibit 2), and we obtained information for points-of-contact in these institutions from the project. We have scheduled meetings to interview representatives of these institutions. Exhibit 2: Institutions Engaged with CCAP by City Maputo Instituto Nacional de Gestao de Calamidades – INGC Ministerio de Terra, Ambiente e Desenvolvimento Rural – MITADER Africa Climate Change Resilience Alliance – ACRA Eduardo Mondlane University Pemba Municipality of Pemba Provincial Office of INGC – National Disaster Management Institute Direção Provincial de Terra, Ambiente e Desenvolvimento Rural – DEPTADER Universidade de Lúrio – UNILURIO Universidade Católica Radio Sem Fronteira Theatre Tambu-tambulani-tambu Paquitequete neighborhood Cariaco neighborhood Quelimane Municipality of Quelimane Provincial Office of INGC – National Disaster Management Institute Direção Provincial de Terra, Ambiente e Desenvolvimento Rural – DEPTADER Eduardo Mondlane University – High School of Marine and Coastal Sciences Universidade Católica Associação dos Naturais e Amigos de Madal – ANAMA Associação dos Jovens Amigos de Quelimane – AJAQ Theatre Retratistas Icidua neighborhood MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 66 Each of the seven general evaluation questions posed in the SOW for this midterm evaluation is complicated and multi-faceted, because each relates to many different project activities, accomplishments, and challenges. Evaluation Question 1a, for example, deals with municipal planning capacity and “tools” for climate change adaptation planning, and there are at least five or six tools that the project has developed and/or supported to enhance municipal planning capacity. Therefore, we need sub-questions for each of the seven main evaluation questions, in order to “unpack” what is inside them. We will ask our key informants a series of short, specific questions to unpack and understand their opinion about each general evaluation question. A summary of the answers to the sub-questions should then provide, in a general way, the answer to each main evaluation question. A complete list of the sub-questions we propose to use in our key informant interviews, organized under the seven general evaluation questions posed for the evaluation, is given in Annex A: Evaluation Questions and Sub-Questions Guide/Script. The semi-structured interviews with key informants will be analyzed to provide both qualitative and quantitative information for the evaluation. Depending on their role in, or engagement with, CCAP, project managers, partners, and beneficiaries have different perspectives on the project, and can be divided into different types of key informant groups. The information-gathering instruments we have developed ask these different groups for information about the project in tailored ways, although many of the questions are the same for all groups. We plan to use four versions of the interview guide/script, tailored with questions that are relevant for the following types of key informants:  Interview Guide/Script #1 – for CCAP Staff  Interview Guide/Script #2 – for Municipalities  Interview Guide/Script #3 – for National Institutions and University Partners  Interview Guide/Script #4 – for Communities, Associations, Media As much as possible we want to get the opinions of key informants using a semi-quantitative, rating or ranking (Likert-type) scale. We have therefore written many of the sub-questions as if they were part of a survey or questionnaire, although we expect to ask these questions orally during our interviews and meetings. The semi-quantitative scales will be used, as much as possible, to help the interviewers note the relative ranking of opinions on the topic they are asking about. However, we expect that each question will also stimulate further open discussion about the topic, and we will analyze the content of the notes taken during the interviews for qualitative information regarding that particular topic or issue. Depending on the key informant group, the order in which the evaluation questions and sub-questions are asked may vary (i.e., interviews will not necessarily always start with Evaluation Question 1a, Sub-Question 1). We have not yet had the opportunity to pilot test these semi-structured interview and discussion guides with key informant groups in the field. Therefore, we expect to have to adapt them somewhat based on how well they function to elicit the needed information during our first series of interviews. We consider this adaptation part of the normal process of information-gathering in an evaluation of this type (i.e., a midterm, mainly adaptive/formative, evaluation). Because our team members from Verde Azul will have the opportunity for a second round of visits to Pemba and Quelimane during Weeks 9 and 10 of the evaluation, we will be able to take advantage of our preliminary analysis of the results of information-gathering during Weeks 6-8 to modify or MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 67 supplement the interview scripts and questions, and gather any further information needed for the evaluation. Exhibit 3 provides more details about how we intend to adjust the questions comprising various versions of our interview guide for different key informant groups. Exhibit 3: Key Informant Groups and Interview Questions Key Informant Type Subgroup Relevant Evaluation Questions Interview Guide Version CCAP Staff Maputo Office 1a-b, 2a-b, and 3a-c Interview guide/script #1; all questions (see Annex A: Evaluation Questions and Sub￾Pemba and questions Guide/Script) Quelimane Offices 1a-b, 2a-b, and 3a-c Municipality Mayor 1a-b, 2a-b, 3a, 3c Interview guide/script #2; omit questions Councilmen/Staff 1a-b, 2a-b, 3a, 3c 23, 24 INGC National 1a-b, 2a, 3a, 3c Interview guide/script #3; omit questions Provincial 23, 24 MITADER/DEPTADER National 1a-b, 2a, 3a, 3c Provincial University Eduardo Mondlane U. 1a-b, 2a, 3a, 3c Interview guide/script #3; omit questions 23, 24 UNILURIO Universidade Católica Community 1b, 2a-b, 3c Interview guide/script #4; omit questions 1-7, 17-22 Associations 1b, 2a-b, 3c Interview guide/script #4; omit questions 1-7, 17-22 Media 1b, 2a-b, 3c Interview guide/script #4; omit questions 1-7, 17-22 Other ? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 68 We expect the interviews with key informants to follow a format approximately as described in Exhibit 4. Our experience in other evaluations has taught us that flexibility and adaptability is needed when conducting such interviews, and we intend in all cases to conduct the interviews in a participatory spirit. Exhibit 4: Hypothetical Interview with Key Informants Meetings with key informants (individuals or small groups) will be scheduled for one hour, but with flexibility to go longer in most cases. Meetings with communities will be scheduled in at least a two-hour block of time. Opening of meeting (approximately 5-10 minutes): We generally expect the meetings to be conducted in Portuguese. Members of the Evaluation Team will introduce themselves to the key informant, and present business cards. A Verde Azul team member will generally take the lead (except in cases where the key informant wishes to speak in English), and explain briefly the purpose of the evaluation. They will explain that this is a midterm evaluation, and is “friendly,” participatory, and mainly focused on adaptive learning to improve the project in its last few years of implementation. They will explain that we are generally familiar with the CCAP project, and need to gather certain kinds of information in order to answer the questions USAID has posed, and so we need to be aware of our time, to make sure that we can discuss all of the required questions. The interviewer(s) will try to stick fairly closely to the “script,” or list of questions, although the order in which these are asked may vary depending on the person or group being interviewed. We expect that many/most questions will elicit a short discussion and additional opinions from the key informants. With timekeeping help from other team members, the interviewer will move on to the next question if the discussion rambles too far afield from the specific topic. For as many of the questions as possible the interviewer(s) will try to ask the key informant for a semi-quantitative opinion, suggesting that they qualify their answers with words like those used in questionnaires with ranking or rating (Likert-type) scales, such as “extremely,” “very,” “moderately,” “somewhat,” or “not very.” Any time left at the end of the meeting can be used to come back to any topic that deserves more time to explore. Questions from Interview Guide/Script (approximately 50 minutes): See Annex A: Evaluation Questions and Sub-questions Guide/Script Explanation of evaluation timeline and next steps, make plans to re-contact for more information if necessary, goodbyes and thank-yous (approximately 5 minutes) MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 69 3.2 Review and analysis of secondary/existing information contained in documents and reports Review and analysis of secondary and existing information is another source of evidence for this evaluation. Details of the data sources and analyses that will contribute to answering the evaluation questions are given in Exhibit 5. Exhibit 5: Secondary/Existing Data Sources and Analysis by Evaluation Question Evaluation Question Secondary/Existing Data Sources and Analysis 1a. To what extent has CCAP been successful in assisting Pemba and Quelimane municipalities incorporate climate change adaptation into their planning processes?  Review baseline LGSAT surveys, climate change vulnerability maps, and municipal adaptation plans  Municipal adaptation plans scored for inclusion of international best practices  Review CCAP M&E data, quarterly reports, etc. to determine number and extent of outputs (e.g., training, workshops, technical assistance) to teach/transfer tools to build planning capacity 1b. To what extent has CCAP been successful in helping the relevant stakeholders of municipalities implement adaptation measures? (communities, civil societies, NGOs, and universities)?  Review CCAP M&E data, quarterly reports, etc. to determine number and extent of inputs outputs (e.g., training, workshops, technical assistance) to various stakeholder groups to implement adaptation measures 2a. To what extent has CCAP increased climate resiliency of the most vulnerable populations of Pemba and Quelimane municipalities, including those living in the most vulnerable areas?  PMP/M&E data reviewed to determine number and extent of “climate resilient” adaptation practices/measures achieved by CCAP  CCAP metrics for “climate resiliency” compared with international “best practices” (e.g., UNISDR LGSAT scores, IPCC 2012 adaptation best practices, USAID ARCC Project analyses)  Baseline LGSAT scores for Quelimane and Pemba compared with midterm scores (if available) 2b) To what extent has CCAP incorporated gender considerations and youth into implementation of its activities?  M&E data reviewed to determine number and extent of male/female and youth incorporation into implementation activities  M&E data analyzed against targets for gender and youth incorporation in project activities 3a. To what extent is CCAP prepared to achieve the project’s objectives over the next 2.5 years?  M&E data reviewed to determine progress toward targets for all indicators  All project progress reports reviewed  M&E data analyzed against targets 3b) What are some challenges/obstacles (related to staffing, finances, etc.), reported by project personnel and what is the project’s capacity to respond to those challenges?  Progress reports and M&E data reviewed and analyzed for challenges/obstacles MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 70 3c) What are some major implementation obstacles/challenges and opportunities (reported by the municipalities and other stakeholders) anticipated over the next 2.5 years of implementation?  Climate related news, information about stability in the region, municipal budgets, political challenges, etc., reviewed and analyzed MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 71 4 Work Schedule The Work Schedule we propose has been designed to accomplish all of the needed evaluation activities in an efficient and timely manner (see Exhibit 6). The intensive information-gathering from primary sources through meetings, interviews, surveys, and site visits that will take place in Weeks 6-8, when the Washington, DC-based Team Leader and Climate Change Specialist are in Mozambique, will involve close collaboration and coordination with USAID/Mozambique staff, CCAP staff, and project partners and stakeholders. The Evaluation Team plans to travel to both Pemba and Quelimane, and coordinate schedules with CCAP staff and partners there. We recognize that rapid, responsive communication, and flexibility and adaptation, will be especially essential during these weeks. During Weeks 9-10 team members from our local evaluation partner Verde Azul will be able to revisit Pemba and Quelimane, if and as needed, to continue and finalize information gathering there, as well as to complete any remaining meetings and interviews in Maputo. The following Work Schedule will guide the implementation of the Final Evaluation Design we describe above and our work to produce the Evaluation Report. A Detailed Work Schedule for Weeks 6-8 is provided in Annex B: Detailed Work Schedule – Information-Gathering Phase in Mozambique. Exhibit 6: Evaluation Work Schedule Week Dates Activities 1 8/29-9/2  ECODIT and Verde Azul (VA) conduct an interactive planning conference via teleconference and email  Team begins background documents review 2 9/5-9/9  Team continues reviewing background documents  Team prepares Draft Evaluation Design and Work Plan 3 9/12-9/16  Team continues to prepare Draft Evaluation Design and Work Plan  Draft Evaluation Design and Work Plan submitted to USAID/Mozambique by COB on Friday 16 Sept. 4 9/19-9/23  Team Leader and Climate Change Specialist meet Washington, DC-based Chemonics staff and USAID Africa Bureau Senior Climate Change Advisor  USAID/Mozambique (and relevant stakeholders) review draft Evaluation Design and Work Plan and submit comments to COR  Preliminary list of contacts and meetings finalized in discussion with USAID/Mozambique and CCAP staff  COR consolidates comments and sends them to the Evaluation Team by COB on Friday 23 Sept.  Team prepares for field work and Verde Azul contacts key informants in Maputo, Pemba, and Quelimane to schedule meetings and interviews 5 9/26-9/30  Team addresses comments and prepares Final Evaluation Design and Work Plan to be submitted to USAID/Mozambique by COB on Friday 30 Sept.  Team finalizes logistical preparations for field work and schedule of meetings and interviews 6 10/3-10/7  Expat team members travel from Washington, DC (on weekend) to arrive in Maputo on Sunday 2 Oct.  3 Oct., Monday AM: Expat team members meet with Verde Azul team members for work/planning session MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 72  3 Oct., Monday 1:30-3:00 PM: In-briefing with USAID/Mozambique to clarify expectations and launch field work  3 Oct., Monday 4-5:30 PM: Meet Chemonics CCAP staff in Maputo Office  4 Oct., Tuesday: Evaluation Team travels to Quelimane on 12:30 PM LAM flight  5 Oct., Wednesday AM: Meet Quelimane-based Chemonics CCAP staff  Meetings in Quelimane with local CCAP project staff, representatives of local stakeholder/partner institutions, local communities, and site visits as relevant – Wednesday to Friday of Week 6  Evaluation Team returns to Maputo (on Friday night LAM flight) 7 10/10- 10/14  10 Oct., Monday: Meetings with selected/priority key informants in Maputo  11 Oct., Tuesday AM: Team travels to Pemba on 8:05 AM LAM flight, arrives 10:35 AM  11 Oct., Tuesday PM: Begin meetings and site visits, to continue Tuesday PM – Friday AM  12 Oct., Wednesday AM: Midterm update with USAID/Mozambique by telephone  14 Oct., Friday: Team returns to Maputo on 13:40 PM LAM flight 8 10/17- 10/21  Team conducts preliminary analysis of information from trips to Pemba and Quelimane  Meetings with selected/priority key informants in Maputo  Team prepares PowerPoint Presentation of Preliminary Results for Exit Briefing  Exit Briefing and Presentation to USAID/Mozambique – Wednesday PM 19 Oct.  Expat Team members return to Washington, DC (depart Thursday 20 Oct. at 3:35 PM) 9 10/24- 10/28  Local Team members from Verde Azul return to Pemba and/or Quelimane to continue information gathering from primary sources, as needed  Team begins writing Draft Evaluation Report 10 10/31-11/4  Local Team members from Verde Azul complete all information gathering, returning to Pemba and/or Quelimane if needed  Team continues writing Draft Evaluation Report 11 11/7-11/11  Team continues writing Draft Evaluation Report 12 11/14- 11/18  Team continues writing Draft Evaluation Report 13 11/21- 11/25  Team completes Draft Evaluation Report  Draft Evaluation Report submitted to USAID/Mozambique by COB on Friday 25 Nov. 14 11/28-12/2  USAID/Mozambique reviews Draft Evaluation Report 15 12/5-12/9  USAID/Mozambique reviews Draft Evaluation Report – continued 16 12/12- 12/16  USAID/Mozambique reviews Draft Evaluation Report – continued  Comments on Draft Evaluation Report sent to ECODIT by COB on Friday 16 Dec. 17 12/19- 12/23  Evaluation Team addresses USAID comments on Draft Evaluation Report 18 12/26- 12/30  Evaluation Team addresses comments on USAID comments on Draft Evaluation Report – continued 19 1/2/17- 1/6/17  Evaluation Team addresses comments on USAID comments on Draft Evaluation Report – continued  Revised Final Draft Report submitted to USAID/Mozambique by COB on Friday 6 Jan. 20 1/9/17- 1/13/17  USAID/Mozambique reviews Revised Final Draft Report and sends comments to ECODIT by COB on Friday 13 Jan. 21 1/16/17- 1/20/17  Final comments from USAID/Mozambique addressed by Team and report finalized  Final Evaluation Report submitted to USAID/Mozambique by COB on Friday 20 Jan. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 73 Note: Evaluation Deliverables are highlighted in bold Annex A: Evaluation Questions and Sub-Questions Guide/Script Evaluation Question 1a: To what extent has CCAP been successful in assisting Pemba and Quelimane municipalities to incorporate climate change adaptation into their planning processes? 1) In general, how successful has CCAP been in assisting this municipality to incorporate climate change adaptation into their planning processes? 1) extremely 2) very 3) moderately 4) somewhat 5) not very 2) In your opinion, what are the three biggest successes? 1) _________________ 2) _________________ 3) _________________ 3) We have learned about a number of “tools” that CCAP has developed or promoted to assist the municipality with climate change planning, for example: ______ SIGIC – Integrated Disaster Information Management System (Sistema Integrado de Gestão de informação sobre Calamidades) ______ SIGIU – Integrated Urban Information Management System (Sistema Integrado de Gestão de informação Urbana) ______ Vulnerability Maps ______ Local Adaptation Plans (Planos Locais de Adaptação) – PLAs ______ Local Government Self-Assessment Tool (LGSAT) ______ Cadaster-linked Vulnerability and Mitigation Scoring ______ Any others? ______________________________ Please rank each of these tools, if you are familiar with them, on a scale of 1-10, where 10 means “excellent or most useful,” and 1 means “not very useful.” 4) Are there any other tools that you would recommend CCAP develop or promote to help this municipality in planning for climate change adaptation? 5) Besides the CCAP project, where do you find climate information that is useful in municipal planning and decision making? (for example, reports, consultants, the internet, media sources?) MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 74 6) How effective has CCAP been in helping this municipality to reduce economic risks of climate-related events through, for example, insurance plans and contingency funds for at-risk urban infrastructure? 1) extremely 2) very 3) moderately 4) somewhat 5) not very 7) How effective has CCAP been in helping this municipality to adopt new laws, policies, regulations, or standards addressing climate change adaptation? 1) extremely 2) very 3) moderately 4) somewhat 5) not very For example? _______________________________________________________________ Evaluation Question 1b: To what extent has CCAP been successful in helping the relevant stakeholders of municipalities to implement adaptation measures? (communities, civil societies, NGOs, and universities)? And Evaluation Question 2a: To what extent has CCAP increased climate resiliency of the most vulnerable populations of Pemba and Quelimane municipalities, including those living in the most vulnerable areas? 8) How effective has CCAP been in engaging communities in implementing adaptation measures? 1) extremely 2) very 3) moderately 4) somewhat 5) not very 9) 2) In your opinion, what are the three most important adaptation measures that have been implemented? 1) _________________ 2) _________________ 3) _________________ 10) How much have CCAP activities increased the climate resilience in the most vulnerable populations in this municipality? 1) very significantly MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 75 2) significantly 3) moderately 4) a little 5) very little if at all 11) What was the most important way in which CCAP activities increased resilience? 12) What are the three most important remaining needs for improving climate resilience in the most vulnerable areas in this municipality, in your view? 1) __________ 2) __________ 3) __________ Evaluation Question 2b. To what extent has CCAP incorporated gender considerations and youth into implementation of its activities? 13) How well has CCAP incorporated gender considerations in implementing its activities. 1) extremely well 2) very well 3) moderately 4) somewhat 5) not very well 14) List three CCAP activities that effectively incorporated gender considerations in implementation: 1) __________ 2) __________ 3) __________ 15) How effectively has CCAP incorporated youth in implementing its activities. 1) extremely well 2) very well 3) moderately 4) somewhat 5) not very well 16) List three CCAP activities aimed at young people or to which young people contributed significantly: 1) __________ 2) __________ 3) __________ MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 76 Evaluation Question 3a. To what extent is CCAP prepared to achieve the project’s objectives over the next 2.5 years? 17) To what degree is CCAP prepared and on target to achieve the project’s objectives over the next 2.5 years, in your opinion: 1) completely, 100% 2) significantly, 75% 3) moderately, 50% 4) partially, 25% 5) marginally, 10% 18) How successful has CCAP been in improving the provision of climate-resilient urban services by municipalities? 1) extremely 2) very 3) moderately 4) somewhat 5) not very 19) How successful has CCAP been in increasing the adoption of climate resilience measures by communities and civic and community organizations? 1) extremely 2) very 3) moderately 4) somewhat 5) not very 20) How successful has CCAP been in increasing capacity to use economic risk-management tools, such as insurance plans and contingency funds? 1) extremely 2) very 3) moderately 4) somewhat 5) not very 21) How successful has CCAP been in identifying and using existing opportunities within the scope of the project? 1) extremely 2) very 3) moderately 4) somewhat 5) not very 22) A good example of how CCAP identified and took advantage of an existing opportunity within the scope of the project is (please explain briefly): ____________________________________________________________________________ MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 77 Evaluation Question 3b. What are some challenges/ obstacles (related to staffing, finances, etc.), reported by project personnel and what is the project’s capacity to respond to those challenges? 23) The three biggest challenges/obstacles to achieving the project’s objectives are: 1) _______________ 2) _______________ 3) _______________ 24) How well has CCAP done in adaptively managing the challenges/obstacles encountered in project implementation? 1) extremely 2) very 3) moderately 4) somewhat 5) not very Evaluation Question 3c. What are some major implementation obstacles/challenges and opportunities (reported by the municipalities and other stakeholders) anticipated over the next 2.5 years of implementation? 25) The three biggest challenges facing this city in preparing for climate change are: 1) _______________ 2) _______________ 3) _______________ MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 78 Annex B: Detailed Work Schedule – Information-Gathering Phase in Mozambique Week Day/date Time/Activities 6 Sun. 2 October  10:45 AM: Expat team members arrive in Maputo from Washington, DC Mon. 3 October  9:00AM-12:00 PM: Work planning session with whole Evaluation Team  1:30-3:00 PM: In-Briefing with USAID/Mozambique to clarify expectations and launch field work  4:00-5:30 PM: Meet Chemonics CCAP staff in Maputo Office Tues. 4 October  12:30 PM: Team departs for Quelimane on LAM flight, arrive 14:30 PM Wed. 5 October  8:30-10:00 AM: Meet Quelimane-based Chemonics CCAP staff  11:00 AM-12:00 PM: Meeting to be arranged  2:00-3:00 PM: Meeting to be arranged  4:00-5:00 PM: Meeting to be arranged Thurs. 6 October  9:00-10:00 AM: Meeting to be arranged  11:00 AM-12:00 PM: Meeting to be arranged  2:00-3:00 PM: Meeting to be arranged  4:00-5:00 PM: Meeting to be arranged Fri. 7 October  9:00-10:00 AM: Meeting to be arranged  11:00 AM-12:00 PM: Meeting to be arranged  2:00-3:00 PM: Meeting to be arranged  19:40 PM Team returns to Maputo on LAM flight Sat. 8 October Weekend individual work or rest day Sun. 9 October Weekend individual work or rest day 7 Mon. 10 October (in Maputo)  9:00-10:00 AM: Meeting to be arranged  11:00 AM-12:00 PM: Meeting to be arranged  2:00-3:00 PM: Meeting to be arranged  4:00-5:00 PM: Meeting to be arranged Tues. 11 October  8:05 AM Team travels to Pemba, arrive 10:35 AM  Lunch  1:30-3:00 PM: Meet Pemba-based Chemonics CCAP staff  4:00-5:00 PM: Meeting to be arranged Wed. 12 October  9:00 AM: Midterm update with USAID/Mozambique by telephone (Team Leader only)  9:00-10:00 AM: Meeting to be arranged  11:00 AM-12:00 PM: Meeting to be arranged  2:00-3:00 PM: Meeting to be arranged  4:00-5:00 PM: Meeting to be arranged Thurs. 13 October  9:00-10:00 AM: Meeting to be arranged  11:00 AM-12:00 PM: Meeting to be arranged  2:00-3:00 PM: Meeting to be arranged  4:00-5:00 PM: Meeting to be arranged Fri. 14 October  9:00-10:00 AM: Team work session (or meeting to be arranged if necessary)  13:40 PM Team returns to Maputo on LAM flight Sat. 15 October Weekend individual work or rest day Sun. 16 October Weekend individual work or rest day 8 Mon. 17 October  All day: Team conducts preliminary analysis of information from trips to Pemba and Quelimane MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 79 Tues. 18 October  All day: Team prepares PowerPoint presentation of preliminary results for Exit Briefing Wed. 19 October  9:00 AM-1:00 PM: Team finalizes and practices Exit Briefing Presentation  2:00-3:30 PM: Exit Briefing and Presentation to USAID/Mozambique  4:00 PM: Whole Team debrief and planning session Thurs. 20 October  3:35 PM: US-based team members depart for Washington, DC Fri. 21 October  3:35 PM: US-based team members arrive Washington, DC MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 80 Annex E: Key Informants and Other Persons Contacted NAME OF INSTITUTION ACRONYM NAME TITLE EMAIL PHONE (+258) WASHINGTON, DC AND OTHER USA USAID Africa Bureau USAID-AFR Tegan Blaine Senior Climate Change Advisor tblaine@usaid.gov (202) 712- 0943 Chemonics Brian App Director, East and Southern Africa bapp@chemonics.com (202) 955.3358 Chemonics Carlos Quintela Former CCAP COP cquintela@chemonics.com Chemonics Debora Freitas Lead consultant, CCAP SCCB study dfreitas@chemonics.com United States Forest Service USFS Carl Trettin Team Leader, Southern Research Station ctrettin@fs.fed.us (843) 336- 5602 MAPUTO Coastal City Adaptation Project CCAP Olanda Bata Chief of Party mobata@ccap-mz.org CCAP Casimiro Antonio Deputy Chief of Party cantonio@ccap-mz.org 845595028 CCAP Gilberto Muai Monitoring and Evaluation Specialist gmuai@ccap-mz.org 82/84467485 0 CCAP Brant Paulson bpaulson@ccap-mz.org Africa Climate Change Resilience Alliance ACCRA/Save the Children Saide Anlaue ACCRA Consortium member saide.anlaue@savethechild ren.org 824548820 Universidade Eduardo Mondlane EMU António Queface EMU – Physics department Antonio.queface@gmail.co m 845181186 Universidade Eduardo Mondlane EMU Elidio Massuanganhe EMU – Geology department geomuzaza2000@yahoo.co m.br 827651840 Universidade Eduardo Mondlane EMU Celia Macamo EMU – Marine Biology celiamacamo@yahoo.com, 843983290 Ministry of Land, Environment and Rural Development MITADER Luis Buchir Head of National Directorate of Environment Management buchirmz@yahoo.com.br 827282890 Ministry of Land, Environment and Rural Development MITADER Celestino Salência Conselho Nacional de Desenvolvimento Sustentável (CONDES); csalencia@gmail.com 825874845 PEMBA Coastal City Adaptation Project CCAP Sharmila Moiane Senior Technical Advisor 824732300 Coastal City Adaptation Project CCAP Culsumo Carimo Community Engagement Advisor 825514070 Municipality of Pemba CMCP Zaide Abubacar Councilman for transport former councilman for climate zaideabubacar@yahoo.co m 843117653 MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 81 change and sanitation division Municipality of Pemba CMCP Abdulcarimo Fadile councilman for climate change and sanitation division abdulcarimofadile@yahoo. com.br 823842068 Municipality of Pemba CMCP Silvestre Macie Director Sanitation division maciesilvestre@gmail.com 823537750 Municipality of Pemba CMCP Abdulremane Chaca Head of international administration and information manager abdulchaca@gmail.com 845151689 Municipality of Pemba CMCP Marques Naba Council man for urban land management marques.naba@yahoo.com .br 843137671/ 827065000 Municipality of Pemba CMCP Rosário Agostinho Vereador de Infraestrutura e Obras Provincial Directorate of land, Environment and Rural development DEPTADER Izidine Opressa Director opressa@gmail.com 823025419 National Institute of Disaster Management INGC Elisete Manuel INGC provincial delegate elisetemanuel@hotmail.co m, elizetemanuel@gmail.com 844021834/ 824468200 Cariaco neighborhood Adelino Savelo Community leader 823898581 Paquitequete neighborhood Sofia Camal Community leader 822843874 Paquitequete neighborhood Salé Abudo Community leader Universidade Católica UCM Bianca Wamusse Director-Faculty of tourism and computer science bgerente@ucm.ac.mz 848270728 Universidade Católica UCM Nielete Oliveira Amado Lecture and Head of environment management course onielete@gmail.com 827099977 Universidade Católica UCM Miguel Nata Universidade Católica UCM Nilete Amado Universidade Católica UCM Dominique Niquele Universidade de Lúrio – UNILURIO UNILURIO Serafim Mucova Lecture and head of environment department smucova@fcn-unilurio.com 825953231/8 60215501 Universidade de Lúrio – UNILURIO UNILURIO Elídio Tomás da Silva Director Faculty of engenneering Elidio.silva@unilurio.ac.mz 828701839 Universidade de Lúrio – UNILURIO UNILURIO Santos Jemuce (a.i) Director faculty of natural science sjemuce@fcn-unilurio.com 828546002 Universidade de Lúrio – UNILURIO UNILURIO Daly Raposo Lecture UNILURIO dalyraposo@gmail.com 825530060 Universidade de Lúrio – UNILURIO UNILURIO Yelen da Rocha Lecture – environment management and community development Yelen.rocha@unilurio.ac.m z 845865866 QUELIMANE Coastal City Adaptation Project CCAP Ivete Fato Community Advisor Coastal City Adaptation Project CCAP Adalberto Moulinho Technical advisor MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 82 Associação dos Naturais e Amigos de Madal ANAMA Tomas Victorino Amissande Executive Secretary tamissande@inss.gov.mz 848906140/ 862356793 Eduardo Mondlane University – High School of Marine and Coastal Sciences EMU-ESCMC Noca Furaca Head of the Engineering and Marine Technology Department nocafuraca@yahoo.com.br 847103808/ 823195185 Eduardo Mondlane University – High School of Marine and Coastal Sciences EMU-ESCMC Fialho Nehma Deputy Director extension and research Fialho.nehama@uem.mz 848261151 Municipality of Quelimane Quelimane Antonio Olimpio Luis Agriculture technician Olimpioall78@gmail.com 842690619 Provincial Delegation – Land, Environment and Rural development DEPTADER Jose Manuel Dias Officer – environment officer dias.josemanuel@yahoo.co m.br 843854119 Provincial Delegation – National Institute of Disaster Management INGC Paulo Tomas INGC Head of technical department paulo.tomas2011@gmail.c om Quelimane Municipality CMCQ Almeida Colasso Quelimane Municipality CMCQ Joao de Brito Director Climate Change, sanitation and garden department britoara2002@yahoo.com. br 847531394/ 825681708 Quelimane Municipality CMCQ Jorge Fernandes Director – Health, Environment and social action fernandes.jorge8@gmail.co m Quelimane Municipality CMCQ Arlindo Rajabo Urban planning and land use Victor.rajabo2014@gmail.c om 826050274 Radio Zambeze FM Quelimane RZFM Timotio Castiano Journalist Radiozambezefm106.6@g mail.com 823112686 Provincial Directorate of land, Environment and Rural development DEPTADER Assane Provincial Directorate of land, Environment and Rural development DEPTADER Conjo Icidua neighborhood Fatima Guerra Community member 842272158 MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 83 Annex F: Evaluation Questions and Sub￾Questions Guide/Script KEY INFORMANT GROUPS AND INTERVIEW QUESTIONS Key Informant Type Subgroup Relevant Evaluation Questions Interview Guide Version Municipality Mayor 1a-b, 2a-b, 3a, 3c Interview guide #1 Councilmen/Staff 1a-b, 2a-b, 3a, 3c Community 1b, 2a-b, 3c Focal discussion Associations 1b, 2a-b, 3c group guide #2 INGC National 1a-b, 2a, 3a, 3c Interview guide #3 Provincial MITADER/DEPTADER National 1a-b, 2a, 3a, 3c Provincial University Eduardo Mondlane U. 1a-b, 2a, 3a, 3c UNILURIO Universidade Católica Media 1b, 2a-b, 3c Interview guide #3 modified for media CCAP Staff Maputo Office 1a-b, 2a-b, and 3a￾c Open contextual discussion Pemba and Quelimane Offices 1a-b, 2a-b, and 3a￾c Open contextual discussion MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 84 Interview Guide #1 – For Mayors and Municipal Staff 1) What is your role/area of expertise and how have you interacted with the CCAP? Qual é a seu papel ou área de trabalho? E como tem interagido com o projecto CCAP? 2) Have you used any of these tools in your work with the municipality? Tem utilizado alguma das ferramentas no seu trabalho com o município? (mark Yes, Y, if mentions tool) SIGIC – Integrated Disaster Information Management System Sistema Integrado de Gestão de informação sobre Calamidades SIGIU – Integrated Urban Information Management System Sistema Integrado de Gestão de informação Urbana Vulnerability Maps Mapas de Vulnerabilidade Local Adaptation Plans (Planos Locais de Adaptação) – PLAs Local Government Self-Assessment Tool (LGSAT) Ferramenta de Auto-avaliação do Governo Local Cadaster-linked Vulnerability and Mitigation Scoring Vulnerabilidade ligada ao Cadastro e à pontuação de mitigações Any others? Outros 3) What has been the importance of this tool? Qual tem sido a importância ou uso desta ferramenta? 4) What have been the challenges of using the tool? Quais tem sido as dificuldades no uso da ferramenta? 5) Will you continue to use this tool? Why? Irá continuar a trabalhar com a ferramenta, porquê? 6) How could it be improved? Como poderia melhorar a ferramenta? 7) Are there any other tools that you would recommend CCAP develop or promote to help this municipality in planning for climate change adaptation? Existem outras ferramentas que recomendaria a desenvolver e promover para ajudar o município no planeamento para a adaptação às alterações climáticas ao CCAP? 8) Has CCAP been helping this municipality to adopt new laws, policies, regulations, or standards addressing climate change adaptation? O CCAP tem ajudado o município a adotar novas leis, políticas, regulamentos, ou normas que abordem a adaptação às alterações climáticas? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 85 9) In your opinion, what are the three biggest successes? Na sua opinião, quais são os três grandes sucessos? 1. _____________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ 10) The three biggest challenges facing this city in preparing for climate change are: Os três maiores desafios / obstáculos para alcançar os objectivos do projecto são: 1. _____________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ 11) In general, how successful has CCAP been in assisting this municipality to incorporate climate change adaptation into their planning processes? Em geral, em que medida é que o CCAP tem sido bem-sucedido a ajudar os Municípios de Pemba e Quelimane a incorporar a adaptação às alterações climáticas em seus processos de planeamento? 1. extremely extremamente 2. very muito 3. moderately moderadamente 4. somewhat um pouco 5. not very Não muito 12) Which adaptation activities started under CCAP will the mucipality continue? If applies to the technical staff, ask about the community: 13) How has the CCAP been in engaging communities in implementing adaptation measures? Como é que o CCAP tem envolvido as comunidades na implementação de medidas de adaptação? 14) How much have CCAP activities increased the climate resilience in the most vulnerable populations in this municipality? Em que medida é que as actividades do CCAP aumentaram a resiliência climática nas populações mais vulneráveis neste município? 1. very significantly Muito significativamente 2. significantly significativamente 3. moderately moderadamente 4. a little pouco 5. very little if at all muito pouco, se de todo MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 86 15) What was the most important way in which CCAP activities increased resilience? Qual foi a maneira mais importante em que as atividades CCAP aumentaram a resiliência? 16) What are the three most important remaining needs for improving climate resilience in the most vulnerable areas in this municipality, in your view? Na sua opinião, quais são as três necessidades remanescentes mais importantes para melhorar a resiliência climática nas áreas mais vulneráveis neste município? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 87 Focal Discussion Group Guide #2– Focal Discussion Group For Community Leaders and Associations 1) Has CCAP been working with this community? (Yes/No) O CCAP tem trabalhado na sua comunidade? (Sim/Não) a. If yes, what are the objectives of the projecto? Se sim, quais são os objectivos do projecto? 2) What has CCAP been doing here? From these activities, which ones have you (community) developed/learned? Quais as actividades que o CCAP tem desenvolvido? Das actividades do projecto/iniciativa quais o que a comunidade tem vindo aprender a desenvolver? Actividades Participação da comunidade 1 2 3 3) What is the most important way CCAP has helped your community? Qual é a actividade mais importante que o CCAP realizou para ajudar a sua comunidade? 4) In the process of planting the mangrove, what have you (community) learned? No plantio do mangal, o que é que a comunidade aprendeu? 5) Will this activity have continuity after the end of the CCAP? Esta actividade/aprendizado vai continuar depois do CCAP ir-se embora? 6) What are the three most important remaining needs related to climate and natural disasters in this community? Quais são as três necessidades mais importantes da comunidade em termos de mudanças climáticas e medidas de adaptação? (o que vão fazer para proteger contra os ventos fortes, contras as chuvas, inundações, contra a seca …?) 1. _____________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ 7) Has CCAP worked with both women and men in this community (activity)? O CCAP tem trabalhado com homens e mulheres nesta comunidade? 8) Has CCAP worked with young people here? O CCAP tem trabalhado com os jovens? 9) In your opinion, what is right and what is wrong in the CCAP project? Na vossa opinião o que está mal e o que está bom na iniciativa/Projecto? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 88 Works well Need improvements 10) In your opinion, what can improve in the project? Se fosse o dono do projecto, o que iria melhorar ou fazer neste projecto na sua comunidade? 11) Would you like to add anything? Gostaria de acrescentar ou partilhar mais alguma coisa? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 89 Interview Guide #3 – for Universities, INGC, and DEPTADER 1) What is your role/area of expertise and how have you interacted with the CCAP? Qual é a seu papel ou área de trabalho? E como tem interagido com o projecto CCAP? 2) Have you used any of these tools in your work with the municipality? Tem utilizado alguma das ferramentas no seu trabalho com o município? (mark Yes, Y, if mentions tool) SIGIC – Integrated Disaster Information Management System Sistema Integrado de Gestão de informação sobre Calamidades SIGIU – Integrated Urban Information Management System Sistema Integrado de Gestão de informação Urbana Vulnerability Maps Mapas de Vulnerabilidade Local Adaptation Plans (Planos Locais de Adaptação) – PLAs Local Government Self-Assessment Tool (LGSAT) Ferramenta de Auto-avaliação do Governo Local Cadaster-linked Vulnerability and Mitigation Scoring Vulnerabilidade ligada ao Cadastro e à pontuação de mitigações Any others? Outros 3) What has been the importance of this tool? Qual tem sido a importância desta ferramenta? 4) What have been the challenges of using the tool? Quais tem sido as dificuldades no uso da ferramenta? 5) Do you think the tool will be used without the CCAP? Acha que continuarao a usar a ferramenta no seu trabalho? 6) How could it be improved? Como poderia melhorar a ferramenta? 7) Are there any other tools that you would recommend CCAP develop or promote to help this municipality in planning for climate change adaptation? Existem outras ferramentas que recomendaria a desenvolver e promover para ajudar o município no planeamento para a adaptação às alterações climáticas ao CCAP? 8) What changes would you recommend to CCAP if the project was to be replicated in other municipalities or another country? O que recomendaria de alterações ao CCAP para que este fosse aplicado noutros municípios ou noutro país? MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 90 Questionnaire #1 – For Possible Use with CCAP Staff (Weeks 9-10, To Be Decided) 1) What is your role in the CCAP and what activities have you developed? Qual é o seu papel e quais actividades tem desenvolvido no CCAP? 2) How successful has CCAP been in identifying and using existing opportunities within the scope of the project? Em que medida é que o CCAP tem sido bem-sucedido em identificar e utilizar as oportunidades existentes no âmbito do projecto? 1. extremely extremamente 2. very muito 3. moderately moderadamente 4. somewhat um pouco 5. not very Não muito 3) A good example of how CCAP identified and took advantage of an existing opportunity within the scope of the project is (please explain briefly): Dê um bom exemplo de como CCAP identificou e aproveitou uma oportunidade existente dentro do objectivo do projeto: ________________________________________________________________________ Gender and youth 4) List three CCAP activities aimed at young people or to which young people contributed significantly: Enumere três atividades CCAP dirigidas aos jovens ou às quais os jovens contribuíram significativamente 1. _____________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ 5) How effectively has CCAP incorporated youth in implementing its activities? Como efetivamente tem o CCAP incorporado os jovens na implementação das suas actividades? 1. extremely well extremamente bem 2. very well muito bem 3. moderately moderadamente 4. somewhat um pouco 5. not very well Não muito bem 6) List three CCAP activities that effectively incorporated gender considerations in implementation: Enumere três actividades do CCAP que efetivamente incorporaram as considerações de gênero na sua implementação: 1. _____________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 91 7) How well has CCAP incorporated gender considerations in implementing its activities? Em que medida é que o CCAP incorporou as considerações de género na implementação das actividades? 1. extremely well extremamente bem 2. very well muito bem 3. moderately moderadamente 4. somewhat um pouco 5. not very well Não muito bem 8) What have been the challenges and how have you overcome? Quais tem sido os desafios em incorporar o género e os jovens nas vossas actividades? Desafios Soluções Gender 1 2 3 Youth 1 2 3 9) How successful has CCAP been in improving the provision of climate-resilient urban services by municipalities? Em geral, em que medida é que o CCAP tem sido bem-sucedido na melhoria da prestação de serviços urbanos climáticos resilientes pelos municípios? 1. extremely extremamente 2. very muito 3. moderately moderadamente 4. somewhat um pouco 5. not very Não muito 10) How successful has CCAP been in increasing the adoption of climate resilience measures by communities and civic and community organizations? Em que medida é que o CCAP tem sido bem-sucedido em aumentar a adoção de medidas de resiliência climática pelas comunidades e organizações cívicas e comunitárias? 1. extremely extremamente 2. very muito 3. moderately moderadamente 4. somewhat um pouco 5. not very Não muito 11) How successful has CCAP been in increasing capacity to use economic risk-management tools, such as insurance plans and contingency funds? Em que medida é que o CCAP tem sido bem-sucedido no aumento da capacidade de usar ferramentas de gestão de riscos económicos, tais como planos de seguro e fundos de contingência? 1. extremely extremamente MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 92 2. very muito 3. moderately moderadamente 4. somewhat um pouco 5. not very Não muito 12) The three biggest challenges/obstacles to achieving the project’s objectives are: Os três maiores desafios / obstáculos para alcançar os objectivos do projecto são: 1. _____________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ 13) How well has CCAP done in adaptively managing the challenges/obstacles encountered in project implementation? Em que medida é que o CCAP tem feito uma gestão adaptativa os desafios / obstáculos encontrados na implementação do projeto? 1. extremely extremamente 2. very muito 3. moderately moderadamente 4. somewhat um pouco 5. not very Não muito 14) To what degree is CCAP prepared and on target to achieve the project’s objectives over the next 2.5 years, in your opinion: Na sua opinião, em que medida é CCAP está preparado para alcançar os objectivos do projecto ao longo dos próximos 2,5 anos: 1. completely, 100% completamente, 100% 2. significantly, 75% significativamente, 75% 3. moderately, 50% moderadamente, 50% 4. partially, 25% parcialmente, 25% 5. marginally, 10% marginalmente, 10% MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 93 Annex G: Lessons Learned in CCAP Mangrove Restoration The Quelimane mangrove restoration component of CCAP is a highly visible project activity and is touted as an example of successful community engagement. The emphasis on mangrove restoration that emerged in our interviews with key informants seemed to suggest that it was a relatively large component of the work there. However, we learned from the CCAP office that it accounts for only about 10% of the project’s budget for activities in Quelimane, a much smaller share than we would have judged based on the visibility and emphasis on this component on the ground. Five out of 11 CCAP quarterly reports have cover photos related to mangrove restoration issues, providing further evidence that CCAP sees this aspect of the project as noteworthy and visible, and in some ways given it disproportionate emphasis relative to the resources being expended on the activity. However, the Evaluation Team found serious design issues with the mangrove restoration component, and we recommend CCAP reassess and redesign the activity to ensure that it follows scientifically-based and internationally-accepted best practices – which it does not as it is currently implemented. Because of this “disconnect,” and although our finding is not a positive one, we believe that the CCAP mangrove restoration experience can provide some valuable “lessons learned” for the current activity, and for potential future USAID mangrove restoration initiatives in Mozambique, East Africa, and perhaps around the world. We therefore offer this more detailed analysis of the mangrove work, which is too lengthy to fit within the page limitations of the main Evaluation Report. HISTORY OF CCAP MANGROVE RESTORATION EFFORTS CCAP apparently followed the recommendation made in 2013 Mozambique Environmental Threats and Opportunities Assessment (USAID, 2013), which was to “Incorporate Mangrove Conservation and Restoration Into the CCAP Program.” In fact, the report made the even stronger recommendation that “We recommend that mangrove conservation and restoration be a much stronger component of the CCAP program than currently seems to be planned, and that one or more additional cities be chosen (e.g., Angoche) in which mangroves may provide the main infrastructure for coastal city protection.” However, it also cautioned that “Mangrove restoration is needed in many places, but the silvicultural science of how to restore each of the main species (there are nine species in Mozambique) in its proper intertidal zone is not complete. More pilot work on mangrove restoration needs to be done, and to be linked with CBNRM in coastal communities of fisher-farmers.” (USAID, 2013). Although CCAP did incorporate mangrove restoration as ecologically-based climate change adaptation measure, it apparently did not heed the warning about the need for pilot work on mangrove silviculture. A rapid assessment of mangrove areas in Quelimane conducted in October 2014 by CCAP (Garrido and Culsumo, 2014). The assessment noted that the clearing of mangroves and construction of dykes to create ponds for evaporative salt-making had disrupted the normal tidal flows in those areas of former mangroves, which in some areas were preventing the natural dispersal of mangrove seeds and droppers that would have allowed natural regeneration to occur. Nevertheless, the assessment documented some areas of natural regeneration in the abandoned salt-making ponds, and noted that, without the dykes, it had a high potential for natural regeneration. The assessment’s first recommendation was to remove the dykes from the old salt-making ponds in order to restore natural tidal flows that were preventing natural seed dispersal and regeneration. The assessment also mentioned that artificial reforestation of mangroves was a possible option. At about the same time, two years ago, Drs. Salomao Bandeira and Celia Macamo of UEM conducted a preliminary assessment of the mangrove situation in Quelimane, with Drs. Carl Trettin and Christina MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 94 Stringer of the US Forest Service. Their assessment found that a hydrological study of the area was needed, but that apparently was not done. Their assessment, like that of Garrido and Culsumo, suggested that restoration of natural tidal flows would induce natural regeneration. However, CCAP apparently wanted results immediately, so the project decided to plant mangrove seedlings to accelerate the process (Macamo, personal communication). It appears thatremoval of dykes, dams, and levees was not attempted (at least it is not mentioned in CCAP documents), but planting of mangrove seedlings began soon after the initial assessment was done, in April 2015. Planting was carried out in areas in the intertidal zones adjacent to the neighborhoods of Icidua and Mirazane, which were designated by the Municipality of Quelimane as “Areas of Environmental Conservation,” intended for “restoration and protection of mangroves.” Area of mangrove planting near Mirazane neighborhood, Municipality of Quelimane. Photo credit: B.Byers/ECODIT Seedlings of Aviccenia marina, the most common mangrove species in the intertidal zone close to Icidua and Mirazane, were grown in plastic tubes from seeds or droppers before planting in the tidal zone. At first these were produced for CCAP by ANAMA, the Association of the Inhabitants and Friends of Madal (Associação dos Naturais e Amigos da Madal), and more recently by the neighborhood of Icidua itself. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 95 Seedlings of Aviccenia marina grown in Icidua neighborhood. Photo credit: B.Byers/ECODIT For some reason that was never adequately explained to us, the communities dug furrows in the mud of the areas to be replanted, as if they were terrestrial crop fields – not a standard practice for mangrove restoration anywhere in the world. In fact, such a disruption of the natural surface of a mangrove further disrupts natural tidal flows and hydrology, likely hindering regeneration. CCAP field staff described this process of furrowing and planting as slow and expensive. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 96 Mangrove seedlings planted in furrows near Mirazane neighborhood; note some natural regeneration at top of photo. Photo credit: B.Byers/ECODIT The timeline of mangrove planting from CCAP quarterly reports shows the following: Quarter Area planted (ha) Cumulative area (ha) FY2015 Q3 (Apr-Jun 2015) 6.5 (p. 9) 6.5 (p.9); indicator #9 not reported FY2015 Q4 (Jul-Sep 2015) 0.5-1.5 (est. from cumulative) 7 (p, 10) or 8 (p.6) FY2016 Q1 (Oct-Dec 2015) None reported 7 or 8 (as in FY2015 Q4) FY2016 Q2 (Jan-Mar 2016) None reported No change in area of Indicator #9 reported since Q1 FY2016 Q3 (Apr-Jun 2016) 3 10 or 11 FY2016 Q4 (Jul-Oct 2016) 2 – No quarterly data reported, but FY2016 cumulative total of 5 ha given, so est. 2 ha for Q4 12 or 13 Recent assessments by scientists from Eduardo Mondlane University (Bandeira and Macamo, 2016) suggest need for modification of mangrove restoration approach and methods. The Executive Summary of their report, in Portuguese and English, is given at the end of this annex. We now understand that CCAP is in the process of awarding grants to NGOs in Quelimane to begin to implement the recommendations of the recent assessment. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 97 BEST PRACTICES FOR MANGROVE RESTORATION Information on science-based best practices for mangrove regeneration and restoration are easy to find through internet searches (Lewis, 2009; Lewis, 2010, Primavera, et al., 2012). They in general suggest the following steps:  Understanding the natural hydrological regime and the effects of human alterations of the regime as a first step in designing a mangrove regeneration activity;  Restoring the natural hydrology of the area to be restored (e.g., removing dykes, dams, levees, salt-making ponds, and “entulhamento”);  Establishing a hydrological monitoring program to document the effects of the activity;  Establishing a plan for protection and sustainable use of mangroves at the scale necessary to prevent “leakage” of mangrove loss from areas being restored to areas that have not yet been damaged; and  Planting of mangroves only if natural regeneration fails, selecting the appropriate species for each intertidal level. It is estimated that approximately 5,700 hectares of mangroves exist in the vicinity of Quelimane (Bandeira, 2016, personal communication). Google Earth views and measurements of this area show that approximately one-half of the original mangroves have been cut, cleared, or degraded – an area of at least 2,500 hectares. Given this context, it seems likely that the 22 hectares now designated by the Municipality of Quelimane for mangrove restoration is much too small an area to provide any significant restoration of the ecosystem services that mangroves provide, and which increase climate resilience. The ecological scale needed for a sustainable mangrove use and protection plan that would restore a significant fraction of degraded mangrove ecosystems and provide sustainable mangrove ecosystem products and services would likely be the entire delta of the Bons Sinais River in the vicinity of Quelimane for example. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 98 Google Earth view of intact and degraded mangroves near Quelimane, showing locations of Icidua and Mirazane. Image shows an area of approximately 12.8 km X 8.0 km. Icidua Mirazane MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 99 Residents of Mirazane, with Evaluation Team member Rui Mirira. Mirazane, like Icidua, is a neighborhood of Quelimane Municipality situated on land only a few meters above sea level, and therefore highly vulnerable to flooding from high tides and storm surges associated with tropical cyclones, and from rising sea level. Photo credit: B.Byers/ECODIT Dr. Roy “Robin” Lewis is an international expert on mangrove restoration. He provides the following steps for what he calls “Ecological Mangrove Restoration.” More information can be found at the website MangroveRestoration.com. “1. Understand the autecology (individual species ecology) of the mangrove species at the site, in particular the patterns of reproduction, propagule distribution, and successful seedling establishment. 2. Understand the normal hydrologic patterns that control the distribution and successful establishment and growth of targeted mangrove species. 3. Assess the modifications of the previous mangrove environment that occurred that currently prevent natural secondary succession. 4. Select appropriate mangrove restoration sites through application of Steps 1–3 above that are both likely to succeed in restoring a sustainable mangrove forest ecosystem, and are cost-effective given the available funds and manpower to carry out the projects, including adequate monitoring of their progress towards meeting quantitative goals established prior to restoration. This step includes resolving land ownership/use issues necessary for ensuring long-term access to and conservation of the site. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 100 5. Design the restoration program at appropriate sites selected in Step 4 above to initially restore the appropriate hydrology and utilize natural volunteer mangrove propagule recruitment for plant establishment. 6. Only utilize actual planting of propagules, collected seedlings, or cultivated seedlings after determining through Steps 1–5 above that natural recruitment will not provide the quantity of successfully established seedlings, rate of stabilization, or rate of growth of saplings established as quantitative goals for the restoration project. Step number 6 is still the most controversial step of EMR. If natural recruitment fails, that may mean the site has not been adequately rehabilitated to facilitate volunteer mangrove recruitment where propagule limitation does not exist. For example, if the hydrology has not been adequately restored, or at an excavated site, the final topographic grade may be too high or too low. Under these circumstances, planting will not overcome these physical limitations on plant establishment, but planting does often occur and the plants then die.” Based on his extensive worldwide experience, including on USAID mangrove restoration projects, Lewis wrote: “In the future mangrove restoration projects should be more carefully designed to ensure successful establishment of plant cover at minimal cost over large areas. This can be achieved for example by restoring hydrologic connections to impounded mangrove areas as has been done in Florida (Brockmeyer et al., 1997), Costa Rica, and the Philippines (Stevenson et al., 1999). Funding agencies typically fund mangrove restoration projects with minimal funds dedicated towards quantitative monitoring and reporting over a reasonable and ecologically based time period (5 years minimum). Both failures and successes thus go undocumented, and mistakes are repeated and lessons learned are lost. Funding agencies and governments need to realize that large amounts of limited restoration funds are now being wasted because of these shortsighted efforts, and at a minimum they should regularly review, publish, and teach the lessons learned from both past successes and failures. These same funding agencies and governments are very loath to fund careful examination of the ecological functions of restored mangrove areas.” (Lewis, 2009). MANGROVE HYDROLOGICAL MONITORING IN QUELIMANE A mangrove hydrological monitoring system was designed and installed in the CCAP mangrove restoration area in 2015with the help of Dr. Christina Stringer and Dr. Carl Trettin, mangrove and wetlands restoration experts from the US Forest Service. This effort was funded by USAID. In their report about this effort, Assessing the Effectiveness of Mangrove Restoration to Mediate Flooding on the Quelimane Site, USAID Coastal City Adaptation Project, Mozambique: Hydrology Monitoring Program Establishment Report, Stringer and Trettin summarize the objective of the monitoring program: “ A comprehensive monitoring plan is being designed and implemented by the Universidade Eduardo Mondlane- Quelimane (UEM-Q) to test the effectiveness of restoration activities. The US Forest Service designed a mangrove hydrologic monitoring strategy for CCAP that will be implemented by UEM-Q., The hydrologic monitoring focuses on evaluating trends in water table depth as restoration progresses and comparing the water levels in the restoration area to that in an undisturbed mangrove ecosystem. This will allow UEM-Q to evaluate whether restoration activities are contributing to returning the hydrologic regime comparable to the reference area.” The design is based on monitoring wells in which the water table, and salinity of the groundwater, are recorded. The hydrologic conditions in portions of the CCAP mangrove restoration area were supposed to be compared to conditions in an undisturbed mangrove reference area near the UEM￾Quelimane campus. The second part of the design is to monitor water level and salinity conditions in the residential areas of Icidua where flooding has previously occurred, and where residents complain about the health effects of salty drinking water. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 101 MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 102 Girl in Icidua drawing water from an open well; mangrove cover reduces saline intrusion into the ground water. Photo credit: B.Byers/ECODIT The Evaluation Team has tried to follow up on and evaluate the success of this monitoring effort. CCAP has assisted us to obtain a year of monitoring data from Dr. Fialho Nehma, the faculty member at the UEM School of Marine and Coastal Sciences who is in charge of the monitoring. We have facilitated the transfer of the data to Dr. Carl Trettin of the US Forest Service, who helped design the monitoring program. It appears there are still some problems with the data that complicate or prevent its analysis. We recommend that CCAP facilitate obtaining and analyzing these data, which would be very valuable in documenting any local effects of the small-scale mangrove restoration efforts supported by CCAP, such as effects on the salinity of wells in the low-lying community of Icidua. The monitoring data could also provide a valuable record of sea levels in the vicinity project activities in Quelimane.------- The Executive Summary from the recent rapid assessment of the CCAP mangrove restoration efforts in Quelimane is provided below, in the original Portuguese with English translation. The full citation of the report is: Bandeira, S. and C. Macamo. 2016. Avaliação Rápida do Programa de Restauro e de Práticas de Protecção da Floresta de Mangal e, Avaliação de Oportunidades para Meios de Subsistência (Livelihoods) para as Comunidades na Cidade Costeria de Quelimane. Universidade Eduardo Mondlane. Maputo, Setembro 2016. EXECUTIVE SUMMARY This document focuses on the project evaluation of mangrove forest restoration in the coastal city of Quelimane. The process covered different nurseries and restoration practices developed in Icidua, Mirazane and Madal; it made not only a detailed analysis of the activities in these places but proposed improvements, presenting specific recommendations. Este documento debruça-se sobre a avaliação do projecto de restauro da floresta de mangal na cidade costeira de Quelimane. O processo cobriu diferentes viveiros e práticas de restauro desenvolvidas em Icídua, Mirazane e Madal e fez uma análise detalhada das actividades nestes locais bem como proposta de melhorias, apresentando recomendações específicas. The need for mangrove restoration comes from profound changes to the ecosystem that significantly changed its own capacity of natural recovery [regeneration] and its ability to provide ecological functions. The restoration process may be passive or active. Even though positive aspects were identified, the evaluation found the following key facts: A necessidade de restauração de uma floresta de mangal surge em resultado de alterações profundas do ecosistema, alterando de forma significativa a sua capacidade de recuperação natural e de desempenhar as funções ecológicas. A restauração pode ser feita de forma passiva ou activa. Apesar de aspectos positivos identificados, a avaliação constatou as seguintes questões principais: MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 103 • The absence of previous studies recommending the need for hydrologic restoration, although it is known that the hydrological regime and topographical landscape has changed in many places, [affecting] mangrove restoration. • Consequently, the restoration of the mangrove using furrows/channels supported by an irrigation pump led to a low survival rate of planted mangrove seedlings. This method is not part of standard techniques for mangrove restoration in any country. • Awareness-raising activities for community in mangrove restoration is still at an embryonic stage or even nonexistent, except for the people directly involved in developing nurseries and planting seedlings. • The project is providing incentives to the people growing and planting the seedlings, but it should promote awareness of the need for mangrove restoration in the whole community, so people would participate in mangrove restoration campaigns for free or for a symbolic reward/payment. • The mangrove nurseries should all be installed in places with regular flooding (daily or similar), in order to achieve low levels of seedling mortality. • We note the absence of experts in mangrove ecology and the lack of knowledge that the restoration should attempt to restore [imitate almost entirely acceptable standards?] the natural distribution of mangroves.  Inexistência de estudos prévios que recomendassem a necessidade de restauração hidrológica, embora se saiba que o regime hidrológico e panorama topográficos tenha mudado em muitos dos locais para restauração com mangal.  Consequentemente o restauro de mangal conjugou a construção de sulcos com o auxílio da irrigação a base de motobomba conduzindo assim a taxas de sobrevivência do mangal replantado baixos. Este método não faz parte de técnicas padrão de restauro de mangal implementadas em vários Países.  As actividades de sensibilização as comunidades nos locais de restauro ainda é incipiente ou inexistente. Excepção às pessoas que estão directamente envolvidas no desenvolvimento do viveiro e replantio.  A prática de incentivar aos viveiristas é comum mas deve-se promover a sensibilização da sociedade para que esta, a custo zero ou simbólico, para o projecto, participe nas campanhas de restauração do mangal.  Os viveiros de mangal deverão todos ser instalados em locais com inundação regular (de forma diária ou similar); permitindo assim que se alcancem níveis mínimos de mortalidade em viveiros.  Constatamos a ausência de técnicos abalizados em ecologia de mangais e a falta de conhecimento de que o restauro deveria imitar quase na totalidade os padrões aceitáveis na distribuição de mangais. In terms of recommended good practices, the establishment of nurseries in the appropriate season coinciding with the availability of seeds stands out; shading nurseries to reduce direct sunlight at the time of high insolation; good plot organization to facilitate record-keeping; establishment of nurseries in the intertidal zone to allow irrigation with tidal waters- watering the seedlings only at high tides; and not using stagnant water from the canals for irrigation because it has a high salinity level. To consolidate the local knowledge we suggest the inclusion of topics related to mangroves in the current curriculum in local higher education institutions. Also recommended is the necessity of hydrological restoration that will re-establish natural hydrological flows and allow normal standard tidal circulation. This action aims at stimulation of natural regeneration of mangroves through recruitment, dispersal, establishment, and growth of new seedlings in degraded areas. MOZAMBIQUE COASTAL CITY ADAPTATION PROJECT MIDTERM EVALUATION 104 Em termos de boas práticas recomendadas destacam-se, o estabelecimento de viveiros coincidindo com época apropriada de disponibilidade das sementes; sombreamento dos viveiros para reduzir a incidência directa dos raios solares na época de insolação elevada; boa organização dos canteiros para facilitar as estatísticas; estabelecimento de viveiros na zona entre marés, para permitir rega com águas da maré; rega das plântulas do viveiro apenas nas marés enchentes; abster-se do uso da água estagnada nos canais para irrigação porque tem um alto índice de salinidade. Para consolidar o conhecimento localmente sugere-se a inclusão de tópicos ligados aos mangais nos curricula actuais em instituições de ensino superior locais. Recomenda-se também a necessidade da restauração hidrológica que visa restabelecer o fluxo hidrológico normal, permitindo a circulação normal das marés. Esta acção visa estimular a regeneração natural do mangal através de recrutamento, dispersão, estabelecimento e crescimento das novas plântulas na área degradada. In general, to improve mangrove restoration practices the project should stop using furrows and motorized irrigation pumps. It should give greater attention to the replanting campaigns as well as awareness-raising activities. Em geral, para a melhoria das práticas de reflorestamento do mangal recomenda-se o abandono da técnica de sulcos com o auxílio das motobombas. Deve-se dar maior atenção para que as campanhas de replantio seja acompanhadas com actividades de sensibilização. The report also contains information and guidelines about hydrological restoration and the development of seedling nurseries. O relatório contém também informação e textos didácticos e orientadores sobre a restauração hidrológica e o desenvolvimento de viveiros.