1 Resilience in the Limpopo Basin (RESILIM) Program Evaluation: Final Evaluation Report October 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Molly Hellmuth (evaluation team leader), Dr. Andrés Gómez, John van Mossel, Angela Wong, Maya Bruguera, Jamie Liu, Jessica Kyle, Joanne Potter, and Mark Wagner of ICF; and Zoe Parr, Dr. Sam Braid, Maimuna Ibraimo, and Batayani Gwangwawa, sub-consultants to ICF. 2 Cover Photo: A ranger surveys an escarpment in the Greater Limpopo Transfrontier Park. Credit: Gideon Mendel Source: Resilience in the Limpopo Basin (RESILIM) Program Final Report, Chemonics 2017 3 ABSTRACT The United States Agency for International Development (USAID) is conducting the Resilience in the Limpopo Basin (RESILIM) program to respond to challenges of transboundary water resource management, with the goal to improve the resilience of the Basin’s communities and ecosystems to better adapt to climate change impacts. The program is composed of two implementing mechanisms: RESILIM-B and RESILIM-O. ICF conducted a final evaluation of RESILIM using a theory-based, mixed method approach to analyze data from a desk study, key informant interviews, and questionnaires, with a synthesis rooted in triangulation. High-level recommendations for Resilient Waters and future USAID programming stem from the key findings of the evaluation. 4 TABLE OF CONTENTS Abstract ........................................................................................................................ 3 Executive Summary.................................................................................................... 6 Purpose and Background .............................................................................................................................. 6 Evaluation Questions...................................................................................................................................... 6 Methods............................................................................................................................................................. 7 Findings and Conclusions.............................................................................................................................. 7 Recommendations.......................................................................................................................................... 9 1 Introduction..................................................................................................... 11 1.1 About Resilience in the Limpopo Basin (RESILIM)........................................................................ 11 RESILIM-O ....................................................................................................................................... 12 RESILIM-B ........................................................................................................................................ 14 1.2 Evaluation Purpose and Scope............................................................................................................ 15 2 Evaluation Methodology ................................................................................ 16 2.1 Evaluation Design .................................................................................................................................. 16 2.2 Data Collection and Analysis.............................................................................................................. 16 2.3 Limitations............................................................................................................................................... 19 3 Evaluation Findings & Conclusions.............................................................. 20 3.1 Two Mechanism Approach: Scalability/Transferability ................................................................. 20 3.2 Cross-sectoral Resilience .................................................................................................................... 22 RESILIM-O ....................................................................................................................................... 23 RESILIM-B ........................................................................................................................................ 25 3.3 Partner Institutions............................................................................................................................... 27 3.4 Sustainability of Interventions............................................................................................................. 29 RESILIM-O ....................................................................................................................................... 30 RESILIM-B ........................................................................................................................................ 33 Grants and sub-grants................................................................................................................... 36 3.5 Social Inclusion....................................................................................................................................... 37 5 RESILIM-O ....................................................................................................................................... 37 RESILIM-B ........................................................................................................................................ 39 4 Recommendations for Future Programming............................................. 41 Annex I. Evaluation Statement of Work............................................................. 45 Annex II. Evaluation Matrix ................................................................................... 46 Annex III. Data Collection Instruments.............................................................. 49 Interview Protocols...................................................................................................................................... 49 USAID RESILIM, USAID, AND OTHER USG STAFF ................................................................... 50 CONSORTIUM OF IMPLEMENTING PARTNERS .............................................................................. 53 INTERGOVERNMENTAL INSTITUTIONS AND GOVERNMENT....................................................... 56 KEY STAKEHOLDERS (NGOS, CIVIL SOCIETY ORGANIZATIONS, ACADEMIC/RESEARCH, INTERNATIONAL COOPERATION PARTNERS) ............................................................................... 59 Questionnaire ................................................................................................................................................ 62 Annex IV. Sources of Information ....................................................................... 63 Documents Reviewed.................................................................................................................................. 63 RESILIM-B Documents.................................................................................................................. 63 RESILIM-O Documents................................................................................................................ 64 Informants Interviewed ............................................................................................................................... 64 USAID, Chemonics and AWARD Interviewees..................................................................... 64 RESILIM-B Interviewees................................................................................................................ 65 RESILIM-O Interviewees.............................................................................................................. 66 Questionnaire Results.................................................................................................................................. 67 RESILIM-B ........................................................................................................................................ 67 RESILIM-O ....................................................................................................................................... 68 Annex V. Evaluation Team .................................................................................... 69 6 EXECUTIVE SUMMARY Purpose and Background The Limpopo River Basin is considered to be a closed river basin, where water use exceeds the amount of renewable water available, for part or all of the year. Climate change is projected to stress water resources and exacerbate water quality and water supply deficits. Due to climate change, the Limpopo Basin may experience more frequent flooding and droughts, affecting livelihoods, agricultural productivity, and ecosystem health (Risk, Vulnerability, and Resilience in the Limpopo River Basin, OneWorld 2015). Other stressors, such as poaching and land conversion, can further put the region’s biodiversity at risk. The United States Agency for International Development (USAID) is conducting the Resilience in the Limpopo Basin (RESILIM) program to carry out transboundary water resource management in the Basin and improve the resilience of its communities and ecosystems to better adapt to climate change impacts. The program is composed of two implementing mechanisms: • The RESILIM-B (Basin) program, implemented by Chemonics International, ran from 2012 to 2017 and was carried out in all four Limpopo Basin countries: Botswana, Mozambique, South Africa, and Zimbabwe. • The RESILIM-O (Olifants) program began in 2012 and will run until March 2020, focusing on the Olifants Catchment, a sub-catchment of the Limpopo Basin, located in South Africa and Mozambique. The RESILIM-O cooperative agreement is carried out by the Association for Water and Rural Development (AWARD). ICF conducted a final evaluation of RESILIM from July 16, 2019 to October 25, 2019. The purpose of the evaluation was to determine the extent to which RESILIM met its objectives and to develop recommendations to inform future USAID work and programming. Evaluation Questions ICF evaluated both the RESILIM-B and RESILIM-O programs with the objective of answering the following questions: • How effective was the RESILIM-O program at creating scalable activities that are ready for Basin￾wide implementation? To what extent was the two-mechanism approach effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? • How has resilience been defined in each the RESILIM-O and RESILIM-B programs? To what extent has the cross-sectoral approach been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? • To what extent has the RESILIM program improved the effectiveness of the Limpopo Watercourse Commission (LIMCOM) and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? • How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? How could USAID/Southern Africa’s Resilient Waters Program be adapted to incorporate lessons learned under the RESILIM program? 7 • How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? Methods This program evaluation used a theory-based, mixed method approach grounded in an evaluation matrix, that aligned each evaluation question with a tailored evaluation design strategy in the form of sub-questions and specific data sources. The mixed-method approach analyzed the following data sources: • Desk Study. ICF reviewed key documents and outputs of RESILIM, as well as relevant documents from external partners and key stakeholders working on resilience in the Limpopo Basin. These included work plans, progress reports, final reports, results frameworks, theories of change, monitoring and evaluation plans and reports, knowledge products, a workshop report, questionnaires conducted post-training, and external literature. • Key Informant Interviews. ICF conducted interviews with 86 key internal and external informants, which served as a critical source of evaluative evidence. Internal informants included USAID, the implementers, and other partners, while external interviews included regional actors, non-governmental and civil society organizations, international cooperation partners, and other key stakeholders. • Questionnaire. ICF developed a questionnaire for grantees and non-implementing stakeholders to fill out that measured the effectiveness of the RESILIM program in carrying out program activities. The questionnaire contained six Likert scale questions about partner institutions, sustainability of interventions, and social inclusion. A total of 49 participants responded to the questionnaire. ICF used contribution analysis as our primary method of analysis. We gathered and analyzed the evidence to determine the extent to which RESILIM interventions contributed to the changes observed. In the final stage of the evaluation, ICF conducted a synthesis of data analyses rooted in triangulation, in which the team examined the evidence supporting each preliminary finding to determine which findings were confirmed by multiple sources or methods. Once key findings were confirmed, the team developed overall evaluation conclusions and recommendations that flowed logically from the findings. Findings and Conclusions Two Mechanism Approach: Scalability/Transferability RESILIM-O employed a systems approach to Olifants Catchment-level activities that may be replicable in the rest of the Limpopo Basin, but the degree to which the specific individual activities that RESILIM-O supported are replicable outside of the catchment varies. The “two-mechanism” approach created opportunity for transfer of scalable approaches from RESILIM-O to RESILIM-B; however, there is little evidence that transfer of approaches between the programs was formally implemented, limiting effectiveness of the approach in supporting the achievement of the RESILIM goals. Despite a lack of cross￾program transfer, ‘ad-hoc’ uptake and transfer did occur. Cross-sectoral Resilience 8 Across the programs, evidence suggests that cross-sectoral integration contributed to building resilience. However, evidence with which to identify or quantify specific, additive contributions of integration compared to single-sector implementation was lacking. RESILIM-O’s understanding of resilience contributed to a focus on improved practices, particularly within livelihoods and governance. RESILIM-O also recognized that achieving resilience is inherently a systems￾level challenge. As such, RESILIM-O interventions often demonstrated mutually reinforcing objectives, and stakeholders were aware of the linkages across sectors. While RESILIM-O’s ability to design clear and specific indicators with which to track progress toward resilience was limited, this challenge was partially overcome by the innovative Monitoring, Evaluation, Reporting and Learning (MERL) approach, which substantiated indicators through qualitative evidence gathering. There is evidence that the cross-sectoral approach enhanced program effectiveness, including building resilience. RESILIM-B applied a flexible definition of resilience that helped foster collaboration in a broad range of contexts. However, while the program’s output-focused indicators documented activity across sectors, they do not provide the evidence needed to measure cross-sectoral integration or ultimate impacts. Implementation of the Nexus approach under RESILIM-B created unique opportunities for cross-sector interaction across the Basin and resulted in some successful outcomes, but did not result in a cohesive set of program activities specifically geared towards achieving resilience. Partner Institutions While RESILIM-B was not well placed to resolve some of LIMCOM’s biggest obstacles and improve its effectiveness in key areas, stakeholders agreed that the effectiveness of LIMCOM was enhanced with RESILIM support. RESILIM-B was able to ultimately strengthen transboundary initiatives and increase the effectiveness of bi-lateral and multi-lateral players through the support of targeted activities. RESILIM-B also made efforts to engage Southern Africa Development Community (SADC), particularly to garner buy-in and support for program activities and outputs where SADC’s strategic guidance to the LIMCOM Secretariat was critical. Sustainability of Interventions RESILIM-O leveraged established and developed new relationships with partner organizations and vulnerable communities, built capacity through a bottom-up approach, and connected organizations and communities with decision-makers to develop platforms for engagement. These efforts, including the use of a social learning approach and real-time learning and adjustment through MERL outputs, supported the development of enabling conditions for sustainability. Some RESILIM-O activities and approaches will be sustained through new funding streams. In some cases, the long-term sustainability of activities is not clear without RESILIM-O’s presence. RESILIM-B leveraged established activities and grantee contributions to catalyze action and reduce dependency on the RESILIM program resources. Some RESILIM-B activities and approaches will be sustained through new funding streams. In some cases, leveraging existing activities and resources helped to enhance outcomes from ongoing sustainable interventions; in other cases, the long-term sustainability of activities is not clear without RESILIM’s support. Social Inclusion Each program addressed social inclusion to some extent. Data was collected to track the number of youth and women that participated in RESILIM programs, but there were few indicators designed to assess the level of effectiveness of this participation. RESILIM-O informants reported a high level of social inclusion, and high agreement about the effectiveness of this inclusion in planning and delivery. RESILIM-B informants 9 reported strong levels of inclusion, but less satisfaction with the degree of consideration of social inclusion in program planning and delivery. Recommendations The recommendations for Resilient Waters and future USAID programming, summarized below, stem from the findings of the evaluation. 1) Maximize learning and outcomes from RESILIM and incorporate into Resilient Waters and future programming in the region. USAID should consider continued contact with the RESILIM program’s two implementing partners in the short and medium term. There remains an opportunity to identify best practices from RESILIM, including program methods and innovations, and the underlying enabling conditions required for scale up and transfer of methods and activities. Best practices identified from the RESILIM program could be more systematically incorporated into Resilient Waters programming. 2) Intentionally incorporate program innovations into the theory of change and design of future programming. • To achieve cross-programmatic coordination and knowledge transfer, this objective should be explicitly defined within program design from the outset. For future USAID programming, coordination and knowledge transfer between concurrent and consecutive programs should be reflected in the statement of work, theory of change, resource distribution, and work planning. This is particularly critical for a “two-mechanism” approach. • Tailor design and implementation to local contexts and with the potential for scalability in mind. Interventions under Resilient Waters and future USAID programming should be designed and implemented with scaling in mind. This should entail identifying and extracting the conditions necessary for scaling up, and accounting for the different social, human, and political capacities in different locations within the region. • Adopt and apply a systems-based understanding of resilience that recognizes the complexity of resilience and the multiple and interactive components involved. USAID should encourage implementers to clearly define their program goals within the context of the ultimate goal of resilience, and to identify outcome objectives that will support increased resilience within a complex and interactive feedback system, addressing climate stressors, ecological factors, and development concerns. USAID should emphasize the importance of iterative learning as part of program design, such as adoption of a MERL approach, given the complexity of the resilience challenge and emerging best practices to address it. • Attention should be given to address social vulnerabilities and capacities. Future USAID programming and Resilient Waters must intentionally recognize and accommodate different groups’ vulnerabilities, needs, capacities, and interests through attention to social inclusion approaches. 3) Coordinate and co-develop with partners and stakeholders on transboundary initiatives in order to foster ownership. • Design and implement interventions that enhance cross-sectoral stakeholder collaboration and interaction in the region. Resilient Waters and future programs in the region should work to maintain existing collaborative networks created through RESILIM and increase efforts to further improve cross-sectoral coordination and communication. Collaboration across multiple stakeholders should be explicitly included in the activity’s 10 implementation plan with specifically designed, implemented, and monitored activities, resources, and indicators. • Identify and engage a broad range of partners and stakeholders through a systems￾based approach. Future USAID programming should embed institutional and individual capacity building into the fundamental structure of the program, through the program’s theory of change and activity design. During activity design, Resilient Waters and future USAID programs should identify key organizations and champions within the organizations who need to be engaged and who should be the focus of the program’s capacity building in order to ensure that program outcomes persist and continue to strengthen past the lifetime of the program. • Co-develop programming with regional partners in order to increase their effectiveness. Resilient Waters, which is currently working with LIMCOM and OKACOM, should co-develop programming and align activities with needs and ongoing activities to strengthen transboundary initiatives. • Recognize that significant effort is required to reach stakeholders in Zimbabwe and Mozambique to provide targeted support for enhanced livelihood, water resource, and ecosystem resilience. Outreach through established partners, through local media, are a necessary first step in identifying potential partners with capacities and experiences that are aligned with future programming. 4) Implement, monitor, evaluate, and iteratively learn in order to measure and promote sustainable, transformational change. • Adopt the iterative learning approach from the MERL framework to work towards enhanced outcomes. Under Resilient Waters, intentional monitoring and learning from new activities can better position the program to assess scaling potential and to actively engage partners to promote scaling. • Explicitly incorporate cross-sectoral integration in MERL to better identify opportunities. Under future USAID programming, include cross-sectoral integration in the MERL plan to help build the evidence around the contributions of integrated approaches to USAID’s investments. • Develop overarching guiding principles for resilience M&E. USAID should consider developing a set of overarching guiding principles for resilience M&E, including identifying a preliminary results framework for resilience M&E, and defining and testing a set of resilience indicators. • Link indicators of Gender Equity and Social Inclusion to monitoring of direct outcomes, impacts, or behavioral changes (where applicable). This emphasis would extend to grant and sub-grant recipients, to demonstrate meaningful GESI activities and ways to measure them. 11 1 INTRODUCTION 1.1 About Resilience in the Limpopo Basin (RESILIM) Due to climate change, Southern Africa may experience in the future more frequent floods, droughts, wildfires, and changes in agricultural productivity and ecosystem ranges. The Limpopo River Basin is particularly risk-prone and climate change is projected to stress water resources and exacerbate the water quality and quantity issues that are already occurring in the area. Other hazards, such as poaching and land conversion, further put the region’s biodiversity and communities at risk. The Limpopo River Basin spans Botswana, Mozambique, South Africa, and Zimbabwe (Figure 1), and includes the Olifants Catchment which is located primarily in the north of South Africa and flows northeast into Mozambique. The Olifants River is the largest contributor of flow to the Basin and is critical to its health. The United States Agency for International Development (USAID) designed the Resilience in the Limpopo Basin (RESILIM) program to respond to the challenge of transboundary water resource management in the Basin with the overall goal of improving the resilience of communities and ecosystems of the Limpopo Basin. The overall program had three objectives: 1. Reduce climate vulnerability by promoting the adoption of science-based adaptation strategies for integrated transboundary water resource management 2. Conserve biodiversity and sustainably manage high-priority ecosystems 3. Build the capacity of stakeholders to sustainably manage water and key ecosystems The RESILIM program was co-funded with biodiversity conservation (BID) and global climate change adaptation (GCC-AD) funds, and designed to support integrated biodiversity and adaptation interventions. Program activities are required to meet both the USAID Biodiversity Code and Adaptation criteria, incorporating an explicit biodiversity objective as well as increasing the resilience of people, places and livelihoods to climate change. The RESILIM program is composed of two implementing mechanisms, Resilience in the Limpopo Basin Program (RESILIM-B) and Resilience in the Limpopo Basin – Olifants (RESILIM-O). The two mechanisms were meant to be integrated such that activities ranging from the sub-catchment level to the regional level would be coordinated for successful scaling up across the Basin. The mechanisms are described below. 12 Figure 1. Limpopo River Basin, Olifants Catchment, and distribution of key informants interviewed for evaluation. RESILIM-O RESILIM-O was initiated in 2012 as a multi-year cooperative agreement to improve transboundary governance and management of the Olifants Catchment in the Limpopo River Basin. RESILIM-O uses science￾based strategies and social learning approaches to better understand the systemic causes of vulnerability. RESILIM￾O is implementing activities to promote new ways of thinking and acting to carry out integrated water and biodiversity management, and to develop an understanding of water security and vulnerability in the Olifants Catchment and that support adaptation strategies to build catchment-wide resilience. RESILIM￾O’s Theory of Change model is rooted in activities to support social learning (Figure 2). RESILIM-O Cooperative Agreement • Contract Number: AID-674-A-13- 00008 • Program Dates: December 1, 2012 – March 31, 2020 • Countries: Mozambique, South Africa • Agreement Value: $10.7 million • Prime Implementing Organization: The Association for Water and Rural Development 13 Figure 2. RESILIM-O Theory of Change (Source: RESILIM-O MERL Framework) Key Result Areas (KRAs) for RESILIM-O include: • KRA 1: To institutionalize systemic, collaborative planning and action for resilience of ecosystems and associated livelihoods through enhancing the capacity of stakeholders to sustainably manage natural resources of the Olifants Catchment under different scenarios. • KRA 2: To enhance long-term water security and protection by supporting collective action, informed adaptation strategies and practices and tenable Institutional arrangements for transboundary Integrated Water Resources Management. • KRA 3: To conserve biodiversity and sustainably-managed high-priority ecosystem conserved through supporting collective action, informed adaptation strategies and practices and tenable institutional arrangements. • KRA 4: To reduce vulnerability to climate change and other factors by supporting collective action, informed adaptation strategies and practices and tenable Institutional arrangements. • KRA 5: To facilitate the sharing of experiences and lessons within the Olifants Catchment and other basins. • KRA 6: To strengthen organizational learning, integration and coherency through continuous reflective and collaborative process. • KRA 7: To ensure good governance through developing and maintaining organizational capacity and effectiveness through tenable management systems and sub-contract management. 14 RESILIM-B RESILIM-B was a five-year program that aimed to reduce climate vulnerability in the Limpopo River Basin and enhance the resilience of its people and ecosystems by promoting adaptation strategies for integrated, transboundary management. RESILIM-B was designed to work with LIMCOM and other basin stakeholders across four countries. Additionally, RESILIM-B supported activities with local, bilateral and multilateral, and international partners, and developed a grants program to carry out a diverse set of initiatives, such as creating eco-friendly jobs, restoring degraded coastal habitats, and developing youth education. RESILIM-B’s causal model maps out outputs, outcomes, and intended impact (Figure 3). Figure 3. RESILIM-B Casual Model (Source: RESILIM-B Year Four Performance Monitoring Plan) Key Results Areas (KRAs) for RESILIM-B include: • KRA 1.1: Science, technology, and capacity for decision-making and development of climate change adaptation strategies improved. • KRA 1.2: Water conservation and water demand management responding to climate change improved. • KRA 1.3: Integration of climate change adaptation strategies into long-term management plans and policies increased. • KRA 2.1: Natural resource management practices that mitigate threats to biodiversity improved. RESILIM-B Contract • Contract Number: AID-674-A-12- 00006 • Program Dates: June 4, 2012 – December 31, 2017 • Countries: Botswana, Mozambique, South Africa, Zimbabwe • Agreement Value: $14.6 million • Prime Implementing Organization: Chemonics International • Subcontractors: Global Water Partnership South Africa (GWP-SA), OneWorld, Overseas Strategic Consulting 15 • KRA 2.2: Ecological integrity and resilience to climate change in key/priority conservation areas improved. • KRA 2.3: Integration of climate change adaptation and biodiversity conservation into basin water and resource management plans. • KRA 3.1: Capacity of all levels of government to formulate transboundary and integrated water, ecosystem management and climate change adaptation policies and strategies improved. • KRA 3.2: Knowledge and awareness of climate change impacts and adaptation measures increased. • KRA 3.3: Knowledge and awareness of integrated and sustainable water management strategies and practices increased. 1.2 Evaluation Purpose and Scope ICF conducted a final evaluation of USAID’s RESILIM program from July 16, 2019 to October 25, 2019. The purpose of the evaluation was to determine the extent to which RESILIM met its objectives and develop recommendations to inform future work and programming. ICF evaluated both the RESILIM-B and RESILIM-O programs with the objective of answering the following questions: • How effective was the RESILIM-O program at creating scalable activities that are ready for Basin￾wide implementation? To what extent was the two-mechanism approach effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? • How has resilience been defined in each the RESILIM-O and RESILIM-B programs? To what extent has the cross-sectoral approach been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? • To what extent has the RESILIM program improved the effectiveness of the Limpopo Watercourse Commission (LIMCOM) and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? • How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? How could USAID/Southern Africa’s Resilient Waters Program be adapted to incorporate lessons learned under the RESILIM program? • How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? 16 2 EVALUATION METHODOLOGY 2.1 Evaluation Design This program evaluation used a theory-based, mixed method approach grounded in an evaluation matrix, detailed in Annex II. Evaluation Matrix. The RESILIM-B and RESILIM-O theories of change served as the system-level theories, with more detailed, mid-level theories elaborated and tested for individual components and activities as needed. The mixed method approach included qualitative methods, such as key informant interviews and document review; and quantitative methods, including assessment of the program’s Monitoring and Evaluation Plan results and questionnaire responses. The evaluation relied on contribution analysis to test causality between the different levels of a program’s results chain, and analyze whether the interventions from RESILIM were plausible, causal factors in the changes observed, in combination with other factors and actors. As feasible and appropriate, we captured the contribution of process and design elements as they affected program outcomes and results. The evaluation matrix (Annex II. Evaluation Matrix) served as the framework for this evaluation. The evaluation matrix aligned each evaluation question with a tailored evaluation design strategy in the form of sub-questions and specific data sources. The development of the evaluation matrix was informed by an initial desk review and initial internal consultations with the USAID Purchase Order Contracting Officer’s Representative (POCOR) and RESILIM Contracting Officer’s Representative (COR), and with RESILIM implementers. 2.2 Data Collection and Analysis The project team collected qualitative and quantitative data to support the mixed method approach. The evaluation team employed a desk study, key informant interviews, and questionnaires for data collection. Desk Study. ICF reviewed key documents and outputs of RESILIM, as well as relevant documents from external partners and key stakeholders working on resilience in the Limpopo Basin (see Annex IV. Sources of Information: Documents Reviewed). For instance, ICF assessed the programs’ indicator data and results reported under each Monitoring and Evaluation Plan, using quantitative methods to the extent feasible. RESILIM documents reviewed included: work plans; progress reports and final reports; program results frameworks, theories of change, and associated monitoring and evaluation plans and reports; knowledge products; a workshop report; and responses to questionnaires conducted post training workshops. The ICF team also reviewed external literature to contextualize the RESILIM program. These documents included: sector management plans, such as national and/or basin-level climate change adaptation strategies and plans; documents from other resilience initiatives in the region (e.g., African Development Bank initiatives); and other Basin-related policies and regulations. Key Informant Interviews. ICF conducted interviews with key internal and external informants, which served as a critical source of evaluative evidence (Annex IV. Sources of Information: Informants Interviewed). Figure 1 shows the geographic distribution of informants and Figure 4 shows the breakdown of these informants by mechanism, country, and stakeholder group. A list of priority internal and external informants was developed by the POCOR and provided to the evaluation team. The evaluation team also independently identified additional stakeholders to address potential selection bias. 17 • Internal consultations. ICF consulted with key USAID officials and program staff to obtain valuable insights on substantive evaluation questions. ICF conducted initial consultations with the Evaluation POCOR and RESILIM COR on July 25, 2019 in order to identify priority areas of investigation for the evaluation, identify key external informants, and inform development of the evaluation’s Design Report. ICF also consulted with the RESILIM program teams (including Chemonics, AWARD, and consortium partners); USAID/Southern Africa Regional Environment, Education and Democracy (REED) staff; and other USAID and U.S. Government agency partners supporting the program. In addition to consultations at the beginning of the evaluation, ICF held follow-up consultations with key individuals later in the evaluation process, to triangulate and validate information gathered through other methods. • External interviews and focus group discussions with key informants. External informants included bilateral, regional, sub-basin- and basin-level actors (e.g., LIMCOM); program grantees; non￾governmental organizations (NGOs), civil society organizations, and academic and research organizations; international cooperation partners; and other key stakeholders involved in climate change adaptation in the Limpopo Basin and region. Many of these external stakeholders were interviewed during fieldwork from August 19 to August 30, 2019. ICF also conducted brief key informant interviews with a sub-sample of participants in capacity building and technical assistance activities to determine the extent of subsequent application of skills, as well as to identify barriers if the tool or method was not used. For interviews with key informants in the field, ICF developed interview protocols as part of the Field Work Plan to guide our interviews with key informants to ensure that parallel information was solicited. For all interviews, group consultations, and fieldwork, ICF’s Team Leader ensured that ethical standards were upheld, and that data collection was conducted with the free and informed consent of the key informants, and in a replicable manner. Interview notes were coded into Dedoose, a software platform that supports mixed methods research and facilitates the organization and analysis of content from interviews and other qualitative and quantitative data. 18 Figure 4. Summary of Key Informants. ICF conducted a total of 86 Key Informant Interviews; 47 participated in RESILIM-B and 35 participated in RESILIM-O (some participated in both). Top: Most of the key informants were based in South Africa; Mozambique had the fewest stakeholders. The “regional” stakeholders, which make up a fifth of interviews, include USAID, the implementing partners, and international cooperation partners. Bottom left: Stakeholders in RESILIM-B mostly included NGOs/civil society organizations (CSOs) and government representatives, followed by the implementers. Bottom right: For RESILIM-O, stakeholders were primarily government, followed by NGOs/CSOs and implementers. 45% 8% 14% 13% 20% Country Breakdown South Africa Mozambique Botswana Zimbabwe Regional 30% 24% 14% 8% 6% 10% 8% Stakeholder Groups: R-B NGOs/CSOs Government Implementers USAID International cooperation partners Private sector Regional 18% 49% 13% 10% 10% Stakeholder Groups: R-O NGOs/CSOs Government Implementers USAID International cooperation partners Private sector Regional 19 Questionnaires. ICF conducted a short questionnaire (Annex IV. Sources of Information: Questionnaire Results) with government, NGOs/CSOs, and international cooperation partners after the interviews to collect additional data for specific evaluation questions. A total of 49 participants responded to the questionnaire. Upon completion of data collection, ICF used contribution analysis as our primary method of analysis. We gathered and analyzed the evidence to determine whether the changes observed were due to the RESILIM interventions themselves, or whether changes observed were not due to the programmatic approach but to other factors (e.g., alternative explanations that could have realized the same outcomes). In the final stage of the evaluation, ICF conducted a synthesis of data analyses, rooted in triangulation. After most of the evaluative evidence was collected and analyzed, the process of triangulation was facilitated via a working group session with key members of the evaluation team. At the working group session, evaluation team members offered up preliminary findings for each evaluation question based on the evidence they collected and analyzed. The team examined the evidence supporting each preliminary finding to determine which findings were confirmed by multiple sources or methods. 2.3 Limitations The Evaluation Team is confident that it collected sufficient evidence to make strong and robust conclusions. However, certain limitations on the data should be borne in mind, including: • Social response bias. Most stakeholders expressed strong appreciation for support from the program. The Evaluation Team is confident that overall, most interviewees were frank and forthcoming. Nevertheless, it is also important to acknowledge that some were, at least partly, presenting material in a way that was consistent with their own institutional interests. Such bias is normal in qualitative research and is one reason why it is critical to interview a range of individuals and triangulate the results. • Insufficient data to evaluate outcomes and impacts. It was not possible to assess the ultimate contributions to long-term resilience and sustainability, as these effects will be evidenced over decades, beyond the implementation time frames of the programs. In order to evaluate the sustainability of interventions past the life of the programs, the evaluation therefore relies on data and qualitative information about outputs, and immediate and intermediate outcomes as reported, consistent with the theory of change for each program. • Material interest bias. Some RESILIM stakeholders are currently applying for funding from the Resilient Waters program. There is a presumed influence – at least to some extent – of this factor on their responses, given that these stakeholders would not want to jeopardize their opportunity of being awarded funds. • Limited questionnaire sample size. The questionnaire responses may not be representative given the small number of respondents. The results should not be interpreted alone but rather triangulated with other evidence. 20 3 EVALUATION FINDINGS & CONCLUSIONS 3.1 Two Mechanism Approach: Scalability/Transferability How effective was the RESILIM-O program at creating scalable activities that are ready for Basin-wide implementation? To what extent was the two-mechanism approach effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? RESILIM-O employed a systems approach to Olifants Catchment-level activities that may be replicable in the rest of the Limpopo Basin, but the degree to which the specific individual activities that RESILIM￾O supported are replicable outside of the catchment varies. The “two-mechanism” approach created opportunity for transfer of scalable approaches from RESILIM-O to RESILIM-B; however, there is little evidence that this transfer of approaches was formally implemented. This limited the effectiveness of the approach in supporting the achievement of the RESILIM goals. Despite a lack of planned cross￾program transfer, ‘ad-hoc’ uptake and transfer did occur. RESILIM-O employed a systems approach that is replicable; this approach was applied to develop a suite of program activities that were tailored to local context and need. Based on key informant interviews, RESILIM-O stakeholders believed that the overall systems approach is sound and can be applied to other regions, but that the individual activities that were supported may or may not be scalable outside the Olifants Catchment. RESILIM-O partners indicated that a transfer of the systems approach would be feasible, but the resulting activities may differ because the specific context is critical in determining and designing program activities. However, participants indicated that there are opportunities for scaling RESILIM-O supported activities across the Limpopo Basin particularly where stakeholders face a similar set of issues; RESILIM-O’s support of Communal Property Associations is one example (see box below). In fact, the Department of Environmental Affairs (DEA) has indicated interest in supporting AWARD to undertake on-the-ground scoping in other catchments facing similar problems, to co-develop plans and activities to support the implementation of the DEA’s environmental Expanded Public Works Programme ‘Working For’ suite of programs.1 Co-management of high biodiversity, strategic water source areas through Communal Property Associations In South Africa’s former Homeland areas, there are many land claims to redress the negative impacts of the former Lands Act. However, many of these lands fall within ecologically sensitive areas including water resource sensitive areas and biodiversity hotspots. RESILIM-O supported Communal Property Associations (CPAs) in a variety of ways--including support for improved co-management agreements, conservation-based entrepreneurship, governance, advocacy, resolution of legal issues, and conflict resolution. In addition, networking opportunities for CPAs in the Olifants Catchment were supported. Addressing disputes over rights to natural resources can improve conservation outcomes and reduce the vulnerability of local communities who depend on those resources. The successful support provided to CPAs within the Olifants Catchment could be replicated to other CPAs throughout the broader Limpopo Basin. Similarly, the support provided to the CPAs in South Africa could also be provided to land claimants in Zimbabwe, and other areas affected by displacement through previous colonial governance policies. 1 ‘Working for’ programs: https://www.environment.gov.za/projectsprogrammes/#workingfor 21 There is little evidence that the “two-mechanism” approach was formally implemented, limiting its effectiveness in supporting the achievement of the RESILIM goals. The transfer of scalable activities and approaches from RESILIM-O to RESILIM-B did not occur during the lifetime of the RESILIM-B program. While the two programs did coordinate to some extent, initial plans to hold monthly check-in calls did not come to fruition. Given the potential for transferring approaches, this was largely a missed opportunity. There were several reasons for this lack of scaling from RESILIM-O to RESILIM-B: • Lack of incentive: Given that coordination between programs was not explicit in either program’s contract/cooperative agreement, design, or theory of change, there was little incentive for collaboration. Related to this, there are limited program indicators focused on inter-program collaboration and scaling. Further, given the contract terms, RESILIM-B is legally obligated to deliver the program as designed. Without explicit integration into the programs, collaboration between program implementers was ‘left to interpretation’. Related to this, there was not an earmarked budget for inter-program collaboration/scaling of approaches and activities. • Program onset dates, timelines, and locations: Different start and end dates hindered collaboration; RESILIM-O effectively began 18 months after RESILIM-B and had an extension of the end date, and RESILIM-B was finished by the time RESILIM-O had experiences to share/scale. While RESILIM-O’s Objective 5 focuses on information sharing and scaling up outside of the Olifants Catchment, by the time useful information was available, RESILIM-B had closed. In addition, the geographic distance between program offices made collaboration slightly more challenging (‘out of sight, out of mind’). RESILIM-O is operating under a cooperative grant, where this is more latitude on the approach and deliverables. • Cultural differences: The work styles and institutional norms of the two implementors – one a small NGO and the other a large private sector organization – contributed to different institutional cultures and expectations that needed to be overcome for effective collaboration to occur. Both programs indicated that cultural differences between their organizations hampered collaboration. • Geographic, political complexities: Both programs noted the complexity associated with their programs’ geographic areas, and that to the challenge of achieving their program objectives at these scales afforded little opportunity for cross-program collaboration. For example, given the political and institutional challenges (particularly within Mozambique and Zimbabwe), RESILIM-B indicated that scaling could be a challenge. While RESILIM-O indicated that the large geographic scale, high vulnerability, and complexity of the Olifants Catchment, kept them focused on working within the catchment. Despite a lack of cross-program transfer, ‘ad-hoc’ uptake and transfer did occur between RESILIM-O and RESILIM-B and their respective program partners. The evaluation identified examples of uptake and transfer of methods from RESILIM-B and RESILIM-O activities to neighboring catchments, estuaries, biodiversity areas, and basins through their respective partners. These primarily included the uptake of processes and methodologies that support RESILIM objectives by government or international cooperation partners working on similar themes or working with RESILIM implementers (see text box, below). However, the formal transfer of practices from RESILIM-O to RESILIM-B was not the focus of scalability and transferability efforts. 22 Uptake and transfer of RESILIM methods Several examples of ‘ad-hoc’ transfer of RESILIM program approaches and methods were noted: • United Kingdom Department for International Development Climate Resilient Infrastructure Development Facility (CRIDF) are incorporating vulnerability mapping into their work with Okavango River Basin Water Commission (OKACOM). • Center for the Sustainable Development of Coastal Zones (CDS-ZC) are using mangrove nursery guidelines and techniques in Inhambane and Maputo Provinces. • Department of Environment, Forestry, and Fisheries (DEFF) are applying the Great Limpopo Trans Frontier Conservation Area (GLTFCA) Livelihoods strategy in Kwa Zulu Natal and Western Cape, South Africa. • The Water Security work, systems thinking, and social learning methods have influenced the Tsitsa Catchment, SanParks, the Inkomati-Usuthu Catchment Management Agency, and the Kingfisher Project in South Africa. When properly designed from the program outset, a “two-mechanism” approach offers an opportunity to maximize investment through transferring learning and best practices; but there are important trade-offs to keep in mind. The evidence above indicates that some approaches were adopted in an ad-hoc manner, attesting to their perceived value and transferability. Ideally, this transfer leads to enhanced resilience of people and ecosystems. However, the identified barriers to cross programmatic transfer provide a window into the potential real costs of implementing a “two-mechanism” approach: systematic knowledge transfer requires dedicated time and resources for programmatic coordination, capture of enabling conditions and lessons learned, identification of contextual synergies across programs, and investment in transfer and implementation of approaches. The trade-offs become sharper where funding and time are limited, given that dedicating resources to knowledge transfer and uptake competes directly with other program priorities. 3.2 Cross-sectoral Resilience How has resilience been defined in each the RESILIM-O and RESILIM-B programs? To what extent has the cross￾sectoral approach been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? Across the programs, evidence suggests that cross-sectoral integration contributed to building resilience. However, evidence with which to identify or quantify specific, additive contributions of integration compared to single-sector implementation was lacking. This evaluation considered several key factors to determine the extent to which the cross-sectoral Nexus approach was conceptualized and implemented to support resilient ecosystems and livelihoods in the face of climate change. These include the extent to which: • Resilience was defined, and incorporated the Nexus approach, • The Nexus approach was incorporated into the theory of change, and • Monitoring and evaluation systems were designed to measure resilience, and the related influence of the Nexus approach on resilience. 23 The programmatic approaches to defining resilience, incorporating the Nexus approach into program design, and measuring resilience are described for each program below. RESILIM-O RESILIM-O’s understanding of resilience contributed to a focus on improved practices, particularly livelihoods and governance. RESILIM-O also recognized that achieving resilience is inherently a systems￾level challenge. As such, RESILIM-O interventions often demonstrated mutually reinforcing objectives, and stakeholders were aware of the linkages across sectors. While RESILIM-O’s ability to design clear and specific indicators with which to track progress toward resilience was limited, this challenge was partially overcome by the innovative MERL approach, which substantiated indicators through qualitative evidence gathering. There is evidence that the cross-sectoral approach contributed to program effectiveness, including building resilience. RESILIM-O was intentional about developing a definition of resilience, holding discussions over several months to think through how resilience would be defined to guide the program’s work. RESILIM-O drew on the Stockholm Resilience Center’s seven principles of resilience;2 and focused on building the resilience of ecosystems through improved livelihoods and governance. The path to resilient ecosystems was largely conceptualized within activities through a “whole of systems lens,” focused on identifying the causal relationships and critical factors that could be addressed through changing practices and behavior. For example, communities identified vulnerabilities to the natural resources that underpin their livelihoods, and then identified activities to tackle these vulnerabilities. Communities could then select priorities for implementation by assessing the feasibility and near-term benefits of potential activities, and prioritizing the options considered to be “low-hanging fruit”. Stakeholder networks were effectively developed or leveraged to support activities, bringing together a broader range of sector viewpoints, and creating buy-in and awareness of the activities. Improving governance included the provision of support for Communal Property Associations (CPAs) to address disputes over rights to natural resources, helping to improve conservation outcomes and reduce the vulnerability of communities who depend on those resources. This conceptualization of resilience was embedded within the cross-sectoral approach, whereby the interventions often demonstrated mutually reinforcing objectives. For example, stakeholders working on biodiversity conservation interventions understood that their goals had implications for, and were influenced by, climate change and competing water uses and users (see text box, below). Other examples of progress towards cross-sectoral resilience include: subgrants for a turnaround process for management of municipal wastewater treatment plants, which are a major contributor to poor water quality in the Lower Olifants; inclusion of biodiversity and climate change into municipal plans; and cross-sector coordination for restoration of the Blyde catchment. Additionally, several stakeholders reported that their work with communities was aided by building on the understanding of the linkages between their immediate needs (e.g., water, food security, protection from extreme climate events) and other sector goals (e.g., biodiversity conservation). For example, support for ‘climate-smart agriculture’ provides adaptation, sustainable landscape, and food security benefits. 2 Biggs, R., Schlüter, M., & Schoon, M. (Eds.). (2015). Principles for Building Resilience: Sustaining Ecosystem Services in Social-Ecological Systems. Cambridge: Cambridge University Press. doi:10.1017/CBO9781316014240 24 Biodiversity-driven water security in the Olifants Catchment The upper reaches of the Olifants Catchment are heavily industrialized, the middle reaches are characterized by agriculture, and the lower reaches flow through the Kruger National Park – a critical biodiversity hotspot – before flowing across the international border into Mozambique. The ongoing drought has placed the system under extreme stress and threatened the job security of agricultural workers, as well as the survival of wildlife and ecotourism of the Kruger Park, with little or no flow reaching the Mozambique border. To address these issues, RESILIM-O developed the InWaRDS decision support system (integrating water quality and quantity monitoring and thresholds of potential concern) together with various stakeholders of the Catchment including the South African National Parks Authority (SANParks). InWaRDS modeled the seasonal environmental water requirements (EWR) of the river together with the gazette resource quality objectives, and licensed abstractions as an operating system for the River. Using InWaRDS, RESILIM-O demonstrated that water released from the De Hoop dam could secure the necessary critical EWR requirements needed for the Kruger Park without a detrimental impact to the De Hoop catchment. The National Department of Water and Sanitation supported this approach; regular releases have been made, resulting in water security of the Lower Olifants River. Water security has been driven by protection of biodiversity and associated eco-tourism in the Kruger Park; in this process, approximately 30,000 agricultural jobs have been secured. For environmental sustainability, SANParks requested that releases only be implemented as an emergency resolution of the drought rather than as an ongoing practice, as the releases were becoming the norm to secure water rather than regulating water abstraction in the system to secure the flow. RESILIM-O attempted to define and apply indicators across multiple dimensions of resilience. An illustrative set of resilience-related output indicators (Table 1) show that most program targets were met or surpassed, but by themselves do not assess the magnitude or likelihood of cross￾sectoral resilience. In at least one case, a RESILIM-O grantee (the Mahlathini Development Foundation) tracked resilience using several environmental and social factors, but other grantees did not use a similar framework. The RESILIM-O indicators help track progress in creating the enabling conditions necessary for cross-sectoral resilience, but by themselves, they do not directly demonstrate cross-sectoral integration, nor do they provide an indication of ultimate impact. The Monitoring, Evaluation, Reporting and Learning (MERL) approach, through its qualitative reporting process, captured some of the attributes of cross-sectoral resilience within activities, capturing intermediate outcomes including behavioral change in the face of climate shocks or stressors (see box above). RESILIM-O recognized that achieving resilience is inherently a systems-level challenge, that would be difficult to measure through indicators alone. In their report on their MERL approach, the program noted that resilience is the result of a complex, multi-faceted, and dynamic process, and that “…conventional linear-logic and indicator-based forms of evaluation are inadequate both for guiding these interventions and for determining their success or failure” (Monitoring, Evaluation, Reporting and Learning for the USAID RESILIM-O Program, Annex 2, updated 2017). The program developed a series of indicators largely focused on outcomes, as a part of their commitment to MERL, while recognizing the limitations of these measures. RESILIM-O also brought in the dimension of Reporting and Learning, which allowed for program iteration and adjustment based on early evidence of activity success and failure. The MERL approach, through qualitative reporting, sought to identify dimensions of behavior change and learning that better captured the enabling conditions and attributes of resilience across sectors. 25 Table 1. Illustrative cross-sectoral resilience-related indicators from RESILIM-O. # Indicator 2015- 2018 2018- 2019 (Q1-Q3) AWARD Number of institutions with improved capacity to address natural resource management (NRM) and biodiversity conservation issues as a result of US Government (USG) assistance Actual Target 399 381 -- 133 EG. 10.2-5 Number of laws, policies, or regulations that address biodiversity conservation and/or other environmental themes officially proposed, adopted, or implemented Actual Target 123 72 41 58 EG. 11-2 Number of institutions with improved capacity to assess or address climate change risks supported by USG assistance Actual Target 437 438 -- 167 EG. 11-3 Number of laws, policies, or regulations addressing climate change adaptation formally proposed, adopted, or implemented as supported by USG assistance Actual Target 96 59 39 42 RESILIM-B RESILIM-B applied a flexible definition of resilience that helped foster collaboration in a broad range of contexts. However, while the program’s output-focused indicators documented activity across sectors, they do not provide the evidence needed to measure cross-sectoral integration or ultimate impacts. Implementation of the Nexus approach under RESILIM-B created unique opportunities for cross-sector interaction across the Basin and resulted in some successful outcomes, but it did not result in a cohesive set of program activities specifically geared towards achieving resilience. RESILIM-B applied a flexible definition of resilience that helped foster collaboration in a broad range of contexts. Implementing partners judged that the use of a narrower definition would have been too prescriptive and rigid to support the broad range of stakeholder needs and varying contexts in the Basin and limit the ability of communities to address resilience in ways that would be most relevant and meaningful to them. By applying a broad definition of resilience, RESILIM-B’s stakeholders in different locations could then use and further refine the definition to guide their specific activities to align with their unique context and needs. For example, the South African DEFF developed their own programs and developed a specific definition of resilience tailored to these programs, after discussions with RESILIM-B. The output-focused indicators of RESILIM-B documented activity but do not provide the evidence needed to measure cross-sectoral integration or ultimate impacts. Examples of the indicators applied by RESILIM-B program are provided below (Table 2). The indicators are largely focused on resilience-related outputs, including institutional agreements, training, and capacity building. These examples show that the programs often met, and sometimes exceeded, their output targets. As with RESILIM-O, these output indicators demonstrate activity to develop the enabling conditions for cross￾sectoral integration, but they do not alone demonstrate cross-sectoral integration, nor do they provide an indication of outcomes or ultimate impact. 26 Table 2. Illustrative cross-sectoral resilience-related indicators from RESILIM-B. Related Sector # Indicator (RESILIM-B) Target Total % of target reached Climate change C1.2 Number of institutions with improved capacity to address climate change adaptation issues as a result of the RESILIM program assistance 40 58 145 C1.7 Number of climate change adaptation strategies approved or adopted by stakeholder groups as a result of RESILIM support 4 4 100 Biodiversity & conservation C2.2 Number of agreements, tools, and strategies promoting sustainable natural resource management and conservation that are officially proposed, adopted, or implemented as a result of RESILIM program support 26 22 85 Water & ecosystem management C3.1 Number of stakeholders with increased capacity to address issues related to water and ecosystem management as a result of RESILIM program assistance 510 527 103 Under RESILIM-B, the Nexus approach (water, biodiversity, and climate change adaptation) created unique opportunities for cross-sector interaction. Interventions often demonstrated mutually reinforcing objectives across sectors. Even within sectors, there is evidence of new opportunities for collaboration among different sector actors that were brought together by the program (e.g., sustainable forestry in South Africa), or strengthening of existing processes (such as LIMCOM Integrated Water Resources Management [IWRM] planning) by approaching them from a different angle (e.g., see text box, right). The Nexus approach did not always lead to a cohesive set of program activities specifically geared towards achieving resilience. In some notable activities supported by the program, cross￾sector resilience building activities were dynamic and multi-stakeholder, linking bottom up with top down (e.g., the work on livelihoods in GLTFCA). However, the emphasis on “stimulus” type funding meant that many of the cross-sector interventions were short-term (e.g., for the community-based natural resource management interventions, workshops, conferences, and learning events). Evidence suggested that these activities were broad in scope, atomized and not cohesive, except for a thematic focus on natural resources management. For example, a broad range of pilot activities were opportunistically supported from training to community-based natural resources management to strategy development, spanning from community to national to regional scales. RESILIM-B obtained broad geographic coverage through this process, but at best, there were loose linkages across sectors. “I think the Nexus approach is more of an enabler of IWRM than existing IWRM mandates and approaches, which are owned by the water sector.” – A key informant. 27 3.3 Partner Institutions To what extent has the RESILIM program improved the effectiveness of the Limpopo Watercourse Commission (LIMCOM) and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? While RESILIM-B was not well placed to resolve some of LIMCOM’s biggest obstacles and improve its effectiveness in key areas, stakeholders agreed that the effectiveness of LIMCOM was enhanced with RESILIM support. RESILIM-B was able to ultimately strengthen transboundary initiatives and increase the effectiveness of bi-lateral and multi-lateral players through the support of targeted activities. RESILIM-B also made efforts to engage Southern Africa Development Community (SADC), another important regional entity. RESILIM-B was not well placed to resolve some of LIMCOM’s biggest obstacles and improve its effectiveness as intended, given the lack of an operational secretariat and limited level of country buy-in. Evidence suggests that there was an initial over-estimation of the institutional and political situation of LIMCOM in its member countries in the RESILIM-B statement of work; therefore, initial ambitions to support the implementation of the strategic objectives of LIMCOM were not realistic. The structural and political challenges facing LIMCOM (intended to support a closed, highly politicized Basin) to deliver its mandate are sufficiently different to those of other river basin organizations in the region. For example, the past experience gained and lessons learned from supporting OKACOM (a relatively pristine basin), are not immediately transferable to LIMCOM. In particular, key informants in South Africa noted that the power dynamics between the South African government and users (mines, farmers, and ranchers) has intensified, and South Africa was not prepared to commit to changes in water management where they would have limited control. LIMCOM was not in a position to move quickly, nor to implement the 2011-2015 IWRM Plan. A functioning LIMCOM is a pre-requisite for developing a shared vision, coordination, and implementation. Based on stakeholder interviews, there was a perception that in the absence of leadership from a LIMCOM secretariat, RESILIM assumed a de-facto LIMCOM secretariat role, driving meetings and outputs. While this helped move some activities forward, it ultimately undercut the necessary buy-in and ownership by member countries that is necessary for long-term efforts to be sustained. For example, RESILIM struggled to gain buy-in from country-level actors for technical outputs, which delayed the completion of key technical support outputs, including the Limpopo Basin Disaster Risk Reduction Strategy, and the Basin Communication Strategy. This approach has raised concerns about the sustainability of this support, not only due to low ownership by LIMCOM, but also due to a lack of implementation resources: LIMCOM has an ‘unfunded mandate’ due to the lack of buy-in and financial support from the member countries. At the same time, despite these constraints to engaging LIMCOM and its member countries, based on the questionnaire results, there was recognition from the key informants that RESILIM contributed to strengthening collaboration among countries and regional institutions to manage the Limpopo Basin (as shown in Figure 5), as well as to strengthening the enabling environment that supports LIMCOM and other regional institutions that manage the Limpopo Basin. 28 Figure 5. RESILIM-B questionnaire results for partner institutions. RESILIM has contributed to strengthening collaboration among countries and regional institutions to manage the Limpopo Basin. (13 responses) RESILIM has contributed to strengthening the enabling environment that supports LIMCOM and other regional institutions that manage the Limpopo Basin. (12 responses) In recognition of LIMCOMs capacity constraints, RESILIM-B pivoted early on by broadening its focus, making some substantial gains through sustained support of bi-lateral activities (e.g., Botswana￾South Africa Joint Permanent Technical Commission (JPTC) Subcommittee on Water Quality) and increased emphasis on multi-lateral activities (e.g., GLTFCA), which ultimately strengthen transboundary initiatives. These activities were approached from a bi-lateral/multi-lateral perspective rather than from the perspective to achieve LIMCOM’s strategic objectives; however, objectives for transboundary coordination were aligned. RESILIM received endorsement from LIMCOM to support bilateral IWRM activities in the Basin, engaging the respective National Water Departments in Botswana and South Africa under the JPTC (see box below). Partnership with the Joint Management Board of the Great Limpopo Transfrontier Park (GLTP) and its partners resulted in the development of an Integrated Livelihoods Diversification Strategy for the GLTFCA. Several key informants remarked that given the programmatic focus on climate change and biodiversity, the GLTFCA may have been a more appropriate regional activity to support, rather than the LIMCOM (which is limited to water management). The JMP and partners also proved to be a less complex and more operational set of stakeholders, relative to LIMCOM, with whom gains could be made. 38% 54% 8% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 83% 17% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 29 Transboundary Initiatives through the JPTC Subcommittee on Water Quality RESILIM-B supported the Botswana-South Africa JPTC Subcommittee Joint Permanent Technical Commission on Water Quality and Water Hyacinth (JPTC), providing convening and technical assistance support. RESILIM-B supported a workshop “Strengthening Communications on Water Quality and Water Hyacinth Control,” that brought together delegates from both countries to build consensus on strategy goals, key messages, target audiences, communications vehicles, and monitoring and evaluation. A joint communications strategy expanded into self-sustaining, regular collaboration of the two governments on water quality monitoring and public reporting, data-sharing, and a first-ever agreement on managing transboundary groundwater in the Ramotswa Aquifer (RESILIM Program Final Report, Chemonics, 2017). Fostering this collaboration is critical to strengthening transboundary relationships, and the ecosystem health and water quality of the Limpopo Basin. Another important regional entity is the Southern Africa Development Community (SADC). Despite difficulties in developing a formal relationship, RESILIM-B made efforts to engage SADC with mixed success. One strategic area of SADC engagement was to enlist their support for program activities and outputs where SADC’s strategic guidance to the LIMCOM Secretariat was critical. For example, the implementation of the Disaster Risk Reduction Action Plan for the Limpopo River Basin requires coordinated interventions at the regional level, which would fall under the auspices of the Water Division of SADC. In addition, SADC’s new Groundwater Management Institute is aligned with the objectives of the RESILIM-B supported Ramotswa transboundary aquifer management initiative, providing opportunities for other aquifer studies in the Limpopo Basin. 3.4 Sustainability of Interventions How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? In some cases, the RESILIM programs helped to develop enabling conditions for sustainability and to enhance outcomes; in other cases, the long-term sustainability of activities is not clear in the absence of RESILIM’s support. The sustainable potential of USAID RESILIM varies by activity and outcome. While RESILIM-B has recently been completed, RESILIM-O remains ongoing and while there are signs of sustainable interventions, it is still premature to assess the sustainability of outcomes. This evaluation considered several key factors that indicate the extent to which sustainability may be achieved. These include the extent to which: • Enabling conditions are in place for potential sustainability of activities, • The theory of change for each program is aligned with long-term sustainability, and • Indicators used to measure activities and outputs suggest the activities are sustainable in the absence of RESILIM’s presence. As is normal for any USAID program, some activities are not expected to be sustainable. For example, without USAID RESILIM funding, the provision of expert consultants to support program activities will cease, as will the financial support for government officials or others to convene or to attend third-party trainings. Many other elements of the USAID RESILIM program have a potential to leave a legacy, however, as described in the sections below. 30 Further, the prospect of new resources through USAID Resilient Waters provides a concrete pathway for continued support of effective initiatives that have been supported by RESILIM. This USAID support represents one of several resources that can help continue promising approaches and build on the progress to date. RESILIM-O RESILIM-O leveraged established and developed new relationships with partner organizations and vulnerable communities, built capacity through a bottom-up approach, and connected organizations and communities with decision-makers to develop platforms for engagement. These efforts, including the use of a social learning approach and real-time learning and adjustment through MERL outputs, supported the development of enabling conditions for sustainability. Some RESILIM-O activities and approaches will be sustained through new funding streams. In some cases, the long-term sustainability of activities is not clear without RESILIM-O’s presence. Sustainability was an important concern for RESILIM-O stakeholders. The majority of key informants opined that it was critically important for them to implement activities that could be sustained, and that RESILIM-O worked to build conditions for sustained impacts (see questionnaire results below, Figure 6). Some stakeholders shared their hypotheses about the potential for sustainability of different types of interventions; these theories could be tested in the future to contribute to future programming and building the evidence base. For example, it was proposed that interventions with short-term quantifiable benefits are more likely to be sustained. Community engagement and buy-in may also contribute to sustainability, for example when RESILIM’s interventions resulted in capacity development (e.g., new agricultural practices). On the other hand, some stakeholders reported that they anticipated some initiatives to stop without RESILIM’s presence (e.g., community natural resource management committees). The RESILIM-O theory of change and social learning approach helped foster sustainability through a bottom-up capacity-building approach. The theory of change is based on building capacity through collaboration; supporting governance structures; building relationship, trust, and credibility; providing trainings; and co￾developing information. The social learning approach focuses on building sustainability into activities from the beginning, by promoting co-development and buy￾in from stakeholders. There are some positive signals that some RESILIM-O activities and approaches will be sustained (and scaled), through new funding streams. For example, the South African DEA is in the process of negotiating funding for AWARD to continue facilitating work in the Blyde Sub-Catchment. The DEA has determined that it is valuable to have a third party to bring government and other stakeholders together and has defined the important role that AWARD could play in undertaking on the ground scoping to develop a plan for other catchments. DEA would like to play a regulatory, not implementation role, and therefore seeks to engage AWARD to support planning and implementation. The sustainability-related indicators confirm that RESILIM-O undertook substantial capacity building and effected policy change. An illustrative set of sustainability-related indictors are shown in Table 3 below. Additionally, most interviewees agreed that their entity used or leveraged RESILIM interventions and results during the life of the program, and that RESILIM-O implemented effective “RESILIM-O focused on capacity development of the beneficiaries. This approach is the essence of sustainability. We helped communities learn how to make decisions, realize they have a say, and build their confidence. The project never made beneficiaries dependent.” – A key informant 31 strategies to increase the likelihood of the sustainability of its interventions. The MERL approach allowed for systematic learning across activities, capturing qualitative lessons and identify enabling conditions for sustainability. Figure 6. RESILIM-O questionnaire results for sustainability of interventions. Your entity has used or leveraged RESILIM’s interventions and results past the life of the program. (18 responses) RESILIM implemented effective strategies to increase the likelihood of sustainability of its interventions. (19 responses) Table 3. Sample of RESILIM-O sustainability of interventions indicators. # Indicator 2015- 2016 2016- 2017 2017- 2018 2018- 2019 (Q1-Q3) EG. 11-3 Number of laws, policies, regulations, or standards addressing climate change adaptation formally proposed, adopted, or implemented as supported by USG assistance Actual Proposed Adopted Target 3 N/A N/A -- 35 21 14 20 58 44 14 39 39 N/A N/A 42 (annual target) EG. 11-1 Number of people trained in climate change adaptation Actual Target 439 700 1,475 1,700 1,585 2,110 1,530 2,463 (annual target) Beyond these indicators and questionnaire findings, several qualitative evaluation findings illustrate aspects of RESILIM-O’s work that will likely affect its sustainability: 17% 56% 5% 11% 11% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 21% 53% 21% 5% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 32 • Communication, coordination, and relationships were understood as critically important to change behavior and to create outcomes from RESILIM-O initiatives and outputs. Stakeholders confirmed that AWARD set up networks and platforms that facilitated communication and coordination among groups and individuals with years of experience in key topic areas. These networks and platforms were seen as relevant in a context of land and water conflicts/profoundly unequal access to water, emerging crises from the impacts of climate change, long standing patterns of communities being disempowered/marginalized, and increasing vulnerability of the natural resources base in the Olifants Catchment. • The capacity of RESILIM-O participants has increased. The AWARD training initiatives have increased the capacity of the RESILIM-O participants in many institutions, groups, and communities. Confidence has grown particularly among participants that sought to be empowered to resolve conflicts and find solutions. The South African National Parks Authority (SANParks) felt that training provided through RESILIM-O was beneficial for the Kruger to Canyon (K2C) Biosphere Region. • Adaptive management is a key capacity that has increased. Adaptive management was the focus of much of AWARD’s training and learning by doing approach. This included engaging participants directly in taking a systems approach to problem or situation analysis, including planning climate change adaptation measures. Multi-sector problem analysis, multi-stakeholder participation and multi-intervention approaches were central to new and effective ways of working, building on the capacities and interests of stakeholders. Strengthening these capacities among stakeholders helps lay the foundation for sustainable program activities. • RESILIM-O built community knowledge and capacity, which is crucial for sustainability. The creation of beneficiary groups in marginalized communities was essential. This included, for example, communities in the national parks, water deficit communities and marginal agricultural areas, including communities that endure significant livelihood challenges. The focus with beneficiaries was on learning, on sustainable natural resources management practices, on adaptive management, and assessing outcomes and results of program interventions. • Some institutions are able to leverage additional funds to continue their activities. A few institutions are in the process of sourcing alternative funding or resources for activities/practices started with RESILIM-O support. This will increase the likelihood that some initiatives will continue past the lifetime of the program. Other partners, including AWARD sub￾grantees, have applied to Resilient Waters for funds to support activities started under RESILIM￾O that fit within the Resilient Waters framework, scope and focus. Despite these positive developments, a range of challenges are likely to threaten sustainability. These issues include: • Under-capacitated government agencies. Some agencies have little incentive to support initiatives or to help embed them. Interventions that involve these agencies may not be sustained without continued investment and maintenance. For example, some of the most useful interventions to stakeholders were developed in parallel to the Department of Water and Sanitation (DWS) systems (e.g., flow tracker and water quality monitoring), which cannot be sustained without continued investment and maintenance. Ideally, interventions and tools should be embedded within, and owned by the appropriate government agencies, to ensure long-term sustainability. • Corruption and power dynamics are a clear threat to sustainability. Several stakeholders commented on challenging operating environments for institutions such as the ones that 33 participated in RESILIM-O activities and mentioned that these concerns may create distinct challenges for continued work. In addition, the power-balance discrepancy between community members, different genders and races, and institutions, can lead to inaction. • The effectiveness of capacity building activities that serve as the basis of the RESILIM-O theory of change may be stunted by high turnover – resulting in loss of institutional knowledge and capacity. One contributing factor is a lack of sustained employment opportunities for staff with specialized skills. RESILIM-B RESILIM-B leveraged established activities and grantee contributions to catalyze action and reduce dependency on the RESILIM program resources. Some RESILIM-B activities and approaches will be sustained through new funding streams. In some cases, leveraging existing activities and resources helped to enhance outcomes from ongoing sustainable interventions; in other cases, the long-term sustainability of activities is not clear without RESILIM-B’s support. While it is still too early to assess the long-term sustainability of interventions, some RESILIM-B interventions have been sustained past the life of the program (e.g., support for the successful application to designate the Marico Biosphere Reserve under UNESCO’s Man and Biosphere Programme). The sustainability-related indicators confirm that RESILIM-B effected change; illustrative sustainability-related indicators are shown in Table 4 below. The efforts reflected in the indicators-- coupled with RESILIM-B’s work to provide catalytic support, develop cross-sectoral plans and policies, and improve transboundary management-- align with the RESILIM results framework and are intended to contribute to foster sustainability. A majority of questionnaire respondents indicated that they have used or leveraged RESILIM-B products and results past the life of the program, and that RESILIM-B implemented strategies to increase the likelihood of sustainability (Figure 7). Table 4. Sample of RESILIM-B sustainability of interventions indicators. # Indicator 2015- 2016 2012- 2018 C1.7 Number of climate change adaptation strategies approved or adopted by stakeholder groups as a result of RESILIM support Actual Target 0 1 2 4 C2.4 Number of hectares in areas of biological significance and/or natural resources showing improved biophysical conditions as a result of RESILIM program assistance Actual Target 0 40,252 19,748 60,000 C2.5 Number of management plans promoting integrated natural resources management officially proposed, adopted, or implemented for vulnerable hotspots as a result of project support Actual Target 1 2 2 5 34 Figure 7. RESILIM-B questionnaire results for sustainability of interventions. Your entity has used or leveraged RESILIM’s interventions and results past the life of the program. (13 responses) RESILIM implemented effective strategies to increase the likelihood of sustainability of its interventions. (16 responses) Beyond these indicators and questionnaire findings, several qualitative evaluation findings illustrate aspects of RESILIM-B’s work that will likely affect its sustainability. RESILIM-B funded multi-stakeholder initiatives, developed knowledge products and databases, and supported participatory learning experiences. These activities appear to have engaged parties and resulted in sustainable institutional linkages, generating longer-term plans for collaborative multi-level and multi-sector problem solving: • RESILIM-B supported multi-lateral and bi-lateral activities, including activities with the JPTC, GLTFCA, and in the Ramotswa Aquifer (see box on Ramotswa Aquifer, below), that successfully leveraged grantee contributions (Table 5), reducing dependency and enhancing sustainability. Work in the Ramotswa Aquifer is ongoing, and key informants indicated that the lessons learned and tools developed can be taken and applied to the other aquifers in the Basin. • Knowledge products and databases on the Limpopo Basin were developed with the support of RESILIM-B in ways that engaged specialized institutions within the Region and involved comprehensive consultations with stakeholders to elicit input. The development of products such as the Risk, Vulnerability and Resilience in the Limpopo River Basin or the Limpopo River Basin: Atlas of our Changing Environment involved the engagement of stakeholders across the Limpopo Basin over an extended period of time. Key knowledge generated by the programs included the Limpopo Information Management System and the Ramotswa Management Information System databases. • RESILIM-B selected effective and knowledgeable groups, institutions, and other entities to support and carry out activities. RESILIM-B provided targeted support to already existing initiatives, supporting strong individual leaders who were, and are, determined to succeed in their organizations’ endeavors. 62% 38% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 6% 62% 13% 19% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 35 RESILIM-B Support to Promote Sustainable Groundwater Management in the Ramotswa Aquifer RESILIM-B supported the hydrogeological and socio-economic assessment of the Ramotswa Transboundary Aquifer and development of an Information Management System, of which the process and products have the potential to generate sustained outcomes beyond the life of the project. The research that was funded engaged a range of relevant parties, from regional water officials (SADC), to the Departments of Water Affairs and Water and Sanitation from Botswana and South Africa, to community representatives of water users (e.g., farmers). SADC-GMI provided ongoing coordination for the Ramotswa Aquifer study, empowering the respective countries to apply the research in other aquifers within the Limpopo Basin. During the period of RESILIM’s support, SADC set up a new Groundwater Management Institute (SADC-GMI) to lead on similar initiatives in other aquifers across the Limpopo Basin. Table 5. RESILIM-B grantee and subcontractor contributions to activities in USD. (RESILIM Program Final Report, Chemonics 2017) Partner Year of contribution Activity RESILIM contribution Partner contribution SANBI (Africa Rising) 2015 Conference 1,597 436,845 CDS-ZC 2014/2016 Mangrove mapping 180,303 28,277 GWPSA 2016 Limpopo Atlas 95,876 17,245 GRID-Arendal 2016 Limpopo Atlas 19,025 Hwange GLTFCA 2014 Conference 12,006 14,000 IGRAC 2016 Ramotswa Info. Mgt. System 49,800 12,400 IWMI 2015 Ramotswa mapping, planning, data tools 420,563 78,390 Kwalata Community Development Initiative 2015 Youth Education 28,478 34,311 MRCA 2016 Training + biosphere reserve application 41,265 24,266 Peace Parks Foundation 2015/2016 Livelihoods – GLTFCA 155,443 173,894 South African Dept. of Environmental Affairs 2016 Workshop 0 3,458 SAWC 2014 Curriculum revision 135,808 53,122 Total 1,121,140 895,232 Despite these positive achievements, some challenges were identified to achieving sustainability: • Key informants involved in community-based natural resource management activities indicated that programs were not sustainable. For example, sustainability was compromised where: support was provided for a short duration (e.g., Kgetsi Ya Tsie [KYT] Women’s marula oil production project in Botswana), low levels of investment were made (e.g., Zimbabwe Humanitarian and Livelihoods Development Trust [ZHLDT]), activity planning was deficient (e.g., the Tati River clean-up campaign in Botswana), and follow on activities were not supported (e.g., implementation of Campfire’s diversified livelihood strategies in the buffer area of the GLTFCA). Key informant interviews indicated that RESILIM-B took an opportunistic and activity-driven approach that was motivated in part by the need to disseminate resources quickly and to spread resources across the Basin. 36 Grants and sub-grants How effective was the RESILIM grant approach to achieving RESILIM goals? As a subcomponent of sustainability of interventions, the evaluation team evaluated the effectiveness of the RESILIM-O sub-grants approach and RESILIM-B grants approach to achieving RESILIM goals. The RESILIM grants and sub-grants were awarded through several channels: broad open calls, direct calls, and sole source contracting. One common theme that emerged across the programs was that grantees and sub-grantees found the grant process to be overly onerous and bureaucratic, requiring significant time and resources to pull together paperwork, financial and management protocols, which in some instances delayed implementation. In addition, the prolonged back and forth with USAID, once the request for approval was submitted, led to delays in awarding the grants and sub-grants. While the same ‘boxes need to be ticked’ for grant and sub-grant awards regardless of their monetary size, larger awards require more investment in terms of time and evaluation. The process to access these resources was perceived by grantees and sub-grantees to be onerous relative to the award size. RESILIM-O RESILIM-O sub-grants have been instrumental for developing networks and capacity in the catchment. RESILIM-O took a focused approach to the sub-grants program to explicitly target and align awards with RESILIM-O’s key results areas. Nearly 14% of RESILIM-O’s total budget expenditures through 2017-2018 fiscal year was dedicated to sub-grants; through the 2017-2018 fiscal year, they had only expended about 54% of the total sub-grant budget allocation—just over half of the amount budgeted. This initial problem was caused by both staffing and administrative hurdles. At the outset, due to limited experience in managing USAID sub-grants, RESILIM-O had difficulty identifying a sub-grants manager, and developing sub-grants protocols and procedures. RESILIM-O indicated that the sub-grant process transferred potential financial risks from USAID to RESILIM-O, a responsibility that they had not anticipated at the outset. These difficulties ultimately added an unexpected administrative cost and burden and delayed the sub-grants program. However, once the processes were in place, the sub-grants have been instrumental for developing networks and capacity in the catchment, and for filling expertise gaps within the catchment via external experts. RESILIM-O took a focused approach to the sub-grants program, using sole source and direct sub-grants mechanisms to explicitly target and align with the key results areas. This type of targeted approach helped them fill expertise and knowledge gaps to support activities (e.g., climate smart agriculture); develop important technical outputs (e.g., wastewater treatment turnaround plans); and develop training, curricula, and capacity building materials. RESILIM-B The RESILIM-B grant program leveraged grantee contributions, to reduce program dependency and maximize grant funding. RESILIM-B was able to quickly spin up the grants program, given prior grant management experience. A key strength of the RESILIM-B grant program was the ability to leverage grantee contributions, to reduce program dependency and maximize grant funding. As noted above, reducing dependence on RESILIM resources can support future sustainability. 37 RESILIM-B had an overarching goal to distribute grant resources across the four Limpopo Basin countries, but it proved challenging to extend the grant financing reach into Zimbabwe and Mozambique. While Mozambique and Zimbabwe posed special challenges (administrative, financial) for grants, the lack of a RESILIM- B program office in these countries was also a contributing factor to the low number of grants within Mozambique and Zimbabwe. In addition, initial advertising efforts focused on advertising in the Mail and Guardian, which is only published in Botswana and South Africa. As a result, the majority of the grants were provided in Botswana and South Africa. Some of the grants supported activities in the action areas identified under Risk, Vulnerability, and Resilience (OneWorld 2015), but key informant interviews suggested that the grant process could have been better designed to target those areas. 3.5 Social Inclusion How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? Each program addressed social inclusion to some extent. Data was collected to track the number of youth and women that participated in RESILIM programs, but there were few indicators designed to assess the level of effectiveness of this participation. RESILIM-O informants reported a high level of social inclusion, and high agreement about the effectiveness of this inclusion in planning and delivery. RESILIM-B informants reported strong levels of inclusion, but less satisfaction with the degree of consideration of social inclusion in program planning and delivery. RESILIM-O RESILIM-O effectively integrated youth, gender sensitive, and social inclusion approaches, and were motivated by several factors (e.g., AWARD’s mission, South Africa requirements). Most questionnaire respondents agreed that RESILIM-O effectively engaged youth, gender, and socially marginalized groups in outreach and activities, and thoughtfully included these populations into programming and delivery (Figure 8). Many key informants indicated that “social inclusion” was clearly important for AWARD, that it was explicit in AWARD’s own organizational mission, and/or that it was a key aspect of effective implementation. In South Africa, there are strict requirements for employment and social inclusion around youth, gender, and disabled; these requirements were typically adhered to under RESILIM-O and frequently referenced by key informants. For example, the “Working for” programs must adhere to strict quotas (60:40 Female:Male, 2% Disabled). 38 Figure 8. RESILIM-O questionnaire results for social inclusion. RESILIM effectively engaged youth, gender (female and male), and socially marginalized groups in its outreach and activities. (17 responses) RESILIM thoughtfully incorporated youth, gender sensitivity, and social inclusion considerations in the planning and delivery of its activities. (16 responses) Youth and women were engaged in a variety of targeted ways, through education and training, as community representatives, and through other mechanisms. Efforts to empower youth were explicitly incorporated into the design of activities. For example, RESILIM-O deliberately developed a youth program that shadows the main CPA management forum (there are 12 members of the youth forum) to learn for the future and empower them. In addition, special attention was paid to issues that related specifically to women’s social burden – such as access to water and access to biofuels (firewood and charcoal). At the same time, there were limits to this support – primarily due to institutional characteristics (institutions or sectors (e.g., mining) dominated by male staff) that contributed to a practical approach of engagement and sometimes resulted in less representation of women and/or youth than desired. Integration of women and youth participation, in both targeted and general programming, is particularly recognizable in M&E reporting of grant activities and trainings (Table 6, Table 7). Table 6. RESILIM-O indicators for youth and women by activity. This table shows the number/percentage of youth and women engaged through the RESILIM-O program (Source: Summary of Institutional Capacity Development in RESILIM-O, AWARD 2019). Youth indicators – # of youth engaged Gender indicators – % women engaged Youth-targeted program # of youth Program % women Lekgalameetse Youth Forum K2C Environmental Monitors Blyde Restoration Champions Moletele Youth Project 12 13 8 8 Agriculture Support Initiative Moletele Youth Project Lekgalameetse Youth Forum K2C Environmental Monitors 65–70 88 50 41% 30% 29% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 44% 31% 19% 6% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 39 Youth indicators – # of youth engaged Gender indicators – % women engaged Youth-targeted program # of youth Program % women Youth in agro-ecology Internship program 24 19 Blyde Restoration Champions Mangrove Restoration Project Changing Practice course for CSOs CSO Indabas Internship program 50 50 82 65 50 84 Table 7. RESILIM-O indicators for number of women trained. This indicator shows that women were actively involved in trainings and activities for capacity building. However, the indicators only tracked participation by gender and were not designed to track level of support to and inclusion of socially marginalized populations. # Indicator (RESILIM-O) 2015- 2016 2016- 2017 2017- 2018 2018-2019 (Q1-Q3) EG. 11-1 Number of people trained in climate change adaptation Total % Women 439 49.8 1,475 63.9 1,585 63 1,530 60.8 RESILIM-O addressed social inclusion through its work with marginalized communities (e.g., CPAs), although there was less emphasis on disabled populations. Interviews with activity beneficiaries support the conclusion that RESILIM-O worked closely with marginalized communities, empowering them through knowledge, advocacy, and in representation in decision-making platforms to better understand their rights, a foundation for the fair and effective development of adaptive capacity and resilience. In general, RESILIM-O did not focus on disabled populations; however, the CDS mangrove restoration activity, it was remarked that people with disabilities were involved in the awareness process, but due to the physical nature of the activities, they were not brought into the restoration activities. There is some evidence that social inclusion enhanced the effectiveness of interventions. Based on key informant interviews, there was a sense that social inclusion was important, and some evidence that it enhanced the effectiveness of interventions (e.g., the division of labor in mangrove restoration activities). It is also possible that broader inclusion increased trust and/or willingness to participate. However, it is difficult to assess the direct impacts of inclusion on program effectiveness. RESILIM-B RESILIM-B had indicators in place to track women and youth participation in trainings and capacity building activities (Table 8) and many questionnaire respondents agreed that RESILIM-B engaged youth, gender, and socially marginalized groups (Figure 9). These data and feedback demonstrate that some progress was made in this area. For example, community-level activities often targeted women, youth, and poor communities mainly through Community-based Natural Resource Management interventions (e.g., KYT, Marico Biosphere Reserve, Campfire). However, RESILIM-B implementers indicated that social inclusion could have been better achieved in their programming. Vulnerability at the village level and household level is strongly determined by gender roles and capacity. In the Limpopo Basin, there is the added stress of male out 40 migration, leaving female-headed households to respond to climate change impacts. A Gender Equity and Social Inclusion (GESI) analysis was not taken up at the outset of the RESILIM-B program, included in the theory of change, nor was there an opportunity for key beneficiaries to define what GESI means for them and how to incorporate it into activities. Under the Resilient Waters program, a more in-depth approach using the GESI framework is being taken, through which the program has the opportunity to identify key barriers to inclusion, sensitize potential partners and beneficiaries, and develop outcome-focused strategies to address them. Figure 9. RESILIM-B questionnaire results for social inclusion. RESILIM effectively engaged youth, gender (female and male), and socially marginalized groups in its outreach and activities. (13 responses) RESILIM thoughtfully incorporated youth, gender sensitivity, and social inclusion considerations in the planning and delivery of its activities. (10 responses) Table 8. RESILIM-B social inclusion indicators for women and youth. These indicators show that women and youth were actively invited to trainings and activities for capacity building; however, the indicators were not designed to track support to socially marginalized populations. Indicator (RESILIM-B) Total Number of people with increased ability to adapt to climate change as a result of RESILIM support Total % Women 4,435 52 Number of youth with increased ability to adapt to climate change as a result of RESILIM support Total % Women ~2,000 >50 25% 42% 33% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 10% 30% 50% 10% Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree 41 4 RECOMMENDATIONS FOR FUTURE PROGRAMMING The following recommendations for Resilient Waters and future USAID programming stem from the findings of the evaluation. These recommendations touch on all phases of the Program Cycle, but most would be especially relevant in the earlier stages of a new activity, as they relate to leveraging learning, ensuring knowledge transfer, and address several points that can informing activity design. 1) Maximize learning and outcomes from RESILIM and incorporate into Resilient Waters and future programming in the region. • USAID should consider continued contact with the RESILIM program’s two implementing partners in the short and medium term to maximize learning. There remains an opportunity to identify best practices, including program methods and innovations, and the underlying enabling conditions required for scale up and transfer of methods and activities. To build on learnings and successes from the $25.3 million overall RESILIM program, USAID may wish to create, disseminate, and continually update a dedicated summary of potentially scalable interventions – including any available evidence for effectiveness – and critically important enabling conditions. This could include identifying additional potential locations, or key partners for future implementation. Best practices identified from the RESILIM program could be more systematically incorporated into Resilient Waters programming. 2) Intentionally incorporate program innovations into the theory of change and design of future programming. • To achieve cross-programmatic coordination and knowledge transfer, this objective should be explicitly defined within program design from the outset. For future USAID programming, coordination and knowledge transfer between concurrent and consecutive programs should be reflected in the statement of work, theory of change, resource distribution, and work planning. This is particularly critical for a “two-mechanism” approach. A coordinated approach should be facilitated by a common understanding of Basin and catchment level vulnerabilities, pathways to resilience that underlie the theory of change, and activity selection. The design should include realistic timelines to allow for learning and transfer of approaches; specifically, allow for enough time to test approaches, identify enabling conditions, identify potential opportunities (locations, communities, biodiversity areas) for scale-up/transfer, and implement scaling and transfer of approaches. Under Resilient Waters, a baseline understanding of vulnerability within the Limpopo Basin based on research from RESILIM, and a baseline understanding of vulnerability in the Okavango Basin based on a pre-design assessment of existing conditions, could help to inform the theory of change and pathways to resilience, and result in more targeted activities and transferable approaches. • Tailor design and implementation to local contexts and with the potential for scalability in mind. Design of a cross-catchment/cross-basin approach that spans ecological, socio-economic, and political contexts should balance a whole-of-activity approach to implementation (to avoid a fragmented set of interventions across the activity’s sites) with tailored interventions designed to be effective under each site’s unique set of conditions and context. Interventions under Resilient Waters and future USAID programming should be designed and implemented with scaling in mind. This should entail identifying and extracting the conditions necessary for scaling up, and accounting for the different social, human, and political capacities in different locations within the region (e.g., through a political economy analysis or a stakeholder analysis). 42 • Adopt and apply a systems-based understanding of resilience that recognizes the complexity of resilience and the multiple and interactive components involved. Promoting catchment and cross-basin resilience across sectors requires integrated planning and design. USAID should encourage implementers to clearly define their program goals within the context of the ultimate goal of resilience, and to identify outcome objectives that will support increased resilience within a complex and interactive feedback system, addressing climate stressors, ecological factors, and development concerns. USAID should emphasize the importance of iterative learning as part of program design, such as adoption of a MERL approach, given the complexity of the resilience challenge and emerging best practices to address it. When resilience is one of the goals of future programming, adopting a standard operational definition would provide a common vocabulary and scope from which to monitor and evaluate interventions. • Attention should be given to address social vulnerabilities and capacities. Impacts and vulnerabilities are felt differently across various social groups (e.g., men, women, youth, and the elderly). Future USAID programming and Resilient Waters must intentionally recognize and accommodate different groups’ vulnerabilities, needs, capacities, and interests through attention to social inclusion approaches. 3) Coordinate and co-develop with partners and stakeholders on transboundary initiatives in order to foster ownership. • Design and implement interventions that enhance cross-sectoral stakeholder collaboration and interaction in the region. Multiple stakeholders working in a complex, large-scale activity with multiple sectors and goals may default to siloed work unless collaboration and communication are carefully included in the theory of change and activity implementation. Resilient Waters and future programs in the region should work to maintain existing collaborative networks created through RESILIM and increase efforts to further improve cross-sectoral coordination and communication. Collaboration across multiple stakeholders should be explicitly included in the activity’s implementation plan with specifically designed, implemented, and monitored activities, resources, and indicators. Specific actions may include: • Design collaboration activities and define outcomes and benchmarks. • Develop a community of practice that undertakes activities such as: scheduling periodic communication and collaboration opportunities, potentially including conference calls or webinars open to all partners; convening events such as Pause and Reflect workshops or learning exchanges; producing and sharing newsletters, short videos, or other strategic communications products to share progress, success stories, and lessons learned. • Identify and engage a broad range of partners and stakeholders through a systems￾based approach. Particularly in a large Basin, multiple stakeholders are linked to outcomes; a system-based approach that leverages and co-develops information with existing organizations and stakeholders may contribute to enhanced sustainability. Future USAID programming should embed institutional and individual capacity building into the fundamental structure of the program, through the program’s theory of change and activity design. During activity design, Resilient Waters and future USAID programs should identify key organizations and champions within the organizations who need to be engaged and who should be the focus of the program’s capacity building in order to ensure that program outcomes persist and continue to strengthen past the lifetime of the program. For example, reinforcement of a traditional leader’s role as natural resource custodian helps to ensure natural resources and the community they support are better protected, while also garnering community buy-in. 43 • Co-develop programming with regional partners in order to increase their effectiveness. Program design and activities intended to increase effectiveness of regional entities such as LIMCOM should be coordinated and aligned with these entities’ objectives and priority needs. Resilient Waters, which is currently working with LIMCOM and OKACOM, should co￾develop programming and align activities with needs and ongoing activities (e.g., SADC 4th Water Action Plan) to strengthen transboundary initiatives. Sub-grant programs intended to increase effectiveness of regional entities should ensure buy-in from regional partners by engaging them in the design of open and targeted calls, so that they align with partner missions and are disseminated through a variety of mechanisms. • Recognize that significant effort is required to reach stakeholders in Zimbabwe and Mozambique to provide targeted support for enhanced livelihood, water resource, and ecosystem resilience. The lessons learned from RESILIM-B’s experience in Zimbabwe and Mozambique should be applied to developing a tailored approach for Resilient Waters and future USAID programming in these countries. Outreach through established partners, through local media, are a necessary first step in identifying potential partners with capacities and experiences that are aligned with future programming. It is important to engage these stakeholders early on, at the start of an initiative, in order to garner longer-term support that could result in sustainable outcomes. 4) Implement, monitor, evaluate, and iteratively learn in order to measure and promote sustainable, transformational change. • Adopt the iterative learning approach from the MERL framework to work towards enhanced outcomes. Under Resilient Waters, intentional monitoring and learning from new activities can better position the program to assess scaling potential and to actively engage partners to promote scaling. Implementation of the approach requires centering around reflection (e.g., regular monthly or bi-monthly reflection meetings), supported by visible valuing of the staff’s inputs into MERL processes from senior management. • Explicitly incorporate cross-sectoral integration in MERL to better identify opportunities. Under future USAID programming, include cross-sectoral integration in the MERL plan to help build the evidence around the contributions of integrated approaches to USAID’s investments.3 Resilient Waters can provide many unique opportunities to reinforce current USAID efforts to better understand the benefits of integration and best practices for integrated programs. • Develop overarching guiding principles for resilience M&E. Measurement of progress towards resilience is complex, and indicators and reporting can be insufficient and inconsistent among programs especially given the relatively short timeframe of projects. USAID should consider developing a set of overarching guiding principles for resilience M&E, including identifying a preliminary results framework for resilience M&E, and defining and testing a set of resilience indicators. These should include measurement of intermediate outcomes to assess changes in both positive and negative behaviors in the face of shocks.4 • Link indicators of Gender Equity and Social Inclusion to monitoring of direct outcomes, impacts, or behavioral changes (where applicable). This emphasis would extend 3 See for example USAID’s Biodiversity Results and Integrated Development Gains Enhanced Project (BRIDGE) contract. 4 Sturgess, P.; DFID. Measuring Resilience. Evidence on Demand, UK (2016) 51 pp. http://dx.doi.org/10.12774/eod_tg.may2016.sturgess2 44 to grant and sub-grant recipients, to demonstrate meaningful GESI activities and ways to measure them. 45 ANNEX I. EVALUATION STATEMENT OF WORK Issue Date: May 3, 2019 Deadline for Questions: May 13, 2019, 2:00 p.m. Pretoria time Closing Date: May 22, 2019, 2:00 p.m. Pretoria time Subject: Request for Quotations (RFQ) No. 72067419Q00001 Final Resilience in the Limpopo Basin (RESILIM) Program Evaluation To: Holders of Climate Integration Support Facility Blanket Purchase Agreements AID￾OAA-E-17-00007, AID-OAA-E-17-00008, AID-OAA-E-17-00009 and AID-OAA-E-17- 00010 The United States Agency for International Development (USAID) is seeking quotations for the final evaluation of USAID’s Resilience in the Limpopo Basin (RESILIM) Program, which is composed of two implementing mechanisms. The services are further described in the Statement of Work (SOW) included in Section A of this RFQ. This procurement will be conducted under the Climate Integration Support Facility (CISF) Blanket Purchase Agreement (BPA) pursuant to Federal Acquisition Regulation (FAR) 8.405- 3(c), 8.405-4, and 8.402(f). Only holders of the BPA are eligible to submit quotations. This RFQ contemplates the issuance of a firm-fixed price purchase order (PO), with an additional unpriced line item for open market items. Quotations must be received by USAID/Southern Africa no later than the time on the due date stated above and in the RFQ. Prior to the submission of quotations, any amendments to this RFQ will be provided to all BPA holders. It is the offeror’s responsibility to check email for official updates and amendments. This RFQ in no way obligates USAID to award a PO, nor does it commit USAID to pay any cost incurred in the preparation and submission of a quotation in response hereto. Furthermore, USAID reserves the right to reject any and all quotations, if such action is considered to be in the best interest of USAID. Offerors should read the entire RFQ, which includes all pertinent technical sections and the terms, conditions and instructions required for submitting a quotation. By email only, please submit all questions about this RFQ no later than the date stated above to proposals@usaid.gov and npiper@usaid.gov. Oral instructions, answers or guidance from any USAID source prior to the award of the contract shall not be binding. This RFQ and subsequent order is intended exclusively for authorized CISF BPA holders. Quotes from other than these vendors will not be considered. 72067419Q00001 2 Sincerely, Nathan Piper Regional Contracting Officer 72067419Q00001 3 SECTION A – STATEMENT OF WORK (SOW) A.1 PURPOSE The purpose of this purchase order is to conduct a final evaluation of USAID’s RESILIM Program, which is composed of two implementing mechanisms: Mechanism #1: First Activity Name: Resilience in the Limpopo Basin Program (RESILIM-B) Contract No: AID-674-A-12-00006 Project Dates: June 4, 2012-December 31, 2017 Agreement Value: $14.6 Million Places of Performance: Botswana, Mozambique, South Africa, Zimbabwe Prime Implementing Organization: Chemonics International Contracting Officer’s Representative: Jeanette Normand Mechanism #2: Second Activity Name: Resilience in the Limpopo Basin – Olifants (RESILIM-O) Agreement No: AID-674-A-13-00008 Project Dates: December 1, 2012-March 31, 2020 Agreement Value: $10.7 Million Places of Performance: South Africa, Mozambique Prime Implementing Organization: The Association for Water and Rural Development Agreement Officer’s Representative: Jeanette Normand A.2 BACKGROUND 72067419Q00001 4 Climate change is projected to have profound impacts on Southern Africa. Water scarcity is a growing concern throughout Southern Africa. Frequent floods and droughts, characterized by high spatial and temporal variability leave many of the region’s poor without access to adequate and safe water supplies. Declines in crop productivity, change in wildlife ranges, frequent wildfires, and the possible expansion of the malaria transmission zone are further consequences of climate change. These challenges are further exacerbated by high poverty levels and low adaptive capacity linked to a high dependence on ecosystem goods and services, weak governance, high levels of migrant labor, and an HIV pandemic. Trans-boundary cooperation and political commitment that can support in institutional, policy and legal reforms is lacking. The Limpopo River Basin is one of the most risk-prone areas in the region and climate change is expected to further stress water resources, in an area where maintaining water quantity and quality is already a challenge. In the Limpopo basin, the key threats to biodiversity in the region include poaching, land conversion for agriculture and other uses, competition for water and other natural resources, weak environmental governance, and unsustainable land and resource planning. USAID’s theory of change was that transitioning to sound, science based water management based on rationalized access to and utilization of water resources would both improve biodiversity conservation in the region and improve the region’s (and the basin’s) resilience to climate change. In 2012, the USAID designed the Resilience in the Limpopo Basin Program (RESILIM) to respond to the challenges of trans-boundary water resource management with the overall goal of improving the management of the Limpopo River Basin resulting in enhanced resiliency of people and ecosystems. RESILIM supports equitable access to water that balances urban and rural needs with ecosystem requirements under changing climate scenarios. RESILIM has three integrated objectives, namely: Objective 1: Reduce climate vulnerability by promoting the adoption of science-based adaptation strategies for integrated, trans-boundary water resource management. Objective 2: Conserve biodiversity and sustainably manage high-priority ecosystems. Objective 3: Build the capacity of stakeholders to sustainably manage water and ecosystem resources. The RESILIM program is being implemented through two coordinating mechanisms: the RESILIM-Basin (RESILIM-B) contract (which has expired), and the RESILIM-Olifants (RESILIM-O) cooperative agreement. The RESILIM program was funded with a combination of funds that were earmarked to USAID’s water and biodiversity earmarks, as well as adaptation funding from President Obama’s Global Climate Change initiative, so all activities under the two mechanisms had to support at least one of those directives in some way. 72067419Q00001 5 The RESILIM program was predicated on significant integration between the two mechanisms. For example, research and analysis at the regional level will feed into activities at the catchment level. Approaches tested at the sub-catchment level will inform policy and priorities at the regional level. Progress and outcomes will be shared at the regional level to determine strategies for successful scaling up and for influencing policy enabling environment. The coordinated, active participation and constructive engagement of national and sub-national stakeholders across the two awards is essential. The program uses participatory project management frameworks to ensure that decision-making, work-planning and monitoring are coordinated across the two mechanisms. A.2.1 THE RESILIM MECHANISMS A.2.1.a Resilience in the Limpopo Basin Program (RESILIM-B): RESILIM-B was a five-year, $14.6 million activity implemented by Chemonics International. The activity reduced climate vulnerability by promoting adaptation strategies for integrated, trans-boundary water resource management. By building the capacity of local river basin organizations and communities to improved trans-boundary governance and management of natural resources, high priority ecosystems/biodiversity were aimed to be be preserved and become more resilient to climate-induced pressure. The RESILIM-B activity, implemented through a contract managed by Chemonics International, focused on the Limpopo River Basin, which includes Botswana, South Africa, Mozambique, and Zimbabwe. RESILIM-B worked with national governments, the Limpopo Watercourse Commission (LIMCOM), other regional organizations, including the Great Limpopo Trans￾frontier Conservation Area (GLTFCA), and international organizations to improve governance of trans-boundary water and natural resources. Goal: The overall goal of the RESILIM-B activity was to improve the trans-boundary management of the Limpopo River Basin to enhance resilience of people and ecosystems to better adapt to climate change impacts. Objectives: Objective 1: Reduce climate vulnerability by promoting the adoption of science-based adaptation strategies for integrated, trans-boundary water resource management. Objective 2: Conserve biodiversity and sustainably manage high-priority ecosystems. Objective 3: Build the capacity of stakeholders to sustainably manage water and ecosystem resources. Intended Results: 72067419Q00001 6 • Climate vulnerability of the Limpopo river basin reduced • Conservation and management of ecosystems improved • Capacity of stakeholders to manage water and ecosystem resources improved Implemented Activities • Awareness creation, science-based diagnostics analyses, development of strategic action plans, study tours, workshops and seminars • Assessments to inform development of a bilateral South Africa-Botswana water quality and water hyacinth management plan • Basin-wide climate vulnerability assessment • Establishment of partnerships to restore mangrove ecosystems at the mouth of the river basin • Development of resiliency training and capacity building curriculum initiated for multi-sectoral participants • Updating disaster risk reduction strategy and action plan for droughts and floods in the river basin A.2.1.b RESILIM-OLIFANTS (RESILIM-O): RESILIM-Olifants is a $10.7 million activity implemented by the Association for Water and Rural Development (AWARD) to improve transboundary governance and management of the Olifants Catchment in the Limpopo River Basin in partnership with a consortium of local stakeholders. Initially scheduled to end in November 2017, the award was extended to March 2020. The activity reduces vulnerability of people and ecosystems through improved transboundary governance and management of natural resources. The activity is grounded in a grassroots approach to understanding the systemic causes of vulnerability, including climate vulnerability, and promoting new ways of thinking and acting to promote integrated water and biodiversity management. The RESILIM-O activity, pursuant to a cooperative agreement managed by the Association for Water and Rural Development (AWARD), operates in the Olifants sub-catchment, which includes South Africa and Mozambique. The Olifants River is the largest contributor of flow to the Basin and is critical to the health of the Basin. The assistance activity demonstrates integrated water, climate change adaptation and natural resource management strategies in biodiversity-rich areas at the sub-catchment level. Goal: 72067419Q00001 7 The overarching aim is to reduce vulnerability to environmental (climate) change through building improved transboundary water and biodiversity governance and management of the Olifants Basin through the adoption of science-based strategies that enhance the resilience of its people and ecosystems through systemic and social learning approaches. Specific Objectives: • To reduce climate vulnerability by promoting science-based adaptation strategies • To enhance water security and integrated water resources management • To conserve biodiversity and improve management of high priority ecosystems • To develop stakeholder capacities to manage water and ecosystem resources • To ensure continuous, reflective and collaborative learning • To facilitate exchanges across the Basin and with other Basins Implemented Activities: • Develop a collaborative, systemic understanding of water security and the role of water resources protection • Contribute to a systemic vulnerability assessment of the Olifants Catchment • Support adaptation strategies that build catchment-wide resilience Monitoring, Evaluation and Learning (MEL) activities have been undertaken during the duration of both awards. This includes monitoring of both USAID and custom indicators used to measure progress made towards the achievement of higher-level activity level outcomes. The MEL plans for both awards are attached (Attachment 1). In addition, during the period of June-July 2017, a learning event took place amongst stakeholders from the RESILIM program to discuss lessons learned from the past 5 years of program implementation. The final report can be found in Attachment 2. Contact information for the relevant stakeholders is to be provided post-award. A.3 EVALUATION QUESTIONS The final evaluation of the RESILIM program must answer the following evaluation questions: • The RESILIM program is implemented by two complementary mechanisms, RESILIM-O and RESILIM-B. Specifically, the RESILIM-O program was meant to demonstrate investments at a catchment level, whereas the RESILIM-B program is meant to work at a basin-wide scale. How effective was the RESILIM-O program at 72067419Q00001 8 creating scaleable activities that are ready for Basin-wide implementation? To what extent was the two-mechanism approach effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? • The RESILIM program used a cross-sectoral approach to building resilience by looking at water resources management, climate change adaptation, and biodiversity conservation. How has resilience been defined in each the RESILIM-O and RESILIM-B programs? To what extent has the cross sectoral approach been successful in building resilience in the Limpopo River Basin? How could cross￾sectoral integration have been improved? • To what extent has the RESILIM program improved the effectiveness of the Limpopo Watercourse Commission (LIMCOM) and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? • How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? How could USAID/Southern Africa’s Resilient Waters Program be adapted to incorporate lessons learned under the RESILIM program? • How effective was the RESILIM program in integrating youth gender sensitive and social inclusion approaches into implementation? How effective were those approaches? These questions may be refined during the evaluation design process in consultation with the Evaluation Contracting Officer’s Representative and USAID/Southern Africa Regional Environment, Education and Democracy (REED) staff. A.4 DESIGN AND METHODOLOGY While maintaining independence, the evaluation team is expected to carry out its work in a participatory manner seeking the views and assessments of all stakeholders. At minimum, the contractor must produce the deliverables described in Section B.3, including: 1. Evaluation Design Report (including the methodology; implementation plan; and travel plan) 2. Draft and Final Evaluation Reports 3. Briefing to USAID on the final evaluation with slides 72067419Q00001 9 The evaluation design must meet the requirements described in Section B.3, including with regard to: • Review of existing project documentation. • Interviews with relevant stakeholders (i.e, government counterparts from the Department of Water and Sanitation and Environment from each country, stakeholders from the Limpopo River Basin Commission (LIMCOM), relevant local communities and non-government organizations and program staff). • Approximately four Visits to partners and communities in all four countries (Mozambique, Zimbabwe, South Africa, and Botswana). • Analysis of pertinent reports, assessments, and laws/bills/regulations associated with the activities. The contractor must disaggregate project data by gender and other relevant categories. The contractor will have access to routine project data but may need to collect additional primary data in order to get the most objective evaluation possible. Team Composition: Offerors must propose a team structure and candidates to fulfill those positions, including no more than three (3) “Evaluation PO Key Personnel”. Evaluation PO Key Personnel are the individuals primarily responsible for the management of this PO and may be distinct from Key Personnel as defined in the BPA. The evaluation team must have a mix of expertise that covers the following capabilities, at minimum of 5 years of experience in monitoring and evaluation and learning and experience in at least one of the following technical areas; water resources management; climate change adaptation; biodiversity conservation; capacity building; institutional development; gender; and regional experience in the four Limpopo countries (South Africa, Mozambique, Botswana and Zimbabwe). A.5 EVALUATION SCHEDULE The contractor will be responsible for all off-shore and in-country logistical support, including international and in-country travel (including vehicle rentals), hotel bookings, working/office space, computers, printing and photocopying. The evaluation team, in collaboration with USAID/Southern Africa and the RESILIM implementing partners, will arrange all meetings, interviews, site visits, and in-briefing and out-briefing presentations. In all other respects, the evaluation team should be self-sufficient. [END OF SECTION A] 72067419Q00001 10 SECTION B – DELIVERIES OR PERFORMANCE B.1 PERIOD OF PERFORMANCE The period of performance shall be six months from the effective date of the signed contract. B.2 PLACE OF PERFORMANCE The Contractor shall perform the services described in Section A in Botswana, Mozambique, South Africa, and Zimbabwe. B.3 DELIVERABLES, TASKS, AND TIMELINES The following deliverables are required: i. Briefings a. Introductory Meetings/In-briefs – The Contractor must organize and conduct separate entry briefings for USAID/Southern Africa and the RESILIM implementers (Chemonics and AWARD). b. Out-Brief – At the end of field work and prior to proceeding to the first draft of the evaluation report, the Contractor must provide separate out-briefs for USAID/Southern Africa, Chemonics and AWARD, and other potential stakeholders. While the content of the meetings shall vary as appropriate, the Contractor shall provide a general outline of preliminary findings, conclusions and recommendations prior to drafting the evaluation report. In addition, the Contractor shall provide a final presentation to relevant USAID staff and implementing partners following the report completion. ii. Reports a. Evaluation Design Report (EDR) - a detailed evaluation plan based on an outline agreed upon with USAID shall be submitted to the Evaluation COR for review, feedback and concurrence within 15 business days after the purchase order start date. The EDR must include the Offeror’s proposed methodology, a list of any data to be collected and the potential sources, and an implementation plan, including a final schedule (which must include nominal number of field trips that will be approved in the Field Work Plan, see paragraph b) below. The EDR must also detail the methods and tools to generate the highest quality and most credible evidence available to answer the evaluation questions, and procedures/protocols to collect organize and interpret quantitative and qualitative data, as applicable, must be included (not to exceed 20 pages, plus annexes). 72067419Q00001 11 b. Field Work Plan (FWP) - The purpose of the FWP is to ensure that USAID/Southern Africa is aware of all of the Contractor’s movements in the field. The FWP must be submitted to the Evaluation Purchase Order Contracting Officer’s Representative (POCOR) at least 10 working days before staff first travel to the field to conduct fieldwork. It must outline the number of field trips to be taken by identified evaluation team members, include a final list of interviewees and day-by-day schedules of the field trips. After approval, any changes in itinerary must receive concurrence from the Evaluation POCOR. c. Draft Evaluation Report – The Contractor shall submit an electronic copy of the draft evaluation report to the Evaluation POCOR fifteen working days after completing the field data collection and analysis process. In the report, the Contractor shall provide separate findings, conclusions, and recommendations for each question. The Evaluation POCOR will share the draft evaluation report for written feedback with key stakeholders, including the RESILIM implementers, within ten working days from the time the draft report was shared. d. Final Evaluation Report – The final evaluation report shall incorporate changes as agreed by the Contractor and USAID/Southern Africa. The Contractor is not obligated to modify the report to incorporate changes that will alter the findings. When applicable, the evaluation report will include statements regarding any significant unresolved differences of opinion on the part of USAID, Chemonics, AWARD and/or members of the evaluation team. The Contractor shall submit an electronic copy of the final report to the Evaluation POCOR within ten working days after the Evaluation POCOR shares comments on the second draft evaluation report from key stakeholders (Not to exceed 30 pages, plus annexes). Within 10 days of approval by the Evaluation POCOR, the final evaluation report must be uploaded to the Development Evaluation Clearinghouse (DEC). The final report shall meet the following quality standards set forth in the USAID Evaluation Policy found here: https://www.usaid.gov/sites/default/files/documents/1868/201mah.pdf: a) The report must represent a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not and why; b) The report must address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; 72067419Q00001 12 d) The evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex to the final report; e) Evaluation findings must assess outcomes and impact by gender and on vulnerable groups; f) Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; and j) Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. The final evaluation report must include the following sections: 1. Executive Summary: Summarizes purpose and background, key evaluation questions, evaluation design and methods, findings, conclusions, and recommendations for the two RESILIM projects 2. Table of Contents 3. Acronym List 4. Introduction and Background: Describes purpose, audience, and synopsis of task, brief overview of the two projects, USAID program strategy and activities implemented in response to the stated problem, brief description of implementing partners 5. Evaluation Design and Methods: Describes evaluation design and data collection methods, including constraints and gaps 6. Findings/Conclusions/Recommendations for each of the two projects 7. References: Includes bibliography and summaries of meetings, interviews, and focus group discussions 72067419Q00001 13 8. Annexes: Documents the evaluation tools, schedules, interview lists, tables, sources of information, statement of work, and statements of differences Final Report Quality Criteria • The final evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the projects, what did not work, and why. • The final evaluation report must address all evaluation questions included in the scope of work. • The final evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, evaluation design and methods or timeline need to be agreed upon in writing. Only the Purchase Order Contracting Officer is authorized to make changes to the final evaluation Statement of Work. • Evaluation design and methodology shall be explained in detail in the final report and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex. • Evaluation findings must assess all project outcomes, including observed impacts on males and females. • Evaluation findings and analysis should be presented as analyzed facts, evidence, and data, and should not be based on anecdotes, hearsay, or opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be attributable to certain findings, and should be action-oriented, practical, and specific, with defined responsibility for the recommended action. e. Deliverable Format 72067419Q00001 14 All deliverables should be submitted in Microsoft product formats (e.g., Word, Excel, Power point, etc.) and 12-point type font should be used throughout, with page margins of one inch top/bottom and left/right. The Offeror must submit the underlying data to any conclusions reached. Final versions of the reports must be submitted in both MS formats and PDFs. B.4 EVALUATION SCHEDULE Timeline: The offeror will suggest a timetable/schedule not exceeding completion by September 15, 2019 including submittal of the final evaluation report. However, it is preferable that the evaluation be concluded with all reasonable speed. Additionally, the offeror’s timeline/schedule for the completion of the evaluation services shall be an evaluation criterion. When planning the schedule for the deliverables, the offeror should consider the following illustrative schedule: Table A: – Illustrative Weekly Work Schedule # Description of Task 1 2 3 4 5 6 7 8 9 10 11 12 13 1 Purchase order signed 2 USAID provides additional documents to winner of the award X 3 Introductory meetings, review of documentations and consultations with USAID and implementing partners X 4 Preparation and submission of work plan X X 5 USAID reviews/approves work plan X 6 In-brief - Presentation of the evaluation approach to USAID/SA X 7 Team travel days to field X 8 Field Work by the team – Interviews with stakeholders/program staff X X X 9 USAID out-brief (Team) X 10 Draft and submit draft evaluation report to USAID for review X X X 11 USAID reviews report X X 12 Final report compiled by evaluation team X X 13 Final report submitted to USAID X 14 REED environment team develops plan to address recommendations X X 15 Submission to DEC X 72067419Q00001 15 A six-day work week is authorized during travel status if it is not in conflict with the Offeror’s policies regarding work week. The evaluation team will be responsible for all off-shore and in￾country logistical support. This includes international and in-country travel (including vehicle rentals), hotel bookings, working/office space, computers, printing and photocopying. The evaluation team, in collaboration with USAID/Southern Africa, will arrange all meetings, interviews, site visits, and in-briefing and out-briefing presentations. In all other respects, the evaluation team should be self-sufficient. B.5 PROPOSED PAYMENT SCHEDULE Line Item #1: This is a firm-fixed-price line item for scheduled services. Based on satisfactory completion and subject to acceptance of all work and services, including the submission of required reports/deliverables, payment shall be as follows: Description of Deliverables % of Line Item #1 Price [offerors to propose] [offerors to propose] The ceiling for Line Item #1 is $222,000. Line Item #2: Line Item #2 is an unpriced line item in accordance with FAR 8.402(f) and FAR 13.302-2 for incidental open market items consisting of travel in support of Line Item #1. The monetary limitation for Line Item #2 is $70,000. The monetary limitation is an obligation subject to adjustment when the firm price is established. Once approved travel has been completed, the Contractor and USAID will agree on a firm price based on the Contractor’s documented travel costs. B.6 BPA and GSA Clauses Incorporated by Reference This RFQ incorporates all applicable Federal Acquisition Regulation (“FAR”), Agency for International Development Acquisition Regulation (“AIDAR”) clauses, and special in full text or incorporated by reference, and Section H—Special Contract Requirements, as contained in BPA Nos. AID-OAA-E-17- 00007, AID-OAA-E-17-00008, AID-OAA-E-17-00009 and AID-OAA-E-17-00010. This BPA incorporates all of the applicable clauses in GSA Contract Numbers GS00F252CA, GS00F227CA, GS00F010CA andGS10F083CA by reference, with the same force and affect as if they were given in full text. Upon request, the cognizant contracting officer will make their full text available. All clauses applicable to items not on the Federal Supply Schedule will be included in the order. 72067419Q00001 16 [END OF SECTION B] 72067419Q00001 17 SECTION C - INSTRUCTIONS, CONDITIONS, AND NOTICES TO OFFERORS C.1 GENERAL INSTRUCTIONS TO OFFERORS a) Instructions: If an Offeror does not follow the instructions set forth herein, the Offeror’s quotation may be eliminated from further consideration, or the quotation may be downgraded and not receive full or partial credit under the applicable evaluation criteria. b) Accurate and Complete Information: Offerors must set forth full, accurate and complete information as required by this acquisition. The penalty for making false statements to the Government is prescribed in 18 U.S.C. 1001. c) Offer Acceptability: The Government may determine an offer to be unacceptable if the offer does not comply with all of the terms and conditions of the BPA. d) Quotation Preparation Costs: The U.S. Government will not pay for any quotation preparation costs. C.2 DELIVERY INSTRUCTIONS a) Submission, Marking and Copies The Offeror should submit the quotation electronically - Internet email with up to 4 attachments (2MB limit per attachment), compatible with recent Windows-compatible version of MS Word (version 2000 or later); spreadsheets must be in MS Excel (version 2000 or later); and Adobe Acrobat (.pdf) usable in a MS Windows environment. Technical and Price/Cost Quotations shall be submitted in separate documents and emails, as described below, with all materials and supporting documentation required in English. All quotation materials should be submitted to: proposals@usaid.gov, with a copy to Nathan Piper at npiper@usaid.gov. Zipped files attachments are not allowed. Any budget spreadsheets shall be sent in unlocked Excel format (formula shown). The Technical and Price/Cost Quotations must be clearly marked on the email as follows: Technical Quotation: 72067419Q00001 – RESILIM – [Name of Offeror] Technical Quotation [Email # of #] Price/Cost Quotation: 72067419Q00001 – RESILIM - [Name of Offeror] Cost Quotation [Email # of #] 72067419Q00001 18 These same requirements exist for the submission of subsequent revised technical and/or cost quotations and technical and/or cost clarifications. b) Closing Date and Time All quotations in response to this acquisition shall be due no later than the Time and Date indicated on the cover letter. C.3 INSTRUCTIONS FOR THE PREPARATION OF THE TECHNICAL QUOTATION a) Technical Quotation The technical quotation must be concise and complete. The quotation will demonstrate the Offeror’s expertise with respect to achieving the evaluation program results. (1) General The Technical Quotation in response to this solicitation should address the technical evaluation factors listed in Section D. The technical quotation shall consist of the technical quotation itself, organized as further described below, and an annex. These together will constitute the Offeror’s technical quotation. The quotation must be well organized, complete, clear, and succinctly presented. (2) Page Limitation The technical quotation itself shall not exceed five (5) pages, not including the annex. Any additional pages will not be evaluated. Quotations shall be written in English and typed on standard 8 1/2" x 11" paper (210mm by 297mm paper) or A4 paper, single spaced, 12 characters per inch with each page numbered consecutively. Font size of less than 12 characters per inch may be used only in tables, charts and footnotes. Margin must normally be 1”. (3) Organization The technical quotation should be organized into the following sections as follows: Table of Contents A. Technical Approach B. Proposed Personnel C. Annex (4) Technical Quotation Body A. Technical Approach: 72067419Q00001 19 General: USAID is open to offeror suggestions on the evaluation design and data collection methods that are most appropriate and effective in answering the evaluation key questions. Offerors shall develop innovative approaches to conducting this evaluation, using activity descriptions of both activities referenced in the Statement of Work (SOW), and the material attached in Attachments 1 and 2. In addition, Offerors shall make use of the USAID Evaluation Policy (see link: https://www.usaid.gov/sites/default/files/documents/1868/201mah.pdf) and any other relevant information. Additionally, Offerors may propose additional deliverables but at a minimum must include the deliverables stated in Section B.3. USAID expects that both quantitative and qualitative methods will be used in the evaluation. Methodology The quotation must describe the process to be followed and the methodology to be used for the tasks set forth in the SOW and in relation to the various stakeholders involved, particularly with respect to the requirements for data collection and analysis. It must demonstrate reasonable expectations toward producing the intended outcomes. This includes an explanation of how activities will be carried out, tracked and reported and sample protocols or instruments. The methodology will clearly describe quantitative and qualitative methods, as applicable, to be used in the evaluation, and demonstrate gender aggregated data. The quotation must discuss the relative strengths and limitations of methodology proposed. The quotation must also demonstrate a sufficient knowledge of the technical and regional challenges the RESILIM program was attempting to address. Data Collection and Analysis Plan The quotation must include a data analysis plan— a detailed outline of the procedures/protocols to collect, organize and interpret quantitative and qualitative data, as applicable—that demonstrates the Offeror’s ability to collect and analyze data that will help transform the collected data into credible evidence to draw conclusions about the performance of the activity and to address the evaluation questions in Section A.3 of the SOW. Schedule and Implementation Approach The quotation must include a proposed mobilization plan (maximum two pages) in the annex that demonstrates feasibility in accomplishing all tasks within the given time frame. The schedule and implementation approach must be detailed enough to include realistic time frames and human and other resource needs to fully carry out the proposed evaluation approach. B. Proposed Personnel: The Offeror must propose the project team for the evaluation, including up to three (3) Evaluation PO Kkey Ppersonnel positions based on key personnelthe labor categories positions 72067419Q00001 20 described in the BPA. The Offeror must name candidates for the Evaluation PO Kkey Ppersonnel positions and include resumes. The evaluation team must have a mix of expertise that covers the following capabilities, at minimum: monitoring and evaluation; water resources management; climate change adaptation; biodiversity conservation; capacity building; institutional development; gender; and regional experience in the four Limpopo countries (South Africa, Mozambique, Botswana and Zimbabwe). At least one member of the team must have Portuguese language capability. C. Annex: The technical quotation annex shall contain: 1) CVs for all Evaluation PO Kkey Ppersonnel with letters of commitment; and 2) Mobilization Plan. A plan to initiate the program including activity start up (staffing, office set up, procurements), stakeholder consultation process with some level of detail, and additional analysis that will be conducted during this time; 2 pages maximum. b) Price/Cost Quotation Any cost assumptions should be explained in a narrative, including the hourly rates and level of effort for each personnel. The quote and cost assumptions should be written in response to the requirements in Section A - Scope of Work. The price quote and narrative should not exceed three pages. Prices must be expressed in US Dollars, showing separately the Value Added Tax (VAT). As a matter of US Law, United States Government foreign assistance is not subject to taxation. However, when the activity is subject to taxation (i.e. VAT), the total cost is considered to be inclusive of VAT. VAT recovered will not be added to the Purchase Order, nor will it increase its total estimated cost. The price quote must include a proposed payment schedule for Line Item #1 based on the following: Description of Deliverables % of Line Item #1 Price [offerors to propose] [offerors to propose] 72067419Q00001 21 Pricing for travel and transportation in support of the evaluation will be covered by an unpriced line item in accordance with FAR 13.302-2 with a monetary limitation of $70,000, with the final amount to be determined as a firm price based on documented costs for approved travel. [END OF SECTION C] 72067419Q00001 22 SECTION D – EVALUATION FACTORS FOR AWARD D.1 GENERAL INFORMATION (a) The Government intends to evaluate Offerors in accordance with Section D of this solicitation and make contract award to the responsible Offeror whose quotation represents the best value to the U.S. Government. (b) The submitted technical quotations will be evaluated using the technical factors shown below. The evaluation committee may include experts who are not employees of the Federal Government. (c) For overall evaluation purposes, evaluation factors other than cost or price, when combined, are more important than cost or price. D.2 EVALUATION FACTORS The factors below are presented by major category, with relative weights identified, so that Offerors will know which areas require emphasis in the preparation of quotations. The factors below reflect the requirements of this particular solicitation. Evaluation criteria are as follows: Technical Approach (60%) Proposed Personnel (40%) Total (100%) Specific information on each evaluation factor and sub-factor is provided below. A. Technical Approach: Methodology, Schedule and Implementation Approach (45%) The quotation will be evaluated based on the extent to which it describes the process to be followed and the methodology to be used for the tasks and in relation to the various stakeholders involved. The quotation will be evaluated based on the extent to which it demonstrates reasonable expectations, including with regard to schedule and implementation approach, toward producing the intended outcomes described in the Statement of Work. The quotation will also be evaluated based on the explanation of how activities will be carried out, tracked and reported and sample protocols or instruments. Data Collection and Analysis Plan (15%) 72067419Q00001 23 The data analysis plan— a detailed outline of the procedures/protocols to collect, organize and interpret quantitative and qualitative data, as applicable—will be evaluated based on the extent to which it demonstrates the Offeror’s ability to collect and analyze data that will help transform the collected data into credible evidence to draw conclusions about the performance of the activity and to address the evaluation questions in the SOW. B. Proposed Personnel: Evaluation PO Key Personnel (20%) The quotation will be evaluated based on the extent to which the designated Evaluation PO Key Personnel represent the most critical positions to successful completion of the evaluation. The quotation will be evaluated based on the extent to which there is a match between the roles and responsibilities proposed and the associated required experience and qualifications. The quotation will be evaluated based on the extent to which the candidates proposed for the Evaluation PO Key Personnel positions meet the requirements of the position descriptionsfunctional labor categories contained in Sections C.4 and C.5 of the BPA. Team Structure and Composition (20%) The quotation will be evaluated based on the extent to which the Offeror demonstrates a team composition and structure that can successfully implement the proposed technical approach and answer USAID’s evaluation questions stated in the SOW. The quotation will be evaluated based on the extent to which the team has a mix of expertise that sufficiently covers the following capabilities, at minimum: monitoring and evaluation; water resources management; climate change adaptation; biodiversity conservation; capacity building; institutional development; gender; and regional experience in the four Limpopo countries (South Africa, Mozambique, Botswana and Zimbabwe). The evaluation will consider whether the quotation meets the requirement that at least one member of the team must have Portuguese language capability. The quotation will be evaluated based on the extent to which the positions proposed for non￾Evaluation PO Kkey Ppersonnel are, individually and collectively, appropriate and necessary to successfully accomplish the evaluation. The quotation will be evaluated based on the extent to which the quotation clearly shows roles, responsibilities and relationships of proposed Evaluation PO Key and non-Key Ppersonnel that, in combination, reflect the necessary skill sets to accomplish the evaluation’s technical and logistical requirements. Price/Cost Evaluation Price/Cost will not be scored but will be used for best value determination with Technical score being more important than cost. USAID will evaluate the Offeror’s quoted price for completeness, reasonableness, and realism. USAID will evaluate to determine whether the estimated quoted costs elements are realistic for the work to be performed, reflect a clear 72067419Q00001 24 understanding of the requirements, and are consistent with the unique methods of performance described in the Offeror’s technical quotation. [END OF SECTION D] 46 ANNEX II. EVALUATION MATRIX The evaluation matrix (Table 9) served as the framework for this evaluation. The evaluation matrix aligned each evaluation question with a tailored evaluation design strategy in the form of sub-questions and specific data sources. The development of the evaluation matrix was informed by an initial desk review and an initial internal consultation. Table 9. Evaluation Matrix Key Questions Sub-questions Sources of Information / Methods Scalability To what extent was the two￾mechanism approach (catchment and Basin-level) effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? ▪ Were the goals of the RESILIM program met? How did RESILIM-B and RESILIM-O each contribute to achievement of the program goals? ▪ How has the catchment/Basin-level dynamic helped or hindered the achievement of program goals? How could the catchment/Basin-level approach be improved? ▪ Document review of final report, results frameworks, M&E plans ▪ Interviews with USAID program staff, implementing and collaborating partners, intergovernmental institutions working across the Basin (including LIMCOM), governments within the Basin, and key stakeholders at both catchment and Basin-levels How effective was the RESILIM-O program at creating scalable activities that are ready for Basin-wide implementation? ▪ What approach did the RESILIM-O program take to identify activities that could be scaled up to the Basin￾level? Did this approach appropriately consider opportunities and risks at the Basin-level? What was the engagement of Basin-level actors in this approach? ▪ Did the RESILIM-O program create activities that are scalable? Were those activities adopted at the Basin￾level? How and why? ▪ Document review of RESILIM-O work plan, progress reports, knowledge products, M&E plans ▪ Interviews with key program staff, RESILIM-O implementing and collaborating partners, intergovernmental institutions and governments, and key stakeholders including at Basin-level Cross-sectoral resilience How has resilience been defined in each of the RESILIM-O and RESILIM-B programs? ▪ Has resilience been defined by the RESILIM-O and RESILIM-B programs? If not, how has it been understood in practice? What indicators were used to measure it? What is the timeframe at which resilience outcomes should be measured? ▪ Document review of SOWs and work plans, knowledge products about resilience, M&E plans for resilience indicators, Workshop Report ▪ Interviews with key program staff and implementing partners ▪ Benchmark to commonly accepted definitions To what extent has the cross-sectoral approach (across water resources management, climate change ▪ Has the resilience of people and ecosystems changed in the Basin? Have there been changes in the quality of ecological, social, and economic conditions in the Basin? ▪ Document review of sector management plans and policies, national and/or basin-level climate change adaptation strategies and plans, documents from other 47 Key Questions Sub-questions Sources of Information / Methods adaptation, and biodiversity conservation) been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? How would the natural systems and institutions that manage them respond differently to changes in climate conditions compared to prior to the program? Has resilience been integrated into the management plans and policies for these sectors? ▪ To what extent were the sectors integrated in the project approach and delivery, and what were the implications of integration for building resilience? ▪ What lessons can be learned from the cross-sectoral approach? What were the benefits of working on resilience of the Basin through the sectors of climate change adaptation and biodiversity conservation? resilience initiatives in region (e.g., AFDB), and M&E plans for resilience indicators ▪ Interviews with implementing and collaborating partners, resource management institutions (including LIMCOM), government departments for each sector, USAID/Southern Africa REED, and key stakeholders including multilateral and bilateral donors working in each sector in the region Partner institutions To what extent has the RESILIM program improved the effectiveness of the Limpopo Watercourse Commission (LIMCOM) and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? ▪ Have LIMCOM and other regional institutions’ capacity and practices changed in managing the Limpopo Basin in an environmental and socially informed way as a result of RESILIM? Have these institutions strengthened Basin￾level collaboration, or internal organizational structures and processes? Have there been changes in the enabling environment in which they operate? ▪ How effective was the RESILIM grant approach to achieving RESILIM goals? What are alternative approaches? ▪ Are there additional institutions that should be included as partners to build on RESILIM’s outcomes? ▪ Document review of M&E plans, responses from questionnaires conducted post training workshops, RESILIM Workshop Report, new Basin-related policies and regulations ▪ Interviews with key program staff, implementing and collaborating partners, LIMCOM, other regional/ intergovernmental institutions, government departments involved in managing the Basin ▪ Questionnaires for LIMCOM, other regional/ intergovernmental institutions, governments departments, and other key stakeholders at the Basin￾level Sustainability of interventions How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? How could USAID/Southern Africa’s Resilient Waters Program be adapted to incorporate lessons learned under the RESILIM program? ▪ What strategies did RESILIM employ to improve the likelihood of sustainability? ▪ How effective were these strategies in ensuring the sustainability of its results? Have certain interventions or results been more sustainable than others? How and why? ▪ What are institutional barriers to sustainability? What lessons can be learned for the USAID/Southern Africa’s Resilient Waters Program? ▪ Document review of M&E plans ▪ Interviews with key program staff, implementing and collaborating partners, USAID officials including those working on Resilient Waters Program, intergovernmental institutions including LIMCOM, governments, and key stakeholders including international cooperation partners ▪ Questionnaires for intergovernmental institutions and governments and key stakeholders 48 Key Questions Sub-questions Sources of Information / Methods Social inclusion How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? ▪ To what extent was implementation of RESILIM activities informed by youth, gender sensitive, and social inclusion considerations (e.g., through consultation, design elements, implementation support, etc.)? ▪ How effective were these approaches in terms of increasing engagement and improving results? ▪ Document review of work plans, youth gender indicators in M&E plans ▪ Interviews with key program staff, implementing and collaborating partners, USAID officials, intergovernmental institutions and governments, and key stakeholders representing youth, gender, and socially marginalized groups ▪ Questionnaires for key stakeholders representing youth, gender, and socially marginalized groups 49 ANNEX III. DATA COLLECTION INSTRUMENTS Interview Protocols Interview protocols provide the basis for semi-structured interviews tailored to the experience and knowledge of each group of key informants: • USAID RESILIM, USAID, and other USG staff • Consortium of implementing partners • Intergovernmental institutions and government o Includes transboundary institutions, river basin authorities, transfrontier conservation area management, catchment management agencies, and government departments • Key stakeholders o Includes NGOs, civil society organizations, academic/research, international cooperation partners At the outset of each interview, the Evaluation Team will provide a brief introduction that describes: • The aims of the interview and the evaluation; • The expected length of the interview; and • How interview information will be used — i.e., that comments made by the interviewee will not be attributable. In each protocol, the list of questions is intended to provide guidance for interviewers. Please note that these are overarching research questions; interviewers will ask ones that are shorter, more straightforward, and conversational in style, with probes (e.g., That’s“’s interesting. Can you tell me a specific example about that?”)” to elicit more detailed answers. The interview protocols are a starting point; the semi-structured format employed by the Evaluation Team is a flexible approach that ensures that specific topics are addressed per the evaluation objectives while remaining open to capturing unanticipated data. More specific lines of questioning will follow on to the questions provided in each interview protocol, based on the individual informants’ experience and knowledge of topics. Not all interviewees may be familiar with USAID RESILIM as a whole (i.e., both RESILIM-B and RESILIM￾O), and they will not be expected to comment on components of the program that they are unfamiliar with. Interviewers are thus expected to cover the key themes outlined in each protocol below, but each interview will be selected and tailored appropriately. 50 USAID RESILIM, USAID, AND OTHER USG STAFF Program Evaluation of USAID RESILIM Date:______________________ Name:_______________________________________ Country:___________________ Title:__________________________________ ______ Sex: ☐ Male ☐Female Organization:_________________________________ Mechanism(s): ☐ RESILIM-B ☐ RESILIM-O Interviewer(s):________________________________________________________________ Introduction • Inform the interviewee of the overall aim of the interview, the time allotted, and that their comments will not be attributed. • If appropriate, ask the interviewee to begin with a brief description of their engagement with RESILIM. Guiding Research Questions for Interviews Scalability 1. To what extent was the two-mechanism approach (catchment and Basin-level) effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? a. Were the goals of the RESILIM program met (program objectives listed below)? How did RESILIM-B and RESILIM-O each contribute to achievement of the program goals? • Reducing climate vulnerability by promoting the adoption of science-based adaptation strategies for integrated transboundary water resource management • Conserving biodiversity and sustainably managing high-priority ecosystems • Building the capacity of stakeholders to sustainably manage water and key ecosystems b. How could the design of the RESILIM Program two-mechanism approach have been adjusted to effectively work at both the catchment and Basin-levels to meet program objectives? c. What are the benefits and drawbacks of working across the two geographic scales to achieve the program objectives? 2. How effective was the RESILIM-O program at creating scalable activities that are ready for Basin-wide implementation? a. What approach did the RESILIM-O and RESILIM-B programs take, separately and jointly, to identify activities that could be scaled up to the Basin-level? b. Did the RESILIM-O program create activities that are scalable? Cross-sectoral resilience 3. How has the RESILIM program defined resilience? 51 a. Has resilience been defined by the RESILIM-O and RESILIM-B programs? If not, how has it been understood in practice? What indicators were used to measure it? 4. To what extent has the cross-sectoral approach (across water resources management, climate change adaptation, and biodiversity conservation) been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? a. How were the sectors (water resources management, climate change adaptation, biodiversity conservation, and livelihoods) integrated in the project approach and delivery? Are there opportunities to strengthen a) resilience in the Olifants catchment and Limpopo River Basin and b) cross-sectoral integration that were not covered through the RESILIM program? b. What lessons can be learned from the cross-sectoral approach? What were the benefits of working on resilience of the Basin through the sectors of climate change adaptation, biodiversity conservation, and livelihoods? c. Have there been changes in the quality of ecological, social, and economic conditions in the Olifants catchment and Limpopo River Basin during and after the RESILIM program? d. How would the natural systems and institutions that manage them respond differently to changes in climate conditions compared to prior to the program? Partner institutions 5. To what extent has the RESILIM program improved the effectiveness of LIMCOM and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? a. Have LIMCOM and other regional institutions’ (ORASCOM, OKACOM, GLTFCA, etc.) capacity and practices changed in managing the Limpopo Basin in an environmentally and socially informed way? Have there been changes in the enabling environment in which they operate? b. Are there additional institutions that should be included as partners to enhance RESILIM’s outcomes? Sustainability of interventions 6. How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? a. What strategies did RESILIM employ in program design and throughout implementation to improve the likelihood of sustainability? b. How effective were these strategies in ensuring the sustainability of its results? Have certain interventions or results been more sustainable than others? How and why? c. How effective was the RESILIM grant approach to achieving RESILIM goals? What are alternative approaches? d. What are institutional barriers to sustainability? What lessons can be learned for USAID/Southern Africa’s Resilient Waters Program? Social inclusion 52 7. How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? a. What strategies did RESILIM employ to consider youth, gender sensitive, and social inclusion throughout the program (e.g., through consultation, design elements, implementation support, etc.)? b. How effective were these approaches in terms of ensuring engagement and improving results? 53 CONSORTIUM OF IMPLEMENTING PARTNERS Program Evaluation of USAID RESILIM Date:______________________ Name:_______________________________________ Country:___________________ Title:__________________________________ ______ Sex: ☐ Male ☐Female Organization:_________________________________ Mechanism(s): ☐ RESILIM-B ☐ RESILIM-O Interviewer(s):________________________________________________________________ Introduction • Inform the interviewee of the overall aim of the interview, the time allotted, and that their comments will not be attributed. • If appropriate, ask the interviewee to begin with a brief description of their engagement with RESILIM. Guiding Research Questions for Interviews Scalability 1. To what extent was the two-mechanism approach (catchment and Basin-level) effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? a. Were the goals of the RESILIM program met (program objectives listed below)? How did RESILIM interventions at the scale at which you worked [catchment-level for RESILIM-O and Basin-level for RESILIM-B] contribute to achievement of the program goals? • Reducing climate vulnerability by promoting the adoption of science-based adaptation strategies for integrated transboundary water resource management • Conserving biodiversity and sustainably managing high-priority ecosystems • Building the capacity of stakeholders to sustainably manage water and key ecosystems b. How could the design of the RESILIM Program two-mechanism approach have been adjusted to effectively work at both the catchment and Basin-levels to meet program objectives? c. What are the benefits and drawbacks of working across the two geographic scales to achieve the program objectives? How could the approach be improved? 2. How effective was the RESILIM-O program at creating scalable activities that are ready for Basin-wide implementation? c. What approach did the RESILIM-O and RESILIM-B programs take, separately and jointly, to identify activities that could be scaled up to the Basin-level? 54 d. Did RESILIM-O engage with Basin-level actors? Did RESILIM-B engage with actors within the catchment-level? How effective was that engagement? e. Did the RESILIM-O program create activities that are scalable? How could program design and support from USAID have been adjusted to enhance achievement of this objective? Cross-sectoral resilience 3. How has the RESILIM program defined resilience? f. Has resilience been defined by the RESILIM-O and RESILIM-B programs? If not, how has it been understood in practice? g. How did the programs measure resilience (e.g., indicators and other methods)? Were these methods effective in capturing changes in resilience? What methods or indicators might they have chosen instead? 4. To what extent has the cross-sectoral approach (across water resources management, climate change adaptation, and biodiversity conservation) been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? h. How were the sectors (water resources management, climate change adaptation, biodiversity conservation, and livelihoods) integrated in the project approach and delivery? Are there opportunities to strengthen a) resilience in the Olifants catchment and Limpopo River Basin and b) cross-sectoral integration that were not covered through the RESILIM program? i. What lessons can be learned from the cross-sectoral approach? What were the benefits of working on resilience of the Basin through the sectors of climate change adaptation, biodiversity conservation, and livelihoods? j. Have there been changes in the quality of ecological, social, and economic conditions in the Olifants catchment and Limpopo River Basin during and after the RESILIM program? k. How would the natural systems and institutions that manage them respond differently to changes in climate conditions compared to prior to the program? Partner institutions 5. To what extent has the RESILIM program improved the effectiveness of LIMCOM and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? a. Have LIMCOM and other regional institutions’ (ORASCOM, OKACOM, GLTFCA, etc.) capacity and practices changed in managing the Limpopo Basin in an environmentally and socially informed way? Have there been changes in the enabling environment in which they operate? b. Are there additional institutions that should be included as partners to enhance RESILIM’s outcomes? 55 c. For RESILIM-O: How did the RESILIM-O program adjust its approach for engagement with partner institutions after disbandment of the Olifants Catchment Management Agency? Sustainability of interventions 6. How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? a. What strategies did RESILIM employ to improve the likelihood of sustainability? b. How effective was the RESILIM grant approach to achieving RESILIM goals? Did the grants align with the theory of change and contribute to program objectives? What are the tradeoffs of managing small grants vs. providing resources for multiple grantee institutions? What are alternative approaches? c. What are institutional barriers to sustainability? What lessons can be learned for the USAID/Southern Africa’s Resilient Waters Program? Social inclusion 7. How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? a. What strategies did RESILIM employ to consider youth, gender sensitive, and social inclusion throughout the program (e.g., through consultation, design elements, implementation support, etc.)? b. How effective were these approaches in terms of ensuring engagement and improving results? 56 INTERGOVERNMENTAL INSTITUTIONS AND GOVERNMENT Program Evaluation of USAID RESILIM Date:______________________ Name:_______________________________________ Country:___________________ Title:__________________________________ ______ Sex: ☐ Male ☐Female Organization:_________________________________ Mechanism(s): ☐ RESILIM-B ☐ RESILIM-O Interviewer(s):________________________________________________________________ Introduction • Inform the interviewee of the overall aim of the interview, the time allotted, and that their comments will not be attributed. • If appropriate, ask the interviewee to begin with a brief description of their engagement with RESILIM. Guiding Research Questions for Interviews Scalability 1. To what extent was the two-mechanism approach (Olifants catchment and Limpopo Basin-level) effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? a. Describe how your entity worked across the catchment and/or Basin-level through RESILIM. How did the RESILIM program work with your entity at the catchment and/or Basin-level on its three program objectives (listed below)? • Reducing climate vulnerability by promoting the adoption of science-based adaptation strategies for integrated transboundary water resource management • Conserving biodiversity and sustainably managing high-priority ecosystems • Building the capacity of stakeholders to sustainably manage water and key ecosystems b. How effective was the RESILIM program in working at the catchment-level and Basin-level? c. What are the benefits and drawbacks of working across the two geographic scales to achieve the program objectives? How could the approach be improved? 2. How effective was the RESILIM-O program at creating scalable activities within the Olifants Catchment that are ready for Basin-wide implementation? a. For Basin-level actors: Did RESILIM-O engage with your entity to identify activities that could be scaled up at the Basin-level? How effective was that engagement? b. For catchment-level actors: Did RESILIM-O engage with your entity to identify activities that could be scaled up at the Basin-level? Did RESILIM-B engage with your entity to identify activities that could be scaled up at the Basin-level? How effective was that engagement? c. Are there activities under the RESILIM-O program that are scalable across the regions where your entity works? Were those activities adopted at the Basin-level? How and why? 57 Cross-sectoral resilience 3. How has the RESILIM program defined resilience? a. Has resilience been defined by the mechanism your entity worked with (RESILIM-B and/or RESILIM-O)? If not, how has it been understood in practice? b. Does the RESILIM-B and/or RESILIM-O’s use of the term resilience differ from how your entity defines resilience? 4. To what extent has the cross-sectoral approach (across water resources management, climate change adaptation, and biodiversity conservation) been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? a. How were the sectors (water resources management, climate change adaptation, biodiversity conservation, and livelihoods) integrated in the project approach and delivery? Are there opportunities to strengthen a) resilience in the Olifants catchment and Limpopo River Basin and b) cross-sectoral integration that were not covered through the RESILIM program? b. What lessons can be learned from the cross-sectoral approach? What were the benefits of working on resilience of the Basin through the sectors of climate change adaptation, biodiversity conservation, and livelihoods? c. Have there been changes in the quality of ecological, social, and economic conditions in the Olifants catchment and Limpopo River Basin during and after the RESILIM program? d. Has your entity changed sector management practices to respond to changes in climate conditions as a result of the RESILIM program? To what extent has resilience been integrated into the management plans and policies for these sectors? Partner institutions 5. To what extent has the RESILIM program improved the effectiveness of LIMCOM and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? a. For intergovernmental and regional institutions managing Limpopo Basin (e.g., LIMCOM): i. How effective was the support your entity received through the RESILIM program to help manage the Limpopo Basin in an environmentally and socially informed way? What changes has your entity experienced as a result of that support (e.g., strengthened Basin￾level relationships, organizational structures, and processes; changes in practices in managing the Basin)? ii. Are there additional institutions that should be included as partners to enhance RESILIM’s outcomes? b. For other intergovernmental and regional institutions and country government institutions: i. To what extent has your entity changed its engagement and interaction with LIMCOM and other related regional institutions to manage the Limpopo Basin in an environmentally and socially informed way as a result of the RESILIM program? ii. Are there additional partner institutions, funders, or government agencies that should be included as to enhance RESILIM’s outcomes? 58 Sustainability of interventions 6. How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? a. How has the RESILIM program work with your entity to ensure the sustainability of its interventions and/or results past the life of the program? b. How is your entity currently utilizing the RESILIM interventions and their products? c. Have you utilized interventions and/or products from the RESILIM grants? d. What are institutional barriers to sustainability? What lessons can be learned for the USAID/Southern Africa’s Resilient Waters Program (5-year program to build more resilient and water-secure communities and ecosystems through improved management of transboundary natural resources and increased access to safe drinking water and sanitation services)? Social inclusion 7. How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? a. How did the RESILIM program consider youth, gender sensitive, and social inclusion as part of its implementation? b. How effective were these approaches in terms of ensuring engagement and improving results? 59 KEY STAKEHOLDERS (NGOS, CIVIL SOCIETY ORGANIZATIONS, ACADEMIC/RESEARCH, INTERNATIONAL COOPERATION PARTNERS) Program Evaluation of USAID RESILIM Date:______________________ Name:_______________________________________ Country:___________________ Title:__________________________________ ______ Sex: ☐ Male ☐Female Organization:_________________________________ Mechanism(s): ☐ RESILIM-B ☐ RESILIM-O Interviewer(s):________________________________________________________________ Introduction • Inform the interviewee of the overall aim of the interview, the time allotted, and that their comments will not be attributed. • If appropriate, ask the interviewee to begin with a brief description of their engagement with RESILIM. Guiding Research Questions for Interviews Scalability 1. To what extent was the two-mechanism approach (catchment and Basin-level) effective in meeting the goals of the RESILIM Program? What were the strengths and weaknesses of this design approach? a. Describe how your entity worked across the catchment and/or Basin-level through RESILIM. b. How did the RESILIM program work with your entity at the catchment and/or Basin-level on its three program objectives (listed below)? • Reducing climate vulnerability by promoting the adoption of science-based adaptation strategies for integrated transboundary water resource management • Conserving biodiversity and sustainably managing high-priority ecosystems • Building the capacity of stakeholders to sustainably manage water and key ecosystems c. How effective was the RESILIM program in working at the catchment-level and Basin-level? d. What are the benefits and drawbacks of working across the two geographic scales to achieve the program objectives? How could the approach be improved? 2. How effective was the RESILIM-O program at creating scalable activities that are ready for Basin-wide implementation? a. For Basin-level actors: Did RESILIM-O engage with your entity to identify activities that could be scaled up at the Basin-level? How effective was that engagement? b. For catchment-level actors: Did RESILIM-B engage with your entity to identify activities that could be scaled up at the Basin-level? How effective was that engagement? 60 c. Are there activities under the RESILIM-O program that are scalable across the regions where your entity works? Were those activities adopted at the Basin-level? How and why? Cross-sectoral resilience 3. How has the RESILIM program defined resilience? a. Has resilience been defined by the RESILIM-O and RESILIM-B programs? If not, how has it been understood in practice? b. Does the RESILIM program’s use of the term resilience differ from how your entity defines resilience? 4. To what extent has the cross-sectoral approach (across water resources management, climate change adaptation, and biodiversity conservation) been successful in building resilience in the Limpopo River Basin? How could cross-sectoral integration have been improved? a. How were the sectors (water resources management, climate change adaptation, biodiversity conservation, and livelihoods) integrated in the project approach and delivery? Are there opportunities to strengthen a) resilience in the Olifants catchment and Limpopo River Basin and b) cross-sectoral integration that were not covered through the RESILIM program? b. What lessons can be learned from the cross-sectoral approach? What were the benefits of working on resilience of the Basin through the sectors of climate change adaptation, biodiversity conservation, and livelihoods? c. Have there been changes in the quality of ecological, social, and economic conditions in the Olifants catchment and Limpopo River Basin during and after the RESILIM program? d. Has your entity changed sector management practices to respond to changes in climate conditions as a result of the RESILIM program? To what extent has resilience been integrated into the management plans and policies for these sectors? Partner institutions 5. To what extent has the RESILIM program improved the effectiveness of LIMCOM and other related regional institutions in managing the Limpopo Basin? Are there alternative approaches or partner institutions that can better optimize this outcome of the RESILIM program? a. Have LIMCOM and other regional institutions’ (ORASCOM, OKACOM, GLTFCA, etc.) capacity and practices changed in managing the Limpopo Basin in an environmentally and socially informed way? Have there been changes in the enabling environment in which they operate? b. Are there additional institutions that should be included as partners to enhance RESILIM’s outcomes? Sustainability of interventions 6. How have Program interventions been sustained past the life of the program? Which types of interventions have been the most sustainable? a. For grant recipients: How did you utilize the grant to contribute to RESILIM goals? What were opportunities and challenges? 61 b. How has the RESILIM program work with your entity to ensure the sustainability of its interventions and/or results past the life of the program? c. How is your entity currently utilizing the RESILIM interventions and their products? d. What are institutional barriers to sustainability? What lessons can be learned for the USAID/Southern Africa’s Resilient Waters Program (5-year program to build more resilient and water-secure communities and ecosystems through improved management of transboundary natural resources and increased access to safe drinking water and sanitation services)? Social inclusion 7. How effective was the RESILIM program in integrating youth, gender sensitive, and social inclusion approaches into implementation? How effective were those approaches? a. How did the RESILIM program consider youth, gender sensitive, and social inclusion as part of its implementation? b. How effective were these approaches in terms of ensuring engagement and improving results? 62 Questionnaire FINAL RESILIENCE IN THE LIMPOPO BASIN (RESILIM) PROGRAM EVALUATION: QUESTIONNAIRE FOR INTERGOV’, GOVERNMENT, KEY STAKEHOLDERS Please indicate the extent of your agreement with the following statements. All responses will be held confidentially by the evaluation team. Mechanism(s): ☐ RESILIM-B ☐ RESILIM-O Affiliation: ☐ Intergovernmental Institution ☐ Government ☐ Civil Society / Academia ☐ International Cooperation Partner ☐ Other Partner Institutions 1. RESILIM has contributed to strengthening collaboration among countries and regional institutions to manage the Limpopo Basin. Strongly Agree ☐ Agree ☐ Neither Agree nor Disagree ☐ Disagree ☐ Strongly Disagree ☐ N/A or Don’t Know ☐ 2. RESILIM has contributed to strengthening the enabling environment that supports LIMCOM and other regional institutions that manage the Limpopo Basin. Strongly Agree ☐ Agree ☐ Neither Agree nor Disagree ☐ Disagree ☐ Strongly Disagree ☐ N/A or Don’t Know ☐ Sustainability of Interventions 1. Your entity has used or leveraged RESILIM’s interventions and results past the life of the program. Strongly Agree ☐ Agree ☐ Neither Agree nor Disagree ☐ Disagree ☐ Strongly Disagree ☐ N/A or Don’t Know ☐ 2. RESILIM implemented effective strategies to improve the likelihood of sustainability of its interventions. Strongly Agree ☐ Agree ☐ Neither Agree nor Disagree ☐ Disagree ☐ Strongly Disagree ☐ N/A or Don’t Know ☐ Social inclusion 1. RESILIM effectively engaged youth, gender, and socially marginalized groups Strongly Agree ☐ Agree ☐ Neither Agree nor Disagree ☐ Disagree ☐ Strongly Disagree ☐ N/A or Don’t Know ☐ 2. RESILIM thoughtfully incorporated youth, gender sensitive, and social inclusion considerations in its activities Strongly Agree ☐ Agree ☐ Neither Agree nor Disagree ☐ Disagree ☐ Strongly Disagree ☐ N/A or Don’t Know ☐ COMMENTS: 63 ANNEX IV. SOURCES OF INFORMATION Documents Reviewed RESILIM-B Documents Title Author Year Resilience in the Limpopo Basin (RESILIM) Program Final Report Chemonics 2017 Building Resiliency in the Limpopo River Basin: RESILIM Workshop Report USAID 2017 Resilience in the Limpopo Basin: Year Four Performance Monitoring Plan (PMP) Chemonics 2016 Annual Progress Report: Resilience in the Limpopo Basin (RESILIM) Program Year Four: October 2015 – September 2016 Chemonics 2016 Annual Work Plan October 2015 – September 2016: Resilience in the Limpopo Basin Program (RESILIM) Chemonics 2016 Limpopo River Basin Disaster Preparedness Action Plan: Enhancing resilience to water-related disaster risks GWP SA 2016 Great Limpopo Transfrontier Conservation Area (GLTFCA): Integrated Livelihoods Diversification Strategy 2016–2030 GLTFCA 2016 Summary Report on Hydrogeological Information for the Development of Subsurface Mapping for the Transboundary Ramotswa Aquifer, using Airborne Geophysics XRI Blue 2016 Risk, Vulnerability, and Resilience in the Limpopo River Basin OneWorld 2015 Resilience in the Limpopo Basin: The Potential Role of the Transboundary Ramotswa Aquifer Chemonics 2015 Annual Progress Report: Resilience in the Limpopo Basin (RESILIM) Program Year Three: October 2014 – September 2015 Chemonics 2015 Resilience in the Limpopo River Basin (RESILIM) Program Presentation to Global Development Lab Kerry Reeves 2015 Resilience in the Limpopo Basin: the Potential Role of the Transboundary Ramotswa Aquifer IWMI 2015 Annual Work Plan October 2014 – September 2015: Resilience in the Limpopo Basin Program (RESILIM) Chemonics 2014 Annual Report: Resilience in the Limpopo Basin (RESILIM) Program: October 2013 – September 2014 Chemonics 2014 Economic Valuation of the mangroves ecosystem in the Limpopo River Estuary CDS-ZC 2014 Annual Work Plan October 2013 – September 2014: Resilience in the Limpopo Basin Program (RESILIM) Chemonics 2013 Annual Report: Resilience in the Limpopo Basin (RESILIM) Program: October 2012 – September 2013 Chemonics 2013 Fact Sheet: Resilience in the Limpopo Basin Program (RESILIM) USAID 2013 Annual Work Plan February – September 2013: Resilience in the Limpopo Basin Program (RESILIM) Chemonics 2012 2012 Annual Report: Resilience in the Limpopo Basin (RESILIM) Program Chemonics 2012 RESILIM Statement of Work USAID 2012 64 RESILIM-O Documents Title Author Year Designing a complex resilience building programme within a systemic framing: The RESILIM-Olifants experience AWARD 2019 Summary of Institutional Capacity Development in RESILIM-O AWARD 2019 Limpopo Basin Curriculum Innovation Network (LBCIN) AWARD 2019 Progress Report Quarter 3 2018-2019 FY AWARD 2019 Annual Report 2017/2018 Financial Year AWARD 2018 Changing Practice: Olifants catchment Final Report AWARD 2018 Highlights, innovations and challenges 2018 RESILIM-O AWARD 2018 Annual Report 2016/2017 Financial Year AWARD 2017 Monitoring, Evaluation, Reporting and Learning for the USAID RESILIM-O Program AWARD 2017 Ba-Phalaborwa Local Municipality Biodiversity Sector Plan Handbook AWARD 2017 Overview of Progress on the Blyde Catchment Restoration for Enhanced Biodiversity and Ecosystem Services (RESILIM-O Programme) AWARD 2017 Drought mitigation for water security: Interim Operating Rules for the Lower Olifants River Progress and Reflection Reprot: September to November 2016 AWARD 2016 Annual Report 2015/2016 Financial Year AWARD 2016 Annual Work Plan 2016: Resilience in the Limpopo Basin – Olifants (RESILIM-O) AWARD 2016 Annual Report Resilience in the Limpopo Basin – Olifants (RESILIM-O) AWARD 2015 USAID RESILIM-O Annual Report AWARD 2014 A systemic framework for context-based decision making in natural resource management: reflections on an integrative assessment of water and livelihood security outcomes following policy reform in South Africa Pollard, Biggs and Du Toit 2014 Annual Report to USAID for period 1 December 12 to 30 September 2013 RESILIM-O: Resilience in the Olifants Program AWARD 2013 Fact Sheet: Resilience in the Limpopo Basin Program (RESILIM): Olifants Catchment USAID 2013 River management under transformation: The development of Strategic Adaptive Management as an approach to river management within the Kruger National Park, South Africa Pollard and Du Toit 2010 Baseline Report Olifants River Basin in South Africa IWMI 2008 Informants Interviewed USAID, Chemonics and AWARD Interviewees Organization Name USAID Jeanette Normand Biggie Chidzvondo Judith Zvikaramba Nathan Piper 65 Organization Name Chemonics Kule Chitepo Mayford Manika Nkobi Moleele Vimbai Chasi Wonder Jonamu AWARD Sharon Pollard Derek du Toit Jan Graf Julia Williams Karen Kotschy RESILIM-B Interviewees Organization Name South Africa Africa Safari Foundation Steve Collins CRIDF Charles Reeves National Department of Environment, Forestry, and Fisheries (DEFF) Wadzi Mandivenyi Northwest Provincial Department of Rural, Environment, and Agricultural Development Mashudu Nemutandani National Department of Water and Sanitation (DWS) Tendani Nditwani Global Water Partnership (GWP) Andrew Takawira International Water Management Institute (IWMI) Karen Villhoth Marico Biosphere Reserve Beatrice van der Merwe Daan van der Merwe Matchboxology Paris Pitsillides OneWorld Belynda Petrie SADC James Sauramba Botswana Department of Meteorological Services Dorcas Masisi Department of Water and Sanitation Galejewe Kago Robert Wantle Saniso Sakuringo Thato Setloboko GIZ Martin Leineweber Thomas Schild Independent Consultant Nancy Kgengwenyane Kalahari Conservation Society Baboloki Autlwetse Keneilwe Mathaba Onkemetse Joseph Kgetsi ya Tsie (KYT) Women’s Group Mercy Masego OKACOM Phera Ramoeli The Nature Conservancy Sekgowa Motsumi Mozambique BIOFUND Maria Alexandra Jorge Center for the Sustainable Development of Coastal Zones (CDS-ZC) Henrique Balidy Micas Mechisso 66 Organization Name LIMCOM Sergio Malo Sergio Sitoe Zimbabwe CAMPFIRE Zimbabwe Kudakwashe Chigodo DAMBARI Wildlife Trust Nicky Pegg Verity Bowman Matobos Conservation Society (MCS) Gavin Stephens Jean Whiley Ministry of Lands, Water, Climate Change, Agriculture, and Rural Gilbert Mawere Southern Africa Research and Documentation Center (SARDC) Egline Tauya Zimbabwe Humanitarian Livelihoods Development Trust (ZHLDT) Norbert Dube Zimbabwe National Water Authority (ZINWA) Eng. Farai George Nzira Thulani Sibanda Tommy Rosen RESILIM-O Interviewees Organization Name South Africa Ba-Phalaborwa Municipality Johan Brehrens Nhlamulo Mdungazi Renske Venter Creating Sustainable Value Mike Ward Department of Agriculture, Forestry, and Fisheries Richard Green Siya Kobese Department of Environmental Affairs – Mpumalanga Office (MDEA) Brendon Mashabane National Department of Water and Sanitation (DWS) Betty Mnugni Harold van Niekerk Theresa Colyn Department of Water and Sanitation – Mpumalanga Regional Office Sydney Nkuna Nonceba Nogayi Steven Shibambu Institute of Natural Resources (INR) Fonda Lewis Kruger 2 Canyon Dimakatso Monyane Mari-Tinka Uys Lekgalameetse CPA (Communal Property Association) Aaron Mangena Mahlathini Development Foundation Erna Kruger Mopani Disaster Centre Bernnie Altenroxel Donald Makhubele Freda Mathebula Hannes Steyn South African National Biodiversity Institute (SANBI) Sagwata Manyike SANParks Eddie Riddell Frans Lesoka Ukuvuna Harvest John Nzira Water Group Holdings Marlene van der Merwe-Botha Water Resources Planning Consulting Engineers William Wegelin Mozambique 67 Organization Name Verde Azul Kemal Vaz Rui Mirira Questionnaire Results RESILIM-B Category Question Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Partner institutions RESILIM has contributed to strengthening collaboration among countries and regional institutions to manage the Limpopo Basin. 5 7 1 0 0 RESILIM has contributed to strengthening the enabling environment that supports LIMCOM and other regional institutions that manage the Limpopo Basin. 0 10 2 0 0 Sustainability of interventions Your entity has used or leveraged RESILIM’s interventions and results past the life of the program. 8 5 0 0 0 RESILIM implemented effective strategies to increase the likelihood of sustainability of its interventions. 1 7 2 3 0 Social inclusion RESILIM effectively engaged youth, gender (female and male), and socially marginalized groups in its outreach and activities. 3 5 4 0 0 RESILIM thoughtfully incorporated youth, gender sensitivity, and social inclusion considerations in the planning and delivery of its activities. 1 3 5 1 0 68 RESILIM-O Category Question Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Sustainability of interventions Your entity has used or leveraged RESILIM’s interventions and results past the life of the program. 3 10 1 2 2 RESILIM implemented effective strategies to increase the likelihood of sustainability of its interventions. 4 10 4 1 0 Social inclusion RESILIM effectively engaged youth, gender (female and male), and socially marginalized groups in its outreach and activities. 7 5 5 0 0 RESILIM thoughtfully incorporated youth, gender sensitivity, and social inclusion considerations in the planning and delivery of its activities. 7 5 3 1 0 69 ANNEX V. EVALUATION TEAM The core ICF evaluation team was composed of three evaluation Key Personnel. Dr. Molly Hellmuth served as the Team Leader and Technical Lead. Dr. Hellmuth was responsible for the overall implementation of the evaluation, including quality assurance and timeliness of evaluation tasks and deliverables. She also provided technical leadership on climate change and water resources. Dr. Andrés Gómez served as a Senior Evaluation Specialist. As a conservation biologist, Dr. Gómez provided deep technical and evaluation expertise to the team. Mr. John van Mossel served as a Senior Evaluation Specialist. He has extensive experience designing and evaluating climate resilience programs in Africa and brought his evaluation knowledge to this project. The core evaluation team received support from senior advisors and research analysts. Ms. Jessica Kyle served as a Senior Evaluation Advisor who provided senior technical backstopping and peer review. She provided leadership to the core team on evaluation design and developing evidence￾based findings and recommendations. Ms. Joanne Potter served as a Senior Resilience Advisor who provided senior level review of resilience principles and practice. She contributed to the review and analysis of findings and development of the final evaluation report. Mr. Mark Wagner served as a Senior Advisor who also provided senior technical backstopping and peer review. He provided his experience in project management, accounting, and quality assurance systems to ensure successful project delivery. Ms. Angela Wong, Ms. Maya Bruguera, and Ms. Jamie Liu served as Research Assistants who provided evaluative research and analysis support. The core evaluation team was also supported in the field by the following in-country Evaluation Specialists: • Ms. Zoe Parr (Botswana, South Africa) • Dr. Sam Braid (South Africa) • Ms. Maimuna Ibraimo (Mozambique, Portuguese Speaker) • Ms. Batayani Gwangwawa (Zimbabwe)