This publication was produced at the request of the United States Agency for International Development for the E3 Analytics and Evaluation Project. It was prepared independently by Fernanda Zermoglio of Palladium and by Gwynne Zodrow and Jorge Salinas of Management Systems International, A Tetra Tech Company. EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 FINAL REPORT MARCH 11, 2021 USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | i (THIS PAGE INTENTIONALLY LEFT BLANK) USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | ii EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENT SINCE 2012 FINAL REPORT Contracted under AID-OAA-M-13-00017 E3 Analytics and Evaluation Project Cover photo: Building capacity in forest inventory and monitoring techniques in the Democratic Republic of Congo. Credit: Eva McNamara, U.S. Forest Service International Programs. DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | iii (THIS PAGE INTENTIONALLY LEFT BLANK) USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | iv CONTENTS ACRONYMS......................................................................................................... VI EXECUTIVE SUMMARY ...................................................................................VII INTRODUCTION.................................................................................................. 1 BACKGROUND ON CLIMATE SERVICES ....................................................... 1 DEVELOPMENT HYPOTHESIS FOR CLIMATE SERVICES .......................................................................... 1 EVALUATION OVERVIEW................................................................................. 2 EVALUATION PURPOSE, AUDIENCES, AND INTENDED USES............................................................. 2 EVALUATION QUESTIONS................................................................................................................................. 2 EVALUATION APPROACH AND METHODS ................................................. 2 DATA COLLECTION............................................................................................................................................. 3 PHASE 1: SECONDARY DATA.................................................................................................................. 3 PHASE 2: DOCUMENT REVIEW ............................................................................................................... 4 PHASE 3: ONLINE SURVEY AND KEY INFORMANT INTERVIEWS ............................................. 4 DATA ANALYSIS..................................................................................................................................................... 5 EVALUATION LIMITATIONS.............................................................................................................................. 6 EVALUATION TEAM.............................................................................................................................................. 6 FINDINGS AND CONCLUSIONS...................................................................... 7 OVERVIEW OF THE CLIMATE SERVICES INVENTORY............................................................................. 7 DATA PROVISION ACTIVITIES................................................................................................................. 9 DECISION-FOCUSED SUPPORT ACTIVITIES....................................................................................... 9 LEARNING ACTIVITIES..............................................................................................................................11 GEOGRAPHIC DIFFERENCES IN CLIMATE SERVICES INVESTMENTS BASED ON THE SURVEY RESULTS...................................................................................................................................................................12 OVERARCHING LESSONS FROM CLIMATE SERVICES INVESTMENTS..............................................15 EQ 1: LESSONS LEARNED, GOOD PRACTICES, AND KEY CHALLENGES ALONG THE VALUE CHAIN......................................................................................................................................................................18 OVERARCHING CHALLENGES AND BARRIERS ..............................................................................18 A CONCEPTUAL MODEL OF THE USAID CLIMATE SERVICES VALUE CHAIN....................20 LESSONS AND GOOD PRACTICES ALONG THE VALUE CHAIN.............................................21 CONCLUSIONS FOR EQ 1: LESSONS LEARNED, GOOD PRACTICES, AND KEY CHALLENGES ALONG THE VALUE CHAIN ......................................................................................30 EQ 2: EXPECTED RESULTS FROM CLIMATE SERVICES INVESTMENTS.............................................30 CONCLUSIONS FOR EQ 2: EXPECTED RESULTS FROM CLIMATE SERVICES INVESTMENTS ............................................................................................................................................................................33 EQ 3: ELEMENTS OF ACTIVITY SUSTAINABILITY ....................................................................................33 CONCLUSIONS FOR EQ 3: ELEMENTS OF ACTIVITY SUSTAINABILITY................................37 GAPS IN THE CLIMATE SERVICES VALUE CHAIN....................................................................................38 FUTURE PROGRAMMING PRIORITIES..........................................................................................................39 REGIONAL BARRIERS.................................................................................................................................40 VALUE CHAIN NEEDS ...............................................................................................................................41 SUMMARY AND RECOMMENDATIONS........................................................42 KEY CONCLUSIONS AND GOOD PRACTICES........................................................................................42 USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | v RECOMMENDATIONS FOR FUTURE PROGRAMMING .........................................................................43 ANNEX 1: EVALUATION SOW.......................................................................45 ANNEX 2: INVENTORY OF USAID CLIMATE SERVICES ACTIVITIES REVIEWED...........................................................................................................56 ANNEX 3: DOCUMENTS INCLUDED IN THE DOCUMENT REVIEW .....58 ANNEX 4: CODES FOR PHASE 2 DOCUMENT REVIEW............................60 ANNEX 5: SURVEY INSTRUMENT .................................................................62 ANNEX 6: KEY INFORMANTS INTERVIEWED ............................................69 ANNEX 7: KEY INFORMANT INTERVIEW GUIDE ......................................70 ANNEX 8: ADDITIONAL SURVEY RESULTS................................................73 USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | vi ACRONYMS ADS Automated Directives System APIK Climate Change Adaptation and Resilience Project (Adaptasi Perubahan Iklim dan Ketangguhan) BRCCC Building Regional Climate Capacity in the Caribbean CCCCC Caribbean Community Climate Change Center CINSERE Climate Information Services for Increased Resilience and Productivity in Senegal CIMH Caribbean Institute of Meteorology and Hydrology COVID-19 Coronavirus Disease of 2019 DEC Development Experience Clearinghouse E3 Bureau for Economic Growth, Education, and Environment (USAID) ENACTS Enhancing National Climate Services EQ Evaluation Question GCC Office of Global Climate Change (USAID/E3) KII Key Informant Interview LAC Latin America and the Caribbean MCCAA Mali Climate Change Adaptation Activity MSI Management Systems International NGO Nongovernmental Organization PARA-Agua Partnering for Adaptation and Resilience-Agua PICSA Participatory Integrated Climate Services for Agriculture SMS Short Message Service SOW Statement of Work USAID United States Agency for International Development USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | vii EXECUTIVE SUMMARY INTRODUCTION This report provides results from a performance evaluation that the Office of Global Climate Change in the United States Agency for International Development’s (USAID’s) Bureau for Economic Growth, Education, and Environment commissioned to examine USAID’s climate services investments (e.g. activities, interventions, projects) since 2012 and identify lessons learned across various climate services interventions. The evaluation results will help to inform the predesign, design, and implementation of future Agency climate services investments, including whether to make such investments, what type of results can be expected, where to invest along the climate services value chain, and whether and how to scale up climate services activities. As global temperatures rise and rainfall becomes more variable, many development practitioners consider climate services to be a way to improve risk management and safeguard key productive sectors and vulnerable populations. Climate services involve the production, transfer, and use of climate information to help individuals and organizations make climate-smart decisions. Climate services equip decision makers in climate-sensitive sectors (e.g., agriculture, water, health, disaster risk management, transportation) with better information to help society adapt to climate variability and change. 1 Thus far, efforts to deploy climate services effectively have relied on diverse approaches, often building on previous investments and/or capacity assessments that laid the groundwork for documenting the core elements, the enabling conditions, or the development benefits of climate services. EVALUATION OVERVIEW EVALUATION QUESTIONS (EQS) The evaluation sought to answer three questions: 1. What lessons learned (including challenges and barriers) and good practices do USAID staff and other key partners perceive from USAID climate services investments since 2012? 2. What kinds of results can USAID expect from investing at different levels (e.g., institutional level, field level) of the climate services value chain? 3. What elements of USAID climate services investments are likely to be sustained beyond the end of an activity and why? APPROACH AND METHODS The evaluation was conducted in three phases, with each building on information obtained in the previous phase. In Phase 1, the evaluation team prepared an initial inventory of USAID’s climate services investments worldwide since 2012 that captured the investments’ key characteristics and results, based on available documentation and information provided by activity stakeholders. While the team sought to include all relevant USAID climate services investments, some may be missing due to the difficulty in identifying all activities that contained a climate services component and changes in staff. The team then used multiple methods to assess activities identified in the inventory to answer the EQs, including a document review during Phase 2 and an online survey and virtual key informant interviews during Phase 3. Field visits and in-person interviews were not possible due to COVID-19-related travel restrictions. 1 This definition is based on the World Meteorological Organization’s Global Framework for Climate Services site: https://www.wmo.int/gfcs/what-are-climate-services. USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | viii KEY FINDINGS AND CONCLUSIONS OVERARCHING CHALLENGES AND BARRIERS The climate services investments the evaluation examined faced challenges including issues of data (in terms of quality, timeliness, and management structures); confidence in the information being presented; the applicability and usability of the information; and the agency of the target beneficiaries. USAID climate services activities have successfully addressed many of these challenges. Despite the diversity of approaches taken and regardless of the entry points of investment in the value chain, several lessons about the overarching challenges and barriers emerged that can inform future climate services programming. ● Poor interoperability of data management systems: weather and climate data management systems are often outdated, making data access difficult for both users and those interested in improving services such as forecasts. ● Limited reach and adequacy of communications and translation systems: meteorological warnings, for example, come with several challenges. First, stakeholders cannot benefit from warnings if they do not receive the information – either because they are out of radio and mobile network coverage or because the climate information is delivered too late. Second, illiterate stakeholders may find it difficult to grasp written information about risks associated with climate services. Third, the information is often not sufficiently tailored to user needs. Finally, inconsistent funding sources for climate services delivery often undermine these services’ effectiveness by negatively affecting sustainability. ● Lack of end user confidence in the information: the historical disconnect between meteorological services, researchers, and decision makers – or the poor quality or insufficient explanations of the inherent uncertainties in any forecast – can diminish the confidence users place in the information and thus in their willingness to act in response to it. ● Insufficient consideration of the needs and agency of marginalized groups in project designs: the information needs of women, youth, ethnic minorities, the landless, and other marginalized groups – in terms of timing, delivery mechanisms, and variables of interest – can significantly differ as their roles and ability to respond are inexorably tied to their status and social roles. ● Limited expertise in using climate information: although weather and climate information has been available for decades, the ability to use this information to inform decisions has been limited. This is partly because the information is often not available at the scales of interest (both in space and time) but also because target users are often unsure how to apply forecasts and other information in their decision-making processes. ● Lack of access to resources and assets (e.g., land, input, equipment) required to act on the information in a timely manner: the Learning Agenda for Climate Services in sub￾Saharan Africa, for example, explored how gender can influence agency and access to climate services. The differential resources and influence under women’s and men’s control affect their ability to make use of climate services as well as the needs and demands for information2. ● Inadequate modelling and forecast capacities: a lack of capacities can pose seemingly insurmountable barriers and require sustained, extensive investments before the potential benefits of a climate service can be realized, if at all. 2 See: https://www.climatelinks.org/resources/gender-responsive-rural-climate-services-review-literature USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | ix CONCEPTUAL MODEL OF THE USAID CLIMATE SERVICES VALUE CHAIN The conceptual model used here to frame USAID’s climate services investments along a value chain (Figure 1) represents a continuum of activities from translating and packaging climate and weather data to information to dissemination and communication ultimately leading to informing action. The production of data and systematic observations are the foundation of informed decisions, but they must be packaged and communicated effectively for those who need to act. Lessons are captured below within each component to identify entry points for climate service investments. However, it is well understood that in practice these components function within a complex continuum in which each component builds and loops back on the other to form an interactive system that delivers value by supporting a goal or outcome (e.g., reducing vulnerability, increasing resilience, safeguarding livelihoods). FIGURE 1: CONCEPTUAL MODEL OF USAID’S CLIMATE SERVICES VALUE CHAIN • Data: facts or systematic observations collected from the environment or records as accurately and as timely as possible, not all of which will be relevant to the decisions made. Relevant data are distilled and synthesized in context to become information. • Information: data interpreted in a specific context to best support the decisions stakeholders need to make. Information is communicated either directly or in an interactive process, via fit for purpose communication channels to those who need to act. Communication alone does not guarantee that users will take advantage of the information. Activity experiences point to the need to provide resources and skills to take advantage of this new information in making changes to practice. • Action: what someone does in response to available information. Examples cover a range of sectors and users, including farmers’ decisions about when and what to plant and water management action related to floods. This model applies across all USAID-supported climate services activities. In practice, each climate services investment worked on several components of the value chain with varying emphasis to achieve the stated goals, building on local and regional realities. Decisions on where in the value chain to invest were driven largely by the local context, stakeholders’ requests and needs, and the role and investment plans of other donors in the space. OVERVIEW OF USAID CLIMATE SERVICES INVESTMENTS The evaluation identified and examined 40 USAID climate services investments ranging from learning agendas to field activities/projects. The largest regional grouping was in Africa, followed by Latin America and the Caribbean and then Asia. Most investments worked in specific sectors (e.g., water resource management, infrastructure, biodiversity conservation, agriculture and food production including livestock rearing and fisheries). USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | x These investments fell into three broad categories of activities: (1) data provision: focused on data including data capture and building data providers’ capacity; (2) decision-focused support: targeted investments across the data-to-action continuum to support improved decision making and planning; and (3) learning: aimed at synthesizing lessons, good practices, barriers, and opportunities in climate services. Several of the 40 investments engaged in interventions across the three activity categories. EQ 1: LESSONS LEARNED AND GOOD PRACTICES Overarching lessons learned from USAID’s climate services investments included: ▪ Climate information services have played a significant role in achieving place-based resilience, with the resulting information products informing sound, evidence-based decision making across many USAID activities. As weather patterns become more variable and the manifestations of climate change bring uncertainty, the need for climate services has grown significantly and the ways these investments have addressed identified barriers have increased their impact. In Rwanda, for example, participating farmers increased yields by 47 percent and their income from crops by 56 percent. ● Inexpensive or low-tech solutions such as stakeholder-monitored rainfall gauges and digitized historical data are not only cost-effective means of filling data gaps and building local information; they also build confidence in the climate information presented. Reliable meteorological information is essential to plan for weather impacts and to better understand and identify future climate change impacts. As climate change makes weather patterns less predictable, the reliability and accuracy of meteorological information (products and services) are critical. Moreover, the diversity of agro-ecological zones across nations makes national-scale weather forecasts relatively useless for local planning. Detailed data specific to local characteristics are necessary to make relevant regional or district-level forecasts. Digitizing historical data enhances meteorological agencies’ ability to show and assess climate variability over time. In Tanzania under the Planning for Resilience in East Africa through Policy, Adaptation, Research and Economic Development activity, digitizing historical data helped strengthen datasets and models, improve national and regional forecasts, and add to information that future generations will be able to use. While more technologically advanced investments (e.g., hydrometeorological stations) are needed, inexpensive or low-tech approaches can add local specificity to available climate and weather information. ● A participatory, user-centered activity design can build trust in the products and thus foster buy-in and long-term commitment to information use. Many USAID climate services investments have demonstrated that climate information adoption rates are based on the perceived utility of the information and adaptation techniques suggested. If households or target stakeholders are aware of the climate information and adaptive technologies offered and have access to them, they will assess their utility and value before they decide to act on the information. A critical factor for success is to work with key stakeholders in an “incubation” period prior to activity implementation to define their options, help them understand the risks they face, and determine what additional support they require to make decisions informed by climate information. This participatory process helps address issues that feed into user assessments (e.g., trust, reliability, relevance, timeliness) by responding specifically to their needs, allows for the proper evaluation of population needs, and offers the opportunity to tailor approaches to meet stakeholders’ needs. Key results from the evaluation’s examination of lessons learned and good practices from across the climate services value chain included: USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | xi ● Action: Climate services are important in helping stakeholders manage climate-related risks and adapt to climate change. However, climate services alone are not sufficient to build resilience. It is also critical to help stakeholders understand what they can do on the ground with the climate information and provide resources for action. Such support can include technical assistance and resources such as improved seeds and climate-smart agricultural practices. There are several examples of climate services demonstrating value in risk management to safeguard lives and/or increase productivity. ● Information: The information component offers an opportunity to translate signals of climate variability and change (e.g., drought, temperature changes) into meaningful indicators for target sectors (e.g., agriculture, hydrology, health), so that salient and relevantly synthesized data are distilled and subsequently communicated to the intended recipients. Investments in the information component have also offered an opportunity to foster engagement and build both supply and demand for climate services. However, as noted above climate services alone are not sufficient to build resilience. They require resources and technical support to those who need to act. ● Data: Systematic observations are fundamental to providing climate services, whether from “rescued” (i.e., digitized) paper records to build a robust historical knowledge base; newly collected through automatic weather stations; blended, remotely sensed products; or crowdsourced techniques such as rain gauges. The primary goal is to improve the resolution and localized quality of available data. However, systematic observation involves not only collecting data but also supporting the capacity of meteorological services to maintain weather stations; improving the provision of critical spare parts for this maintenance through public-private partnerships; and managing, analyzing, and distributing that data to key stakeholders. Sustainability in data production requires supporting the business models to support financial management, procurement processes, and human resources necessary to maintain these observational networks. EQ 2: EXPECTED RESULTS FROM CLIMATE SERVICES INVESTMENTS Survey respondents and interviewees noted several successful outcomes for climate services investments, including how these investments played a significant role in: ● Building resilience to climate shocks by improving productivity, planning, and response capacities at localized scales. Climate services have a demonstrable role in this space. In Rwanda, for example, farmers participating in climate services investments increased yields by 47 percent and income from crops by 56 percent. ● Establishing robust baselines and a knowledge base to inform climate risk management and development activities. The Caribbean Institute of Meteorology and Hydrology’s strengthened capacity allowed it to develop the first analysis of climate trends and projections for the Eastern Caribbean region. This analysis sets the scientific foundation for the Organization of Eastern Caribbean States’ forthcoming climate change adaptation strategy, steering regional efforts and opportunities. ● Strengthening data analysis to improve decision making. In the Andes region of Latin America, for example, a hydroclimatological monitoring system for the Guatapuri river watershed is now operational, providing early response capacities to safeguard the lives of approximately half a million Colombians from flooding. ● Strengthening awareness and understanding of climate variability and change as well as adaptation needs, including the scientific foundations of these needs and potential responses. In Indonesia, climate adaptation and disaster management activities such as mangrove USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | xii protection/reforestation and silvofisheries are now funded by government budgets thanks to the incorporation of climate adaptation and disaster management activities into village plans. EQ 3: ELEMENTS OF ACTIVITY SUSTAINABILITY USAID’s experiences have demonstrated that building the sustainability of local climate services goes beyond simply providing support to create information and deliver weather and climate advisories. Providing financial support for the enabling environment – the conditions that can facilitate adoption of climate-informed practices – can help targeted stakeholders confront the risks that may deter them from a proactive stance. Examples of key programming modalities that enhanced sustainability through enabling environments included: Integration of climate considerations into relevant plans and policies, which can help secure financial support for climate services. Integrating climate risks and adaptation concerns into development plans can boost their prominence in budgeting processes, potentially ensuring the availability of the requisite financial resources. USAID/Indonesia’s five-year Climate Change Adaptation and Resilience activity (APIK) supported the Indonesian government to strengthen climate and disaster resilience. APIK worked in an integrated manner from the national level down to the regional and community levels. For example, APIK worked with the National Adaptation Plan Secretariat in partnership with the Indonesian Ministry of National Development Planning to integrate climate resilience into the new National Midterm Development Plan, 2020–2024. The plan now includes a priority to enhance development efforts by addressing environment, disaster resilience, and climate impacts together. APIK led scientific research and cost-benefit assessments for the Secretariat revision, which was an essential input for the plan and resulted in $2.4 billion being allocated to climate resilience in the agriculture, water, coastal/marine, and health sectors. APIK then worked with the government on a detailed annual plan to translate this into concrete activities across the country. Demonstration of proof of concept to secure public-private financing. Following on the successful implementation of weather and climate services in support of increased productivity for farmers, livestock herders, and fisherfolk, the Climate Information Services for Increased Resilience and Productivity in Senegal activity conducted a business case analysis to identify innovative strategies to sustain weather and climate services provision and transition to service delivery on a user-pay basis. The activity also identified weather and climate services production cost-recovery opportunities for the national meteorological agency. In addition, the activity sought to build a sustainable business case for weather and climate services in Senegal by identifying partnership opportunities with the private sector along a structured value chain. The study assessed the market opportunities for weather and climate services by mapping, sizing, and characterizing potential users and understanding target clients’ pain points, motivations, and willingness to pay. It also identified potential private-sector partners interested in operationalizing the business case and other required partners by exposing business models along the activity’s weather and climate services value chain. The analysis roadmap offers important insights on the engagement process required to support the evolution of a climate services investment into a viable business case. Promoting the sustained involvement of relevant partners and institutions. Engaging local stakeholders such as universities, nongovernmental organizations, and private businesses enhances the potential for lessons learned and capacity building to remain beyond the activity. In the Philippines’ Be Secure activity, members of the outreach, research, and planning departments of the Central Philippines University in Ilolo City actively participated in the participatory climate vulnerability assessment, disaster risk management climate change adaptation orientation, and development of local climate change action plans, as well as the disaster resilience and risk management plan. Their experiences in these processes led to the integration of disaster resilience and risk management awareness into the curriculum of their USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | xiii national service training program courses. San Augustin University, which offers architecture degrees, also engaged in the activity and is now looking to develop course materials in environmentally friendly and disaster-resilient structures. Development of context-relevant tools to support climate-resilient decision making. USAID’s Climate Change Adaptation Project worked in partnership with the Caribbean Community Climate Change Center to assist 10 Eastern and Southern Caribbean Community countries in their responses to climate change by developing and implementing adaptation policies and initiatives for sustainable economic development and disaster risk reduction and to achieve maximum impact from the scarce resources employed. The project developed an online decision support system for climate-resilient decision making that helps users in the Caribbean determine the most appropriate action in response to a variable and changing climate. The system was socialized and disseminated throughout the region. Grenada, St. Kitts, and Antigua and Barbuda formally adopted it as a decision support instrument, and Guyana is embedding it into the country’s National Climate Change Adaptation Plan. Establishment of institutional arrangements and relationships to promote lasting impact. Efforts to support informed action using climate services are constrained by a lack of coordination among relevant institutions whose support (e.g., technical, resources, policies) decision makers need to act. Strengthening institutional collaboration can safeguard investments made by: • Ensuring climate services data provision continues beyond an activity. APIK partnered with OpenStreetMap Indonesia Group to set up a geographic information system–based platform to support its climate and disaster information database management system. The OpenStreetMap team has committed to host the platform, a promising commitment for the sustainability of APIK’s work. • Establishing a collaboration plan. The Caribbean Climate Change Center activity, which funded work on data management in Belize, also established a formal collaboration plan for securing partners’ engagement as well as their roles and responsibilities in program implementation. The activity also guided the development of several Green Climate Fund proposals in the region. GAPS IN THE CLIMATE SERVICES VALUE CHAIN Despite significant recent progress in addressing the overarching barriers and challenges mentioned in the preceding sections, it could be argued that most of USAID’s climate services investments to date have been proof of concept activities that will need to evolve into concerted investments that continue to safeguard vulnerable livelihoods at scale as the climate continues to change. Survey respondents identified the following programming gaps, which align well with the challenges and barriers the evaluation team identified along the value chain: ● Action: little work has been dedicated to understanding how a changing livelihood and political landscape will alter the need for and therefore the use and value of climate services on the social constraints and opportunities to use climate information. o Communication and resources: further exploration is needed of the roles of gender, social equity, and agency in how information is consumed and utilized so appropriate communication channels and relevant resources are coupled to reach different users. ● Information: substantially more work is needed to define the information needs of various user communities. Existing studies are very context specific, so having more cases and applications could help identify shared challenges across contexts and user groups, supporting scale-up. For example, do commonalities exist in the information needs of rainfed maize farmers subject to prolonged dry periods and could this help inform climate services programs in other locations? USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | xiv o Distillation and analysis: sectors and risks that were not the focus of USAID’s climate services investments, in particular, will require further analysis and a grounded determination of indicators of interest, as well as their uncertainty bounds. ● Data: continued investments are needed to build out observation networks and capacity for managing these networks, including repairs and acquisition of spare parts through local businesses, harmonization of data management tools to enable faster and seamless analyses, and continued training and capacity building in key data analysis methods such as drought forecasts. The rescue and digitization of historical climate data has proven to be a low-cost opportunity to build an early foundation for developing countries and should continue. While the private sector has begun to fill some of these gaps3, the challenges – particularly for areas that do not currently offer market opportunities – are large. FUTURE PROGRAMMING PRIORITIES ACROSS REGIONS The evaluation’s findings also point to important regional differences in the programming priorities on climate services. For example, respondents from Asian countries indicated a need to focus on extreme heat, drought, sea level rise, and saltwater intrusion. Respondents in Latin America and the Caribbean suggested that future climate services programs should focus on urban water management and building resilience in the hydropower production sector. Respondents from African countries suggested that the risks being addressed (e.g., extreme events, altered rainfall patterns, longer dry periods) should continue to be emphasized, and extreme heat events should be examined for future interventions. There is also regional diversity in the challenges to implementing climate services activities. Respondents working in Asia emphasized a need to increase knowledge and awareness of climate risks, address other priorities competing for funding and political attention, and translate climate information into understandable formats. Those working in Latin America and the Caribbean cited challenges including limited availability of usable data, the need to address a diverse user base, and poor coordination among donors and relevant institutions. Respondents in Africa cited limited observation networks, the cost of meteorological equipment, limited granularity of forecasts, poor communication of the concrete impacts of investing in climate services, and underutilization of critical observation networks as priority challenges to be addressed. There were also important regional differences in the barriers limiting use of climate and climate risk data. In Asia, respondents prioritized a lack of awareness of available information and limited technical expertise in using climate data, whereas those in Africa prioritized limited availability of relevant data, lack of awareness of available data, limited technical expertise in using climate data, and barriers related to data access, particularly in remote areas. RECOMMENDATIONS FOR FUTURE CLIMATE SERVICES INVESTMENTS Climate services have evolved significantly in the last decade, offering critical insights and valuable proof of concepts to inform future initiatives. Nevertheless, more remains to be done. The evaluation team recommends USAID take the following actions to guide future climate services investments: • Widely share diverse examples of climate services programs. Despite a growing demand for examples of climate services programming, there are relatively few in the broader development community that go beyond a simple description. 3 https://www.climatelinks.org/sites/default/files/asset/document/2020_USAID_Learning-Agenda_Spotlight-Series-Private-Sector￾Solutions-for-Climate-Services.pdf USAID. GOV EVALUATION OF USAID’S CLIMATE SERVICES INVESTMENTS SINCE 2012 | xv • Prioritize momentum and experience on the ground, which are catalysts for scaling climate services programming in the current geographic focal areas. Since engagement and consultation – particularly at localized scales – are costly and time intensive, many activities had to scale back their pilot investments to account for funding constraints. However, across activities, the trust built with target communities and the improvements supported in dialogue with key research entities and meteorological services remain. • Finance studies that explore how to build financial sustainability in climate services investments. Market potential studies for climate services show consistent benefits to target populations, reflected in their willingness to pay for the services and demonstration of increases in yields. There are clear opportunities to continue to explore the “business case” for climate services, which offers the promise of sustainability beyond donor financing. This could include a mix of public and private financing that fills critical gaps in available resources for effective climate services delivery while also considering equity issues in cost-recoverable models. • Continue investments in activities that fill data gaps critical for effective climate services. The quality, timeliness, and lead time of critical weather and climate information needed to support climate services have markedly improved in recent years, in part due to scientific and technological advances, an enhanced understanding of user needs, and significant experiences in the deployment of these services around the world. The generation and use of climate information continue to require donor support, particularly in light of limited public budgets. • Pay attention to remaining research gaps, particularly for sectors and risks that were not the focus of previous USAID climate services work. There is abundant evidence across the sectors within which climate services worked (e.g. agriculture, livestock production, water) that climate services have saved lives and livelihoods. However, for other sectors of priority to USAID, such as health and specifically heat risks, fundamental questions will need to be addressed before the role and potential benefit of climate services can be realized. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |1 INTRODUCTION The Office of Global Climate Change (GCC) in the United States Agency for International Development’s (USAID’s) Bureau for Economic Growth, Education, and Environment (E3) commissioned USAID’s E3 Analytics and Evaluation Project4 to conduct an evaluation of the Agency’s climate services investments since 2012. The evaluation identified lessons learned and best practices across these climate services investments to help inform future USAID programming. Annex 1 provides USAID’s statement of work (SOW) for the evaluation. BACKGROUND ON CLIMATE SERVICES Climate services equip decision makers with better information to help society adapt to climate variability and change.5 Climate services involve the production, transfer, and use of climate information to help individuals and organizations make climate-smart decisions. National and international databases provide high-quality data on temperature, rainfall, wind, soil moisture, and ocean conditions. Collected data are transformed into customized informational products such as projections, trends, and services for various user communities. Climate services help guide countries, communities, households, and individuals to anticipate and manage climate and weather risks and opportunities. Climate services must be timely, spatially appropriate, relevant, accurate, and understandable and can take a variety of forms, such as seasonal forecasts, weather information, drought forecasts, and flood early warnings. USAID and other development organizations invest in activities to support climate services across the developing world, working to build resilience to weather and climate variability as well as to support long￾term planning in the face of climate change. These investments have taken a variety of approaches, targeted multiple actors in the climate services value chain (e.g., action, communication, information, and data),6 engaged a range of partners, and sought to develop a variety of usable products. Annex 2 lists USAID’s climate services investments that the evaluation team identified. DEVELOPMENT HYPOTHESIS FOR CLIMATE SERVICES A theory of change for climate services7 is that providing decision makers at various levels (e.g., government, builders, farmers or local health care workers) with sufficient, usable, quality, and reliable climate information and data will help inform decision making, reduce climate change risks, and improve livelihoods by facilitating adaptation. Climate services investments can lead to more effective and ultimately cost-saving implementation of adaptation measures. 4 Management Systems International (MSI) implements the E3 Analytics and Evaluation Project in partnership with Palladium and NORC at the University of Chicago. 5 This definition is based on the World Meteorological Organization’s Global Framework for Climate Services site: https://www.wmo.int/gfcs/what-are-climate-services. 6 A climate services product cycle is an end-to-end process that comprises the consideration of all required processing steps from data to decision-making as well as all involved stakeholders. Such an end-to-end production cycle is characterized by one or several steps of value adding, which might be tailoring of data or provision of information and services, etc. to make climate information usable. Source: (GIZ 2018). 7 See Global Framework for Climate Services: https://gfcs.wmo.int/what-are-climate-services. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |2 EVALUATION OVERVIEW EVALUATION PURPOSE, AUDIENCES, AND INTENDED USES This evaluation identified lessons learned and good practices from successes and challenges experienced by USAID’s climate services investments in partner countries since 2012. The evaluation results will inform decisions on the predesign, design, and implementation of future USAID climate services investments, including whether to make such investments, the expected results, where to invest along the climate services value chain, and whether and how to scale up climate services activities. The evaluation’s intended audiences in USAID include the cross-bureau adaptation team supporting Washington- and mission-based Agency staff, the new Bureau for Resilience and Food Security, and missions and other operating units designing and providing climate services activities across countries and sectors. EVALUATION QUESTIONS The evaluation sought to answer three evaluation questions (EQs) per USAID’s approved SOW (Annex 1) that fell under three broad themes: lessons learned (EQ1), expected results (EQ2), and sustainability (EQ3): 1. What lessons learned (including challenges and barriers) and good practices do USAID staff and other key partners perceive from USAID climate services investments since 2012? a. How do lessons learned and good practices vary at different points in the climate services value chain (e.g. data collection, capacity building, dissemination)? b. How do these results translate into recommendations for the design of new climate services activities? 2. What kinds of results can USAID expect from investing at different levels (e.g., institutional level, field level) of the climate services value chain? a. In what ways have climate services investments added value or contributed to other USAID activity components? 3. What elements of USAID climate services investments are likely to be sustained beyond the end of an activity, and why? a. What types of enabling environments/local contexts appear to support positive outcomes across different levels of climate services investments? EVALUATION APPROACH AND METHODS This evaluation was conducted in three phases, with each phase building on information obtained in the previous phase. In Phase 1, the evaluation team prepared an initial inventory of USAID’s climate services investments worldwide since 2012 that captured key characteristics and results based on available documentation and information that activity stakeholders provided. In Phases 2 and 3, the team used multiple methods to assess activities identified in the inventory to answer the EQs, including a document review during Phase 2. In-person key informant interviews (KIIs) and field observations were originally envisioned for Phase 3. However, due to travel restrictions stemming from the Coronavirus Disease of 2019 (COVID-19), the team instead conducted an online survey and virtual KIIs. This evaluation relied mostly on secondary data and information from a sub-set of the USAID climate services community and activities. This evaluation was not designed to be representative of all of USAID USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |3 climate services programming or staff, but rather to collect key lessons learned across global investments utilizing available data and internal USAID knowledge. DATA COLLECTION As detailed below, the team used a mixed-methods approach that included: • A review of secondary documents to collect data for the inventory of 40 identified USAID climate services investments (Phase 1). • A qualitative document review of 44 documents covering 18 climate services activities in Latin America and the Caribbean (LAC), Africa, and Asia (Phase 2). • An online survey with 50 respondents and 9 KIIs with regional technical staff and climate integration leads across all regions (Phase 3). PHASE 1: SECONDARY DATA In Phase 1, the team compiled a list of all USAID-funded climate services investments (e.g., projects, activities, research) it could identify and gathered key information across these investments to create an inventory. This inventory, structured in a matrix, included the following key data for each investment: location, status, budget, description/objectives, funding dates, entry point in the climate services value chain, service type, scale of service, sector focus, and variables measured. The team was not able to fill out all columns for every activity based on the available documentation. Budget information was particularly difficult to find. In addition, the team separated all identified activities into three types: learning activities, data provision activities, and decision-focused support activities (discussed further in subsequent sections). This breakdown helped to frame the analysis and allowed the team to examine different investment types and better understand their goals and scope. Annex 2 lists the investments the team identified. Identified investments ranged from activities to improve climate data (including investments in meteorological stations) to decision-focused support activities to provide better information for targeted beneficiaries, such as farmers to decide when to plant their crops. The review included all levels of climate services investments – a considerable task given the diversity of activities USAID has supported in recent years. The team collected general information about all activities, but to accommodate USAID’s request for a relatively rapid inventory, the team did not comprehensively examine all activities but instead more closely examined decision-focused support activities. The team identified 40 climate services investments that covered 40 countries and 3 regions. The largest regional grouping was Africa, followed by LAC and then Asia (Figure 2). Most investments in Africa were in Sub-Saharan Africa. In South America, activities were implemented in Colombia and Peru, and in Central America activities were implemented throughout the region. In the Caribbean, activities were implemented regionally and in Barbados. In Asia, activities were implemented in Indonesia, Kazakhstan, Mongolia, Nepal, the Philippines, Tibet, and Vietnam. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |4 FIGURE 2: GEOGRAPHIC DISTRIBUTION OF CLIMATE SERVICES INVESTMENTS IN THE INVENTORY (N=40) To identify these investments, the evaluation team started with an initial list of climate services investments that USAID’s adaptation team provided. The evaluation team then compiled a more complete inventory based on data it collected from a variety of sources, including an internet search of USAID’s Development Experience Clearinghouse (DEC) and climatelinks.org resources; key USAID regional bureau staff for Africa, Asia, and LAC; mission staff involved in the activities; and a few implementing partners. Notes from informational interviews and information from relevant reports (e.g., annual and final reports, case studies, performance evaluations, monitoring and evaluation plans, technical reports) provided information to fill in the inventory. PHASE 2: DOCUMENT REVIEW In Phase 2, the evaluation team mined 44 documents for 18 key activities in the inventory across all regions to draw lessons learned and good practices across USAID’s climate services investments since 2012. The team gathered relevant documents that it could not obtain during Phase 1 through resources from climatelinks.org, the DEC, and USAID staff and stakeholders. After reviewing all collected documents, the team selected the 18 key activities based on the availability of final or annual reports, evaluation reports, and other key learning agendas. The team used the Dedoose online platform to upload, organize, and code each document. Annex 3 lists the documents the team reviewed, and Annex 4 provides the codes the team used for the review, which were informed by the EQs. PHASE 3: ONLINE SURVEY AND KEY INFORMANT INTERVIEWS Online Survey In Phase 3, the evaluation team administered an online survey targeted at learning from USAID activity key points of contact (e.g., contracting or agreement officer’s representatives, implementing partner staff such as the chief of party or deputy chief of party, host country officials, other appropriate USAID staff) for each climate services investment identified in Phase 1. The team used the online survey tool SoGoSurvey to develop, administer, and manage the survey. USAID sent the survey link to 80 people (65 USAID staff and 15 implementing partner organizations). USAID respondents were selected to represent Africa (17) 43% Asia (7) 18% LAC (16) 40% USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |5 all regions, with a focus on regional technical staff and climate integration leads. Implementing partners were from the 18 key activities identified in Phase 2. The survey took about 15 minutes to complete and consisted of multiple choice and open-ended short answer questions that were based on document review findings, to help validate findings and collect additional data related to the EQs. USAID sent multiple reminder emails to encourage participation. The survey remained open from August 3 to 13, 2020. Fifty people responded8 (39 USAID staff and 11 implementing partners). Annex 5 provides the survey instrument and Annex 8 provides detailed survey results. Key Informant Interviews In Phase 3, the team conducted KIIs with key USAID climate services stakeholders across various missions and USAID/Washington bureaus. The team requested interviews with 30 people but only conducted 9 interviews with 12 USAID respondents, based on responses and availability.9 Annex 6 lists the interviewees. Due to COVID-19-related travel restrictions, the team conducted all KIIs remotely through Google Meet. The team recorded all interviews with the respondents’ informed consent, and then had a transcription firm produce interview transcripts for analysis. The selection of interviewees was based on findings from Phases 1 and 2 as well as USAID input. The team conducted all interviews in August 2020. Annex 7 provides the semi-structured KII guide the team used for all interviews. The interviews lasted about 60 minutes. All interview respondents received the online survey link as well as a copy of the interview guide. Privacy and Confidentiality Considerations The evaluation team obtained informed consent from respondents before carrying out the online survey and KIIs. All responses to the online survey were confidential and anonymous. The team recorded KIIs through Google Meet and all respondents were asked for separate consent to record. DATA ANALYSIS The team analyzed data and information from each phase, building on the learning through each phase. The evaluation findings are based on data and information from the Phase 1 inventory (across 40 activities), Phase 2 qualitative data from reports covering 18 of those activities, and Phase 3 responses from the 50 online survey responses and 9 KIIs. Pattern/Content Analysis The evaluation team examined qualitative data from activity documents, KIIs, and the online survey for patterns and comparisons. The team identified themes and trends of good practices, challenges, barriers, and outcomes relevant to each EQ to better understand meaning and context. The team coded all activity documents in Dedoose. The team analyzed information by the codes and then summarized and compiled relevant information to answer the EQs. During Phase 3, the team analyzed KII transcripts using content analysis techniques to code text according to key EQ themes. The team summarized responses related to each EQ, including highlighting outlier 8 The survey response rate was 62.5 percent, which is above average for online surveys to the public according to general literature and likely reflective of the age of the activities (e.g., some of the respondents’ contact information may have been outdated and incorrect). 9 The response rate for the KIIs was 30 percent. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |6 responses and experiences. The team triangulated data from activity documents, KIIs, and the online survey to determine key findings for each EQ. Descriptive Analysis Using data from the inventory categories (i.e., data, decision-focused, and learning) and the online survey’s multiple-choice questions, the team calculated summary statistics to analyze the distribution and number of responses. The team also used outputs from SoGoSurvey and Excel to produce cross-tabs and other comparative analyses to explore data for detailed findings based on geographic location and activity sector and to differentiate between respondent types (e.g., USAID positions). EVALUATION LIMITATIONS The evaluation team recognizes the following limitations and challenges with the evaluation approach, and notes how it sought to mitigate these issues: ● Response Bias. The evaluation’s analysis is only as strong as the data and information provided by stakeholders. The KIIs and the online survey both collected data based on respondent experiences and perceptions. The team sought to mitigate this risk by triangulating data sources. ● Recall bias: Also, given that some activities took place several years ago, recall bias could be an issue. Although there is always a risk of response and recall bias, the team pretested the instruments, and kept the survey short and user friendly. ● Attribution: A nonexperimental methodology lacks direct linkages to attribution. However, the evaluation approach of exploring multiple perspectives allowed for a deeper understanding of the climate services investments and their results. The evaluation could not test for causality directly and did not use rigorous methods to determine attribution, but rather sought to identify key contributing factors. ● Limited observation: Due to COVID-19-related travel restrictions, the team could not visit any sites or conduct in-person interviews. This limited the team’s ability to explore climate services interventions firsthand. The team worked to compensate for this restriction by conducting a wider range of interviews across all regions and by relying on multiple methods of data collection. ● Information gaps: Although the team made every effort to secure resources for each activity it examined, the available documentation varied. Some activities had many reports available (e.g., annual or final reports, evaluations) while others were limited to just annual reports or presentations. In addition, the team was unable to conduct any field visits or collect data from direct users or key stakeholders such as government officials. This could certainly have influenced the results of the evaluation’s analysis. EVALUATION TEAM The core team for this evaluation consisted of Team Leader Fernanda Zermoglio, Evaluation Specialist Gwynne Zodrow, and Researcher Jorge Salinas. • Ms. Zermoglio is an adaptation specialist with extensive experience in the development and deployment of pragmatic tools and approaches to support decision managers in adaptation planning. She was responsible for the quality of the evaluation design and its execution, including the evaluation report. • Ms. Zodrow is a technical manager with MSI who provides monitoring and evaluation support to multiple government and private sector clients in areas including health, agriculture, and food USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |7 security. She supported Ms. Zermoglio in developing the evaluation methods and data collection instruments and with data collection, analysis, and reporting. • Mr. Salinas has completed extensive coursework in global environmental policy and has managed USAID projects in Mexico and elsewhere. He worked closely with Ms. Zermoglio and Ms. Zodrow to support the development of the inventory and data collection and analysis activities. The E3 Analytics and Evaluation Project home office supported the core team with quality assurance, administrative oversight, and logistical support. FINDINGS AND CONCLUSIONS OVERVIEW OF THE CLIMATE SERVICES INVENTORY Of the 40 USAID climate services investments since 2012 that the evaluation team identified, the largest regional grouping was in Africa (17), followed by the Latin America and the Caribbean (LAC, 16) and Asia (7). Looking at sub-regions (Figure 3), most climate services investments in Africa were in Sub-Saharan Africa. In South America, climate services activities were implemented in Colombia and Peru. In Central America, activities were implemented in Honduras, Nicaragua, Costa Rica, Guatemala, and Panama, while in the Caribbean, activities were implemented in Barbados, Guyana, Jamaica, the Dominican Republic and the Eastern Caribbean states. In Asia, activities were implemented in Kazakhstan, Vietnam, Indonesia, Nepal, Tibet, Mongolia, and the Philippines. FIGURE 3: GEOGRAPHIC DISTRIBUTION OF NUMBER OF CLIMATE SERVICES INVESTMENTS BY SUB-REGION (N=40) Most of these investments were in specific sectors such as infrastructure, water resources, agriculture, or food production. The investments either fell strictly within three overarching activity categories or worked 0 1 2 3 4 5 6 7 8 West Africa East Africa Southern Africa Across Sub-Saharan Africa South America Caribbean Central America South East Asia Central Asia Pacific Across Asia Number of Projects in Inventory USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |8 across them (Figure 4). The team used this initial categorization to define a subset of the available activities for further analysis and it informed the conceptual value chain model of climate services investments presented later in this report. ● Data provision activities: focused on data overall, including data capture, whether through data rescue activities to digitize historical records or the installation of weather and other monitoring stations, as well as strengthening data providers’ capacity to deliver timely and trusted information. ● Decision-focused support activities: targeted investments across the data-to-action continuum to support improved decision making and planning. Whereas data provision activities strictly focused on the aspects noted above, decision-focused support activities often also included data provision components along with more action-oriented components. ● Learning activities: aimed at synthesizing lessons, good practices, barriers, and opportunities in climate services. Several investments in the inventory engaged in interventions across these categories. For example, an activity may have invested in strengthening capacity for data provision or supported research for specific challenges while also engaging stakeholders in beneficiaries’ use of this information. The team catalogued these activities in the inventory as decision-focused support as they incorporated all climate services components. As shown in Figure 4, most activities in LAC were focused on data provision (69 percent of all climate services investments in the region), reflecting in-region requests to improve observation networks in the region. The LAC region includes the relatively poorly monitored Caribbean and Central American countries as well as those in South America. In Africa, decision-focused support activities dominated (59 percent of all climate services investments in the region). In Asia, 57 percent of the activities implemented in the region were also decision-focused support. FIGURE 4: CATEGORIZATION OF CLIMATE SERVICES INVESTMENTS IN THE INVENTORY BY REGION (N=40) Note: Labels represent percent of total number of activities in each region. 59% 6% 35% 57% 14% 29% 25% 13% 69% 0 2 4 6 8 10 12 Decision Focused Learning Data Provision Africa Asia LAC Number of Activities USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |9 DATA PROVISION ACTIVITIES The activities in this category span a range of action, either through direct, “hard” investments (e.g., those used to purchase automated weather stations that fill observational gaps) or through strengthening the capacity of institutions responsible for data provision, including international institutions focused on improving historical data records such as those funded under the Enhancing National Climate Services (ENACTS) activity. All the data provision activities addressed specific development challenges such as disaster risk reduction (particularly for floods) or decision-making needs (e.g., municipal planning) that required significant investment in building the data and information space to support these tasks. Most of these activities were implemented in the Caribbean, followed by Sub-Saharan Africa and South America, potentially reflecting a perceived need to fill data and observation gaps in these regions. For example, under the Building Regional Climate Capacity in the Caribbean (BRCCC) activity, USAID partnered with the World Meteorological Organization to strengthen the capacity of the Caribbean Institute of Meteorology and Hydrology (CIMH) to deliver its forecasts in support of disaster management while also installing automatic weather stations in places previously not monitored, such as Guyana. Data provision interventions focused on areas such as infrastructure renovations, increasing the range of source information that could eventually be used to deliver services to stakeholders. For example, to improve disaster risk reduction and planning, the Eye Kutoloka project in Angola’s Cuvelai Basin partnered with the Angolan government to establish a network of weather stations to provide data that could be sourced in short-term climate forecasts. Other data provision activities focused on improving the quality and reliability of forecasts. For example, in Jamaica drought forecasts were improved through the use of the Climate Predictability Tool. This category also included activities that sought to enhance the science and knowledge base for decision makers. For example, Peru’s National Infrastructure for Water Security activity sought to mainstream natural infrastructure approaches to reduce water risks such as droughts, floods, and water pollution, including activities to improve the climate and weather knowledge base to support investments. Examples of intended outcomes from data provision activities that the team identified from inventory documents that were also reflected in the survey responses included: • Enhanced capacity at meteorological agencies to effectively convert climate data to products and services that better inform decision making in key climate-sensitive sectors. • Enhanced climate monitoring and forecasting that feed into early warning systems. • Improved data acquisition networks. DECISION-FOCUSED SUPPORT ACTIVITIES While data collection and provision activities focused on building and sharing the information evidence base on weather and climate, decision-focused support activities translated weather forecasts and climate information into localized knowledge products and advisory services that could inform action to be taken. Most of these activities focused on agriculture and food production, disaster risk management and climate risk management, infrastructure, and water resource management. These activities assisted in the following ways: • Developing advisory services for target groups (e.g., farmers, fisherfolk). • Investing in research and science (e.g., hydrologic modeling to inform water distribution and potential risks from floods) to support partnerships in water resource management. • Providing feedback mechanisms for locally collected information to improve observation networks. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |10 • Translating and applying available climate information into planning mechanisms to help build resilience. Some decision-focused support activities in Sub-Saharan Africa developed customized advice for farmers based on local context, transforming data into knowledge to support adaptation and risk management and build resilience (see Box 1 for an example in Rwanda). Examples of intended outcomes sourced from inventory documentation and also corroborated through the survey and KIIs for decision-focused support activities included improved monitoring systems for water management and developing tools such as ClimaPesca, an app that provides early warning information to fishermen regarding sea safety, early warning systems to better address local risks, and timely and appropriate climate information to support household decision making. Several decision-focused support climate services activities benefited from and built on previous investments in strengthening local capacity for forecasting, such as the data provision activities discussed above. For example, a previous activity helped improve Mali’s meteorological agency’s forecast capacity and skill. This effort paved the way for new activities to improve information use in local decision making through participatory techniques that engage providers and users, making it more likely that target beneficiaries would use this information. Clearly, the investments were also guided by the contextual realities of the activity, including the relative availability of climate and weather information, state of climate services in the location, and other donors’ investments plans. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |11 LEARNING ACTIVITIES Climate services activities in this category were conducted primarily in Sub-Saharan Africa under the Learning Agenda for Climate Services. The learning agenda sought to generate and analyze new information, evidence, and learning on the effective and sustainable production, delivery, and use of climate information to improve decision making and outcomes for rural agricultural livelihoods. Two streams of work were carried out under the learning agenda: 1. The Climate Information Services Research Initiative, which sought to address the following overarching questions: o At the farm level, what factors influence farmers’ access to and use of climate information services and the circumstances under which climate information services benefit livelihood outcomes? What are the barriers to accessing and using climate information services? o What methodologies can be used to evaluate the impact of climate information services? o How can learning and evidence be incorporated into processes for improving the design, implementation, and evaluation of climate information services in the future? 2. Assessing Sustainability and Effectiveness of Climate Information Services, which aimed to identify sustainable and effective climate services models. This included developing BOX 1: RWANDA’S CLIMATE SERVICES FOR AGRICULTURE ACTIVITY Rwanda’s Climate Services for Agriculture activity trained agricultural extension staff, nongovernmental organizations, and other intermediaries on how to integrate climate services into their ongoing work with farming communities to offer advice on farm-level decisions, such as when to plant, seed selection, and what inputs to use on the field. This activity used Participatory Integrated Climate Services for Agriculture (PICSA), a facilitative participatory process, to identify decisions made by farmers and determine how climate information can be used to inform these decisions. Additionally, the PICSA approach helps strengthen knowledge of the assets farmers call on to respond to climate stresses and shocks, with the understanding that farmers have varying needs and adaptive capacity depending on their social roles and status (e.g., gender, age, socioeconomic status). The process helps identify locally appropriate solutions to help communities and households adapt to climate risks. These could include a combination of approaches such as: • Improving soil fertility or promoting soil conservation. • Improved agricultural practices. • Pest control and integrated pest management. • Plant disease management. • Improved seed variety availability and selection. • Water management. • Improved post-harvest storage. Access to and the diffusion of climate information is facilitated through a variety of channels including community focal points and radio and television programs in local languages. Post-season, the activity tracked the use of this information by farmers and the degree to which the information contributed to enhanced resilience. The activity demonstrably improved farm yields and productivity, increasing yields from 6 to 21kg per plot for bean grains, while also increasing yield mass weight overall from 10-36kg per plot (Figure 15). USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |12 metrics to assess sustainable and effective provision of climate services and identifying relevant business models to improve the sustainability of national meteorological and hydrological services. Other learning activities included supporting the establishment of communities of practice such as the Climate Services Partnership, an informal, interdisciplinary network of climate information users, providers, donors, and researchers that share an interest in climate services and are actively involved in the climate services community. GEOGRAPHIC DIFFERENCES IN CLIMATE SERVICES INVESTMENTS BASED ON THE SURVEY RESULTS The 50 survey respondents were principally from USAID (78 percent), along with a few implementing partners (18 percent), non-governmental organizations (NGOs, 2 percent), and private businesses (2 percent). Of the USAID respondents, 61 percent were agreement or contracting officer’s representatives for climate service activities, 37 percent were mission-based climate integration leads, 58 percent were technical staff, and 8 percent classified themselves as part of a program office or leadership. Figure 5 shows the diverse geographic distribution of the online survey respondents. Regional insights gleaned through the survey, document review, and KIIs on the role that climate services activities have played in supporting larger development programming reflect these activities’ geographic diversity. FIGURE 5: GEOGRAPHIC DISTRIBUTION OF ONLINE SURVEY RESPONDENTS Latin America and the Caribbean, 32% Asia, 21% Africa, 37% Middle East, 3% Multiple Regions, 3% Regional Bureau, 5% Pillar Bureau, 3% Other , 3% USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |13 There was some geographic variability in how survey respondents described the purpose of the climate services activity on which they worked. Across regions, the greatest weight was given to translating information, improving the quality of climate services data, and informing field-level investments. Respondents in Africa also emphasized co-production with key stakeholders and making the case for future investments. Respondents in Asia mostly did not emphasize making the case for future climate services investments. Guiding further climate services research was a relatively lower priority for global, LAC and Africa respondents. (Figure 6). 25% 25% 67% South America Central America Caribbean Latin America and Caribbean Sub-Regions (N=12) 7% 7% 50% 57% Central Africa Southern Africa East Africa West Africa Africa Sub-Regions (N=14) 25% 25% 100% Asia Pacific Central Asia Southeast Asia Asia Sub-Regions (N=8) USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |14 FIGURE 6: PURPOSE OF THE CLIMATE SERVICES ACTIVITY (N=47) Whereas all sectors queried on through the survey received attention from climate services activities, activities in Asia emphasized agriculture, water management, and disaster risk reduction; activities in Africa prioritized forecasting, agriculture, and natural resource management; and activities in LAC focused on water management, natural resource management, and agriculture (Figure 7). FIGURE 7: SECTORAL FOCUS OF CLIMATE SERVICES ACTIVITIES BY REGION (N=47) The weather and climate-related risks that activities addressed also varied regionally. All activities prioritized extreme weather-related events such as droughts and floods, whereas extreme heat received less emphasis across all activities. Unreliable rainy seasons were of high priority in the Africa and LAC activities as well and less of a priority in the activities in Asia (Figure 8). 0% 20% 40% 60% 80% 100% To inform field level investments To translate information into usable formats to improve key stakeholder utilization To improve the quality of weather and climate services data To make a case for future climate services investments To co-produce weather and climate information with key stakeholders To guide further climate services research Global LAC Asia Africa 0% 20% 40% 60% 80% 100% Agriculture Water Management Natural Resource Management Fisheries Disaster Risk Reduction WASH Global LAC Asia Africa USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |15 FIGURE 8: MAIN CLIMATE RISKS ADDRESSED IN THE CLIMATE SERVICES ACTIVITY (N=39) OVERARCHING LESSONS FROM CLIMATE SERVICES INVESTMENTS This section summarizes lessons learned that emerged through the evaluation process. Climate information services have played a significant role in achieving place-based resilience, with the resulting information products informing sound, evidence-based decision making across many USAID activities. As weather patterns become more variable and the manifestations of climate change bring uncertainty, the need for climate services has grown significantly and the ways these investments have addressed identified barriers has increased their impact. In Rwanda, for example, participating farmers increased yields by 47 percent and their income from crops by 56 percent. Inexpensive or low-tech solutions such as stakeholder-monitored rainfall gauges and digitized historical data can fill data gaps and build local information in a cost-effective manner, while also establishing confidence in the information. Reliable meteorological information is essential to plan for weather impacts and to better understand and identify future climate change impacts. As climate change makes weather patterns less predictable, the reliability and accuracy of meteorological information (products and services) are critical. Moreover, the diversity of agro-ecological zones across nations makes national-scale weather forecasts relatively useless for local planning. Detailed data specific to local characteristics are needed to make relevant regional or district-level forecasts. Although more expensive or high-tech investments are also clearly needed, inexpensive or low-tech approaches can helpfully add local specificity to available climate and weather information. The Mali Climate Change Adaptation Activity (MCCAA), for example, provided vulnerable communities with 980 rain gauges (3 in each of 170 new villages and 2 in each of the original 235 villages) and trained focal points at the cercle and commune levels to collect and track rainfall measurements. MCCAA also worked to build awareness on how to make decisions based on information collected from the rain gauges. Further, to ensure that focal points understood the terms used in the Agromet Toolbox and in the daily, weekly, and 10-day forecasts that are broadcast through 8 radio stations in Mopti Region, MCCAA translated key terms into 3 local languages and distributed them to all focal points (Box 2). These efforts 0% 20% 40% 60% 80% 100% Unreliable rains Rising temperatures Extreme heat Altered or unpredictable seasons Extreme weather￾related events such as floods, droughts, or cyclones Global LAC Asia Africa USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |16 were clearly successful in building resilience by increasing adaptive capacity and improving yields. For example, households receiving the climate service over three years reported significantly greater adoption of climate adaptation technologies than households without services and an increase in their perceived ability to cope with climate risks. MCCAA’s 2019 effectiveness study also showed that participating households improved their resilience due to the activity’s interventions (a 34 percent increase on adaptive capacity indicators, a 30 percent increase on absorptive capacity indicators, and an 18 percent increase in transformative capacity indicators). Further, households that received the climate services reported significantly larger increases in staple crop production, livestock production, and household income (64 percent versus those outside of the activity reporting no increase). Another approach that Uganda’s Strengthening of Meteorological Products, Services, and Use in the Agriculture and Water Sectors activity employed was to provide financial support to the National Meteorological Authority to “rescue” (i.e., digitize) 30 years of historical rainfall information for Uganda based on paper files housed in the region’s former data hub in Kenya. Similar data rescue activities filled critical information gaps in northern Tanzania through the Planning for Resilience in East Africa through Policy, Adaptation, Research, and Economic Development activity, information subsequently used to support water allocation planning in Tanzania through the Sustainable Water Partnership. Providing information is not enough. Activities must also address issues of trust, as well as the resources and skills to use this information. Although it is useful and important to provide partners with scientific data, information, and tools, it is not enough if stakeholders lack confidence in the information or have difficulty understanding how to interpret and use it to make decisions – particularly given inherent uncertainties in the information. These decisions can include when to plant, what kinds of inputs and seeds to use on a farm and when, when to go fishing, or how to calibrate disaster management plans in light of emerging risks. Furthermore, even if trust exists, many users lack the resources and know￾how to employ alternative strategies considering this information. The evidence from climate services investments overwhelmingly points to a need to couple the climate service with extension support and resources that support actions. User-centered activity design can foster buy-in and long-term commitment to information use. Many USAID climate services investments have repeatedly demonstrated that climate information adoption rates are based on the “perceived utility” of the information and adaptation techniques suggested. If households or target stakeholders are aware of the climate information and adaptive technologies offered and have access to them, they will assess the utility or value of these factors before they decide to adopt them. Working at the outset with key stakeholders in an “incubation” period prior to activity implementation is crucial for success as it allows issues to be addressed that feed these assessments, including trust, reliability, relevance, and timeliness, by responding specifically to user needs. This participatory process allows for the proper evaluation of population needs and offers the opportunity to tailor approaches and resources to meet stakeholders’ needs. An added benefit of co-producing climate services is that the dialogue itself can raise awareness of climate variability and change, educating users and identifying and helping shape demand for information in support of their specific decisions (Table 1). USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |17 TABLE 1: ILLUSTRATIVE CLIMATE INFORMATION NEEDS OF STAKEHOLDERS Stakeholder Segment Climate Information Needs Government (ministries, agencies, local authorities) Development of adaptation and mitigation plans, design of infrastructure, civil protection Universities and research institutions Training, modeling tools development, studies on climate change and environmental degradation NGOs and development partners Design and implementation of programs to increase resilience to the effects of climate change Private sector companies Climate-related risk management to improve business performance General public (individuals, communities) Development of adaptation and mitigation action to increase safety and resilience Producer groups (farmers, pastoralists, fisherfolk) Climate-related risk management to increase productivity Modified from Senegal – Weather and Climate Information Services business case presentation, April 22, 2020 BOX 2: MALI CLIMATE CHANGE ADAPTATION ACTIVITY MCCAA At a Glance Total Budget $13.5 million Period of Performance Five years (2016-2020) Service Types • Localized weather information collection • Awareness-raising and improving access to climate information • Advisory services to farmers including adaptive technologies and options Sector of Focus Agriculture/food security Target Beneficiaries Farmers in rainfed areas of the Mopti region As shown in the figure on the right, MCCAA was designed to test how and in what ways producing, improving, and providing access to climate information can help identify actionable adaptive strategies that farmers, communities, and households can use to be more resilient to climate variability and change, thereby increasing their potential for sustainable and equitable economic growth. MCCAA sought to make Mali’s rainfed farmers more resilient by helping increase the adaptive capacity of targeted communities, households, individuals, and systems; the inclusion of climate change considerations to accelerate Mali’s transition to climate-resilient and sustainable economic development; and the adoption of local solutions to climate variability and change by communities and individual households. MCCAA built on previous analyses that point to robust forecast accuracy provided by Mali’s meteorological agency. Daily, weekly, 10-day, and seasonal forecasts are combined with an overview of a locale’s climatology and linked to local rain gauge observations that community members monitor. This bottom-up, top-down approach helps build confidence and awareness of the local dynamics of climate (which are poorly captured by the meteorological agency’s networks) and strengthens beneficiaries’ understanding of climate variability and how these dynamics play into a probabilistic forecast. The forecast directly addresses local farmers’ planning needs for sowing dates, timely application of fertilizer and pesticide, seed selection, and field tasks. MCCAA facilitated access to and diffusion of climate information through a variety of channels, including community monitors and radio and television. MCCAA DEVELOPMENT PATHWAY USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |18 EQ 1: LESSONS LEARNED, GOOD PRACTICES, AND KEY CHALLENGES ALONG THE VALUE CHAIN USAID’s climate services investments work across a variety of sectors and with many kinds of decision makers. Some involve detailed, localized work with specific farmers or fisherfolk, whereas others engage with municipal managers to integrate climate resilience into local plans and policies. Many have benefited from previous, tangential initiatives that promoted other aspects of the value chain required for effective delivery of climate services. This section describes the overarching challenges and barriers identified, the conceptual value chain, key lessons learned along the value chain. and conclusions for EQ 1. OVERARCHING CHALLENGES AND BARRIERS The climate services investments the evaluation examined faced challenges including issues of data (in terms of quality, timeliness, and management structures); confidence in the information presented; applicability and usability of the information, and agency of the target beneficiaries. ● Poor interoperability of data management systems: weather and climate data management systems are often outdated, making data access difficult for both users and those interested in improving services such as forecasts. ● Limited reach and adequacy of communications and translation systems: meteorological warnings, for example, come with several challenges. First, stakeholders cannot benefit from warnings if they do not receive the information – either because they are out of radio and mobile network coverage or because the climate information is delivered too late. Second, illiterate stakeholders may find it difficult to grasp written information about risks associated with climate services. Third, the information is often not sufficiently tailored to user needs. Finally, inconsistent funding sources for climate services delivery have undermined the long-term effectiveness of these services. ● Lack of end user confidence in the information: the historical disconnect between meteorological services, researchers, and decision makers – which can lead to poor quality or insufficient explanations of the inherent uncertainties in any forecast – can diminish the confidence users place in the information and thus in their willingness to act in response to it. ● Insufficient consideration of the needs and agency of marginalized groups in project designs: the information needs of women, youth, ethnic minorities, the landless, and other marginalized groups – in terms of timing, delivery mechanisms, and variables of interest – can significantly differ as their roles and ability to respond are inexorably tied to their status and social roles. ● Limited expertise in using climate information: although weather and climate information has been available for decades, the ability to use this information to inform decisions has been limited. This is partly because the information is often not available at the scales of interest (both in space and time) but also because target users are often unsure how to apply forecasts and other information in their decision-making processes. ● Lack of access to resources and assets (e.g., land, input, equipment) required to act on the information in a timely manner: the Learning Agenda for Climate Services in sub￾Saharan Africa, for example, explored how gender can influence agency and access to climate “I think there is already a wealth of data and information that can be drawn from, but communication remains a challenge for climate services.” - Survey respondent USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |19 services. The differential resources and influence under women’s and men’s control affect their ability to make use of climate services as well as the needs and demands for information10. ● Inadequate modelling and forecast capacities in some countries can pose seemingly insurmountable barriers and require sustained, extensive investments before the potential benefits of a climate service can be realized, if at all. USAID climate services activities have successfully addressed many of these challenges. Despite the diversity of approaches taken and regardless of the entry points of investment in the value chain, several lessons emerged that can inform future climate services programming. When asked about barriers limiting the use of climate services in investments, survey respondents highlighted a range of issues, including limited technical expertise in using climate and climate risk data (73 percent), lack of awareness of available climate or climate risk data (69 percent), and limited availability of relevant data (55 percent) (Figure 9). FIGURE 9: WHAT DO YOU THINK ARE THE MAIN BARRIERS THAT LIMIT THE USE OF CLIMATE AND/OR CLIMATE RISK DATA IN YOUR REGION? (N=49) Survey respondents also noted several challenges and barriers in climate services implementation, with the most prominent being the varying capacity of key personnel in different organizations to both understand and use climate services. Other challenges in implementation included limited dissemination reach, limited resources for stakeholders to act on the information, and a disconnect between the available research base and the needs of target decision makers (Figure 10). 10 See: https://www.climatelinks.org/resources/gender-responsive-rural-climate-services-review-literature 20% 2% 39% 49% 55% 69% 73% 0% 20% 40% 60% 80% 100% Other I have not experienced any barriers to my climate or climate risk data use Access issues due to information being in a different language Access issues with obtaining climate or climate risk data (e.g., hard to download, cost) Limited availability of relevant data Lack of awareness of available climate or climate risk data Limited technical expertise in using climate or climate risk data USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |20 FIGURE 10: DID THE ACTIVITY FACE ANY OF THE FOLLOWING CHALLENGES OR BARRIERS DURING IMPLEMENTATION?11 (N=45) A CONCEPTUAL MODEL OF THE USAID CLIMATE SERVICES VALUE CHAIN The conceptual model used to frame USAID’s climate services investments along a value chain (Figure 11) represents a continuum of activities from translating and packaging climate and weather data into information to dissemination and communication of that information, ultimately leading to informing action. The production of data and systematic observations are the foundation of informed decisions, but they must be packaged and communicated effectively for those who need to act. The lessons captured below for each component are intended to identify entry points for climate services investments. While the clear lines indicated between each component help to conceptualize the investments evaluated, in practice the components function within a continuum of a highly iterative and complex system. 11 Survey respondents could select more than one of the challenges, and Figure 10 represents a summary of their responses. 16% 7% 20% 22% 27% 40% 44% 44% 47% 73% 0% 20% 40% 60% 80% 100% Other The activity did not face any of these challenges or barriers Stakeholders did not use the same data management software and tools Timing issues (e.g., data are collected at irregular intervals or not concurrently with other relevant data or user needs) End users did not have confidence in the information. Limited engagement of women accessing climate services Disconnect between researchers and decision makers (e.g., lack of partnership and collaboration between key stakeholders or poor communication) Limited dissemination and overall reach of climate services information. Limited resources for users to implement change based on climate information (e.g., farmers do not have knowledge or resources to change farming practices) Varying capacity among personnel in different organizations (e.g., government, private sector, research institutions) USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |21 FIGURE 11: CONCEPTUAL MODEL OF CLIMATE SERVICES VALUE CHAIN USAID’s climate services investments begin with a goal such as improving livelihoods, reducing vulnerabilities, or improving the capacity and information base to respond. The goal at the outset defines: • Who will act? • When will the information to support the action be needed? • What kind of knowledge can help support decisions? The continuum can then be roughly divided into three components, each building and looping back on the other to form an iterative system that delivers value by supporting a goal or outcome, whether reducing vulnerability, increasing resilience, or safeguarding livelihoods. At each step below in the process, the same strategic questions (i.e., Who? When? What?) need to be asked: • Data: individual facts or systematic observations collected from the environment or digitized from paper records, not all of which will be relevant to the decisions made. Relevant data are distilled and synthesized in context to become information. • Information: data interpreted in a specific context to best support the decisions stakeholders need to make. Information is communicated either directly or in an interactive process, via fit￾for-purpose communication channels to those who need to act. However, communication alone does not guarantee that users will take advantage of the information. Activity experiences point to the need to provide resources and skills to take advantage of the new information in making changes to practice. • Action: what someone does in response to available information. Examples cover a range of sectors and users, including farmers’ decisions about when and what to plant and water management action related to floods. This climate services value chain model applies across all USAID-supported activities. In practice, each climate services investment worked on several components of the value chain with varying emphasis to achieve the stated goals, building on local and regional realities. Decisions on where in the value chain to invest were driven largely by an activity’s context, stakeholders’ requests and needs, and the role and investment plans of other donors in the space. Examples of these activities are presented throughout this report. LESSONS AND GOOD PRACTICES ALONG THE VALUE CHAIN Given the diversity of experiences across USAID activities, several lessons and helpful practices can be discerned along the climate services value chain. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |22 DATA Systematic observations are fundamental to providing climate services, whether from “rescued” (digitized) paper records to build a robust historical knowledge base or newly collected through automatic weather stations, blended satellite products, or crowdsourced techniques such as rain gauges. Improving the resolution and localized quality of available data is the principal goal. However, production involves not only collecting data but also supporting the capacity of meteorological services to maintain weather stations; improving the provision of critical spare parts for this maintenance through public-private partnerships; and managing, analyzing, and distributing that data so it can be analyzed and synthesized in context (becoming information) to key stakeholders. Sustainability in production requires supporting the business models, procurement processes, and human resources necessary to sustain these observational networks (Box 3 provides an example from the Caribbean). The long-term value of investing in observation networks can catalyze demand for climate services. Investments in the data provision and information components are well documented across the USAID climate services experience. The BRCCC activity, for example, upgraded and enhanced hydrometeorological networks by installing new monitoring equipment in eight countries including Guyana, whose meteorological coverage was extended to areas not previously monitored. BRCCC not only provided finance and training for the network but also built the capacity of regional centers of excellence such as the Caribbean Community Climate Change Center (CCCCC) and CIMH, which have mandates as regional data repositories, by providing training and technical support in applied meteorology, climatology, and forecasts and improved data delivery mechanisms to partner countries in the region. Additionally, BRCCC helped forge and strengthen relationships and initiate dialogues between these climate and weather organizations and sector-specific regional organizations such as the Caribbean Agricultural Research and Development Institute, the Caribbean Disaster Management Agency, and the Caribbean Public Health Agency, furthering discussions on the needs for climate and weather information across the region. Who? Typically, meteorological services of individual countries or regional meteorological organizations. What? Build the observational database necessary to establish climate services. Data are the foundation of climate services. How? Support observation network expansion, train staff in maintenance and data management, fill in data gaps with blended station/satellite products, “rescue” paper-based information. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |23 BOX 3: BUILDING REGIONAL CLIMATE CAPACITY IN THE CARIBBEAN The BRCCC activity’s primary objective was to strengthen CIMH’s capacity to deliver its programs and to facilitate the establishment of the World Meteorological Organization’s Regional Climate Centre for the Caribbean to be housed at CIMH. Achieving these objectives required (1) renovating infrastructure; (2) increasing the range of products and services delivered to stakeholders; (3) enhancing human and technical capacities at CIMH and national meteorological and hydrological services in small Caribbean developing states; and (4) improving service delivery mechanisms to national, regional, and international stakeholders. BRCCC aimed to support the Caribbean’s sustainable development by making regional societies and economies more resilient to extreme weather, various facets of climate variability, and long-term climate change. The activity outcomes included: • Enhanced capacity at CIMH and across the Caribbean to effectively convert climate data to products and services to better inform decision making in critical climate-sensitive sectors. • Enhanced CIMH climate monitoring and forecasting, feeding into early warning systems, and improved data acquisition networks across the Caribbean. • Establishment of the Caribbean Centre for Climate and Environmental Simulations to provide CIMH staff and regional scientists with the necessary resources to simulate regional environmental and climate processes to better inform regional decision making on disaster risk reduction, water resources management, and adaptation to climate change and increasing climate variability. • Enhanced CIMH infrastructure to enable it to sustain and expand core activities as well as the activities envisioned as a climate center under the Global Framework for Climate Services. BRCCC invested heavily in strengthening the capacity of Guyana’s meteorological services. The picture below provides an example of the outputs of this investment: a customized Maproom that the meteorological services serves online to all users and that includes blended product outputs (merged station and satellite data). The Maproom is a collection of maps and other figures used to monitor past and present climate and societal conditions. The maps and figures can be manipulated and linked to the original data and users can choose which datasets are particularly useful for monitoring current conditions. Source: http://181.199.253.14/maproom/Climatology/Climate_Analysis/monthly.html?T=Jan®ion=bb%3A￾58.449999999999996%3A6.6000000000000005%3A￾58.4%3A6.65%3Abb&YearStart=1981&YearEnd=2018&seasonStart=Jul&seasonEnd=Sep&var=.precip#tabs-2 USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |24 INFORMATION The information component of the value chain focuses on translating available data into information relevant to key users and recognizes that these users’ information needs vary depending on the context in which they live. The information component offers an opportunity to: • Translate signals of climate variability and change (e.g., drought, changes in temperature and precipitation) into meaningful indicators for target sectors (e.g., agriculture, hydrology, health, ecosystems, urban energy demand). For example, some stakeholders need improved short-term meteorological and hydrological forecasts of the onset of rains, while others require long-term climate information on water availability. • Work in a participatory manner to define information needs and foster engagement. Investments in the information component have significantly improved the available information supply while also building demand. The supply of available information has evolved and improved significantly since 2012. For example, farmers in Rwanda can now interpret relatively complex probabilities on the onset of rains to make decisions. On the other hand, not all potential users understand the need, value, and possibilities of getting climate-related information, so in some instances, the demand for information must be constructed. Building this demand involves creating awareness of available climate and weather information, its relative confidence, and how it could be used. The Partnering for Adaptation and Resilience–Agua (PARA-Agua) activity, implemented in Peru and Colombia, worked to create partnerships with decision makers, scientists, and communities to strengthen water management capacities and planning while improving water security and climate change resilience. Originally designed to address the needs of communities and stakeholders reliant on disappearing glaciers in the Andean region of South America, PARA-Agua included a focus on integrating climate projections with hydrological modeling techniques to develop a robust information base to use in planning water allocation. In contrast to the decision-focused support activities in Sub-Saharan Africa, which focused principally on food production and agriculture, PARA-Agua’s climate services investments aimed to increase the research community’s capacity to provide the data needed for better-informed choices on watershed management and climate change and systematically optimize water usage over entire watersheds in the context of climate change adaptation (Box 3). ● Build trust through established and continued dialogue. According to survey respondents, the challenges that climate services investments faced included a disconnect between researchers and decision makers (e.g., lack of partnership and collaboration between key stakeholders and poor communication) as well as issues of timing, whereby data presented were not aligned with user needs. The workshops and participatory dialogues that took place under climate services investments addressed these issues, building ownership and trust. Translating data into information requires (1) engaging with stakeholders through participatory processes to identify their informational needs and (2) combining multiple lines of evidence, from local knowledge to Who? NGOs, extension services, and boundary organizations. What? Translate available data into information that can be used to support actions. How? Analyze data in context and focus on decision needs through co-production and participatory methods. “Availability of climate data may not necessarily lead to their uptake by itself. Climate information must be made available to users who need to be engaged on the value and application of climate information products.” - John Ntaganda Semafara, Director General, Meteo Rwanda USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |25 locally collected information and contextual characteristics. Climate services aim to understand both the information and target stakeholders’ needs. USAID’s climate services investments have implemented several approaches, including: o Facilitating dialogue between the science community and management authorities. In Colombia, PARA-Agua worked to create partnerships with decision makers, scientists, and communities to strengthen water management capacities and planning while improving water security and climate change resilience (Box 4). One of the activity’s many results was the design of a hydroclimatological modeling system for the Guatapuri river watershed to protect basin users from floods. o Supporting district and community planning mechanisms. In the Philippines, through the support of the Climate Change Adaptation and Resilience Project (APIK), 13 villages in KPPN Tinanggea allocated climate adaptation and disaster management activities into their village budgets. These activities included mangrove protection and reforestation, silvofishery, vulnerability assessments, climate field schools for seaweed and rice, and installation of early warning systems. The South Konawe government funded these activities, demonstrating the strong buy-in of APIK approaches and activities by the local government. o Using established participatory methodologies such as the PICSA tool (see Box 1) that the Consultative Group for International Agricultural Research’s Program on Climate Change, Agriculture, and Food Security developed with national meteorological agencies, government extension agents, and NGOs. This involved agriculture extension staff working with groups of farmers in Rwanda ahead of the agricultural season to analyze historical climate information and use participatory tools to develop and choose crop, livestock, and livelihood options best suited to individual farmers’ circumstances. Then, soon before and during the season, extension staff and farmers considered the practical implications of seasonal and short-term forecasts on the farmers’ decisions. The information provided was locally specific and tailored to farmers’ needs and written in the local language to enhance farmers’ understanding. The information included insights on the start and end of the rains, length of the growing season, total seasonal rainfall amount, longest dry spell within a cropping season, and short- and long￾term forecasts. These methods continue to evolve by bringing together key stakeholders across the climate service system, strengthening the capacity of local actors to contribute meaningfully to service design, and facilitating dialogue and consensus building for actions, including locally driven solutions12. o Segmenting user groups to understand the different information needs of marginalized groups including women, youth, ethnic minorities, and the landless. Survey respondents highlighted the need to engage women specifically in the climate services investment process, noting that their needs, both in timing and variables of interest, differ from those of men. 12 See: https://www.climatelinks.org/sites/default/files/asset/document/2020_USAID_Learning-Agenda_Spotlight%20Series￾Learning-Agenda-on-Climate-Services.pdf USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |26 According to survey respondents, the most common types of weather or climate information USAID activities worked with were seasonal forecasts (77 percent), short-term weather forecasts (66 percent), and historical climate trends (66 percent) (Figure 12). Training technical staff in improved forecasting techniques and specialized data analysis, such as downscaling seasonal weather prediction and historical trend analysis, improved technical officers’ skills in providing climate services that are suitable to end-user needs. FIGURE 12: WHAT KIND OF WEATHER OR CLIMATE INFORMATION DID THE ACTIVITY COLLECT, ANALYZE, OR DISSEMINATE? (N=47) 11% 2% 51% 66% 66% 77% 0% 20% 40% 60% 80% 100% Other The activity did not work with weather or climate data Climate projections Short term weather forecasts Historical climate trends analyses Seasonal forecasts BOX 4: PARTNERING FOR ADAPTATION AND RESILIENCE–AGUA PARA-Agua At a Glance Total Budget ~$20 million Period of Performance Two years with two year-long extensions (2013–2017) Service Types Create partnerships with the government, research institutions, and communities to strengthen watershed management and resilience capabilities Sector of Focus Water resources Target Beneficiaries Government, institutions, and communities across four watersheds in Colombia and Peru PARA-Agua was designed to strengthen the capacity of Colombia and Peru to manage watersheds at risk from glacial melt in the Andes. The activity integrated scientific research into policy through four building blocks: sustainable knowledge transfer, institutional framework, prioritization of climate adaptation options, and climate adaptation financing. PARA-Agua combined traditional capacity-building activities with resources and implementation expertise to create linkages between researchers and decision makers to mainstream various types of data into watershed management. This required facilitating dynamic exchanges and improved collaboration by, for example, creating toolkits to train scientists to effectively communicate findings to policymakers so action could be taken. PARA-Agua also fostered regional cooperation through an online community of practice, partnered with a network of female scientists to work with community leaders, and used world-class modeling to build up water management programs that involve a sustainable flow of information to meet beneficiary needs. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |27 For target beneficiaries to use the information generated, there needs to be significant investment in methods of communication that use fit-for-purpose approaches to reach the intended recipients. USAID climate services investments employed diverse tools and methods to communicate information to target beneficiaries directly, including using information and communication technologies such as radio programs, short message service (SMS), email, reports, presentations, television, and social media (Figure 13; Box 5). Some activities emphasized interactive approaches such as local dialogues and workshops, supported local focal-point farmers who could offer tangible examples of the value of climate services. The Philippines’ Be Secure activity established a community of practice on resilient disaster risk management and water operations with officials in cities and towns to ensure the larger community’s safety. Composed initially of water district decision-making staff and later supplemented with municipality staff, this core group received sustained and long-term capacity-building assistance from experts both in￾country and virtually. The experts themselves were practitioners and had developed resilience measures successfully in their own communities. These formal training sessions aimed to expose Filipino participants to strategies that had been successful in other areas and help them adapt, plan, and implement similar strategies for their own communities and institutions. Twinning, in which experienced practitioners from one area mentor and provide in-classroom (formal) training followed by a practicum period, enabled greater learning. Geographic areas compared their strategies and budgets with those of others in the Philippines, shared ideas, and learned about their own weaknesses. The principal teaching methods were formal objectives, lesson plans, and relevant handouts and PowerPoint presentations supplemented by discussion and consultation. Peer consultation sessions and pre- and post-test results of each session allowed both the participants and the trainer to evaluate the learning taking place. “The information so far generated is not localized, very general - subnational level that affect the confidence and uptake of the information though it is contextualized and developed into appropriate communication formal. We need to improve on the quality of the product to respond to subnational-level variations.” - Survey respondent BOX 5: FACEBOOK DISCUSSIONS HELD IN THE PHILIPPINES The Social Media Campaign on Water Awareness, which was implemented through a partnership between Be Secure and the Philippine Star Digital Edition and Newspaper, featured infographics, water and sanitation facts shared on social media, and a 90-minute Facebook Live event on July 22, 2016 on water awareness. Be Secure experts discussed the Philippines’ current water situation and answered questions. The discussion was viewed 8,407 times within the first 24 hours. The “Your Water, Your Choice” video was viewed 3,216 times from the Philippine Star’s Facebook page as of August 10, 2017 (https://www.facebook.com/PhilippineSTAR/videos/781738631979820/). In addition, USAID’s Global Waters online magazine features the video, making it accessible to audience outside of the Philippines. Source: Be Secure Final Report 2017. “Evidence supports an early pan-Africa assessment that concluded that poor relevance of available climate information, and weak capacity of users to act on information and to articulate demand for more usable information, interact in a way that impedes progress. Breaking this cycle requires overcoming longstanding usability constraints, and building user capacity and delivery channels, in parallel.” - Survey respondent USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |28 Finally, the completed plans and their passage by relevant local authorities showed that the participants could promote their efforts convincingly, adapt the plans when needed on their own, and in turn ensure that mitigation and adaptation measures (the long-term objective) had indeed taken place. FIGURE 13: HOW DID THE ACTIVITY DISTRIBUTE OR SHARE WEATHER OR CLIMATE DATA WITH TARGET STAKEHOLDERS? (N=42) ACTION Reliable meteorological information is essential to plan for weather impacts and to better understand and identify future climate change impacts. For example, in the agriculture sector, farmers need reliable information to plan seed selection, sowing, and harvesting. Although providing partners with scientific data, information, and tools is useful and important, it is not enough without stakeholders trusting, understanding how to interpret, and having the agency to use this information to alter their decisions. These decisions can include when to plant, what kinds of inputs and seeds to use on a farm and when, when to go fishing, or how to calibrate disaster management plans in light of emerging risks. Climate services are important in helping stakeholders manage climate-related risks and adapt to climate change. However, climate services alone are not sufficient to build resilience. It is also critical to help stakeholders adjust what they can do on the ground in response to the climate information. Such support can include technical assistance and additional resources such as improved seeds and climate-smart agricultural practices. Getting people to act requires not just giving them information but also having significant interaction to build trust and providing skills and resources to respond. Examples include (1) farmer field schools and farmer promoters to increase experimentation of improved seed varieties and short-cycle varieties when the forecast requires them, and incorporating climate-smart agricultural practices, (2) multidisciplinary working groups, and (3) monitoring and evaluation techniques to understand and demonstrate the value of implemented practices. As an example of this last point, Rwanda’s Climate Services Activity conducted post-season surveys to assess productivity differences 26% 0% 19% 33% 50% 52% 60% 67% 71% 0% 20% 40% 60% 80% 100% Other Mail Reports or journal articles Email Bulletins/newsletters Television or radio Text/SMS messages Online (e.g., website, open dashboard, social… Presentations/discussions at meetings or events Who? Target stakeholders, farmers, fisherfolk, municipal workers, and policy makers. What? Activities and decisions undertaken in light of the information provided. How? Responses defined in coordination with local conditions and supported with additional resources. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |29 across farmer plots with and without access to the climate services and extension support provided. Climate services were disseminated directly to more than 111,000 farmers in Rwanda through PICSA, radio listener clubs, and cellphones and broadcast by radio networks accessible to 70 percent of the population. While the one-way dissemination via radios and cellphones allowed for wider reach of the available information, the PICSA process supported discussions by recipients on field-level actions that could take advantage of the available information. The post-season survey assessed the influence of PICSA training and listener clubs on awareness, access, and updating of climate services by smallholder farmers and their impact on household welfare (e.g., crop productivity, income, food security). The analyses13 showed the following: • Farmers use climate services to make decisions on the types of crops to grow (75 percent), the types of crop varieties to plant (58 percent), timing of planting and land preparation (75 percent), and when and how to prepare land (65 percent). • Participation in PICSA and radio listener clubs, alone and in combination, is associated with a substantial increase in the proportion of farmers who report changing crop, livestock, and livelihood management practices in response to weather and climate information. Relative to the control group, PICSA participation raised the value of crop production by 24 percent and income from crops by 30 percent. The combination of PICSA and radio listener clubs was associated with a 47 percent increase in the value of 13 See: https://cgspace.cgiar.org/handle/10568/108052. 14 CINSERE Final Evaluation Report. crop production and a 56 percent increase in income from crops. Several examples exist in which climate services have demonstrated value in risk management to safeguard lives and/or increase productivity. Under the Climate Information Services for Increased Resilience and Productivity in Senegal (CINSERE) activity in Senegal, climate services in the form of meteorological forecasts delivered by SMS, voice calls, radio, and vigilance flags are tailored to specific stakeholders and sectors. These forecasts have helped provide timely guidance and, when combined with technical support and resources, decision-making tools for farmers as they evaluate their planting and input choices to safeguard their yields and for fisherfolk as they weigh the risks and take appropriate measures before venturing out to sea.As a result, climate services in Senegal have helped increase field productivity (Figure 14) and reduce untimely deaths of fisherfolk at sea14. Further, in Mali the final MCCAA evaluation report showed promising evidence of increased household resilience, as measured by household-level absorptive capacity (up 30 percent), adaptive capacity (up 34 percent), and transformative capacity (18 percent greater than in households without services). When beneficiaries were equipped with the knowledge, skills, and resources to respond to use the climate service, their yields and incomes improved significantly. In Senegal’s Feed the Future activity in Naatal Mbay, for example, gains in productivity and farm gate value experienced by farmers “Because of the uncertainty associated with forecasting, trust is the building block for an effective climate services project to be successful. Understanding the social dimensions that influence farm level decision making and the role of indigenous knowledge are two very key ingredients for this to work” - Ousmane Ndiaye. Senegal National Meteorological Agency FIGURE 14: FARMERS IN SENEGAL LEARN FROM CIS Source: CINSERE. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |30 were made possible by a sharp growth in financial resources channeled to cereal value chains. When using climate services information to guide field-level decisions, yield increases were achieved for millet (14 percent increase), maize (25 percent increase), and rain fed rice (18 percent increase). Gross margins for the 2019 fiscal year showed increases against the Naatal Mbay baseline for all crops (irrigated rice: 31 percent increase, rain-fed rice: 148 percent increase, maize: 204 percent increase, and millet: 38 percent increase) despite some activity areas having been affected by erratic rainfall patterns during the year. CONCLUSIONS FOR EQ 1: LESSONS LEARNED, GOOD PRACTICES, AND KEY CHALLENGES ALONG THE VALUE CHAIN The breadth and diversity of USAID’s recent climate services investments have clearly enabled target audiences to be proactive rather than reactive in managing weather and climate risks, thereby improving decision-making processes, enhancing efficiency, and boosting performance of farmers, fisherfolk, and resource managers. The challenges identified include those related to available data (including data management mechanisms that are often outdated and not easily adapted for broader use), limited observational information at localized scales, lack of awareness of available climate and weather information that could support specific user needs, varied access to available information, and limited technical expertise to make use of available information. The value chain defines iterative steps along a continuum, from data to action, which themselves feed back into other components of the chain. Lessons are identified along the components of the value chain. • Data: the value of investing in improved systematic observations cannot be understated, particularly where additional granularity would be of value to local users. Activities covered the full gamut of data collection, including investing in automatic weather stations, crowdsourcing rain gauge information, and rescuing data from paper archives to build a longer observational record. Equally important are activities that support institutional capacity (usually meteorological services) to maintain weather stations as well as manage and analyze this information. • Information: translating available data into information relevant to key users can help define what signals from climate variability and change are useful for stakeholders, as users have diverse needs. Additionally, investments in the information component can foster engagement and trust with target users through the co-production of climate services. USAID’s experience points to significant progress in established methodologies available for engaging stakeholders in the co￾production of climate services. • Action: providing climate information to target stakeholders is only one part of the process of designing climate services. The evaluation found that providing additional resources and options to act considering this information is critical to the usefulness of the service. Several USAID activities demonstrated that climate services can safeguard lives, increase productivity, improve incomes, and build resilience. The greatest evidence comes from activities that sequenced and integrated efforts across the value chain with layered programming. These lessons offer important information for future climate services programming. Recommendations emerging from these lessons are outlined later in this report. EQ 2: EXPECTED RESULTS FROM CLIMATE SERVICES INVESTMENTS USAID’s climate services investments varied in purpose, including improving the quality of weather and climate data, translating data into usable formats for key stakeholders, and targeting field-level investments (Figure 15). USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |31 FIGURE 15: WHAT IS/WAS THE CLIMATE SERVICES ACTIVITY’S MAIN PURPOSE? (N=47) Survey and KII respondents outlined what they considered successful outcomes of the climate service investments. These included: • Building resilience to climate shocks by improving productivity, planning, and response capacities at localized scales. As previously noted, recognizing the importance of adapting to climate variability and change impacts and other related hazards has, in some cases, demonstrably enabled key stakeholders to respond to these risks in real time, improving livelihoods and food security (Figure 16). FIGURE 16: MASS WEIGHT OF YIELDS (KGS) IN KEY CROPS BETWEEN FARMER’S PLOTS WITHOUT INTERVENTIONS AND PICSA PLOTS • Establishing robust baselines and a knowledge base to inform climate risk management and development activities through investments in observation networks, blended satellite and meteorological station products, and farmer-sourced local rainfall information, as well as a variety of other investments in monitoring sea level and coral health. The value of these activities in supporting informed decisions will endure, especially when coupled with strengthened data analysis, as discussed below. 17% 40% 40% 62% 64% 77% 83% 0% 20% 40% 60% 80% 100% Other To make a case for future climate services investments To guide further climate services research To co-produce weather and climate information with key stakeholders To inform field-level investments To improve the quality of weather and climate services data To translate information into usable formats to improve key stakeholder utilization 0 10 20 30 40 Farmer's Plot PICSA Plot Mass (bean grains + pods) in Kgs Mass (bean grains without pods) in Kgs USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |32 • Strengthening the role of regional and national meteorological services and agencies in providing data analysis. Training meteorological staff in data management, station maintenance, and improved forecast techniques for drought and rains, as well as building their capacity for communicating this information to a wider range of users, has enabled these services to contribute to broader development goals at more localized scales (e.g., communes, watersheds, regions) (Box 6). Furthermore, helping these meteorological agencies develop business plans to ensure their long-term ability to provide climate services may enhance the sustainability of these tasks. • Delivering assistance to key stakeholders, including through scientific assessments and prioritization, to enable appropriate and timely allocation of funds. Across climate services investment geographies, governments recognize the importance of investing in climate and disaster resilience and are willing to allocate significant resources to these tasks. However, budgeting is often hampered by institutions not clearly and actively working at the interface between climate science and policy or by the fragile evidence base on risks and potential responses. PARA-Agua helped fill this gap by providing technical assistance to watershed management bodies. An analysis conducted at the start of the activity identified gaps between actual and potential development of those institutions and interaction between them to identify capacity-building needs. The analysis examined the interaction and interplay between a wide swath of stakeholders, including researchers, donors, and institutions that fund and help steer research; policy formulation specialists (who process information for decision makers); decision makers themselves (who eventually approve policy proposals presented to them at various decision levels); practitioners who implement decisions in specific policy frameworks; and knowledge mediators. By working with local water resource councils on the development of watershed management plans, PARA-Agua strengthened participatory and research systems to increase the production of plan- and policy-oriented data. • Responding effectively to key stakeholder needs. Working with participatory methods, as well as with boundary organizations and extension agencies, to translate and communicate when and how to address climate risks has yielded critical insights on how to build stakeholder resilience and on intervention scale-up. The climate services activities have: o Improved flood forecasting and rapid information dissemination methods by connecting researchers and end users in a process of co-production, yielding timely information for disaster management. o Encouraged the adoption of local solutions to improve resilience by initiating and supporting engagement between data providers and data users. • Strengthening awareness and understanding of climate variability and change, as well as adaptation needs, including the scientific foundations of these needs and potential responses. A variety of activities implemented under climate services investments have helped: o Support local universities to develop and use a climate change adaptation curriculum. BOX 6: ENHANCING NATIONAL CLIMATE SERVICES The ENACTS approach, developed by the International Research Institute for Climate and Society, focuses on enabling countries to produce reliable quality climate information suitable for national and local decision making. Through this initiative, Rwanda’s National Meteorological Agency quality controlled its observational data, and merged its observations with global climate proxy satellite and reanalysis data using years with large amounts of data to calibrate global proxies for years where data are sparse. The result is a 30+ year historical database of rainfall and temperature (minimum and maximum) at a 4-5 kilometer spatial resolution, with no gaps in space or time, readily accessible through the web-based “Maprooms” that are built using the Institute’s Data Library software and hosted on the Agency’s website. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |33 o Build farmers’ and local stakeholders’ knowledge of climate risks. o Secure political buy-in to respond to climate risks by promoting adaptation technical capacity, policy, leadership, and action readiness of individuals and institutions. Additionally, and critically, key stakeholders’ capacity to respond to a changing climate to safeguard their activities has improved through the provision of information, tools, and resources coupled with climate services (Table 2). TABLE 2: DECISIONS SUPPORTED ACROSS SECTORS USING CLIMATE SERVICES15 Agriculture and Livestock Fisheries Disasters (Early Warning) Water Resource Management • Informing choice of field crops and crop varieties, timing of agricultural tasks, application of inputs, and negotiation of annual loans • Where and when to shift livestock to new grazing areas, seeking safe passage to those areas, purchase of veterinary drugs When it is safe to venture into fishing grounds, when it is not, and for how long • When and how to evacuate amid risk of flash floods • What to do and where to go during extreme rainfall amid increased risk of landslides • How to include climate considerations in drought management plans Establishing long￾term dialogue and relationships, plus capacity to address risks as they emerge CONCLUSIONS FOR EQ 2: EXPECTED RESULTS FROM CLIMATE SERVICES INVESTMENTS Weather and climate-related shocks and stressors are becoming and will become even more severe under a changing climate. Climate services investments can offer critical information to support decisions that help farmers, policymakers, water managers, and others address these risks in real time, safeguarding their lives and livelihoods. Across USAID’s climate services experience, the choice to invest in one or more components of the value chain was informed by the activity context, on-the-ground needs, and other potentially complementary initiatives. As evidenced in this evaluation, climate services activities can help build resilience to climate shocks, improve productivity, and support planning and response capacities at more local scales. Furthermore, even when working at a single component of the value chain such as building meteorological observation and data management capacity, the baselines established from these efforts will endure and offer valuable information to future climate risk management activities. Other lessons are discussed throughout this document. EQ 3: ELEMENTS OF ACTIVITY SUSTAINABILITY USAID’s experiences with climate services investments have shown the need to go beyond just providing information and weather and climate advisories. Providing financial support for the enabling environment – the conditions that can facilitate adoption and continuation of climate-informed practices – can help targeted stakeholders confront the risks that may deter them from a proactive, sustainable stance. Such 15 Based on the evaluation’s KIIs, document review, and online survey. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |34 enabling environments can shape risks and barriers that stakeholders face in scaling up investment in solutions for climate change and scaling back investments in action that are reactive and business as usual. The following are examples of key programming modalities that enhanced the likelihood that these efforts will continue beyond the lifetime of a specific donor program. Integration of climate considerations into relevant plans and policies, which can help secure financial support for climate services. Integrating climate risks and adaptation concerns into development plans can boost their prominence in budgeting processes, potentially ensuring the availability of the requisite financial resources. USAID/Indonesia’s five-year APIK activity supported the Indonesian government to strengthen climate and disaster resilience. APIK worked in an integrated manner from the national level down to the regional and community levels. For example, APIK worked with the National Adaptation Plan Secretariat in partnership with the Indonesian Ministry of National Development Planning to integrate climate resilience into the new National Midterm Development Plan, 2020–2024. The plan now has a stated priority to enhance development efforts by addressing environment, disaster resilience, and climate impacts together. The Secretariat revision, with scientific research and cost-benefit assessments that APIK led, was an essential input for the plan, resulting in $2.4 billion being allocated to climate resilience work in the agriculture, water, coastal/marine, and health sectors. APIK then worked with the government on a detailed annual plan to translate this into concrete activities across the country. Demonstration of proof of concept to secure public-private financing. Following on the successful implementation of weather and climate services in support of increased productivity for farmers, livestock herders, and fisherfolk, CINSERE in Senegal conducted a business case analysis to identify innovative strategies to sustain weather and climate services provision and transition to service delivery on a user-pay basis. It also identified weather and climate services production cost-recovery opportunities for the national meteorological agency. In addition, it sought to build a sustainable business case for weather and climate services in Senegal by identifying partnership opportunities with the private sector along a structured value chain. The study assessed the market opportunities for weather and climate services by mapping, sizing, and characterizing the potential users and understanding target clients’ pain points, motivations, and willingness to pay. It also identified potential private partners interested in operationalizing the business case and other required partners by exposing business models along the activity’s weather and climate services value chain, which resembles the one detailed in previous sections (Figure 17). The analysis roadmap offers important insights on the engagement process required to support the evolution of a climate services investment into a viable business case (Figure 18). This is further supported by insights from the Learning Agenda on Climate Services, which suggested that many obstacles in securing public-private partnerships can be overcome through strong national strategic plans that (1) support the autonomy of national meteorological services while also recognizing the opportunity to capitalize on the value add of services that private sector actors can develop through shared data policies and sector roles, and (2) develop financial planning mechanisms that offer the flexibility to fund improvements in technology through private-sector partnerships16. 16 See: https://www.climatelinks.org/sites/default/files/asset/document/2020_USAID_Learning-Agenda_Spotlight-Series-Private￾Sector-Solutions-for-Climate-Services.pdf USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |35 FIGURE 17: WEATHER AND CLIMATE INFORMATION SERVICES VALUE CHAIN IN SENEGAL Promoting the sustained involvement of relevant partners and institutions. Engaging local stakeholders such as local universities, NGOs, or private businesses enhances the potential for activity lessons, training, and capacity building to remain beyond the life of the activity. In the Philippines’ Be Secure activity, for example, members of the outreach, research, and planning departments of the Central Philippines University in Ilolo City actively participated in the participatory climate vulnerability assessment, disaster risk management climate change adaptation orientation, and development of local climate change action plans, as well as the disaster resilience and risk management plan. Their experiences in these processes led to the integration of disaster resilience and risk management awareness into the curriculum of their national service training program courses. Likewise, San Augustin University, which offers architecture degrees, also engaged in the activity and is now looking to develop course materials in environmentally friendly and disaster-resilient structures. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |36 FIGURE 18: POST-CINSERE PHASED APPROACH TO SUPPORT WEATHER AND CLIMATE INFORMATION SERVICES (WCIS) VALUE CHAIN DEVELOPMENT IN SENEGAL Development of context-relevant tools to support climate-resilient decision making. USAID’s Climate Change Adaptation Project worked in partnership with CCCCC to assist 10 Eastern and Southern Caribbean Community countries in their responses to climate change by developing and implementing adaptation policies and initiatives for sustainable economic development and disaster risk reduction and to achieve maximum impact from the scarce resources employed. The CCORAL tool developed for this project is an online decision support system for climate-resilient decision making that helps users in the Caribbean determine the most appropriate action in response to a more variable and changing climate. The tool, which was socialized and disseminated throughout the region, has now been formally adopted USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |37 by Grenada, St. Kitts, and Antigua and Barbuda as a decision support instrument, and Guyana is embedding the tool into its National Climate Change Adaptation Plan. Establishment of institutional arrangements and relationships to increase the likelihood of lasting impact. Efforts to support informed action using climate services are constrained by a lack of coordination among relevant institutions whose support (e.g., technical, resources, policies) decision makers need to act. Strengthening institutional collaboration can safeguard investments made by: • Ensuring climate services data provision continues beyond the life of an activity. APIK in Indonesia partnered with OpenStreetMap Indonesia Group to set up a geographic information system–based platform to support its climate and disaster information database management system.17 The OpenStreetMap team has committed to host the platform, a promising commitment for the sustainability of APIK’s work. • Establishing a collaboration plan. The Caribbean Climate Change Center activity, which funded work on data management at the Center in Belize, also established a formal collaboration plan that outlines the process for securing various partners’ engagement as well as their roles and responsibilities in program implementation. The activity also guided the development of several Green Climate Fund proposals in the region. Support of regional and national centers of excellence that provide access to weather and climate information for risk management. Investment in improved practices in collecting, analyzing, and disseminating climate services is likely to be sustained. Building staff capacity at climate service institutions to provide more regionalized information and downscaled products of relevance (e.g., drought forecasts) and to engage with the wider user community to understand their data and information needs has been critical to sustainability of climate services investments. BRCCC in the Caribbean, for example, provided training and resources to CIMH, which provides climate and weather information to the region. BRCCC is itself a sustainability intervention for CIMH, as it provided seed funding for training staff who remain, as well as surge support for new staff, whose salaries are now covered by subsequent donors. Further, these investments paved the way for CIMH to secure financing to provide analysis and data in support of other climate change projects (e.g., the French Development Agency’s Adapt‘Action Facility). Additionally, CIMH secured a Regional Climate Center designation from the World Meteorological Organization and funding of approximately $4 million from a combination of sources to continue to expand its role in climate services regionally. CONCLUSIONS FOR EQ 3: ELEMENTS OF ACTIVITY SUSTAINABILITY USAID’s approach to climate services has been flexible, accommodating the diverse socioeconomic, climatic, and political needs and characteristics of the populations whose vulnerability these efforts were tasked with reducing. Additionally, as seen across the issues of sustainability addressed above, the responsiveness of these services enabled stakeholder needs to be addressed at the required scales. The collaborative nature of these services suggests that significant institutional capacity exists across countries and regions and that augmenting the skills sets of these groups is a sustainable strategy since these institutions (e.g., CIMH and CCCCC in the Eastern Caribbean; meteorological services in Rwanda, Mali, and Senegal; regional centers of excellence such as AGRHYMET) will remain after the activity ends. Clearly, many of the investments have been integrated into core elements of local, regional, and national adaptation and mitigation investments and have targeted both public and private investment. Climate services deployment has fostered a co-learning and co-producing approach that supports continuous 17 See: https://openstreetmap.id/api-prb-sultra. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |38 improvement and learning delivered with transparency. While there is more to be done, there are clear examples of climate services to draw from in developing new programs. As stated above, in addition to improving livelihoods by safeguarding crop yields and introducing improved methods of livelihood security, as well as saving lives, potential benefits from climate services investments include: • Mainstreaming climate risks and climate considerations into relevant plans and policies, which can further secure financial support. This can boost their prominence in budgeting processes, potentially ensuring the availability of the requisite financial resources. • Promoting the sustained involvement of relevant partners and institutions including local stakeholders such as universities, NGOs, and private businesses. • Establishing institutional arrangements and relationships that can increase the likelihood of lasting impact. Efforts to support informed action using climate services are constrained by a lack of coordination among relevant institutions whose support (technical, resource, policy) decision makers need to act. • Supporting regional and national centers of excellence that provide access to weather and climate information for risk management. Investment in improved practices for collecting, analyzing, and disseminating climate services is likely to be sustained. • Assisting key stakeholders with scientific assessments and prioritization to enable appropriate and timely allocation of funds. Across climate services investment geographies, governments recognize the importance of investing in climate and disaster resilience and are willing to allocate significant resources to these tasks. However, budgeting is often hampered by institutions not clearly and actively working at the interface between climate science and policy or by the fragile evidence base on the risks and potential responses. GAPS IN THE CLIMATE SERVICES VALUE CHAIN Despite significant progress in recent years in addressing the challenges of implementing climate services, it could be argued that climate services investments must evolve from proof of concept to real opportunities in safeguarding vulnerable livelihoods as the climate continues to change. Future USAID climate services investments should focus on scaling experiences, including built capacities and awareness, as well as the tools available to understand and respond to climate risks. The remaining programming gaps were identified by survey respondents and align well with the challenges and barriers this evaluation identified along the value chain, including: ● Action: little work has been dedicated to understanding how a changing livelihood and political landscape will alter the need for and therefore the use and value of climate services. For example, as livelihood diversification efforts provide households with substantial non-farm income, the need for climate services will evolve. o Communication and resources: understanding the roles of gender, social equity, and agency in how information is consumed and utilized needs further exploration so appropriate communication channels and relevant resources are coupled to reach differential users. ● Information: substantially more work is needed to define the information needs of various user communities. Existing studies are very context specific, so having more cases and applications could help identify shared challenges across contexts and user groups, supporting scale-up. For example, do commonalities exist in the information needs of rainfed maize farmers subject to prolonged dry periods in the different regions and could this help inform climate services programs in other locations? USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |39 o Distillation and analysis: sectors and risks that were not the focus of climate services investments (e.g., heat) will require further analysis and a grounded determination of indicators of interest, as well as their uncertainty bounds. ● Data: continued investments are needed to build out observation networks and capacity for managing these networks, including repairs and acquisition of spare parts through local businesses, harmonization of data management tools to enable faster and more seamless analyses, and continued training and capacity building in key data analysis methods such as drought forecasts. The rescue and digitization of historical climate records has proven to be a low-cost opportunity to build an early foundation for developing countries and should continue. While the private sector has begun to fill some of these gaps18, the challenges – particularly for areas that do not currently offer market opportunities – are large. Regional-level insights and gaps in programming are presented in EQ3: Elements of Activity Sustainability section. FUTURE PROGRAMMING PRIORITIES Direct insights from the evaluation’s analyses point to important differences in the programming priorities on climate services across the three regions (i.e., Africa, LAC, and Asia). While these insights may or may not reflect the priorities of the larger donor community, they could help in future decisions about where, how, and in what areas to invest in climate services. For example, survey respondents indicated a need in Asian countries to focus on extreme heat, drought, sea level rise, and saltwater intrusion, whereas LAC countries suggested that future climate services programs should focus on urban water management, ocean acidification and reef health, and the impacts on hydropower production. Respondents from Africa suggested that the risks being addressed (e.g., agriculture, fisheries, extreme events) should continue to be emphasized in future activities, and extreme heat events should be examined as potential target sectors for future programming. There is also some regional diversity in the challenges to implementing climate services activities, likely linked to differences in context, risks, needs, and available expertise. Respondents in Asia emphasized a need to increase knowledge and awareness of climate risks, address other priorities competing for funding and political attention, and translate climate information into understandable formats. Challenges for those in LAC included limited availability of usable data, the need to address a diverse user base, and poor coordination among donors and relevant institutions. Those in Africa, however, cited the cost of meteorological equipment, limited granularity of forecasts, poor communication of the palpable impacts of investing in climate services. and underutilization of critical observation networks (Table 3). 18 See: https://www.climatelinks.org/sites/default/files/asset/document/2020_USAID_Learning-Agenda_Spotlight-Series-Private￾Sector-Solutions-for-Climate-Services.pdf USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |40 TABLE 3: REGIONAL PROGRAMMING FUTURE PRIORITIES FOR CLIMATE SERVICES THAT SURVEY RESPONDENTS HIGHLIGHTED Issue Asia Africa LAC Climate risks that are not currently the focus of climate services activities but should be • Extreme heat • Drought • Sea level rise • Saltwater intrusion • Continuity of existing risks • Extreme heat • Urban water management considering floods and glacial melt • Sea level rise • Heat as a fire prediction tool • Ocean acidification and reef health • Reduced water availability for hydropower Challenges to implementation • Limited knowledge and awareness • Scaling • Competing attention • Translation into understandable formats • Equipment is expensive • Forecast granularity • Limited government support to meteorological services • Creative partnerships to address resource shortages • User capacity • Demonstrating/convincing evidence of integration • Limited palpable impacts • Weak coordination • Underutilization of regional networks such as SERVIR • Lack of usable data • Funding • Diverse user base • Buy-in from policymakers • Coordination among institutions • U.S. policy on climate adaptation • Donor coordination REGIONAL BARRIERS There were also important regional differences in the barriers limiting use of climate and/or climate risk data. Respondents in Asia prioritized a lack of awareness of available information and limited technical expertise in using climate data, particularly in the urban landscape, while those in Africa prioritized limited availability of relevant data, lack of awareness of available data, and limited technical expertise in using climate data while also highlighting barriers related to data access, particularly in more remote areas (Figure 19). USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |41 FIGURE 19: WHAT DO YOU THINK ARE THE MAIN BARRIERS THAT LIMIT THE USE OF CLIMATE AND/OR CLIMATE RISK DATA IN YOUR REGION? (N=49) VALUE CHAIN NEEDS While all survey respondents prioritized investments in the communications and decisions (action) component of the value chain across all regions, with less weight given to the data and information components, those in the LAC and Asia regions also prioritized information (data put into context) (Figure 20). FIGURE 20: WHERE DO YOU THINK IS THE GREATEST NEED FOR CLIMATE SERVICES INVESTMENT IN YOUR REGION? (N=49) 0% 20% 40% 60% 80% 100% Limited availability of relevant data Lack of awareness of available climate or climate risk data Limited technical expertise in using climate or climate risk data Access issues with obtaining climate or climate risk data (e.g., hard to download, cost) Access issues due to information being in a different language Global LAC ASIA AFRICA 0% 20% 40% 60% 80% 100% Data (Observations or Models) Information (Data put in context) Communication (Translating information to impacts) Decisions (Understanding choices) GLOBAL LAC USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |42 SUMMARY AND RECOMMENDATIONS Climate variability and change present a significant threat to lives and livelihoods across the world. Water shortages, rising temperatures, longer and more intense droughts and floods, and extreme heat events are already impacting the health, life, and livelihoods of millions whose reliance on the vagaries of climate makes them more vulnerable to these impacts. By recognizing climate variability and change as a critical development challenge, climate services can inform decisions about the measures needed to tackle climate impacts and safeguard livelihoods. As shown in this evaluation, climate services investments can and do build and strengthen the resilience of people and communities to climate risks. Significant progress has been made in recent years across USAID activities in providing climate information to decision makers. Addressing climate variability and change requires all who are engaged in development in high-risk areas to constantly learn and retain information about an enormous range of topics and issues that change rapidly. Good practices abound, some of which are discussed below. KEY CONCLUSIONS AND GOOD PRACTICES Inexpensive or low-tech solutions such as stakeholder-monitored rainfall gauges and digitized historical data not only fill data gaps and build local information cost effectively but also build user trust in the information. Reliable meteorological information is essential to plan for weather impacts and to better understand and identify future climate change impacts. As climate change makes weather patterns less predictable, the reliability and accuracy of meteorological information (products and services) are critical. Moreover, the diversity of agro-ecological zones across nations makes national-scale weather forecasts relatively useless for local planning. Indeed, detailed data specific to local characteristics are needed to make relevant regional or district-level forecasts. Although hard investments such as with hydrometeorological stations are clearly needed, inexpensive or low-tech approaches can add local specificity to available climate and weather information. User-centered activity design can foster buy-in and long-term commitment to information use. Many USAID climate services investments have repeatedly demonstrated that climate information adoption rates are at least partially based on the perceived utility of the information and adaptation techniques suggested. If households or target stakeholders are aware of the climate information and adaptive technologies offered and have access to them, they will then assess the utility or value of these factors before they decide to adopt new behaviors. Working at the outset with key stakeholders in an “incubation” period prior to activity implementation is crucial for success. It allows issues to be addressed that feed these assessments, including trust, reliability, relevance, and timeliness, by responding specifically to user needs and concerns, while also allowing for the proper evaluation to tailor approaches to meet stakeholders’ needs and address their concerns. Climate information services have played a significant role in achieving place-based resilience, with the information products informing sound, evidence-based decision making that lead to improved outcomes. The relevance of and need for climate services have grown significantly due to these investments and the ways they have addressed previously identified barriers. Owing to the diverse needs of activity beneficiaries, USAID’s climate services investments varied in purpose including improving the quality of weather and climate data, translating data into usable formats for key stakeholders, and targeting field-level investments. Deciding where in the value chain to invest should be driven by the activity context, which can identify opportunities as well as gaps in the value chain. Donor discussions and coordination can help identify the relative strengths of individual partners in filling existing gaps. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |43 RECOMMENDATIONS FOR FUTURE PROGRAMMING Climate services have evolved significantly in the last decade, offering critical insights and valuable proof of concepts to inform future climate services initiatives. Nevertheless, more remains to be done. The following are key recommendations for future USAID climate services programming based on the evaluation’s findings and conclusions: Widely share the diverse examples of climate services activities. Despite a growing demand for examples of climate services programming, there are relatively few in the broader development community that go beyond a simple description. The activities this evaluation examined are valuable illustrations of how to integrate climate services into larger projects, with documented benefits to target stakeholders. Knowing “what” (e.g., what kinds of climate services have been deployed and across what aspects of the value chain); “who” (e.g., who the target users have been); and “how” (e.g., how they have been designed, including the methods applied in determining user needs, delivering the services effectively, and securing service sustainability) will further the case for investing in climate services. Activity designers, implementing partners, project planners, donors considering priorities, and organizations working in climatically variable areas could all benefit from USAID’s significant climate services’ portfolio. There are several international communities of practice that can be leveraged to ‘amplify’ these experiences, ideas and practices, including online platforms focused on climate adaptation (e.g., WeADAPT and Prevention Web). Prioritize momentum and experience on the ground, which are catalysts for scaling climate services programming in the current geographic focal areas. Given that engagement and consultation, particularly at localized scales, is costly and time intensive, many activities had to scale back their pilot investments to account for funding constraints. However, across activities, the trust built with target communities and the improvements supported in dialogue with key research entities and meteorological services remain. Priority should be placed on scaling these investments in the geographic areas where USAID has worked, placing greater emphasis on elements of continued sustainability such as private-public partnerships. Finance targeted studies that explore how to build financial sustainability in climate services investments. Market potential studies for climate services show consistent benefits to target populations; these benefits are reflected in their willingness to pay for the services. There are clear opportunities to continue to explore the “business case” for climate services, which offers the promise of sustainability beyond donor financing. This could include a mix of public and private financing that fills critical gaps in available resources for effective climate services delivery while also considering equity issues in cost-recoverable models. Continue investments in activities that fill data gaps critical for effective climate services. The quality, timeliness, and lead time of critical weather and climate information needed to support climate services have improved markedly in recent years, in part due to scientific and technological advances, an enhanced understanding of user needs, and significant experiences in the deployment of these services around the world. The generation and use of climate information continue to require donor support, particularly in light of limited public budgets. Clearly not every gap can, or should, be filled, but examples abound on how to support systematic observations beyond the installation of weather stations through participatory rain gauges, blended satellite and station products, or earth observation systems, as well as through technical support, know-how, and spare parts for maintenance. Regional donor coordination could help identify the relative strengths of individual partners in filling other gaps in the value chain. Pay attention to remaining research gaps, particularly for sectors and risks that were not the focus of climate services work such as health. There is abundant evidence across the sectors within which climate services worked (e.g. agriculture, livestock production, water), that climate services USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |44 have saved lives and livelihoods. However, in the case of other sectors of priority to USAID, such as health, and specifically heat risks, which were highlighted by survey respondents, fundamental questions remain that will need to be addressed before the role and potential benefit of climate services can be realized. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |45 ANNEX 1: EVALUATION SOW Performance Evaluation of USAID’s Climate Services Investments This statement of work (SOW) is for a performance evaluation commissioned by the Office of Global Climate Change in the Bureau for Economic Growth, Education, and Environment of the United States Agency for International Development (USAID/E3/GCC). The evaluation will examine climate services (CS) investments funded by USAID’s climate change adaptation budget since 2012. The evaluation will identify lessons learned across different CS interventions of use for future programming. Background on Climate Services Climate services equip decision-makers in climate-sensitive sectors (e.g., agriculture, water, health, disaster risk management) with better information to help society adapt to climate variability and change.19 Climate services involve the production, transfer, and use of climate information to help individuals and organizations make climate-smart decisions. National and international databases provide high-quality data on temperature, rainfall, wind, soil moisture, and ocean conditions. Other outputs include maps, risk and vulnerability analyses, assessments, and long-term projections and scenarios. Socio-economic variables and non-meteorological information (e.g., data on agricultural production, health trends, human settlements in high-risk areas, road and infrastructure maps for the delivery of goods) may be combined, depending on user needs. Collected data are transformed into customized products such as projections, trends, economic analysis, and services for different user communities. Climate services are intended to guide countries, communities, households, and individuals to anticipate and manage climate and weather risks and opportunities. In general, CS must be timely, spatially appropriate, relevant, accurate, and understandable. They can exist in a variety of forms, such as seasonal forecasts, weather information, drought forecasts, and flood early warnings. USAID and other development organizations invest in activities to support CS across the developing world, working to build resilience to weather and climate variability as well as support long-term planning in the face of climate change. These investments have promoted a variety of approaches, targeted different actors in the value chain, engaged a number of different partners, and sought to develop a variety of different usable products. As part of an initial scoping exercise for this evaluation, the E3 Analytics and Evaluation Project20 prepared an initial list of USAID CS investments, which Annex B provides. Development Hypothesis USAID is interested in better understanding potential results from engagement strategies at different entry points or stages of the CS value chain. There are many existing diagrams of the CS value chain and how CS are intended to contribute to development objectives.21 During the design stage of this evaluation, the evaluation team will work with USAID’s Adaption team to clarify Agency perspectives regarding the CS value chain and how recent CS investments have contributed to broader Agency objectives. 19 This definition is based on the World Meteorological Organization’s Global Framework for Climate Services site: https://www.wmo.int/gfcs/what-are-climate-services. 20 Management Systems International, A Tetra Tech Company, leads implementation of the E3 Analytics and Evaluation Project in partnership with Development and Training Services, a Palladium company; and NORC at the University of Chicago. 21 See for example https://www.researchgate.net/figure/Weather-and-climate-services-value-chain_fig9_272094720 and https://www.slideshare.net/fionapercy/connectivity-and-codevelopment-of-climate-services, slides 9-14. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |46 Existing Performance Information Sources The following documents related to the Agency’s CS investments have already been shared by USAID’s AD team or obtained by the evaluation team: • An initial list of USAID CS investments since 2012. • A Google Spreadsheet listing additional USAID CS investments, including intended beneficiaries of the investment and links to data sources. • Websites with relevant information on USAID CS investments, including Climatelinks and The CGIAR Research Program on Climate Change, Agriculture, and Food Security. • Activity documents and studies related to CS, including those indicated in the table below. Activity Country Type/Year Implementer Mali Cereal Value Chain (CVC) Mali Annual Report 2017 ACDI/VOCA Adaptasi Perubahan Iklim dan Ketangguhan (APIK) Indonesia Annual Report 2017 DAI Assessing Sustainability and Effectiveness of Climate Information Services in Africa (Sustainable CIS) Africa Final Report 2019 Winrock International Climate Change Adaptation Project Preparation Facility for Asia and the Pacific (ADAPT Asia￾Pacific) Asia-Pacific Final Report 2017 AECOM Climate Information Services (CIS) Research Initiative – Africa Sub￾Saharan Africa Learning Agenda 2017 Mercy Corps Climate Information Services (CIS) Research Initiative – Africa Sub￾Saharan Africa A Learning Agenda 2019 Winrock/Mercy Corps Climate Information Services (CIS) Research Initiative – Africa Sub￾Saharan Africa A Learning Agenda 2018 Mercy Corps Climate Information Services (CIS) Research Initiative – Africa Sub￾Saharan Africa Final Report 2017 Mercy Corps Climate Services for Africa Africa Annual Report 2018 International Research Institute for Climate & Society Planning for Resilience in East Africa through Policy, Adaptation, Research and Economic Development (PREPARED) East Africa Annual Report 2017 Tetra Tech/ARD USAID’s AD team will help the evaluation team obtain additional activity documents from USAID staff and implementing partners as needed to carry out the evaluation. Annex A provides a draft list of USAID CS activities. Evaluation Purpose, Audiences, and Uses Purpose and Audiences This evaluation seeks to draw lessons learned from USAID CS investments in partner countries since 2012, including successes and challenges. These lessons learned will increase the understanding of past and ongoing interventions to inform future climate service activities and how they contribute to broader development programming. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |47 This evaluation is aimed at several audiences. First, USAID’s AD team has been supporting Washington￾and Mission-based Agency staff who have a wide range of experiences with CS programming, and is interested in ways to advance Agency goals through activities along the CS value chain in the context of different sectors. The new Bureau for Resilience and Food Security will be at the forefront of much of the Agency’s future CS programming, and is a key audience for this evaluation to inform its food security, resilience, and water programs. Additionally, the information from this evaluation may be useful to USAID missions and other operating units for the provision and design of CS across various countries and sectors (e.g., resilience and food security, health, disaster risk reduction, water). Intended Uses Results from this evaluation will be used to inform the pre-design, design, and implementation of future USAID CS investments, including where to invest along the CS value chain, expected results from such investments, and insights that may inform decisions around scaling up CS activities. The evaluation will also provide evidence and lessons learned to help USAID Mission staff determine, at the pre-design and design stages, whether to make CS investments and at what entry point or stage in the CS value chain. Evaluation Questions Prior to the finalization of this SOW, the USAID AD team commissioned the E3 Analytics and Evaluation Project to conduct a scoping exercise to help determine the best options for evaluating the Agency’s CS portfolio. Comparing CS activities presents challenges because of the diversity of CS provided, the types of partners engaged, the contextual differences in partner countries, and the types of U.S. government engagement to support these services. The Project team’s scoping support included consultations and interviews with key evaluation stakeholders, which resulted in a briefing note summarizing potential evaluation questions, methodologies, and CS investments for inclusion in the evaluation. Annex B provides the potential evaluation questions the Project team identified based on the scoping exercise. Following that scoping exercise, USAID prioritized and selected the evaluation questions (EQs) listed below to guide the evaluation. USAID expects that the evaluation will be conducted in three phases (see the following section for more details), with each phase building upon information obtained in the previous phase. In Phase 1, the evaluation team will prepare an inventory of USAID CS investments since 2012 that captures key characteristics and achievements of those investments based on available documentation and information provided by activity stakeholders. In Phases 2 and 3, the evaluation team will conduct a survey and key informant interviews (KIIs) to gather data to answer the following evaluation questions: 1. What lessons learned (including challenges and barriers) and good practices do USAID staff and other key partners perceive from USAID climate services investments since 2012? a. How do lessons learned and good practices vary at different points in the climate services value chain (e.g. data collection, capacity building, dissemination)? b. How do these results translate into recommendations for the design of new CS activities? 2. What kinds of results can USAID expect from investing at different levels (e.g., institutional level, field level) of the climate services value chain? a. In what ways have climate services investments added value or contributed to other USAID activity components? 3. What elements of USAID climate services investments are likely to be sustained beyond the end of an activity, and why? a. What types of enabling environments/local contexts appear to support positive outcomes across different levels of CS investments? USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |48 Evaluation Approach This evaluation will take an exploratory, phased approach utilizing existing secondary documentation and data as well as collecting primary data from relevant stakeholders of USAID CS investments. As described in the following section, USAID expects that the evaluation team will collect and analyze data to answer each EQ from activity data and documentation in conjunction with qualitative data collected from USAID staff, implementing partners, host-country officials, and/or activity beneficiaries. USAID expects that the evaluation will focus on key decision-focused support activities such as PREPARED and those implemented in Mali, Indonesia, Senegal, and Rwanda listed in Annex A). The evaluation team will also draw from other CS evaluations and studies and related data (where available and relevant). The evaluation will be conducted in three phases described below, with each phase building off information from the previous phase to ensure informed decisions are made throughout the evaluation process. Phase 1 – Inventory of USAID CS Investments. In Phase 1, the evaluation team will use available activity documentation22 and data to develop an inventory of all relevant CS activities the Agency has invested in since 2012 that captures various characteristics (e.g., investment type, entry point/stage of the CS value chain, outcomes). USAID AD team members will provide this documentation and data, to ensure all relevant activities are captured as well as the key aspects of each activity. This inventory will be useful for establishing a typology of CS activities and activity components that can be drawn upon for the two subsequent evaluation phases. In preparing the inventory, the evaluation team will collect data related to categories such as: activity name, country, budget, period of performance, short description of the activity and relevant interventions, type of CS, entry point/stage of CS investments, intended results, intended end users/beneficiaries, key partners/stakeholders, and reported results from activity monitoring and evaluation data/reports, and identified lessons learned from activity reports and evaluations (if available). The evaluation team will refine these categories in collaboration with USAID’s AD team and propose a list of categories in its evaluation design proposal. In addition to reviewing available activity documents and data, the evaluation team may conduct informational interviews or correspond with missions and other relevant USAID and implementing partner staff to fill in gaps from the available documentation. Additionally, the AD team may share a draft of the inventory with key missions to get feedback and validation on the draft list. The evaluation team will prepare two documents at the end of Phase 1: an Excel version of the inventory and a brief analysis of the inventory to inform decisions for Phase 2. The brief analysis will provide a concise overview of the inventory, summarizing the different types of activities/sub-activities mapped along the CS value chain. It will also include a discussion of other attributes of these activities (e.g. distribution across sectors, variations over time or in different regions). Prior to Phase 2, the evaluation team will convene with the AD team to discuss key areas of focus to frame survey questions around lessons learned the group wants to learn about (e.g., monitoring, evaluation, and learning good practices, activity design, integration). In addition, the team will begin discussing the focus and target activities or countries for Phase 3 field work. This discussion will include decisions around the overall sampling approach and whether to conduct deep dives into specific cases or take a more comparative analysis approach across activities. Once countries and activities are selected, the AD team will support the evaluation team in communicating and coordinating with the relevant Missions to prepare for the field work. 22 USAID’s AD team will assist the evaluation team as needed to obtain activity documents and data (e.g., activity annual reports, relevant assessment and evaluations, monitoring data). USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |49 Phase 2 – Lessons Learned from USAID CS Investments. Phase 2 of the evaluation will focus on identifying and documenting lessons learned and good practices from the implementation of CS activities as perceived by relevant activity stakeholders. The evaluation team will develop an online survey to be emailed by the team or a point of contact from USAID’s AD team to key CS activity points of contact (e.g., the contract or agreement officer’s representative; implementing partner staff such as the chief of party or deputy chief of party; host-country officials) for each investment identified in Phase 1. This survey will ask respondents to reflect on lessons learned and good practices from CS interventions based on key areas of interest drawn from Phase 1 (e.g., different outcomes of investments at different points in the CS value chain, outcomes with different types of implementing partners, outcomes with different degrees of integration with other USAID activities). The survey will focus on capturing some qualitative data (e.g., short answers) on design strategies, implementation successes and challenges, activity outcomes, partnerships, and other insights into good practices. Rather than attempting to comprehensively assess or compare specific CS activities, this Phase would focus on obtaining a broadly representative sample of the kinds of CS investments USAID has made since 2012 from which lessons learned and good practices can be extracted. Although online survey response rates can vary, this will allow the team to reach out to a broader group of respondents and a larger number than will be possible during Phase 3 data collection. Even with low response rates, surveys can still provide important and fruitful data to help answer the EQs. In addition, based on information gathered in the survey, the Phase 1 inventory will be updated if the survey data can fill in any gaps. Following the survey, the evaluation team will present some of the key quantitative findings to the AD team to help guide the discussion on the final planning for the site visits and KIIs, and key areas to focus data collections around (e.g., different entry points/stages of the value chain). Phase 3: KIIs and Site Visits - Informed by findings from the first two phases, the evaluation team will work with the AD team to determine the approach to collect qualitative data (e.g., KIIs) for Phase 3. Data collection for this phase may require short, targeted visits to a sample of up to three activities or countries, and may include interviews with host-country meteorological services professionals, infrastructure specialists, local farmers or other beneficiaries, or senior policymakers, depending on the activity element to be examined. During the discussion of the draft inventory and planning for the survey design at the end of Phase 1, the team will decide on the selection criteria for the data collection sites, as well as a data collection approach and format (e.g., case studies, comparative analysis). Prior to each country visit, the evaluation team may be requested to submit a trip statement of work to the relevant Mission(s) and/or conduct a debriefing for the Mission(s) (likely at the end of each field data collection visit). Following each country visit, the team will submit a short trip report summarizing the data collection process and interviewees. All field visits will require Mission concurrence prior to the team’s arrival in-country. The evaluation team will work with the AD team and relevant Mission(s) as required to secure this concurrence. After the completion of Phase 3 the evaluation team will provide USAID with two deliverables: the evaluation report summarizing results from all three phases, and a user-friendly summary of best practices, lessons learned, and recommendations from the evaluation. The user-friendly summary document will be designed to be easily shared with non-climate-oriented mission staff to help them determine whether to invest in a CS activity and if so, at what entry points/stages of the CS value chain and what good practices and lessons learned to consider in designing and managing a CS activity. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |50 Primary Data Collection and Analysis Methods It is anticipated that the evaluation will use a mixed-methods approach. In its evaluation design proposal, the evaluation team responding to this SOW will propose specific data collection and analysis methods to answer USAID’s EQs on a question-by-question basis. The proposal will also detail how choices will be made related to selecting the sample of respondents for each phase, and for selecting which activity components will be the subject of the data collection. USAID anticipates that data collection methods for this evaluation will likely include, but not necessarily be limited to: • A review of secondary documents and data from USAID CS investments, as well as non-USAID studies related to CS activities, and informational interviews if necessary; • An online survey to key CS activity points of contact identified in Phase 1; and • Key informant interviews and site visits for activities selected for Phase 3. Data analysis methods to be proposed by the evaluation team will follow closely from the methods used to collect each type of data needed to answer the EQs. USAID expects that the evaluation team will use content analysis to identify themes and trends relevant to each EQ and to better understand the meaning of, and context in which, statements were made by interviewees. The evaluation team will use qualitative data software to code interview notes/transcripts for content analysis. Gender Aspects of the Evaluation USAID evaluation policy guidance calls upon Agency staff and evaluation teams to examine EQs and processes from a gender perspective as well as report sex-disaggregated data and consider gender-specific and differential effects as appropriate. Gender aspects must be considered and incorporated during all phases of an evaluation. In answering each EQ, the evaluation team will seek to recognize and cut across the heterogenous and often interacting social and cultural strata to which respondents belong. The goal of gender and sub-group analysis for this evaluation will be to assess whether and how differences may exist in outcome sustainability based on identity and status. The evaluation team will conduct further inquiry on gender themes as they emerge during data analysis. The evaluation team will also be expected to apply gender-sensitive methods while conducting interviews to ensure that accurate data are collected. Strengths and Limitations The strengths and limitations of this evaluation will depend on the final design proposed by the evaluation team, in consultation with USAID. Anticipated strengths and limitations of the general evaluation design are described below. The final design should reflect a robust approach to answering the EQs and should address any further limitations and anticipated challenges to implementing the study design, along with proposed mitigation strategies for those limitations. Potential limitations that are likely relevant to this evaluation in comparing CS across multiple activities relate to the wide range of climate services provided, the different types of partners engaged, the differences in CS context in partner countries, and the types of U.S. government engagement to support these services. Strengths • Participatory and phased approach: It is expected that the evaluation will employ an exploratory and participatory approach in which the AD team will provide input at different stages on various components including the evaluation design, progress, and outcome. This will USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |51 help ensure the relevance and utilization of evaluation findings and lessons learned within the AD team as well as across the Agency. Limitations • Limited data: If documents and information from only a few CS interventions are provided, this could present a skewed picture of the overall services and their impact. The evaluation team will work hard to obtain information from all relevant stakeholders. • Response bias: The evaluation’s analysis will only be as strong as the data and information provided by stakeholders. Although there is always a risk of response bias, the evaluation team will develop data collection tools and sample to try to mitigate any bias. • Attribution: A non-experimental methodology lacks direct linkages to attribution. However, the expected evaluation approach of exploring multiple perspectives will allow for a deeper understanding of the interventions. The evaluation cannot test for causality directly, but results could identify key factors contributing to results. When possible, the evaluation team will incorporate empirical evidence to support the qualitative findings. Deliverables and Reporting The evaluation team will be responsible for the following deliverables and will provide a final list of proposed deliverables and due dates in its evaluation design proposal for USAID’s approval. Deliverable Estimated Due Date 1. Draft Evaluation Design Proposal o/a four weeks following USAID approval of this SOW 2. Revised Evaluation Design Proposal o/a two weeks following receipt of all written USAID feedback on the Draft Evaluation Design Proposal 3. Draft Inventory for Phase 1 To be proposed in Evaluation Design Proposal 4. Final Inventory for Phase 1 To be proposed in Evaluation Design Proposal 5. Draft Overview of Phase 1 Inventory To be proposed in Evaluation Design Proposal 6. Final Overview of Phase 1 Inventory To be proposed in Evaluation Design Proposal 7. Slide Deck for Oral Debrief to USAID on Phase 2 Results To be proposed in Evaluation Design Proposal 8. Trip Reports for Each Country Visit To be proposed in Evaluation Design Proposal 9. Draft Evaluation Report To be proposed in Evaluation Design Proposal 10. Draft Summary Document on Key Best Practices, Lessons Learned, and Recommendations To be proposed in Evaluation Design Proposal 11. Slide Deck for Presentation on Evaluation Results To be proposed in Evaluation Design Proposal 12. Final Evaluation Report To be proposed in Evaluation Design Proposal 13. Final Summary Document To be proposed in Evaluation Design Proposal The evaluation team will provide all documents and reports electronically to USAID. All qualitative and quantitative data will be provided in electronic format to USAID in a format consistent with Automated Directives System (ADS) 579 requirements. All debriefs will include a formal presentation with slides delivered both electronically and in hard copy for all attendees. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |52 Prior to the submission of the evaluation design proposal, the evaluation team will discuss with USAID whether its preliminary dissemination plan for this evaluation indicates other deliverables that should be prepared. Such additions as agreed with USAID will then be included in the evaluation design proposal. Team Composition The core team for this evaluation will consist of a team leader with extensive climate services experience, an evaluation specialist, and a research assistant. The evaluation specialist will serve as the evaluation coordinator and act as the primary E3 Analytics and Evaluation Project home office liaison to the team; she will oversee the team's progress towards the required deliverables and support data collection and analysis activities as needed. Support and quality assurance from the E3 Analytics and Evaluation Project home office will also be provided as required to ensure successful, on-time completion of the required deliverables. Additional team members for field work might be brought on later once Phase 3 is defined. The proposed team members for the evaluation are listed below and Annex C provides their CVs. The evaluation team will also sign USAID’s conflict of interest statement and return it to the E3 Analytics and Evaluation Project home office before data collection starts. TEAM LEADER AND SUBJECT MATTER EXPERT – FERNANDA ZERMOGLIO Fernanda Zermoglio, the team leader, is an adaptation specialist with extensive experience in the development and deployment of pragmatic tools and approaches to support decision managers in adaptation planning. With an education in geography and natural sciences, she combines applied research, innovative programming and policy analyses into the design of integrative tools and knowledge sharing platforms to evaluate the impacts of climate change on vulnerable systems. Her work in the last 20 years has been dedicated to improving the use of climate information in support evidence-based decision making, and building tools, training and communication packages for the use of climate information in policy and decision making. Mrs. Zermoglio is a skilled data analysist. Data management and using spatial and statistical tools for data analysis is at the root of her professional training and experience. She has leveraged these skills to strengthen the available climate information services to support strategic management capacities of municipal and national governments as they confront risks. Mrs. Zermoglio is an internationally renowned thought leader in data science related to development planning. She has worked in consultation with various development professional groups to evaluate climate and related risks across various sectors including health, transportation, emergency management and emergency response. She has served on several committees regarding her work on climate services, including the Intergovernmental Panel on Climate Change Task Group on Data and Scenario Support for Impact and Climate Analysis and the World Climate Research Programme’s Working Group on Regional Climate. Ms. Zermoglio will be primarily responsible for the quality of the evaluation design and its execution, and for drafting all evaluation deliverables. EVALUATION SPECIALIST AND COORDINATOR – GWYNNE ZODROW Gwynne Zodrow, the evaluation specialist and coordinator, is a technical manager with the Strategy, Evaluation, and Analysis team at MSI. Ms. Zodrow provides monitoring and evaluation support to multiple government and private sector clients in a variety of areas, including health, agriculture, and food security sectors. Ms. Zodrow has been involved in various evaluations and research studies from design to final report with experience in both qualitative and quantitative design and analysis methods. In addition to having expertise with evaluations and monitoring systems, Ms. Zodrow is also experienced in strategic planning and results-based management. Ms. Zodrow has worked on a range of evaluations and studies across multiple countries and topic areas. Ms. Zodrow currently works with USAID’s Ghana Evaluate for Health project providing technical assistance to the local team supporting implementing partners, the USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |53 Mission, and managing multiple health evaluations and a national survey. Previously, she coordinated an ex-post evaluation for an orphan and vulnerable children’s project in Uganda where she managed the teams progress from design to report writing, worked on a USAID evaluation on land tenure and with the World Cocoa Foundation to measure the impact of their multi-country farmer livelihood program in West Africa. Ms. Zodrow holds a Masters of Global Public Health from George Washington University and her Bachelor of Science in Sociology from Portland State University. Ms. Zodrow will support the team leader with the design and implementation of the evaluation. In coordination with the team leader, she will be responsible for developing the evaluation methods and data collection instruments, as well as supporting data collection and analysis. RESEARCH ASSISTANT – JORGE SALINAS Jorge Salinas, the research assistant, holds a master’s degree in sustainability leadership, for which he completed a capstone project focused on building environmental sustainability on the grassroots level in minority communities. During his program, he completed extensive coursework in global environmental policy, communication for sustainability, and development and global technology. These academic frames of reference have endowed him with holistic, global knowledge of sustainability programs, policies, and topics, and the ability to effectively analyze both the private and public sectors. Mr. Salinas has over five years of USAID project management experience, which include managing MSI’s PROJUST project in Mexico for USAID and previous departmental leadership of global translation staff. Mr. Salinas will work in close coordination with the team leader and evaluation specialist to support the development of the inventory and data collection and analysis activities. LOCAL LOGISTICIANS/RESEARCHERS – TBD Local consultants may be utilized in countries where field work takes place to assist in coordinating interviews, setting up focus group discussions, arranging in-country travel, taking notes, obtaining data and relevant documents, conducting additional research, providing translation support, and other in-country activities as required by the team leader and evaluation specialist. HOME OFFICE SUPPORT The E3 Analytics and Evaluation Project home office will support the core evaluation team through technical coordination and guidance, quality control assurance, research assistance, quantitative and qualitative data analysis, administrative oversight, and logistical support. USAID Participation Regular communication between the evaluation team and the designated USAID activity manager for this evaluation will be essential to the successful execution of evaluation activities. An interactive and collaborative process is envisioned between the evaluation team and USAID’s AD team to carry out the evaluation. The AD team will be engaged during the design process to ensure agreement on the focus and approaches for the design and delivery of the evaluation. The evaluation team will keep the USAID activity manager apprised of changes and developments that necessitate/require any significant decision-making or modification of the approved evaluation design proposal. Possible USAID participation in the data collection phase of the evaluation will be determined prior to the start of field work. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |54 Schedule and Logistics The following illustrative chart provides an overview of the anticipated timeframe for the activities described in this document. The evaluation design proposal will include a detailed schedule and proposed delivery dates. The overall period of performance for completion of the evaluation is expected to last from approximately October 2019 to August 2020. The schedule below assumes USAID approval of this SOW by late November 2019. The evaluation team will be responsible for all logistics, including coordinating all in-country travel, lodging, printing, office space, equipment, car rentals, etc. USAID’s AD team will provide support to set up initial meetings with USAID stakeholders and implementing partners, and other stakeholders as appropriate. Reporting Requirements The evaluation report will follow USAID guidelines set forth in the agency's Evaluation Report Template23 and How-To Note on Preparing Evaluation Reports24 as well as the Mandatory Reference for Automated Directives System 201 on USAID Evaluation Report Requirements. The final evaluation report should not exceed 30 pages, excluding references and annexes. The evaluation team will deliver a copy of the final evaluation report to USAID’s Development Experience Clearinghouse (DEC) within 30 days of COR approval to post it on the DEC. The evaluation report must contain at least the following: • Abstract: A summary of the key evaluation results in no more than 250 words, including relevant information about the evaluation background, objectives, methods, and key findings, conclusions, and recommendations. • Executive Summary: This section should be no more than five pages in length and describe the purpose, background, evaluation design and methodology including the EQs, and key findings, conclusions, recommendations, and lessons learned (if applicable) from the evaluation. • Background: This section will provide a brief description of USAID CS investments. • Methodology: This section will detail the methodology and related research protocols undertaken in conducting the evaluation, including the relevant data collection and analysis methods, sampling approach, and related challenges or limitations encountered during the evaluation and mitigation approaches employed. • Findings: This section will present findings collected from the evaluation relevant to each EQ. The evaluation findings must be presented as analyzed facts, evidence, and data and not be based on anecdotes, hearsay, or the compilation of people’s opinions. The findings must be specific, concise, and supported by the quantitative and/or qualitative evidence analyzed through scientifically plausible methodologies. Sources of information used in arriving at the findings must be properly acknowledged and listed in an annex. • Conclusions: The evaluation report will present evaluation conclusions that are interpretations and judgments based on the findings described, and must logically follow from the gathered data and findings and be explicitly justified. If necessary, the evaluation team will state its assumptions, judgments, and value premises in presenting a conclusion so that readers can better understand and assess them. • Recommendations: This section will concisely and clearly present recommendations that are drawn from specific findings and conclusions provided in the report. The recommendations must be stated in an action-oriented fashion and be practical, specific, and with defined target audience(s). 23 See http://usaidlearninglab.org/library/evaluation-report-template 24 See http://usaidlearninglab.org/library/how-note-preparing-evaluation-reports USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |55 Following receipt of USAID’s comments on the draft evaluation report, the evaluation team will prepare a final version that incorporates and responds to this feedback. The final evaluation report should contain the same sections as noted above for the draft evaluation report and should also include: ● References: This section should include a list of all documents reviewed as well as individuals interviewed (keeping respondent anonymity as applicable). ● Annexes: These may include, but are not limited to, the evaluation statement of work, instruments used in conducting the evaluation, any statements of differences received, as well as other relevant sources of information. All members of the evaluation team should be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet, shown in the text box below. Data Management The storage and transfer of data collected for this evaluation will adhere to the requirements laid out in ADS 579.25 Final datasets are expected to be submitted to USAID’s Development Data Library as required in a format consistent with ADS 579. 25 See http://www.usaid.gov/sites/default/files/documents/1868/579.pdf MANDATORY REFERENCE FOR ADS CHAPTER 201 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Pursuant to 201.3.5.17, draft evaluation reports must undergo a peer review organized by the office managing the evaluation. The following criteria should serve as the basis against which the report is reviewed. To help ensure a high-quality evaluation report, these criteria must be included in the evaluation Statement of Work to communicate to evaluators USAID’s quality criteria. • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |56 ANNEX 2: INVENTORY OF USAID CLIMATE SERVICES ACTIVITIES REVIEWED # Initial Typology Continent Country/Countries Subregion Activity Name 1 Learning Africa Senegal, Ethiopia, Rwanda, Malawi, Mali, Cote d’Ivoire, and Niger Africa regional Learning Agenda on Climate Services 2 Field Based Africa West Africa West Africa West Africa Biodiversity and Climate Change (WA-BiCC) 3 Data Provision Africa Ghana Sub-Saharan Africa Agricultural Development and Value Chain Enhancement (ADVANCE) 4 Data Provision Africa Africa regional Famine Early Warning Systems (FEWSnet) 5 Field Based Africa Kenya East Africa Scaling the Impact of USAID Resilience Programming Through Local Systems (USAID Kuza) 6 Data Provision Africa East Africa East Africa Enhancing National Climate Services (ENACTS) 7 Data Provision Africa Burundi, Kenya, Rwanda, Tanzania, and Uganda Africa regional Planning for Resilience in East Africa through Policy, Adaptation, Research, and Economic Development (PREPARED) 8 Data Provision Africa Angola Southern Africa Angola GCCI 9 Field Based Africa Ethiopia, Cote d'Ivoire, Malawi, Niger, Rwanda, and Senegal Africa regional Assessing Sustainability and Effectiveness of Climate Information Services in Africa 10 Field Based Africa Mali West Africa USAID Cereal Value Chain 11 Field Based Africa Mali West Africa USAID Mali Climate Change Adaptation Activity (MCCAA) 12 Data Provision Africa Mozambique Southern Africa Climate Resilient Infrastructure Services (CRIS) - Nacala Pilot Site 13 Field Based Africa Rwanda East Africa Climate Services for Agriculture (CCAFS) 14 Field Based Africa Senegal West Africa Climate Change Vulnerability Assessment and Options Analysis, and Assessing Climate Service Needs Reports 15 Field Based Africa Senegal West Africa Climate Information Services for Increased Resilience and Productivity in Senegal (CINSERE) 16 Field Based Africa Uganda Sub-Saharan Africa Strengthening Meteorological Products, Services and Use in the Agricultural and Water Sectors 17 Field Based Asia Indonesia Pacific Adaptasi Perubahan Iklim dan Ketangguhan (APIK) 18 Field Based Asia Kazakhstan Central Asia Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security project (integration pilot through CCRD) 19 Field Based Asia Kazakhstan Central Asia Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security (CRW) 20 Learning Asia Nepal, Tibet, Mongolia Asia regional High Mountains Adaptation Partnership (HiMAP): Quantifying Supraglacial Lake Changes: Contributions to Glacial Ice Volume Loss and Runoff Inputs to Rivers in Nepal and Tibet (implemented by Ulyana Nadia Horodyskyj) 21 Field Based Asia Philippines Pacific Be Secure USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |57 # Initial Typology Continent Country/Countries Subregion Activity Name 22 Data Provision Asia Vietnam Southeast Asia Climate Resilient Infrastructure Services (CRIS) - Hue Pilot Site 23 Data Provision North America Barbados & Eastern Caribbean Caribbean Programme for Building Regional Climate Capacity in the Caribbean 24 Data Provision North America Caribbean Caribbean Caribbean Community Climate Change Centre (CCCCC) 25 Data Provision North America Dominican Republic Caribbean Improved Climate Information 26 Data Provision North America Dominican Republic Caribbean Climate Resilient Infrastructure Services (CRIS) - Santo Domingo Pilot Site 27 Data Provision North America East Caribbean and Guyana Caribbean Building Regional Climate Capacity in the Caribbean (BRCCC) 28 Data Provision North America Jamaica Caribbean Climate Predictability Tool (CPT) 29 Learning North America Jamaica Caribbean Climate Service Partnership: International Conference on Climate Services 30 Data Provision Asia Mekong Asia regional SERVIR – Mekong 31 Field Based Central America Honduras, Nicaragua, Costa Rica, Guatemala, Panama Central America CentroClima 32 Data Provision Central America Honduras, Nicaragua, Costa Rica, Guatemala, Panama Central America Clima Pesca 33 Data Provision South America Colombia Andean Clima y Café 34 Field Based Africa RISE I and RISE II Sahel Resilience in the Sahel Enhanced 35 Field Based South America Colombia, Peru Andean Partnering for Adaptation and Resilience Agua (PARA-Agua) 36 Data Provision South America Colombia, Peru Andean CIFEN - cuencas climaticamente resilientes Increasing Climate Change Resilience and Reducing Vulnerability in Chinchina (Colombia) and Mantaro (Peru) Climate Resilient Basins Project 37 Field Based South America Colombia, Peru, Ecuador Andean CAFTA-RD Tratado de Libre Comercio de Centroamérica y la República Dominicana 38 Learning South America Peru Andean High Mountains Adaptation Partnership (HiMAP): Community Water Management in the Tres Cuencas Commonwealth (implemented by Laura Reed) 39 Data Provision South America Peru Andean Climate Resilient Infrastructure Services (CRIS) - Piura River Pilot Site 40 Data Provision South America Peru Andean Natural Infrastructure for Water Security USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |58 ANNEX 3: DOCUMENTS INCLUDED IN THE DOCUMENT REVIEW # Activity Name Report Title 1 Angola Global Climate Change Initiative (GCCI) Mid-Term Assessment (2015) 2 Assessing Sustainability and Effectiveness of Climate Information Services in Africa (Sustainable CIS) Final Report 2018 3 Enhancing National Climate Services (ENACTS) Article – ENACTS for development in Africa 2015 4 ENACTS Article – ENACTS: Transforming Climate Services across Africa 2019 5 Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security (CRW) Mid-Level Assessment: Climate Forecasting in Kazakhstan 2014 6 CRW Final Report 2014 7 CRW Evaluation CRW 2015 8 Mekong ARCC USAID Mekong ARCC Climate Change Impact and Adaption Study for Lower Mekong Basin: Main Report 2013 9 Learning Agenda on Climate Services Summary Report: Innovative Qualitative approaches for CIS monitoring and Evaluation (2019) (under CISRI) 10 Learning Agenda on Climate Services Identifying climate information services users and their needs in sub-Saharan Africa: A learning agenda (2017) (under CISRI) 11 Learning Agenda on Climate Services Evaluating Agricultural Weather and Climate services in Africa 2017 (under CISRI) 12 Learning Agenda on Climate Services Climate information for those who need it most: Contributions of a participatory systems mapping approach in Nigeria (2018) (under CISRI) 13 Learning Agenda on Climate Services Synthesis Report: Improving the monitoring and evaluation of CIS to facilitate learning and improve outcomes (2019) (under CISRI) 14 Learning Agenda on Climate Services National Meteorological and Hydrological Service Financial Planning Tool: User Manual (2018) (under Sustainable CIS) 15 Learning Agenda on Climate Services Approaches to combine technologies for weather observation, storage and analysis (2018) (under Sustainable CIS) 16 Learning Agenda on Climate Services Approaches to collect, exchange and integrate national and global datasets (2018) (under Sustainable CIS) 17 CIFEN Final Report 2019 (Spanish) 18 Mali IRI Climate services partnership – Mali Meteo Final Technical Report (2016) 19 Mali IRI Climate services partnership – Mali Meteo Development of Metrics to assess National meteorological services in Africa (2018) (Under Sustainable CIS) 20 Mali IRI Climate services partnership – Mali Meteo Institutional analysis of l’agence de l’environnement et du développement durable (aedd) and l’agence nationale de la météorologie (Mali-météo) (2014) USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |59 # Activity Name Report Title 21 Partnering for Adaptation and Resilience Agua Project (PARA-Agua) Evaluation 2015 22 PARA-Agua Gap Analysis 2014 (Spanish) 23 PARA-Agua Final Report 2017 24 Learning Agenda on Climate Services Climate information services market assessment and business model review (2018) (under Sustainable CIS) 25 Adaptasi Perubahan Iklim dan Ketangguhan (APIK) Fact Sheet Sugarcane Climate Field School (CFS) for community economic Resilience (2018) 26 APIK Annual Report Year 4 (2018/19) 27 Be Secure Final Report 2016: Technical services for participatory vulnerability assessment, capacity building, and participatory development Planning in Iloilo City 28 Be Secure Final Report 2017 29 Climate Services for Agriculture (CCAFS) Annual Highlights 2019 30 CCAFS Estimating the economic benefits of alternative options for investing in agricultural climate services in Africa: A review of Methodologies, 2018 31 CCAFS Review of climate service needs and opportunities in Rwanda, 2016 32 Planning for Resilience in East Africa through Policy, Adaptation, Research and Economic Development (PREPARED) Mid-term Evaluation 2017 33 PREPARED Tanzania Meteorological Agency Climate data rescue pilot project report December 2016 34 Strengthening Meteorological Products, Services and Use in the Agricultural and Water Sectors Evaluation Report 2018 35 The Caribbean Community Climate Change Centre (CCCCC) Annual Report 2019 36 The Programme for Building Regional Climate Capacity in the Caribbean (BRCCC) Final Report 2017 37 BRCCC Evaluation Report 2017 38 BRCCC OECS – USAID RRACC final report 39 USAID Cereal Value Chain (CVC) Mid-term Eval (2017) 40 CVC Report Launch of Initiative Enhancing National Climate Services (ENACTS) in Mali 2016 41 CVC Assessing Mali’s Direction Nacionale De la Meteorologie advisory program: Preliminary report on Climate science and farmer use of Advisories (2014) 42 USAID/Mali Climate Change Adaptation Activity (MCCAA) Effectiveness Study 2019 43 CVC Final Report 2018 44 Strengthening Meteorological Products, Services and Use in the Agricultural Sectors Final Report 2017 USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |60 ANNEX 4: CODES FOR PHASE 2 DOCUMENT REVIEW EQ Code Definition General Information General project/activity description Background/descriptive information about the project or activity Project goal or theory of change The top objective or theory of change the activity or project is working toward Project/activity budget Overall activity budget or funding from USAID Value chain focus Where on the value chain the activity is working to contribute to (e.g. data, information, knowledge, action) Dates Dates of activity implementation Country Country where the activity is being implemented or targeting Value Chain Definition Data The capture of data. The generation of observations and models of weather and climate information through remote sensors, etc. This can include through external providers or in-country providers (e.g. national hydrometeorological services). Information The provision of data in an accessible form to make it useful. This can be through media, ICT, radio/SMS, agricultural extension agents, NGOs, government, etc. Knowledge Engaging users in understanding and contextualizing or tailoring the information for specific contexts and needs. This can be through media, ICT, radio/SMS, agricultural extension agents, NGOs, government, etc. Action The dissemination of knowledge together with recommended decisions into action by end users at community level (e.g., farmers, pastoralists, vulnerable populations) or national-level users (e.g., rural development planners, disaster managers, public health, dam builders, private sector) CODES FOR EQ 1 Lessons learned Any text that specifically states something as a lesson learned and/or has lesson learned label or reference. These should be lessons learned explicitly called out in the document and not determined to be a lesson learned by the coder. (Explicit lessons learned will be coded to other codes and pulled out as lessons learned during the analysis.) 1 Challenges/barriers (programing) Any reference to obstacles or barriers that prevented or slowed down implementation or delivery of CS or the achievement of the overall investment goal. These should be challenges or barriers related to programming. (External challenges should be coded under limiting environment.) 1 Recommendations for future CS programming This includes recommendations from activity reports and evaluations that are relevant to future CS programming (e.g., design, implementation) Other recommendations All other recommendations not related to programming should be coded here (e.g., for government partners, policy recommendations) 2 Mid-term results Any activity results reported at mid-term and NOT end-of-activity results. This should be used for data in quarterly reports or any annual reports that are not the final year. 2 End-of-activity results Any result data including activity reported indicators, evaluation findings, and other data collected at the end of the activity. This includes findings from end-of-activity reports, final evaluations, etc. 2 Contribution toward other activities Explanation or examples of ways CS investments have added value or contributed to other USAID activity components or sectors. This includes text that connects the CS invention with larger activities or partner activities. 3 Sustainability Reference to whether or how the investment will be sustained in the future. 3 Post-activity results Any findings from an ex-post evaluation or other reports that have data at least one year after the activity ended. This could be information from ex-post evaluations or USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |61 EQ Code Definition other reports describing results of interventions in which USAID programming/investment had already ended. 3 Enabling environments (external) Explanation or examples of any external situation or context that helped support the implementation of the activity or the achievements of the investment or CS. This can be related to infrastructure, policy, government funding, etc. 3 Limiting environments (external) Any external situation or context that inhibited the implementation of the activity or the achievements of the investment or CS. This can be related to infrastructure, policy, government funding, etc. 3 Existing gaps in CS programming Explanation or examples of any gaps in programming or investments that could have helped the activity be more successful or that can be supported with future programming. These should be specific things called out by the activity or evaluation report as missing or a gap in the USAID programming or investment. USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |62 ANNEX 5: SURVEY INSTRUMENT 1. Introduction This survey is part of an evaluation of USAID’s investments in climate services. These investments use climate information to assist decision making across sectors. The Office of Global Climate Change in USAID’s Bureau for Economic Growth, Education, and Environment commissioned the evaluation to identify best practices and lessons learned that can inform future climate services programming. USAID contracted the E3 Analytics and Evaluation Project, led by Management Systems International (MSI), to conduct the evaluation. Based on your experience with USAID climate work and/or programming, your perspective and input will provide us with helpful information to understand USAID’s past work and how it can be improved in the future. This survey should take approximately 10-20 minutes. All interested participants can also receive a copy of the final report by providing an email address at the end of the survey. For the purposes of this survey, “climate services” are defined as the collection, production, transfer, and use of climate information to guide countries, communities, households, and individuals to anticipate and manage climate and weather risks and opportunities. If you have any questions or technical issues, please contact Management Systems International’s evaluation coordinator Gwynne Zodrow at gzodrow@msi-inc.com. This survey is completely voluntary and your responses are confidential. You can stop at any point during the survey or decline to answer any of the questions. We will assign a numeric ID to each survey and remove any information that could identify you personally from the data and analysis. We will use personal information only for evaluation research purposes and will not share it with any third party. 1. Before you start, do we have your consent to participate in this survey? (Select one option) o Yes o No 2. General information 2. What type of organization do you currently work for? (Select one) (All non-USAID selections go to question 7 or 8, if select USAID go to 3). ▪ USAID (Go to question 3) ▪ Private business ▪ Research institution or university ▪ Non-governmental organization (NGO) ▪ Government agency ▪ Civil society organization ▪ I currently do not work for an organization (Go to question 8) ▪ Other (fill in the blank) 3. Please indicate your position within USAID (check all that apply) (USAID ONLY RESPONDENTS) o AOR/COR for climate services activities o Climate Integration Lead (CIL) o Program office o Technical staff USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |63 o Leadership o Other (fill in the blank) 4. What is your main region of focus for your climate work at USAID? (Select all that apply) (USAID ONLY RESPONDENTS) ▪ Latin America and the Caribbean i. South America ii. Central America iii. Caribbean ▪ Asia i. Southeast Asia ii. Central Asia iii. Asia Pacific ▪ Africa i. West Africa ii. East Africa iii. Southern Africa iv. Central Africa ▪ Middle East ▪ I work across multiple regions and have no specific focus ▪ Regional Bureau ▪ Pillar Bureau ▪ Other (fill in the blank) 5. What sectors do you primarily work with? (Select all that apply) (USAID ONLY RESPONDENTS) ▪ Agriculture ▪ Water management ▪ Natural resource management ▪ Forestry ▪ Fisheries ▪ Disaster risk reduction ▪ Climate services ▪ Climate adaptation ▪ Health ▪ Water, Sanitation and Hygiene (WASH) ▪ Other: ______________ 6. Have you worked on USAID climate services-related activities in the last five years? USAID ONLY RESPONDENTS) ▪ Yes (Go to question 8) ▪ No (Skip to question 23) ▪ I have not worked directly on any USAID climate services-related activities, but I have experience with climate service-related programming. (Go to Question 8) 7. What sectors does your organization work in? (Select all that apply) (NON-USAID RESPONDENTS ONLY – go to question 8) ▪ Agriculture ▪ Water management ▪ Natural resource management ▪ Forestry USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |64 ▪ Fisheries ▪ Disaster risk reduction ▪ Climate services ▪ Climate adaptation ▪ Other (fill in the blank) 3. USAID Activity Information Note: For the purposes of this survey, “climate services” is defined as the collection, production, transfer, and use of climate information to guide countries, communities, households, and individuals to anticipate and manage climate and weather risks and opportunities. 8. What USAID or other climate services-related activities have you worked on (or have knowledge of) over the last 5 years? (List up to 5) (USAID that did not select “no” on question 6, and all other respondents) ▪ ______________________ ▪ _____________________ ▪ ____________________ ▪ ______________________ ▪ ________________________ Thinking about any one of the climate services activities you listed above, please answer the rest of the questions on this page regarding your experiences with that activity 9. Which sectors does/did the activity work in? (Select all that apply) ▪ Agriculture ▪ Water management ▪ Natural resource management ▪ Forestry ▪ Fisheries ▪ Disaster risk reduction ▪ Climate services ▪ Climate adaptation ▪ Water, Sanitation and Hygiene (WASH) ▪ Other (fill in the blank) 10. What is/was the climate services activity’s main purpose? (select all that apply) ▪ To inform field-level investments ▪ To translate information into usable formats to improve key stakeholder utilization ▪ To improve the quality of weather and climate services data ▪ To make a case for future climate services investments ▪ To co-produce weather and climate information with key stakeholders ▪ To guide further climate services research ▪ Other (fill in the blank) 11. In which region(s) is/was this activity implemented? (Select all that apply) ▪ North America ▪ South America ▪ Central America ▪ Caribbean ▪ Southeast Asia USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |65 ▪ Central Asia ▪ Asia Pacific ▪ West Africa ▪ East Africa ▪ South Africa ▪ This activity had no specific regional focus ▪ Other (fill in the blank) 12. What are/were the main climate risks that you saw being addressed across the activity’s region(s)? (Select all that apply) ▪ Unreliable rainy seasons ▪ Increased temperatures ▪ Extreme heat ▪ Extreme weather-related events such as floods, droughts, or cyclones ▪ Other (fill in the blank) 13. What kind of weather or climate information did the activity work to collect, analyze, or disseminate? (Select all that apply) ▪ Short term weather forecasts ▪ Seasonal forecasts ▪ Historical climate trends analyses ▪ Climate projections ▪ The activity did not work with weather or climate data ▪ Other (fill in the blank) 14. Please select the main targeted stakeholders who were engaged in the activity. (Select all that apply) ▪ Government (e.g. ministries) ▪ Extension services ▪ Local or community governments ▪ Farmers ▪ Meteorological service providers ▪ Research institutions or universities ▪ Civil society organizations ▪ Media organizations ▪ Other (fill in the blank) 15. Did the activity distribute or share weather or climate data with stakeholders? ▪ Yes ▪ No (skip to question 17) 16. How did the activity distribute or share weather or climate data with stakeholders? (Select all that apply) (IF QUESTION 15 IS “YES”) ▪ Online (e.g., website, open dashboard, social media, datasets) ▪ Bulletins/newsletters ▪ Text/SMS messages ▪ Email ▪ Mail ▪ Reports or journal articles ▪ Presentations/discussions at meetings or events USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |66 ▪ Television or radio ▪ Other (fill in the blank) 17. Did the activity use any of the following methods to track how stakeholders were using the climate services information they were provided? (Select all that apply) ▪ Online surveys ▪ Polls or surveys through phone (e.g., WhatsApp, SMS) ▪ In-person surveys or interviews ▪ Feedback from workshops or meetings ▪ The activity used none of these methods ▪ Other (fill in the blank) 18. Did the activity face any of the following challenges or barriers during implementation? (Select all that apply) ▪ Stakeholders did not use the same data management software and tools ▪ Varying capacity among personnel in different organizations (e.g., government, private sector, research institutions) ▪ Timing issues (e.g., data is collected at irregular intervals or not concurrently with other relevant data or user needs) ▪ End users did not have confidence in the information. ▪ Disconnect between researchers and decision-makers (e.g., lack of partnership and collaboration between key stakeholders or poor communication) ▪ Limited dissemination and overall reach of climate services information. ▪ Limited resources for users to implement change based on climate information (e.g., farmers do not have knowledge or resources to change farming practices) ▪ Limited engagement of women accessing climate services ▪ The activity did not face any of these challenges or barriers ▪ Other (fill in the blank) 19. What were the main climate services-related accomplishments or successes of this activity? (List up to 3) ▪ _______________ ▪ _______________ ▪ ________________ 20. What good practices or specific approaches to climate services did the activity use that you believe could be useful to improve results of other future climate services activities? (List up to 3) ▪ _______________________ ▪ _______________________ ▪ ________________________ 21. Based on your experience, what could the activity have done differently to improve the outcomes/impact of this activity? (List up to 3) (ALL non-USAID respondents go to section 5, all USAID respondents continue to question 22) ▪ _______________________________ ▪ ______________________________ ▪ ________________________________ USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |67 4. General Information on Climate Risk and Services 22. Based on your experience, what are the main climate risks that are currently being addressed across your region(s) of focus? (Select all that apply) (ALL USAID respondents) ▪ Unreliable rains ▪ Rising temperatures ▪ Extreme heat ▪ Altered or unpredictable seasons ▪ Extreme weather-related events such as floods, droughts, or cyclones ▪ Other: _____________________________________ 23. What climate risks that are not currently being addressed do you think should be the focus of new activities in your region(s) of focus? (List up to 3) (ALL USAID Respondents) ▪ _______________________________ ▪ ___________________________________ ▪ __________________________________ 24. What do you think are currently the largest challenges in the implementation of climate services programming in your region(s)? (List up to 3) (All USAID Respondents) ▪ _____________________________ ▪ ______________________________ ▪ ______________________________ 25. Based on the climate services value chain provided below, where do you think is the greatest need for climate services investment in your region? (Select all that apply) (All Respondents) ▪ Data (observations, models) ▪ Information (data put into context) ▪ Communication (translating information into key impacts) ▪ Decisions (understanding the set of choices that can be made based on the information ▪ Other (fill in the blank) 26. Please briefly explain why you selected this area on the value chain. (All Respondents) a. ______________________________________ 28. What do you think are the main barriers that limit the use of climate and/or climate risk data in your region? (Select all that apply) (All Respondents) ▪ Limited availability of relevant data ▪ Lack of awareness of available climate or climate risk data ▪ Limited technical expertise in using climate or climate risk data ▪ Access issues with obtaining climate or climate risk data (e.g., hard to download, cost) ▪ Access issues due to information being in a different language ▪ I have not experienced any barriers to my climate or climate risk data use ▪ Other (fill in the blank) Data Information Communication Decisions USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |68 5. Conclusion Thank you for taking the time to complete our survey. This information will be very helpful in informing the evaluation. 29. Do you have any additional comments about your experience with climate services you would like to share? (all respondents) ▪ _________________________________________________ We expect that the final evaluation report of USAID’s climate services investments will be publicly available on USAID’s Development Experience Clearinghouse by the fall of 2020. 30. If you would like a copy of the report, please enter your email here: ____________________________________ USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |69 ANNEX 6: KEY INFORMANTS INTERVIEWED All interviews were conducted virtually over Google Hangout. All interviews were recorded and transcripts were developed for analysis. # Name Organization Country/Region Interview Date 1 Oumou Ly USAID Senegal August 5, 2020 2 Luis Ramos USAID El Salvador August 5, 2020 3 Maria Elena Santana USAID Colombia August 5, 2020 4 Samantha Wapnick USAID Sahel (West Africa) August 5,2020 5 Kuhong Tran Chinh USAID Vietnam August 5,2020 6 Jean Damascene Nyamwasa USAID Rwanda August 6. 2020 7 Napak Tesprasith USAID RDMA/Thailand August 10, 2020 8 Roopa Karia USAID RDMA/Thailand August 10, 2020 9 Mansfield Blackwood USAID Caribbean August 14, 2020 10 Nikki Hassell USAID Caribbean August 14, 2020 11 Amadou Diane USAID Mali August 11, 2020 12 Katia Villanueva USAID Peru August 31, 2020 USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |70 ANNEX 7: KEY INFORMANT INTERVIEW GUIDE Evaluation of USAID’s Climate Services Investments Interview Guide for Projects included in the Inventory Introduction Hello, my name is ___________. I am a consultant with MSI/Palladium conducting an evaluation of USAID’s climate services investments. This evaluation is being conducted for the Office of Global Climate Change in USAID’s Bureau for Economic Growth, Education, and Environment to identify best practices and lessons learned that can inform future climate services programming. Based on your experience with USAID climate work and programming, your perspective and input will provide us with helpful information to understand USAID’s past work and how it can be improved in the future. This interview will take 45-60 minutes. We will digitally-record the session, but your responses will remain anonymous. However, if you agree, we will include you in a list of people interviewed in the report. To help frame our discussion, for the purposes of this interview, “climate services” are defined as the collection, production, transfer, and use of climate information to guide countries, communities, households, and individuals to anticipate and manage climate and weather risks and opportunities. Do you have any questions about the evaluation? Do you consent to this interview? If NO, Stop and thank the person for their time. If YES, Do you consent to being recorded? If NO, tell the interviewee that you will just be taking written notes instead and continue to interview questions. If YES, state that you thank them for consenting to being recorded and then continue to interview questions. Interview Guide Questions Background and General Climate Services (CS) questions 1. What is your current role at USAID? a. How do CS relate to your work at USAID? b. How long have CS been relevant to your work at USAID? c. What USAID or other projects have you worked on either directly or seen that have a CS component? d. Does your CS work focus on certain regions or countries? e. Does your CS work focus on certain aspects of the CS value chain? (i.e. investing in new observation networks (data), communicating available information to specific users (communication) translating relevant data for specific users (information), or specifically looking at actions that can be taken in response to climate information?) 2. How have CS investments been used in support of project goals in the country or region you work in? a. In what ways have you seen CS investments add value to the project goals? b. Who would you describe as the main in-country stakeholders for these investments? c. What were the successes of these CS investments from your perspective? Why? d. What types of enabling environments/local contexts do you believe supported these successes? e. What types of environments/local contexts do you believe were a hindrance to the success of the project? USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |71 f. What types of environments/local contexts do you believe are most suitable for the success of CS investments? 3. What do you think are currently the largest gaps in CS programming in the country or region you work in? a. Please explain why you think these are the largest gaps. b. What gaps exist across the CS value chain that could inform future investments in the country or region you work in? Where are the weakest links in the value chain in your opinion and what could be done programmatically to address these links? Programming specific questions 4. Do you think USAID CS investments have adequately responded to the needs of the different stakeholders in your country/region of focus? a. If so, how? Please provide some examples. b. If no, why not? c. How has this response been different for each stakeholder type (e.g. end users, government institutions)? d. What would you have done differently knowing what you know today? Or How would you design a new program now based on this knowledge? 5. Do you think USAID CS investments adequately responded to the differential needs of women? a. If so, how? Please provide some examples. b. If no, why not? c. How has this response been different for each stakeholder type (e.g. end users, government institutions)? d. What would you have done differently knowing what you know today? Or How would you design a new program now based on this knowledge? 6. Do you think USAID CS investments have led to changes in the way stakeholders engaged in the projects that have CS components make their decisions based on CS information? a. If yes, how? Please explain and provide examples. For example, what decisions were you trying to influence? b. If no, why not? Please explain and provide examples. 7. What barriers did you encounter in the delivery of CS to end users in your region/country of focus? a. Where are the obstacles preventing climate information from reaching end users when and where needed? b. What conditions (e.g., institutional arrangements, available technology, personnel, and others) have contributed to the improved delivery of CS by end users? c. Do you have any good practices to share from your experience in improving the delivery of CS services among end users? 8. What barriers exist in the uptake of CS by end users in your region/country of focus? a. Where are the obstacles preventing climate information from being applied in support of decisions by end users when and where needed? b. What conditions (e.g., institutional arrangements, available technology, personnel, and others) have contributed to the increased use of CS by end users? c. Do you have any good practices to share from your experience in increasing the uptake of CS services among end users? USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |72 9. What barriers exist to the delivery of CS to end users in your region/country of focus? a. What are the main challenges when trying to deliver CS to end users? b. What conditions (e.g., institutional arrangements, available technology, personnel and others) have contributed to enhance the delivery of CS to end users? c. Do you have any good practices to share from your experience in increasing the delivery of CS services to end users? External Factors and Sustainability 10. In general, what do you think are the largest challenges or barriers to improving livelihoods through implementation via USAID CS programming? a. Do you have any recommendation based on your experience on how to address or mitigate these challenges or barriers? 11. What factors do you think contribute to the sustainability of CS investments? a. Why? Please explain. b. Do you think these factors have been incorporated into USAID CS-related programming in your region? Why or why not? c. How could these factors be better incorporated into CS programs? 12. What factors do you think hinder or limit the sustainability of CS investments? a. Why? Please explain. b. Do you think these factors are usually considered during USAID project/activity design? Why or why not? Lessons learned and other questions 13. In hindsight, if you could redesign the USAID CS activities you’ve worked on with the benefit of your experience, what would you do differently? a. How can these insights help inform the design of new CS activities? b. What would be your main recommendations for new CS programs? 12. What do you see as the role of the private sector in CS programming? 13. Is there anything else you would like to add that we haven’t covered? USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |73 ANNEX 8: ADDITIONAL SURVEY RESULTS Most survey questions allowed for respondent to select multiple answers, so percentages do not always add up to 100 percent. The online survey had a 63% response rate with 50 out of 80 people responding. Of those 78% (38/50) were USAID staff and were from 22% other organizations (e.g. NGO, research, org, etc.). Of the USAID respondents about 61% (23/38) were agreement or contracting officer’s representatives of climate services activities, 37% (14/38) were climate integration leads (CIL), 58% (22/38) were technical staff, 5% (2/38) were leadership, 3% (1/38) were in the program office. Majority of USAID respondents focused on climate work in Africa (37%, 14/38), followed by Latin America and the Caribbean (32%, 12/38) and Asia (21%, 8/38). Only 3% of the respondents focus on the Middle East. In addition, the majority of the USAID respondent’s generally focus on the following sectors: Natural resources management (77%), climate adaption (79%), climate services (49%), agriculture (49%), Water Management (44%), Disaster risk reduction (41%), forestry (36%), WASH (33%), fisheries (21%), and health (15%). Of the USAID (38) respondents 69% worked directly on USAID projects and 23% had experience with climate services related programming. Forty-seven respondents answer questions about climate services activities they had worked with. Of these activities, 85% worked in agriculture, 85% in climate adaption, 77% worked in climate services, 53% in national resources, 40% water management, and 40% disaster risk reduction. FIGURE A: WHICH SECTORS DOES/DID THE ACTIVITY WORK IN? (N=47) The main purposes of the 47 activities was to (83%) translate information into usable formats to improve key stakeholders utilization and (77%) to improve the quality of weather and climate services data. 17% 21% 23% 26% 40% 40% 53% 77% 85% 85% 0% 20% 40% 60% 80% 100% Other Forestry Fisheries Water, Sanitation and Hygiene (WASH) Water management Disaster risk reduction Natural resource management Climate services Agriculture Climate adaptation USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |74 FIGURE B: WHAT IS/WAS THE CLIMATE SERVICES ACTIVITY’S MAIN PURPOSE? (N=47) Most activities were implemented in West Africa (45%) and East Africa (32%), with 15% being implemented in the Caribbean and Southeast Asia. The most commonly addressed risks were extreme weather (91%) (related events such as floods, droughts or cyclones), and unreliable rainy seasons (83%), and increased temperatures (64%). Few activities addressed extreme heat (13%). FIGURE C: WHAT ARE/WERE THE MAIN CLIMATE RISKS THAT YOU SAW BEING ADDRESSED ACROSS THE ACTIVITY’S REGION(S)? (N=47) The most common weather or climate information that the activities worked to collect, analyze or disseminate was seasonal forecasts (77%), short-term weather forecasts (66%), historical climate trends analysis (66%), climate projections (51%). Only 2% did not work with weather or climate data. 17% 40% 40% 62% 64% 77% 83% 0% 20% 40% 60% 80% 100% Other To make a case for future climate services investments To guide further climate services research To co-produce weather and climate information with key stakeholders To inform field-level investments To improve the quality of weather and climate services data To translate information into usable formats to improve key stakeholder utilization 19% 13% 64% 83% 91% 0% 20% 40% 60% 80% 100% Other Extreme heat Increased temperatures Unreliable rainy seasons Extreme weather - related events such as floods, droughts, or cyclones USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |75 FIGURE D: WHAT KIND OF WEATHER OR CLIMATE INFORMATION DID THE ACTIVITY WORK TO COLLECT, ANALYZE, OR DISSEMINATE? (N=47) The main targeted stakeholders who were engaged with the activity were meteorological services providers (85%), farmers (79%), governments (e.g. ministries) (77%), local or community governments (72%), research institutions or universities (64%), civil society organizations (57%), extension services (53%), media organizations (11%). 89% of the activities distributed or shared weather or climate data with stakeholders. Of these the majority distributed or shared weather or climates data with stakeholders through presentations/discussion at meetings or events (71%), online (e.ge. websites, open, dashboard, social media, datasets) (67%), text/SMS messages (60%), TV or radio (52%), bulletins or newsletters (50%), email (33%), Reports or journal articles (19%) and no mail and 26% other. FIGURE E: HOW DID THE ACTIVITY DISTRIBUTE OR SHARE WEATHER OR CLIMATE DATA WITH STAKEHOLDERS? (N=42) To track how stakeholders were using climate services information the majority of these activities used workshops or meetings to get feedback (74%) or in-person surveys or interviews (63%). About only 17% used polls or surveys through phone (e.g. WhatsApp, SMS) and 15% used online surveys. Four percent of 11% 2% 51% 66% 66% 77% 0% 20% 40% 60% 80% 100% Other The activity did not work with weather or climate data Climate projections Short term weather forecasts Historical climate trends analyses Seasonal forecasts 26% 0% 19% 33% 50% 52% 60% 67% 71% 0% 20% 40% 60% 80% 100% Other Mail Reports or journal articles Email Bulletins/newsletters Television or radio Text/SMS messages Online (e.g., website, open dashboard, social media, datasets) Presentations/discussions at meetings or events USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |76 the activities did not use any methods to track and 17% used other methods including, evaluations and field visits. FIGURE F: DID THE ACTIVITY USE ANY OF THE FOLLOWING METHODS TO TRACK HOW STAKEHOLDERS WERE USING THE CLIMATE SERVICES INFORMATION THEY WERE PROVIDED? (N=46) The main reported challenge or barrier these activities faced during implementation was the varying capacity among personnel in different organizations (e.g. government, private, sector, research institutions) (73%), while 47% reported limited resources for users to implement change based on climate information. While 44% reported a disconnect between researchers and decision-makers (e.g. lack of partnership and collaboration between key stakeholders or poor communications) and 44% reported limited dissemination and overall reach of climate services information. FIGURE G: DID THE ACTIVITY FACE ANY OF THE FOLLOWING CHALLENGES OR BARRIERS DURING IMPLEMENTATION? (N=45) 17% 4% 15% 17% 63% 74% 0% 20% 40% 60% 80% 100% Other The activity used none of these methods Online surveys Polls or surveys through phone (e.g., WhatsApp, SMS) In-person surveys or interviews Feedback from workshops or meetings 16% 7% 20% 22% 27% 40% 44% 44% 47% 73% 0% 20% 40% 60% 80% 100% Other The activity did not face any of these challenges or barriers Stakeholders did not use the same data management software and tools Timing issues (e.g., data are collected at irregular intervals or not concurrently with other relevant data or user needs) End users did not have confidence in the information. Limited engagement of women accessing climate services Disconnect between researchers and decision makers (e.g., lack of partnership and collaboration between key… Limited dissemination and overall reach of climate services information. Limited resources for users to implement change based on climate information (e.g., farmers do not have knowledge… Varying capacity among personnel in different organizations (e.g., government, private sector, research institutions) USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |77 The reported main climate risks that are currently being addressed in the different regions was extreme weather-related events such as floods, droughts or cyclones (92%), about reported 77% unreliable rains, 59% reported altered or unpredictable seasons, 36% reported rising temperatures, and only 15% reported extreme heat. FIGURE H: BASED ON YOUR EXPERIENCE, WHAT ARE THE MAIN CLIMATE RISKS THAT ARE CURRENTLY BEING ADDRESSED ACROSS YOUR REGION(S) OF FOCUS? (N=39) Looking at the climate service value chain, Data, information, communication and decisions, 80% of respondents reported communication (translating information into key impacts) as the area of greatest need. About 76% reported decisions (understandings the set of choices that can be made based on the information), 57% reported information (data put into context), and 45% reported data. FIGURE I: BASED ON THE CLIMATE SERVICES VALUE CHAIN PROVIDED BELOW, WHERE DO YOU THINK IS THE GREATEST NEED FOR CLIMATE SERVICES INVESTMENT IN YOUR REGION? (N=49) The reported main barriers that limits the use of climate and/or climate risk data were limited technical expertise in using climate or climate risk data (73%) and lack of awareness of available climate or climate 10% 15% 36% 59% 77% 92% 0% 20% 40% 60% 80% 100% Other Extreme heat Rising temperatures Altered or unpredictable seasons Unreliable rains Extreme weather-related events such as floods, droughts, or cyclones 10% 45% 57% 76% 80% 0% 20% 40% 60% 80% 100% Other Data (observations, models) Information (data put into context) Decisions (understanding the set of choices that can be made based on the information Communication (translating information into key impacts) USAID. GOV EVALUATION OF CLIMATE SERVICES INVESTMENTS SINCE 2012 |78 risk data (69%). In addition, 55% reported limited availability of relevant data and 49% reported access issues with obtaining climate or climate risk data (e.g. hard to download or cost). FIGURE J: WHAT DO YOU THINK ARE THE MAIN BARRIERS THAT LIMIT THE USE OF CLIMATE AND/OR CLIMATE RISK DATA IN YOUR REGION? (N=49) 20% 2% 39% 49% 55% 69% 73% 0% 20% 40% 60% 80% 100% Other I have not experienced any barriers to my climate or climate risk data use Access issues due to information being in a different language Access issues with obtaining climate or climate risk data (e.g., hard to download, cost) Limited availability of relevant data Lack of awareness of available climate or climate risk data Limited technical expertise in using climate or climate risk data U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20004