December 22, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Development and Training Services and Management Systems International for the E3 Analytics and Evaluation Project.   EVALUATION Performance Evaluation: Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security Project FINAL EVALUATION REPORT PERFORMANCE EVALUATION: IMPROVING THE CLIMATE RESILIENCY OF KAZAKHSTAN WHEAT AND CENTRAL ASIAN FOOD SECURITY PROJECT DECEMBER 22, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Development and Training Services and Management Systems International for the E3 Analytics and Evaluation Project COVER PHOTO Caption: Demonstration plots of various kinds of wheat in Kostanay. Credit: Lyubov Palyvoda, dTS FINAL EVALUATION REPORT PERFORMANCE EVALUATION: IMPROVING THE CLIMATE RESILIENCY OF KAZAKHSTAN WHEAT AND CENTRAL ASIAN FOOD SECURITY PROJECT Management Systems International Corporate Offices 200 12th Street, South Arlington, VA 22202 USA Tel: + 1 703 979 7100 Contracted under AID-OAA-M-13-00017 E3 Analytics and Evaluation Project Prepared by: Lyubov Palyvoda (dTS) Svetlana Negroustoueva (dTS) Gregory Gust (dTS) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Final Evaluation Report – Performance Evaluation: CRW Project i Contents Acronyms .................................................................................................................................................. iv Executive Summary ................................................................................................................................. v Evaluation Purpose and Scope ............................................................................................................................................... v CRW Project Background and Rationale ........................................................................................................................... v Evaluation Methodology .......................................................................................................................................................... vi Evaluation Findings and Conclusions ................................................................................................................................... vi Recommendations ...................................................................................................................................................................... x Recommendations to Improve the Performance of the CRW Project .................... x Recommendations for the Design of a Subsequent Phase of the Project or another Similar Project ................................................................................................................................................................................... xi Recommendations for the Government of Kazakhstan ............................................................................................... xi Introduction ............................................................................................................................................... 1 Evaluation Purpose and Scope ............................................................................................................................................... 1 Evaluation Questions ................................................................................................................................................................. 1 CRW Project Background and Rationale ........................................................................................................................... 2 Country Context ........................................................................................................... 2 Gender Issues in Agriculture in Kazakhstan ...................................................................... 2 CRW Project Summary ........................................................................................................... 3 CRW Project Development Hypothesis ............................................................................................................................ 4 Evaluation Methodology ......................................................................................................................... 5 Evaluation Design ....................................................................................................................................................................... 5 Sustainability Analysis ........................................................................................................... 5 Data Collection and Analysis Methods ............................................................................................................................... 6 Evaluation Sampling and Site Selection Approach ........................................................................................................... 8 Evaluation Team .......................................................................................................................................................................... 9 Study Limitations ........................................................................................................................................................................ 9 Evaluation Scope ........................................................................................................... 9 Terminology ........................................................................................................... 9 Respondent Fatigue ........................................................................................................ 10 Selection Bias ........................................................................................................ 10 Limited Ability to Contact CRW Project Beneficiaries .............................................. 10 Non-KHM Weather and Climate Information as a Confounding Factor.............. 10 Non-Experimental Evaluation Design ................................................................................ 10 Findings and Conclusions ...................................................................................................................... 11 Evaluation Question 1: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg to collect and analyze meteorological data, and make weather and climate predictions? ................................................................................................................... 11 Kazhydromet (KHM) ........................................................................................................ 12 National Space Research Institute (NSRI) ....................................................................... 13 Ministry of Agriculture (MinAg) .......................................................................................... 14 Final Evaluation Report – Performance Evaluation: CRW Project ii Evaluation Question 2: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather- and climate-related data to farmers and other key stakeholders, as well as to each other? ........................................................ 15 Weather and Climate Data Sources .................................................................................. 16 KHM ........................................................................................................ 18 MinAg ........................................................................................................ 18 NSRI ........................................................................................................ 19 CRW-Supported Dissemination Practices ...................................................................... 19 Evaluation Question 3: To the extent that there are improved practices in the collection, analysis and dissemination of weather and climate information, are these improvements likely to be sustained? .................................................................................................................................................... 23 Policy ........................................................................................................ 23 Participation and Ownership ................................................................................................ 24 Management and Organization ............................................................................................ 25 Training Needs ........................................................................................................ 26 Financial ........................................................................................................ 26 Technology ........................................................................................................ 27 Evaluation Question 4: To what extent are other institutional stakeholders using Kazhydromet weather and climate information? .............................................................................................................................. 27 Evaluation Question 5: To what extent are farmers basing their decisions on Kazhydromet weather and climate information? Why or why not? .......................................................................................... 30 Agro-Producer Perceptions of KHM Forecasts ............................................................ 31 KHM Short-Term Weather Forecasts .............................................................................. 33 KHM Medium and Long-term Climate Forecasts ......................................................... 34 Training/Training of Trainers (TOT) ................................................................................. 39 Publications/Videos ........................................................................................................ 41 Demonstration Plots ........................................................................................................ 42 Evaluation Question 7: Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries? ............................................... 48 Challenge 1: Accessibility of High-Quality and Reliable Weather and Climate Forecasts ..................................................................................................... 48 Challenge 2: Limited Understanding of the Impact of Climate Change on Agricultural Practices .............................................................................................. 50 Challenge 3: Lack of Knowledge about Advanced Farming Practices among Farmers and Agro-Producers ................................................................ 51 Challenge 4: Old and Outdated Farmer Equipment ..................................................... 51 Challenge 5: Access to High-Quality and Appropriate Seeds ................................... 52 Challenge 6: Low Wheat Storage Capacity ..................................................................... 53 Challenge 7: Decreased Prioritization of Wheat Production by MinAg ................ 53 Findings Related to Gender ................................................................................................................................................. 56 Recommendations ................................................................................................................................. 59 Recommendations to Improve the Performance of the CRW Project ................................................................ 59 Recommendations for the Design of a Subsequent Phase of the Project or another Similar Project ................................................................................................................................................................................. 60 Recommendations for the Government of Kazakhstan ............................................................................................. 61 Annex A: Evaluation Statement of Work ........................................................................................ 63 Final Evaluation Report – Performance Evaluation: CRW Project iii Annex B: Overview of Gender Considerations for the Kazakhstan Agriculture Sector 73 Annex C: Data Collection Instruments ............................................................................................. 75 Key Informant Interview Guide: Institutional Respondents ...................................................................................... 75 Individual Interview Guide: Farmer Beneficiaries ......................................................................................................... 81 Focus Group Discussion Guide: Farmers ....................................................................................................................... 89 Telephone Survey Questionnaire: Farmer Beneficiaries ............................................................................................ 94 Annex D: List of CRW Project Publications .................................................................................. 106 Annex E: List of Documents Reviewed for the Evaluation ......................................................... 109 Annex F: Schedule of Interviews Conducted by Evaluation team ............................................ 111 Final Evaluation Report – Performance Evaluation: CRW Project iv ACRONYMS ADB Asian Development Bank ADS Automated Directives System CCRD Climate Change Resilient Development project CEDAW Convention on the Elimination of all forms of Discrimination against Women CPT Climate Predictability Tool CRW Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security dTS Development and Training Services E3 USAID Bureau for Economic Growth, Education and Environment EQ Evaluation Question FGDs Focus Group Discussions GAO Government Accountability Office GCC Global Climate Change Office, USAID/E3 GDP Gross Domestic Product GEF Global Environment Fund GOK Government of Kazakhstan IP Implementing Partner IRB Institutional Review Board IRI International Research Institute for Climate and Society KAI KazAgroInnovation KHM KazHydroMet KIIs Key Informant Interviews MinAg Ministry of Agriculture MSI Management Systems International NDVI Normalized Difference Vegetative Index NSRI National Space Research Institute PMP Performance Monitoring Plan SOW Statement of Work SPI Standardized Precipitation Index SSMI Special Sensor Microwave Imager TPM Team Planning Meeting UNDP United Nations Development Program UNIFEM UN Women USAID United States Agency for International Development VCI Vegetative Condition Index WMO World Meteorological Organization WB World Bank Final Evaluation Report – Performance Evaluation: CRW Project v EXECUTIVE SUMMARY Evaluation Purpose and Scope The United States Agency for International Development, through its Office of Global Climate Change in the Bureau for Economic Growth, Education, and Environment (USAID/E3/GCC), requested that the E3 Analytics and Evaluation Project1 design and implement a performance evaluation of the Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security (CRW) project that is being delivered through the USAID/Central Asia Regional mission, based in Kazakhstan. The purpose of this evaluation was to assess the effectiveness of specific CRW interventions related to:  The collection, analysis, dissemination, and use of weather and climate information, including forecasting; and  The mainstreaming of adaptation techniques by Kazakhstan farmers. The evaluation also assessed whether improved practices for weather and climate analysis and prediction are likely to be sustained, and the relevance of the suite of activities delivered by the CRW project to address the challenges of climate change to the wheat sector in Kazakhstan. Evaluation Questions The evaluation questions presented below are identical to those found in USAID’s Statement of Work for the evaluation, which is attached as Annex A. 1. To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for collecting and analyzing agrometeorological data, and making seasonal and climate predictions? 2. To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather- and climate-related data to farmers and other key stakeholders, as well as to each other? 3. To the extent that there are improved practices in the collection, analysis and dissemination of agrometeorological and climate information, are these improvements likely to be sustained? 4. To what extent are other institutional stakeholders using Kazhydromet weather and climate information? 5. To what extent are farmers basing their decisions on Kazhydromet weather and climate information? Why or why not? 6. To what extent have farmers adopted climate change adaptation techniques promoted by the demonstration plots and through CRW-supported education initiatives (such as publications and circulars)? Why/Why not? If adopted, from which initiative? 7. Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries? CRW Project Background and Rationale Kazakhstan is a major wheat producer and the largest grain producer under predominantly rain-fed conditions in the world. The main wheat production comes from three regions: Akmola, Kostanay, and North Kazakhstan. 1 The E3 Analytics and Evaluation Project consists of a lead implementer, Management Systems International (MSI), and partners Development and Training Services (dTS) and NORC at the University of Chicago. Final Evaluation Report – Performance Evaluation: CRW Project vi Climate change is one of the major challenges faced by Kazakh farmers. Studies have demonstrated that the air temperature in Kazakhstan is increasing, and have linked this increase to the desertification of large areas and to increased climate variability marked by an increasing number of droughts coming in closer succession. These more frequent and intermittent droughts can cause harvests to vary dramatically from one year to the next, depending on when the soil moisture deficits occur relative to critical stages in plant growth and seed production. The CRW project was launched to address the vulnerability of the Kazakhstan wheat sector to climate change. This $2.2 million initiative is intended to “catalyze the process of adaptation in Kazakhstan’s wheat sector, while also opening a regional dialogue around the challenges of climate change to Central Asian food security.”2 The project is implemented by the United Nations Development Programme (UNDP) in Kazakhstan, with significant support and technical assistance provided by USAID’s Climate Change Resilient Development (CCRD) project. The CRW project is being carried out in two phases, each funded for $1.1 million, with the first phase implemented from September 2012 to September 2014 and the second phase running from October 2014 to September 2016. Through the CCRD project, USAID provided an additional $450,000 for technical assistance interventions. There are three components to the CRW project:3 1. Improved Information for Climate-Resilient Wheat Production in Kazakhstan 2. Mainstreaming Climate Resilience into Wheat Production in Kazakhstan 3. Regional Dialogue on Wheat, Climate Change, and Regional Food Security Evaluation Methodology This performance evaluation was designed to examine the effectiveness of delivering certain components of the CRW project. It did not rely on counterfactual groups to assess the extent to which results can be attributed to the project. Instead, the evaluation incorporated mixed-methods approaches to obtain data of behavior change on the part of institutional stakeholders and farmer-beneficiaries, and information on the perceptions of those stakeholders concerning the project’s effectiveness, utility, and the sustainability of results. To address USAID’s evaluation questions, the evaluation team undertook secondary analysis of project documents and conducted semi-structured interviews and focus group discussions with CRW project stakeholders including UNDP and other project implementing partners, USAID, Kazakh government officials, and project beneficiaries. A telephone survey was also conducted of farmers who attended CRW-related training activities. Evaluation Findings and Conclusions The evaluation conclusions are presented below by evaluation question. The findings that support these conclusions are presented in the main body of the report for each evaluation question. Evaluation Question 1: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for collecting and analyzing agrometeorological data, and making seasonal and climate predictions? 2 This project goal is stated in the CRW Final Project Report (UNDP 2015), although the evaluation team identified several iterations of the project goal in the CRW documentation. For instance, the 2013 UNDP Logical Framework identified the principal objective of CRW as “To increase understanding of climate variability to regional food security by developing resilience of Kazakhstan's wheat sector.” 3 CRW Project Document, September 20, 2012, page 5 Final Evaluation Report – Performance Evaluation: CRW Project vii  Conclusion 1.1: While the CRW project was instrumental in bringing key stakeholders together to examine new practices and discuss new policies which address their shared concerns, there remains little cooperation and collaboration between agencies.  Conclusion 1.2: New tools and techniques, especially the introduction of new imagery (SSMI), new indices (SPI, NDVI), and new database management (IRI Data Library), have improved the production of drought, crop, and climate assessments by both Kazhydromet (KHM) and the National Space Research Institute (NSRI).  Conclusion 1.3: The KHM climate forecasting process remains hampered by an inadequate surface reporting network and large amounts of un-digitized historical station data.  Conclusion 1.4: While CRW project support has led to improvements in the production of drought, crop, and climate assessments, KHM-produced seasonal climate forecasts have seen only marginal gains, and those are largely due to improvements in data handling via the International Research Institute for Climate and Society Data Library and Digital Map Library. Techniques to improve the accuracy of forecasting have not been fully adopted. Evaluation Question 2: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather- and climate-related data to farmers and other key stakeholders, as well as to each other?  Conclusion 2.1: The fee-based structure of most agrometeorological information produced by KHM, NSRI, and the Ministry of Agriculture (MinAg) significantly impedes the dissemination of this information both to farmers and among the agencies.  Conclusion 2.2: Although the CRW project has provided support to improve the structure and dissemination of the MinAg monthly agricultural bulletins, it does not appear that farmers actually receive these bulletins.  Conclusion 2.3: CRW project support to the development of a geoportal for the expanded dissemination of weather and climate data has not been effective to date, as development has been delayed and there remains no agreement on what information will be contained on the geoportal, how it will be made accessible, and to whom.  Conclusion 2.4: The presence of many unsolved issues makes the possibility of creating and launching the geoportal in the upcoming year unclear. Evaluation Question 3: To the extent that there are improved practices in the collection, analysis and dissemination of agrometeorological and climate information, are these improvements likely to be sustained?  Conclusion 3.1: CRW project support was consistent with the Government of Kazakhstan’s (GOK) climate change policies and objectives and beneficiary institutional policies.  Conclusion 3.2: The introduction of new agrometeorological practices responded to institutional needs and garnered institutional support.  Conclusion 3.3: The CRW project objective was aimed at improving agrometeorological practices without consideration of their sustainability within partnering institutions. Final Evaluation Report – Performance Evaluation: CRW Project viii  Conclusion 3.4: Most institutional stakeholders agreed that the training was appropriate and responsive to institutional needs, though the evaluation found that both baseline and ongoing needs were mainly assessed through informal means.  Conclusion 3.5: GOK institutions have committed to meeting the minimal recurring costs associated with the new meteorological practices.  Conclusion 3.6: The technology and software provided by the CRW project and CCRD are appropriate, and ongoing maintenance will be minimal and within the capacity of the relevant institutions.  Conclusion 3.7: Improved practices in the collection, analysis, and dissemination of weather and climate information are likely to be sustained. Evaluation Question 4: To what extent are other institutional stakeholders using Kazhydromet weather and climate information?  Conclusion 4.1: KHM is acknowledged as the official source for weather and climate information, and observational data, analyses, and short-term forecasts are used by other institutional actors.  Conclusion 4.2: There is little apparent systematic sharing or use of KHM medium- and long-term climate forecasts by other Kazakh institutional actors. Evaluation Question 5: To what extent are farmers basing their decisions on Kazhydromet weather and climate information? Why or why not?  Conclusion 5.1: Farmers trust and rely upon KHM short-term forecasts in agricultural decision￾making.  Conclusion 5.2: Farmers believe that KHM medium- and long-term forecasts are not sufficiently specific as to location to be useful in agricultural decision-making.  Conclusion 5.3: While medium and large farms can afford to pay for KHM medium- and long-term forecasts, smaller farmers view the cost as prohibitive.  Conclusion 5.4: Farmers do not generally view the KHM medium- and long-term forecasts to be reliable.  Conclusion 5.5: Farmers do not generally rely on KHM medium- and long-term forecasts.  Conclusion 5.6: Although farmers believe that overall KHM forecasting has improved over the last two years, the evidence suggests that farmers perceive that short-term forecasts have improved and that medium- and long-term forecasts largely have not. Evaluation Question 6: To what extent have farmers adopted climate change adaptation techniques promoted by the demonstration plots and through CRW-supported education initiatives (such as publications and circulars)? Why/Why not? If adopted, from which initiative?  Conclusion 6.1: Exposure by farmers to CRW-sponsored trainings was consistent with CRW objectives and targets, although small in number relative to the overall population. Final Evaluation Report – Performance Evaluation: CRW Project ix  Conclusion 6.2: The CRW-sponsored training module was effective at communicating the risks posed by climate change and encouraging the adoption of adaptation techniques for a significant minority of the farmers exposed to the trainings.  Conclusion 6.3: Few farmers are aware of CRW-sponsored publications and media.  Conclusion 6.4: Among farmers who viewed the CRW-sponsored demonstration plots, there is evidence that a significant proportion at least partially adopted an adaptation technique showcased.  Conclusion 6.5: As farmers prefer to adopt technologies where there is successful evidence of adoption on neighboring farms, the impact of the CRW project will depend on the extent to which this demonstration effect occurs and likely can only be determined ex-post.  Conclusion 6.6: There are many reasons that farmers choose not to adopt showcased adaptation practices, including their (for some) high cost, concerns about whether the practices are applicable to the soil and climate conditions on their farms, and a lack of expertise.  Conclusion 6.7: Farmers of medium and large farms are more likely to undertake adaptive techniques connected to crop diversification than adopt techniques connected with no or minimum tillage, use of fertilizers, water accumulation, and wheat variety.  Conclusion 6.8: The partnership with Agro-centers allowed the CRW project to gain access to farmers and leverage the expertise and resources of the Agro-centers, but the infrequency of CRW￾supported field days resulted in limited exposure of farmers who could be monitored by the project. Evaluation Question 7: Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries?  Conclusion 7.1: The CRW project’s focus on improving the accuracy and availability of medium- and long-term climate forecasts addressed a key challenge for wheat production, although the project focused insufficiently on issues of access to and cost of data, which are faced by small farmers in particular.  Conclusion 7.2: Aside from access to and quality of the climate and weather information, the most significant challenges to wheat production identified by stakeholders are outdated farm equipment, a lack of access to high-quality seeds, and inadequate storage for farm yields – none of which were comprehensively addressed by the CRW project, largely due to the limited project scope.  Conclusion 7.3: At the time of this evaluation, the decision to focus on improved climate resiliency of wheat production did not fully align with the GOK focus to improve the production of other grains and livestock. However, the CRW project facilitated strengthening GOK’s attention to the issues of food security and adaptation in the context of climate change in the region. Conclusion Related to Gender  Conclusion G1: There was no focus in CRW project design, implementation, or monitoring on an inclusive project model or gender-differentiated approaches to project delivery.  Conclusion G2: Capacity-building assistance benefited fewer women than men, as women are underrepresented among farmers and agro-producers. Final Evaluation Report – Performance Evaluation: CRW Project x  Conclusion G3: The CRW project lacked specific activities to encourage equal access for female agro-producers and female farm owners, although due diligence was taken to collect and report sex￾disaggregated data, which illuminated gender gaps.  Conclusion G4: Although the CRW project did not systematically address gender considerations, the evaluation did not reveal that women or vulnerable populations were actively excluded from project design and implementation. Recommendations Recommendations to Improve the Performance of the CRW Project Recommendation 1: The CRW project should support efforts to bring into KHM existing surface agrometeorological data and information from agricultural research center station observations and NSRI satellite-derived analyses. To achieve the objectives of CRW, such data should be freely shared and the assessments should be collaborative. Recommendation 2: The CRW project should continue working with KHM and MinAg on prioritizing increasing the usability of information produced by MinAg-supported climate stations. This would require training, site review, the installation of telemetry, the expansion of analysis processes, and administrative oversight. Recommendation 3: The CRW project should support the continued digitization of a large archive of historical observational data (over the past few decades) that currently exists only in paper form. Recommendation 4: The CRW project should take efforts to improve the effectiveness and efficiency of the free information-sharing mechanisms including the MinAg bulletins and the geoportal. As the Internet is the primary mechanism through which farmers access meteorological data, the development of the geoportal and the provision of additional information through the KHM and MinAg websites (e.g., the bulletins currently provided freely at akimat meetings) could be immediate objectives. Recommendation 5: The CRW project should consider conducting a cost-benefit analysis of the required potential support necessary for MinAg to take the lead in distributing climate information. Recommendation 6: Given the close collaboration between the CRW project and MinAg, the project should work with MinAg to identify and resolve bottlenecks in the distribution flow of bulletins to farmers. Recommendation 7: The CRW project should work with key stakeholders and build on the success of professional networking established to create an action plan and timeline for the development of the geoportal. Recommendation 8: The CRW project should extend the use of the Training of Trainers model to include actual farmers and agro-producers with a good success rate and reputation in the community, to increase the likelihood of adoption and sustainability, and to increase experience and knowledge sharing among farmers. Recommendation 9: Due to the high utilization of the Internet by farmers to learn about new agricultural techniques, the CRW project should work with MinAg and KazAgroInnovation to provide additional information about climate-resilient adaptation techniques on the appropriate websites. Recommendation 10: The CRW project should improve its monitoring of project outputs and outcomes, including specifically assessments of knowledge gained as a result of project training and Final Evaluation Report – Performance Evaluation: CRW Project xi innovative approaches such as outcome mapping, to capture the outcomes of farmers participating in project-sponsored field days. The CRW project should also take steps to monitor visits to the pilot demonstration plots other than during field days. Recommendation 11: The CRW project should consider a strategy for the sustainability of project activities and outcomes following the conclusion of the project. This strategy should at a minimum involving planning for the sustainability (if sought) of CRW-sponsored demonstration plots. Recommendation 12: The CRW project should use sex-disaggregated monitoring data, the results of the farmer survey, information from gender assessments, and consultations with local gender experts with sectoral expertise in agriculture to understand and address relevant gender gaps and barriers as feasible within the project scope. Recommendations for the Design of a Subsequent Phase of the Project or another Similar Project Recommendation 13: USAID should consider supporting additional surface meteorological stations and a specific meteorological office capacity-building project across the Central Asia region. Recommendation 14: Design a capacity building project, specifically geared towards a particular agricultural sector (like wheat) or array of sectors in Kazakhstan, considering the following aspects: (1) hydro-agro-met observations, which should be made to a World Meteorological Organization standard for accuracy; (2) data analyses or assessments, which should have certain criteria for site representativeness and spacing as well as data frequency; and (3) forecast processes, the potential accuracy of which will be greatly influenced by the extreme continentality of the Kazakhstan wheat￾growing areas. Recommendation 15: Diligence should be taken to consult available national normative guidance related to promoting women’s empowerment at the project design stage, especially when cooperating with the government and semi-government institutions as the key stakeholders. Recommendation 16: Openly recognize and commit adherence to USAID’s Gender Equality and Female Empowerment Policy and Automated Directives System 205. This should include: (1) stating in project design, planning, and monitoring documents that achieving gender equality and women’s empowerment is a project priority; and (2) undertaking a gender analysis to understand relevant gender gaps and identify the project’s beneficiary and stakeholder groups within which women should be represented as leaders and members. Recommendation 17: USAID should support policy advocacy intended to remove barriers (such as the current requirements for leasing farming land in Kazakhstan) that impede women’s participation in wheat farming in Kazakhstan. Recommendation 18: Promote the use of agro-meteorological and climate information communication methods that are likely to be effective in reaching women farmers and managers in the wheat farm sector. Recommendations for the Government of Kazakhstan Recommendation 19: GOK should consider introducing principles for long-term weather forecasting and agricultural crop productivity/harvest outlooks in strategic government documents. Final Evaluation Report – Performance Evaluation: CRW Project xii Recommendation 20: GOK, through consultation with relevant ministries, should develop an action plan, with assigned responsibilities, for the development of the geoportal. Recommendation 21: GOK should consider establishing an early warning system for drought to promote drought mitigation. Recommendation 22: Kazhydromet and the NSRI should consider developing mechanisms for the free exchange of data and collaboration on assessments that would include bringing together existing surface agrometeorological data and information from agricultural research center station observations and NSRI satellite-derived analyses. Recommendation 23: GOK should consider financing the construction of additional surface meteorological stations and engage in capacity building for the meteorological office. Recommendation 24: MinAg should consider how to improve the utility and dissemination of weather and climate data, including through improvements in weather and climate analysis and improved effectiveness and efficiency of free information-sharing mechanisms such as the MinAg website and bulletins. Recommendation 25: MinAg and KazAgroInnovation should consider expanding the Training of Trainers model to include additional farmers and agro-producers with a good success rate and reputation in the community. Recommendation 26: MinAg should consider collaboration with the National Commission on Women’s Issues and Family and Demographic Status to better understand and address relevant gender barriers faced by female agro-producers. Final Evaluation Report – Performance Evaluation: CRW Project 1 INTRODUCTION Evaluation Purpose and Scope The United States Agency for International Development, through its Office of Global Climate Change in the Bureau for Economic Growth, Education, and Environment (USAID/E3/GCC), requested that the E3 Analytics and Evaluation Project4 design and implement a performance evaluation of the Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security (CRW) project that is being delivered through the USAID/Central Asia Regional mission, based in Kazakhstan. The purpose of this evaluation was to assess the effectiveness of specific CRW interventions related to:  The collection, analysis, dissemination, and use of weather and climate information, including forecasting; and  The mainstreaming of adaptation techniques by Kazakhstan farmers. The evaluation also assessed whether improved practices for weather and climate analysis and prediction are likely to be sustained, and the relevance of the suite of activities delivered by the CRW project implementing partner (United Nationals Development Programme [UNDP] in Kazakhstan) to address the challenges of climate change to the wheat sector in Kazakhstan. The primary audiences for this evaluation are USAID/Kazakhstan, USAID/E3/GCC, the UNDP office in Kazakhstan, CRW project staff, and the Government of Kazakhstan (GOK). Secondary audiences for this evaluation include other USAID Missions and donors delivering projects that address the impacts of climate change on agriculture or improved climate data and information for use in the agriculture sector. The evaluation provides practical information from which to draw lessons and refine approaches for current and future projects. Specifically, this evaluation allows the primary audiences to:  Consider whether mid-course corrections or adaptations are required to increase the effectiveness and impact of the CRW project during its final year of implementation;  Inform future approaches that USAID may undertake to mitigate the effects of climate change and enhance food security in Kazakhstan and Central Asia; and  Inform the GOK strategy in the climate change adaptation and wheat sector, as well as streamline its approach in working with USAID and other development partners. Evaluation Questions The evaluation questions (EQs) were developed through a series of consultations between the E3 Analytics and Evaluation Project and USAID that culminated in the Statement of Work (SOW) for this evaluation (see Annex A). The EQs presented below are identical to those found in the SOW. 1. To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for collecting and analyzing agrometeorological data, and making seasonal and climate predictions? 2. To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather- and climate-related data to farmers and other key stakeholders, as well as to each other? 4 The E3 Analytics and Evaluation Project consists of a lead implementer, Management Systems International (MSI), and partners Development and Training Services (dTS) and NORC at the University of Chicago. Final Evaluation Report – Performance Evaluation: CRW Project 2 3. To the extent that there are improved practices in the collection, analysis and dissemination of agrometeorological and climate information, are these improvements likely to be sustained? 4. To what extent are other institutional stakeholders using Kazhydromet weather and climate information? 5. To what extent are farmers basing their decisions on Kazhydromet weather and climate information? Why or why not? 6. To what extent have farmers adopted climate change adaptation techniques promoted by the demonstration plots and through CRW-supported education initiatives (such as publications and circulars)? Why/Why not? If adopted, from which initiative? 7. Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries? CRW Project Background and Rationale Country Context Kazakhstan is one of the world’s largest wheat producers and the largest wheat exporter in Central Asia.5 Wheat accounts for 80 percent of total value of the country’s grain production, 25 percent of gross agricultural output, 65 percent of total cultivated area,6 and 2 percent of the total value of exports. The main wheat production comes from three regions: Akmola, Kostanay, and North Kazakhstan – where the total volume of production reached 11.9 million tons in 2013, or 81.3 percent of all wheat production in the country. The GOK owns all agricultural land and leases it to farmers under 49-year leases. Grain production is dominated by two categories of farms: peasant farms and large agricultural enterprises. Peasant farms are typically family farms, and 95 percent of them are smaller than 1,000 hectares. About 200,000 peasant farms produce grains and account for about 35 percent of the country’s output. There are also about 5,000 agricultural enterprises with an average size of 3,000 hectares each. These account for about 65 percent of Kazakhstan’s grain production.7 Grain yields on peasant farms are much lower than on agricultural enterprises because of outdated machinery and financial limitations to leasing or purchasing new equipment and purchasing the fertilizers needed for no or minimum tillage agriculture. Climate change is one of the major challenges faced by Kazakh farmers. Studies have demonstrated that the air temperature in Kazakhstan is increasing,8 and have linked this increase to the desertification of large areas and to increased climate variability marked by an increasing number of droughts coming in closer succession.9 These more frequent and intermittent droughts can cause harvests to vary dramatically from one year to the next, depending on when the soil moisture deficits occur relative to critical stages in plant growth and seed production. Gender Issues in Agriculture in Kazakhstan10 While there is no Ministry of Gender or similar body in Kazakhstan, gender issues are addressed at the highest level of the National Commission on Women’s Issues and Family and Demographic Status to the 5 In 2014, Kazakhstan was the world’s 11th largest producer and 7th biggest exporter of wheat. US Department of Agriculture, Grain: World Markets and Trade (December 2015), available at http://apps.fas.usda.gov/psdonline/circulars/grain.pdf. 6 World Bank, Agricultural Sector Risk Assessment in Kazakhstan, June 2015. 7 Islamic Development Bank, Enhancing Competitiveness and Diversification of the Kazakhstan Economy, December 2011. 8 While there is no single study demonstrating all impacts of climate change on wheat production in Kazakhstan, scientific reports register an increase of air temperature of 0.3 degrees Celsius every decade since the 1930s. 9 A Review of Drought Occurrence and Monitoring and Planning Activities in the Near East Region, 2008. FAO and National Drought Mitigation Center University of Nebraska-Lincoln, Nebraska, USA. 10 Additional information is provided in Annex B. Final Evaluation Report – Performance Evaluation: CRW Project 3 President, created in 1995. The commission’s most recent annual work plan, for 2015, has an extensive list of activities to be applied in GOK agencies to promote gender equality, without specific reference to particular agencies or sector-specific interventions,.11 By Article 101 of the Land Code, women and men have the same right to own and manage land. According to the official Convention on the Elimination of all forms of Discrimination against Women (CEDAW) report, women head 11 percent of all farms and agricultural processing businesses.12 However, the shadow report submitted to the CEDAW Committee in 2006 noted that overall, women continued to experience discrimination in regards to access to land (especially in rural areas), in part because in order to obtain land for farming, the applicant must be able to prove that she has an agricultural qualification and experience managing an agricultural business, which is unusual for Kazakh women.13 The official 2012 CEDAW report noted that women and men participate on an equal footing in the various government-run training and support programs for farmers. A 2010 Asian Development Bank Gender Assessment cited persistent gender gaps in key poverty reduction indicators. At the time, when new GOK programs for economic diversification and agricultural revival were being developed and delivered, the ADB assessment called for equitable access to the new resources being made available to women through public investments. CRW Project Summary The CRW project was launched to address the vulnerability of the Kazakhstan wheat sector to climate change. This $2.2 million initiative is intended to “catalyze the process of adaptation in Kazakhstan’s wheat sector, while also opening a regional dialogue around the challenges of climate change to Central Asian food security.”14 The CRW project is implemented by UNDP Kazakhstan, with significant support and technical assistance provided by USAID’s Climate Change Resilient Development (CCRD) project. The CRW project is being carried out in two phases, each funded for $1.1 million, with the first phase implemented from September 2012 to September 2014 and the second phase running from October 2014 to September 2016. Through the CCRD project, USAID provided an additional $450,000 for technical assistance interventions.15 There are three components to the CRW project:16 1. Improved Information for Climate-Resilient Wheat Production in Kazakhstan – To overcome deficiencies in climate information services, improve the understanding of climate change impacts in Kazakhstan’s wheat growing regions and develop a system to continuously deliver climate information to key stakeholders. Key project outputs include:17  Needs assessment and stakeholder consultations  Improving data collection and dissemination mechanisms 11 The document is available here. 12 http://genderindex.org/sites/default/files/datasheets/KZ.pdf 13 Idem. 14 This project goal is stated in the CRW Final Project Report (UNDP 2015), although the evaluation team identified several iterations of the project goal in the CRW documentation. For instance, the 2013 UNDP Logical Framework identified the principal objective of CRW as “To increase understanding of climate variability to regional food security by developing resilience of Kazakhstan's wheat sector.” 15 E-mail from Rebecca Nicodemus, USAID/E3/GCC, November 3, 2015 16 CRW Final Project Report (UNDP 2015), p. 4. 17 The CRW documentation refers to these as “activities”, but they are more appropriately conceived of as “outputs” in USAID LogFrame terminology. Final Evaluation Report – Performance Evaluation: CRW Project 4  Development of forecasting models  Improved data sharing and use 2. Mainstreaming Climate Resilience into Wheat Production in Kazakhstan – Bring together key stakeholders to identify viable climate adaptation practices and provide technical support to mainstream these practices into existing decision-making processes. Key project outputs include:  Mainstreaming wheat climate resilience into relevant climate change adaptation and agricultural strategies  Priority adaptation options demonstrated  Capacity development and awareness raising  Improving wheat production, storage and distribution 3. Regional Dialogue on Wheat, Climate Change, and Regional Food Security – Bring together Kazakhstan and Central Asian wheat importers to discuss and identify solutions to deal with fluctuations in wheat availability and price. Key project outputs include:  Gap analysis and assessment review  Awareness raising CRW Project Development Hypothesis In early 2013, the CRW project convened a series of stakeholder workshops in Astana and the three northern oblasts (regions) of Kostanay, Akmola, and Northern Kazakhstan to better understand the challenges facing the country’s wheat sector. Participants included national ministry officials (e.g., the Ministry of Agriculture [MinAg], Kazhydromet [KHM], National Space Research Institute [NSRI]), representatives from the Farmers’ Union, agricultural specialists, meteorologists, and farmers. Through these workshops, the representatives identified challenges facing the Kazakhstan wheat sector in responding to climate change and climate variability. Among the challenges highlighted by participants were:  The availability of high-quality seed varieties;  The lack of skilled farmers and agricultural specialists; and  The lack of reliable and precise climate forecasts and information. The CRW project, informed by the stakeholder consultations and with the support of CCRD, developed a suite of interventions designed to address the challenges posed to farmers in Kazakhstan resulting from climate change and climate variability. Embedded in the CRW project Performance Monitoring Plan (PMP)18 and LogFrame is a hierarchy of intended results and outputs that USAID and UNDP hypothesized would flow from implementing project activities in Kazakhstan and in the region. The logic can be restated into the hypotheses below, and a graphic depiction of the theory of change is included in the evaluation SOW in Annex A.  If access to Kazakhstan climate data for extension workers and farmers to schedule planting and harvesting is improved and climate-resilient wheat practices are adopted, the resilience of Kazakh wheat production is improved.  If national policies that hinder regional food security are removed, and there is an increase in the availability of new drought-resistant wheat varieties across Central Asia, food security in Central Asia is improved. 18 “Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security” PMP, UNDP. Submitted to USAID in October 2013. Final Evaluation Report – Performance Evaluation: CRW Project 5  If there is an improved resilience of Kazakh wheat production coupled with a regional dialogue and efforts to increase the availability of new drought-resistant wheat varieties across Central Asia, food security in Central Asia is improved. CRW project documentation also indicates that key learning objectives would include the recommendations and implications that are captured and disseminated regionally in Central Asia and nationally in Kazakhstan among key partners and the general public. This issue is particularly important for improving regional food security. EVALUATION METHODOLOGY Evaluation Design This section describes the overall research design and approaches employed under this evaluation to answer the EQs, which are clustered here according to the data sources that were relied upon.  EQs 1 through 4 examine changed behaviors and practices within Kazakh institutions associated with the technical assistance and capacity building provided through the CRW project. The primary data sources used to answer these questions were technical specialists affiliated with the CRW project and institutional stakeholders from KHM, NSRI and MinAg.  EQs 5 and 6 relate to changed behaviors and practices on the part of farmer-beneficiaries. The primary data sources to answer these questions were technical specialists affiliated with the three agricultural extension centers (“Agro-centers”) in the three northern oblasts of Kazakhstan as well as farmer-beneficiaries.  EQ 7 relates to the relevance of the CRW project interventions. Data from both institutional stakeholders and farmer-beneficiaries were collected to answer this question. The evaluation methodology, including data sources and data collection and analysis methods, is described in the following sub-section. This performance evaluation was designed to examine the effectiveness of delivering certain components of the CRW project but did not rely on counterfactual groups to assess the extent to which results could be attributed to the project. Instead, the evaluation incorporated a mixed-methods approach for obtaining data on behavior change on the part of institutional stakeholders and farmer-beneficiaries, and information on the perceptions of those stakeholders concerning the project’s effectiveness, utility, and the sustainability of results. Sustainability Analysis To address EQ 3 and assess the likelihood of sustainability, the evaluation team considered factors and conditions commonly associated with the sustainability of international development interventions as identified in USAID’s Automated Directives System (ADS) 201.3, academic and practitioner literature,19 and other donor-adopted frameworks. 19 Von Wieren-Lehr S (2001) Sustainability in agriculture: an evaluation of principal goal oriented concepts to close the gap between theory and practice. Agric Ecosyst Envir 84:115-129; Hayati et al. (2010) Measuring Agricultural Sustainability. Sustainable Agriculture Reviews 5; Johnson, K., Hays, C., Hayden, C., & Daley, C. (2004). Building capacity and sustainability prevention innovations: A sustainability planning model. Evaluation and Program Planning, 27, 135-149; Goldsmith, Arthur (2015) Defining Sustainability in International Development (available at: http://pdf.usaid.gov/pdf_docs/PA00KD6H.pdf). Final Evaluation Report – Performance Evaluation: CRW Project 6 Based on a review of this literature, the evaluation team identified six domains (i.e., relevant factors and conditions) likely to influence the continued use of agrometeorological practices by GOK institutions.20 Table 1 outlines the domains and provides a description of the key research questions and methods associated with the assessment of each domain. TABLE 1: ANALYTICAL FRAMEWORK FOR ASSESSING SUSTAINABILITY Sustainability Domains Key Research Questions Policy  Does the policy environment support the continued use of the agrometeorological practices?  Was the CRW project support to KHM and NSRI consistent with relevant sector policies? Participation & Ownership  Have key local stakeholders actively participated in the design of the capacity building interventions?  Have key local stakeholders been clearly supportive of the adoption of different practices?  Do the new agrometeorological practices meet a clearly expressed need on the part of key stakeholders? Of ultimate beneficiaries? Management & Organization  Did the CRW project include sustainability of agrometeorological practices as a project objective?  Has the CRW project assessed the capacity of the relevant institutions to sustain the practices?  Has a sustainability monitoring framework been proposed in the CRW design or implemented as part of the project? Training Needs  Have the needs for ongoing training been assessed and provided for by the CRW project?  Has a training strategy been developed and described that addresses sustainability issues? Financial  Will there be ongoing and recurrent costs associated with the continued use of the agrometeorological practices?  If so, are recurrent costs likely to be met?  Have the host institutions made a commitment to meeting recurrent costs? Technology  Is the new technology (e.g., IRI software, geoportal) provided of appropriate quality and responsive to stakeholder needs?  Have training and maintenance requirements been specifically assessed and addressed? The evaluation team assessed the evidence for each research question either dichotomously or on the basis of a continuous scale of 1 to 5 (with 1 representing a strongly affirmative response and 5 representing a strongly negative response) in order to allow USAID to better understand the various dimensions of sustainability. However, it was not possible for the evaluation team to assign weights associated with each of the domains and their associated research questions, as this determination is context specific; it is possible that considerations entirely within one domain are of such importance that these outweigh considerations associated with other domains. Therefore, the overall assessment of the likelihood of sustainability was qualitative in nature and based upon the totality of the factors and their relative influence. Data Collection and Analysis Methods To address the EQs, the evaluation team undertook secondary analysis of project documents and conducted semi-structured and group interviews with CRW project stakeholders including UNDP, 20 Although these factors were discussed (in greater and lesser detail) throughout the literature, the clearest articulation of the six factors identified by the evaluation team is reflected in the 2000 AusAID submission to the OECD-DAC Working Party on Aid Evaluation. See AusAID (2000). Promoting Practical Sustainability. DAC Working party on aid evaluation. 33rd Meeting 22-23 November 2000. Final Evaluation Report – Performance Evaluation: CRW Project 7 USAID, implementing partners, government officials, and project beneficiaries. A telephone survey was also conducted of farmer-beneficiaries of CRW-related training activities. A description of research methods and analytical approaches used to address each of the EQs is presented below. TABLE 2: RESEARCH AND ANALYTICAL APPROACHES UTILIZED Evaluation Questions Research Methods Sampling Approach Analytical Approach 1: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for collecting and analyzing agrometeorological data, and making seasonal and climate predictions? - Document review - Key informant interviews - Unstructured observation - Purposeful sampling - Triangulation - Content analysis - Document analysis 2: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather- and climate-related data to farmers and other key stakeholders, as well as to each other? - Document review - Key informant interviews - Unstructured observation - Purposeful sampling - Triangulation - Content analysis - Document analysis 3: To the extent that there are improved practices in the collection, analysis and dissemination of agrometeorological and climate information, are these improvements likely to be sustained? - Document review - Key informant interviews - Purposeful sampling - Content analysis - Triangulation - Document analysis 4: To what extent are other institutional stakeholders using Kazhydromet weather and climate information? - Document review - Key informant interviews - Purposeful sampling - Content analysis - Triangulation - Document analysis 5: To what extent are farmers basing their decisions on Kazhydromet weather and climate information? - Telephone survey - Focus group discussions - Key informant interviews - Purposeful sampling - Convenience sampling - Census sampling - Content analysis - Triangulation - Descriptive statistics - Analytical induction 6: To what extent have farmers adopted climate change adaptation techniques promoted by the demonstration plots and through CRW-supported education initiatives (such as publications, and circulars)? Why/Why not? If adopted, from which initiative? - Telephone survey - Focus group discussions Key informant interviews - Purposeful sampling - Census sampling - Convenience sampling - Descriptive statistics - Content analysis - Triangulation - Analytical induction 7: Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries? - Document review - Telephone survey - Focus group discussions - Key informant interviews - Purposeful sampling - Census sampling - Convenience sampling - Descriptive statistics - Content analysis - Triangulation - Document analysis All collected data have been stored on a secure server and can be transferred in electronic format to USAID within 30 days of its approval of the Final Evaluation Report. Final Evaluation Report – Performance Evaluation: CRW Project 8 Evaluation Sampling and Site Selection Approach The evaluation team collected primary data in July and August of 2015 from a purposive/convenience sample of beneficiaries and stakeholders identified through a review of CRW project documents and in consultation with the project team. Selection criteria focused on ensuring geographic coverage as well as representative coverage of national and sub-national levels of project beneficiaries and informed stakeholders’ groups. The evaluation team sought to reach particular key informants who could contribute to a more in-depth understanding of rationale behind the CRW project design, challenges, and successes with implementation, as well as gender issues. Key Informant Interviews (KIIs): The evaluation team conducted a total of 76 KIIs. This respondent group included 63 institutional stakeholders who were purposively sampled based upon their institutional role, association with the CRW project, and relevant technical knowledge. The KIIs also included 13 farmer-beneficiaries (out of 15 planned) who were selected based upon a convenience sample drawn from participants of a training session held at the Agro-center in Shortandy. Focus Group Discussions (FGDs): Only one of two planned FGDs was conducted. The FGD included 10 farmers (8 men and 2 women) who had not been exposed to CRW project-supported educational activities. The discussion topics included key constraints and opportunities that farmers face and that impact agricultural production, the availability and utilization of relevant agrometeorological information, and overall challenges to wheat production. While a FGD was planned in Kostanay with 10 farmers identified by the manager of that Agro-center, 8 of the planned participants did not show up on the day of the FGD or otherwise refused to participate, and it was not possible to reschedule that FGD during the field research period. Telephone Survey: BRIF Research Group, a Kazakhstan-based survey research company, was contracted to carry out a telephone survey of the 295 farmers and agricultural specialists identified as having participated in CRW-supported capacity building activities in the target oblasts. These individuals were identified by the project, which provided participant lists for trainings. BRIF Research Group was provided with contact details for these participants as well as with the survey instrument (see Annex C). In total, 77 participants (26 percent of the total identified) responded to the survey, which included 42 farmers and 25 agricultural specialists. The primary reasons for non-response were disconnected telephone lines and a failure to answer the telephone. Unstructured Observations: The evaluation team conducted onsite observations at the KHM National Weather/Climate Forecast Center and at two demonstration plots of the Shortandy and Kostanay Agro-centers. The purpose of these unstructured observations was to observe the methods used by different institutional actors to assess current conditions, integrate large-scale, short-term climate forecast model guidance, and produce downscaled regional climate forecast materials for their respective sectors. Table 3 summarizes the sample sizes for in-country data collected by the evaluation team by data collection method, type of informant, and location. The total sample size for each method is shown along with the number of women in parentheses. Final Evaluation Report – Performance Evaluation: CRW Project 9 TABLE 3: SAMPLE SIZE FOR DATA COLLECTION Method and Respondent Type Astana/ Almaty Akmola Oblast Kostanay Oblast North-Kaz. Oblast TOTAL KIIs 41(11) 8(2) 25(4) 2(2) 76(19) USAID 6(4) 6(4) UNDP 10(2) 10(2) National stakeholders 20(4) 20(4) Regional stakeholders 4(1) 16(4) 2(2) 22(7) Other (CCRD, IRI, etc.) 5(1) 5(1) Farmers/agro-producers 4(1) 9(0) 13(1) FGD with farmers 10(2) 10(2) Telephone survey of beneficiaries of CRW educational activities 21 34 22 77(18) Unstructured observations 1 1 1 3 KHM National Weather/Climate Forecast Center 1 1 Demonstration Plots 1 1 2 Evaluation Team The core evaluation team was external to USAID and consisted of: a Team Leader/Evaluation Specialist, Dr. Lyubov Palyvoda; a Kazakh Evaluation Specialist, Baurzhan Zhussupov; two Kazakh Agricultural Specialists, Dr. Yerlan Dutbayev and Dr. Inna Savenkova; and an operational meteorologist, Gregory Gust. Home Office support by the E3 Analytics and Evaluation Project team was provided to the core evaluation team throughout the design and implementation of the evaluation, including technical coordination and guidance, quality control assurance, research assistance, and logistical support. The evaluation team was provided with USAID’s mandatory statement of evaluation standards (see Annex A) and signed conflict of interest disclosure statements indicating that no conflicts were present. Copies of those statements are available upon USAID’s request. Study Limitations Evaluation Scope The scope of this performance evaluation was limited to assessing CRW project performance under the two out of three intermediate results. The evaluation team acknowledges the hard work done by the CRW project team in the domains not covered by this evaluation, and notes that the lack of discussion in this report about those aspects is not a negative reflection of the CRW project’s activities to support the third intermediate result. Terminology Field research and data analysis conducted by the evaluation team uncovered terminology issues in the evaluation SOW. Specifically, the SOW repeatedly uses the term "agrometeorological" to describe the data and information being promoted by the CRW project, whereas all collected data and information were in fact "meteorological" in nature and represented either weather or climate-related information. Final Evaluation Report – Performance Evaluation: CRW Project 10 Additionally, the term "data" in meteorological usage should be used to represent only those meteorological elements as measured and reported at meteorological (weather and climate) stations or plant conditions as measured and reported at agrometeorological stations. Published summaries and analyses of these data, and the forecasts that result from these analyses, would constitute additional forms of meteorological information. These distinctions were validated during the field research and are well understood by the CRW project team and key stakeholders. Respondent Fatigue The evaluation team notes that a separate evaluation of the CRW project was conducted on behalf of UNDP in the spring of 2015, per agreement with USAID/Central Asia Regional.21 Many of the key informants interviewed as part of that evaluation were also re-interviewed under this evaluation. There is the possibility that respondents may have suffered fatigue by these nearly consecutive interviews, and may not have disclosed the same depth of information that they would otherwise have done. Selection Bias For some of the EQs, there is a significant risk of selection bias. Specifically, for research conducted with farmer-beneficiaries, the evaluation team relied upon lists of participants of educational activities of the Agro-centers implementing CRW-supported initiatives – even where the EQ does not strictly relate to the effectiveness of these trainings (e.g., EQs 5 and 7). For these questions, the research was not representative of all wheat farmers in the northern oblasts of Kazakhstan, nor of all farmers in northern Kazakhstan. The evaluation team has been careful to not suggest that findings resulting from this research apply to the greater population from which they are drawn. Limited Ability to Contact CRW Project Beneficiaries The evaluation was conducted up to two years after many of the CRW project activities took place. As such, the sample from which data were collected was limited to those respondents with up-to-date contact information, as provided to the evaluation team by the Agro-centers. Non-KHM Weather and Climate Information as a Confounding Factor Determining the extent to which farmers rely on KHM weather and climate data required asking farmers to recall the extent to which they weighed this information against other sources of information available at the time. In addition to the problems of recall and courtesy bias, there is also the possibility that KHM weather and climate data and predictions were consistent with other sources. In such a circumstance, the evaluation team would only have been able to ask farmers ex-post to assess how they would have weighed the sources of information were they different. Non-Experimental Evaluation Design Since this evaluation did not include a counterfactual group as part of an experimental or quasi￾experimental design, the findings do not support strong causal inference.22 Thus, the evaluation is not able to determine whether the CRW project specifically caused the identified outcomes. 21 “Evaluation of the project ‘Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security,’” April 20, 2015. UNDP. 22 This is a performance evaluation to examine the effectiveness of delivering certain components of the CRW project. As such, the evaluation did not rely on control group comparisons to assess the extent to which results can be attributed to the CRW project. Instead, the evaluation primarily relied on non-experimental assessments of behavior change on the part of beneficiaries and perceptions of key stakeholders of the project’s effectiveness, utility, and sustainability of results. Final Evaluation Report – Performance Evaluation: CRW Project 11 FINDINGS AND CONCLUSIONS Evaluation Question 1: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg to collect and analyze meteorological data, and make weather and climate predictions?  Finding 1.1: The CRW project brought key stakeholders together to begin addressing their corporate weather and climate forecast service concerns.  Finding 1.2: Historically, there was little information sharing between agencies responsible for weather and climate data collection and analysis, and there is significant evidence that this remains the case.  Finding 1.3: KHM’s process for producing weather and climate forecasts and crop yield projections for MinAg was time-consuming and could not regularly produce the information by the time required. Historically, the Kazakh agencies responsible for the analysis and dissemination of weather and climate information have not collaborated, with each agency relying on its own resources to separately:  Gather and archive meteorological, climatological, and agrometeorological datasets;  Analyze these datasets, in either point-source or areal manner, and produce descriptive bulletins;  Generate weather and climate forecasts using whatever techniques they each had available; and  Disseminate weather and climate information within limited networks, generally unique to each agency. Where collaboration did exist, the results were sub-optimal. KHM was contracted by MinAg to provide seasonal climate outlooks (mid-February and mid-September) and a crop yield projection (mid-June) to MinAg weeks before public release, allowing for their production of crop planting or harvesting-related information. However, KHM’s processes to collect and analyze climate data have been cumbersome and antiquated, so that even regular climate and crop condition reports every 10 days were often produced so long after the fact that they were no longer considered pertinent. The CRW project’s capacity￾building work in digitizing data and maps brought about a notable increase in the speed of data analysis for the forecasting process, which will be discussed throughout this section. Similarly, NSRI was contracted by MinAg to provide satellite-derived, areal analyses of soil conditions, crop extent, crop conditions, and areal crop yield projections. Although NSRI had some advantages in budget and technological resources, it was not able to cross-calibrate its satellite-derived surface data nor collaborate on its analyses and forecasts with other agencies. The CRW project provided fora for representatives from each of the three key stakeholder agencies to discuss factors that have limited their capacities to improve weather- and climate-related forecast services. These fora included training sessions, project review meetings, and activities such as a series of initial project planning meetings (February-March 2013) and a U.S. study tour (March 2013) that culminated in a Climate Services Roundtable held in Almaty (May 2013). As a result of the Roundtable, participants identified a distinct set of tools, techniques, and training requirements, mainly at two KHM sites and one NSRI location, aimed at improving climate data analyses and climate forecast skills. Final Evaluation Report – Performance Evaluation: CRW Project 12 During an evaluation findings validation workshop held in August 2015, agency representatives reported that these activities have fostered greater cooperation and advanced thinking among their respective staff. However, interviews conducted by the evaluation team with the agencies’ representatives found that each agency continues to collect data, perform analyses, and produce forecast products that suit their respective needs but little if any of these data and information are freely shared with the other agencies.23 For example:  While NSRI uses spatial analyses to produce drought assessments and crop yield forecasts for MinAg, it does not share this analysis with KHM.  NSRI does not collaborate on the “official” drought assessments and crop yield forecasts produced by KHM. This lack of collective action may result from differing agency standards for collecting such data and inter-departmental security issues. However, it also appears to be influenced by the monetization process by which each agency funds its respective programs (as discussed in later EQs). Kazhydromet (KHM)  Finding 1.4: KHM has incorporated the CRW-provided Standardized Precipitation Index (SPI) methodology in the production of monthly drought bulletins.  Finding 1.5: KHM uses the International Research Institute for Climate and Society (IRI) Data Library to conduct digital analyses of digitized historical climate station records and recent station data to support its drought and climate assessments.  Finding 1.6: The use of digitized historic weather and climate data to produce analog seasonal climate forecasts is less labor and time intensive than the previous approach of manual analysis of hard-copy data.  Finding 1.7: KHM has not fully adopted the Numeric Forecasting approach for which CCRD has provided training and software – except on an experimental basis. The CRW project provided software and training in the use of the IRI Data Library to both the Almaty and Astana branches of KHM. The IRI Data Library is an online data repository and analysis tool that allows users to view, analyze, and download hundreds of terabytes of climate-related data through a standard web browser – provided that historical data have been digitized in the library. Currently, both the Almaty and Astana branches of KHM are now able to make use of IRI Data Library software for the computation and storage of meteorological data from stations across Kazakhstan. Although this has reduced the staff hours previously required to perform such tasks, this effort has only incorporated a portion of the historical station data, and there are still large quantities of station data that are virtually inaccessible to the analysis process. In fact, several decades’ worth of such paper files exist in an archive in Almaty and are unavailable for routine analysis as well as subject to the ravages of time.24 According to the KHM team, all the historical weather map files used by KHM for its analog climate forecasting process have been completely digitized and can now be more easily reviewed via digital files on a computer workstation, as opposed to leafing through volumes of paper maps. While digitization has markedly reduced the time necessary for forecasting, by perhaps dozens of hours per month, it was not possible for the evaluation team to make a determination as to whether it improved seasonal forecast 23 A singular exception to this may be the use of surface-based soil moisture sensors (operated by KHM) to calibrate satellite￾based sensors (used by NSRI), as was described by NSRI Division Director, Azamat Kauazoy. See also, Presentation before the AGU, December 16, 2014, in San Francisco. 24 Interview with Mereke Akbuzay, KHM Deputy Director General, on July 21, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 13 accuracy. The KHM Deputy Director General estimated that "the overall quality of [short-term] weather forecasts has increased by as much as 10 to 15 percent, so if before a forecast was 65 percent accurate it is now 75 percent accurate.”25 This was echoed by other KHM staff, who stated that “seasonal forecasts have not improved much yet, only to 55 to 60 percent accurate, and it is important to increase the validity of those.”26 Although no independent analyses are available to substantiate either claim, they are consistent with anecdotal comments made by other several local experts including the Director of the Barayev Research Institute for Grain.27 The CRW project also provided software and training in the use of the SPI to the Almaty branch of KHM, which it has successfully incorporated into its monthly drought assessment processes. Prior methods for drought assessment required a labor-intensive point-by-point hand analysis, while the new approach uses data that have been ingested and processed through the IRI Data Library to produce the analyses and maps automatically. The output is now available for publication in a much timelier fashion, several days ahead of previous schedules, and in a more polished, graphic-enhanced digital format. This has brought KHM into accordance with World Meteorological Organization (WMO) recommendations and has reduced the amount of time required by KHM to undertake routine monthly tasks.28 The CRW project also provided training on the use of the IRI/Climate Predictability Tool (CPT) for statistical prediction of monthly or seasonal climate patterns. This numeric approach to forecasting is generally regarded as more accurate than the traditional analog approach to forecasting utilized by KHM. While KHM does undertake numeric forecasting, to date this method is only used in tandem with the traditional analog methods while research is being done to establish an improved set of predictors. According to KHM respondents, the successful implementation of the IRI/CPT may hinge on (1) the development of a set of improved predictors, and (2) the digitizing of additional Kazakh temperature and precipitation records from the last few decades. According to IRI respondents, the ongoing use of the CPT as part of KHM’s actual seasonal forecast process will help to “train” the equations being used and to improve their predictive skill. The Astana and Almaty offices of KHM do different tasks. During a visit to KHM-Astana, the evaluation team learned that this office does fully assess current conditions and fully integrate various climate forecast model guidance into its forecast products. The team found that KHM-Almaty offices are mainly involved in water- and drought-related assessments or research, and do not conduct routine forecast operations. KHM-Almaty staff have begun issuing experimental drought forecasts as part of their monthly drought bulletin, but they do not yet incorporate monthly or seasonal climate forecasts, as issued by KHM-Astana, in an operational manner. Based upon interviews conducted and a review of available CRW project literature, the evaluation team determined that the improvements noted by respondents in forecast process or accuracy are most directly traceable to an increase in the speed of data ingest and analysis, brought about by the digitization of data and maps and their subsequent use as part of the IRI Data Library and the Analog Map Viewer. National Space Research Institute (NSRI)  Finding 1.8: NSRI now uses techniques for tracking soil moisture (Special Sensor Microwave Imager – SSMI) and monitoring crops (Normalized Difference Vegetative Index – NDVI), based on training and access received through CRW project activities. 25 Idem. 26 Interview with KHM-Astana staff on July 21, 2015. 27 Interview with Dr. Kanat Akshalov at Barayev Institute on July 27, 2015. 28 Interview with Svetlana Dolgikh on July 24 2015. Final Evaluation Report – Performance Evaluation: CRW Project 14  Finding 1.9: NSRI produces drought assessments and crop yield forecasts for MinAg without collaborating with KHM forecasters. Researchers at NSRI are involved in the routine analysis of satellite data to monitor the areal extent of snow cover, soil moisture, seeded lands, crop status, and progress of the harvest. Such satellite-derived analyses provide areal coverage of all of Kazakhstan, including areas that have a sparse distribution of surface-based sensors and reports. NSRI’s Space Monitoring and Natural Processes Division conducts surface soil moisture assessments, and based on these it produces a short-term drought outlook specifically for MinAg. As noted earlier, there is little information sharing in the production of NSRI drought forecasts. NSRI does not incorporate any KHM-produced climate model output or climate forecasts into its drought outlook products, which are specifically produced for MinAg. However, KHM-Almaty recently hired a Satellite Meteorologist to increase its capacity to analyze NSRI satellite imagery. Interview respondents expected that this individual will help provide a technical bridge between these two Almaty-based offices and contribute to an increased exchange in products and services among the agencies.29 As previously noted, NSRI participated in project planning activities, a U.S. Study Tour, the Climate Roundtable, and other CRW project-sponsored trainings and meetings including those provided by CCRD. Following these activities, NSRI adopted several new agrometeorological tools and techniques, including:  SSMI imagery for tracking soil moisture. This was provided free-of-charge by the U.S. National Oceanic and Atmospheric Administration (NOAA) Polar Orbiting Earth Satellites. The imagery is used by NSRI in concert with data from Kazakhstan’s satellites to provide greater detail and accuracy in soil moisture assessments.  NDVI and Vegetative Condition Index (VCI) for satellite-based crop monitoring. These are used by the U.S. National Drought Mitigation Center (NDMC), and NSRI has adopted these indices for operational use and is investigating more advanced methodologies. In addition, NSRI management was impressed with the design and function of the NDMC, which they had visited in Lincoln, Nebraska as part of the U.S. Study Tour. NSRI has since applied for and been awarded a three-year grant to fund the development of such a center in Almaty. One NSRI participant commented that trainings and consultations delivered by CRW project experts were “concrete, informative, and relevant.”30 In particular, all respondents who went to the U.S. noted that the Study Tour program was of great interest and they were shown the latest achievements of the leading centers. The participants had a good understanding of how these centers operated and gained a greater understanding of the steps required for them to develop such a center. One such NSRI representative noted, “on the study tour, at UNL/National Drought Mitigation Center we saw that such a center was not so monumental of a task. It consisted of some 20 people. So we have applied for a national grant to develop our own Kaz National Drought Center, for technologies, etc.”31 Ministry of Agriculture (MinAg)  Finding 1.10: MinAg has Climate and Agrometeorological Station data that could be made available to other stakeholders and integrated into their analyses. 29 Svetlana Dolgikh, op cit. 30 Interview with NSRI representative, July 24, 2015. 31 Idem. Final Evaluation Report – Performance Evaluation: CRW Project 15  Finding 1.11: MinAg has expertise in local drought analysis, climate analysis, and climate forecasting at their Agricultural Institutes that could inform the work of KHM and NSRI. Like the other key stakeholders, MinAg was a regular participant in CRW-sponsored trainings and activities, and to a certain degree benefitted from the increase in interagency communication and collaboration produced by these processes. Since MinAg does not officially collect meteorological data or produce climate analyses and forecasts, there were no CRW-related activities that specifically targeted such technical functions within MinAG. Thus, within MinAg there are no apparent improved practices related to the collection and analysis of weather data or in the forecast of climate. However, the evaluation team did find that Agricultural Research Institutes, such as the ones visited in Shortandy and Kostanay, do collect certain weather, climate, and agrometeorological data from their local micro-networks. Most of these data are kept in-house at the Institutes, and are used for their local production of climate analyses and local seasonal climate forecasts.32 KHM does not currently have ready access to (or it chooses not to incorporate) surface meteorological data gathered at these MinAg stations, and also does not have access to any locally-produced climate analyses and forecast products. It was clear from discussions with staff at the Institutes that they feel they have information and expertise that could improve the quality of “official” drought and climate assessments, crop assessments, crop yield forecasts, and seasonal climate forecasts, which are primarily produced by KHM. It would seem that a pairing of such expertise at the local level with KHM staff, and an integration of such data and information into the assessment and forecast mix, could be beneficial to the agencies and to farmers. 32 Site visit and Kanat Akshalov, op cit. Conclusions:  Conclusion 1.1: While the CRW project was instrumental in bringing key stakeholders together to examine new practices and discuss new policies that address their shared concerns, there remains little cooperation and collaboration between agencies. [Supported by Findings 1.1, 1.2, 1.3, 1.9, 1.10, and 1.11]  Conclusion 1.2: New tools and techniques, especially the introduction of new imagery (SSMI), new indices (SPI, NDVI), and new database management (IRI Data Library), have improved the production of drought, crop, and climate assessments by both Kazhydromet and NSRI. [Supported by Findings 1.4, 1.5, 1.6, 1.7 and 1.8]  Conclusion 1.3: The KHM climate forecasting process remains hampered by an inadequate surface reporting network and large amounts of un-digitized historical station data. [Supported by Findings 1.5 and 1.10]  Conclusion 1.4: While CRW project support has led to improvements in the production of drought, crop, and climate assessments, KHM-produced seasonal climate forecasts have seen only marginal gains, and those are largely due to improvements in data handling via the IRI Data Library and Digital Map Library. Techniques to improve the accuracy of forecasting have not been fully adopted. [Supported by Findings 1.5, 1.6, 1.7, 1.9, 1.10, and 1.11] Final Evaluation Report – Performance Evaluation: CRW Project 16 Evaluation Question 2: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather- and climate-related data to farmers and other key stakeholders, as well as to each other? Weather and Climate Data Sources  Finding 2.1: KHM, NSRI, and MinAg each collect, analyze, and disseminate weather and climate-related data.  Finding 2.2: Weather and climate-related data are available primarily through fee￾based mechanisms, even when the recipient is another Kazakh agency.  Finding 2.3: Fee-based provision of weather and climate data is an important revenue stream for both KHM and NSRI.  Finding 2.4: Each agency (KHM, NSRI, and MinAg) requires data from the others to undertake agrometeorological analyses.  Finding 2.5: MinAg has contracts with KHM and NSRI for the provision of agrometeorological information, although NSRI data are used only for internal analytic purposes.  Finding 2.6: MinAg-procured KHM agrometeorological information is made available to farmers through monthly bulletins distributed locally through meetings at local administrations (akimats) and to the Farmers’ Union, although the evaluation team identified only one farmer who had received the bulletin. All three key stakeholders (KHM, NSRI, and MinAg as well as its local Agricultural Research Institutes) produce, analyze, and disseminate weather- and climate-related data and information. The form and method of dissemination depend on the type of information and target audience, and is provided through both fee (primarily) and non-fee mechanisms, as summarized in Table 4. Final Evaluation Report – Performance Evaluation: CRW Project 17 TABLE 4: INFORMATION DISSEMINATION MECHANISMS Source Recipients Type of Information Form Paid/Free KHM MinAg, farmers Agrometeorological monitoring data Inquiries and bulletins (agro￾meteorological decadal bulletin, monthly drought monitoring bulletin) Paid Overview of agrometeorological data Decadal (10 days) overview on the KHM website Free Medium and long-term weather forecasts Bulletins (monthly forecast, seasonal forecast), maps Paid Agrometeorological forecasts33 Forecasts, recommendations, guidelines Paid KHM MinAg, farmers, insurance companies Historical data of adverse weather conditions (drought, hail) Inquiries Free KHM Available to all Current meteorological data Hourly reports, via media and KHM website Free Short-term weather forecasts and warnings Websites, media Free NSRI MinAg Spatial data: soil moisture Reports, maps Paid Spatial data: seeded lands and the crop status Reports, maps Paid Spatial data: crop yield forecast Forecasts, maps Paid MinAg Research Institutes Farmers/agro￾producers Determining optimum sowing/harvesting and other field works dates/ periods Recommendations Free Research Institutes Farmers/agro￾producers Agrometeorological monitoring data Recommendations Free Agrometeorological forecasts Farmers Union Farmers/agro￾producers Agrometeorological monitoring data and forecasts Bulletins, forecasts Free As Table 4 illustrates, only limited information is available from multiple sources:  Basic meteorological and agrometeorological observational data were available from KHM and through certain MinAg Research Institutes that operate stations to collect that data for their own use.  High-resolution surface satellite imagery offers improved land cover characterization and is only available from NSRI, although certain analyses based off the images are supplied to MinAg officials.  Several types of information are produced by NSRI by processing and triangulating data from various sources. For example, the determination of optimum sowing dates as defined by the Agricultural Research Institutes, which are affiliated with MinAg’s KazAgroInnovation (KAI), is based on KHM seasonal forecasts and other agrometeorological data. Thus, each of these stakeholders is dependent on one another for critical information:  KHM is interested in NSRI data because it allows for a better extrapolation of meteorological observations when meteorological stations are located far from each other. Additionally, NSRI helps KHM to provide independent assessments of the crop yield data using satellite imagery. 33 Forecast of available soil moisture reserves, determining optimum sowing date, crop status, forecast of agro-meteorological conditions for the harvest, and crop yield forecast. Final Evaluation Report – Performance Evaluation: CRW Project 18  NSRI is interested in KHM surface observational data for calibration/verification of its own satellite-derived data.  Agricultural Research Institutes need data from both KHM and NSRI in order to study the impact of weather and climate on productivity, as well as to develop more valid and precise recommendations for implementation of agricultural technologies. Despite these stakeholders’ reliance on each other’s work, the evaluation team found significant challenges and barriers to information exchange, including the fee-based system, which prevents access to and the exchange of data among these organizations and between these organizations and farmers and agro-producers. KHM Almost all long-term and seasonal meteorological or agrometeorological information provided by KHM is fee-based. Only hourly airport-based weather observations, storm reports, and short-term weather forecasts (up to three days) are provided free of charge by KHM and these are available through a variety of sources including the KHM website, media outlets, and other Internet-based sources. The annual costs for access to daily weather data (temperature and precipitation) measured at one nearby KHM meteorological is about 100,000 tenge (U.S. $529).34 This cost is equivalent to approximately 7 percent of the average annual salary in Kazakhstan35 and according to interviews with informants from institutional stakeholders (KHM, NSRI, and Agricultural Research Institutes), such fees are a major obstacle limiting access to such critical information:  "If farmers want to get a long-term weather forecast directly from KHM, it would be very expensive for them. We instruct people to obtain the weather forecasts for free from the local administration or the extension center. These organizations receive this information from us."36  “Six years ago, we asked KHM to provide the meteorological observational data. They requested us to pay a large amount of money for one year data. A year ago, we again asked KHM for the monitoring weather data. Again, we were asked to buy the data and their analysis.” KHM and NSRI representatives noted that user fees constitute a substantial source of their operating budgets. Therefore, the two institutions are currently not in the position to provide the weather, soil condition, and climate-related data for free. MinAg MinAg is a major customer of KHM for agrometeorological information, and repackages or distributes such information to farmers. KHM issues this information in the form of an agrometeorological bulletin once every ten days. According to interviews with representatives from MinAg, KAI, and the CRW project, these bulletins are made available to farmers via MinAg for free in three ways: 1. MinAg  local authorities (regional affiliates of the MinAg)  farmers; 2. MinAg  KAI (regional Agro-centers)  farmers; and 3. MinAg  Farmers Union  farmers/agro-producers The first is a traditional information dissemination pathway and was established in the Soviet era. Almat Kunakov37 from MinAg remarked, "there is a long-term contract between MinAg and KHM. They give us 34 Based upon exchange rate of 7/28/2015. 35 The average salary in Kazakhstan is estimated at 1,461,804 tenge per year as of September 2015. See http://www.tradingeconomics.com/kazakhstan/wages 36 Interview with Almat Kunakov, MinAg, on August 5, 2015. 37 Interview on August 5, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 19 various information. KHM produces its bulletins, mostly on monthly basis. KHM gives the bulletins to us, then we distribute them to akimats and agro-extension centers." The CRW project helped with the bulletin redesign, appearance, and content to redesign it from being a purely meteorological bulletin to more agriculturally-focused, with such features as:  Additional infographics;  Information on different crops and agricultural technologies, with summaries under each crop section;  Simple tips and methodologies related to planting periods, fertilization, harvesting, interpretation of climate data, climate forecast, example of farmer record books, news on harvesting periods based on a comparative summary of the space satellite imaginary and field-based data testing;  Calendar of events organized by extension centers and other organizations; and  Contact information for different agricultural stakeholders and KHM staff. While respondents did indicate that akimat meetings took place before planting and harvesting and included presentations by KHM, it does not appear that the bulletins are regularly disseminated at these meetings. Only one respondent to the telephone survey indicated having received bulletins through the akimats and none of the farmers who participated in individual interviews or FGDs reported recently receiving any agrometeorological bulletins: “we previously obtained agrometeorological Bulletins produced by KHM from the Ministry of Agriculture. Now we don’t receive them.”38 NSRI NSRI has been developing crop yield forecasts based on its monitoring of grains, snowmelt, and spring sowing, and is now working on soil moisture and vegetative monitoring using the NDVI. While this work is based on the season, its capacity includes providing inter-seasonal comparisons through the VCI. Its satellite-based drought assessment is currently not compared to the KHM surface-based drought analysis, although it could be useful for farmers and has been noted as desirable by at least two KHM representatives. MinAg has an exclusive annual contract with NSRI for the provision of certain satellite-derived information, including periodic ground and crop condition reports and seasonal crop yield forecasts. MinAg uses this information for internal analytical purposes only, and does not retransmit it to other stakeholders. NSRI representatives noted a lack of regular operational collaboration on monthly drought assessments. Although one representative praised the CRW project for promoting NSRI and KHM collaboration on the geoportal, that tool has not yet been developed. Additionally, there is an ongoing joint research with IRI and KHM on using data from the KHM surface observations to calibrate the NSRI satellites. CRW-Supported Dissemination Practices  Finding 2.7: The farmer and agro-producer respondents predominantly access weather and seasonal climate information via the Internet to a significantly greater extent than other mechanisms.  Finding 2.8: CRW project technical assistance resulted in changes to the structure and format of the monthly bulletins to make them more accessible and understandable to farmers. 38 Interview with Vladimir Skoblikov, Barayev Research Institute for Grain (Shortandy) on July 24, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 20  Finding 2.9: In collaboration with CCRD, the CRW project facilitated stakeholder engagement on the development of a geoportal for expanded dissemination of weather and climate data.  Finding 2.10: There is still no agreement between agencies on many aspects of geoportal implementation and the development of the tool has been delayed. The CRW project provided support to improved dissemination practices in two ways: to improve the structure and format of MinAg-produced monthly bulletins, and through the development of a geoportal. Interestingly, data from the telephone survey suggest that the primary means through which farmers access weather and climate information is via the Internet, with only limited access obtained through other sources. Of the 42 farmer respondents to the survey (with respondents allowed to identify multiple sources of information), 33 obtained seasonal climate forecasts through computer-based Internet service and 14 through phone-based Internet service. Among other means, three farmers obtained these forecasts via television and two relied on meetings in akimats or personal experience; only five accessed information via the radio, newspapers, MinAg bulletin, their own meteorological station, or Research Institutes. Over the year prior to this evaluation, the CRW project helped improve the overall structure and format of these agrometeorological bulletins. The monthly drought monitoring bulletin is now based on SPI data, with technical assistance provided through CCRD. In addition, the Farmers Union has offered its members access to agrometeorological information by signing a Memorandum of Understanding (MOU) with KHM. The signing of this MOU was not something that the CRW project intentionally promoted. Rather, the agreement was at their own initiative, although the project provided a forum to discuss the exchange of information. The CRW project has begun working with the Farmers Union to help disseminate bulletins and other materials produced by KHM to farmers through their network, and to explore a way for farmers to have a voice with KHM by establishing a feedback mechanism. However, none of the farmers who participated in the evaluation team’s individual interviews or FGD have received these bulletins to date. One of the main means by which data dissemination should have been improved was through the development of a geoportal to disseminate weather- and climate-related information. The CRW project collaborated with CCRD to develop the tool, which has been under development by a U.S.-based company, SpatialDev, located in Seattle, Washington. The geoportal will have five main functions:39 1. A search service enables users to search collections of spatial data and geo-tools based on the corresponding metadata and to display the content metadata. 2. Imaging services to provide the ability to view data, navigation images and their graphic overlay data, and also display map legends and related information contained in the metadata. 3. Services for downloading information to allow for copying of spatial data sets or fragments thereof, and the possibility of providing direct access to data. 4. Data transformation services to enable transforming a set of spatial data for interoperability. 5. Invoking other (remote) services. In an automatic mode, it will be able to connect or provide users with services such as an insurance calculator or connect users to the other geoportals, depending on their needs. 39 NSRI, Geoportal Concept Note, Almaty 2015. Final Evaluation Report – Performance Evaluation: CRW Project 21 Figure 1 presents the conceptual structure of the portal. FIGURE 1: GEO-PORTAL CONCEPTUAL STRUCTURE40 The geoportal concept note identifies sources of information and automation of information flow processes, and envisions the provision of key information between agencies. Among the key data sources are: 1. Space pictures. Satellite images of various spatial resolution obtained by downloading from the server (NOAA, ModisTerra/Aqua and other channels HTTP, FTP), as well as by direct discharges and other means. 2. Meteorological data. Meteorological data obtained from weather stations, weather online resources, and KHM. The evaluation team found that the only surface weather observations available online in Kazakhstan are from sites operated by KHM. While private individuals and MinAg may operate local meteorological stations, these are not available online or via standard WMO networks. 3. Thematic information based on remote sensing. Subject information represented by vector data on the basis of satellite images. The USAID Environmental Officer involved with the CRW project design confirmed that buy-in was obtained from both NSRI and KHM for the geoportal, and both agencies committed to sharing data. The geoportal concept note specified that as a leader in the field of remote sensing data processing, NSRI is best positioned to manage the portal: “NSRI already has good material resources and sufficient staff of experts to perform the tasks of monitoring and forecast droughts.” NSRI also has a grant from the Ministry of Education and Science for “information technology monitoring and forecasting of droughts based on long-term series of data remote sensing of the territory of Kazakhstan." Nonetheless, the evaluation team did not find complete agreement on the leading role of NSRI in geoportal implementation. KAI officials noted that they may have the information technology professionals, server capacity, and budget resources in place within MinAg to develop and manage such a geoportal.41 40 Geoportal Concept Note, 2015. 41 Interview with KazAgroInnovation on July 21, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 22 Notably, the legal framework to ensure sustainability of the geoportal could be fixed in the MOU between the CRW project and NSRI. Specifically, the project is supposed to procure a server with other software to support the operation of geoportal, to be completed “as soon as the geoportal is completed.”42 However, it is not clear the extent to which the concept note obligates NSRI as the key agency responsible for the development of the geoportal or KHM and NSRI to share data to further this implementation. At the time of the evaluation, there were significant delays in the geoportal launch and no clear path or formal agreements on who will complete the tool, what data and information will be available through it, how access to these data and information will be regulated, or what fee structures may be involved to support it. In light of the telephone survey evidence concerning the ways that farmers access seasonal climate information, the delays to the development of the geoportal potentially represent a significant impediment to the effectiveness of the CRW project’s interventions to promote more effective dissemination. 42 Correspondence with the CRW project team in September 2015. Conclusions:  Conclusion 2.1: The fee-based structure of most agro-meteorological information produced by KHM, NSRI, and MinAg significantly impedes the dissemination of this information both to farmers and among the agencies. [Supported by Findings 2.1, 2.2 and 2.3]  Conclusion 2.2: Although the CRW project has provided support to improve the structure and dissemination of the MinAg monthly agricultural bulletins, it does not appear that farmers actually receive these bulletins. [Supported by Findings 2.6 and 2.8]  Conclusion 2.3: The CRW project’s support to the development of a geoportal for the expanded dissemination of weather and climate data has not been effective to date, as development has been delayed and there remains no agreement on what information will be contained on the geoportal, how it will be made accessible, and to whom. [Supported by Finding 2.10]  Conclusion 2.4: The presence of many unsolved issues makes the possibility of creating and launching the geoportal in the upcoming year unclear. [Supported by Finding 2.10] Final Evaluation Report – Performance Evaluation: CRW Project 23 Evaluation Question 3: To the extent that there are improved practices in the collection, analysis and dissemination of weather and climate information, are these improvements likely to be sustained? As detailed in the Evaluation Methodology section of this report, to assess the sustainability of new practices for the collection, analysis, and dissemination of weather and climate data, the evaluation team considered six factors (see Table 5) that contribute to the sustainability of development interventions. TABLE 5: SUSTAINABILITY FACTORS Policy Support is consistent with and supported by relevant government sectoral policies Participation & Ownership Local stakeholders support the intervention, which responds to clearly expressed local needs. Management & Organization The project was cognizant of the capacities of local actors and incorporated sustainability as a key aspect of project delivery. Training Needs The project assessed the training needs of project stakeholders and developed training that addressed these needs. Financial Host institutions are capable and committed to meeting the ongoing costs associated with the use of new practices. Technology New technology/software provided by the project is appropriate for the required tasks and the project has accounted for any ongoing training and maintenance. Policy  Finding 3.1: The CRW project provided climate change-related recommendations for various national policy documents, although not directly related to the collection, analysis, and dissemination of weather and climate information.  Finding 3.2: Neither KHM nor NSRI have amended formal policies or protocols to reflect the use of new analysis and forecast practices promoted by the CRW project.  Finding 3.3: KHM has modified its internal operating procedures to incorporate the use of certain CRW-promoted practices. With respect to climate policy generally, climate change issues are under the control of the President of the Republic of Kazakhstan, and climate change mitigation strategies are being developed at the national level.43 All relevant GOK ministries have their respective strategic plans, although a unified policy paper is not yet in place. This policy limitation was recognized by the CRW project in the analysis it presented in early 2014,44 which pointed out that “the introduction of principles for long-term weather forecast and agricultural crop productivity/harvest outlook in strategic documents is a cornerstone of sustainability.” The evaluation team found that the CRW project contributed to increased awareness among some institutional stakeholders regarding the effects of climate change and the need for adaptation and mitigation measures, including through providing recommendations for various national policy documents: 43 Interview with Ministry of Energy representatives, Astana, August 3, 2015 44 Increasing Wheat Resistance/Hardening in Kazakhstan Against the Background of the Climate Change to Ensure Food Security in the Central Asia Presentation by Yerlan Zhumabayev, January 30, 2014, Astana. Final Evaluation Report – Performance Evaluation: CRW Project 24  The CRW project provided recommendations on improving the wheat production sector and market, which were incorporated into the Concept of the Transition of Republic of Kazakhstan to Green Economy that was endorsed by the GOK on May 30, 2013 (#577). The documents formalized the roles of key stakeholders including KHM. A representative from KHM stated, “our state has a concept for the transition to the green economy up to 2030. The Kazakh Hydrometeorological Center has a role in the process.”45  The CRW project provided recommendations adopted by the Agrarian Science Development Program – 2020 developed by KAI, including those related to an improved geoportal, agricultural technologies that are resilient to climate vulnerability, and measures to address the adverse impacts of climate change on the wheat production sector.  The CRW project is currently assisting the Ministry of Energy to collect proposals to develop a concept of a draft law on climate change adaptation, which the Ministry plans to include in the Parliament work plan/agenda for 2016. Regarding seasonal climate prediction practices, neither KHM nor NSRI have yet modified their formal policies regulating any practices introduced as a result of the CRW project. However, KHM departments have modified their internal operating procedures, forecasting work assignments, and the activities of their technical staff on a daily and monthly basis, to ensure the application of new practices. Institutional informants stated that internal procedures regarding the implementation of the new practices can be adopted at the institutional science and technology council level without ministerial approval: “As a rule, the protocols we follow in our work do not require any external approval.”46 This finding is inconsistent with observations made during the recent CRW project mid-term evaluation,47 a Mid-Level Assessment,48 and interviews with a CCRD representative,49 where it was indicated that there may be “legal barriers prohibiting the use of different, perhaps more appropriate means of climate prediction.”50 Participation and Ownership  Finding 3.4: The CRW project held consultations to garner institutional buy-in and ensure that interventions responded to institutional needs.  Finding 3.5: KHM and NSRI representatives actively participated in the selection and design of the CRW project capacity-building interventions.  Finding 3.6: KHM and NSRI respondents expressed that the practices promoted by the CRW project were important and useful to the agencies, even if not all were adopted. At the beginning of the CRW project, CCRD consultants and CRW experts held a number of meetings and consultations with project stakeholders and beneficiaries at the national and regional levels in order to present new practices, forecast models, and climate change adaptation techniques, and to obtain buy￾in. Former UNDP manager Stanislav Kim commented, “the participation of local experts in the meetings with their peers from the U.S. was beneficial since they learned about the latest international experience and practices as well as they were in position to influence the project training program. As a result, the 45 Evaluation Findings Validation Workshop, August 5, 2015 46 Ibid. 47 Evaluation of the CRW Project, Roberth et al, April 2015 48 Mid-Level Assessment, Glen Anderson, Oct 2014 49 Interview with Glen Anderson, CCRD Chief of Party, July 2015 50 Evaluation of the CRW Project, Roberth et al, April 2015 Final Evaluation Report – Performance Evaluation: CRW Project 25 local experts have become more interested and engaged in the workshops and seminars. Besides, the relevance of the new knowledge and skills has also contributed to their higher engagement.”51 The CRW project’s selection of capacity building interventions was informed by critical self-assessments and dissatisfaction from KHM and NSRI experts as well as recipients of weather information with the quality and reliability of seasonal forecasts. These views were expressed during consultations with stakeholders including those conducted by CCRD consultants.52 For example:  NSRI expressed a need for modeling and correlating snow melt to spring soil moisture, in order to improve forecasts for the timing of sowing. This issue was addressed by CRW project interventions.  KHM recognized the importance of strengthening its climate forecasting and capacity to prepare numerical climate forecasts. As a result, the CRW project provided training on the use of CPT. By clearly expressing their needs during the initial stages of the CRW project, key institutional stakeholders participated in and influenced the design of the project and helped make the interventions more focused and specific. The importance of the KHM capacity building efforts that were required to implement the new practices was clearly recognized and supported by the institution’s Deputy Director. KHM and NSRI respondents stated that the new practices addressed their needs and were consistent with emerging international standards encouraged by organizations such as the WMO. However, they acknowledged that some practices are not fully utilized (e.g., the continued use of analogue forecasting). KHM and NSRI respondents reported that these practices had become part of day-to-day operations and would continue, stating that they “would not be turning back – only going forward.”53 The CRW project has also promoted networking between institutions including KHM and NSRI, including through joint research on droughts. This has led the KHM office in Almaty to hire a new staff member to interpret space data and evaluate soil hydration conditions with the help of satellite imagery, which may serve as a stepping stone to improved collaboration and ownership in the future. Management and Organization  Finding 3.7: The CRW project did not have an explicit focus on sustainability as part of project planning and design.  Finding 3.8: KHM and NSRI share information obtained from CRW project trainings among their staff and continue to have access to some support from CCRD partners. A review of project documents and monitoring frameworks demonstrates that the sustainability of new agrometeorological practices was not explicitly addressed in CRW project planning and was not subject to monitoring. The CRW project was primarily output focused (e.g., number of people trained and practices introduced). Based on stakeholder consultations conducted by UNDP and USAID, CCRD compiled a baseline report of stakeholder capacity and needs in order to develop capacity-building interventions. The evaluation team did not obtain access to this study and thus cannot determine which capacities were assessed and what criteria were used. CRW project trainings contributed to the professional development of many KHM and NSRI staff members, and both institutions54 have protocols to share knowledge among staff. In addition, there is 51 Interview with Stanislav Kim, August 20, 2015 52 CCRD trip reports from 2012 and Report on stakeholders’ consultations from 2013 53 Interviews with KHM Astana-Almaty and NSRI, July 21 and 24, 2015 54 Evaluation Findings Validation Workshop, August 5 and KII, Almaty, July 24, 2015 Final Evaluation Report – Performance Evaluation: CRW Project 26 evidence that some staff continue to take advantage of follow-up support from CCRD partners such as IRI, and continue to use the software and information available on the IRI website. Training Needs  Finding 3.9: Needs for ongoing training have been routinely assessed and addressed by the CRW project.  Finding 3.10: A training plan for new practices was developed and described in the CRW project annual plans. The CRW project informally assesses training needs and provides capacity building activities on an ongoing basis. Training plans are developed as a part of the project’s annual work plans,55 and specify the participating institutions, training needs, topics covered, and expected results. The training plans also consider how trained participants will share new knowledge with other staff. Institutional stakeholders largely felt that the training was relevant and important. For example, KHM staff commented that the knowledge, skills, and tools acquired have helped them reduce the time necessary to prepare their short-term forecasts, to make those forecasts more detailed (down to the oblast level), and to extend the timeframe of those forecasts to 15 days.56 There are many different types of training activities delivered by the CRW project, most of which incorporated a Training of Trainers approach, including:  Formal training;  Informal training at the workplace;  Technical assistance on a demand-driven basis; and  Remote consultations. An informal stakeholder needs assessment at the beginning of the CRW project helped establish the general framework for the project’s ongoing training approach. However, as the CRW project entered its fourth year, some respondents felt that another comprehensive stakeholder needs analysis was needed: “The project began three years ago and back then we had a number of questions for the American experts. Many things have changed since then. We have accomplished a lot. Perhaps, we should get together one more time and define our training needs and priorities.”57 Financial  Finding 3.11: There are minimal recurring costs associated with the practices promoted by the CRW project.  Finding 3.12: To the extent that recurrent costs do exist, KHM has signed agreements to meet the recurring cost for new practices adopted through the CRW project. Most of the CRW-promoted practices are free, and only occasional software upgrades are needed (e.g., SPI, SSMI). KHM has made assurances that they will meet the cost of new equipment and software updates.58 Additionally, the use of some practices, such as the IRI Data Library, reduces the labor required to perform routine tasks and would therefore represent a cost savings to KHM. 55 3rd year work plan, October 1, 2013 – September 30, 2014. 56 Interview with KHM-Astana representatives, July 21, 2015. 57 Representative from the Geography Institute, Evaluation Findings Validation Workshop, August 5, 2015. 58 Interview with UNDP Chief Technical Advisor. Final Evaluation Report – Performance Evaluation: CRW Project 27 During the CRW project, a synoptic long-term forecaster position was created by KHM. This imposes a recurrent cost, but KHM has taken full responsibility for the ongoing costs for the new forecaster and any related equipment and software needs. Technology  Finding 3.13: New technologies provided by the CRW project were of appropriate quality and responsive to KHM/NSRI needs. Several tools and software applications were installed at KHM and NSRI, including new data archival and retrieval methods, data analysis routines, access to new satellite imagery sources, and new indexing methodologies that have been cited throughout this report. Most new technologies were significantly advanced as compared to previous ones. According to respondents from these institutions in both Astana and Almaty,59 these new technologies have addressed their needs. NSRI received a three-year grant from the Ministry of Education and Science to develop the methodology for both a National Drought Center and a geoportal. Representatives for KHM and NSRI hoped to maintain support from CCRD for the geoportal development, finalization, and launch. At the time of this evaluation, CRW project staff were not aware that the CCRD project will end in October 2015, but were informed by the CCRD project leader that work on the geoportal would not continue. The geoportal, or its equivalent, is therefore not operational, but the concept remains under development and NSRI believes it has the staff and funding capability to maintain a geoportal once it is established. 59 Interview with KHM and NSRI, Astana and Almaty, July 21 and 24, 2015. Conclusions:  Conclusion 3.1: The CRW project’s support was consistent with GOK climate change policies and objectives and beneficiary institutional policies. [Supported by Findings 3.1, 3.2, and 3.3].  Conclusion 3.2: The introduction of new agrometeorological practices responded to institutional needs and garnered institutional support. [Supported by Findings 3.4 and 3.5].  Conclusion 3.3: The CRW project objective was aimed at improving agrometeorological practices without consideration of their sustainability within partnering institutions. [Supported by Finding 3.7]  Conclusion 3.4: Most institutional stakeholders agreed that the CRW project training was appropriate and responsive to institutional needs, though the evaluation found that both baseline and ongoing needs were mainly assessed through informal means. [Supported by Findings 3.4, 3.9 and 3.10]  Conclusion 3.5: GOK institutions have committed to meeting the minimal recurring costs associated with the new meteorological practices. [Supported by Findings 3.11 and 3.12]  Conclusion 3.6: The technology and software provided by the CRW and CCRD projects are appropriate, and ongoing maintenance will be minimal and within the capacity of the relevant institutions. [Supported by Findings 3.13]  Conclusion 3.7: Improved practices in the collection, analysis, and dissemination of weather and climate information are likely to be sustained. [Supported by Conclusions 3.1 – 3.6] Final Evaluation Report – Performance Evaluation: CRW Project 28 Evaluation Question 4: To what extent are other institutional stakeholders using Kazhydromet weather and climate information?  Finding 4.1: There is little evidence of systematic data sharing from KHM to other institutional actors including the Agricultural Research Institutes, which appear to prefer their locally-produced guidance.  Finding 4.2: KHM provides medium- and long-term weather forecasts to the Committee of Emergency Situations.  Finding 4.3: KHM provides weather data information to the Farmers Union pursuant to a MOU, for dissemination to its members.  Finding 4.4: Individual beneficiaries of the CRW-supported trainings and field days, representing selected institutional stakeholders, indicated use of and reliance on KHM seasonal and drought forecasts. EQ 4 focuses on institutions outside of the CRW project’s key institutional partners and targeted direct institutional beneficiaries such as MinAg, KHM and NSRI. Since CRW project objectives include increasing the quality of KHM agrometeorological data and the dissemination of such data to farmers, it should not be surprising that there is little evidence of the project’s work on increasing data use among Kazakh institutions. Except for MinAg’s uses of KHM data that were discussed under EQ 2, the evaluation team found that little data and information are freely shared among the relevant GOK institutions, including KHM which is widely considered the official source for meteorological data.60 Almost half of the telephone survey respondents (35 out of 77) were representatives of these institutions, including but not limited to the research institutes, government agencies, and private organizations. When prompted, 93 percent of these institutional respondents indicated using KHM data: 56 percent cited using seasonal forecasts in their work and 37 percent indicated using information on both seasonal and drought forecasts. Additionally, when asked about their reliance on KHM data, 71 percent of respondents indicated modest and 20 percent indicated heavy reliance on KHM data. However, when asked about sources of data in an open-ended question, only two respondents stated that they received agro-meteorological data directly from KHM, while seminars, for example, were noted by five respondents. This may indicate a lack of understanding of KHM as being the actual source of data and but also demonstrates the lack of a systematic process of institutional information sharing. Interviews with representatives of the Shortandy and Kostanay research institutes revealed that these institutes produce their own local climate analyses and statistics-based seasonal climate forecasts based upon local meteorological observation equipment. Neither of these 60 A singular exception to this may be the use of surface-based soil moisture sensors (operated by KHM) to calibrate satellite￾based sensors (used by NSRI). Members of the evaluation team visit the Department of Long￾Term Forecasts in KHM-Astana. Credit: Svetlana Negroustoueva, dTS. Final Evaluation Report – Performance Evaluation: CRW Project 29 institutions access KHM large-scale forecast climate model guidance. There was also some evidence that these institutes may prefer their locally-produced guidance to that of KHM, given cost considerations and legal issues: “We [at the Research Institute] conclude a contract with local KHM office [that belongs to the Ministry of Energy] to provide us with precipitation and cumulative temperature forecasts. However, this approach does not work. We [the Institute] sign a contract as a legal entity, and MinAg has a separate KHM contract, so the contract is signed twice”61 In discussions with a host of other actors, including the Institute of Geography, KazAgroMarketing, the Organic Farming Public Association, and the Biological Diversity Public Fund, there was no evidence that any of these actors use weather and climate information developed by KHM in their work. The only identified instances of use and reliance on KHM weather and climate data by other institutional actors (excluding the uses discussed with respect to EQ2) are:  The Farmers Union ostensibly offers its members access to KHM seasonal climate data, pursuant to the MOU signed with KHM (see EQ2). According to a KHM staff member, the first example of cooperation between these two organizations occurred in the spring of 2015.62  KHM works with the Committee of Emergency Situations in the field of risk assessment, monitoring potential adverse weather, early warning, and forecasting of severe weather, with support from another UNDP project. Svetlana Dolgikh from KHM stated, “KHM works closely with the Committee of Emergency Situations at national and regional levels. As long as we know about emergency coming we contact the Committee. Each oblast has its own passport, which is regularly updated; it contains information about all possible climate, hydrological and meteorological risks. KHM works with the Committee to develop the passports and the Atlas of Emergency Situations (one map for each situation. However, drought is not mentioned as a risk category, but flood is).” 63 The Deputy Head of KHM also stated that the institution’s data are used by other non-institutional actors in Kazakhstan: “Our monitoring data and long-term forecasts are used by insurance companies, large agro producers in regions.”64 61 Interview at the Kostanay Research Institute, July 30, 2015 62 The CRW project September 2014 report to USAID lists this as achievement #18 of the project. The evaluation team could not identify the source of this discrepancy. 63 Interview on July 24, 2015 64 Interview on July 21, 2015 Conclusions:  Conclusion 4.1: KHM is acknowledged as the official source for meteorological information and its observational data, analyses, and short-term forecasts are used by other institutional actors. [Supported by Findings 4.1, 4.2, 4.3, and 4.4]  Conclusion 4.2: There is little apparent systematic sharing or use of KHM medium- and long-term climate forecasts by other Kazakh institutional actors. [Supported by Findings 4.1, 4.2, and 4.3] Final Evaluation Report – Performance Evaluation: CRW Project 30 Evaluation Question 5: To what extent are farmers basing their decisions on Kazhydromet weather and climate information? Why or why not? The evaluation team’s discussions with farmers and agro-producers highlighted that each farm is different in terms of soil structure, management style, staffing mechanisms, seed types, level of production, and diversification of production. According to the farmer respondents, seasonal forecasts are significant factors in planning for the agricultural year. Traditionally, farmers have relied upon short￾term (1-, 3-, and 10-day) forecasts, long-term seasonal forecasts, and retrospective monitoring. These forecasts influence decisions on the area of risk (rainfed) agriculture, the arrangement of culture, the selection precursor65, the level of tillage, the selection of seeds and seeding rate, and a host of other considerations. Each of these considerations directly or indirectly depends on the season (see Table 6).66 TABLE 6: AGROMETEOROLOGICAL OBSERVATION BY SEASON Season Observed Variable Use in Decision-making Spring  Soil temperature for planning the leading agricultural crop  Topsoil moisture  Selecting time to plant  Selecting seeds and seeding rate Summer  Onset of the main phases of plant development and their condition  Height and thickness of the haulm strand  Productivity elements  Soil moisture: productive moisture in the different soil levels, moisture of agricultural crops  Tillage techniques to be used  Use and selection of fertilizers  Selecting time to harvest Fall  Topsoil moisture  Use and selection of fertilizers  Selecting time to harvest Winter  Height and nature of the snow cover  Selection and preparation of seeds  Selection precursors General  Damage to crops by adverse weather phenomena  Insurance recovery Table 7 summarizes the types of information provided by KHM according to the three types of weather and climate data used by farmers and agro-producers. 65 In areas where cereal monoculture is prevalent and famers are unable to afford high prices of chemical fertilizers and pesticides, crop rotation method is used to reduce crop pests and improve soil fertility and hence productivity. Crop rotation means changing the type of crop grown on a particular piece of land from year to year. It is primarily a management decision based on a desire to optimize financial, agricultural or environmental objectives through profit and yield maximizations as well as through minimized pesticide use (Castellazzi et al., 2008). A leguminous crop usually precedes cereals for the aim of improving soil fertility. Therefore, the benefits of rotations could arise from increased nitrogen supply, soil organic matter, and improvement in soil structure, and decreased pests, disease or weed competition. Hence, the choice of appropriate precursor crop to wheat planting for rotation can affect wheat yield. Journal of Economics and Sustainable Development www.iiste.org; Vol.5, No.3, 2014. 66 IRG Mid-Level Assessment: Climate Forecasting in Kazakhstan, October 2014 Final Evaluation Report – Performance Evaluation: CRW Project 31 TABLE 7: METEOROLOGICAL INFORMATION USED BY AGRO-PRODUCERS Forecast Characteristic Short-Term Forecasts Medium and Long￾Term Projections Retrospective (Monitoring) The most important parameters for farmers  Temperature & rainfall primarily  Wind speed and direction and atmospheric pressure secondarily Periods of rain and drought Temperature and volume of rainfall by the days and terms Sources of information on projections Varied - RSE "Kazhydromet" and its regional/local weather stations, online resources, regional research institutes, TV, periodicals, etc. Limited - RSE "Kazhydromet", meeting at regional and local departments of MinAg Varied - RSE "Kazhydromet" and its regional/local weather station, regional research institutes, personal data Accessibility and availability of information on weather and climate  Area-specific and detailed forecasts can be obtained for a fee  Farmers can sometimes obtain specific forecasts without payment through personal relationships  General area forecasts are available for free  Farmers can only get from an official source (fee based) or through meetings at local akimats;  The availability of this information is limited  Farmers can get this information from official sources (paid), or from the personal archive records;  Access may be bounded or free, depending on the source Quality of data and confidence  Farmers consider this type of forecast to be the most accurate  Degree of confidence – high  Farmers do not believe this type of forecasts to be accurate  Degree of confidence - low (farmers often rely on personal observations, experience and self-measurement)  Farmers believe this type of forecasts is accurate  Degree of confidence – relatively high Data characteristic Short-term forecasts - zonal and territorially bound Long-term projections – are not zonal and geographically bound This forecast is zonal and territorially bound Type of work  Planting  Harvest  Seasonal work  Planting  Harvest  Planning (winter break)  Planting  Harvest Agro-Producer Perceptions of KHM Forecasts  Finding 5.1: The majority of farmers believe that KHM forecasting is somewhat reliable, although a significant minority find it to not be reliable at all.  Finding 5.2: A significant majority of farmers rely on KHM forecasting to at least a moderate extent, although it is only one source of information utilized.  Finding 5.3: A bare majority of farmers believe that KHM forecasts have become more reliable over the preceding two years. Final Evaluation Report – Performance Evaluation: CRW Project 32 The telephone survey conducted for this evaluation of farmers and agro-producers who had participated in CRW-promoted training activities (n=42) included questions on their perceptions and use of KHM seasonal climate forecasts. Although the survey questions were most immediately focused on seasonal climate and drought forecasts, a review of the survey responses and comparison of quantitative and qualitative evidence suggest that these questions were likely interpreted to refer to all of KHM’s weather and climate forecasts. Therefore, general findings are presented here while a more detailed discussion of their use and reliance upon different types of (short-, medium- and long-term) forecasts is subsequently presented and primarily based on the qualitative evidence. In the telephone survey, farmers and agro-producers were asked to provide an assessment of the reliability of KHM seasonal climate forecasts. As shown in Figure 2, a clear majority (74 percent) of respondents believed KHM forecasts to be somewhat reliable, while a significant minority of respondents (22 percent) felt that the forecasts were not reliable at all. Interestingly, this finding from the telephone survey (which was based on a relatively small sample) is not consistent with qualitative evidence collected by the evaluation team, which identified a sharp distinction between perceptions of the reliability of KHM short-term forecasts (which were generally considered to be reliable) and medium- and long-term forecasts (which were generally believed to be unreliable). Again, this suggests that many or most telephone survey respondents did not interpret the question to distinguish seasonal climate forecasts from short-term weather forecasts produced by KHM. When questioned about levels of reliance on KHM forecasts, the significant majority (84 percent) of farmers and agro-producer respondents to the telephone survey indicated that they did rely to some extent on KHM weather and climate forecast information (see Figure 3). Twenty-four percent of these respondents reported heavy reliance on KHM data, while 60 percent indicated that they relied on KHM forecasts to some extent, suggesting a reliance on other forecasting methods as well – a finding also borne out by the qualitative data collected. Few respondents indicated that they did not rely on KHM forecasts to any degree. FIGURE 2: FARMERS’ ASSESSMENT OF THE RELIABILITY OF KHM FORECASTS FIGURE 3: FARMER RELIANCE ON KHM FORECASTS 2% 74% 22% 2% Don't know Somewhat reliable Unreliable Very Reliable n=42 Final Evaluation Report – Performance Evaluation: CRW Project 33 Notably, as shown in Figure 4, slightly over half of these telephone survey respondents indicated that the reliability of KHM forecasts had improved in the last two years, although a significant share of respondents had not seen any changes or believed KHM’s forecasts had become less reliable. A further discussion of the findings with respect to KHM weather and climate information and the reasons for reliance (or lack of reliance) upon KHM information is broken down in the rest of this section by the type forecast (short-term and medium/long-term) and monitoring information, as discussed with the farmers. KHM Short-Term Weather Forecasts  Finding 5.4: Farmers use short-term weather forecasts for defining the time for different field work, including planting and harvesting.  Finding 5.5: Farmers use multiple sources of information for short-term forecasts, including internet sources, local weather stations, and their historical experience.  Finding 5.6: Farmers obtain access to KHM short-term weather forecasts from the KHM website, where the forecasts are provided for free.  Finding 5.7: Farmers generally trust and have confidence in the accuracy of the KHM short-term forecasts. Farmers rely on short-term weather forecasts to obtain temperature and precipitation data to make decisions related to planting and harvesting during the spring – fall period. Examples of the role of these forecasts include:  Farmers closely monitor temperature, rainfall, and soil moisture to make decisions about crops, tillage, fertilization, seed and seedbed preparation, and sowing during the planting period in May.  Decisions relating to pests and infestation are important for mechanical and chemical care of wheat and take place during the summer.  Mechanical and chemical processing must finish 10 to 20 days before the harvest (August￾September or early October) and are possible only under certain weather conditions (i.e., dry, rain, no wind)  Yield collection must take place before rainfall starts. Farmers may obtain short-term weather forecasts from a variety of sources including television, the Internet (including the KHM website), regional/local weather stations, written recommendations of regional research institutes, periodicals, and other media. One- and three-day KHM forecasts are FIGURE 4: FARMERS’ PERCEPTIONS OF WHETHER KHM FORECASTS HAVE IMPROVED IN THE LAST TWO YEARS n=42 Final Evaluation Report – Performance Evaluation: CRW Project 34 provided for free and farmers obtain these through the KHM website. The ten-day forecast is not free, but some farmers can obtain these through personal relationships at their local KHM office. Two-thirds of the farmers (12 out of 18) who the evaluation team interviewed accessed meteorological data and information from Internet sites; 4 of these 12 farmers noted that they primarily relied on KHM websites, while the remaining 8 used multiple Internet sites (e.g., KHM, GISmeteo). While this was not examined in detail, it would appear that most data and information on these other websites are data that originate in some form from KHM and the worldwide meteorological network system, although most farmers interviewed do not understand or distinguish between such sources. The KHM website provides two different versions of the 3-day short-term forecasts, and farmers generally rely on the more descriptive version that provides information about the different regions of Kazakhstan and additional forecast information (e.g., wind direction). In addition, a couple of the farmers interviewed were only accessing KHM short-term information through traditional broadcast media methods. Thus, a total of 14 out of 18 farmers from the individual or group interviews stated that they do rely on KHM short-term forecasts. The reasons that farmers cited for using KHM short-term forecasts included: they are zone bound, easily accessible, and the data are generally free through broadcast media or Internet sources.67 Farmers generally stated that they have a very high-degree of confidence (characterized by a level of trust between 80 and 100 percent) in the short-term forecasts.  “We trust data on the current weather status – this in fact translates for us into a short-term weather forecast, as we are not 100 percent relying on the Internet-resources - they cannot give you exactly accurate forecast even for the coming week, we take only forecasts for three days to come.”  “I trust Internet and my own observations… Let's say 80 percent of short-term forecasts from the Internet prove to be correct”  “We trust and use short-term forecasts.” The four farmers interviewed who do not use KHM short-term forecasts stated that they have their own meteo-stations, use barometers, and rely on their own experience (e.g., clouds pulling together is considered a sign that there will be rain in three days). KHM Medium and Long-term Climate Forecasts68  Finding 5.8: Farmers use medium- and long-term seasonal climate forecasts to determine the optimal time for planting and harvesting.  Finding 5.9: KHM medium- and long-term seasonal climate forecasts are only available for free through meetings at the oblast akimats, including in the form of bulletins, which are also distributed through Union of Farmers.  Finding 5.10: KHM forecasts cover very large areas (i.e., south, north, east, and west of the country) and are not sufficiently zone bound69 or specific 67 Interviews and FGD with farmers, July 23 to 31, 2015 68 Although not directly responsive to this EQ, the evaluation team also asked farmers about their use and assessment of the reliability of drought forecasts (primarily produced and disseminated by NSRI). Seventy-five percent of respondents thought these forecasts were somewhat or very reliable, and 44 percent felt these forecasts had improved in the last 2 years – as opposed to 31 percent who thought their reliability unchanged and 25 percent who believed they had become less reliable. 69 The terms “zonal bound,” “‘territorially bound,” and “geographically bound” arose from the interviews. These terms refer to geographic area for which a forecast is considered accurate and reliable. This is an especially relevant consideration for Final Evaluation Report – Performance Evaluation: CRW Project 35  Finding 5.11: Some large-scale farmers pay for KHM medium- and long-term forecasts but small-scale farmers cannot or will not bear the cost.  Finding 5.12: Farmers expressed generally negative opinions about the reliability and accuracy of the KHM long-term forecasts and most do not believe they have improved. Farmers use medium- and long-term seasonal climate forecasts to determine the best times for planting and harvesting. The most important parameters are the periods of rain and drought. Based upon this information, farmers select seed type and tillage approach, and prepare equipment and mechanisms for planting or harvesting. Medium- and long-term forecasts are used for planning agricultural work like planting, harvesting, and mechanical and chemical care, while short-time forecasts determine the specific days and times for implementing these plans.  KHM medium-term forecasts cover a one-month period and describe in one or two sentences forecasted changes in temperature, precipitation, and wind direction for different regions of Kazakhstan.  KHM long-term forecasts cover specific seasons (e.g., winter, summer) and include temperature, precipitation, and wind direction tendencies for selected parts of the country by month. KHM medium and long-term forecasts are obtained from the KHM website or its national or regional offices. The weather and climate information available on the KHM website is not suitable for making informed agricultural decisions as it is too general – covering either one of the four major regions of the country or regional centers and providing a wide range of potential temperature and precipitation outcomes. Moreover, the information that is provided is more accurately described as situation monitoring (i.e., providing historical monitoring information from previous seasons) than forecasting. However, farmers can pay to receive specific forecasts for their areas from KHM. Only a minority of the farmers the evaluation team spoke with (5 of 18) stated that they would be willing to pay for KHM medium- and long-term forecasts. Farmers can obtain KHM medium- and long-term forecasts from oblast akimat planting/harvesting meetings and ostensibly the MinAg bulletin. Meetings in oblast akimats are scheduled before planting time and harvest, where representatives from local KHM divisions present seasonal and long-term forecasts. All farmers are invited to these meetings, although most participants are representatives from medium and large businesses. Farmers use this opportunity to ask questions and compare the forecasts with their weather monitoring observations and/or data from their meteo-stations. Even where farmers purchase KHM forecasts, they still triangulate it with other data: “We have our own analytical unit that works with data from Kazhydromet and our weather station. We also have weather records for the last 50 years. Every day I find data on temperatures and precipitation on my table. We get it from the state. We analyze everything.”70 At the time of this evaluation, MinAg claimed that the bulletin was being disseminated to farmers. While none of the respondents from two regions at the individual and group interviews could recall even having heard of such bulletins, findings from the telephone survey indicated that six respondents have used the bulletins for obtaining either weather and climate information or information on adaptation techniques (which will be discussed later).71,72 northern Kazakhstan, which features many micro-climatic zones for which area-specific and more detailed forecasts are required. 70 Interview with farmer, July 27, 2015. 71 Interviews with key informants and farmers in Kostanay, Shortandy from July 23 to 31, 2015. 72 Telephone survey of farmers, August 2015. Final Evaluation Report – Performance Evaluation: CRW Project 36 The majority of farmers from the interviews and FGD (14 out of 18) professed to not trust or rely on the KHM medium- and long-term forecasts, and provided three main reasons. First, long-term KHM forecasts cover very large areas (i.e., south, north, east and west of the country) and are not zone bound and specific. Even for large-scale farmers with 20,000 to 40,000 hectares, forecasts that cover a quarter of the country are too general. Second, farmers reported that medium- and long-term KHM forecasts are too expensive. Small-scale farmers either cannot or will not pay for the forecasts: “We don't use Kazhydromet data – this service is way too expensive for us.”73 Those farmers that did pay for the KHM forecast data represented medium and large farms (2,000 to 40,000 hectares). Even these farmers did not rely fully on KHM forecasts, and obtained additional data from the akimats, their own weather stations, and even NSRI:  “We don't trust Kazhydromet data, but check weather forecast on the Internet. We have our own weather station – for our fields this indeed provides trustworthy information.”74  “We cooperate with the NSRI since 2003. We have their data along with long-term weather forecast for a year ahead.”75 The third reason cited by farmers was that KHM weather forecasts are inaccurate and of low quality, with 14 of the 18 farmers reporting that they did not recognize any recent improvements in the forecasts:  “Quality has not improved, but the information has been commercialized. Ten years ago these forecasts were more accurate. Now they are making mistakes.” 76  “We do not trust long-term weather forecasts, the quality level has not changed in two years' time.”77 One farmer disagreed with the general assessment and believed that KHM “started improving its long￾term forecasts in terms of their accuracy recently.”78 Finally, three farmers mentioned that because of climate change, it is quite difficult to trust any forecasts and recommendations based on historical monitoring data, and they did not distinguish between KHM and other sources.79 Institutional stakeholders80 expressed that farmers are conservative in approaches and changes will have to be heavily and actively promoted, and the use of new dissemination techniques should be considered. In this vein, one farmer stated: “I wouldn't mind if, for example, I could get weather forecasts or newsletters in texts on my phone. So far I do nothing to try and get this information.”81 Farmers appeared to be more likely to rely on local Research Institutes,82 each of which develops and publishes recommendations for farmers. These recommendations are based on MinAg materials and 73 Interview with farmer, July 24, 2015 74 Interview with farmer, July 31, 3015 75 Interview with farmer, July 27, 2015 76 Interview with farmers, July 29, 2015 77 Idem. 78 Interview with farmer, July 24, 2015 79 FGD Shortandy, July 27, 2015 80 Interviews in Shortandy and Kostanay, July 23 to 27, 2015 81 Interview with farmer, July 29, 2015 82 KazAgroInnovation includes 23 research institutions, 26 affiliates of research institutes and 10 extension centers. Each research institute has a specialization. For example, Barayev research institute coordinates research projects on the development of soil conservation farming systems for growing crops in all grain-producing regions of the country through improving regional systems of conservation agriculture, enhancing soil fertility, integrating mechanization in cultivation of crops, land reclamation, the development of alkaline and unproductive land, the development of new high-yield, high-quality varieties and hybrids of seed production, as well as the development and introduction of technologies of their cultivation, ensuring high Final Evaluation Report – Performance Evaluation: CRW Project 37 consider the forecasts of the local KHM divisions and their own meteo-stations, as well as short-term weather data. The Research Institutes’ recommendations are oblast-specific as well as being easy to understand, user-friendly, and come in a pocket-size format valued by farmers: “We use Research Institute recommendations for our region….there is enough information.” 83 Representatives from these local Research Institutes expressed strong opinions on the type and quality of KHM data:  "KHM forecasts are only recommendatory in nature and don’t contain any in-depth analysis. Therefore, every spring we participate in the region meetings devoted to the planting/harvesting campaign where we explain what the current weather conditions and weather forecasts mean from a practical point of view."84  “KHM produces the general weather forecast. In contrast, we make specific recommendations for the technology of cultivation of wheat. We also give advice on optimal planting time.”85 Although farmers are more willing to rely on information from the Research Institutes (which do themselves consider and incorporate KHM data with other sources), they largely do not rely directly or fully on KHM data for agricultural decision-making, but will use the data to receive compensation for weather-induced crop damage by insurance companies. However, the evaluation team found evidence that in many cases insurance companies will not pay out – possibly as a result of the long distances between meteo-stations that do not provide data specific enough to confirm conditions on large farmers’ fields. The CRW project planned to review insurance criteria and mechanisms allowing farmers to get insurance in case of disaster.86 However, as was acknowledged by the a UNDP Chief Technical Advisor during a presentation to the evaluation team,87 the CRW project so far has not succeeded in establishing contacts with the national partner Agency of Financial Enforcement/Control. productivity of crops with the least expenditure of labor and resources. Extension centers are specialized scientific and technical assistance entities that disseminate advanced agricultural knowledge and technologies through educational programs. These centers are established on the basis of the research institutions and their lecturers consist of domestic and foreign experts, scholars, leading scientists of research institutes, university professors, etc. Lecturers and participants concurred that the agricultural extension service helps professionals to improve their knowledge and gain agricultural best practices and technologies through practical and educational activities that contribute significantly to higher production efficiency. 83 Interviews with farmers, July 23 to 31, 2015. 84 Interview with Kanat Akshalov, Barayev Research Institute for Grain (Shortandy), July 24, 2015. 85 Interview with Vladimir Skoblikov, Barayev Research Institute for Grain (Shortandy), July 24, 2015. 86 CRW Project Description for Proposed Extension, 2014. 87 July 20, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 38 Conclusions:  Conclusion 5.1: Farmers trust and rely upon KHM short-term forecasts in agricultural decision-making. [Supported by Findings 5.6 and 5.7]  Conclusion 5.2: Farmers believe that KHM medium- and long-term forecasts are not sufficiently specific as to location to be useful in agricultural decision-making. [Supported by Finding 5.10]  Conclusion 5.3: While medium and large farms can afford to pay for KHM medium- and long￾term forecasts, smaller farmers view the cost as prohibitive. [Supported by Finding 5.11]  Conclusion 5.4: Farmers do not generally view the KHM medium- and long-term forecasts to be reliable. [Supported by Finding 5.12]  Conclusion 5.5: Farmers do not generally rely on KHM medium- and long-term forecasts. [Supported by Conclusions 5.5, 5.6 and 5.7 and Findings 5.10, 5.11 and 5.12]  Conclusion 5.6: Although farmers believe that overall KHM forecasting has improved over the last two years, the evidence suggests that farmers perceive that short-term forecasts have improved and that medium- and long-term forecasts largely have not. [Supported by Findings 5.3, 5.7 and 5.12] Final Evaluation Report – Performance Evaluation: CRW Project 39 Evaluation Question 6: To what extent have farmers adopted climate change adaptation techniques promoted by the demonstration plots and through CRW-supported education initiatives (such as publications and circulars)? Why/Why not? If adopted, from which initiative?  Finding 6.1: Less than one-third of farmer respondents to the telephone survey indicated that the Agro-Centers through which the CRW project promoted trainings were delivered and adaptation practices showcased were a source of information for new agricultural techniques – less than the number who relied on internet sources. The CRW project’s second component focused upon mainstreaming climate resilience into agricultural techniques. The associated activities included:  Revising the learning curriculum of the agricultural extension agencies;  Setting up pilot demonstration plots to showcase advanced adaptive wheat growing technologies resilient to climate shocks; and  Conducting analysis for informed decisions and raising the awareness of national and local stakeholders, including farmers, about new agricultural technologies. It was expected that through different CRW project activities, the following results would be achieved:88  Skills and knowledge of institutional stakeholders at local and national levels on new adaptive wheat growing technologies will be improved;  Farmers received knowledge on effective climate resilient agriculture practices. Under agreement with MinAg, the CRW project implemented these activities in partnership with KAI, which is represented at the local level by agricultural extension centers and research institutes (referred to here as Agro-centers) selected through a competitive process from the three main wheat-growing regions of Kazakhstan. Agro-centers conduct practical training, seminars, consultations, and online advisory sessions with farmers, in addition to sharing printed materials. Interestingly, the telephone survey indicated that Agro-centers were not the main source of information on climate change adaptation techniques for farmers and agro-producers: only 12 out of 42 respondents use Agro-centers as a source of information on new adaptation techniques for wheat production. The other sources of information mentioned by farmers included: the Internet (15 respondents), newspapers (6), MinAg bulletin (4), and akimat meetings (2). Radio, television, other farmers, and conferences each received one mention. Findings are presented below according to the nature of the knowledge dissemination delivered through the Agro-centers (i.e., training, the publication of research and learning materials, and demonstration plots). Training/Training of Trainers (TOT)  Finding 6.2: The CRW project focused on raising awareness of climate change and promoted climate change adaptation techniques through training events, cascade trainings, pilot demonstration plots, publications, and the use of social media. 88 CRW project Performance Monitoring Plan, 2013. Final Evaluation Report – Performance Evaluation: CRW Project 40  Finding 6.3: Approximately 700 farmers received the CRW-sponsored climate-change adaptation training module.  Finding 6.4: Although approximately half of the farmers who participated in CRW￾sponsored climate-change adaptation training could not recall attending, most of those farmers who could recall attending viewed the trainings positively, believed it helped them understand climate change risks, and subsequently adopted some adaptation practice (fully or partially). The vulnerability of the Kazakhstan wheat sector to climate change and the resulting need to mainstream climate-resilient adaptation measures were recognized by CRW project designers, but only to a limited extent by key stakeholders and beneficiaries – especially farmers. To increase awareness about climate change threats, the CRW project provided technical assistance to Agro-centers in three regions to revise the training curriculum on improving crop cultivation and production. The training provided by the Agro-centers are generally conducted by specialists from the Agro-centers and include theoretical, practical, and demonstration components. In order to sensitize Agro-center training participants about climate resilience, the CRW project developed a four-hour component for inclusion in the standard training. This session focused upon the challenges facing the wheat sector as a result of climate change, accessing climate information, and options for addressing climate vulnerability. To deliver the training, the CRW project – with support from CCRD – conducted two Training of Trainers (TOT) sessions89 in 2014 for approximately 30 trainers from Kazakhstan and other Central Asian countries. The first TOT round in February 2014 initiated a six-day program that included the first stakeholder workshop by Kazakh trainers conducted at the beginning of the CRW project. The first day and a half of the TOT session provided background on climate variability and change, described USAID’s Climate-Resilient Development Framework, and helped trainers design stakeholder workshops to elicit inputs from participants to address climate and non-climate challenges facing farmers and the sector, and identify and prioritize options for overcoming these challenges. The second TOT session, dedicated to the issues of effective farm-level adaptation strategy, was conducted in Almaty from November 11 to 14, 2014. Participants included representative of three extension centers, including a KAI representative and five representatives from other Central Asian countries. Since the first TOT session, the Agro-centers have conducted eight follow-up trainings in Kazakhstan and two regional trainings in Tajikistan. These trainings were conducted based on the TOT sessions, and certain modules were adapted from TOT for other training programs that were conducted independently. The local trainers delivered workshops to farmers, agronomists, and other technical specialists under the supervision of CCRD for agro-producers. Interviews with CCRD staff involved with the training program indicated that nearly 400 farmers were trained in Shortandy and Kostanay shortly following the February 2014 ToT session.90 However, the evaluation team’s review of the monitoring data and interviews with Agro-center staff found that no more than 300 farmers (out of approximately 15,000 in the three target regions) have participated in training sessions, primarily representatives of large- and medium-sized farms (60 and 24 percent, respectively, of the telephone survey respondents).91 The evaluation team cannot explain this discrepancy, although it may be the case that CRW project staff did not consider and include the earlier trainings as CRW-promoted trainings, which would suggest that the total number of farmers trained is approximately 700. 90 Interview with Glen Anderson, CCRD Chief of Party, April 1, 2015. 91 CRW monitoring data and interviews with Agro-center staff on July 23 and July 28, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 41 Out of the 77 participants of the training and other CRW capacity building activities who responded to the phone survey, 55 affirmed that they had participated in a training, workshop, or seminar focused on climate change adaptation. Three-quarters of these respondents (41) indicated that they had a better understanding of the risks posed by climate change as result, and 38 out of the 55 said they had gained a better understanding of what they can do to mitigate the impacts of climate change. Among farmers, only 22 of the 42 respondents who were identified through participant lists as having attended a training, workshop, or seminar focused on climate change adaptation could affirm having participated in a training. All 22 of them stated that they had acquired a better understanding of climate change risks, and 17 acknowledged having adopted new agriculture approaches based on information received during the training. The most popular new approaches cited included adoption of new planting times and a new approach to tilling (7 answers each), and initiating use of alternative fertilizers and/or seed varieties (11 answers). Other approaches adopted that were less frequently cited included crop diversification and the use of alternative fertilizers. Nearly every farmer interviewed or surveyed as part of this evaluation who could recall attending the trainings expressed the view that the training itself was good and the training materials were useful in terms of understanding climate change-related challenges, different climate adaptation techniques, and local specifics. Institutional actors and most farmers believed that the training sessions allowed farmers to gain a better comprehension of the connection between climate change and extreme weather conditions that affect their crops.92 The majority of farmers interviewed indicated that the training helped them make this link. However, there were divergent views on the issue of relevance. As one respondent remarked: “The climate change subject is not exactly a popular matter for discussion, as it is vague, there is nothing clear about it, and it does not have vital influence on our lives. Moreover, it is regarded as something of a far-away nature that has nothing to do with the local situation. But it was on offer for the participants, and they decided to go for it. The climate change was deemed to matter in longer time frames, whereas seasonal weather forecasts were considered more important.”93 The evaluation team noted that the CRW project does not have a formal process for following up with training participants to assess the success of the training or the utilization of the knowledge gained. Publications/Videos  Finding 6.5: The CRW project published studies and media focused on climate change variability and resilience, but 75 percent of these materials targeted policy makers.  Finding 6.6: There was no evidence that published materials and media targeting farmers influenced the adoption of adaptation techniques. The CRW project published studies, recommendations, and documents registering lessons learned, the aim of which were to provide better information for new agricultural policies related to crop simulations, new climate-resilient agricultural technologies, and measures to address the adverse impacts of climate change (Annex D includes a list of publications provided by the CRW project). The stated target audience for a majority of the publications (12 out of 16) were “all project partners and other interested counterparts.” The evaluation team’s analysis found that the main audiences for these publications were policy makers and government authorities, local experts, and CRW project partners, although a small selection of publications and media (25 percent) were relevant to farmers. 92 Interviews with Stanislav Kim, UNDP and Ashley King, USAID/CAR 93 KII, Shortandy, July 27, 2015 Final Evaluation Report – Performance Evaluation: CRW Project 42 A Facebook webpage was launched for dissemination of the CRW best practices of agricultural adaptation techniques.94 This website is regularly updated with information about conferences and CRW partner news, and notices for upcoming events in Kazakhstan and abroad including CRW field days and references to analyses related to the wheat sector. However, the website actually features few CRW best practices.95 The evaluation team was not able to assess the level of activity on the website, although there were no responses to an open call for participation in evaluation interviews and the telephone survey. The CRW project planned to publish a brochure on innovative approaches for climate resilient wheat production in Kazakhstan, but the evaluation team could not confirm that this brochure was actually published. The CRW project, with CCRD supervision and support, planned to produce three videos to target: (1) staff from USAID and UNDP, their partners, and international practitioners in climate adaptation, agriculture, and security; (2) the GoK, businesses, farmers, and the general public to promote climate￾resilient development of the Kazakh wheat sector; and (3) farmers who participate in training at the extension centers. However, only one video (“Kazakhstan World of Wheat”), which addressed the importance of climate change and enumerated the goals of the project, was mentioned as being produced.96 The video also provided some educational information about technologies important for increasing the climate resilience of the wheat sector. According to the CRW project, the video targeted the general public and was disseminated through the KAI network, national television network, and was shown at the TOT events, but it was not mentioned in any of the evaluation team’s discussions with farmers. Demonstration Plots  Finding 6.7: The total number of farmers who visited the pilot demonstration plots has not been monitored and cannot be determined.  Finding 6.8: The number of farmers who visited the pilot demonstration plots during CRW-supported field days is consistent with CRW project targets, but low relative to the overall population.  Finding 6.9: Famers who attended the demonstration plots found them interesting and informative, and there is quantitative and qualitative evidence of adoption.  Finding 6.10: Where farmers did adopt adaptation techniques, they generally did so partially and without fidelity to the implementation guidelines.  Finding 6.11: When farmers expressed reluctance about adopting adaptation techniques, it was due to concerns about their applicability, cost, and inadequate expertise.  Finding 6.12: Farmers are reluctant to adopt new technologies without evidence that the technology has been applied successfully on farms in the same geographic area.  Finding 6.13: Due to regional micro-climates, farmer respondents were not fully convinced that the showcased adaptation results were replicable on their farms. 94 www/facebook.com/altynalkap 95 The CRW project prepared a publication that provides a detailed explanation of 22 documented best practices, including conventional practices being applied in Kazakhstan and other Central Asian nations. It also describes basic agro-technological means broken down to the specific seasons when farmers need to apply particular technologies. 96 Glen Anderson, Trip Report, March 4, 2014 Final Evaluation Report – Performance Evaluation: CRW Project 43  Finding 6.14: Farmers on medium and large farms express an interest in, or have focused upon, diversification of crops – one of the recommendations for mitigation of climate risk espoused by CRW-supported activities. The CRW project supported the establishment of three demonstration plots (one in each of the three regions of northern Kazakhstan) to showcase climate-resilient adaptive wheat growing technologies to local farmers and decision makers. Following a feasibility study in three northern regions of Kazakhstan, it was decided to select three areas with different zones: the forest steppe, steppe, and dry steppe. A total of 22 adaptation techniques were demonstrated in the three regions, although not all 22 techniques were demonstrated in any one region. The following categories of adaptation techniques were demonstrated: 1. Akmola region: Cereal crop diversification methods were applied using different cropping and rotation technologies with different sowing periods of the spring wheat crops (May 1 to June 10), under no tillage and soil conservation technologies. This demonstration plot is set up to determine the best sowing, rotation, tilling, and organic farmer approaches for the agro￾ecological condition of the Akmola region. 2. Kostanay region: Cultivation of spring wheat varieties using no tillage agriculture technological conservation methods as well as cultivation of the winter cereal crops. These plots were set up to assess the best minimum tillage approaches and their impact on soil and crop conservation, organic farming, and economic efficiency or wheat growing. This method also showcases the best relationship during rotation of the cereals with cereals of other family lines, such as triticale and rye, in order to improve the fodder cropping and soil conservation technology. 3. Northern-Kazakhstan region: The experimental works consist of determining optimal sowing periods for the spring varieties of the wheat crops and external root fertilization using nitrogen and micronutrients during different vegetative periods. This pilot plot was established to determine a best sowing period to the agro-ecological conditions of the Northern￾Kazakhstan region taking into consideration the shifting winter season, increased snow pack, and conservation of soil moisture through multi-cropping diversification and improving of soil fertility. Demonstration plots served as a prime tool for the dissemination of information on wheat planting dates, zero tillage technology, and resilience of wheat production in the three regions. Demonstration plots in Kazakhstan (and in Tajikistan with the focus on all Central Asian countries) were quite open to the farmers, and the terms of reference for these pilot plots were designed in a way that all the field activities were carried out by the farmers through the “learning by doing” approach, with scientists supervising the overall activities and the farmers implementing. The CRW project supported the demonstration plots by, among other things, reimbursing travel costs for participants, funding the production of visibility materials, and promoting project-specific work of specialists from local research institutes. Once per year, the CRW project supported a field day conducted at each project location. These field days were an opportunity for farmers to visit and review the CRW-promoted pilot demonstration plots. However, the pilot plots were open to the public year-round. Although early in the project, CRW estimated that 111 farmers97 (out of approximately 15,000 farmers in the three regions) would visit the 97 “IR.2.2: Developed [by 9/14] the knowledge of 111 farmers through implementation of effective climate resilient agriculture practices” CRW PMEP, 2013. Final Evaluation Report – Performance Evaluation: CRW Project 44 demonstration plots during Phase I of implementation, it is not possible to determine how many farmers actually visited sites since site visits throughout the year were not recorded.98,99 Among telephone survey respondents, 26 out of 42 surveyed farmers confirmed that they had visited the pilot demonstration plots, and 80 percent of these farmers stated that they learned addition information about what to do in order prepare for and address the impacts of climate change – although approximately 25 percent could not identify a specific approach they had learned. The most commonly cited measures identified were:  Using new equipment and new production technologies (6 answers)  Implementing moisture-keeping techniques (4 answers)  Adopting optimal planting times (5 answers)  Using new kinds of seeds or new information from trusted sources (2 answers each) Both in the survey and in qualitative interviews, farmers stated that they found the demonstration plots interesting and informative. Furthermore, 71 percent (15 of the 21 farmers) acknowledged adopting the climate change adaptation techniques, however selectively, using elements of the 22 adaptive technologies based on the availability of different types of wheat seeds, technical equipment, and main farm activities.100 Based on the qualitative and quantitative evidence collected, the most widely cited adaptation techniques that are being used included:  Use of new fertilizers and fertilizers by ploughing a depth of 25 cm and planting in May;  Wheat cultivation on the plots where snow had accumulated or collected through half straw left on the field;  Use of new kinds of seeds;  Use of oil seeds as predecessors promoted by GOK subsidies; and  Wheat cultivation with minimum and no tillage. The evaluation team’s interviews with farmers found that those who experimented with new approaches and crop diversification – a key recommendation from the TOT sessions and the CRW project – tended to be from large and medium farms and could afford experimentation due to a greater access to capital:  “Every year we add new grain legumes, oil-bearing and forage crops due to diversification of our production. We take seeds from seed-plots of the third reproduction year and keep seeding them till the third reproduction. We are doing everything to update the diversity of our crops in line with the market demands. We are trying to seed new crops, two to three wheat cultivars that differ in terms of seeding and ripening phases.”  “The harvesting capacity depends on the weather (precipitation) and applied technologies. 90 percent of our equipment is brand new. We change crops in line with the rotation and requests from the Ministry of Agriculture, formed based on the consumption demands. In general we are trying to follow all the recommendations. We do have exceptional business.”. 98 While the evaluation team does not have sufficient evidence to state the number of farmers who visited the demonstration plots, a simple extrapolation from this sample would suggest that the CRW project achieved its targets. 99 The CRW project attempted to assess the impact from the overall demonstration plots and field day activities by keeping track of farmers who visited the demonstration sites and replicated results, and instances of on-the-job trainings conducted by farmers and researchers based on the knowledge received after visiting the demonstration plots. See the CRW Catalogue on Best Practices. However, the evaluation team found that there was no follow-up with farmers to assess the extent to which adaptation techniques were adopted, and only limited anecdotal evidence exists. Although the Agro-centers conduct annual surveys of their membership base, the questionnaires are generic in nature and do not provide evidence related to the demonstration plots. 100 Farmer telephone survey, September 2015, QQs. D7, D10, D11 Final Evaluation Report – Performance Evaluation: CRW Project 45  “We seed a lot, and we seed diversified crops. We cannot deal in No-Till technologies, as our climate and equipment does not allow it... After chemical treatment we seed rapeseed and cotton. It is driven by diversification approach. We are driven by information. We need to work more with new cultivars.” One CRW project expert noted, “the diversification policy is now being promoted for the last five years. It's not always successful. We are showing everybody that diversification if economically beneficial.” In addition, GoK supports the diversification by providing subsidies to those farmers practicing crop rotation. When those farmers adopting adaptation techniques were asked during the telephone survey what benefits they expected to accrue from adoption, 11 of 15 respondents indicated that they expected increased harvests and 3 respondents indicated that they expected the quality of harvests to improve. During the interviews and FGD, the evaluation team was also able to explore some of the factors that explain why some farmers choose not to adopt adaptation techniques or adopt them only partially. Farmers and agricultural specialists cited several factors that constrain adoption:  Applicability: Within each of the northern regions of Kazakhstan, there is considerable variability in soils and climates (micro-climates). Famers are concerned about the feasibility of adopting the showcased techniques due to differences in soil and climate between their farms and the demonstration plots. Farmers prefer to see the results of applying a new technology in neighboring farms to make a decision on whether to apply it or not.  Economic constraints: The cost of some techniques is high and requires significant financial investment (e.g., different seed varieties). Moreover, applying new technologies may require modernizing machinery and other equipment, as well as chemicals (e.g., required for zero tillage farming). Farmers are not confident in the time-scale of return on investment and understand that the relative cost of techniques is greater where applied on less land.  Insufficient knowledge base: Farmers do not feel that they have adequate understanding for the application of new technologies, and there are few opportunities to consult experts during implementation. One example cited was that farmers are not always clear on which adaptation technology to use, as in the case of minimum- versus zero-tillage farming.  Human resources: There is a relatively low level of education and qualifications among farmers (especially small-scale farmers), and it can be challenging for farmers to identify well￾trained specialists familiar with these techniques: “Yield depends on 80 to 85 percent of how well agronomist and economist planned.” A couple of quotes summarize some farmers’ attitudes toward adaptive technologies: “We are satisfied with what we have, meaning agricultural crops and technologies,” and “The variety of crops depends on the market demand of the year and the needs of the cattle-breeding sector. That is why you don't see much new in the business.” Despite the challenges described earlier, some farmers felt forced to start using adaptive techniques because of the changing life conditions: “Because of the rains we got new weeds we have never had before. We almost lost ripe wheat. We were forced to desiccate by plane. I really liked it. We applied 100 percent aerial soil treatment. As a result, first and foremost, the wheat was ready for harvesting, and we didn't need Final Evaluation Report – Performance Evaluation: CRW Project 46 to thresh it, even not fully ripe. I won big time. Secondly, there were no weeds. For two years it was like living in a dream. And what is the most important: I did no pre-seeding treatments!”101 Institutional respondents across the board emphasized that farmers do not maintain fidelity to the implementation guidelines for adaptation techniques. Where adoption was occurring, farmers were likely to adopt techniques partially, whether due to a lack of resources or lack of trust. The CRW project did not establish mechanisms to provide support and monitor the fidelity of implementation. In addition (but somewhat related) to the constraints noted above, the evaluation team identified several other factors that to date have limited, or may in the future limit, the impact of the demonstration plots. 1. Three demonstration plots were insufficient in number to demonstrate different adaptive techniques. Each of the three regions is comprised of micro-climatic zones and feature different types of soil. Only farmers within a 200 to 400 km radius of the demonstration plots would have similar climate and soil conditions such that the showcased results could be reliably replicated. 2. The pilot plots and adaptive techniques demonstrated have only been established for two agricultural seasons. Hence, there has been insufficient time for a significant demonstration effect to take hold – especially given the size of the three oblasts. Farmers prefer to see positive results from their neighbors before trusting that new techniques will be effective. 3. The CRW project supported only one field day a year (out of 20 to 25 days organized by each extension center). Although the pilot plots are open to the public at any time, the field days play an important role in educating farmers about new agriculture adoptive techniques. 4. The evaluation team did not find evidence that there is a financial commitment from the Agro￾centers to sustain the demonstration plots following the end of the CRW project. Due to both the difficulty of identifying visitors to the demonstration plots and the short time period that they have been in existence, the evaluation team was unable to assess more specific potential impacts of the pilot demonstration plots. However, due to both demonstration site accessibility and the expressed importance of a demonstration effect to agricultural decision-making, it would seem likely that farmer exposure to the demonstration plot adaptation techniques would continue to grow over time. 101 Interview with farmer, July 29, 2015 Final Evaluation Report – Performance Evaluation: CRW Project 47 Conclusions:  Conclusion 6.1: Exposure by farmers to CRW-sponsored trainings was consistent with CRW objectives and targets, although small in number relative to the overall population. [Supported by Findings 6.3]  Conclusion 6.2: The CRW-sponsored training module was effective at communicating the risks posed by climate change and encouraging the adoption of adaptation techniques for a significant minority of the farmers exposed to the trainings. [Supported by Findings 6.2, 6.3, and 6.4]  Conclusion 6.3: Few farmers are aware of CRW-sponsored publications and media. [Supported by Findings 6.5 and 6.6]  Conclusion 6.4: Among farmers who viewed the CRW-sponsored demonstration plots, there is evidence that a significant proportion at least partially adopted an adaptation technique showcased. [Supported by Findings 6.7, 6.8, and 6.9]  Conclusion 6.5: As farmers prefer to adopt technologies where there is successful evidence of adoption on neighboring farms, the impact of the CRW project will depend on the extent to which this demonstration effect occurs and likely can only be determined ex-post. [Supported by Findings 6.4, 6.7, 6.8, 6.9, 6.10, 6.11, and 6.12]  Conclusion 6.6: There are many reasons that farmers choose not to adopt showcased adaptation practices, including their (for some) high-cost, concerns about whether the practices are applicable to the soil and climate conditions on their farms, and a lack of expertise. [Supported by Findings 6.11, 6.12 and 6.13]  Conclusion 6.7: Farmers of medium and large farms are more likely to undertake adaptive techniques connected to crop diversification than adopt techniques connected with no or minimum tillage, use of fertilizers, water accumulation, and wheat variety. [Supported by Findings 6.14]  Conclusion 6.8: The partnership with Agro-centers allowed the CRW project to gain access to farmers and leverage the expertise and resources of the Agro-centers, but the infrequency of CRW-supported field days resulted in limited exposure of farmers who could be monitored by the project. [Supported by Findings 6.2, 6.7, and 6.8] Final Evaluation Report – Performance Evaluation: CRW Project 48 Evaluation Question 7: Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries? The goal of the CRW project has been to “catalyze the process of adaptation in Kazakhstan’s wheat sector”102 by contributing “to strengthen the livelihoods and resilience in Central Asia by strengthening the wheat production sector as a whole and especially its ability to anticipate, cope with, and recover from climate-related risks.”103 As part of this process and at the outset of the project, the CRW project (with CCRD support) conducted a series of workshops with key stakeholders including government officials, agricultural specialists, and farmers (although turnout for this last group was relatively low) to identify the main challenges confronting the wheat production sector in the face of climate change. On the basis of these workshops and additional consultations with stakeholders, the CRW project focused its activities in Kazakhstan on the twin objectives of improving weather and climate information and information dissemination, and promoting climate-resilient wheat production adaptation techniques. Another component of the project also supports Kazakh policies, strategies, and coordination across Central Asia but is not the subject of this evaluation – although some elements of this work are considered in the discussion under this EQ. To address EQ 7, the evaluation team examined whether the challenges identified by the CRW project are considered by stakeholders to be the major challenges to climate change wheat resilience, i.e., the “main problems and general challenges facing the wheat sector, climate impacts [and] vulnerability,”104 or whether other challenges should have been addressed by the project or should be considered during a subsequent phase of the project or a similar project. This EQ was addressed primarily through interviews with a variety of Kazakh stakeholders, farmers, agro-producers, agricultural specialists, and institutions. The primary challenges faced by the Kazakhstan wheat sector to anticipate, cope with, and recover from climate-related risks that were identified by the evaluation team are described and explained below. Challenge 1: Accessibility of High-Quality and Reliable Weather and Climate Forecasts  Finding 7.1: Beneficiaries and stakeholders agreed that access to high-quality and reliable weather and climate forecasts for farmers was a key challenge. CRW project beneficiaries and stakeholders almost universally agreed that obtaining high-quality and reliable weather and climate forecasts was important for farmer decision-making. According to nearly all farmer respondents to the telephone survey, weather and climate information is either the most important factor or one of the most important factors when making decisions on when to plant and harvest (see Figures 5 and 6). 102 CRW Final Close Out Report (UNDP 2014), page 4. 103 Ibid., page 5. 104 Ibid., page 7. Final Evaluation Report – Performance Evaluation: CRW Project 49 FIGURES 5 AND 6: FARMER PERCEPTIONS OF THE IMPORTANCE OF WEATHER AND CLIMATE INFORMATION FOR HARVEST AND PLANTING DECISIONS Discussions with farmers also identified obtaining reliable medium- and long-term forecasts as a major challenge. The quality, accuracy, accessibility, timeliness, and cost of medium- and long-term forecasts were cited as key challenges by agro-producers as well as institutional researchers and stakeholders from the World Bank and Global Environment Fund. KHM experts themselves stated that low quality and lack of access to medium- and long-term forecasts caused losses to farmers and a lack of trust in KHM data. Respondents to the telephone survey were asked if access to accurate and reliable seasonal climate data/information would increase wheat production. Eighty-one percent believed that having such access would increase wheat production either to a large or moderate degree. Among the key benefits that farmers had hoped to receive from using KHM seasonal climate data/information were: 48% 40% 10% 2% Importance of Weather and Climate Information for Farmer Planting Decisions One important factor The most important factor Just one of the many factors Not considered n=42 41% 39% 15% 5% Importance of Weather and Climate Information for Farmer Harvest Decisions One important factor The most important factor Just one of the many factors Not considered n=41 Final Evaluation Report – Performance Evaluation: CRW Project 50  Better planning of planting and harvesting works (18 respondents);  Prepare more reliable and on-time forecasts (9 respondents);  Higher yields (6 respondents); and  Increased profit (8 respondents). As findings to EQs 1 and 2 demonstrated, the CRW project responded to this challenge by facilitating collaboration between KHM and institutional counterparts tasked with distributing this information to farmers and agro-producers. However, the CRW project focused on the upper level of information flow, and did not adequately address the accessibility of this information by farmers, including through bottlenecks in data flow from the Farmers Union to its membership or through akimats. While the geo￾portal was supposed to be the tangible output to address the issue of data accessibility for all stakeholders, this has not occurred and at the time of this evaluation the geoportal’s future remains uncertain. Challenge 2: Limited Understanding of the Impact of Climate Change on Agricultural Practices  Finding 7.2: Institutional stakeholders agreed that there was a need to promote better understanding of the impact of climate change on the Kazakh wheat sector.  Finding 7.3: CRW project activities focused on improving farmers’ understanding of climate change and measures that could be taken in response. KAI experts and a former CRW project expert cited low use of new technologies as a challenge for adaptation, due to costs as well as habits and the overall passiveness of farmers. These experts saw a solution in the continued distribution of information on innovative approaches and farmer engagement. The vulnerability of the Kazakh wheat sector to climate change and the resulting need to mainstream climate-resilient adaptation measures were recognized by the designers of the CRW project. The project made a strategic decision to work through the knowledge distribution centers to teach farmers about climate change risks and adaptation techniques and promote demonstration sites. As noted by one Agro-center representative,105 as a result of being exposed to the CRW project￾supported module on climate change within the training materials, some farmers and agro-producers were introduced to the connection between climate change and extreme weather conditions that affect crops.106 Quantitative findings from this evaluation somewhat support this assessment:  Ninety-six percent of farmers surveyed who could recall participating in the climate change trainings (25 of 26 respondents) affirmed that they had developed a better understanding of what they could do to prepare for and address the impacts of climate change.  Almost all respondents to the survey (73 out of 77) affirmed that climate change had an effect on agriculture in Kazakhstan in general, and their respective farms in particular. Furthermore, they all were confident in their understanding of the risks associated with climate change.  When asked to identify risks associated with climate change as part of an open-ended question, 3 respondents cited unfavorable weather conditions and 2 each identified varying planting and harvest times and decreased ripening (out of 10 who responded to the open-ended question), which were more than responses about other risks. Twenty-two out of 32 (70 percent) 105 Interview at Shortandy, 106 Interviews with Stanislav Kim, UNDP and Ashley King, USAID/CAR Final Evaluation Report – Performance Evaluation: CRW Project 51 representatives of research institutes, academia, and government entities responding to the survey noted bad weather conditions as a risk posed by climate change. While the rates of adoption of adaptation techniques were not commensurate with the high levels of understanding evidenced through the research, the evaluation team found that this was largely due to challenges outside the scope of the CRW project to address and the need for appropriate time for the application of active versus passive knowledge. Challenge 3: Lack of Knowledge about Advanced Farming Practices among Farmers and Agro-Producers  Finding 7.4: Institutional actors view the lack of knowledge among farmers and agro￾producers as a key challenge to improving wheat production. KAI institutional representatives and analysts cited a lack of knowledge about new technologies, even ones not involving actual new equipment, as a key challenge. Consistent with CRW project activities, these stakeholders saw a need to distribute information and educate farmers about new practices in order to raise awareness and interest in innovative approaches to mitigate the impact of climate change: “Farmers are old school, they don’t want extra expenses.”107 Stakeholders including from the World Bank also noted the relatively low knowledge base with respect to approaches to weeding and alternative approaches to the use of chemicals and pesticides. Farmers, on the other hand, did not indicate that a lack of knowledge prevented them from applying techniques. However, attempts to generalize from this evidence should be treated with caution. While farmers did not indicate that a lack of knowledge was deterring adoption, this could be a manifestation of response bias and a desire not to look uninformed to the interviewer. On the other hand, since the telephone survey respondents had elected to attend CRW project training and capacity building activities, these farmers – compared to an average farmer – may be more concerned about the risks from climate change, more open to new knowledge, or merely possessed a higher level of knowledge resulting from the trainings themselves. However, nearly all respondents acknowledged increasing learning through the training. The evaluation team cannot make further conclusions on this issue based on the evidence. Challenge 4: Old and Outdated Farmer Equipment  Finding 7.5: The lack of modern equipment required to implement adaptation techniques was seen as a detriment to the fidelity of adoption and production in general. All classes of stakeholders (e.g., institutional actors, farmers, experts) from whom data were collected for this evaluation cited older farm equipment as a main challenge to a better production, especially for small- and medium-scale farmers, as well as to some extent to the fidelity of adoption of adaptation techniques where producers lacked updated machinery to comply with implementation guidelines. Seventeen percent of farmers responding to the telephone survey cited old equipment as an obstacle to higher yields, and 7 other representatives saw the lack of modern machinery as an obstacle to a better production. While this challenge was not addressed by the CRW project, it does appear to have influenced the delivery of CRW activities. While the CRW project did not target specific types of farmers and all farmers could participate in CRW activities, there was a sense among beneficiaries that large-sized farms 107 Interview with KAI, July 21, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 52 are more equipped to use the skills and knowledge promoted by the CRW project because small and medium-sized farms do not have the financial resources to purchase necessary modern equipment. Some farmers noted that they had arrangements in place to share equipment, but with the tight schedules for planting/harvesting, it would be challenging to adhere to technological recommendations if equipment had to be shared. Several farmers and experts called for grants to help modernize equipment.108 Challenge 5: Access to High-Quality and Appropriate Seeds  Finding 7.6: Institutional stakeholders stated that the lack of appropriate seed varieties and the high cost of seeds had a detrimental effect on wheat production.  Finding 7.7: The GOK procedure for approving new seed varieties is viewed by stakeholders as slow and inefficient.  Finding 7.8: The CRW project provided evidence of its work in the seed domain, but this did not come up in interviews with stakeholders. Several stakeholders (including the KAI team and independent agronomists) cited the poor condition and state control of seed science and the high costs of seed as key challenges; a couple of farmers also indicated the poor quality of seeds as an obstacle to higher wheat production. Only 10 percent new seed varieties have been introduced since 2005,109 and the current variety does not respond to the needs of the sector. For planting, farmers are supposed to only use seed types that are listed in the State Registry of Seeds maintained by the State Committee on Seed Testing. The testing and introduction process for the State Registry takes three years and results in a seed rayoing certificate for each type of seed. The rayoned seed is tested in different zones and conditions that specify its use to certain oblasts/districts. Representatives of the Agro-centers as well as farmers questioned why one kind of seed was allowed for use in one oblast but not in the bordering district that belongs to another oblast. Related to this, KAI and other agro-experts mentioned the genetic homogeneity of wheat varieties, 51 percent of which were not adapted to the Kazakh soil110. KAI challenges to the system of state licensing of seed varieties have in some cases reduced the licensing process to one or two years, but within the context of changing climatic conditions, this is still slow and inefficient: “Even the shift from three to one years required for approval was too long.”111 Farmers do produce their own seeds to reduce reliance on the costly seeds and limited varieties, but the World Bank and Global Environment Fund experts noted that this reduced the quality of seeds. The CRW project helped conduct two international conferences where these challenges were addressed.112 Additionally, the project established a seed laboratory that brought 60 kg of wild wheat varieties from other Central Asian countries.113 Two new drought-resistant wheat varieties have been produced, with a focus on Tajikistan and Afghanistan, and official certification has been received.114 108 Interviews with KAI and Bakhtyar Sadyk. 109 Interview with KAI, July 21, 2015. 110 Ibid. 111 Ibid. 112 CRW project documents and interview with the CRW project team. 113 Ibid. 114 Ibid. Final Evaluation Report – Performance Evaluation: CRW Project 53 Challenge 6: Low Wheat Storage Capacity  Finding 7.9: There is inadequate storage capacity to accommodate high volumes of grain production. In both the quantitative and qualitative research for this evaluation, agro-producers cited a lack of storage capacity as a key challenge, noting that this was actually more of a problem than production and could become an even more significant issue with stable harvests. A lack of adequate storage capacity requires small- and medium-scale farmer to sell their yield at a lower price immediately after harvest in order to avoid costs for its storage. It also minimizes the amount that is retained for the following year’s planting. “Having necessary infrastructure, technology and equipment (drying equipment and storage) decreases vulnerability to extreme weather conditions.”115 This problem mainly concerns small- and medium-scale farmers, as large producers have warehouses and financial resources to maintain yield during the winter season. However, even for large-scale farmers, storage costs are extensive and require not only physical facilities but also human resources to keep yields in good conditions. As one large-scale farmer noted, “We do not have storage, we save crop yield in sacks and sell it directly from fields. In this way we harvest all 100 percent of yield and buyers pick up sacks from the field.”116 Article 10 of the Kazakh Grain Law, “assistance for the application of new technologies on grain production and storage,” is among the support measures (educational and consulting services of 10 extension centers) that exist for grain producers and grain operators. However, given the extent to which this challenge was mentioned by not only farmers but also analysts, it is not clear whether farmers are aware that this legislation exists or whether there is difficulty in the application of the legislation. Challenge 7: Decreased Prioritization of Wheat Production by MinAg  Finding 7.10: UNDP’s work with the Ministry of Environment and MinAg prior to the design of the CRW project laid the groundwork for the attention to climate adaptation and food security issues in Kazakhstan and region as a whole.  Finding 7.11: At the time of this evaluation, MinAg’s strategy is to prioritize other grains and meat instead of wheat.  Finding 7.12: The reprioritization from wheat to other agricultural products has influenced the level of subsidies and other kinds of government support for wheat producers. Prior to the CRW project design, UNDP/Kazakhstan had worked on sensitizing the Ministries of Agriculture and Environment Protection to the development of an adaptation strategy. The Environmental Protection Ministry was only able to define its strategy due to the GOK requirement in place at the time that strategies had to be accompanied by mandatory financial commitments, which were lacking at the GOK level. USAID recognized the UNDP work and picked up the issue for the CRW project design, as it also started considering food security issues in the region.117 As evidenced by Finding 3.1, the CRW project has been facilitating cooperation among the experts and providing recommendations towards developing a climate change adaptation law. Furthermore, a group 115 FGD with farmers, July 27, 2015 116 Interview with farmer, Kostanay, July 31, 2015. 117 Interviews with Stanislav Kim and Ashley King Final Evaluation Report – Performance Evaluation: CRW Project 54 of experts at the Ministry of Energy has been established to deal with climate change, including its effect on wheat production. However, at the time of this evaluation, in prioritizing wheat over other grains and livestock, the CRW project’s priorities differed from those of MinAg and the latest World Bank recommendations. A KAI expert noted: “Wheat will still be the main sector for Kazakhstan in the future, but MinAg has become more interested in livestock lately, and has pulled back from other activities.”118 The reprioritization of meat by the GOK is worth mentioning in this context because, on the positive side, it promotes crop diversification through forage crops. However, at the same time, it draws resources at KAI and other MinAg entities away from wheat-related projects. If wheat production is not a priority for the GOK and subsequently others, building resilience to climate change in that sector appears implausible. However, the CRW project team noted that Kazakhstan had recently signed two long-term wheat export agreements with the governments of Saudi Arabia and Afghanistan and that the GoK focus on livestock “does not mean that the area of wheat crops is going to be diminished.” This discussion is related to the nature of focus of GOK subsidies to the agricultural sector more generally. To support the GOK diversification strategy, subsidies for “priority” crops such as barley, corn for grain, rapeseed, soybeans, pasture grasses, and corn for silage increased in 2014 at the expense of those for wheat. Furthermore, unlike wheat acreage, the acreage sown with oilseeds nearly doubled between 2009 and 2014, and continues to grow.119 According to the literature, “the trend of land shifting from wheat to other grains and oilseeds was expected to continue in 2015, and the GOK would encourage this diversification.”120 The primary reasons given for this shift are:  The difficulty in getting wheat to export markets due to Kazakhstan’s landlocked status;  The desire to increase domestic consumption of grain in Kazakhstan; and  The GOK’s goal to turn Kazakhstan into an exporter of meat products. The evaluation team found diverse and strong opinions on the issue of wheat subsidies. While agro￾producers believe in the need for subsidies for wheat production, a representative of KazAgroMarketing encourage subsidies for adopting new technologies and marketing approaches in grain production and an expert from the World Bank expressed an opinion that farmers should be subsidized based on their production results, rather than planting.121 118 Interviews, July 21, 2015. 119 Kazakhstan-Republic: Grain and Feed Annual report 2015, USDA. 120 Kazakhstan-Republic: Grain and Feed Annual report 2015, USDA. 121 Interview, August 6, 2015. Final Evaluation Report – Performance Evaluation: CRW Project 55 Conclusions:  Conclusion 7.1: The CRW project’s focus on improving the accuracy and availability of medium- and long-term climate forecasts addressed a key challenge for wheat production, although the project focused insufficiently on issues of access to and cost of data, which are faced by small-scale farmers in particular. [Supported by Finding 7.1]  Conclusion 7.2: Aside from access to and quality of the climate and weather information, the most significant challenges to wheat production identified by stakeholders are outdated farm equipment, a lack of access to high-quality seeds, and inadequate storage for farm yields – none of which were comprehensively addressed by the CRW project, largely due to the limited project scope. [Supported by Findings 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8 and 7.9]  Conclusion 7.3: At the time of this evaluation, the decision to focus on improved climate resiliency of wheat production did not fully align with the GOK focus on improving the production of other grains and livestock. However, the CRW project facilitated strengthening the GOK’s attention to the issues of food security and adaptation in the context of climate change in the region. [Supported by Findings 7.11, 7.12 and 3.1] Final Evaluation Report – Performance Evaluation: CRW Project 56 Findings Related to Gender  Finding G1: Institutional gender experts were consulted at USAID and the United Nations, but a deliberate decision was made not to address gender in the CRW project implementation beyond collecting sex-disaggregated data.  Finding G2: The results of the available gender analysis and literature were not taken into consideration when the CRW project was designed and implemented.  Finding G3: Sex-disaggregated data from CRW project-supported activities were not used to encourage gender balance in the activities going forward.  Finding G4: None of the project beneficiaries or stakeholders reported that the project was deliberately targeting one group of beneficiaries over the other, although farmers with medium- and large-scale farms appear to have disproportionally benefitted from project activities.  Finding G5: Cultural norms and traditions prevail and would require an extensive effort to work with male farmers to open space for females in the agricultural sphere. The USAID environmental officer that oversaw the design and implementation of Phase I of the CRW project stated that although the USAID/Kazakhstan gender focal person had been consulted and an ADB Kazakhstan gender analysis document was reviewed, gender considerations were deliberately not included in the project design. Notably, the evaluation team found a 2010 USAID Gender Assessment of the Central Asian countries that cited “little to no coordinated, substantive attention to gender at the USAID/CAR mission, including lack of gender expertise among mission staff.” At the time of this evaluation, the USAID gender expert was new to the Kazakhstan Mission and did not have any background with the CRW project. Furthermore, the USAID officer who was overseeing the project from early 2013 to early 2015 pointed out that since gender was not a requirement as per the CRW grant agreement, UNDP was not responsible to go beyond collecting and reporting sex￾disaggregated data. In line with the above findings, the current USAID Agriculture Officer overseeing the CRW project expressed that “the Project has not addressed especially the gender issues, because knowledge and skills promoted by the project haven’t got gender specifics. Almost all farmers are males and the project can’t change this situation.” The CRW project met the minimum requirements and ensured that, where applicable, indicator data was sex-disaggregated. However, when the evaluation team requested data from CRW-supported capacity-building activities, data provided from the Agro-centers did not come sex-disaggregated. Evidence from the evaluation did not show that the CRW project had a specific gender focus, and the project indicators (from the 2012 PMP) did not include measures about how access to information, knowledge, or technical assistance, for example, might be different for men and for women. Moreover, the Women’s Empowerment in Agriculture Index122, which USAID would recommend for agricultural projects, was released in early 2012, after the CRW project design phase. Consistent with the evaluation team’s analysis of background documentation and the telephone survey results123, a former USAID manager of the CRW project noted that: 122 http://feedthefuture.gov/lp/womens-empowerment-agriculture-index 123 Seventeen percent of respondents were affiliated with small-size farms, and 60 percent with large-size farms. Final Evaluation Report – Performance Evaluation: CRW Project 57 “the Index is not applicable for the North Kazakhstan, and probably Kazakhstan in general. In the North Kazakhstan (unlike in the South) the vast majority of farms are industrial, commercial size farms, and an extra effort would have had to be made to adapt the value chain on which the index is based, which is not related to the objectives of the CRW project.” When asked about the status of female farmers and agro-producers, and whether they face particular barriers, the former CRW project director noted that “female farmers have resources, and they have perseverance…. We do have to train them more”’. The same attitude was cited by a couple of respondents to the telephone survey. This assessment was supported by available data on the beneficiaries of CRW-supported training and capacity building activities: only 25 percent of the overall number participants from the lists made available to the evaluation team were women. Similarly, females represented only 24 percent of the respondents for the telephone survey. With a few exceptions, their views were consistent with those of male colleagues Notably, there was a difference in the proportion of female beneficiaries by the Agro-centers with which activities were affiliated: 16 percent in Shortandy, 63 percent in Kostanay, 18 percent in North Kyzylzhar, and 27 percent in Petropavlovsk. Staff interviewed at the Agro-centers were predominantly female, and training statistics indicate that both genders interact with knowledge distribution centers and would be involved in making decisions around the purchase and use of inputs and upgrading and receiving technical assistance. Three out of eight Agro-center directors are women. However, when targeting farmers for KIIs, the evaluation team only managed to talk to one female agronomist, who was an employee at a farm but not the owner. She did not identify gender-specific barriers in her work. Overall, about half of the respondents were convinced that there were no barriers for women in the agricultural field: about a quarter were not aware of such issues, and some provided examples. Additionally, several agro-experts identified the following barriers to gender equality in agriculture, and some were supported by findings of the telephone survey as well:  Cultural norms:  Men leading in the traditionally labor-intensive sector, which results in lower enrollment of women into agro-related fields of study  Family inheritance and traditions  Lack of time, including due to household chores and personal issues  Physical:  Women’s physical characteristics preventing them from working in the field  Institutional:  Nominal nature of some posts, where women are assigned leadership posts on paper only  Limited work experience, as many directors come from leadership positions in government￾supported farms during Soviet era, when women held lower positions Qualitative and quantitative findings from this evaluation indicate that beliefs about the roles of women are one of the main barriers to their more significant participation in agriculture. When asked about the role of women and men in the farm, FGD participants noted: “as a rule, women work in the labs, as accountants, deal with seeds, not as much agronomists or managers. Men are mechanics. There is a clear separation of roles: men are in the field, women are at home (for lighter roles, hence their longer life expectancy). We do not foresee changes, women are already emancipated too much, they spend too much time at work and there is nobody to take care of the house”. According to the telephone survey, women were only accountants and cooks, and were not equipped for driving tractors in the fields. A few respondents recognized that as a result women working in agriculture were paid lower wages. Final Evaluation Report – Performance Evaluation: CRW Project 58 The analysis of sex of the institutional key informants shows a bias to men as well: only 26 percent of the identified national and regional key informants were women. As indicated in Annex F, their positions ranged from department head in the Ministry of Energy to forecasting specialist in KHM, and gender issues were not evident. Conclusions:  Conclusion G1: There was no focus in CRW project design, implementation, or monitoring on an inclusive project model or gender-differentiated approaches to project delivery. [Supported by Findings G1, G2 and G3]  Conclusion G2: Capacity-building assistance benefited fewer women than men, as women were underrepresented among farmers and agro-producers.  Conclusion G3: The CRW project lacked specific activities to encourage equal access for female agro-producers and female farm owners, although due diligence was taken to collect and report sex-disaggregated data, which illuminated gender gaps. [Supported by Findings G1, G2 and G3]  Conclusion G4: Although the CRW project did not systematically address gender considerations, the evaluation did not reveal that women or vulnerable populations were actively excluded from project design and implementation. [Supported by Findings G1, G2, G3 and G4] Final Evaluation Report – Performance Evaluation: CRW Project 59 RECOMMENDATIONS The evaluation team provides two sets of recommendations. The first set is focused on actions that can be taken in the near term to improve the performance of this second phase of the CRW project. The second set of recommendations applies to any subsequent project. Recommendations to Improve the Performance of the CRW Project Recommendation 1 The CRW project should support efforts to bring into KHM existing surface agrometeorological data and information from agricultural research center station observations and NSRI satellite-derived analyses. To achieve the objectives of CRW, such data should be freely shared and the assessments should be collaborative. [Supported by Findings 1.10 and 1.11 and Conclusions 1.1 and 1.3] Recommendation 2 The CRW project should continue working with KHM and MinAg on prioritizing increasing the usability of information produced by MinAg-supported climate stations. This would require training, site review, the installation of telemetry, the expansion of analysis processes, and administrative oversight. [Supported by Findings 1.10 and 1.11 and Conclusions 1.1 and 1.3] Recommendation 3 The CRW project should support the continued digitization of a large archive of historical observational data (over the past few decades) that currently exists only in paper form. [Supported by Findings 1.5, 1.6, 1.10 and Conclusion 1.3] Recommendation 4 The CRW project should take efforts to improve the effectiveness and efficiency of the free information-sharing mechanisms including the MinAg bulletins and the geoportal. As the Internet is the primary mechanism through which farmers access meteorological data, the development of the geoportal and the provision of additional information through the KHM and MinAg websites (e.g., the bulletins currently provided freely at akimat meetings) could be immediate objectives. [Supported by Findings 2.6, 2.7 and 2.10 and Conclusions 2.1, 2.2, 2.3 and 2.4] Recommendation 5 The CRW project should consider conducting a cost-benefit analysis of the required potential support necessary for MinAg to take the lead in distributing climate information. [Supported by Findings 2.5, 2.6 and 2.10 and Conclusion 2.1] Recommendation 6 Given the close collaboration between the CRW project and MinAg, the project should work with MinAg to identify and resolve bottlenecks in the distribution flow of bulletins to farmers. [Supported by Conclusion 2.2] Final Evaluation Report – Performance Evaluation: CRW Project 60 Recommendation 7 The CRW project should work with key stakeholders and build on the success of professional networking established to create an action plan and timeline for the development of the geoportal. [Supported by Conclusions 2.3 and 2.4] Recommendation 8 The CRW project should extend the use of the Training of Trainers model to include actual farmers and agro-producers with a good success rate and reputation in the community, to increase the likelihood of adoption and sustainability, and to increase experience and knowledge sharing among farmers. [Supported by Conclusion 6.2, 6.4, and 6.5] Recommendation 9 Due to the high utilization of the Internet by farmers to learn about new agricultural techniques, the CRW project should work with MinAg and KazAgroInnovation to provide additional information about climate-resilient adaptation techniques on the appropriate websites. [Supported by Finding 6.1] Recommendation 10 The CRW project should improve its monitoring of project outputs and outcomes, including specifically assessments of knowledge gained as a result of project training and innovative approaches such as outcome mapping, to capture the outcomes of farmers participating in project-sponsored field days. The CRW project should also take steps to monitor visits to the pilot demonstration plots other than during field days. [Supported by Finding 6.7 and Conclusion 6.8] Recommendation 11 The CRW project should consider a strategy for the sustainability of project activities and outcomes following the conclusion of the project. This strategy should at a minimum involving planning for the sustainability (if sought) of CRW-sponsored demonstration plots. [Supported by Finding 3.7] Recommendation 12 The CRW project should use sex-disaggregated monitoring data, the results of the farmer survey, information from gender assessments, and consultations with local gender experts with sectoral expertise in agriculture to understand and address relevant gender gaps and barriers as feasible within the project scope. [Supported by Conclusions G1 and G3] Recommendations for the Design of a Subsequent Phase of the Project or another Similar Project Recommendation 13 USAID should consider supporting additional surface meteorological stations and a specific meteorological office capacity-building project across the Central Asia region. [Supported by Findings 1.10 and 1.11 and Conclusions 1.1 and 1.3] Final Evaluation Report – Performance Evaluation: CRW Project 61 Recommendation 14 Design a capacity building project, specifically geared towards a particular agricultural sector (like wheat) or array of sectors in Kazakhstan, considering the following aspects: (1) hydro-agro-met observations, which should be made to a World Meteorological Organization standard for accuracy; (2) data analyses or assessments, which should have certain criteria for site representativeness and spacing as well as data frequency; and (3) forecast processes, the potential accuracy of which will be greatly influenced by the extreme continentality of the Kazakhstan wheat-growing areas. [Supported by Conclusions 1.1, 1.3 and 1.4] Recommendation 15 Diligence should be taken to consult available national normative guidance related to promoting women’s empowerment at the project design stage, especially when cooperating with the government and semi-government institutions as the key stakeholders. [Supported by Conclusion G1] Recommendation 16 Openly recognize and commit adherence to USAID’s Gender Equality and Female Empowerment Policy and Automated Directives System 205. This should include: (1) stating in project design, planning, and monitoring documents that achieving gender equality and women’s empowerment is a project priority; and (2) undertaking a gender analysis to understand relevant gender gaps and identify the project’s beneficiary and stakeholder groups within which women should be represented as leaders and members. [Supported by Conclusion G1, G2, G3 and G4] Recommendation 17 USAID should support policy advocacy intended to remove barriers (such as the current requirements for leasing farming land in Kazakhstan) that impede women’s participation in wheat farming in Kazakhstan. [Supported by Conclusion G3] Recommendation 18 Promote the use of agro-meteorological and climate information communication methods that are likely to be effective in reaching women farmers and managers in the wheat farm sector. [Supported by Conclusion G1, G2, G3 and G4] Recommendations for the Government of Kazakhstan Recommendation 19 GOK should consider introducing principles for long-term weather forecasting and agricultural crop productivity/harvest outlooks in strategic government documents. [Supported by Conclusions 1.2, 1.3 and 1.4] Recommendation 20 GOK, through consultation with relevant ministries, should develop an action plan, with assigned responsibilities, for the development of the geoportal. [Supported by Conclusions 2.3 and 2.4] Final Evaluation Report – Performance Evaluation: CRW Project 62 Recommendation 21 GOK should consider establishing an early warning system for drought to promote drought mitigation. [Supported by Conclusions 1.2 and 1.4] Recommendation 22 Kazhydromet and the NSRI should consider developing mechanisms for the free exchange of data and collaboration on assessments that would include bringing together existing surface agrometeorological data and information from agricultural research center station observations and NSRI satellite-derived analyses. [Supported by Conclusions 1.3 and 2.1] Recommendation 23 GOK should consider financing the construction of additional surface meteorological stations and engage in capacity building for the meteorological office. [Supported by Conclusion 1.3] Recommendation 24 MinAg should consider how to improve the utility and dissemination of weather and climate data, including through improvements in weather and climate analysis and improved effectiveness and efficiency of free information-sharing mechanisms such as the MinAg website and bulletins. [Supported by Conclusions 1.1, 2.1, 2.2, 2.3 and 2.4] Recommendation 25 MinAg and KazAgroInnovation should consider expanding the Training of Trainers model to include additional farmers and agro-producers with a good success rate and reputation in the community. [Supported by Conclusions 6.4, 6.5 and 6.6] Recommendation 26 MinAg should consider collaboration with the National Commission on Women’s Issues and Family and Demographic Status to better understand and address relevant gender barriers faced by female agro￾producers. [Supported by Finding G5 and Conclusions G3 and G4] Final Evaluation Report – Performance Evaluation: CRW Project 63 ANNEX A: EVALUATION STATEMENT OF WORK Performance Evaluation: Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security 1. Activity Description In October 2012, with grant funding from the United States Agency for International Development (USAID), the United Nations Development Programme (UNDP) initiated the Improving the Climate Resiliency of Kazakhstan Wheat and Central Asian Food Security (CRW) project. The aim of CRW is to strengthen Central Asian regional food security through increased resilience to climate change at the national and local levels. Of particular concern in this project is the stability of Kazakhstan’s wheat supply, on which the entire region relies, as that supply is highly vulnerable to climate change and a significant decrease in production is projected, absent an adequate adaptive response. As the primary funder for CRW, USAID committed $1.1 million for an initially-projected two-year project duration. In 2014, USAID and UNDP agreed to a two-year extension to have the project run for four years, through 2016. CRW is structured around three components that focus on: (1) improving the supply of information on climate-resistant wheat production, locally, in Kazakhstan; (2) fostering the adoption of improved techniques by Kazakh farmers and (3) initiating a regional dialogue on wheat, climate change and regional food security. To complement its investments in CRW, USAID has commissioned an evaluation of the project through the Agency’s E3 Analytics and Evaluation Project. While initially envisioned as a final evaluation, this study will now focus on Phase I (2012-2014) of CRW and inform Phase II of the project. 2. Development Hypothesis Figure 1 illustrates the causal linkage USAID and UNDP envision for translating results under each of the three project components into CRW’s intended intermediate and final outcomes. In this Theory of Change diagram, which draws on UNDP’s 2014 Final Narrative Report for Phase I and USAID’s Performance Indicator Data Table for CRW, project components are recast as Output-level results, which then lead to the project’s two key outcomes. These, in turn, are expected to result in improved prospects for long-term food security on a regional basis in Central Asia. With respect to implementation, UNDP’s 2014 Final Narrative Report for Phase I describes accomplishments under each of the three project components and reports against indicators agreed upon with USAID. This report also outlines the magnitude of the work that UNDP views as remaining to be done, including capacity building both at the government level and among actors in the wheat production value chain. Final Evaluation Report – Performance Evaluation: CRW Project 64 Figure1: Theory of Change for the CRW Project124 Supporting the intermediate outcomes displayed in Figure 1 are specific activities that fall under each project Output. These activities are listed in Table 1. Table 1: Project Activities under Each Project Component (Output) Output 1: Monitoring and Information Sharing for Climate-Resilient Wheat Production Improved  Activity 1.1: Needs assessment and stakeholder consultations  Activity 1.2: Improving data collection and dissemination mechanisms  Activity 1.3: Development of forecasting models  Activity 1.4: Improved data sharing and use 124 The Theory of Change was developed by the USAID Mission in Kazakhstan in February 2015. Final Evaluation Report – Performance Evaluation: CRW Project 65 Output 2: Climate Resilience Developed Through Mainstreaming of Adaptation Measures  Activity 2.1: Mainstreaming wheat climate resilience into relevant climate change adaptation and agricultural strategies  Activity 2.2: Priority adaptation options demonstrated  Activity 2.3: Capacity development and awareness raising  Activity 2.4: Improving wheat production, storage and distribution Output 3: Regional Dialogue on Wheat, Climate Change and Regional Food Security Strengthened  Activity 3.1: Gap analysis and assessment review  Activity 3.2: Awareness raising 3. Existing Performance Information Sources USAID’s Global Climate Change Office, in coordination with the USAID/Kazakhstan Mission, has provided the evaluation team with the following documents related to existing performance information: 1. CRW project documents from UNDP:  CRW Final Project Report, 2014125  CRW 2013 Annual Project Progress Report  CRW 2013 PMP Report  CRW 2014 Q1 Project Report  CRW 2013 Work Plan  CRW LogFrame (undated) 2. Trip Reports related to the CRW project prepared by the following organizations:  Climate Change Resilient Development (CCRD) (7)  International Research Institute for Climate and Society (IRI) (2)  WeatherPredict Consulting Reports (2)  Development & Training Services (dTS) 3. Presentations, minutes and summaries from a variety of meetings and workshops conducted by or with CRW’s participation. The above, non-exhaustive list highlights the more important sources of performance information that have been shared with the evaluation team. The following additional documents have not yet been provided to the evaluation team but will be shared prior to the start of the evaluation unless UNDP notifies that these documents do not exist:  List of all CRW-supported trainings or workshops (including those held at demonstration plots) on adaptation techniques, including lists of attendees with all contact information available 125 The Final Closeout Report records CRW progress against indicators from the CRW Performance Management Plan. While this may be a useful source of information on project performance, additional information about definitions and data collection is required to allow the evaluation team to assess the relevance and reliability of these measures. Final Evaluation Report – Performance Evaluation: CRW Project 66  List of CRW-supported trainings or workshops provided to Khazhydromet, the National Space Research Institute (NSRI), KazAgroInnovations and the Ministry of Agriculture (MinAg), including lists of attendees (name and role) with all contact information available  Copies of all monthly or quarterly project management and progress reports provided by CRW  Copies or detailed descriptions of circulars and brochures given to farmers through CRW￾supported initiatives  Report on the reliability of forecasts by IRI  The CRW-sponsored paper, “Methods and models for weather forecasting and evaluation of the agro meteorological conditions of the wheat crops development in Kazakhstan” 4. Evaluation Purpose, Audience, and Intended Use The information to be provided by this performance evaluation on the relevance, effectiveness and sustainability of CRW initiatives is expected to be used by USAID and CRW staff to modify project delivery in order to maximize results delivered in Phase II. The primary audiences for this evaluation will be the USAID Mission in Kazakhstan, the USAID Global Climate Change Office, the UNDP office in Kazakhstan and CRW project staff. 5. Evaluation Questions The following evaluation questions will guide this study: 1. To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for collecting and analyzing agrometeorological data, and making seasonal and climate predictions? 2. To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather- and climate-related data to farmers and other key stakeholders, as well as to each other? 3. To the extent that there are improved practices in the collection, analysis and dissemination of agrometeorological and climate information, are these improvements likely to be sustained? 4. To what extent are other institutional stakeholders using Kazhydromet weather and climate information? 5. To what extent are farmers basing their decisions on Kazhydromet weather and climate information? Why or why not? 6. To what extent have farmers adopted climate change adaptation techniques promoted by the demonstration plots and through CRW-supported education initiatives (such as publications and circulars)? Why/Why not? If adopted, from which initiative? 7. Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries? Final Evaluation Report – Performance Evaluation: CRW Project 67 6. Gender Considerations In line with USAID’s Gender Policy, the research design for this evaluation will consider gender-specific and differential effects of the project. The evaluation team will, if possible, disaggregate the UNDP visitor database for demonstration plots and training to understand the extent of participation by gender (such as participation of female farmers and trainings conducted by female trainers), and the potential influence it has on the project’s efficiency. The team will also explore gender-differential access to and participation in the project at multiple points along the Theory of Change diagram. If the evaluation conducts a survey of farmers, the team will ensure that the questionnaire is gender-disaggregated to identify gender differences with respect to results and benefits, and from lessons learned from female farmers. The team will base further inquiry on gender themes that emerge during data analysis. 7. Evaluation Methods In its Evaluation Design Proposal, the evaluation team will propose detailed evaluation methods suitable for addressing the evaluation questions. It is anticipated that theory-based evaluation approaches applying mixed methods will likely be appropriate, in light of the evaluation questions and the anticipated availability of data. Table 2 summarizes the range of methods the evaluation team may consider using to gather evidence needed to address the evaluation questions. Table 2: Getting to Answers Matrix of Performance Evaluation Questions and Methods Evaluation Questions Data Sources Data Collection Methods Sampling or Selection Plan Data Analysis Plan 1: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for collecting and analyzing agrometeorological data, and making seasonal and climate predictions? -KZH/NSRI/MinAg staff -CRW program managers -CRW technical assistants (CCRD) -Document review -Semi-structured interviews -Small surveys -Structured and unstructured observation -Purposeful sampling -Descriptive statistics -Triangulation and synthesis 2: To what extent has the CRW project been effective in improving practices within Kazhydromet/NSRI/MinAg for disseminating weather￾and climate-related data to farmers and other key stakeholders, as well as to each other? -KZH/NSRI/MinAg staff -CRW program managers -CRW technical assistants (CCRD) -Farmer beneficiaries -NGO - Union of Farmers -Document review -Semi-structured interviews -Small surveys -Structured and unstructured observation -Purposeful sampling -Descriptive statistics -Triangulation and synthesis 3: To the extent that there are improved practices in the collection, analysis and dissemination of agrometeorological and climate information, are these improvements likely to be sustained? -KZH/NSRI/MinAg internal documentation -KZH/NSRI/MinAg staff -CRW program managers -CRW technical assistants (CCRD) -Document review -Semi-structured interviews -Small surveys -Purposeful sampling -Content analysis -Triangulation and synthesis -Secondary analysis 4: To what extent are other institutional stakeholders using Kazhydromet weather -KZH/NSRI/MinAg staff -CRW program -Semi-structured interviews -Survey -Purposeful sampling -Descriptive statistics -Triangulation and Final Evaluation Report – Performance Evaluation: CRW Project 68 Evaluation Questions Data Sources Data Collection Methods Sampling or Selection Plan Data Analysis Plan and climate information? managers synthesis -Document Analysis 5: To what extent are farmers basing their decisions on Kazhydromet weather and climate information? -Farmers -Semi-structured interviews -Random stratified sampling (preferred) -Convenience sampling (if required) -Content analysis -Triangulation and synthesis -Descriptive statistics 6: To what extent have farmers adopted climate change adaptation techniques promoted by the demonstration plots and through CRW-supported education initiatives (such as publications, and circulars)? Why/Why not? If adopted, from which initiative? -CRW program managers -Agro-center staff -Farmers -Semi-structured interviews -Survey -Purposeful sampling -Random stratified sampling (preferred for farmers) -Convenience sampling (if required for farmers) -Descriptive statistics -Content analysis -Triangulation and synthesis 7: Does the CRW project address the key challenges to climate change wheat resilience as understood by project stakeholders and beneficiaries? -Officials from KZH/NSRI/MinAg & Kazprodcorporation -NGO - Union of Farmers -Farmers -Agricultural trade associations -CRW program managers -Document review -Semi-structured interviews -Purposeful sampling -Snowball sampling -Content analysis -Triangulation and synthesis -Secondary analysis 8. Data Analysis Methods Data analysis methods to be proposed in the evaluation team’s Evaluation Design Proposal will follow closely from the methods used to collect each type of data needed to answer the evaluation questions. Whatever data analysis methods are chosen for this evaluation, they should be justified in terms of their fit with the data collected for a question and the types of answers that USAID seeks. Time and cost considerations are also important in this area. 9. Strengths and Limitations There are three significant challenges anticipated for this evaluation. 1. The number and geographic spread of farmers who have potentially been influenced by CRW activities is considerable but, for some evaluation questions, indeterminate. It will likely not be possible for the evaluation team to speak with all or even a random representative sample of farmers who were potentially influenced by CRW activities. Therefore, there will be limitations on the extent to which evaluation findings are externally valid. This limitation applies especially to Evaluation Question 4, but also to a lesser extent to Evaluation Question 6. Final Evaluation Report – Performance Evaluation: CRW Project 69 2. Determining the extent to which farmers rely on KZH weather and climate data will require asking farmers to recall the extent to which they weighed this information against other sources of information available at the time. In addition to the problems of recall and courtesy bias, there is also the possibility that KZH weather and climate data and predictions have been consistent with other sources. In such a circumstance, the evaluation team will only be able to ask farmers ex post to assess how they would have weighed the sources of information were they different. 3. The sustainability of an initiative can only be determined ex post. The methods proposed for Evaluation Question 3 will assess factors that theoretically would contribute to the sustainability of the intervention. Further research will be done on the indicators of sustainability of capacity building efforts, and it is possible that there is no consensus on the indicators of sustainability for an intervention of this nature. 10. Evaluation Deliverables The evaluation team will be responsible for the following deliverables. Specific due dates will be proposed in the Evaluation Design Proposal to be prepared by the evaluation team. Deliverable Estimated Due Date 1. Evaluation Concept Paper, including preliminary methodological options for the evaluation. o/a 30 days from client approval of SOW 2. Evaluation Design Proposal, including description of the evaluation methodology, drafts of data collection instruments and a sampling plan, as relevant o/a 30 days from client approval to move forward with preparing Evaluation Design Proposal 3. Field debrief for USAID staff on preliminary study findings prior to team’s departure Following completion of field work but before the expat team returns to the U.S. or begins drafting the evaluation report 4. Draft Evaluation Report o/a 60 days from completion of field research 5. Oral presentation(s) of Draft Evaluation Report key findings, conclusions and recommendations for USAID and its invitees TBD 6. Final Evaluation Report including evaluation data sets, codebooks, etc. o/a 21 days following receipt of USAID feedback on Draft Evaluation Report 7. Debrief for UNDP staff and partners (tentative) As agreed following USAID approval of Final Evaluation Report All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved Evaluation Design Proposal. All qualitative and quantitative data will be provided in electronic format to USAID either by email or by thumb drive, depending on the size of the files being provided. All debriefs will include a formal presentation with slides delivered both electronically and in hard copy for all attendees. Final Evaluation Report – Performance Evaluation: CRW Project 70 Prior to the submission of the Evaluation Design Proposal, the evaluation team will discuss with USAID whether its preliminary dissemination plan for this evaluation indicates other deliverables that should be prepared, such as translation of evaluation materials into other languages and additional presentations or workshops. Such additions as agreed with USAID will then be included in the Evaluation Design Proposal. 11. Team Composition The evaluation will be delivered by a core evaluation team supported by technical and administrative U.S.-based evaluation and project management specialists. The core evaluation team will be composed of a Team Leader who is an Evaluation Specialist, one or two climate change Subject Matter Experts and up to two Local Research Specialists. Team Leader/Evaluation Specialist An evaluation Team Leader with extensive experience leading multi-disciplinary teams conducting field evaluations of complex projects will oversee the evaluation implementation process including field data collection, analysis and report preparation. The Team Leader should hold at least a master’s degree with at least 10 years of experience as an evaluation team leader or team member. Relevant experience and knowledge with agricultural or climate change programs is preferred, as well as prior experience in Central Asia, and specifically Kazakhstan. Fluent English and Russian are required. Subject Matter Experts Subject Matter Experts will provide expertise and guidance to the evaluation team on topics relevant to the evaluation including agricultural production, storage and distribution, meteorology, climate change and climate change policy. They should have familiarity with the relevant literature in their technical area. The specialists should hold advanced degrees with at least 10 years of experience in their technical sector, including experience working on evaluation teams. Prior experience in Central Asia is preferred. Local Research Specialists Depending on the intensity and breadth of research, the evaluation team will also include up to two Local Research Specialists who will contribute substantially to the data collection (interviews, site visits, etc.), data analysis and presentations/debriefs being conducted for the evaluation. They will provide country context for the evaluation and relevant subject matter knowledge or evaluation expertise, as required. They may also be asked to provide translation or logistical support, if needed by the evaluation team. Fluent Russian is required. 12. USAID Participation Regular communication between the evaluation team and the designated USAID Activity Manager for this evaluation will be essential to the successful execution of the evaluation activities. The evaluation team will keep USAID apprised of changes and developments that necessitate/require any significant decision-making or modification of the approved Evaluation Design Proposal. Possible USAID participation in the data collection phase of the evaluation will be determined prior to the start of field work. Final Evaluation Report – Performance Evaluation: CRW Project 71 13. Scheduling and Logistics The following Gantt chart provides a general overview of the anticipated timeframe for evaluation activities and deliverables. This schedule is assuming approval of this SOW in early February 2015, followed by approval of the Evaluation Design Proposal in March or early April, with the evaluation team preparation commencing immediately thereafter. The evaluation implementation is anticipated to run between April and July 2015, with approximately four weeks of data collection in Kazakhstan. The timing of field work related to Evaluation Questions 4 and 6 may be strongly influenced by the timing, duration and intensity of the Kazakhstan agricultural season. The evaluation team will investigate the impact of the agricultural seasons with CRW staff. Estimated CRW Performance Evaluation Timeline (2015) Task/Deliverable Feb Mar Apr May June July Aug Sept Oct Evaluation Concept Paper Evaluation Design Proposal Evaluation Preparation Field Work In-Country USAID Debrief Analysis & Report Writing Presentation of Draft Evaluation Report Draft Evaluation Report Final Evaluation Report The evaluation team will be responsible for procuring all logistical needs such as work space, transportation, printing, translation, and any other forms of communication. USAID will offer some assistance in providing introductions to partners and key stakeholders as needed, and will ensure the provision of data and supporting documents as possible. 14. Reporting Requirements The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing-evaluation￾reports). The final version of the evaluation report will be submitted to USAID and should not exceed 30 pages, excluding references and annexes. All members of the evaluation team will be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet, shown in the following text box below, along with USAID’s conflict of interest statement that they sign and return to the E3 Analytics and Evaluation Project Home Office where necessary before field work starts. Final Evaluation Report – Performance Evaluation: CRW Project 72 15. Budget The evaluation team will propose a notional budget in its Concept Paper for this evaluation, including cost implications of the methodological options proposed. A full detailed budget will then be prepared and included in the Evaluation Design Proposal for USAID’s approval. USAID EVALUATION POLICY, APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. Final Evaluation Report – Performance Evaluation: CRW Project 73 ANNEX B: OVERVIEW OF GENDER CONSIDERATIONS FOR THE KAZAKHSTAN AGRICULTURE SECTOR By law, in Kazakhstan, women and men have the same right to own and manage land, under Article 101 of the Land Code. Access to land is governed solely by civil law, and customary and religious laws have no standing. According to the official Convention on the Elimination of all forms of Discrimination against Women (CEDAW) report, women head 11% of all farms and agricultural processing businesses126. No quantitative data on women’s land ownership was found. The shadow report submitted to CEDAW Committee in 2006 notes that overall, women continued to experience discrimination in regard to access to land (especially in rural areas), in part because in order to obtain land for farming, the applicant must be able to prove that she has an agricultural qualification and experience of managing an agricultural business, which few women have. In addition, local officials may be reluctant to register land titles to women, even when they qualify impeding women’s access to land127. Furthermore, research by USAID found that in southern Kazakhstan interview respondents reported negative attitudes to women’s employment outside the home, and to women interacting with men from outside their household (e.g., government officials, or colleagues)128. The 2012 official CEDAW report notes that the government runs various training and support programs for farmers (including business training), and that women and men participate on an equal footing in these schemes. Women and men have the same equal rights to own and access property other than land. Unmarried women and men have the same property rights, as do married women and men. Joint communal property is the default property regime, unless an agreement is made between the spouses at the time of marriage. All transactions relating to the sale and management of property require the consent of both spouses. Women in unregistered marriages have no legally recognized rights to property owned jointly with their spouses. As such, their rights to property are not effectively protected. Women and men have the same legal right to access credit and bank loans in Kazakhstan. According to financial inclusion data held by the World Bank, 44% of women in Kazakhstan had bank accounts in 2011, as did 40% of men. In the same year, 13% of adults had taken out a loan with a financial institution (defined as a bank, credit union, microfinance institution, or another financial institution such as a cooperative); these data were not disaggregated by gender. According to the Microfinance Information Exchange, women accounted for 72.81% of recipients of micro-credit in Kazakhstan in 2012. The 2010 ADB Gender Assessment cited persistent gender gaps in key poverty reduction indicators, which highlighted the need to adapt poverty reduction programs more effectively, to address the needs of women as well as men. At the time, when Government of Kazakhstan’s (GOK) new programs for economic diversification and agricultural revival were being developed and delivered, ADB assessment called for equitable access to the new resources being made available to women through public investments. While there is no Ministry of Gender or similar body in Kazakhstan, gender issues are addressed at the highest level of the special National Commission on Women’s Issues and Family and Demographic status to the President, created in 1995. The commission annual releases its workplan, and the most recent 126 http://genderindex.org/sites/default/files/datasheets/KZ.pdf 127 Idem. 128 Idem. Final Evaluation Report – Performance Evaluation: CRW Project 74 workplan for 2015 has extensive list of activities to be applied in GOK agencies to promote gender equality129. 129 http://www.akorda.kz/upload/nac_komissiya_po_delam_zhenshin/2.2%20%D0%9F%D0%BB%D0%B0%D0%BD%20%D1%80%D0 %B0%D0%B1%D0%BE%D1%82%D1%8B%20%D0%9D%D0%B0%D1%86%D0%B8%D0%BE%D0%BD%D0%B0%D0%BB%D1%8C %D0%BD%D0%BE%D0%B9%20%D0%BA%D0%BE%D0%BC%D0%B8%D1%81%D1%81%D0%B8%D0%B8%20%D0%BF%D0%BE %20%D0%B4%D0%B5%D0%BB%D0%B0%D0%BC%20%D0%B6%D0%B5%D0%BD%D1%89%D0%B8%D0%BD%20%D0%B8%20 %D1%81%D0%B5%D0%BC%D0%B5%D0%B9%D0%BD%D0%BE- %D0%B4%D0%B5%D0%BC%D0%BE%D0%B3%D1%80%D0%B0%D1%84%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE %D0%B9%20%D0%BF%D0%BE%D0%BB%D0%B8%D1%82%D0%B8%D0%BA%D0%B5%20%D0%BF%D1%80%D0%B8%20%D0% 9F%D1%80%D0%B5%D0%B7%D0%B8%D0%B4%D0%B5%D0%BD%D1%82%D0%B5%20%D0%A0%D0%B5%D1%81%D0%BF%D 1%83%D0%B1%D0%BB%D0%B8%D0%BA%D0%B8%20%D0%9A%D0%B0%D0%B7%D0%B0%D1%85%D1%81%D1%82%D0%B0 %D0%BD%20%D0%BD%D0%B0%202015%20%D0%B3%D0%BE%D0%B4.pdf Final Evaluation Report – Performance Evaluation: CRW Project 75 ANNEX C: DATA COLLECTION INSTRUMENTS Key Informant Interview Guide: Institutional Respondents Instructions This Interview Guide applies to key informant interviews with institutional stakeholders and is modular in nature. It is arranged in sections by topic, with sectional notations indicating the categories of informants (e.g., Kazhydromet Managements, UNDP Project Staff, etc.) for which the subsequent interview questions will likely be relevant. With each section, recommended questions have been provided that address the information that we will seek from the classes of key informants. Following most recommended questions are a series of prompts that the interviewer may wish to consider to solicit follow-up information. Sometimes, additional guidance on specific questions is provided in brackets following the question for consideration by the interviewer. The Interview Guide is not intended to be adhered to strictly and interviewers are encouraged and expected to deviate from specific questions (and possibly topics) and prompts with relevant follow￾up questions. Further, the interviewer is not limited by these sectional notations and may deem topics appropriate to individual key informants even where not indicated by the notation. Within each section, the individual questions are followed by a notation that references the evaluation framework and indicates the EQ to which that question is directed (e.g., EQ3). This should facilitate faster and more efficient analysis of interview transcripts and responses. Interviewers should ensure that they have read the Interview Guide fully and are familiar with the topic, sectional notations and individual questions prior to initiating any interviews. Final Evaluation Report – Performance Evaluation: CRW Project 76 Section A: Introduction Hello, I am _________________ and my organization has been contracted by USAID to evaluate the Improving Climate Resilient Wheat (CRW) project implemented by the UNDP with funding by USAID. We are carrying out this evaluation to assess how well the program is meeting the needs of internal and external stakeholders like you and to find out how various aspects of the project have been working. This interview is voluntary; you can withdraw at any time, either before or during the interview. There are no right or wrong answers. We want to hear your thoughts, based on your experience and your involvement with the project. The interview should not take more than 60 minutes to complete. Following the interview, we may want to contact you again in a few days to confirm or clarify some of the information you have given us. The information you provide us will be important to understand the achievements of the CRW program and we may wish to cite this discussion in support of our findings. However, if you would like to remain anonymous, you may inform us of this now or at any time in the next week following this interview. If so, we will not attribute any information that we receive to you, either in any report, transcript or notes from this discussion, or any conversations that we may have with persons outside of our evaluation team. Does the respondent wish to remain anonymous? Yes  No  If you have no objection, we would like to record this discussion, but wish to assure you that all recordings and notes will remain confidential and will be kept in a safe place. The recordings will be used for analysis purposes only. Do you have any other questions about the study or this interview? Name of Interviewee: Organization/Agency: Title/Role/Position: Date of Interview: The study has been explained to me. My questions have been answered satisfactorily. I understand that I can change my mind at any stage and it will not affect me in any way. Do you agree to participate in this study (automatic if interview is scheduled)? Yes  No  RESPONDENT: ______________ (INITIALS) DATE: _______________ Final Evaluation Report – Performance Evaluation: CRW Project 77 Section B: Background [ALL RESPONDENTS UNLESS OTHERWISE NOTED] “I would like to start off by speaking to you a little bit about your institution and the kinds of support that your institution provides to Kazakhstan farmers, but also to other agencies within Kazakhstan that you may collaborate with. After that, I would like to get your opinions on the support that CRW has provided to your institution and other institutions to the extent that you are aware.” B1. Can you please tell us a little bit about yourself and your role here at _______?  Job Title  Job Responsibilities B2: Can you tell us a little bit about the role of your organization? What are its responsibilities?  Overall objectives of the Institution  Objectives of the department in which the respondent works?  Can the respondent describe the structure, size and funding of the institution? B3: What kinds of services does this institution provide that are relevant to farmers and specifically wheat farmers?  Does the institution provide information to farmers? (E.g., climate prediction, crop yields, etc.)  Does the institution provide training to farmers? (agricultural training)  Does the institution provide funding or subsidies to farmers?  Others? B4: Does the institution work closely with other national agencies or institutions to provide support to Kazakhstan agriculture and farmers? How? [EQ4] B5: What issues and obstacles do you think will be important in the future for Kazakhstan’s development, and more specifically the wheat/agricultural sector?” [Depending on the institution, this could be a very short or a very long answer. It may not be appropriate to ask this question for MinAg officials as the breadth of the answer may be considerable in the context of a short interview.] [EQ7] [B6, B7 and B8 are follow-ups to Question B5 and focus only on meteorological data. It may not be appropriate to ask of all stakeholders, but will likely be appropriate for KZH stakeholders] B6: In reviewing the information about the CRW project and wheat farming in Kazakhstan, it seems like one of the major challenges for farmers is getting accurate and reliable weather and climate predictions to make agricultural decisions. Would you agree that this is a challenge for farmers in Kazakhstan, and if so, could you tell us a little bit more about the difficulties that farmers face? [EQ2,5] B7: Can you describe for me briefly how farmers get their weather and climate data and from which sources? (EQ2,5)  MinAg bulletins?  Internet weather sites?  Foreign sources  Almanacs  Radio, TV, farmer-to-farmer, extension agents, B8: To what extent do you think farmers rely on weather and climate data produced by Kazhydromet to make agricultural decisions? (EQ5) B9: Are you familiar with the CRW project? If yes, can you describe what interactions your organization and you yourself have had with CRW? Final Evaluation Report – Performance Evaluation: CRW Project 78 Section C: Agrometeorological Support [Primarily Kazhydromet, NSI, Ministry of Agriculture] C1: Has your organization received specific training and support on improved agrometeorological practices from the CRW project, including through its partner program CCRD? (EQ1,2) C2: Can you describe as fully as you can the kind of support that the CRW project has provided? (EQ1,2)  Collection and analysis of meteorological data (air temperature, relative humidity, soil moisture)  Methods of seasonal climate forecasting  Disseminating weather and climate related data to farmers  Facilitating the use of weather and climate data by other institutions [The following questions should be asked for each kind of support that was referenced in the preceding question – Perhaps lead in with: “I would like to speak a little bit more about CRW’s support to…”] C3: Did you participate in any of the trainings or were you engaged with the support provided by CRW? If yes, can you give us your opinion of whether the support was well-delivered? (EQ1,2,3)  Was it relevant to the institution’s needs?  Do you feel the training is relevant to what the project is trying to achieve (climate resilient wheat)  Was the institution involved in developing the trainings or training schedule?  Was the support provided long enough?  Were the trainings effective in increasing skills?  If there are other training they would find valuable with regard to climate resilient wheat? For instance, other topics or going more in-depth on a topic?  Was any technological support sufficient? C4: Can you describe for me how your institution did this work before you received CRW support? (EQ1,2)  What were the standard practices before CRW support?  Does the respondent feels these were adequate or insufficient, and if insufficient, how and why? C5: How has the institution applied the training and support provided by CRW? (EQ1,2)  Does the institution apply the practices? (Yes/No/Partially)  If yes, how has this changed how the institution does its work? - Can it do its work faster, more efficiently, more accurately? Why? - Does it work better with other Kazakhstan institutions? - Does it provide a better service to its beneficiaries and constituents?  If no, why doesn’t it apply the practices? - Lack of capacity - Lack of management buy-in - Time/Cost issues C6: Do you think that any of the support provided by CRW or CCRD has contributed to a better relationship and better information sharing between your institution and other Kazakhstan institutions? (EQ4) C7: Do you think that your institution will continue to apply what you have learned from CRW? (EQ3)  Do you have plans to expand the trainings for new staff?  Is there a commitment from other managers and staff to apply these new methods?  Do you think there will be any specific challenges in applying these new methods?  Does the policy environment support the continued use of the agrometeorological practices?  Was the CRW support to KZH and NSI consistent with relevant sector policies?  Have key local stakeholders actively participated design of the capacity building interventions?  Have key local stakeholders been clearly supportive of the adoption of different practices? Final Evaluation Report – Performance Evaluation: CRW Project 79  Do the new agrometeorological practices meet a clearly expressed need on the part of key stakeholders? Ultimate beneficiaries?  Did the CRW project include sustainability of agrometeorological practices as a project objective?  Has the CRW project assessed the capacity of the relevant institutions to sustain the practices?  Has a sustainability monitoring framework been proposed in the CRW design or implemented as part of the project?  Have the needs for ongoing training been assessed and provided for by the CRW project?  Has a training strategy been developed and described that addresses sustainability issues?  Will there be ongoing and recurrent costs associated with the continued use of the agrometeorological practices?  If so, are recurrent costs likely to be met?  Have the host institutions made a commitment to meeting recurrent costs?  Is the new technology (e.g., IRI software, geoportal) provided of appropriate quality and responsive to stakeholder needs?  Have training and maintenance requirements been specifically assessed and addressed? Section D: Climate Resilient Adaptation [Ministry of Agriculture, KazAgroInnovation, Agricultural Extension Centers] D1: Has your organization received specific training and support on climate change adaptation from the CRW project and UNDP? (EQ6) D2: Can you describe as fully as you can the kind of support that the CRW project has provided? (EQ6)  Providing training to extension center staff on climate change  Developing training modules for farmers on climate change and responding to climate change  Support the development of experimental plots to test and demonstrate adaptation practices that can improve wheat yields  Other… [The following questions should be asked for each kind of support that was referenced in the preceding question – Perhaps lead in with: “I would like to speak a little bit more about CRW’s support to…”] D3: Did you participate in any of the trainings or were you engaged with the support provided by CRW? If yes, can you give us your opinion of whether the support was well-delivered? (EQ6)  Was it relevant to the institution’s/ farmer’s needs?  Was the institution involved in developing the trainings or training schedule?  Was the support provided long enough?  Were the trainings effective in increasing skills of staff? Of Farmers?  Was any technological support sufficient? D4: Can you describe a little bit about what kinds of farmers have participated in the CRW￾supported trainings or toured the demonstration plots supported by CRW? Are they representative of the average Kazakhstan wheat farmer? If not, why not? If yes, why yes? (EQ6)  Owners of big farms versus little farmers?  Rich versus poor farmers?  Farmers that live near the extension centers as opposed to farmers that live far from extension centers? D5: Have farmers been enthusiastic about the trainings/support about climate change adaptation offered as a result of CRW? If yes, why? If no, why not? (EQ6, 7)  How many farmers have participated in these trainings or reviewed the demonstration plots?  Have farmers expressed an interest in applying some of the adaptation approaches? D6: Do you know if farmers have tried to apply the adaptation techniques exhibited through CRW￾supported trainings or demonstration plots? (EQ6)  If yes, what has been the result? Final Evaluation Report – Performance Evaluation: CRW Project 80  If no, why not? D7: What do you think are the main challenges in getting farmers to think more about the impacts of climate change and to take steps to mitigate these impacts? (EQ6) D8: Do you have any recommendations about steps that could be taken to promote climate change adaptation by farmers in the future? (EQ6) Section E. Program Design and Implementation [All respondents] “I would like to ask you a couple of questions about the delivery of the CRW program specifically.” E1: How relevant the CRW approach is with regard to resiliency in the wheat sector? Not just have they accomplished what they set out to but are they addressing what should be addressed. E2: What have been the strengths and/or weaknesses of the UNDP approach to delivering the CRW project? (All)  Did it communicate well with partners?  Did it engage partners in decision-making and consultation? E2 Can you identify any key opportunities the project may be able to take advantage of or align with over the next year? (All)  Collaboration with external stakeholders  Buy-in from key stakeholders  Communication Section F. Challenges and Recommendations [ALL RESPONDENTS] “Thank you very much for all of the information that you have provided to us. We are very thankful for your time. If you don’t mind, I would just like to ask you a few more general questions about the challenges for wheat farmers in Kazakhstan and what could be done in the future to address these challenges.” F1: We have already discussed this somewhat, but I just wanted to get your general opinions on what the main challenges are to increasing yields for wheat farmers in Kazakhstan. Do you think you could outline what you think the three most significant challenges are to increasing Kazakh wheat production? (EQ7) F2: If you could provide any recommendations for the CRW project or any donor project that is interested in responding to the challenges of wheat farmers, what would you recommend? (All) Wrap-up “I want to thank you again for your time in meeting with me. If you have any questions, please don’t hesitate to get into contact with the evaluation team. We want this to be a transparent and collegial process. Also, if there are any clarifications that you would like to make or if there is anything else that comes to mind that you would like to convey to us, we would be very happy to hear from you. Thank you again.” Final Evaluation Report – Performance Evaluation: CRW Project 81 Individual Interview Guide: Farmer Beneficiaries Instructions This Interview Guide applies to key informant interviews with farmer beneficiaries. It is arranged in sections by topic with recommended questions. Following most of the questions are a series of prompts that the interviewer may wish to consider to solicit follow-up information. Sometimes, additional guidance on specific questions is provided in brackets following the question for consideration by the interviewer. The Interview Guide is not intended to be adhered to strictly and interviewers are encouraged and expected to deviate from specific questions (and possibly topics) and prompts with relevant follow￾up questions. Further, the interviewer is not limited by these sectional notations and may deem topics appropriate to individual key informants even where not indicated by the notation. Interviewers should ensure that they have read the Interview Guide fully and are familiar with the topic, sectional notations and individual questions prior to initiating any interviews. Final Evaluation Report – Performance Evaluation: CRW Project 82 Section A: Introduction Hello, I am _________________ and my organization has been contracted by USAID to evaluate a United Nations Development Project called Improving Climate Resilient Wheat (CRW) project. This project has worked with various Kazakhstan institutions including Kazhydromet and several agricultural extension centers. We want to assess how well the program is meeting the needs of intended beneficiaries, like you. Your responses will help us make recommendations on the ways to improve the reliability and accuracy of the weather and climate information provided to farmers, and as well as improve the quality of capacity building activities going forward. This interview is voluntary; you can withdraw at any time, either before or during the interview. There are no right or wrong answers. We want to hear your thoughts and opinions on the challenges of wheat farming in Kazakhstan and also about your experiences with some Kazakhstan institutions. The interview should not take more than 60 minutes to complete. Following the interview, we may want to contact you again in a few days to confirm or clarify some of the information you have given us. The information you provide us will be important to understand how well UNDP has supported Kazakhstan agriculture so that we can improve programs in the future. We may wish to cite this discussion in our report to USAID. However, if you would like to remain anonymous, you may inform us of this now or at any time in the next week following this interview. If so, we will not attribute any information that we receive to you, either in any report, transcript or notes from this discussion, or any conversations that we may have with persons outside of our evaluation team. Does the respondent wish to remain anonymous? Yes  No  If you have no objection, we would like to record this discussion, but wish to assure you that all recordings and notes will remain confidential and will be kept in a safe place. The recordings will be used for analysis purposes only. Do you have any other questions about the study or this interview? Section B: Background Name of Interviewee: Date of Interview: Age of Respondent: Gender of Respondent: KZ Region The study has been explained to me. My questions have been answered satisfactorily. I understand that I can change my mind at any stage and it will not affect me in any way. Do you agree to participate in this study (automatic if interview is scheduled)? Yes  No  RESPONDENT: ______________ (INITIALS) DATE: _______________ Final Evaluation Report – Performance Evaluation: CRW Project 83 Section B: Background “I would like to start off by speaking to you a little bit about yourself and your experiences farming in Kazakhstan and specifically farming wheat. After that, I would like to get your opinions on some of the work that Kazakhstan institutions such as Kazhydromet and the agricultural extension centers are doing.” B1. How long have you been a farmer? B2: What kind of farming do you primarily do? Examples/Prompt  Animal Farming/ Crop Farming  What kinds of crops do you primarily farm? B3: How large is your farm in terms of hectares? How far away is your farm from demonstration plot? B4: Do you employ seasonal workers on your farm? Yes  No  If yes, how many workers do you employ during the busiest time of year? ______ B5: Speaking specifically of crop farming, can you describe for us what are the main challenges that you face on an annual basis in crop production? Examples/Prompt  Weather and climate  Drought  Getting the right seeds  Getting agricultural workers  Lack of access to finance  Grain storage B6: in the last 5 years:  What crops did you plant? Same or different?  Do you plan to change them? Why have not you change them?  Have you had problems losing much of your harvest? If yes, what were the reasons for crop losses and/or lower yields? B6: How do you decide each year what crops you are going to plant? Section C: Weather and Climate “The next questions that I have relate primarily to weather and climate and how you make farming decisions, including specifically when to plant and harvest crops.” C1: When you are thinking about when to plant or harvest crops, what are the main weather and climate issues that you consider? Examples/Prompts:  Floods  Drought  Late frosts  Other, please specify C2: Can you describe how you decide when to plant your crops? What are the factors that you consider? Who and when do you consult? Final Evaluation Report – Performance Evaluation: CRW Project 84 C3: Can you describe how you decide when to harvest your crops? What are the factors that you consider? Who and when do you consult? C4: To what extent do you rely on weather and climate forecasts in making your planting and harvesting decisions? [If yes, go to C5] If not, why? Examples/Prompts:  Do not trust  Not right information  Information is not distributed  Do not know where to find information C5. Where do you get your weather and climate information that you rely upon? Examples/Prompts:  MinAg bulletins  Internet  Radio/television C6: Do you use the weather and climate forecasts prepared by Kazhydromet? How much reliance to you place on the weather and climate forecasts prepared by Kazhydromet? Why? Have something been changed recently? C7: Do you believe Kazhydromet weather and climate forecasts are more accurate than 2 years ago, less accurate or the same? C8. What information can be improved to be more useful to you? C9. How much would access to accurate and reliable seasonal climate forecasts improve wheat production? (Very much; Somewhat; Little) C10. What benefits would you expect from using weather/climate data/information provided by Kazhydromet Section D: Climate Change Resilience (Familiarity with the CRW Project) [This section is intended to assess the farmer’s familiarity with the CRW project. The scoping interviews suggest that the familiarity of the CRW project will vary depending on the center that they attended.] “As we noted at the beginning, we are evaluating the UNDP Improving Climate Resilience Project, which has sponsored various trainings and activities at the agricultural extension centers. In the course of these trainings, you may have been made aware of UNDP’s role, but may not have been.” D1: Are you familiar with the CRW Project implemented by UNDP? Yes  No  Any additional comments: [If NO, Continue to section E] D2: Do you know whether you attended any training sessions or visited any demonstration plots associated with the CRW Project? Yes  No  Any additional comments: [If NO, Continue to section E] [The respondent should answer this question in their own words. However, some prompts are provided below to clarify their answer. If the respondent cannot name a specific activity, continue to section E] D3: Which CRW-supported activities have you been involved in/participated in? Examples/Prompts  Field farming schools (demonstrations) set up to show the adaptation techniques Final Evaluation Report – Performance Evaluation: CRW Project 85  Attended seminars organized at the agro-centers  Visited a successful farm that had/was implementing new or a combination of best known practices as part of the Project’s activities  Comparing of a variety of agronomic experiments, side by side, with traditional practices in order to see the difference  Comparing the effectiveness of crop production with other annual crops by setting up comparative trials with the help of the Project  Demonstration of different cultivation times compared with the local traditional sowing season of different crops.  Education of innovative agro-technologies on cultivation of spring wheat crop  Had an wheat or agricultural specialist come and help with a problem or make recommendations for the new crop Types of Activities Respondent Provided: (information to be used in following questions) a. b. c. d. [Answer question D4 for EACH activity that the respondent identified.] D4(a): Can you describe a little more about this activity?  What was taught?  Was the training long enough?  Were the instructors knowledgeable?  Was it useful?  Maybe will be useful in future?  How it can be improved to be more useful? D5: After the activity, did you feel you had learned more about the potential impacts of climate change? Examples/Prompt:  What did you learn that was useful to you?  Did the activity make you more or less concerned about the impacts of climate change or did it not change your opinion? D6: After the activity, did you feel you had learned more about steps that you could take to mitigate the impacts of climate change? What did you learn that was useful to you? D7: As a result of what you learned, did you change your planning or farming approaches? If so, how? Examples/Prompts:  Crop diversification  Seed diversification  Application of new agricultural techniques D8: IF the Respondent indicated that they did visit a demonstration plot showcasing adaptation techniques, but did not adopt any of these, the follow question is “Why not?” [If the respondent indicates that they viewed a demonstration plot showcasing one of the 22 UNDP adaptation techniques and applied an adaptation technique, Annex I of this instrument may be shown to them to clarify with them which adaptation technique they applied.] D9. What benefits would you expect from adaptation of these practices? Or you already experience Final Evaluation Report – Performance Evaluation: CRW Project 86 some benefits – what they are? Section E: Climate Change Resilience (Not Familiar with CRW Project) “The CRW project that we are evaluating is focused on mitigating the impacts of climate change for Kazakhstan farmers. The next series of questions that I will ask you relate to climate change and adaptation techniques to deal with climate change.” Training/Workshops E1: In the last 18 months, have you participated in any trainings, workshops or seminars at the agricultural extension center about the potential impacts of climate change to farming in Kazakhstan? [If yes, continue to E2. If no, continue to E6] Yes  No  E2: Can you describe the training, workshop or seminar in which you participated? Examples/Prompts:  What was taught?  Was the training long enough?  Were the instructors knowledgeable?  Was it useful?  Maybe will be useful? E3: After the training, workshop or seminar, did you feel you had learned more about the potential impacts of climate change? What did you learn that was useful to you? E4: After the training, workshop or seminar, did you feel you had learned more about steps that you could take to mitigate the impacts of climate change? What did you learn that was useful to you? E5: As a result of what you learned at the training session, did you change your planning or farming approaches? If so, how? Examples/Prompts:  Crop diversification  Seed diversification  Application of new agricultural techniques E6. What benefits would you expect to receive from having new knowledge? Or you already experience some benefits – what they are? Demonstration Plots E7: In the last 18 months, did you visit any demonstration plots that were sponsored by the agricultural extension centers and which focused on agricultural techniques to cultivate more climate resilient wheat? [If yes, continue to next questions. If no, continue to Section F] Yes  No  Additional Comments: E8: After visiting the demonstration plot, did you feel you had learned more about steps that you could take to mitigate the impacts of climate change? [If the respondent indicates that they viewed a demonstration plot showcasing one of the 22 UNDP adaptation techniques and applied an adaptation technique, Annex I of this instrument may be shown to them to clarify with them which adaptation technique they applied.] E9. As a result of visiting the demonstration plot, did you adopt any of the climate change adaptation techniques that were taught? If yes, what adaptation technique did you adopt? IF no, why not? [If the respondent indicates that they viewed a demonstration plot showcasing one of the 22 UNDP adaptation techniques and applied an adaptation technique, Annex I of this instrument may be shown to them to clarify with them which adaptation technique they applied.] E10. What benefits would you expect to receive from what you saw and learned on the demonstration Final Evaluation Report – Performance Evaluation: CRW Project 87 plot? Section F. Challenges and Recommendations “Thank you very much for all of the information that you have provided to us. We are very thankful for your time. If you don’t mind, I would just like to ask you a few more general questions about the challenges for wheat farmers in Kazakhstan and what could be done in the future to address these challenges. We have already discussed this somewhat, but I just wanted to get your general opinions on what the main challenges are to increasing yields for wheat farmers in Kazakhstan.” F1: Do you think you could outline what you think the three most significant challenges are to increasing Kazakh wheat production? F2: Can you list three main obstacles for increase of income from wheat production in Kazakhstan? F2: If you could provide any recommendations for the CRW project or any donor project that is interested in responding to the challenges of wheat farmers, what would you recommend? Wrap-up “I want to thank you again for your time in meeting with me. If you have any questions, please don’t hesitate to get into contact with the evaluation team. We want this to be a transparent and collegial process. Also, if there are any clarifications that you would like to make or if there is anything else that comes to mind that you would like to convey to us, we would be very happy to hear from you. Thank you again.” Final Evaluation Report – Performance Evaluation: CRW Project 88 ANNEX I: Adaptation Techniques Evaluation of spring wheat cultivation in depending of soil tillage deep plough with using fertilizers, different previous crops in small scale rotation, and different dates of planting:  Rational use of mineral fertilizer with tillage by 25cm by the planting period of 10 of May.  Rational use of organic fertilizers with deep plough (65-70cm) by the planting period of 10 of May.  Cultivation of wheat and alfalfa used as green manure technology and organic fertilizers by ploughing depth 25 cm and planting period on 25 of May.  Cultivation of wheat with predecessor crop chickpea and corn, all other technologies are per standard. The planting period is 05 June  Deep plough by 70 cm, using drought wheat seed varieties. Using both organic and mineral fertilizers. Planting time is 30 May Spring wheat cultivation in depending of different elements of soil protective and water accumulation techniques  Wheat cultivation with no tillage.  Wheat cultivation with minimum tillage after black fallow.  Wheat seed cultivation by application of organic and mineral fertilizers with no tillage. Planting time is June 10.  Wheat cultivation on minimum tillage.  Small scale rotation wheat plus legumes (sainfoin, lupin). Planting period. 10 of June.  Wheat cultivation on the plots where the snow was accumulated or collected through half straw left on the field  Cultivation of early ripening varieties of wheat after black farrow, using both organic and mineral fertilizers, planting period is 15 of June.  Cultivation of wheat variety "Astana" after the oil crops. No tillage and planting period is 05 June.  Cultivation of wheat variety "Almaty" after the oil crops. No tillage and planting period is 05 June.  Cultivation of wheat variety "Astana" after the legume crops. No tillage and planting period is 05 June.  Cultivation of wheat variety "Almaty" after the legume crops. No tillage and planting period is 05 June  Cultivation of wheat variety "Omsk" after the legume crops. Mineral fertilizers were used but no organic fertilizers. Tillage 25 cm and planting period is 07 June.  Cultivation of wheat variety "Astana" after the legume crops. Mineral fertilizers were used but no organic fertilizers. Tillage 25 cm and planting period is 05 June.  Cultivation of wheat variety "Almaty " after the legume crops. Mineral fertilizers were used but no organic fertilizers. Tillage 25 cm and planting period is 05 June.  Cultivation of wheat variety "Astana" after the oil crops. Mineral and organic fertilizers were used. Tillage 65 cm and planting period is 01 June  Cultivation of wheat variety "Astana" after the oil crops. Deep tillage 65 cm and planting period is 01 June.  Cultivation of wheat variety "Omsk" after the legume crops. Both Mineral and organic fertilizers were used. Tillage 65 cm and planting period is 01 June. Final Evaluation Report – Performance Evaluation: CRW Project 89 Focus Group Discussion Guide: Farmers TO BE FILLED OUT BY FOCUS GROUP PARTICIPANTS MODERATOR: Read this out loud at the beginning of the interview. We will be recording this interview so that we can make sure we accurately document what you are telling us, review the tapes later, and not forget anything that was said. While we may share the opinions that are expressed today with our client or with researchers who are interested in these types of projects, we will not identify you or anyone else as the person who shared those opinions. Your identity will be kept confidential and we will ensure that it will not be possible for you to be identified by any information provided in our reporting. So please feel free to speak your mind and Your participation is completely voluntary and you can choose to not answer any question or stop participating at any time. However, we want to remind you that your participation is very important to make development programs more efficient and tailored to your needs. This discussion will last approximately one hour. Do we have your permission to record the interview? Yes  No  Focus group participants will put their signatures on the list of participants that they agree to the interview being recorded. _________ (INITIALS) All recordings and notes will remain confidential and will be kept in a safe place. The recordings will be used for analysis purposes only and will not be heard by anyone outside the evaluation team. Your name and personal information will not be linked to any quotes in any public reports unless you request this. A list of interviewees will be included in an annex, however, if you do not wish to be included, please indicate that below:  Include my name in the Annex  Do NOT include my name in the Annex Do you agree to participate in this study (automatic, if interview is scheduled)? Yes  No  The study has been explained to me. My questions have been answered satisfactorily. I understand that I can change my mind at any stage and it will not affect me in any way. Name of Participants Focus group participants will put their signatures on the list of participants which indicates that they agree to be interviewed. If you have further questions about this evaluation you can contact Svetlana Negroustoueva at snegroustoueva@onlinedts.com. [For the interviewer/administrator to fill in after interview] I have followed the agreed evaluation protocol in order to obtain informed consent from the participant. S/he understands the nature and purpose of the study, as far as can be ascertained, and Final Evaluation Report – Performance Evaluation: CRW Project 90 consents to participate in it. The respondent has been given the opportunity to ask questions, which have been answered satisfactorily. Evaluator’s Name: ____________________ Signature: _______________________ DATE ______ Final Evaluation Report – Performance Evaluation: CRW Project 91 TO BE FILLED OUT BY FOCUS GROUP PARTICIPANTS Moderator: A copy of this page is provided to each respondent. Each respondent hands this page to the moderator at the end of the discussion. 1. How large is the farm on which you own or work? ________ hectares 2. Are you the primary owner of the farm?  Yes  No 4. How far is your farm from the extension center you are now? ________rv 5. What is your gender?  Male  Female  Prefer Not to Answer 6. How old are you? __________ 3. What kinds of crops do you predominantly grow? Circle all that apply  Wheat  Barley  Cotton  Rice  Other (please specify): _________ Final Evaluation Report – Performance Evaluation: CRW Project 92 TO BE RECORDERD BY MODERATOR INTRODUCTION: EXPERIENCES WITH CRW-SUPPORTED ACTIVITIES Theme 1: Use of Climate Information for Farmers 1. We would like to discuss your experience with using the weather and climate information: a. From what source do you receive this information? Does the information satisfy your needs? How can it be improved? 1) Which CRW activities have you taken part in? (MODERATOR: mark all those that come up in the discussion) Activity List: DO NOT READ Mentioned by Respondents Prompted a. Field farming schools (demonstrations) set up to show the adaptation techniques   b. Attended seminars organized at the agro￾centers   c. Visited a successful farm that had/was implementing new or a combination of best known practices as part of the Project’s activities   d. Comparing of a variety of agronomic experiments, side by side, with traditional practices in order to see the difference   e. Comparing the effectiveness of crop production with other annual crops by setting up comparative trials with the help of the Project   f. Demonstration of different cultivation times compared with the local traditional sowing season of different crops.   g. Education on innovative agro-technologies on cultivation of spring wheat crop   h. Had a wheat or agricultural specialist come and help with a problem or make recommendations for the new crop   i. Reports of ripening of wheat crops, best sowing times for crops   j. Agrometeorological forecasts during the period of crop harvesting (advisory forecast)   2) Have you experienced any challenges due to weather and climate in the past five years? What were they? How did you address them? Final Evaluation Report – Performance Evaluation: CRW Project 93 b. Has the quality of weather and climate data improved since 2011? Has the way/means of receiving this information convenient to you? c. Has having access to this information affected your daily life, the way you plant your crops? And might affect crop productivity and/or net farm income? If so, how? d. Have you used weather and climate information to schedule planting and harvesting or other purposes? e. To what extent access to accurate and reliable seasonal climate data/information increase wheat production? f. What are women and men’s roles on the farm? 2. How do you think the Kazhydromet climate reporting service could be improved? a. What topics or types of information would you like to receive more of? b. What are the best ways/means for you to receiving information? c. What additional information would be the most useful for you? 3. Which climate or environmentally related issue, if any, would you say has the biggest impact on your daily life? Theme 2: Capacity Building Around Adaptation to Climate Change – New Agricultural Techniques 1. What trainings and activities supported by the CRW-project have you participated in since 2011? a. How was your experience of attending trainings/seminars/visiting sample plots? b. How were you invited/selected? Do you think participants are similar in any ways? Are there groups that are missing? [Moderator prompt: women, farmers with less land, ethnic groups, etc.? c. Did you attend all the events you wanted to attend? If not, why? d. Do you feel that they provided you with any useful information? What were the most and least useful topics covered during trainings? e. Do you think the methods employed to raise awareness were effective? f. Did you receive adequate support in terms of teaching materials, facilities, etc? Were distributed materials helpful? g. Were there any topics you would have liked to learn more about, or which were not covered? h. How do you think the trainings/sample demonstration plots could be improved? 2. Have you been able to apply the knowledge and skills received about any of the methods that you learned about on your own land? Why or why not? What benefit(s) did you receive or expect to receive from use of new methods? 3. Do you think there is a difference in adoption of new agrarian techniques between women and men? If YES, why do you say that? 4. What challenges do you expect as on adaptation new methods? 5. What are the greatest challenges for the future for you and your farm? Final Evaluation Report – Performance Evaluation: CRW Project 94 Telephone Survey Questionnaire: Farmer Beneficiaries Introduction BRIF research group company is conducting a survey on behalf of a USAID evaluation of the UNDP program to support climate resilient wheat production in Kazakhstan. Your name and contact information were provided to us by the organizers a UNDP-supported training sessions or other event that you attended within the last 18 months at the agricultural extension center. If you have some time, we would really like to speak to you about the challenges that you face as a farmer in Kazakhstan. We would like to assess how well the UNDP program is relevant to and has met its target beneficiaries such as yourself. Your responses will help us make recommendations on the ways to improve the quality of capacity building activities going forward. Instructions Please read question carefully in the first column and answers to the questions in the second column. Check appropriate answer (underlining, using “bold”) or type it next to the question Great. Then follow skip patters and instructions in the third column. I just want to provide a quick explanation of how the survey is being conducted.  This interview is voluntary.  There are no right or wrong answers.  The interview should not take more than 30 minutes to complete. Your responses will stay anonymous, and collected information will be used for analysis purposes only. Name of Respondent: Gender (Please circle one): Male/ Female Interviewer Name: Age: Section A: Background “We would like to start off by asking you a little bit of background information about yourself so that we can better understand the challenges faced by farmers like yourself. Question Response Instruction A0. Are you a farmer or involved in agricultural production? 1. Yes A1 2. NoSection F -89. Refused If yes, go to A1. If no, go to Section F. A1. Can you tell us how many years you have been farming, not necessarily in the current farm? ___ ___ -88. Don’t know -89. Refused Go to A2 A2. How old are you? ___ ___ -88. Don’t know -89. Refused Go to A3 A3. Approximately how large is your farm (in hectares)? _____________ hectares Go to A4 Final Evaluation Report – Performance Evaluation: CRW Project 95 A4. What is your position at your farm? 1. Owner 2. Manager 3. Seed agronomist/expert 4. Agronomist (other) 5. Mekhanizator/engineer 6. Tallyman 7. Brigadier Arable 8. Technologist 9. Accountant 10. Economist 11. Purchasing manager 12. Livestock Brigadier 13. Other (specify) 88. Don’t know 89. Refused Go to A5 A5. Do you employ seasonal workers on your farm? 3. Yes A6 4. NoA8 -88. Don’t know -89. Refused If yes, go to A6. If no, go to A8. A6. Approximately how many seasonal workers did you employ during the peak of the agricultural season last year? Go to A7 A7. Approximately what proportion of them was women? 1. More than half 2. Less than half 3. Equal 4. None 88. Don’t know 89. Refused Go to A8 A8. What kind of farming do you primarily engage in? (INTERVIEWER: Please read options 1-3 out loud, and circle the best option) 1. Livestock/ Dairy  END 2. Crop cultivation A9 3. Both A9 -88. Don’t know END -89. Refused END If livestock/ dairy, end interview. If crop or both, go to A9. A9. What kinds of crops do you predominantly grow? Circle all that apply 1. Wheat 2. Barley 3. Cotton 4. Rapeseed 5. Peas 6. Sunflower 7. Buckwheat 8. Oatmeal 9. Linen 10. Other (please specify): _________ -88. Don’t know -89. Refused Go to A10. A10. What percentage of your crop was comprised of wheat last year, from August 2014 to August 2015, in %? ___ ___ % -88. Don’t know -89. Refused Go to A11. A11. Can you please tell us what rayon your farm is located in? Name of District_________ Go to A12 A12. Can you please tell us how far is your farm from the knowledge distribution center (agro-center): Distance: ______________ Go to Section B. Final Evaluation Report – Performance Evaluation: CRW Project 96 Section B: Assessment of Climate Change “Now I would like to ask you a little about climate change, one of the areas of focus for the UNDP project that we are studying.” Section C: Weather and Climate “One of the areas of focus of the UNDP project is to ensure that Kazakhstan agro-producers have access to reliable and accurate weather and climate information to inform their decision-making. We would like to ask you a few questions about how you currently use weather and climate information in making farming decisions.” Question Response Instruction B1. Do you believe that climate change is or could have serious impacts on the agricultural sector in Kazakhstan? 1. Yes B2 2. No C1 -88. Don’t know -89. Refused If yes, go to B2. If no, go to Section C. B2. Do you believe you have a good understanding of the risks posed by climate change to your farm’s production? 1. Yes B3 2. No C1 -88. Don’t know -89. Refused If yes, go to B3. If no, go to Section C. B3. Is your farm production at risk because of climate change? 1. YesB4 2. NoB4 -88. Don’t know -89. Refused Go to B4 B4. How concerned are you about the risks posed by climate change to your farm’s production? (INTERVIEWER: Read options 1-3 out loud, and circle the option that matches the response most closely) 1. Very concerned 2. Somewhat concerned 3. Not concerned -88. Don’t know -89. Refused Go to B5 B5. Which specific risks related to climate change are you most concerned about? INTERVIEWER: It is open question, please record every word Go to B6 B6. How prepared do you feel to deal with the risks posed by climate change to your farm? (INTERVIEWER: Read options 1-3 out loud, and circle the option that matches the response most closely) 1. Well prepared 2. Somewhat prepared 3. Not prepared -88. Don’t know -89. Refused Go to Section C Final Evaluation Report – Performance Evaluation: CRW Project 97 Question Response Instruction C1. When making decisions about when to plant your wheat crop, would you say that weather and climate are: 1. The most important factor that you consider 2. One important factor that you consider 3. Just one of the many factors that you consider 4. Weather and climate are not considered -88. Don’t know -89. Refused Go to C2 C2. When making decisions about when to harvest your wheat crop, would you say that weather and climate are: 1. The most important factor that you consider 2. One important factor that you consider 3. Just one of the many factors that you consider 4. Weather and climate are not considered -88. Don’t know -89. Refused Go to C3 C3. From which of the following sources do you obtain information about seasonal climate forecasts? INTERVIEWER: Please circle all that apply 1. Internet (not including phone) 2. Television 3. Radio 4. Newspaper 5. Bulletins from Ministry of Agriculture 6. Local Akimat/Agro￾management 7. Other farmers/ agro￾producers 8. My own labs, metro stations 9. Kazhydromet 10. Phone (through internet) 11. Other (Please specify): ___________________ -88. Don’t know -89. Refused Go to C4 C4. To what extent do you rely on seasonal climate forecasts produced by Kazakhstan’s national weather service – Kazhydromet? 1. Very much 2. Somewhat 3. Not at all -88. Don’t know -89. Refused Go to C5 C5. As compared to two years ago, would you say that you rely on Kazhydromet seasonal climate forecasts more, less or about the same? 1. I rely more on Kazhydromet 2. I rely less on Kazhydromet 3. Not much has changed since 2 years ago -88. Don’t know -89. Refused Go to C6 C6. How reliable do you consider the seasonal climate 1. Very reliable Go to C7 Final Evaluation Report – Performance Evaluation: CRW Project 98 Section D: Climate Adaptation Techniques “The information that we have been provided indicates that within the last 2 years, you have attended at the agricultural extension center a training session, workshop or seminar or toured an experimental demonstration plot showcasing agricultural adaptation techniques for building resiliency to climate change. These events were sponsored by the UNDP program that we are evaluating (although that may not have been made clear to the attendees of these sessions). We would like to ask you a few questions about these events. forecasts produced by Kazhydromet? 2. Somewhat reliable 3. Not at all reliable -88. Don’t know -89. Refused C7. In the previous two years, do you think that the Kazhydromet seasonal climate forecasts have: 1. Become more reliable 2. Become less reliable 3. Have not changed -88. Don’t know -89. Refused Go to C8 C8. To what extent do you rely on drought forecasts produced by Kazakhstan’s national weather service – Kazhydromet? 1. Very much C9 2. Somewhat C9 3. Not at all  C12 -88. Don’t know  C12 -89. Refused Go to C9 or C12 C9. As compared to two years ago, would you say that you rely on Kazhydromet drought forecasts more, less or about the same? 1. I rely more on Kazhydromet 2. I rely less on Kazhydromet 3. Not much has changed since 2 years ago -88. Don’t know -89. Refused Go to C10 C10. How reliable do you consider the drought forecasts produced by Kazhydromet: 1. Very reliable 2. Somewhat reliable 3. Not at all reliable -88. Don’t know -89. Refused Go to C11 C11. In the previous two years, do you think that the Kazhydromet drought forecasts have: 1. Become more reliable 2. Become less reliable 3. Have not changed -88. Don’t know -89. Refused Go to C12 C12. To what extent access to accurate and reliable seasonal climate data/information increase wheat production: 1. Very much 2. Somewhat 3. Little - 88 Don’t know - 89 Refused Go to C13 C13. What benefit(s) did you receive or expect to receive from use of seasonal climate data/information Kazhydromet? INTERVIEWER: It is open question, please record every word Go to Section D. Final Evaluation Report – Performance Evaluation: CRW Project 99 Question Response Instruction D0. From which of the following sources do you obtain information about new adaptation techniques for wheat production? INTERVIEWER: Please circle all that apply 1. Internet 2. Television 3. Radio 4. Newspaper 5. Bulletins from Ministry of Agriculture 6. Other farmers/agro￾producers 7. Official Meetings 8. Exhibits 9. Training/educational events/seminars 10. Research Institute 11. Conference 12. Other (Please specify): __________________ -88. Don’t know -89. Refused Go to D1 D1. Have you attended any training sessions, workshops or seminars at the agricultural extension center that discussed climate change? 1. Yes  D2 2. No D7 -88. Don’t know D7 -89. RefusedD7 If yes, go to D2. If no, go to D7. D2. As a result of the training session, workshop or seminar, did you develop a better understanding of the risks posed by climate change? 1. Yes 2. No -88. Don’t know -89. Refused Go to D3. D3. As a result of the training session, workshop or seminar, did you develop a better understanding of things that you could do to prepare for and address the impacts of climate change? 1. Yes D4 2. No D7 -88. Don’t knowD4 -89. RefusedD7 If yes, go to D4. If no, go to D7. D4. As a result of the training session, workshop or seminar, did you adopt a new agriculture approach or approaches? 1. Yes D5 2. NoD6 -88. Don’t know -89. RefusedD7 If yes, Go to D5 If no, go to D6. D5. As a result of the training session, workshop or seminar, did you: INTERVIEWER: Circle all options that apply. If “Adopt some other approach” is the selection, ask what that approach is and record the answer verbatim. 1. Adopt a new planting time D7 2. Diversify the structure of crops that you plant D7 3. Resort to alternative fertilizers or seed varietiesD7 4. Adopt a new approach to tillingD7 5. Changed fertilizers D7 6. Adopt another D7approach (please specify):______________ D7 -88. Don’t know -89. Refused Go to D7. D6. What were the reasons that you chose not to adopt a 1. It was too expensive Go to D7. Final Evaluation Report – Performance Evaluation: CRW Project 100 new agricultural approach? INTERVIEWER: Circle all that apply [If “ Some other reason not listed above” is selected, ask what that reason was and record the answer verbatim] 2. There was insufficient time to do so in the previous growing season 3. There was too much risk involved 4. I did not feel I possessed sufficient knowledge of the approach 5. Lacking human resources 6. Insufficient equipment 7. Other (please specify): ___________ -88. Don’t know -89. Refused D7. Did you visit an experimental demonstration plot of climate change adaptation techniques for wheat production at the agricultural extension center? 1. YesD8 2. No Section E -88. Don’t know Section E -89. Refused Section E If yes, go to D8. If no, go to Section E. D8. As a result of visiting the experimental demonstration plot, do you know more about what you could do to prepare for and address the impacts of climate change? 1. YesD9 2. No  Section E -88. Don’t know  Section E -89. Refused Section E If yes, go to D9. If no, go to Section E. D9. What, specifically, did you learn to prepare for and address the impacts of climate change? INTERVIEWER: This is an open-ended question. Record verbatim. Go to D10 D10. As a result of a better understanding of things that you could do to prepare for and address the impacts of climate change, did you adopt a new agriculture adaptation technique for wheat production on your farm? 1. Yes D11 2. No D13 -88. Don’t knowSection E -89. Refused Section E If yes, go to D11. If no, go to D13. D11. What agricultural adaptation technique did you adopt? INTERVIEWER: This is an open-ended question. Record verbatim. Go to D12. D12. What benefit(s) did you receive or expect to receive from implementation of adaptation techniques? INTERVIEWER: This is an open-ended question. Record verbatim. Go to E1. D13. What were the reasons that you chose not to adopt a new agriculture adaptation technique for wheat production on your farm? INTERVIEWER: Circle all options that apply. If “ Other” is selected, ask what that reason was and record the answer verbatim 1. It was too expensive 2. There was insufficient time to do so in the previous growing season 3. There was too much risk involved 4. I did not feel I possessed sufficient knowledge of the approach 5. Lacking human resources 6. Insufficient equipment 7. Not enough scientific evidence 8. Other (please specify): ________________ -88. Don’t know -89. Refused Go to Section E. Final Evaluation Report – Performance Evaluation: CRW Project 101 Section I: Wheat Production Challenges “Thank you very much for your time. This has been very helpful to us. We just have one more question that is more general in nature and about which we would really like your feedback” Section F: Assessment of Climate Change “Now I would like to ask you a little about climate change, one of the areas of focus for the UNDP project that we are studying.” Question Response I1. What are the three greatest challenges that you face in increasing wheat production on your farm? INTERVIEWER: This is an open-ended question – record answer verbatim. I2. When it comes to increasing your wheat production, which new challenges do you expect in the near future? I3. When it comes to increasing your wheat production, which new opportunities do you expect in the near future? I4. Are there any problems that female farmers and agro-producers face, as compared to their male counterparts? Question Response Instruction F0: Where do you currently work? What is your affiliation? 1. Research Institute- F1 2. University/college - F1 3. Government office - F1 4. Other (Please specify): ___________________- -88. Don’t know -F2 89. Refused-F2 Go to F1. F1. Do you believe that climate change is or could have serious impacts on the agricultural sector in Kazakhstan? 3. Yes F2 4. No Section G -88. Don’t know -89. Refused If yes, go to F2. If no, go to Section G. F2. Do you believe you have a good understanding of the risks posed by climate change to the agricultural sector in Kazakhstan? 3. Yes F3 4. No F4 -88. Don’t know If yes, go to F3. If no, go to Section C. Final Evaluation Report – Performance Evaluation: CRW Project 102 Section G: Weather and Climate “One of the areas of focus of the UNDP project is to ensure that Kazakhstan agro-producers have access to reliable and accurate weather and climate information to inform their decision-making. We would like to ask you a few questions about how you currently use weather and climate information in making farming decisions.” Section G -89. RefusedSection G F3. Which specific risks related to climate change to the agricultural sector in Kazakhstan are you most concerned about? INTERVIEWER: It is open question, please record every word Go to F4 F4. How prepared do you feel farmers are to deal with the risks posed by climate change? 1. Well prepared 2. Somewhat prepared 3. Not prepared -88. Don’t know -89. Refused Go to Section G Question Response Instruction G1. Do you use information about seasonal climate forecasts or drought forecasts for your work? 1. Yes G2 2. No Section H -88. Don’t know  Section H -89. Refused Section H If yes, go to G2. If other answers, go to Section H. G2. Which forecasts do you use? 1. Seasonal climate forecasts G3 2. Drought forecasts  G8 3. Both G3 -88. Don’t know Section H -89. Refused Section H G3. From which of the following sources do you obtain information about seasonal climate forecasts? INTERVIEWER: Please circle all that apply 1. Internet (excluding phone) 2. Television 3. Radio 4. Newspaper 5. Bulletins from Ministry of Agriculture 6. Local Akimat/Agro￾management 7. Other farmers/ agro￾producers 8. My own labs, metro stations 9. On the phone (through internet) 10. Kazhydromet 11. Other (Please specify): ___________________ -88. Don’t know -89. Refused Go to G4 G4. To what extent do you rely on seasonal climate forecasts produced by Kazakhstan’s national weather service – Kazhydromet? 1. Very much 2. Somewhat 3. Not at all -88. Don’t know -89. Refused Go to G5 Final Evaluation Report – Performance Evaluation: CRW Project 103 Section H: Climate Adaptation Techniques The information that we have been provided indicates that within the last 2 years, you have attended at the agricultural extension center a training session, workshop or seminar or toured an experimental demonstration plot showcasing agricultural adaptation techniques for building resiliency to climate change. These events were sponsored by the UNDP program that we are evaluating (although that may not have been made clear to the attendees of these sessions). We would like to ask you a few questions about these events. G5. As compared to two years ago, would you say that you rely on Kazhydromet seasonal climate forecasts more, less or about the same? 1. I rely more on Kazhydromet 2. I rely less on Kazhydromet 3. Not much has changed since 2 years ago -88. Don’t know -89. Refused Go to G6 G6. How reliable do you consider the seasonal climate forecasts produced by Kazhydromet? 1. Very reliable 2. Somewhat reliable 3. Not at all reliable -88. Don’t know -89. Refused Go to G7 G7. In the previous two years, do you think that the Kazhydromet seasonal climate forecasts have: 1. Become more reliable 2. Become less reliable 3. Have not changed -88. Don’t know -89. Refused Go to G8 G8. To what extent do you rely on drought forecasts produced by Kazakhstan’s national weather service – Kazhydromet? 1. Very much 2. Somewhat 3. Not at all -88. Don’t know -89. Refused Go to C9 G9. As compared to two years ago, would you say that you rely on Kazhydromet drought forecasts more, less or about the same? 1. I rely more on Kazhydromet 2. I rely less on Kazhydromet 3. Not much has changed since 2 years ago -88. Don’t know -89. Refused Go to G10 G10. How reliable do you consider the drought forecasts produced by Kazhydromet: 1. Very reliable 2. Somewhat reliable 3. Not at all reliable -88. Don’t know -89. Refused Go to G11 G11. In the previous two years, do you think that the Kazhydromet drought forecasts have: 1. Become more reliable 2. Become less reliable 3. Have not changed -88. Don’t know -89. Refused Go to G12 G12. To what extent access to accurate and reliable seasonal climate data/information increase wheat production: 1. Very much 2. Somewhat 3. Little -88. Don’t know -89. Refused Go to Section H Final Evaluation Report – Performance Evaluation: CRW Project 104 Question Response Instruction H0. From which of the following sources do you obtain information about new adaptation techniques for wheat production? INTERVIEWER: Please circle all that apply 1. Internet 2. Television 3. Radio 4. Newspaper 5. Bulletins from Ministry of Agriculture 6. Other farmers/agro￾producers 7. Official Meetings 8. Exhibits 9. Training/educational events/seminars 10. From supervisors 11. Research Institute 12. Conferences 13. Other (Please specify): _______________ ____ -88. Don’t know -89. Refused Go to H1 H1. Have you attended any training sessions, workshops or seminars at the agricultural extension center that discussed climate change? 1. Yes  H2 2. No H6 -88. Don’t know H6 -89. RefusedH6 If yes, go to H2. H2. As a result of the training session, workshop or seminar, did you develop a better understanding of the risks posed by climate change? 1. Yes 2. No -88. Don’t know -89. Refused Go to H3. H3. As a result of the training session, workshop or seminar, did you develop a better understanding of things that could be done to prepare for and address the impacts of climate change? 1. Yes 2. No -88. Don’t know -89. Refused Go to H4. H4. As a result of the training session, workshop or seminar, did you adopt new knowledge or a new agriculture approach or approaches in your work 1. Yes H6 2. No H5 -88. Don’t know H6 -89. Refused H6 Go to H5. What were the reasons that you chose not to consider a new agricultural approach? INTERVIEWER: Circle all that apply 1. It was too expensive 2. There was insufficient time to do so in the previous growing season 3. There was too much risk involved 4. I did not feel I possessed sufficient knowledge of the approach 5. Lacking human resources 6. Insufficient equipment Final Evaluation Report – Performance Evaluation: CRW Project 105 Section I: Wheat Production Challenges “Thank you very much for your time. We just have one more question that is more general in nature and about which we would really like your feedback” “Thank you once again for agreeing to participate in this survey. END 7. Not enough scientific evidence 8. Other (please specify): ___________ -88. Don’t know -89. Refused H6. Did you visit an experimental demonstration plot of climate change adaptation techniques for wheat production at the agricultural extension center? 1. Yes H7 2. No Section I -88. Don’t know Section I -89. Refused Section I If yes, go to H7. If other responses, go to Section I. H7. As a result of visiting the experimental demonstration plot, do you know more about what could be done to prepare for and address the impacts of climate change? 1. Yes H8 2. No Section I -88. Don’t know Section I -89. RefusedSection I If yes, go to H8. If other responses, go to Section I. H8. What, specifically, did you learn to address the impacts of climate change? INTERVIEWER: This is an open-ended question. Record verbatim. Go to Section I Question Response I1. What are the three greatest challenges that farmers face in increasing wheat production? INTERVIEWER: This is an open-ended question – record answer verbatim. I2. When it comes to increasing your wheat production, what new challenges do you expect in the near future? I3. When it comes to increasing wheat production, are there new opportunities in the near future? I4. Are there any problems that female farmers and agro-producers face, as compared to their male counterparts? Final Evaluation Report – Performance Evaluation: CRW Project 106 ANNEX D: LIST OF CRW PROJECT PUBLICATIONS 1. CRW Project Bulletins #1, 2 and 3 This is a project owned publication issued once per quarter describing a series of reports on Project activity status, lessons learnt and best practices. Each report updates project activities and outcomes both implemented in the country and Central Asian regions, including the articles received from different project partners. The bulletin also contains different national and international level news in the agricultural sector including best practices and proposals for cooperation. The audience of quarterly bulletin is all project partners and other interested counterparts. 2. CRW Infographics on Wheat The project best practices, results of demonstration plots are usually expressed through such infographics to make the project message to the farmers as well as to any other interested in a simple and understandable mode! 3. Study on The Relationship between Soil Moisture Observed by in situ probes and Satellite Derived Surface Wetness in Northern Kazakhstan The publication expresses the new methodology and approaches to identify the optimum planting period including crop yield estimation based on the experience of the USA and the Canada on the wheat production sector. The publication brings results of the research using space satellite images and field based probe data. This publication sought to understand and monitor how soil moisture affects yields. The authors used field data probe data to detect fluctuations, and relate them to variability in the satellite derived wetness values. 4. Technology of cultivation of wheat crops in different dry climatic condition The publication explains a number of agricultural technologies to cultivate wheat crops in different soil and climatic conditions. Particularly, it focuses on the soil and climatic conditions of CA countries including Afghanistan. The wheat in CA countries except KAZ is mostly grown on the irrigated lands, thus it requires a special approach to get grown especially, taking into consideration the mountainous regions of Tajikistan and Kyrgyzstan. 5. Integrated pest and disease management The publication provides methods and technologies for pursuing an integrated pests and disease management on the wheat cultivation sector. It provides list of effective herbicides and pesticides to combat pests and diseases during a different vegetative growth of the wheat crops. It provides how, wheat and in what proportion and technologies the chemicals should be used. Meantime, the publication provides safety precautions during the use or application of chemicals. It also contains the list of national and international agricultural inputs dealers for better agricultural chain and trade. 6. Plant breeding and seed multiplication in Kazakhstan and Central Asian Countries This is a small brochure that provides overview for some of promising wheat varieties broadly cultivated in the scale of the Central Asian countries. It also provides possibility and potential of some seed multiplication centers in Kazakhstan and Central Asian countries. This is produces with the emphasis to be a desktop reference for the farmers to make a right decision during selection of the seeds for cultivation. Final Evaluation Report – Performance Evaluation: CRW Project 107 7. Catalogue of drought resistant wheat seed in Central Asia The catalogue describes the list of drought resistant wheat varieties at the scale of the Central Asian countries. The publication is structures in a way that the farmers will recognize the seed by a number of criteria mentioned in the catalogues to be as drought resistant seed. Along with detail place based agro technologies means of production, it provides description of the morphological, biological and chemical characteristics of the wheat seeds. It helps the farmers to select the right drought resilient wheat seeds and further respective agro technologies. 8. Atlas of drought resistant wheat seed in Kazakhstan Under this publication, the project provides an atlas of the wheat seeds where it describes the region’s soil and climatic condition and outlines the suitable wheat seed varieties to be grown in these particular regions. The publication shows the results of the studies based on the demonstration plots and expressed in the form of maps of wheat seeds varieties. This helps farmers to carefully select the right wheat seeds and technologies based on their regions and agro climatic conditions. 9. Catalogues on best practices in Kazakh language This publication provides a detail explanation of 22 documented best practices including 44 conventional best practices being applied in Kazakhstan and other Centrals Asian regions. It also describes a basic agro-technological means broken down to the specific seasons where farmers need to apply this or that technology. 10. Agricultural practices for sustained production of spring wheat in different soil and climatic condition of north Kazakhstan The brochure provides a number of agro-technologies for wheat cultivation in different soil and climatic condition in the northern Kazakhstan. Particularly, the following agro technologies are expressed in this brochure. Application of no or minimum tillage, traditional practices, cultivation in different periods of time, wheat crop diversification. 11. Technical instruction for application of laser leveling technology The brochure provides an overview on advantage and drawbacks of application of laser leveling technology during cultivation of different agricultural crops. It describes, to what extent the smooth land surface can play an important role to ensure to minimize the work load, uniform seeds emergence and etc. thus is gives a comprehensive description of laser leveling technology in the wheat production sector. 12. Agro metrological forecasts in Kazakhstan The brochure analyzes the agro-climatic resources and the consequences of climate change for grain production of the Republic of Kazakhstan, analyzes the agrometeorological monitoring, holds agrometeorological and forecasting evaluation, provides recommendations to improve the agrometeorological support and extension in Kazakhstan. Climatic and statistical data is used in the brochure to describe the situation better inn regards the impact of adverse effect of climate change, it provides, the results of earlier conducted researches on vulnerability and adaptation on agriculture and climate change. Yet it provides the results of conducted field work on the adaptation measures for grain production as well as expected climate variability, recommendations for the development of agrometeorological monitoring, analysis and forecasting, recommendations to improve the provision of agricultural sector with agrometeorological information. Final Evaluation Report – Performance Evaluation: CRW Project 108 13. Technology for cultivation of spring wheat verities in the face of climate change The brochure provides a very specific and detail agricultural technologies for cultivation of spring wheat seeds in the face of different climatic conditions. It provides a practices that farmers may apply during the early spring or early autumn frozen as well as during the expected drought or high amount of precipitation. 14. Effectiveness of land use system in the Wheat cultivation sector of Kazakhstan The publication describes the land resources in the agricultural sector of Kazakhstan and provides a comprehensive study results on how land resources in the wheat production sector should be used. It highlights the issues of monoculture and diversification that sustains the production of wheat in the country. 15. Central Asian Food security gap analyses The report reviews regional wheat market dynamics, structure, trends, and constraints confronting the wheat marketing sector in Central Asian Countries, and to explore the role of Kazakhstan wheat market and trade plays in ensuring food security in Central Asian Countries. 16. The current situation and problems in the wheat production sector in the face of climate change The brochure provides the present and main problems in the wheat production sector of Kazakhstan and the consequences of climate change impacts to the sustainability of what production sector in Kazakhstan. It provides problems related the wheat crop yield, farmers’ knowledge and applied experience to cultivate wheat crops in Kazakhstan. The publication provides a number of nationally existing and internationally recognized best practices and adaptation options in the wheat cultivation sector of Kazakhstan. Final Evaluation Report – Performance Evaluation: CRW Project 109 ANNEX E: LIST OF DOCUMENTS REVIEWED FOR THE EVALUATION 1. CRW Project Description, 9/2012 2. CRW Project Description for proposed extension 3. CRW Performance Monitoring Plan, 10/2013, updated October, 2014 4. Kazakhstan Stakeholder Consultations – Workshop Report, February – 03/2013, International Research Group 5. USAID UNDP 3rd Year Workplan 6. Mid-Level Assessment: Climate Forecasting in Kazakhstan, 10/2014, Engility/ International Research Group 7. Roberth I., et al (2015) Evaluation of the CRW project, April 20th, 2015 8. CRW Project Final Report, September 2012 – September 2014 9. Yerlan Zhumabayev, CRW project presentation, 30 January 2014 10. Ashly King, Glen Anderson, Presentation «Responding to Climate Change in the Wheat Sector of Kazakhstan», September 19, 2013 11. CRW Project, 3rd Year Work Plan, October 2013 – September 2014 12. CRW Project, Quarterly Report, Second Quarter 2014 13. CRW Project Annual Report 2013. Engility/ International Research Group Trip Reports, December 2012 – June 2014 14. Hemson D., et al (2014) Kazakhstan Pilot Project Scoping Trip Report 15. Syzdykov R., et al (2015) Country Report: Kazakhstan, 
Analytical Centre of Economic Policy in Agricultural Sector, Kazakhstan 16. Martin Petrick, M., et al (2014) Kazakhstan’s wheat, beef and dairy sectors: An assessment of their development constraints and recent policy responses, Paper prepared for presentation at the symposium “Kazakhstan’s Economic Strategy: Halfway to 2030” 17. World Bank (2015) Agricultural Sector Risk Assessment in Kazakhstan 18. National Center Space Research and Technology (2015) Concept of Geo Portal 19. KHM (2015) Monthly Weather Bulletin, May 20. KHM (2015) Seasonal Forecast Bulletin, April – October 21. KHM (2015) Drought Bulletin, June 22. USDA (2015) Kazakhstan-Republic: Grain and Feed Annual report http://gain.fas.usda.gov/Recent%20GAIN%20Publications/Grain%20and%20Feed%20Annual_Astana_Kazakhsta n%20-%20Republic%20of_4-10-2015.pdf 23. Gender Assessment USAID/Central Asian Republics: Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, Uzbekistan. March 2010 http://pdf.usaid.gov/pdf_docs/Pnads880.pdf 24. Country Gender Assessment: Republic of Kazakhstan. Asian Development Bank, May 2006 http://www.adb.org/documents/kazakhstan-country-gender-assessment 25. Kazakhstan: Country Gender Assessment. Asian Development Bank, 2013 http://www.adb.org/sites/default/files/institutional-document/34051/files/kazakhstan-country-gender￾assessment.pdf 26. The Women's Empowerment in Agriculture Index. May 15, 2014 http://feedthefuture.gov/lp/womens-empowerment-agriculture-index Accessed 8/22015 27. National Commission (12/2014) Workplan of the national commission of women and family-demographic policy for 2015, 28. http://www.akorda.kz/upload/nac_komissiya_po_delam_zhenshin/2.2%20%D0%9F%D0%BB%D0%B0 %D0%BD%20%D1%80%D0%B0%D0%B1%D0%BE%D1%82%D1%8B%20%D0%9D%D0%B0%D1%86 %D0%B8%D0%BE%D0%BD%D0%B0%D0%BB%D1%8C%D0%BD%D0%BE%D0%B9%20%D0%BA%D 0%BE%D0%BC%D0%B8%D1%81%D1%81%D0%B8%D0%B8%20%D0%BF%D0%BE%20%D0%B4%D0 Final Evaluation Report – Performance Evaluation: CRW Project 110 %B5%D0%BB%D0%B0%D0%BC%20%D0%B6%D0%B5%D0%BD%D1%89%D0%B8%D0%BD%20%D0 %B8%20%D1%81%D0%B5%D0%BC%D0%B5%D0%B9%D0%BD%D0%BE- %D0%B4%D0%B5%D0%BC%D0%BE%D0%B3%D1%80%D0%B0%D1%84%D0%B8%D1%87%D0%B5 %D1%81%D0%BA%D0%BE%D0%B9%20%D0%BF%D0%BE%D0%BB%D0%B8%D1%82%D0%B8%D0 %BA%D0%B5%20%D0%BF%D1%80%D0%B8%20%D0%9F%D1%80%D0%B5%D0%B7%D0%B8%D0% B4%D0%B5%D0%BD%D1%82%D0%B5%20%D0%A0%D0%B5%D1%81%D0%BF%D1%83%D0%B1% D0%BB%D0%B8%D0%BA%D0%B8%20%D0%9A%D0%B0%D0%B7%D0%B0%D1%85%D1%81%D1% 82%D0%B0%D0%BD%20%D0%BD%D0%B0%202015%20%D0%B3%D0%BE%D0%B4.pdf Final Evaluation Report – Performance Evaluation: CRW Project 111 ANNEX F: SCHEDULE OF INTERVIEWS CONDUCTED BY EVALUATION TEAM # Name Organization/ Affiliation Position Place of Meeting 20 July, Monday 1 Meeting with UNDP/CRW Team: Yerlan Zhumabayev UNDP, Sustainable Land Management Projects National Coordinator Astana Firuz Ibraghimov, UNDP, Sustainable Land Management Projects Chief Technical Advisor Astana Gulmira Kabanbayeva UNDP, Sustainable Land Management Projects Project Expert Astana Rassul Rakhinbekov UNDP E&E unit Programme Analyst/Portfolio Manager Astana Victoria Baigazina UNDP E&E unit Astana 2 Murad Akshalov KazAgroInnovation former CRW National Director Astana 3 Zein Kabikeev UNDP Expert Astana 4 Asilkhan Asilbekob UNDP GEF Project Project Director Astana 21 July, Tuesday 5 Tolgar Absattar KazAgroInnovation Astana 6 Ayman Absattarova KazAgroInnovation CRW National Director Astana 7 Seric Altkhozhin KazAgroInnovation Astana 8 Mereke Akbuzay Kazhydromet Deputy Director General Astana 9 Yernanat Iskakov Kazhydromet Lead engineer of Department of agro meteorological forecast Astana 10-12 Group interview (3): Dinara Shalabayeva Kazhydromet Lead engineer-synoptic of Department of Long-Term Forecasts Astana Erbolat Mukanov Kazhydromet Head of Department of agro meteorological forecast Astana Aliya Osmanova Kazhydromet Lead engineer-synoptic of Department of Long-Term Forecasts Astana Observation tour Kazhydromet Short – and Long-term forecasts units 22 July, Wednesday 13-14 Group Interview (2): Roman Kusainov Analytical Center of Economic Policy in Agriculture Deputy Director Astana Aydos Mukashybekov Analytical Center of Economic Policy in Agriculture Managing Director Astana 15 Nurlan Mukhamedzhanov JSC AgroMarketing Deputy Director Astana 16 Saken Savetovich Institute of Geography Lead scientific worker Astana 17 Yevgeniy Klimov Organic Farming Association Head Astana Final Evaluation Report – Performance Evaluation: CRW Project 112 18 Satubulyk Zhuligetov Farmer Union of Kazakhstan Head Astana 23 – 24 July, Thursday Shortandy 19 Tatyana Gontarenko Barayev Research Institute of Grain Head of Extension Center Shortandy 20 Kanat Akshalov Barayev Research Institute of Grain Head of department Shortandy 21 Vladimir Skoblikov Barayev Research Institute of Grain Head of department Shortandy 22 Yuriy Pokhorukov Barayev Research Institute of Grain Head of agrotechnique Laboratory 23 Vasiliy Mitrophanov Svobodnoye Farm Farmer Shortandy 24 Askhat Kubyshev Novomartovka 2010 Farm Farmer Shortandy 25 Rinat Kanatovich Farmer Astana 26 Nadezhda Nosachova Kubanskaya Farm Agronomist Shortandy Almaty 27 Bakhtiyar Said Research Institute of Livestock and KORMOPROIZVODSTVO Agriculture expert, former CRW chief technical adviser Almaty 28-29 Group Interview (2): Payizkhan Kozhakhmetov Almaty branch of RSE “Kazhydromet Head of Department of Climate and Water Problem Research Almaty Svetlana Dolgikh Almaty branch of RSE “Kazhydromet Head of Department of Climate Research Almaty 30 Azamat Kauazov National Center For Space Research And Technology Head of Department of Space Monitoring and Nature Processes Almaty 31 Damelya Aitkhozhina UNWomen, Multi-country Office Programme specialist Almaty 32-33 Meeting at USAID Mission for CA￾interview (2): David G. Brown USAID Mission for Central Asia Deputy Regional Mission Director Almaty Laura Gonzalez Economic Development Office Deputy Director Almaty Gulzada Azhetova Economic Development Office Project Management Specialist Almaty Marina Lyaschenko Gender Focal Person Almaty 27 July, Monday 34 Observation of Demonstration plot Shortandy 35 Focus Group Discussion with farmers Shortandy 28 July, Tuesday 36 Yuriy Tulayev Kostanay Agriculture Research Institute Deputy Head of Soil Kostanay 37 Ekaterina Gubert Kostanay Agriculture Research Institute Manager Kostanay 38 Almabek Nugmanov Kostanay Agriculture Research Institute Head of Extension Center Kostanay 29 July, Thursday 39-41 Group Interview (3): Larisa Kuzmina Kazhydromet, Kostanay Branch Director Kostanay Svetlana Melnik Kazhydromet, Kostanay Branch Synoptic Kostanay Final Evaluation Report – Performance Evaluation: CRW Project 113 Irina Zhylyayeva Kazhydromet, Kostanay Branch Synoptic, Marketing specialist Kostanay 41 Zhetes Amantayev Farmer Assosiation of Kostanay oblast Head of Board Kostanay 42 Vadim Lopukhin Association of Organic Farming Head Kostanay 43-49 Group Interview (7): Sharuan Zhelmisov Kostanay State University, Agronomic Department Professor Kostanay Niyazbek Kalimov Kostanay State University, Agronomic Department Dean Kostanay Mikhail Shylov Kostanay State University, Agronomic Department Senior Lecturer Kostanay Bakyntbek Baimbayev Kostanay State University, Agronomic Department Senior Lecturer Kostanay Nikolay Levaznuy Kostanay State University, Agronomic Department Senior Lecturer Kostanay Belet Dyusebayev Kostanay State University, Agronomic Department Senior Lecturer Kostanay Amanzhol Akhmet Kostanay State University, Agronomic Department Professor Kostanay 50 Alexander Grinets, Agriculture Enterprise Farmer Kostanay 51 Tamerlan Aksagov Kostanay Agriculture Research Institute Senior Scientific Worker Kostanay 30 July, Thursday 52 Alexandr Borisenko, Farmer Enterprise IRINA Head of Farm Lisakovsk 53-54 Group Interview (2): Rustam Ablayev Farmer Kostanay Sergey Bezhunar Farmer Kostanay 55 Alexandr Borodin Zuyevka Farm Agronomist Kostanay 56 Salimzhan Isenomanov Kostanay oblast Akimat Head of Agriculture department Kostanay 31 July, Friday 57 Visit of demonstration plot of the Extension center Kostanay 58-59 Group Interview (2): Vladimir Chernenko Zarechnaya Farm Director Nikolay Levadnyy Farmer 60 Valeriy Razumovich KazAgroNo-till Technical Director 1 August, Saturday 11:15am - Return to Astana from Kostanay 2 August, Sunday Day off 3 August, Monday 61 Saule Zhurynova Ministry of Energy 62 Firuz Ibraghimov UNDP, Sustainable Land Management Projects Chief Technical Advisor 63 Baumekhambetov B. UNDP Disaster management project Project Manager Final Evaluation Report – Performance Evaluation: CRW Project 114 4 August, Tuesday  Interview Reports preparation  Preparation of Preliminary Findings for Validation Workshop 5 August, Wednesday 64 Almat Kunakov Ministry of Agriculture Senior Expert Data analysis/validation workshop with stakeholders 6 August, Thursday 65 Alexey Morgunov International Maize and Wheat Improvement Center Head of International Winter Wheat Improvement Program 7-8 August, Friday Depature OTHER INTERVIEWS 66 Ashley King USAID/Indonesia Former USAID/Kaz Envionemtntal officer Phone 67 Stanislav Kim UNDP/Turkey Former head of Enviromental Offce at UNDP/Kaz Skype 68 Evan Meyer USAID/Kazakhstan USAID Environmental Officer In-person 69 Igor Khomyakov IRI Phone Site Observations 1. KazHydroMet, Astana 2. Demonstration plot, Shortandy 3. Demonstration plot, Kostanay