USAID/BHA’S EFFECTIVENESS OF RAINFALL PREDICTION TOOLS FOR FARMERS IN NTB AND NTT EVALUATION REPORT – FINAL FEBRUARY 1, 2023 Contracted under GS-00F-210DA / 72049720M00001 Indonesia Monitoring, Evaluation, and Learning Platform (MEL-P) Authors Prof. Dr. Rizaldi Boer, Team Leader Dr. Muh Taufik Resti Salmayenti, MSc DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development of the United States Government. i | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV ABSTRACT This report answers evaluation questions related to the effectiveness of rainfall prediction tool (PCH) for farmers in Nusa Tenggara. Our implementing partner, World Neighbors, has applied the tool to advise farmers on their agriculture activities (e.g., planting season, cropping patterns) and to support disaster risk assessment at the village level. In the evaluation, we addressed four questions related to (i) tool accuracy and reliability, (ii) tool utility for farmers, (iii) effectiveness of tool dissemination, and (iv) tool utility for disaster preparedness. We combined qualitative and quantitative analyses to obtain precise answers to the questions. Qualitative analysis for the adoption of the prediction tool was based on 54 in-depth interviews (online, offline, and hybrid) with 224 key informants representing climate experts, government at district and village levels, World Neighbors’ local partners, and farmers. Our findings showed that using a single statistical model was unreliable in forecasting rainfall on a seasonal/annual scale. This has been confirmed quantitatively by the low accuracy of the tool based on statistical indicators. Most farmers we interviewed still apply the traditional system in making planting decisions, and some of them also consider rainfall prediction from the tool in making decisions. The traditional system may be more suitable for dissemination advice considering the farmers’ accessibility on smartphones and internet networks. For disaster preparedness, we found that the tool was a complementary source of disaster information. Finally, the evaluation team offers several recommendations to improve future USAID activities/interventions in this sector. Cover photo caption: Evaluation team for the Rainfall Prediction Tools conducted field data collection by interviewing the key informants in Dompu, West Lombok, Central Lombok, East Lombok, and Nagekeo in September 2022. Left top: Interviewing the village leaders in Giri Sasak village, West Lombok. Left bottom: Group discussion with the East Lombok Agriculture District Office. Right: Farmer group discussion in Padak Guar village, East Lombok. Photo credit: USAID/ MEL-P USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | ii CONTENTS ABSTRACT ............................................................................................................................... I CONTENTS ............................................................................................................................ II LIST OF FIGURES..................................................................................................................................................... IV LIST OF TABLES....................................................................................................................................................... IV ACRONYMS............................................................................................................................ V EXECUTIVE SUMMARY...................................................................................................... VII RECOMMENDATIONS EQ1..............................................................................................................................VIII RECOMMENDATIONS EQ2................................................................................................................................IX RECOMMENDATIONS EQ3................................................................................................................................IX RECOMMENDATIONS EQ4................................................................................................................................. X BACKGROUND....................................................................................................................... 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS ......................................... 2 EVALUATION PURPOSE ........................................................................................................................................ 2 EVALUATION QUESTIONS................................................................................................................................... 2 EVALUATION METHODS AND LIMITATIONS................................................................ 4 PRIMARY DATA......................................................................................................................................................... 4 SECONDARY DATA................................................................................................................................................ 4 DATA COLLECTION............................................................................................................................................... 5 LOCATION, SAMPLING, AND SELECTION PROCESS ................................................................................ 5 DATA ANALYSIS....................................................................................................................................................... 8 LIMITATIONS ...........................................................................................................................................................10 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS .......................................... 11 ACCURACY OF RAINFALL PREDICTION TOOLS......................................................... 11 EQ1: TO WHAT EXTENT HAS THE RAINFALL PREDICTION MODEL/TOOL BEEN APPLIED? IS THE MODEL RELIABLE AND ACCURATE?....................................................................................................11 FINDINGS.........................................................................................................................................................11 CONCLUSIONS..............................................................................................................................................24 RECOMMENDATIONS................................................................................................................................. 24 UTILITY AND INFORMATION DISSEMINATION OF RAINFALL PREDICTION...... 26 EQ2: WHAT ARE THE UTILITIES OF THE RAINFALL PREDICTION MODEL/TOOL FOR FARMERS?..................................................................................................................................................................26 FINDINGS.........................................................................................................................................................26 CONCLUSIONS..............................................................................................................................................32 RECOMMENDATIONS................................................................................................................................. 33 EQ3: BY COMPARING THE TRADITIONAL SYSTEM AND ONLINE APPLICATION AS A MEANS OF INFORMATION DISSEMINATION, WHICH SYSTEM IS MORE EFFECTIVE FOR FARMERS? AND WHY?...............................................................................................................................................................33 iii | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV FINDINGS.........................................................................................................................................................33 CONCLUSIONS..............................................................................................................................................36 RECOMMENDATIONS................................................................................................................................. 36 EARLY WARNING SYSTEM CHECKLIST......................................................................... 37 EQ4: IN WHAT WAYS HAS THE TOOL BEEN UTILIZED FOR DISASTER PREPAREDNESS?.......37 FINDINGS.........................................................................................................................................................37 CONCLUSIONS..............................................................................................................................................42 RECOMMENDATIONS................................................................................................................................. 43 ANNEXES............................................................................................................................... 44 ANNEX I: EVALUATION STATEMENT OF WORK.....................................................................................44 ANNEX II: EVALUATION QUESTIONS IN THE APPROVED INCEPTION REPORT........................53 ANNEX III: DATA COLLECTION AND ANALYSIS TOOLS .....................................................................55 ANNEX IV. EVALUATION DESIGN MATRIX ................................................................................................65 ANNEX V: SOURCES OF INFORMATION.....................................................................................................67 ANNEX VI: DISCLOUSRE OF ANY CONFLICTS OF INTEREST .............................................................71 ANNEX VII: EVALUATION TEAM MEMBERS................................................................................................. 74 USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | iv LIST OF FIGURES Figure 1. Flow chart of location selection............................................................................................................. 7 Figure 2. The period for model development was for 1979-2016 ...............................................................14 Figure 3. Distribution of observed rainfall stations (red dots) in West Lombok, Central Lombok, East Lombok, and Dompu...............................................................................................................................................18 Figure 4. Distribution of accuracy of rainfall prediction using CHIRPS satellite rainfall data in Nagekeo for 2020-2022............................................................................................................................................................20 Figure 5. Spatial comparison between PCH forecast and corrected CHIRPS satellite rainfall data .....22 Figure 6. Inconsistency of displaying data between rainfall data in map (color) and graph in Towak Village, Nagekeo........................................................................................................................................................23 Figure 7. General cropping system of agriculture in selected villages..........................................................26 Figure 8. The percentage of climate impacts and climate-related disaster in selected villages..............26 Figure 9. The awareness and the application of PCH by farmers in surveyed villages............................30 Figure 10. Disaster information provided by the tools...................................................................................38 Figure 11. KMPB response on the use of disaster prediction from tool for disaster preparedness....38 LIST OF TABLES Table 1. Priority index of surveyed villages in five districts.............................................................................. 8 Table 2. Number of rainfall stations for development of the PCH tool in each district.........................11 Table 3. Villages for quality assessment of the accuracy of rainfall forecasts.............................................15 Table 4. Criteria for a good accuracy based on three statistical indicators...............................................15 Table 5. Summary of the accuracy checking of rainfall prediction by key informants.............................16 Table 6. Accuracy of PCH based on observed rainfall at station level using the three statistical indicators....................................................................................................................................................................19 Table 7. Accuracy of online tool based on CHIRPS satellite rainfall data using the three statistical indicators....................................................................................................................................................................20 Table 8. The accuracy of PCH forecast (in % of total stations) for 2018-2022 in four districts...........21 Table 9. Cropping system and climate issues on agricultural land in surveyed villages...........................27 Table 10. The impact on the use of climate information in planting decisions in surveyed villages.....29 Table 11. Planting decisions by farmers in selected villages...........................................................................31 Table 12. PCH application impacts on production...........................................................................................32 Table 13. Climate information dissemination received by farmers in surveyed villages..........................34 Table 14. Response of KIs to the use of PCH for disaster preparedness..................................................41 v | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV ACRONYMS APIK Climate Change Adaptation and Resilience Balitklimat Research Institute for Agro-climate and Hydrology BMKG Bureau of Meteorology, Climatology and Geophysics Indonesia BNPB National Disaster Management Agency BPBD District Disaster Management Office BPP Balai Penyuluhan Pertanian/Agriculture Extension Center BRIN National Research and Innovation Agency CHIRPS Climate Hazards Group Infra-Red Precipitation with Station Distan Agriculture division of district government DRR Disaster risk reduction EQ Evaluation Question ET Evaluation Team FPRB Forum Pengurangan Risiko Bencana (Disaster Risk Reduction Forum) FC Field Coordinator FGD Focused Group Discussion GOI Government of Indonesia IP Implementing Partner ITB Bandung Technology Institute ITS Sepuluh Nopember Institute of Technology IPB IPB University (Bogor Agricultural University) KATAM Dynamic crop calendar developed by Ministry of Agriculture Kementan Ministry of Agriculture KII Key Informant Interview KLHK Ministry of Environment and Forestry KMPB Disaster Care Community Group LESPEL Lembaga Studi Pengkajian Lingkungan/Environmental Assessment Study Institute LG Local Government LPSDM Lembaga Pengembangan Sumber Daya Mitra/Partner Resource Development Institute MCH Maternal and Child Health MEL Monitoring, Evaluation, and Learning MEL-P Monitoring, Evaluation, and Learning Platform USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | vi MEWS Multi-Hazard Early Warning Systems NGO Non-governmental organization NTB West Nusa Tenggara Province NTT East Nusa Tenggara Province PSP Pusat Studi Pembangunan/Center for Development Studies PCH Prediksi Curah Hujan (Rainfall prediction tool) PPL Facilitator from Agriculture Office SOW Statement of Work SLI Sekolah Lapang Iklim (Climate Field School) RCDP Rainfall, Cropping Patterns and Disaster Projections RMSE Root-mean-square error TOC Theory of Change USAID United States Agency for International Development USAID/BHA USAID’s Bureau for Humanitarian Assistance WMO World Meteorological Organization WN World Neighbors YMTM Yayasan Mitra Tani Mandiri Flores/Flores Independent Farmer Partners Foundation vii | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY USAID/Bureau of Humanitarian Affairs (BHA) has been supporting the non-governmental organization (NGO), World Neighbors (WN) and its partner Bandung Institute of Technology (ITB), on their activities, including the development of a regency-specific and seasonal rainfall prediction tool. The tool predicts suitable planting time in the season and commodities, the potential for pests and plant infestations, and the likelihood of climate-related disasters. The tool has been adopted in five districts in West Nusa Tenggara (NTB) and East Nusa Tenggara (NTT). Under Monitoring, Evaluation, and Learning Platform (MEL-P), USAID/BHA assigned Panagora to evaluate the utility, accuracy, and effectiveness of the rainfall prediction tool for 2018-2022. Four evaluation questions (EQs) are answered in this evaluation. The EQs are: 1. To what extent has the Rainfall Prediction Tool (PCH) been applied? Is the model reliable and accurate? 2. What is the utility of the Rainfall Prediction Model/Tool for farmers? 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 4. In what ways has the tool been utilized for disaster preparedness? The evaluation employs a mixed-methods approach and uses both primary and secondary data. The evaluation team (ET) gathered primary information through interviews with key informants (KIs) including national experts and governments, related provincial and regional (district) governments, village governments, beneficiaries, farmers, and disaster groups. The ET used secondary data consisting of existing documents and analyzed biophysical data (e.g., the presence of climate-related disasters, level of village vulnerability, and rainfall level). The ET employed a combination of descriptive analysis for the adoption of the rainfall forecast based on focused group discussions (FGDs) and interviews with KIs, quantitative analysis of the observed rainfall, and Causal Attribution Analysis, which is used to understand the causal attribution or link between observed changes and adoption of the forecast information. The ET also used standard statistical methods to evaluate the accuracy of rainfall forecasts from the tool compared to actual rainfall data. EQ1: To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? The ET conducted quantitative and qualitative analyses to assess the reliability and accuracy of the prediction tool (PCH). From documents and interviews, we found: • PCH provides rainfall forecast at decadal (10-day) periods with 5-year lead time on a kriging map (fits to village level), and it was updated every two years. From interviews, we found that the current climate forecast is for 3-6 months ahead or a maximum of up to 12 months lead time with a monthly update. Rainfall forecast beyond 12 months ahead is in the realm of annual-to-decadal prediction. • The Bureau of Meteorology, Climatology and Geophysics Indonesia (BMKG) has provided similar forecast information as PCH. The spatial resolution of the forecast is only up to sub￾district level with three-month lead time, but it is updated every month. Farmers and local government considered that this level of resolution is not very helpful for making cropping decisions. With every monthly update, the forecast information from BMKG provide the update. • The PCH used Fast Fourier Transform (FFT), which relies on a forecasting time series from one data (rainfall) without involving the input of climate elements related to climate control USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | viii factors. The results of the FFT tend to be repeating patterns and are not sensitive to climate anomalies, especially on the interannual scale. • The use of ensemble models that combine statistical and dynamic models will improve rainfall forecasts, especially in regions where the local factor is dominant, such as in Nusa Tenggara. Using a solely statistical model may lead to underestimating extreme events. • From an accuracy analysis using 57 stations rainfall data from Bureau of Meteorology, Climatology and Geophysics (BMKG), PCH showed low accuracy in all stations. Using Climate Hazards Group Infra-Red Precipitation with Station (CHIRPS) data, PCH showed good accuracy in some locations, but the accuracy completely drops in the next year’s prediction throughout all locations. • KIs from villages and farmers said that they believe the PCH was accurate, although they did not directly check the tool’s accuracy with data in the field. • The current direction for developing climate predictions is to make more use of the outcomes from global climate models (dynamic climate models), which are downscaled to local areas, and to utilize the outputs of many models (multi-model ensemble) to overcome uncertainty problems both due to internal factors from the climate system and from the various models used. BMKG has already used a multi-model ensemble. • Adoption of any methods/approaches for rainfall forecast by BMKG is possible. The Government Regulation Number 13/2018 mandates the adoption, and it can be reached and facilitated through the National Climate Expert Forum. RECOMMENDATIONS EQ1 ● Future project activities should have a strong collaboration with BMKG to avoid any potential hindrances (e.g., access to rainfall data) and to ensure the system’s adoption and use meets public information standards. ● Future project activities should facilitate BMKG in developing a Standard Operating Procedure (SOP) for adopting a new rainfall forecast model to improve the national seasonal climate prediction system and enrich the ensemble models. ● The tool developer should integrate the forecast skill in the tool (spatial accuracy of the forecast) to allow farmers to benefit from using the forecast. ● The tool developer should train WN and local partners to identify potential use of rainfall forecasts, taking into consideration the skill of the forecast. EQ2: What are the utilities of the Rainfall Prediction Model/Tool for farmers? The farmers we interviewed have mostly experienced climate-related disasters, such as false rain and season breaks. Under the disaster, crops are exposed to prolonged dry spells that significantly affect crop planting and later yield. Based on interviews, we found the following key findings: • Adoption of climate forecast information can help farmers in reducing expenses (e.g., avoiding planting failure), increase yield, and increase planting intensity with appropriate crop types. • Most farmers interviewed still used natural signals and rainfall occurrences that follow the natural signal in making planting decisions, and PCH information was considered. • Farmers that adopted the PCH self-report that they have experienced increased yields. However, no standard method is used for evaluating the impact objectively. ix | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV RECOMMENDATIONS EQ2 • WN and local partners should support the Agriculture Office at the district level to institutionalize the use of climate forecasts for cropping. The socialization of climate forecasting and assistance to tailor the cropping to the forecast should be a routine activity that does not depend on the project. • The future USAID project should focus more on institutionalizing Sekolah Lapang Iklim (Climate Field School-SLI), which have similar aims and information to this project, at the village level as part of a local government program (Agriculture Office). The collaboration will provide one source of information, which may generate better results. • The Agriculture Office (with assistance from donors/projects) routinely holds training of trainers (ToT) sessions for field facilitators (extension workers), which is in line with the BMKG program at SLI. • WN and local partners should develop synergy with the SLI. By doing so, the facilitators will get acquainted with the Standard Operation Procedure to read and understand the climate info and how to interpret the climate information. • WN and local partners should develop a standard method to evaluate the benefit for farmers from using climate forecast information. EQ3: By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? Dissemination of rainfall forecast information in the traditional system uses the PCH map, and the one in the online system uses an online web or Android application. In the traditional system, PCH information was disseminated by printing PCH maps and distributing them to farmers and village offices. The following key findings are based on interviews: • The traditional system is more suitable for dissemination advice in the project sites given that most farmers lack smartphone access and have limited internet networks. • Most farmers are illiterate and need frequent advice about PCH/climate information from the Field Facilitator from the Agriculture Office (PPL) and its uses for making cropping decisions. • Many farmers that received the forecast information still used natural signals and rainfall events following the signals in making planting decisions. RECOMMENDATIONS EQ3 • The project should continue to support the use of maps for dissemination and socialization of rainfall forecast (traditional system). Future projects should prioritize such socialization at least once every planting season. • The current project should collaborate strongly with field PPL, such as through intensive training on how to effectively provide advice to farmers. • WN and local partners should integrate the use of weather/climate information from regional BMKGs in the training of PPL considering the PCH/apps are updated online once every two years, while BMKG provides updated forecasts on a monthly basismonthly. • WN should collaborate with BMKG to share best practices and lessons learned from the project regarding the approach in communicating forecast information to farmers so BMKG can adopt and integrate the lesson into the Climate Field School program, particularly in producing rainfall forecast information maps relevant for cropping decisions. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | x EQ4: In what ways has the tool been utilized for Disaster Preparedness? The WN and its local partners conducted socialization of the project that involved broad stakeholders at the district and village levels and intense contact with agencies that were directly involved in the project, particularly the Agriculture Office, Local Disaster Office (BPBD), and Disaster Risk Reduction Forums (FPRB) at the district level; and village staff and Disaster Care Community Group (KMPB) at the village level. The following are the key findings from the interviews: • BPBD uses BMKG information as the primary disaster information source, and the PCH was used for additional information. • BMKG information is sent daily through the WhatsApp group of leaders in the Regional Government. This facilitates early coordination between members of the WA group in managing the disaster (particularly for floods and earthquakes) before a formal state of emergency is issued by the Regional Government. • Assessment of Desa Tangguh Bencana (Disaster Resilience Village) has been carried out annually, using 60 indicators divided into several parameters, such as Legislation, Planning, Institution, Funding Capacity Building, and Disaster Management Practice. • Regarding the Multi-Hazard Early Warning System (MEWS) Checklist, Desa Tangguh Bencana generally focuses on the parameters of disaster risk knowledge and preparedness and response capabilities, not on the parameters of detection, monitoring, analysis, and forecasting of the hazards and possible consequences or warning dissemination and communication that are critical for defining Desa Tanguh Bencana. • Some villages and KMPB are still not aware of the Desa Tangguh Bencana status and the change of personnel may contribute to this condition. RECOMMENDATIONS EQ4 • Future project activities should focus on public engagement on climate information. • The future project should support and facilitate local government and stakeholders to increase public awareness of climate-related disasters by integrating climate information into regulations. • The future project should facilitate BNPB/BPBD to adopt the use of EWS Multi-Hazard Criteria of the World Meteorological Organization (WMO) in assessing Desa Tangguh Bencana (Disaster Resilience Village). • The dissemination of Desa Tangguh Bencana should involve local stakeholders and the key person at the village level. Therefore, they understand the meaning behind “resilience.” 1 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV BACKGROUND USAID/BHA has been supporting the NGO World Neighbors (WN) and their activities in the eastern Indonesian provinces of West Nusa Tenggara (NTB) and East Nusa Tenggara (NTT). These activities include the formation and training of Village Disaster Management Committees and working with village governments to achieve resilient village status per Government of Indonesia metrics; support to regency-level disaster risk reduction (DRR) multi-stakeholder forums; village disaster risk assessments and risk management plans; undertaking village-level projects to reduce risk and improve resilience; and promotion of climate-smart agriculture technologies to better cope with rainfall variability. Among the latter support, World Neighbors partnered with Bandung Technology Institute (ITB) staff to develop a regency-specific and Rainfall Prediction Tool to advise farmers on what and when to plant for each rain-fed cropping season and on the potential for pests and plant infestations. The tool also illustrates the likelihood of climate-related disasters (e.g., flooding, landslides, droughts) and the spread of malaria and dengue fever, allowing governments and communities to better prepare for disasters. Two models have been developed: a monthly rainfall forecasting model and a ten-day model. The predictions cover five years, from 2018 to 2022. The tool has been adopted in five districts in NTB and NTT and covers the regencies of Dompu, Central Lombok, West Lombok, and East Lombok in NTB and Nagekeo in NTT. WN facilitated an external assessment of the impact of rainfall, cropping patterns, and disaster projections (RCDP) in the five target project regencies for 2014-2018. From 336 respondents including farmers, village and regency governments, and the DRR Forums, the assessment found that: 1) the accuracy of the rainfall predictions in years one and two is more than 80 percent, and after that decreases; 2) the rainfall prediction is beneficial for farmers in determining their planting calendar and selection of plants suitable for the rainfall; 3) the community and government are more prepared in the allocation of contingency funds to face disasters, especially floods; and 4) regency governments are committed to developing the Rainfall, Cropping Patterns and Disaster Projections (RCDP) in the form of a website or application to make them more accessible to the public. Based on the assessment, the RCDP has affected the community (farmers) and regional government, including 1) the reduction of losses due to crop failure and harvesting and an increase in community preparedness by knowing the potential for disasters; 2) people are increasingly aware and understand the importance of cultivating more local food crops because they can adapt to erratic rainfall; and 3) regional government planning, especially by the BPBD, is more focused, based on the results of both the disaster risk assessment and rainfall/potential disaster projections. Under MEL-P, USAID commissioned Panagora to evaluate USAID/BHA. The evaluation is intended to verify the utility, accuracy, and effectiveness of the Rainfall Prediction Tool developed for farmers in the provinces of NTB and NTT for the period 2018-2022 and to inform future environmental activities related to climate change adaptation and resilience. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 2 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE This evaluation aims to assess the accuracy, effectiveness, and utility of the seasonal Rainfall Prediction Tool that World Neighbors and their partners have developed at ITB. This evaluation will inform USAID/BHA and its implementing partner, World Neighbors, to allow them to verify the usefulness of the tool and the added value to participating farmers in reducing their climate risk. The results of this evaluation will also inform the design of a follow-on activity to the USAID/Climate Change Adaptation and Resilience (APIK) activity in urban settings by providing information on the accuracy and the utility of the tools and data from the BMKG to promote the dissemination and use of similar tools for early warning information, disaster preparedness, and climate change adaptation. The secondary audience includes vital stakeholders, such as BMKG, Kementan (Ministry of Agriculture), and other national and local agriculture departments, to allow them to adopt and institutionalize the rainfall prediction tool to support farmers in other regions of Indonesia. EVALUATION QUESTIONS USAID has provided the ET with evaluation questions as stated in the Statement of Work (SOW Annex 1) to assess the Seasonal Rainfall Prediction Tool’s accuracy, effectiveness, and utility. The evaluation team revised the original questions from the SOW to increase the coherence and alignment of the questions with the evaluation purposes, which were approved by USAID on June 21, 2022 (see Annex II). We provide a brief description of the focus of each question below: 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? a. What are the components/variables used in the model to predict/forecast rainfall? Where do they come from? b. What types of rainfall characteristics are being forecasted (season onset, duration, and intensity for a season)? What are the lead times of the forecast — how much in advance can the forecast be made? c. What are the methods used for rainfall forecasting? Statistical, dynamic, or statistical￾dynamic model? Ensemble models? d. What are the assumptions used in the model? e. What is the area covered by the forecast tool? Is it only for specific sites (districts)? What is the resolution? f. What is the period of data used for developing the model, as well as calibration/validation and verification of the model? And the period used for updating the forecast? g. What are the methods used to evaluate the accuracy of the forecast? h. What criteria are used to determine that the forecast is accurate? i. Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? Is the accuracy of the forecast conducted on a point base or spatial base? j. Are there other forecast models in use by BMKG that could be modified to produce similar forecasts? And if so, what is restricting their adoption? 2. What is the utility of the Rainfall Prediction Model/Tool for farmers? a. How does the climate affect farmers’ cropping system (e.g., change in season onset/false rain, season break, extreme rainfall, etc.)? b. How are the forecast results turned into advice to farmers? What is the advice? c. What percentage of farmers are using the tool? If it is low, why? Do they view it as a useful tool (provide examples), and why? 3 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV d. How is the advice used by the farmers for making planting decisions or other planting operations (provide the example and reason, e.g., adjust technology like delay/accelerate planting, change planting arrangement, irrigation scheduling, applying pesticide/herbicide; change commodity)? e. For farmers who used the rainfall prediction advice, what difference did it make in that seasonal production? f. For the farmers who adopt/follow the rainfall prediction, cropping pattern, and other guidance, what is the percentage increase in food crop production and/or income? g. How do you differentiate the increase in food production due to changes in or adopted improved farming practices from increases due to the adoption of forecasts? 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? a. What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? b. What are the main sources of information that farmers use to guide decision-making each season? What information do farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? c. Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in-person gatherings and those using the online application? d. Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places with no app support? 4. In what ways has the tool been utilized for disaster preparedness? a. What is the awareness level of stakeholders on the tool? b. Do they use the tool in disaster prediction? How is it used? What are the forms of disaster information provided based on the tool? c. Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? d. How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? By whom? e. How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? f. Do the results of rainfall predictions causing disasters facilitate the coordination of related agencies to prepare for disaster management? g. How is the disaster risk information disseminated and communicated broadly? h. Who are the government agencies/stakeholders/ community groups that use it? What are the benefits of the disaster prediction for the community, government agencies/stakeholders? (future benefit vs. current benefit)? The ET developed the interview guides based on the primary and sub-questions. However, the interviews were semi-structured and did not necessarily follow every sub-question depending on the respondent and time allocated. This approach allows respondents to answer questions in more detail and provide richer qualitative data. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 4 EVALUATION METHODS AND LIMITATIONS The evaluation team evaluated the accuracy, effectiveness, and utilization of the seasonal rainfall prediction tools. The evaluation applied a mixed-methods approach, using a combination of quantitative and qualitative data from primary and secondary data collection. Primary data includes qualitative data obtained through direct interviews with climate experts and discussions with WN and WN counterparts, tool developers, government, DRR Forums, KMPB (Disaster Care Community Group), farmer groups, and other stakeholders as appropriate. Secondary data includes existing documents and analyzed biophysical data (e.g., the presence of climate-related disasters, level of village vulnerability, and rainfall level). To ensure that the evaluation process and findings are comprehensive, objective, and evidence￾based, the evaluation team paid careful attention to triangulating data points from a broad range of sources, including a combination of quantitative and qualitative data and discussions with a diverse range of stakeholders as appropriate. PRIMARY DATA The evaluation of the rainfall prediction tool required gathering data from key stakeholders at the national, provincial, district, and village levels with knowledge and experience in rainfall prediction or who have developed, received, or utilized the tool. We provide the list of key informants by respondent category in the form of a stakeholder matrix in Annex III. Before starting fieldwork, the ET and USAID/BHA discussed and finalized documents for the interview. The documents comprise the list of key informants and the corresponding interview templates (Annex II). With approval from USAID/BHA, the ET identified the KIs related to the tool development and application (in the context of WN activity). We arranged for interviews or small group discussions in Jakarta/Bogor and the project locations (Lombok Barat/West Lombok, Lombok Tengah/Central Lombok, Lombok Timur/East Lombok, Dompu, and Nagekeo districts). Some interviews and discussions were also arranged remotely with representatives of USAID, the Government of Indonesia (GOI), and other project partners upon request. If the targeted informant was not available for an interview, the ET sent the list of written questions based on EQs (see Annex II). The availability and preference of respondents likely influenced the structure of the review. For small group interviews, the ET developed semi-structured guidelines (templates) for these interviews based on the respondents and their involvement with WN activities. The interviews were aimed at beneficiaries (farmers), field officers, village government officers, and the head of the farmer community where the tool is being implemented. The ET adjusted the review template based on the respondent’s knowledge. SECONDARY DATA The evaluation team performed a desk study review of the existing documents. We also analyzed biophysical data (e.g., the presence of climate-related disasters, level of village vulnerability, rainfall, and type of cropping system). By reviewing the documents, the ET gained a more profound knowledge of the project, which assisted in developing this evaluation report. The document review involves all WN reports and technical documents, such as final reports (2018 and 2021), an evaluation report (from SATUNAMA), training documents, and other products and publications such as books and technical documents. Also, the ET utilized relevant documents from 5 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV GOI, such as products of data and information from the Regional Statistical Bureau. Annex III provides a list of documents for review. In addition to the documents, the ET required biophysical and socioeconomic data from various sources as an essential requirement to evaluate the tool. The biophysical data comprises a series of daily rainfall for 2010-2021 for each district, the villages where the project was implemented, village boundary data in GIS format, and the village’s vulnerability level. ET required information on cropping patterns and cultivation methods in each village and climate-related disaster events (floods) for socioeconomic data. DATA COLLECTION The ET systematically collected primary and secondary data. The ET used interview templates to guide the interview process consistently for all respondents and locations (villages). The interviews were in Bahasa, Indonesia. Team members recorded and transcribed the notes and cross-checked, edited, and validated the data. Following cross-checking, the notes were finalized through discussion, validation, and approval by the Team Leader. The ET members translated the interview outputs into English accurately, fairly, and without bias. After the interview, the ET members summarized the interview from their notes and recordings, highlighting key findings and recommendations. The ET extracted the findings that needed follow-up and further investigation and verification. Following cross-checking, the notes were finalized through discussion, validation, and approval by the Team Leader. The ET ensured that the evaluation respondents were selected purposely based on their village location (low plane, high plane), gender, and social-inclusion criteria. The ET also interviewed government representatives (at the national, provincial, district, and village levels), BMKG experts, professional climate experts, tool developers, and WN’s local partners. The ET, WN team, and USAID/Indonesia jointly determined the final list of respondents. Due to the COVID-19 pandemic, some interviews were conducted remotely via phone, WhatsApp, Zoom, etc. The ET also requested written responses to the evaluation questions from the tool developer. Fieldwork was carried out three times over a total of 19 days: 1. July 24 – August 2, 2002, to interview farmers, local governments, WN local partners, and field facilitators in Central Lombok and Dompu districts (West Nusa Tenggara). 2. August 7 – 12, 2022, to interview farmers, local governments, WN local partners, and field facilitator in Nagekeo district (East Nusa Tenggara). 3. September 11-16, 2022, to interview farmers, local governments, WN local partners, and field facilitators in West Lombok and East Lombok (West Nusa Tenggara). LOCATION, SAMPLING, AND SELECTION PROCESS The evaluation team followed a framework for selecting locations for a survey (collecting primary data) based on the following criteria: (i) the presence of climate-related disasters, (ii) the level of village vulnerability, (iii) climate threat, and (iv) type of cropping system. These four criteria ensured that the villages selected for the survey were exposed to high climate risks related to existing farming practices. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 6 Most farmers are responsive to climate information in locations exposed to high climate risk. The ET sampled villagers from the villages that have been identified and fulfilled the four defined criteria. Selection of Villages for Interviews Criterion 1: The presence of climate disaster. Each village was identified based on historical climate￾related disasters such as floods. If there is no record of the disaster, the respective village gets a score of zero (0); otherwise, it is one (1). Criterion 2: Level of village vulnerability. The data-related vulnerability was based on the SIDIK system (Ministry of Environment and Forestry, GOI). There are five categories from 1 to 5, in which 1 indicates the lowest vulnerability and 5 shows the highest vulnerability. Criterion 3: Climate threat. The criterion was based on historical rainfall data from BMKG. We processed rainfall data to obtain its statistical probability for false rain occurrence, seasonal break, length of the rainy season, and extreme rainfall. False rain is the condition in the transition season (dry to wet) where rainfall is less than 120 mm/month. Season break refers to the condition in the rainy season with no rainfall events for 15 days in a row. The length of the rainy season represents the last part of the season when each monthly rainfall amount is above 120 mm. The probability is analyzed based on the 5-yr, 10-yr, and 15-yr return period (RP). There are three categories for this probability: (i) 1, if RP>10-year; (ii) 2, if RP is between 5-10-year; (iii) 3, if RP < 5-year. Criterion 4: Cropping pattern. With an assumption that the farmers will start their cultivation in sufficient water conditions, the length of the rains can explain the general cropping pattern in such a village. The probability of the length of the rainy season has been mentioned earlier in Criterion 3. Villages for assessment are determined based on criteria 1-3. We overlaid these criteria to obtain the most suitable villages for assessment (priority village for interview; Figure 1). Then we combined the villages assessment and criterion 4 to get data from the selected villages for interviews (Figure 1). By applying the criteria, we expected that the villages are exposed to high climate risk related to existing farming practices. 7 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Figure 1. Flow chart of location selection The selection of villages was based on the total score of the village using the criteria and considering the type of cropping pattern in the villages. The selected villages should be a high priority (high score) and represent specific cropping systems. Considering the time and the available resources, the interviews were conducted in three villages in each district (two fostered villages and one non￾fostered village). Fostered villages are the villages selected by the project (beneficiary villages) and non-fostered villages are non-beneficiary villages. In addition, we added two more villages for online interviews. In total, 17 villages across five districts were selected to be part of the evaluation sample. We have four villages in West Lombok and East Lombok, while the other districts have three each (Table 1). The villages selected based on the above four criteria are presented in Table 1. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 8 Table 1. Priority index of surveyed villages in five districts DISTRICT VILLAGES BENEFICIARY PRIORITY INDEX LOCATION CATEGORY OF PRIORITY West Lombok Cendi Menik Yes 3.64 Very high Taman Baru Yes 3.09 Very high Giri Sasak Yes 2.98 Very high Batu Layar* No 5.83 Very high Central Lombok Tiwugalih Yes 1.49 Very high Kabul Yes 1.14 High Semayan No 1.55 Very high East Lombok Padak Guar Yes 1.19 High Puncak Jeringo Yes 1.34 High Gunung Malang Yes 1.17 High Surabaya* No 1.38 High Dompu Nusa Jaya Yes 1.55 Very high Pekat Yes 0.99 Medium Nowa No 1.17 High Nagekeo Wajomara Yes 1.30 High Pagamogo Yes 1.37 High Nangaroro No 1.32 High Note: * only online interviews In each selected village, the ET carried out interviews with one or two farmer groups (1-5 members). The ET also conducted a mini focus group (1-3 members) from one non-beneficiary farmer group (KMPB) in each village. From the interview processes, we expected to obtain information from the end-users about their knowledge of the tool and its application. DATA ANALYSIS The evaluation team employed a mixed-methods approach to data analysis. For example, quantitative data analysis primarily focused on using existing data from the document review, self-assessments, biophysical data, etc. Qualitative data analysis was conducted using primary sources, including interview notes from key informant interviews (KIIs) and secondary sources from the WN project. When the evaluation team requested written responses to questions or conducted KIIs, this was also included in the qualitative data analysis. After completing an interview, the team transcribed the notes into a Google document or an MS Word document. According to the primary EQs, the qualitative data in each interview transcript was coded into categories to identify the source and themes. A minimum of two types of coding were applied, including topic coding and open coding. Topic coding involves arranging the raw qualitative data according to the evaluation questions. Data may also be organized into additional relevant codes (or themes) during the coding process, including the data sources determined to answer the key EQs or better capture performance related 9 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV to sub-questions. Open coding refers to notes of the key findings based on the ET’s perspective. The evaluation team looked for common patterns and outliers throughout the coding process. To ensure that the evaluation findings and conclusions are objective and evidence-based, all data were compiled using an Evaluation Matrix and triangulated using the following methods: (i) Source Triangulation: Compare information from different sources; (ii) Method Triangulation: Compare information collected by different methods and techniques, including documents, KIIs, small group discussions, FGDs, direct observation, questionnaires, etc., and (iii) Evaluator Triangulation: The evaluation team members assess the data and the findings and compare common and divergent views. Descriptive Analysis and Causal Attribution – The evaluation team employed a combination of descriptive analyses, which analyze and describe “what happened” or what changes occurred because of the adoption of the rainfall forecast, based on FGDs and KIIs and by comparing the predicted rainfall forecast data to the observed rainfall, and causal attribution analysis, which is used to understand the causal attribution or link between observed changes and adoption of the forecast information. The ET used standard quantitative methods to evaluate the accuracy of rainfall forecasts from the tool. The ET conducted quantitative analysis for this at the station and spatial levels and at different periods of prediction (next 1-5 years). According to the documents, the accuracy was measured based on the number of predictions matched with observed rainfall. The ET focused on testing the tool’s accuracy for the data that are not for calibration purposes but verification purposes for 2019- 2022. Before its wide application to end-users, the tool shall meet a good performance for calibration (developing the tool) and verification processes (testing the tool). The accuracy of rainfall prediction was evaluated on a 10-day basis (dasarian). The ET used daily rainfall data from BMKG stations accumulated over 10 days for each station. The accuracy of prediction was evaluated statistically using three indicators, namely the percentage of bias (PBIAS), root-mean-square error (RMSE), and coefficient of determination (R2). PBIAS helps identify average model simulation bias (over prediction vs. under prediction). The model is good if PBIAS is in the range of -15 percent to +15 percent. Indicator RMSE provides an error of prediction in the same unit, with 0 meaning a perfect model fit. Indicator R2, which ranges from 0 to 1, describes the proportion of the variance in measured data explained by the model. The model is acceptable if R2 > 0.5. The formulas for PBIAS, RMSE, and R2 are as follows: 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 = ∑𝑛𝑛 𝑖𝑖=1 𝑂𝑂𝑖𝑖−𝑃𝑃𝑖𝑖 ∑𝑛𝑛 𝑖𝑖=1 𝑂𝑂𝑖𝑖 (2) 𝑅𝑅𝑅𝑅 𝑅𝑅 = �1 𝑛𝑛 ∑𝑛𝑛 𝑖𝑖=1 (𝑂𝑂𝑖𝑖 − 𝑃𝑃𝑖𝑖)2 (3) 𝑅𝑅2 = � ∑𝑛𝑛 𝑖𝑖=1 �𝑂𝑂𝑖𝑖−𝑂𝑂��𝑃𝑃𝑖𝑖−𝑃𝑃� �∑𝑛𝑛 𝑖𝑖=1 �𝑂𝑂𝑖𝑖−𝑂𝑂� �∑𝑛𝑛 𝑖𝑖=1 �𝑃𝑃𝑖𝑖−𝑃𝑃� 2 � 2 (4) Where O is observed rainfall and P is predicted rainfall; units are in mm. Forecast accuracy is for verification purposes, and the ET selected rainfall for 2019-2022 when the data is not for model calibration. In each station, we calculated the three statistical indicators mentioned above (Equations 2-4). As the output of the prediction tool is on a spatial basis, we USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 10 applied the bias-corrected CHIRPS rainfall for verification purposes against spatial rainfall prediction. Then we applied statistical indicators in Equations 2-4. The ET also conducted a match analysis to evaluate the accuracy of the forecast following participatory match analysis employed by WN whenever time allowed. A match is grouped into three categories: precise (tepat) if the observed and predicted rainfall is accurate in 12 months per year; somewhat precise (kurang tepat) if the observed and predicted rainfall is accurate between 6-11 months per year; and not precise (tidak tepat) if the accuracy only falls < 6 months per year. For verification, selected farmers were interviewed (mini group discussion) on weather conditions in 2021-2022. Criteria for weather conditions at 10-days are heavy rainfall (hujan lebat), medium rainfall (hujan sedang), and light rainfall (hujan ringan). LIMITATIONS It is important to note that the ET was exposed to limitations in performing the evaluation. However, the potential limitations were anticipated beforehand. Some new limitations arose during the data collection and analysis process. Any limitations were shared with USAID, including proposed and acted mitigation measures when necessary and appropriate. The limitations include the following: • A limited number of villages were selected in each district, three villages on average, to be considered representative of each district. • The evaluation of impact of applying the rainfall prediction tool on crop productivity mainly depended on self-reported information from farmers without any field measurement, formal documents, or reports from trusted institutions. Therefore, the evaluation could be subjective. The project can use crop productivity data from foster and non-foster villages collected by the Agriculture Office through field measurement in the season when the majority of farmers in the foster villages have tailored their decisions to the rainfall forecast. The non-foster villages selected for the comparison should be neighbor villages where the difference in crop productivity of non-foster and foster village in seasons before the tools was applied was not significant. • In some cases, the KI that attends the interview may not be representative enough (for example, for a village, only one farmer is available to be interviewed), and some KIs suddenly could not attend the scheduled interview. Additional online interviews with farmers and KIs have been conducted to mitigate this limitation. • The information from the tool developer is limited as an interview with the tool developer could not be scheduled by the ET. The ET conducted evaluation by sending written questions via email and follow-up questions as required to the written response provided by the tool developer. The tool developer provided written response to the questions but not to the follow-up questions. • Access to rainfall prediction data from the model was not available. Thus, the data were manually read directly through maps (.jpeg format) from the color that may not give the precise value (expected deviation of five percent). 11 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS This section includes the findings, conclusions, and recommendations for each EQ. ACCURACY OF RAINFALL PREDICTION TOOLS EQ1: TO WHAT EXTENT HAS THE RAINFALL PREDICTION MODEL/TOOL BEEN APPLIED? IS THE MODEL RELIABLE AND ACCURATE? FINDINGS EQ1.1: What are the components/variables used in the model to predict/forecast rainfall? And where do they come from? The main components for rainfall prediction/forecast are rainfall data. Typically, long-term rainfall data for at least ten years is required to represent or capture extreme climate conditions such as El Niño and La Niña phenomena. For the PCH/online tool, it used 5-10 years of observed rainfall data, which varied from 1979-2018. In rainfall prediction modeling, this data was used to calibrate the Fast Fourier Transform model. Therefore, the model prediction fits the observed rainfall. This step is to check the accuracy of a model’s results. Rainfall data were obtained from regional BMKGs in Lombok, West Nusa Tenggara, and from the local agriculture offices in each district where the project was implemented. Table 2 presents the number of rainfall stations in each district for developing the PCH model, which varies from 6-8 stations in West Nusa Tenggara and 16 stations in East Nusa Tenggara. Also, the prediction required CHIRPS rainfall satellite data, especially for predicting rainfall in the villages where the observed rainfall data is not present. Table 2. Number of rainfall stations for development of the PCH tool in each district NO DISTRICT PROVINCE NUMBER OF STATIONS 1 West Lombok West Nusa Tenggara 7 2 Central Lombok West Nusa Tenggara 6 3 East Lombok West Nusa Tenggara 8 4 Dompu West Nusa Tenggara 7 5 Nagekeo East Nusa Tenggara 16 EQ1.2: What types of rainfall characteristics are being forecasted (season onset, duration, and intensity for a season)? What are the lead times of the forecast — how much in advance is the forecast being made? Based on PCH documents, we summarized the characteristics of PCH forecast as follows: The rainfall characteristic being forecasted/predicted is rainfall intensity at a 10-day (decadal) period. The rainfall prediction for each rainfall station is in Table 2. Then in each 10-day period, the USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 12 prediction was interpolated to the district level with a Kriging technique. During the interpolation, rainfall patterns from the CHIRPS rainfall satellite data were applied. The 10-day rainfall prediction is used to determine the season onset and duration. The PCH/online tool was developed to predict rainfall with a lead time of five years. To get better results, the tool was updated every two years. The online tool was first using the observed rainfall data from the 44 stations (Table 2) for 2019-2020. The PCH got several critiques from the experts we interviewed. The critiques focused on the terminology used, models, and the lead time of forecast/prediction, as follows: • PCH prediction of 1-5 years ahead is in the realm of annual-to-decadal predictions, not the seasonal climate forecast. Risbey et al. (2001) stated that the period of climate forecast is on a monthly scale with a year of lead time. • The ability of PCH to predict rainfall with five years lead time that solely depend on a single FFT statical model may be low and it may result inaccurate forecast. The results of the FFT tend to be pattern repeating and are not sensitive to climate anomalies, especially on the interannual scale. Climate modeling community commonly use dynamic climate model such as ensemble Global Climate Model (GCM) for interannual scale prediction. Goswami et al. (2015,2017) showed that a GCM with an atmospheric component was able to simulate one five-year climate forecast in a tropical region. • All experts working on climate forecasting models (from BMKG, universities, and research agencies) had a similar opinion on the use of seasonal climate prediction of 3-6 months or a maximum of up to 12 months lead time with monthly updates. The longer lead time will lead to a decrease in forecast accuracy (Liu et al. 2019). The most preferred lead time for farming season is one month (Nyadzi et al., 2019). EQ1.3: What are the methods used for rainfall forecasting? Statistical, dynamic, or statistical-dynamic model? Ensemble models? The PCH/online rainfall prediction tool was developed based on a solely statistical method called FFT. It is a purely statistical model that depends on the data length and data itself to predict the future. The results of the FFT tend to be pattern repeating and are not sensitive to climate anomalies, especially on the interannual scale. Rainfall is the most dynamic weather/climate variable that globally depends on the complex interaction of the sea, land, and atmosphere. As an inherently statistical model, FFT does not cover this interaction. This is a weakness of FFT that most interviewed experts are concerned about. The current direction for developing climate predictions is to make more use of the outcomes from global climate models which are downscaled for local areas and to utilize the outputs of many models (multi-model ensemble) to overcome uncertainty problems both due to internal factors from the climate system and from the various models used. In the field of climate, dynamic models are commonly used for seasonal climate forecasts with their ability to forecast climate extremes on a seasonal time scale. For example, the European Centre for Medium-Range Weather Forecasts (ECMWF) can do long-range forecasts monthly from 0 to 7 months (https://www.ecmwf.int/en/forecasts). For the annual scale, ECMWF forecasts quarterly, up to 13 months ahead. 13 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV With different rainfall regimes/types in Indonesia, BMKG uses ensemble models that combine dynamic and statistical models for climate forecasting. One of the reasons is that no single forecasting system works nationally at different rainfall regimes. At the national level, BMKG provides different climate forecasting systems, and then regional BMKG offices will select the best system that fits their (local) conditions. In BRIN, our experts revealed that statistical models were rarely used to make rainfall predictions on a seasonal scale. Instead, they used dynamic models based on global climate data that has been downscaled to finer resolution. Our academic experts stated that no single model applies to every place in climate forecasts. When one statistical model has good accuracy during calibration, it may have poor verification accuracy when applied in other places. Rainfall data for model development (calibration) has specific local properties that differ from other places. Therefore, the use of ensemble models from many approaches/techniques are supported. EQ1.4: What are assumptions used in the model? The assumption used by the FFT, which has been applied to the PCH/online tool, is that the rainfall prediction will follow the pattern of observed historical data for calibration. FFT applies a sinusoidal approach, which results in a similar pattern prediction at any time. FFT is initially applied to process signal data, and later it has been applied for nowcasting1 of rainfall and numerical weather prediction (e.g., Nerini et al., 2017). In Indonesia, Aldrian and Jamil (2008) used FFT as a supporting method to determine the frequency distributions of the dominated spatial pattern. Based on the interviews, our key informants revealed that the output FFT tends to repeat patterns and is not sensitive to climate anomalies/extremes, especially on the inter-annual scale. Research in West Java, Indonesia, showed that 5-year rainfall prediction based on FFT resulted in significant bias (Susilokarti et al., 2015). Our experts from BRIN said that FFT might perform well in prediction when the data has a strong periodicity. However, for West and East Nusa Tenggara, local sea breeze and ocean-land interaction are the main driving factors influencing rainfall. Therefore, ignoring the drivers and solely focusing on FFT might mislead the prediction. Also, combined with the assimilation of rainfall data will improve the prediction accuracy of the FFT model. Our academic experts then added concern about the FFT, especially its ability to detect climate anomalies when El Niño/La Niña strikes. EQ1.5: What is the area covered by the forecast tool? Is it only for specific sites (districts)? What is the resolution? PCH was developed for five districts, four in West Nusa Tenggara and one in East Nusa Tenggara. In West Nusa Tenggara, the districts are West Lombok, Central Lombok, East Lombok, and Dompu, whereas Nagekeo represents East Nusa Tenggara. In 2019, PCH was upgraded into a web-based forecasting tool (online application) for three districts: Central Lombok and Dompu in West Nusa Tenggara and Nagekeo in Nagekeo. We observed that PCH provides a spatial rainfall prediction map with a resolution fitted to the village level. Based on model documentation from the tool developer, the spatial map was a product of interpolated technique using the Kriging method with the support of CHIRPS satellite data. The rainfall stations for the interpolation process varied among districts from 6 to16 stations (Table 1 Up to between two and six hours. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 14 2). PCH’s tool developer integrated CHIRPS data for identifying rainfall pattern in the respective districts in the process of interpolation. EQ1.6: What is the period of data used for developing the model, as well as calibration/validation and verification of the model? And the time period used for updating the forecast? • One year of data used for verification overlaps with data used in developing the model (2016 in the first forecast and 2018 in the second forecast). • The forecast accuracy in the overlapping year is normally very high for many statistics-based rainfall prediction models. • The verification should use independent data (only data not used in developing the model). With the inclusion of overlap year, the verification result suggests that the model performs well in all stations with an R-skill above 0.8. Figure 2. The period for model development was for 1979-2016 EQ1.7: What are the methods and criteria used to evaluate the accuracy of the forecast? The accuracy of the PCH tool was evaluated using the “match” criterion developed by the tool developer. The match was assessed on an annual basis. There are three classes for a match, namely: (i) very accurate, when the monthly rainfall prediction matches with the observed rainfall for the entire 12 months; (ii) quite accurate, when there are 6-11 months of predicted rainfall matched with the observed value; (iii) inaccurate, when the rainfall prediction matches the observed rainfall for 1-5 months. Based on the interviews, farmers and local partners in several beneficiary villages conducted a qualitative assessment of the accuracy of the forecast. The assessments were carried out in Babussalam and Sandik in West Lombok (Table 3). Generally, there were at least three villages for the qualitative assessment for each district. The farmers did match analysis on a 10-day basis by comparing the color of the rainfall map with the historical rainfall intensity (heavy, medium, and light rainfall) based on their memory of 2-10 years ago. 15 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Table 3. Villages for quality assessment of the accuracy of rainfall forecasts NO DISTRICT VILLAGE PERIOD OF VERIFICATION BASED ON FARMERS’ MEMORY 1 West Lombok Babussalam, Mareje, Sandik 2007-2016 2 Central Lombok Batu Jangkih, Pandan Indah, Pendem, Sepakek, Teratak 2015-2016 3 East Lombok Pene, Pringgabaya Utara, Masbagik Utara, Pohgading Timur, Sapait 2015-2016 4 Dompu Keramabura, Nangamiro, Nusajaya, Sampe, Songgajah 2015-2016 5 Nagekeo Aeramo, Bidoa, Focolodorawe, Kelewae, Wajomara 2015-2016 The key climate experts are concerned with verifying rainfall prediction by a qualitative assessment. This assessment is an unconvincing method. More robust techniques are expected to get convincing results. Farmer involvement in measuring daily rainfall at the farmer group level (or village level) will expand the coverage of rainfall observation that benefit to the verification and model improvement. At the spatial scale, there are no documents on PCH evaluation. However, the tool developer evaluated PCH at a spatial scale, but the information was not provided to the evaluation team. The tool developer also used statistical indicators to assess the model’s accuracy. Two well-known indicators were applied, namely the coefficient of determination (R2) and RMSE. EQ1.8: What criteria are used to define that the forecast is accurate? Based on the three statistical indicators, we developed the methodology to define rainfall forecast accuracy with the PCH/online system. The indicators assess the pattern, bias, and error of the prediction. For the pattern, we used the coefficient of determination (R2); for bias assessment, we applied percent bias (PBIAS); and for error of the prediction, we applied root-mean-square error￾standard deviation ratio (RSR). Table 4 provides the formula of each indicator and the criteria for good accuracy. Criteria for accuracy follow Moriasi et al. (2015). Table 4. Criteria for a good accuracy based on three statistical indicators INDICATORS FORMULA CRITERIA FOR GOOD ACCURACY DATA USED FOR Coefficient determination (R2) R2 > 0.5 16 stations in Central Lombok, 13 stations in West Lombok, 20 stations in East Lombok, and 8 stations in Dompu. Observed satellite CHIRPS rainfall data (corrected) in the three districts (Dompu, Nagekeo, and Central Lombok) Percent of bias (PBIAS) PBIAS between -15% and +15% Root-mean squared error-standard deviation ratio (RSR) RSR < 0.7 Note: 𝑂𝑂 and 𝑃𝑃 are observed and predicted rainfall, respectively. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 16 We used decadal rainfall data for 2018-2022 from the BMKG to assess the accuracy of the model prediction at the station level. The assessment at the station level was for four districts in West Nusa Tenggara (West Lombok, Central Lombok, East Lombok, and Dompu). We used decadal￾corrected CHIRPS satellite data for the online tool from 2020-2022. Three districts with the online tool are Dompu and Central Lombok (West Nusa Tenggara) and Nagekeo (East Nusa Tenggara). The prediction accuracy is good if all three indicators are used to meet the criteria presented in Table 3. Then, for each district, we calculated the proportion of stations/villages with good accuracy. Here, the accuracy refers to the number of stations/villages with good accuracy divided by the number of total stations/villages in the respective districts. EQ1.9: Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? Is the accuracy of the forecast conducted on a point base or spatial base? From interviews, most of our key informants said they did not check the accuracy of rainfall prediction (Table 5). All local World Neighbors partners, except PSP in West Lombok, conducted accuracy checking by a qualitative assessment in the field by observing a rainfall event and its intensity based on personal judgment (light, mild and heavy rainfall), although their methods varied among the partners. The ET observed some inconsistencies in the methods used to evaluate the accuracy of rainfall prediction. Most key informants in the field believed that PCH prediction was close to reality. Table 5. Summary of the accuracy checking of rainfall prediction by key informants DISTRICT INSTITUTIONS ACCURACY CHECKING NOTES BMKG NTB Accuracy for Central Lombok on average was 57% PCH accuracy with observation data for the last 6 months Central Lombok FPRB dan BPBD No accuracy checking was conducted PCH is close to actual rainfall Agriculture Office No accuracy checking was conducted PCH is accurate because it uses satellite data Local Partner - Berugak Dese Evaluation is done for one full year, in 2018 Asking farmers every month on the 10th, 21st, and at the end of the month, 3 times a month for a full year, but the data doesn’t exist because the laptop was broken Semayan, non￾foster Farmers believe PCH is accurate Dompu BPBD Accuracy checking was conducted, but the data is with the staff and KI has moved to DLH Accuracy similar to BMKG Agriculture Office No accuracy checking was conducted, but listened to qualitative views from farmers that PCH is correct Field information conveys that the estimated accuracy is relatively accurate Local Partner - LESPEL Accuracy checking was conducted, went straight to the field, every 10 days in 2018. For example, on Sunday there was rain, it was noted and check the form The dasarian data is less accurate than monthly prediction Pekat Accuracy checking was conducted by the farmer, but the document was lost PCH is actually not far from the old pattern Nowa, non-foster No accuracy checking was conducted PCH is not always accurate; sometimes it is right (hit), and sometimes it is wrong (false) 17 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV DISTRICT INSTITUTIONS ACCURACY CHECKING NOTES Mekar Sari Accuracy checking was conducted qualitatively I think it’s 90% accurate because the predictions for different villages will have rainy weather conditions, the results of qualitative discussions Nagekeo BPBD No accuracy checking was conducted PCH is used for comparison only Agriculture Office No accuracy checking was conducted Most PCHs are accurate but no evaluation Local Partner - YMTM Accuracy checking was conducted; one month and three bases. We usually check for 2022, now we check if 2020-2021 is correct or not In the discussion, the farmer who applies it he will testify West Lombok FPRB dan BPBD No accuracy checking was conducted Local Partner - PSP No accuracy checking was conducted Giri Sasak No accuracy checking was conducted PCH is not always accurate, sometimes is right (hit), sometimes is wrong (false) Cendi Manik Farmers have rain records from 2020 PCH as expected; farmers believe PCH is accurate because of socialization by PCH developer who is a professor from a well-known university East Lombok FPRB dan BPBD No accuracy checking was conducted, but did a comparison Agriculture Office PCH is not always accurate; sometimes it is right (hit) and sometimes it is wrong (false) Local Partner - LPSDM Accuracy checking was conducted using form Gunung Malang PCH is actually not far from the old pattern/PCH is not always right Puncak Jeringo PCH is way off the real condition Padak Guar Accuracy checking was conducted quantitatively based on discussion with farmers PCH is not always accurate; sometimes it is right (hit), and sometimes it is wrong (false) • Based on station level The number of observed rainfall stations varies among districts. More stations are found in East Lombok (20 stations), whereas Dompu has the lowest number of stations (eight stations). Based on their distribution, most stations are concentrated in the middle area, as shown in West Lombok, Central Lombok, and East Lombok (Figure 3). Stations partially cover the northern part of West and Central Lombok. In East Lombok, no monitoring station is located in the southern part. For Dompu, the western part is barely covered by monitoring stations. In the stations, we did an accuracy check for each, then computed the proportion accuracy for the whole district, which showed good accuracy. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 18 Figure 3. Distribution of observed rainfall stations (red dots) in West Lombok, Central Lombok, East Lombok, and Dompu Note: The stations are for calculation the accuracy of PCH/online system The accuracy of the PCH/online tool varied among stations. Based on the R2 value, the tool only has a good accuracy of up to 25 percent of all stations in the first year (2018), as shown in Dompu. In later years, the accuracy was below 25 percent, and even in West Lombok, the accuracy was ultimately 0 percent for all years (2018-2022). Based on PBIAS, the accuracy of the model was inconsistent over time. In Central Lombok and Dompu, in the first year, the proportion of PBIAS value with good accuracy was 0, which means the predicted rainfall at all stations has a significant bias. On the other hand, the proportion was up to 39 percent in West Lombok. The best accuracy based on PBIAS was in the fourth year at West Lombok (85 percent of stations). The best accuracy at the fourth year is consistent with other districts in East Lombok (45 percent) and Dompu (25 percent). Based on the RSR value, the accuracy of the PCH/online was very poor, below 15 percent (Table 6). In Central Lombok and West Lombok, the accuracy was zero for all of the years (2018-2022), which means that the prediction error is enormous. 19 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Table 6. Accuracy of PCH based on observed rainfall at station level using the three statistical indicators DISTRICT R2 (> 0.5) PBIAS (+-15%) RSR (<0.7) MEET ALL CRITERIA Central Lombok (16 stations) 2018 0 0 0 0 2019 0 19% 0 0 2020 0 6% 0 0 2021 19% 13% 0 0 2022 0 0 0 0 West Lombok (13 stations) 2018 0 39% 0 0 2019 0 8% 0 0 2020 0 0 0 0 2021 0 85% 0 0 2022 0 54% 0 0 East Lombok (20 stations) 2018 0 25% 0 0 2019 0 35% 0 0 2020 0 5% 0 0 2021 5% 45% 5% 0 2022 10% 10% 5% 0 Dompu (8 stations) 2018 25% 0 0 0 2019 0 0 0 0 2020 0 13% 0 0 2021 13% 25% 13% 0 2022 0 25% 0 0 Note: The italic indicates the forecast data were extracted from the PCH application tool, while the non-bold ones were read manually from the PCH maps. • Based on CHIRPS data The accuracy of the online tool was improved when using CHIRPS satellite rainfall data. In Dompu, based on the R2 value, the first-year prediction was accurate (100 percent) at all points, then it declined to 91 percent and 27 percent for the second and third years. When using PBIAS and RSR, the accuracy was up to 60 percent and 85 percent (first year), then dropped in the second and third years. Therefore, the overall accuracy in the first year is the best, reaching 60 percent, then plummeting below 10 percent in the following years of prediction. For Central Lombok, the accuracy was good based on R2 and RSR values in the first year, but then it was completely inaccurate for the following year. For PBIAS, the accuracy was low. Therefore, overall accuracy was poor as there was no single year that met all criteria used. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 20 Lastly, in Nagekeo, the online tool was more accurate for the first year, as indicated by the three statistical indicators. Then the accuracy dropped in the next year. However, the overall accuracy was relatively low (Table 7). Locations where the accuracy prediction was good were not consistent annually. As an illustration, the distribution of the overall accuracy prediction in Nagekeo is presented in Figure 4. Table 7. Accuracy of online tool based on CHIRPS satellite rainfall data using the three statistical indicators DISTRICT R2 (> 0.5) PBIAS (<15%) RSR (<0.7) MEET ALL CRITERIA Dompu (81 points) 2020 100% 60% 85% 60% 2021 91% 2% 58% 2% 2022 27% 19% 14% 6% Central Lombok (140 points) 2021 100% 0 88% 0% 2022 0 27% 0 0% Nagekeo (100 point) 2020 55% 44% 31% 25% 2021 31% 2% 4% 2% 2022 0 85% 0 0% Figure 4. Distribution of accuracy of rainfall prediction using CHIRPS satellite rainfall data in Nagekeo for 2020-2022 Note: Good accuracy means that all indicators at that point meet all criteria in Table 3 In addition, our expert from Regional BMKG in Mataram investigated the online tool performance in Central Lombok. He used the observed rainfall from 16 stations for January-June 2022 as a reference and compared it with outputs of the BMKG and online tool forecast. The expert applied categorical rainfall, which scaled decadal rainfall into 1 to 9 (representing 0-20 mm, 21-50 mm, 51-100 mm, 101-150 mm, 151-200 mm, 201-300 mm, 301-400 mm, 401-500 mm, >500 mm). His findings revealed that for the online forecast, only 57 percent matched the observed rainfall, which was 25 percent smaller than the BMKG forecast. In February and April 2022, the output of the online forecast was inaccurate as it matched 0 percent and 19 percent, respectively, to the observed rainfall. Further, using a similar categorical approach, we found that the accuracy of rainfall prediction (2018-2022) varied among districts. In West Lombok and Dompu, the accuracy tended to increase with time. For instance, in 2018, the first-year accuracy was 17.5 percent (West Lombok) and almost doubled in the fifth year of prediction. For Central Lombok, the accuracy 21 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV was steadily around 33 percent except in the third year. The best accuracy was 46.5 percent in East Lombok for the fifth-year prediction (Table 8). The findings contradict the output stated earlier by tool modelers, which revealed that the first-year accuracy is the best. Table 8. The accuracy of PCH forecast (in % of total stations) for 2018-2022 in four districts YEAR WEST LOMBOK (13 STATIONS) CENTRAL LOMBOK (16 STATIONS) EAST LOMBOK (20 STATIONS) DOMPU (8 STATIONS) 2018 17.5 33.0 31.4 18.4 2019 14.3 33.2 36.8 10.4 2020 21.4 22.4 24.3 19.8 2021 29.0 33.2 36.8 22.6 2022* 34.2 32.5 46.5 29.2 Note: The accuracy of PCH in four districts is calculated as proportion of the stations that has good accuracy. * only up to September 2022. Then, we compared the output of the PCH forecast with the observed CHIRPS data. For illustration, we analyzed data from October 2018 to January 2019 for West Lombok (Figure 5). This period is critical as the rainy season starts and when false rain events may strike. In October 2018, PCH forecasted the beginning of the rainy season with rainfall amount ~50 mm per 10-day period. In contrast, CHIRPS reported that adequate rainfall only occurred in the first 10-day period (D1-10-18), whereas rainfall was barely observed in the next two 10-day periods. In December 2018, both PCH and CHIRPS forecasts were comparable, but in January 2019 PCH forecast was overestimated. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 22 Figure 5. Spatial comparison between PCH forecast and corrected CHIRPS satellite rainfall data Note: D in the panel means ‘dasarian’ or decadal rainfall; D1-10-18 represents the 1st decadal in October 2018 In addition to the accuracy, the online tool has a problem with consistency in displaying rainfall forecasts. There is inconsistent data between the map and graph, which display different rainfall amounts in the same period. For example, in Towak Village (Nagekeo), on the first 10-day period in December 2023, the graph says that the rainfall forecast is 110 mm, but the color of the map shows a rainfall amount of >200 mm (see Figure 6). A similar inconsistency between map and graph is also found in other villages. The tool developers responded that there was an error with the online system and clarified that the amount of rainfall shown in the graph was the correct prediction. This might show that the online system is not ready for public dissemination. 23 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Figure 6. Inconsistency of displaying data between rainfall data in map (color) and graph in Towak Village, Nagekeo EQ1.10: Are there other forecast models in use by BKMG that could be modified to produce similar forecasts? And if so, what is restricting their adoption? In BMKG, many models have been applied, either statistical models or dynamic models. For statistical models, ARIMA, Welt Fed, and ANFIS are the typical models widely used for forecasting. For the dynamic model, BMKG utilizes the ECMWF forecast and INA MMA. BMKG integrates both models as ensemble models for rainfall forecast at decadal (dasarian), monthly, seasonal forecast (three months), and climate outlook (one year). The forecast is available at the resolution of the sub￾district level. For the statistical model, each BMKG region may select the best model that fits with the local conditions. Forecast of decadal rainfall is provided up to three months ahead and updated every month. The update provides revised forecasts for two months and an additional one month of new forecast. Extending the forecast to a longer lead time will have low accuracy. In addition to climate forecasts, BMKG also provides weather information. It provides information on the early warning for extreme weather, nowcasting weather (one-hour forecast), and weather forecasting (three hours and 1-7 days). The resolution for weather information is 3 km (village). Information on weather and climate is available online at https://www.bmkg.go.id. BMKG also has developed its app to disseminate the weather and climate forecast, which is available for Android: “Info BMKG.” Although BMKG has provided online information, we found from an interview that the information provided online is not attractive. Users do not consider it user-friendly, and the resolution of information does not meet the needs of farmers/related agencies (sub-district level). BMKG understand that the users (farmers) need forecasts at more refined resolution (down to village level), and it is possible to apply interpolation technique to produce more refined spatial resolution of rainfall forecasts. BMKG is still doing research on applying interpolation techniques to increase the resolution of the rainfall forecast. Considering geography and topography have strong influences on USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 24 rainfall, producing very refined resolution of rainfall forecast will need detailed geographical and topographical characteristics. BMKG welcomes scientists from universities and research agencies to research weather and climate forecasts. Their findings on the methods and applications are categorized as experimental research. Adoption and integration of new forecast methods into national forecast systems is possible. A SOP for adopting the new forecast model is being developed as it is mandated by the regulation of Peraturan Pemerintah (Government Regulation) No.13/2018 on research and development of meteorology, climatology, and the geophysics industry. Discussion on the development of new methods/findings/applications is facilitated through the National Climate Expert Forum, which can be the forum to facilitate the adoption of new approaches. CONCLUSIONS 1. The PCH tool uses Fast Fourier Transform (FFT), a statistics-based model, which is not suitable to be used for long-term rainfall prediction (1-5 years) and it is categorized as the annual-to-decadal prediction, not the seasonal climate prediction. 2. The annual-to-decadal prediction needs a dynamic model that includes the air-land￾atmospheric dynamic interaction that controls rainfall. 3. The nature of the FFT enables the tool to mimic the pattern of observed data up to 3-6 months ahead but not able to simulate extreme events. 4. Climate producers commonly use ensemble models that combine dynamic and statistical models to produce seasonal climate prediction as they can simulate intensive and complex interaction of land-sea and atmosphere and thus predict extreme events. 5. The PCH tool performed quite well at the first-year prediction in some locations, but very poor in the next year’s prediction. 6. The statistical-based forecast models need updates regularly within a short period (at least every six months). 7. The PCH tool updated the forecast every two years, which is the tool’s fatal weakness as weather and climate are very dynamic and depend on the intensive and complex interactions of land, sea, and atmosphere. 8. The project has difficulties assessing observed decadal rainfall data, which limits the model developer to updating rainfall forecast regularly within the short period. 9. BMKG is the sole agency that can disseminate weather and climate information and already has the required infrastructure and human resources to support the forecast operation in all regions. 10. Introducing and integrating a new forecast model to BMKG’s regional/national rainfall forecast system is crucial to ensure the sustainability of the service. 11. National Climate Expert Forum is an effective forum in facilitating the BMKG’s process of adopting the new forecast model. RECOMMENDATIONS 1. Future project activities should have strong collaboration with BMKG to avoid any potential hindrances and to ensure the system’s adoption and use meets the public information standards. 2. Future project activities should facilitate BMKG to develop an SOP for adopting a new rainfall forecast model to improve the national seasonal climate prediction system (mandated by Government Regulation Number 13/2018) and enrich the ensemble models. 25 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV 3. The tool developer should integrate forecast skills in the tool (accuracy of the forecast spatially) to allow users to benefit from using forecasts. 4. The tool developer should train WN and local partners to identify potential use of rainfall forecast, taking into consideration the skill of forecast. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 26 UTILITY AND INFORMATION DISSEMINATION OF RAINFALL PREDICTION EQ2: WHAT ARE THE UTILITIES OF THE RAINFALL PREDICTION MODEL/TOOL FOR FARMERS? FINDINGS EQ2.1: How does the climate affect farmers’ cropping systems? The cropping systems in the five districts are varied. Mostly in Lombok, the farmers have monoculture land while the others have an intercropping system with perennial crops like cashew, srikaya, and coconut (Figure 7). The number of planting seasons is also different, ranging from once a year to twice and three times a year. Most types of crops planted by the farmers are paddy and corn and sometimes mixed with vegetables, nuts, cassava, and sweet potatoes. Figure 7. General cropping system of agriculture in selected villages Figure 8. The percentage of climate impacts and climate-related disaster in selected villages 27 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV The first planting seasons are between October and January. Most of the farmers experience false rain. In the case of false rain, farmers commonly have planted the crops, but the crops will be exposed to prolonged dry spells, causing failure to grow, which requires the farmers to replant the crops. This climate impact also causes low yield. Another common climate issue is the season break. In this case, the crop may be exposed to drought stress, and yield may decrease. Some farmers exposed to season break irrigate the crops using pumping, but this relies on the availability of technology (dams, wells), cost, and the sources of water (groundwater, rivers, and ditch). Some farmers apply organic fertilizers (conservation agriculture) to store water during the season break and reduce water stress. Most of the villages, as seen in Figure 8, have issues with drought. Meanwhile, one village, Taman Baru, experiences annual floods, which also cause damage to the crops. Strong winds sometimes also become a problem in the surveyed villages. However, the impact is minor compared to drought and floods. Table 9. Cropping system and climate issues on agricultural land in surveyed villages DISTRICT VILLAGE CROPPING SYSTEM PLANTING SEASON CLIMATE ISSUES OTHER ISSUES Central Lombok Tiwugalih rice-rice-upland crops (corn, soybeans, green beans) The first is in December, January, the second planting is in March, April Semayan, non-foster rice, rice, upland crops. There are corn, soybeans, green beans, tobacco, peanuts, depending on the soil around January seasons change; drought (for rain- fed land) Dompu Pekat corn (intercropped with cashew); corns; corn￾peanut October-November 1 month rain break rat pest, fertilizer availability Nowa, non￾foster paddy-paddy-paddy November￾December 1-2 months rain break, less rain irrigation was being repaired; rat pest Mekar Sari Planting season 1 (MT1) paddy or corn; MT2 corn, tobacco, or beans; MT3 tobacco for those who have a well December (paddy), November (corn) false rain, drought limited irrigation, mostly rain-fed; pig pests; quarrel over the use of well water Nagekeo Wajomara corn-organic vegetables: spinach, green spinach, red spinach, kale, beans, eggplant, and shallots November￾December (corn) drought caterpillar pest intercropping (cashew with the most corn, beans, cassava) November￾December (corn); January (nuts) strong winds soil humus is not good Pagomogo intercropping corn, cassava, paddy October (corn); November (paddy) fertilizer paddies; intercropping of corn, cassava, peanut, paddy, with coconut, clove, cashew November￾December fertilizer stock Nangaroro, non-foster corn or paddy, mixed with cassava, peanuts (once a year) January (paddy), December (corn) false rain caterpillar West Lombok Giri Sasak paddy; paddy; paddy, sometimes add vegetables January (paddy) drought, false rain the cost of taking water is expensive; fertilizer cost Taman Baru paddy-corn-palawija (corn, beans) September, October flood every year dam is being repaired; rats, birds, pig pest Cendi Manik MT 1 paddy, corn (rare); MT2 beans, corn, chili, watermelon, cucumber, tobacco (new); some land is mixed with cashew in the highlands October drought, false rain, flood USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 28 DISTRICT VILLAGE CROPPING SYSTEM PLANTING SEASON CLIMATE ISSUES OTHER ISSUES East Lombok Gunung Malang Intercrop of srikaya, cashew, tree, with corn November￾December, January at the latest drought, false rain, rains later than the surrounding area rats (worst), caterpillars, pig pest Puncak Jeringo paddy-corn mixed with tobacco December false rain; drought rat (worst) pest, pig pest, fewer fertilizer subsidies Padak Guar paddy-corn-corn, mixed with green beans December-January drought In the second planting season, farmers experience changes in the ending of the wet season (Table 9). When the wet season ends earlier, and farmers have already planted the second crop soon after the first harvest, the second crop commonly gives a low yield to harvest. If the rainfall in the dry season is more than average (wet dry season), some crops will grow better and give a high yield (upland crops-corn), but some crops could fail (such as crops that need less water, like tobacco). In addition, certain climate conditions may cause an outbreak of certain pest and diseases, such as caterpillars (season transition). Besides the climate, the agricultural activities in the villages are also affected by other factors. The cost of taking water is considered extremely high for the farmers. In addition to this, only a few farmers have the opportunity to take additional water sources due to the distance between the lands and the water sources. Pests are the natural enemy faced by the crops. The farmers have difficulty wiping out rats, in particular. The existence of pests becomes a threat that lowers the yields. EQ2.2: How are the results of the forecast turned into advice to farmers? What is the advice? Generally, according to all farmers in the selected villages, the rainfall forecast information can assist in determining planting times (Table 10). The forecast provides information on the amount of rainfall and a clear guideline for planting. Some farmers said the forecast information could help decide on the second crop. For example, in some villages in Dompu, some farmers only plant corn once a year. They were advised by the local partner (LESPEL) to grow the second crop after corn, for example, tobacco. In Pekat village, some farmers tried to plant the second season of corn after the first planting. Although the second planting season is not every year, the rainfall forecast information helps the farmers determine if they can grow crops in the second planting season based on the amount of rainfall. Some farmers said that they utilize rainfall forecast information for water management. On the other hand, none of the farmers consider using forecast information to determine the pesticide application. The forecasted rainfall information is also not used to determine the fertilization time. This strongly depends on the availability of fertilizers, which are provided and distributed by the PPL from the Regional Agriculture Office (Distan). Some farmers, like horticultural farmers and farmers with irrigated land, are not concerned with rainfall forecasts as they normally take water from other water sources, such as springs (taps in small ponds) and wells. The facilitators (and PPL) play a significant role in facilitating and improving the understanding of farmers in using climate forecast information for planting decisions. PPLs who participate in SLI facilitated by the BMKG, which is also involved in the project, have good knowledge of using climate information. They also formed WhatApps Group of SLI Alumni that facilitates communication and knowledge exchange among PPL and BMKG. 29 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Table 10. The impact on the use of climate information in planting decisions in surveyed villages DISTRICT VILLAGE CROPPING PATTERN IMPACT ON THE USE OF CLIMATE INFORMATION IN PLANTING DECISION Central Lombok Tiwugalih Rice-rice-upland crops Information planting time from PPL Semayan, non￾foster Rice-rice-upland crops Determine the planting season Dompu Pekat corn-peanut (intercropped by cashew) Determine the planting season; add planting season Nowa, non￾foster Rice-rice-rice Determine planting time; manage water supply Mekar Sari Rice/corn - tobacco Determine the planting season Nagekeo Wajomara Intercropping corn/organic vegetables Determine the planting season Wajomara 2 corn/beans/cassava (intercropping cashew) Determine the planting season Pagomogo Intercropping corn/cassava/Rice Determine the planting season Pagomogo 2 Intercropping corn/peanut/casava under coconut, clove, cashew Determine the planting season Nangaroro, non-foster Mix: corn/paddy/cassava/peanuts (once a year) Determine the planting season West Lombok Giri Sasak Rice-rice/upland crops-vegetables Determine the planting season; determine the commodity (palawija or corn or beans) Taman Baru Rice-corn-nuts Determine the planting season Cendi Manik Rice/corn-vegetables and fruits (chili, watermelon, cucumber) Determine the planting season East Lombok Gunung Malang Intercropping corn under Srikaya, Cashew Determine the planting season Puncak Jeringo Paddy/corn mixed with tobacco. In dryland mostly corn Determine the planting season and determine commodity EQ2.3: What percentage of farmers are using the tool? If it is low, why? Do they view it as a useful tool (provide examples), and why? Most of the interviewed farmers are aware of the rainfall prediction information (PCH) (Figure 9). Only a few farmers stated that they had never heard of PCH, particularly farmers from non-foster villages. In some villages, the head of farmers’ groups attended the socialization or training in using PCH. Some of the farmers said that they already forgot about PCH. Only a few farmers can read PCH maps. Some farmers never use PCH as a reference for their planting activities. Some farmers use it with a combination of observing the actual rainfall and traditional wisdom (called traditional forecast). Interviewed farmers stated the following reasons for not using PCH: • Lack of understanding of the use of the PCH • Lack of confidence in the accuracy of PCH • Not having PCH maps • Having no water problem (irrigated lands) • The reliance on traditional forecast or local planting season • Avoiding different planting times with other farmers to avoid pests’ attacks • Forgetting how to use PCH • Most of the farmers that use PCH in combination with traditional forecast (TF) tend to follow the traditional, and some argued that PCH is not different from the TF • Most farmers still use natural signals and rainfall events following the signals in planting decisions rather than the forecast USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 30 Figure 9. The awareness and the application of PCH by farmers in surveyed villages EQ2.4: How is the advice used by the farmers for making planting decisions or other planting operations? Some farmers who use PCH have stated that the planting season based on PCH is similar to their planting habits. When the time of planting advice by PCH does not match with their local wisdom, the farmers will observe the actual rainfall. Most farmers will start to prepare their land after waiting for 2-3 days of heavy rainfall in one week (Table 11). However, the planting date also relies on other factors, such as labor and pest attacks. Due to a shortage of laborers, in some villages, farmers plant sequentially as they help each other. In other villages, the farmers tend to plant simultaneously to avoid the spread of pests. Most farmers do not follow PCH because they still rely on local wisdom. There are a number of local wisdoms in determining planting time. The most common one is papan or urige. This is a traditional or cultural planting calendar used by farmers to determine the month of planting. The calculation is conducted by senior residents and followed by other farmers. In one village, Pagomogo, some farmers determine the planting date after cultural leaders conduct rituals. Besides the local wisdom, farmers determine the planting time based on natural signals. Several signals indicate the start of wet seasons, including thunder, the sound of birds or insects, bush growth, and the emergence of sprouts of some plants. The appearance of these natural signals determines the time for land preparations. After waiting for rains to happen in several days (2-3 days), the farmers will start seedlings and/or planting. 31 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV In the case of crop type advice based on PCH, some farmers decide their crop type based on the predicted amount of rainfall. Some successful stories came from Dompu District, where some farmers can increase planting intensity from one to two in a year. However, some farmers hardly follow this advice because of habits. In Giri Sasak, some senior farmers cannot be convinced to change their crop type. Although planting corn in the second planting season produces more yield and benefits than rice, farmers cannot follow this advice because the rice harvested from the second season is used for daily needs. EQ2.5-2.6: For farmers who used the rainfall prediction advice, what difference did it make in the production? What is the percentage increase in food crop production and/or income? Before the KIIs it was stated in the WN report, A New Way to Read the Weather, that PCH increased corn production by 77 percent on average in five surveyed districts. However, the method used to determine this figure is absent. Therefore, the ET asked some clarification questions during the KII with the local partners. According to a local partner, LESPEL, the production assessment for the WN report was conducted on pilot sampling land. Adding more information, PSP, a local partner in West Lombok, shared that the production survey was conducted by each local partner in five districts by collecting commodity and productivity data from foster farmer groups. Table 11. Planting decisions by farmers in selected villages DISTRICT VILLAGE PLANTING DECISION Central Lombok Tiwugalih traditional: papan, month calculation (not Jan-Dec) based on sun movement, planting month 6 or 7 (hijriah); wait for 2-3 heavy rains; planting time at the same time/nearby between farmers Semayan, non￾foster traditional: urige; fallen leaves, begadung; see the previous year’s rain Dompu Pekat using PCH; traditional: thunder, grass grows, spray, wait for rain, plant; depending on the readiness of the land Nowa, non￾foster traditional: see tamarind tree flowering; planting simultaneously Mekar Sari traditional: urige (Balinese) planting first, waiting for the leaves to fall or the tangled wind, little hurricane, thunder; depending on the wetness of the soil; MT2 depends on rain; there is a group meeting to discuss land preparation Nagekeo Wajomara traditional: thunder, sound of birds; wait for 2-3 times of heavy rains Pagomogo saw rain 2-3 times Pagomogo 2 traditional: the ritual of the beginning of the rainy season, calculating the moon; wait for 2-3 times of heavy rains Nangaroro, non-foster saw 2-3 times of heavy rains West Lombok Giri Sasak see rain at least 2 times until it can irrigate paddy fields (flat land); Taman Baru traditional: sukulan, planting before the ants rise, hearing the sound of animals (insects, ketended/cicadas); waiting for rain; planting together at the same time because of pests Cendi Manik traditional: urige; waiting for the rain, the soil gets wet and the grass grows; consider PCH East Lombok Gunung Malang using PCH; traditional: see srikaya tree buds, lots of clouds in the west; wait for 2-3 times of heavy rains; after no PCH back to habit Puncak Jeringo traditional: banten tree leaves fall and sprouts appear, shoots of galung tree appears; now waiting for the nonstop rains for a week; the calendar is sometimes used as a reference, planting season together at same time, feel afraid if planting is too late Padak Guar traditional: the tops of the kapok trees or the banten tree appear, the groundwater has risen and starts to clean up; waiting for rain 3 times USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 32 A local partner in Nagekeo, YMTM, shared their survey data in 2020 and 2021 on the productivity of farmers who followed PCH advice and who did not follow PCH advice. According to this data as shown in Table 12, in 2020-2021, the foster farmers who followed PCH had production of paddy, corn, and cassava, around 67 percent, 100 percent, and 64 percent more than the production of foster farmers who did not follow PCH, respectively. Meanwhile, non-foster farmers that followed PCH had production of paddy, corn, and cassava, around 44 percent, 59 percent, and 48 percent more than the production of non-foster farmers that did not follow PCH, respectively. The farmers that follow the PCH planted the crops in the period of December11-20, 2020. The farmers who did not follow the PCH advice plant their crops differently, including at the end of ten days of November 2020, the last ten days of December 2020, or the first ten days of January 2021. According to the interviews with local partners, there is no standard method being used by local partners for evaluating the impact of using PCH on the increase in yield (no field measurement but based on interviews with farmers that follow and do not follow PCH). Furthermore, local partners have no clear explanation on how to differentiate the impact of other interventions after adopting PCH (e.g., change in inputs, crop management, etc.). Besides the local partners, no other key informants conduct production evaluations. Most farmers claimed that the use of PCH has an impact on reducing the risk of crop failure. Some farmers in Dompu reported an increase in production (Table 12). However, some farmers in Central Lombok stated that they had insignificantly increased yield. In contrast, farmers in Desa Gunung Malang, East Lombok, stated that using PCH has no impact on production. The farmers shared their opinion that production is affected not only by the amount of rainfall but also by other key factors such as fertilizers, extreme weather (drought, floods), pests and diseases, and the quality of seeds. CONCLUSIONS 1. False rain, seasonal breaks, and rainy seasons ending earlier are the main factors causing planting and harvesting failure. 2. Adopting climate forecast information can help farmers reduce expenses (e.g., avoiding planting failure), increase yield, and increase planting intensity with appropriate crop types. 3. Helping users understand the meaning of the forecast accuracy (skill of forecast) and how to use forecast information in making decisions is critical to ensuring the forecast’s maximum benefit. Table 12. PCH application impacts on production DISTRICT INCREASE IN YIELD AFTER ADOPTING PCH BASED ON INTERVIEW WN Local Partners Facilitator and Farmers Lombok Tengah Increased, but not significantly/PCH reduced loss due to fail planting Dompu corn: 167% Facilitator: 20%-30%; Farmers: 75%-167%. Can plan crop twice in a year Nagekeo Foster village: Rice: 67%, corn: 100%, Cassava: 64% Facilitator: Depend son diseases, fertilizer, and seeds. Yield increase depends on Non-Foster village: Rice: 44%, corn: 59%, fertilizer Cassava: 48% Lombok Barat corn: 133% Lombok Timur N/A Farmers: Drought can reduce yield up to 60% 33 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV 4. Most farmers interviewed still used natural signals and rainfall occurrences that follow the natural signal in making planting decisions, and they used forecast from PCH as consideration in making the decision. 5. Surveys by local partners on sample farmers and self-reports from interviewed farmers that adopted the PCH report increased yields. However, no standard method is used for evaluating the impact objectively. RECOMMENDATIONS 1. WN and local partners should support the District Agriculture Office to institutionalize the use of climate forecast for cropping. The socialization of climate forecasting and assistance to tailor cropping to the forecast should be a routine activity that does not depend on the project. 2. Future USAID projects should focus more on institutionalizing field climate schools (e.g., SLI) that have similar aims and information to this project at the village level as part of a local government program (Agriculture Office). The collaboration will provide one source of information, which may generate better results. 3. The Agriculture Office (with assistance from donors/projects) routinely holds ToTs for field facilitators (extension workers), which is in line with the BMKG program on SLI. WN and local partners should develop synergy with the SLI. By doing so, the facilitators will get acquainted with the standard operation procedures for reading and understanding climate information. 4. WN and local partners should develop a standard method to evaluate the benefit of using climate forecast information. EQ3: BY COMPARING THE TRADITIONAL SYSTEM AND ONLINE APPLICATION AS A MEANS OF INFORMATION DISSEMINATION, WHICH SYSTEM IS MORE EFFECTIVE FOR FARMERS? AND WHY? FINDINGS EQ3.1: What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? The traditional system of information dissemination was to provide paper PCH maps, which are a set of online pictures. The first system to disseminate PCH information to farmers was by printing PCH maps distributed to farmers and village offices. One of the local partners in Dompu, YMYM, also provides table matrices that provide more understandable information for farmers. Based on the interview with the local partner, PSP, the project emphasized 25 fostered villages (desa binaan). The local partner manages about five villages, and the remaining 20 villages collaborate with PPL from Distan. In Nagekeo, for example, PPL was trained in BPP once at the start of the project. PCH maps are distributed annually by local partners in every district. Socialization with village officers and farmers was conducted, represented mainly by the head of farmer groups. Heads of farmer groups play a prominent role in providing cropping pattern/planting guidance to members since most members follow the head of the farmer group’s cropping pattern. Therefore, not all farmers were exposed to the PCH information directly. Based on interviews in Table 13 below, in USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 34 some villages, socialization was conducted every year before the planting season started. In some villages, socialization was conducted once or a few times (Table 13). From the interviews, we can see that many farmers cannot read the PCH maps. Some have forgotten about the PCH, and some are illiterate and need more assistance to read the PCH. After getting support from the regional government, three of the five districts (Nagekeo, Central Lombok, Dompu) governments worked with the tool developer to build their websites or Android applications to disseminate PCH information. The website provides the same information as PCH maps, with additional information (such as daily weather prediction). This allows farmers to access information on their phones. However, two farmers in the interviewed farmer groups in Pekat, Taman Baru, and Pandak Guar stated that they had accessed the website. Some never use the website for reasons including having no smartphone, lack of internet connection, no internet data, and not being aware of PCH websites. However, according to the interviews with PPL from Distan, the website is significantly helpful in accessing the information. Table 13. Climate information dissemination received by farmers in surveyed villages DISTRICT VILLAGE PCH DISSEMINATION DISSEMINATION PREFERENCES OTHER SOURCES OF CLIMATE INFORMATION Central Lombok Tiwugalih maps are shared PPL assistance. can’t use the app Semayan, non-foster training; there is a farmer school assistance; not many people use smartphones KATAM, BMKG Dompu Pekat socialization, community consultation frequent socialization; use maps instead of application because there is no signal; need IT expert BMKG Nowa, non￾foster socialization with the head of the farmer group many farmers can’t use hand phone (HP) BMKG bulletin Mekar Sari socialization/training of some farmers PPL assistance Nagekeo Wajomara socialization in 2021 Pagomogo never frequent socialization in the last 2 years frequent socialization Nangaroro, non-foster never West Lombok Giri Sasak socialization from PPL, maps are not distributed frequent assistance Taman Baru frequent socialization; use map because farmer can’t use app Cendi Manik socialization in 2018-2019 Socialization and maps; difficult signal, no internet access East Lombok Gunung Malang socialization once every year (head of farmer group mandatory, member occasionally), 2017-2021 more frequent assistance and maps provided to farmers; many can’t use a smartphone Puncak Jeringo socialization 2018, calendar 2020 Padak Guar socialization, map given 2018-2019 calendar provided every year 35 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV EQ3.2: What are the main sources of information that farmers use to guide decision￾making each season? What information do farmers use to guide decision￾making each season? How do farmers make decisions in each season from the provided information? Based on the interviews, most farmers still apply and use natural signals when making decisions. In some villages (Pekat, Dompu and Gunung Malang, Lombok Timur), they may combine natural signals with PCH information in decision-making. For instance, if PCH is predicted earlier or later than the regular planting season, the farmers generally observe the rainfall conditions at the beginning of the planting season to determine the time to prepare the lands and to plant the crops. Before understanding the PCH, most farmers depended on traditional planting calendars or natural signals. Only two farmers in Semayan Village, Central Lombok District, have heard about KATAM (planting calendar from the Ministry of Agriculture) from PPL. Some have heard of and accessed BMKG information but only to see information on earthquakes and daily weather forecasts. At the beginning of the wet season, farmers observed the natural conditions, such as thunder, the appearance of certain songbirds, the sound of animals (insects, like ketended/cicadas), the appearance of buds of srikaya/kapok, the falling of banten tree leaves, and the appearance of sprouts and shoots of the galung tree. These signals indicate that the wet season is coming, and most farmers start to prepare the land. Following the natural signals, and when rainfall occurs 2-3 times or days within a week or ten days, usually the wet season has started, and farmers start planting. The farmers tend to discuss with others (in farmer groups) when to start planting together. However, in some areas, every farmer makes decisions individually depending on the availability of inputs (seeds, fertilizers) and labor for planting. In contrast, others tend to plant simultaneously to avoid the spread of pests. EQ3.3: Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in-person gatherings and those using the online application? Based on the results in Figure 9, almost all farmers we interviewed are aware of PCH, but not all who are aware adopted the advice. The main reasons for not adopting the advice are because they believe the advice is not yet proven. In addition, the number of maps distributed to farmers is limited to one per village. This limited access contributes to farmers’ lack of understanding of the maps, so they tend to follow the farmer groups’ leaders. Few farmers, generally the younger ones, used websites or online information. Some of them also access other online climate information from BMKG. Most farmers cannot use technology like smartphones and the internet because of limited internet access or limited funds to purchase sufficient internet data. Most farmers cannot read PCH maps (printed or online) and stated that they still need regular assistance from PPL and the heads of farmer groups. In the end, it is easier for farmers to directly ask PPL. EQ3.4: Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places where there is no app support? The benefit of a PCH website or mobile application is real-time accessibility. However, most farmers prefer to have face-to-face PCH socialization by PPL. It is easier for farmers to ask others rather than access the online application. Only young farmers tend to have smartphones, most others do USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 36 not. In addition, the farmers are mostly illiterate and do not understand PCH. For these reasons, coupled with limited internet signals, the utility of mobile apps is very limited. However, the availability of online information and apps is helpful for agriculture offices and facilitators (PPLs), which can improve their ability to assist farmers. CONCLUSIONS 1. Frequent socialization with farmers is the key to success in delivering advice, as most farmers are illiterate. 2. Information dissemination through gathering and PPL assistance can generate better understanding of climate information and its utilization among farmers because of low understanding of technology-driven rainfall prediction. 3. Producing more rainfall forecast application relevant for cropping decision, e.g. maps on suitable planting time for crops and cropping pattern based on rainfall forecast etc, is very important in increasing farmers’ capacity to tailor cropping operations and management to rainfall forecasts. 4. Climate information is not the sole aspect in making planting decisions; the economy and social aspects such as cost, price, culture, and farmer collaboration also play vital roles. 5. The online version of PCH is not user-friendly or applicable for farmers, however it is useful for PPL. RECOMMENDATIONS 1. The project should continue to support the use of maps for dissemination and socialization of rainfall forecast. Future projects should prioritize such socialization at least once every planting season. 2. The current project should collaborate strongly with field PPL through intensive training on how to effectively provide advice to farmers. 3. WN and local partners should integrate the use of weather/climate information from regional BMKGs in the training of PPL, considering the PCH/apps are updated online once every two years, while BMKG provides updated forecasts on a monthly basis. 4. WN should collaborate with BMKG to share best practices and lessons learned from the project regarding the approach to communicating forecast information to farmers so BMKG can adopt and integrate the lessons into the Climate Field School program, particularly in producing rainfall forecast information maps relevant for cropping decisions. 37 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV EARLY WARNING SYSTEM CHECKLIST EQ4: IN WHAT WAYS HAS THE TOOL BEEN UTILIZED FOR DISASTER PREPAREDNESS? FINDINGS EQ4.1: What is the awareness level of stakeholders on the tool? The WN and the local partners socialized the project at the start of the project. This socialization involved broad stakeholders at the district level and village level and intense contact with agencies directly involved in the project, particularly the Agriculture Office, Local Disaster Office (BPBD), and DRR Forum at the district level, and village staff and KMPB (Community-Based Disaster Group) at the village level. The tool’s awareness level is relatively high for the stakeholders from these organizations but not in other organizations. In the Dompu District, an interview with a KI from the Irrigation Division of Public Work Office (Dinas PUPR) indicated that they did not know much about the project. In this district, an Irrigation Commission (Public Works or Bappeda chairs the commission) is responsible for defining the cropping pattern (including planting time and irrigation scheduling) for each season. Nevertheless, this committee works closely with BMKG and uses the BMKG rainfall forecasts to define the cropping pattern in the irrigated-rice system. In the fostered villages, the head and/or staff of village offices in the five districts and KMPB are aware of the tool, except for KMPB Nusa Jaya (Dompu) and KMPB Wajomara (Nagekeo). KIs from the KMPB of Wajomara noted that they have never seen the PCH maps, while the KMPB of Nusa Jaya has seen the PCH map printed and also on a laptop but did not use it. In addition, most KIs from village offices said that they received PCH maps and kept them, but some stated they did not have them anymore. EQ4.2: Do they use the tool in disaster prediction? And how it is used? What are the forms of disaster information provided based on the tool? The rainfall prediction tools in the web and application were developed in 2018-2019. These tools provide information on disaster predictions, namely flood, drought, landslide, and crop pests. The disaster information is provided in the form of prediction of disaster vulnerability (kerentanan) level for each dasarian for five years (until December 2024). The information is provided at the village level without providing a specific location in the village (Figure 10). USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 38 Figure 10. Disaster information provided by the tools From interviews with KIs at the villages, most villages do not use the tool to prepare for disasters, with the exception of some villages that are frequently exposed to flood disasters (flood-prone villages). About 55 percent of KMPBs interviewed in the surveyed villages stated that they do not use the tools for disaster preparedness (Figure 11). Figure 11. KMPB response on the use of disaster prediction from tool for disaster preparedness The translation of the disaster prediction tool into an impact outlook that can help the village assess the disaster’s potential impact and the impacted locations is not in place yet. However, disaster prediction has influenced the village government’s decisions, particularly on budget allocation. Predictions that put the village at high flood risk have facilitated the village government to allocate more funds for river normalization and cleaning the drainage system with the community’s involvement through community mutual cooperation (gotong royong). Some villages stated that gotong royong for village drainage cleaning was already routine even before the PCH tool was produced. In addition, the project also helps the village allocate funds for rehabilitating the catchment area of spring water as part of an effort to minimize drought risk. 39 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV EQ4.3: Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? From the interviews, all BPBDs use the early warning information from the BMKG.2 However, some of the BPBDs stated that they use early warning information from the tool as consideration (secondary information), not as primary information. It is also stated that the information used to indicate the potency of having hydro meteorological disasters is weather forecast (not seasonal/climate forecast). In all districts, most of the related stakeholders and institutions (DRR Forum members, including village heads) are connected in a Disaster WhatsApp group that regularly receives weather and climate information. In addition, another tool for disaster prediction in agriculture is KATAM (Dynamic Crop Calendar), developed by the Ministry of Agriculture (https://katam.litbang.pertanian.go.id). The KATAM provides recommendations for planting time, fertilizer, and variety in each season following climate forecast information issued by the BMKG. The KATAM also provides a forecast on potency for disasters in agriculture, which include flood, drought, and crop pest and disease attacks, namely tungro, blast, rice stemborer, brown planthopper, and rats. The information is at the sub￾district level (Kecamatan). All KIs from the District Agriculture Office know KATAM, but the utilization of KATAM varied immensely between districts. KIs from the Agriculture Offices of Nagekeo and East Lombok stated that they used information from KATAM more often than the tools. KIs from Central Lombok District mentioned that the tool is connected to KATAM, which helps in detailing the information from KATAM to the village level. Based on information from the tool developer, the tools adopt the criteria used in KATAM for assessing the climate conditions favorable for the crop pest and disease attacks, not connected to KATAM. However, the tool only provides information on predicting pest vulnerability levels without specifying the type of pest, as in KATAM. EQ4.4: How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? And by whom? Based on interviews with KIs from the village governments and KMPBs, none of them have translated the rainfall forecast/prediction to an impact outlook. The impact outlook provides an assessment of the potential consequence or potential impact of the extreme climate and the affected locations. The project has facilitated the villages in developing a map of disaster-prone areas (peta rawan bencana). However, none of the village staff understand how the rainfall forecast is being translated to the impact outlook (combining rainfall prediction/disaster information with the map of disaster-prone areas to indicate the specific area to be affected and develop anticipated actions). EQ4.5: How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, fourth? Why? 2 Information on the early warning of disaster (e.g., earthquakes, tsunami, floods, disease) used by BPBD is mainly from BMKG since the BMKG is the authorized agency that issues early warning information on disasters. An emergency alert is declared when there is a disaster potency based on early warning information and technical institutions’ recommendations (e.g., Agriculture Office, Health Office). The disaster will occur and threaten people’s livelihood and thus need immediate action. When the emergency state has been declared by the President at the national level, Governor at the provincial level, and the Regent/Mayor at district/city level, efforts to manage the disaster must take place, including a) a rapid assessment of the situation and the need to manage the risk, b) activation of the command system to manage the disaster, like preparing the operational plan based on the contingency plan, c) evacuating threatened people, fulfilling the basic needs of threatened people, protecting vulnerable groups, and controlling the source of the disaster threat. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 40 Based on interviews with KIs from BPBD and DRR Forums and the villages, none of them have evaluated the tool’s accuracy in disaster prediction, except BPBD of Dompu District. The KI of the Dompu District was previously the head of BPBD, which was moved to the Environmental Office (DLH). The KI could not provide documentation or data on the evaluation result as the documentation is stored at the BPDB office by his previous staff. However, he mentioned that the prediction accuracy is only high in the first two years and then decreases in the third and fourth years. Referring to model documentation and information from the tool developers, disaster prediction was produced based on the rainfall conditions and expert judgment. The tool used a set of rainfall criteria to indicate the condition of rainfall that caused the disaster. The tool developer defined the criteria with reference to IRBI (2017) for flood/landslide and drought, and crop pests referred to Estiningtyas & A Hamdani (2015) and Susanti et al. (2018), and human disease (dengue, malaria and filaria referred to Dini et al. (2010). Different types of crop pests/human diseases used different criteria. However, the reference provided by the tool developer only describes dengue, not other human diseases. In contrast, for crop pests, the references only provide a review on the state of the art of the model for predicting crop pests and diseases. Similarly, the reference for flood/landslides and drought does not describe the criteria. Referring to model documentation, the rainfall condition causing flood/landslide is 600 mm per month, while that causing drought is between 40 mm and 50 mm per decade (dasarian). Further clarification requested by the ET on the criteria has not been responded to yet by the tool developer at the time this report was prepared. In addition, the tool developer clarified that the rainfall criteria adopted for each type of disaster are applied to all locations (villages) in the district. As the disaster prediction is based on the rainfall condition, the accuracy of the disaster prediction will depend on the accuracy of the rainfall forecast. EQ4.6: Do the result of rainfall predictions causing disasters facilitate the coordination of related agencies to prepare for disaster management? From interviews with KIs at district and village levels, seasonal-based rainfall predictions (long-term forecasts) do not facilitate the coordination of related agencies to prepare for disaster management. However, it occurred only in a few villages (Table 14). From the interviews, it is stated that the coordination occurs when the state of emergency is declared. 41 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Table 14. Response of KIs to the use of PCH for disaster preparedness DISTRICT INSTITUTION YES/ NO NOTES Dompu BAPPEDA no DLH (ex BPBD) yes WRS information from BMKG was used for the coordination of related agencies KMPB Nusajaya no KMPB Pekat no the predicted disasters, such as tsunami, was communicatede after Friday prayer Lombok Barat BPBD and FPRB (DRR Forum) no (1) coordinated with BMKG to get information on the climate forecast; (2) most coordination was meant for budget planning for emergency response KMPB Giri Sasak no KMPB Cendi Menik yes initiative coordination at village level; there was information for FPRB at district level KMPB Taman Baru no Lombok Tengah BAPPEDA no BPBD and FPRB no only for emergency response KMPB Tiwugalih no Lombok Timur BPBD and FPRB no/yes (1) there are no strong answers/responses that clearly stated “coordination between agencies” to anticipate future disasters; (2) WhatsApp group for sharing information from “top management” KMPB Gunung Malang no KMPB Padak Guar no/yes most activities were related to emergency response and connected to BPBD KMPB Puncak Jeringo no Nagekeo BPBD no information related to disasters is from regional BMKG; the info was distributed through various means including through letters and WA group FPRB no KMPB Wajomara no KMPB Pagamogo no The project has facilitated the village to assess the status of the village disaster resilience (Desa Tangguh Bencana). Disaster Resilient Village is a village that has the independent ability to adapt and face the threat of disasters, as well as recover quickly from the adverse impacts of disasters (Regulation of the Head of BNPB No. 1 of 2012). This assessment is essential for evaluating the state of the capacity of villages to manage disasters, which is related to institutional response to disaster information. The status of village disaster resilience was assessed using several criteria and indicators that partly follow the criteria and indicators used in the Multihazard Early Warning System (MEWS) of the World Meteorological Organization (WMO). The four criteria in the MEWS of WMO include (i) disaster risk knowledge; (ii) detection, monitoring, analysis, and forecasting of the hazards and possible consequences; (iii) warning dissemination and communication; and (iv) preparedness and response capabilities. Criteria and indicators used in Desa Tangguh Bencana (Perka BNPB 1/2012) generally focus on disaster risk knowledge, preparedness, and response capabilities, not on the parameters of detection, monitoring, analysis, and forecasting of the hazards and possible consequences or warning dissemination and communication. Most villages have been trained to use the evaluation form to evaluate the village disaster resilience status. An evaluation group (representatives from different organizations and village governments) is formed to conduct the assessment. The group assesses the village’s status related to each indicator of the criteria and calculates the score to define the status. This evaluation is conducted annually by USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 42 the evaluation group. It assesses the change in the indicators to define whether the status improves—the status of the disaster resilience village (Pratama-basic, madya-medium, and utama￾advance). The project has reported that most of the villages supported by the project have improved their status to advance (utama), indicating that the project has an impact. Nevertheless, from the evaluation made by the ET, some village staff (village leader), including KMPB, are not aware of the disaster resilience status of their village (most forgot the status, and the result of the evaluation has not been shared). A few KIs stated that the change of personnel may cause discontinuity in the information. EQ4.7: How is the disaster risk information disseminated and communicated broadly? Most village leaders are connected to the BPBD and DRR Forum in the district and included in the WA group. The WA group facilitates the communication and dissemination of disaster information. However, formal dissemination of the disaster risk is done using official letters. Dissemination of disaster information at the village level, mainly when a disaster occurs, generally uses speakers in the mosque and private telephones. The project often carries out disaster socialization and simulations, especially related to earthquakes, and are considered very useful by the community. EQ4.8: Who are the government agencies/ stakeholders/ community groups that use it? What are the benefits of the disaster prediction for the community and government agencies/stakeholders (future benefit vs current benefit)? The agency that routinely used climate information was the Agriculture Office through their field assistance facilitator (PPL) and BPBD. There were no agencies/stakeholders that solely used one source of PCH. They combined PCH with other information, such as from BMKG. The current benefit of using PCH/BMKG information is that they could prepare in advance to reduce the disaster impacts. EQ4.9: What policy needs to be in place for increasing the use of climate forecast information for disaster preparedness? Most KIs revealed that intensive field assistance and socialization are encouraged to increase farmers’ awareness. Funding resources for supporting the intensive field assistance and socialization should be integrated into the annual budget planning of the agriculture offices. Integrating climate￾related disasters into government regulation also increases the use of climate information. CONCLUSIONS 1. BPBD uses BMKG information as the primary disaster information, and PCH information for additional and comparative information. 2. Assessment of Desa Tangguh Bencana has been carried out annually following indicators defined by BNPB, which only include part of MEWS indicators of WMO. BNPB does not include indicators related to parameters of detection, monitoring, analysis, and forecasting of the hazards and possible consequences or warning dissemination and communication, which are critical for assessing the state of Desa Tangguh Bencana. 3. Some villages and KMPBs are still not aware of the Desa Tangguh Bencana status. 4. Disaster socialization and simulations are often carried out, especially related to earthquakes, and are considered very useful by the community. 43 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV 5. To accelerate the dissemination and exchange of information on early warning/disaster information between national and local governments (districts, sub-districts, villages), local governments have used an informal communication system via a WhatsApp (WA) group. 6. Dissemination of information in villages when a disaster occurs generally uses speakers in the mosque, and private telephones. RECOMMENDATIONS 1. Future project activities should focus on the public engagement of climate information. 2. The future project should support and facilitate local government and stakeholders to increase awareness of climate-related disasters by integrating climate information into regulations. 3. The future project should facilitate BNPB/BPBD to adopt the use of EWS Multi-Hazard Criteria of WMO in assessing Desa Tangguh Bencana (Disaster Resilience Village). 4. The dissemination of Desa Tangguh Bencana should involve local stakeholders and the key person at the village level. Thus, they understand the meaning behind “resilience” as defined in criteria and indicators of Desa Tangguh Bencana and allow them to take necessary actions to increase the resilience. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 44 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK STATEMENT OF WORK - EVALUATION OF EFFECTIVENESS OF A RAINFALL PREDICTION TOOL FOR FARMERS IN THE PROVINCES OF NUSA TENGGARA BARAT AND NUSA TENGGARA TIMUR 1. Purpose USAID’s Bureau for Humanitarian Assistance (USAID/BHA) would like to commission an independent evaluation of the accuracy, effectiveness, and utility of a seasonal Rainfall Prediction Tool that has been developed by World Neighbors and their partners at Bandung Technology Institute (ITB) and is being used to advise farmers in the provinces of NTB (West Nusa Tenggara) and NTT (East Nusa Tenggara). 2. Audience and Intended Use The primary audience of this evaluation is USAID/BHA and its implementing partner, World Neighbors, to allow them to verify the utility and the usefulness of the tool, as well as the added value to participating farmers-reducing their climate risk. The results of this evaluation will also inform the design of a follow-on to the USAID APIK (Adapatasi Perubahan Iklim dan Ketangguhan – Climate Change Adaptation and Resilience) in urban settings by providing information on the accuracy and the utility of the tools and data from the national weather service (BMKG) in order to promote the dissemination and the use of similar tools for early warning information, disaster preparedness and climate change adaptation. The secondary audience includes key stakeholders, such as BMKG and other national and local agriculture departments, to allow them to adopt and institutionalize this Rainfall Prediction tool to support farmers in other regions of Indonesia. 3. Background USAID BHA has been supporting the non-governmental organization (NGO), World Neighbors, and their activities in the eastern Indonesian provinces of NTB and NTT. These activities included the formation and the training of Village Disaster Management Committees and working with village governments to achieve resilient village status per Government of Indonesia metrics; the support to regency-level DRR multi-stakeholder forums; village disaster risk assessments and risk management plans; undertaking village-level projects to reduce risk and improve resilience; and promotion of climate-smart agriculture technologies to better cope with rainfall variability. Among the latter, World Neighbors partnered with ITB staff who developed a regency-specific and seasonal Rainfall Prediction tool to advice farmers on what and when to plant each main rain-fed cropping season as well as the potential for pests and plant infestations. The tools also illustrate the likelihood of climate-related disasters (flooding, landslides, droughts) as well as the spread of malaria and dengue fever, allowing government and communities to better prepare for disasters. The tool has been adopted in 5 districts in NTT and NTB, including 3 districts in Lombok, Dompu District (Sumbawa), and 1 district in Flores. 45 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV 4. Information Sources USAID recommends the following materials for the desk review of this evaluation: • Quarterly and/or Annual Progress Reports and as well as evaluations focusing specifically on the development and the use of the Rainfall Prediction Tool • An internal evaluation report from World Neighbors • Other project monitoring report documents 5. Evaluation Questions This evaluation will assess four evaluation questions with explanatory questions for each question: 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? • What data is used to feed the seasonal forecast model? And where does it come from? • What are the components/variables used in the model to predict/forecast rainfall timing, duration, and intensity for a season? • Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? How much in advance of the season can the forecast be made? • Are there other models used by BKMG that could be modified to produce similar forecasts? And if so, what is restricting their adoption? 2. What are the utilities of the Rainfall Prediction Model/Tool for farmers? • How are the results of the forecast turned into advice to farmers? • For farmers who used the rainfall prediction advice, what difference did it make in that season’s production? • What percentage of farmers are using the tool? If the percentage is low, why? Do they view it as a helpful tool (provide examples), and why? • What is the percentage increase in food crop production from the farmers who adopt/follow the rainfall prediction and cropping pattern guidance? 3. By comparing the traditional system and online application as means of information dissemination, which system is more effective for farmers? And why? • How do the community and/ or farmers access the rainfall prediction and cropping pattern guidance information? What are the primary sources of information that farmers use to guide decision-making each season? • Is there any difference between farmers receiving rainfall prediction advice from in-person gatherings and those using the online application? Do farmers understand and adopt the advice (in-person and online advice)? • Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places with no app support? 4. In what way has the tool been utilized for Disaster Preparedness? • How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? • Who takes advantage of the disaster predictions? Who are the government agencies/stakeholders/community groups that use it? What are the benefits of the disaster prediction for the community, government agencies/stakeholders? USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 46 6. Gender Considerations In accordance with USAID’s Automated Directive System 201 point 7, the research design for this evaluation shall consider gender-specific aspects. While a detailed analysis by gender might not be relevant for each question, where applicable, the evaluation team shall explore gender aspects in line with the evaluation questions and available data sources. The analysis may require more than simple disaggregation of quantitative data. The evaluation team must refer to relevant USAID guidance on gender and inclusion, specifically ADS 205, and propose specific evaluation designs as appropriate. 7. Evaluation Design and Methodology The evaluation team shall consider a mixed-method evaluation approach to the extent possible. The methodology shall combine a review of the quantitative data and application the qualitative evaluation techniques to obtain information, opinions, and data. The evaluation team will examine the data sources, review the variables used in the model, and compare rainfall vs. actual rainfall in previous seasons. By using a mixed approach, the evaluation team shall gain insight on the effectiveness int the rainfall prediction tool for farmers. The evaluation team shall draft an evaluation methodology or design for USAID approval. The detailed methodology of this evaluation shall be described by the evaluation team in the Work Plan. It shall include an evaluation matrix that explicitly links the evaluation questions to particular data collection approaches and data sources. In choosing possible data collection methods, the evaluation design must consider the implications of an operating environment that have been altered by the COVID-19 pandemic. Depending on the prevailing environment at the time of the evaluation, the evaluation team must propose an evaluation design that considers remote monitoring or data collection methods as appropriate. This may include methods such as the use of cellphones or SMS to conduct interviews, interactive voice responses, and voice calls. The following essential elements shall be included in the methodology: Document Review: USAID/BHA will provide the evaluation team with a core list and/or copies of the activity design, tool design/model, monitoring, and relevant evaluation reports before the evaluation begins. The evaluation team leader shall be responsible for expanding this background documentation as appropriate reviewing, prioritizing, and distributing it to the other team members for their review. All team members shall review relevant documentation before their initial team meetings. Key informant interviews: The evaluation team shall conduct interviews to obtain data and information from farmers on who has adopted the tool(s), who hasn’t, and why; and to compare knowledge and adoption between the user who have the online application and those who do not; NGO and Agriculture staff who deliver the Rainfall prediction information to farmers. Focus Group Discussions: Discussion groups shall include in-depth discussions with the Rainfall Prediction modelers; examination of data sources, review of the variables used in the model; comparison of predicted rainfall vs. actual rainfall in previous seasons. 47 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Site Visits: Ideally, the evaluation team will visit 5 districts in NTB and NTT to gather primary data and information. However, depending on the prevailing environment at the time of the evaluation, the evaluation team must propose an evaluation design that considers remote data collection methods as appropriate. This may include methods such as the use of cellphones or SMS to conduct interviews, interactive voice response, voice calls, or maximizing national data collection activities that are still active, etc. The final site selection for the evaluation will be determined collaboratively by USAID, World Neighbors, MEL-P, and the evaluation team in order to determine an implementation plan that allows for a rigorous evaluation. Staff from MEL-P, World Neighbors, and the USAID/BHA team shall assist the evaluation team in organizing logistics for all site visits or remote-based data collection approaches. Data Analysis: Team members shall analyze program reporting and evaluation documents and information gained from key informant interviews and other data collection methods in order to inform their findings and recommendations. The team shall keep a record of meetings that take place and record the summaries of each meeting. Some quantitative analyses may be featured, for example, in the review of documents or secondary data. The evaluation team shall analyze the information collected to establish credible answers to the questions. 8. Evaluation Tasks Given the above requirements, the evaluation team will be required to perform the below tasks per USAID ADS 201. Component 1 – Design Task 1: Draft Inception or Evaluation Design Report There is no page requirement for the Inception Report, but it shall contain a Data Collection Plan, Analysis Plan, Dissemination Plan, Limitations/Risks, Quality Control Protocols, and Work Plan. The Inception Report shall contain all these components as outlined below. • Data Collection Plan o All data sources are identified and mapped against the evaluation question they are meant to answer. o The method of data collection, including remote data collection, for each data source, who will perform the collection, and the timing for when the collection will take place. o Relevant enumerator training protocols (if applicable), instrument piloting plan, and quality control procedures. o Roles and Responsibilities of the Evaluation Team for completing data collection tasks. • Data Analysis Plan o Analytical methods for each type of data collection method to include relevant quality controls for the method. o Roles and Responsibilities of the Evaluation Team for completing data analysis tasks. • Limitations o The limitation of proposed collection and analysis methods include bias, missing data points for triangulation, timing issues, etc. per evaluation question. o Risk Management strategy for mitigating the effects of limitations, and issues related USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 48 to remote data collection methods as appropriate. • Dissemination o Timeline for producing the In-brief, Inception Report, Final Report, and Out-brief. • Quality Control: For all data collection, analysis, and dissemination tasks, outline the risks to evaluation quality and the steps being taken to mitigate the risks with relevant citations of applicable evidence and best practice. • Work Plan: An outline of all deliverables, collection, analysis, and other tasks set against a timeline that matches the period of evaluation. • Data Collection Instruments: All relevant data collection instruments shall be included in an annex to the Inception Report. Task 2: Submit a draft Inception or Evaluation Design Report • On or about May 6 – 11, 2022, the evaluation team will conduct team planning meetings with the USAID/Indonesia Monitoring, Evaluation and Learning Platform (MEL-P), and USAID. • On or about May 27, 2022, the evaluation team must submit the draft Inception Report to MEL-P for review. • On or about May 31, 2022, MEL-P will share the draft Inception Report with USAID for review. • On or about June 7, 2022, the evaluation team will present the evaluation design to USAID. Task 3: Finalize Inception or Evaluation Design Report • On or about June 8-14, 2022, the evaluation team will integrate all relevant feedback into the Inception Report, including providing responses on how feedback was or was not integrated into the Inception Report and why. • On or about June 15, 2022, MEL-P will submit the final Inception Report to USAID for approval. Component 2 – Data Collection Task 1: Enumerator Training (as applicable) • If applicable, the evaluation team will provide all relevant training to data collectors as outlined in the Inception Report. • If requested, the evaluator will provide relevant enumerator training materials. Task 2: Instrument Piloting • The evaluation team will pilot all data collection instruments before they are used and provide necessary piloting feedback as requested. Task 3: Perform Data Collection • The evaluation team will perform all data collection tasks outlined in the Inception Report and provide timely updates to MEL-P. • All quantified data will be digitally recorded and stored. All qualitative data will have summary sheets and transcripts provided upon request. Component 3 – Data Analysis and Preliminary Findings, Conclusions, and Recommendations Task 1: Data Cleaning 49 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV • The evaluation team will scrub all data sets of personal identification information in accordance with USAID policy. • The evaluation team will follow best practices in preparing data sets for analysis as outlined in the Inception Report. Task 2: Data Management • All data must be stored in a secure drive that is only accessed by the evaluation team. If MEL-P or USAID requires access to the raw data set, it will be done in accordance with USAID policy after removing all personal identification information. Task 3: Data Analysis • All data will be analyzed using the methods outlined in the Inception Report. • Final codebooks and statistical analysis framework will be submitted to MEL-P and USAID upon request. • Relevant quality control protocols (spot checks, inter-rater reliability checks, etc.) as outlined in the Inception Report will be adhered. Task 4: Preliminary findings, conclusions, and recommendations • The evaluation team will compile and present the preliminary findings, conclusions, and recommendations to MEL-P and USAID and other stakeholders for feedback that will be incorporated into the evaluation report. Component 4 – Evaluation Report and Existing Briefings Task 1: Draft and Final Evaluation Report • On or about August 16, 2022, the evaluation team will be submitted a draft report to MEL-P. • On or about August 25, 2022, MEL-P will submit the draft report to USAID and other stakeholders as appropriate for feedback. • On or about September 6 – 14, 2022, the evaluation team will address and integrate all relevant feedback into the evaluation Report. • On or about September 22, 2022, MEL-P will submit the final evaluation report to USAID for approval. Task 2: Final presentations or exit briefings • On or about September 26 – October 3, 2022, the evaluation team will provide exit briefs to MEL-P, USAID, and other relevant stakeholders to present the final results of the evaluation. 9. Estimated Cost The total estimated cost for this independent evaluation is USD 122,180, including an estimated cost for MEL-P staff who will oversee and coordinate the work of the evaluation team. 10. Management This evaluation will be conducted through MELP support. The MELP will establish an evaluation in close coordination with USAID. This evaluation will require an evaluation team consisting of two experts with Agro-Meteorology and weather/climate forecast modeling backgrounds. USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 50 11. Deliverables All deliverables must be submitted to MELP, who will submit them to USAID. The evaluation team must promptly notify MELP of any problems, delays, or adverse conditions which materially impair the evaluation team’s ability to meet the requirements. The evaluation team is expected to deliver the following products or complete the following key tasks at the estimated due dates indicated below: Deliverables or key tasks Estimated due dates Initial team planning meeting with USAID and/or other stakeholders May 11, 2022 Submission of draft evaluation design or inception report May 31, 2022 In-brief to USAID on the evaluation design June 7, 2022 Final evaluation design or inception report June 15, 2022 Presentation(s) to USAID and/or other stakeholders of key findings and preliminary conclusions and recommendations October 18, 2022 Evaluation reports: ● A draft and final report describing the Rainfall Prediction model and addressing its accuracy (max 20 pages) ● A draft and a final report on the effect of adopting/following the Rainfall Prediction advice on farmers’ production (max 20 pages) ● A draft and a final brief report that follows the World Meteorological Organization checklist on Multi-hazard Early Warning Systems, noting where checklist items may not be relevant or where there are data limitations. see https://library.wmo.int/doc_num.php?explnum_id=4463. The report should highlight checklist items that are major contributors to program success or that may limit success. - (draft and final) November 10, 2022 Final evaluation reports December 7, 2022 Final presentations or exit briefings to USAID and other stakeholders December 15, 2022 The evaluation team will ensure that the written products use plain language, concise, audience appropriate, representative of USAID achievements, and the USAID Style Guide 2021 is adhered to. Reporting and Dissemination The evaluation report format shall follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing￾evaluation-reports). Evaluation team members shall be provided with the USAID’s mandatory statement of the evaluation standards they are expected to meet (see Annex A). 12. Schedule and logistics The evaluation will be expected to start from April 26 to October 3, 2022. MEL-P shall be responsible for handling all logistics. 51 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV Estimated Evaluation Timeline Tasks or Deliverables Estimated start date (dd/mm/yy) Estimated end date (dd/mm/yy) Finalize SOW, evaluation budget, and implementation schedule 02-March-22 07-March-22 Evaluation team: Development of SOWs, recruitment, and contracting 22-February-22 22-April-22 Initial team planning meetings and desk review 26-April-22 05-May-22 Team planning meetings with USAID 06-May-22 11-May-22 Develop evaluation design or inception report 12-May-22 20-May-22 Submit draft evaluation design or inception report, including presentation to USAID, and USAID’s review and feedback 31-May-22 14-June-22 Finalize and submit evaluation design or inception report, including USAID’s approval 15-June-22 21-June-22 Data collection and analysis 11-July-22 16-September-22 Compilation of preliminary findings, conclusions, and recommendations. 30-September-22 04-October-22 Presentation(s) to USAID and/or other stakeholders of preliminary findings, conclusions, and recommendations 12-October-22 19-October-22 Report Writing, including internal and editor’s review 20-October-22 09-November-22 Submission of draft evaluation reports 10-November-22 10-November-22 USAID’s and/or other stakeholders’ review of the draft evaluation report 11-November-22 18-November-22 Drafting and submission of final evaluation report 21-November-22 07-December-22 Final presentations or exit briefings to USAID and other stakeholders 08-December-22 15-December-22 13. Evaluation Team Composition The evaluation team will consist of two experts, including: ● A Team Leader, preferably an Indonesian national with a minimum of a Master’s degree, preferably a Doctorate Degree and Agro-Meteorology experience and an understanding of weather and climate forecast modeling. ● An Agriculture and Climate Change Expert, an Indonesian national with experience in Agriculture Extension and skill in interviewing key informants and holding focus group discussions are also required. ● A Research Assistant, an Indonesian national with M&E knowledge and experience, and in research, analysis, or reporting on one or more of the following issues: weather and climate forecast, agriculture, or climate change. Collectively the team must have experience in developing and testing data collection instruments, implementing data quality protocols in the field, collecting data, and training and supervising enumerators (if any). USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 52 MEL-P shall share a copy of the resume of the selected team leader for USAID review and approval prior to the implementation of the evaluation. 14. Regular Communication Regular communication between the evaluation team, MEL-P, USAID/BHA, and USAID M&E Team will be essential to the successful execution of this evaluation. Through MEL-P, the evaluation team shall keep USAID apprised of changes and developments that necessitate any significant decision￾making or modification of the approved evaluation design. 53 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV ANNEX II: EVALUATION QUESTIONS IN THE APPROVED INCEPTION REPORT USAID has provided the evaluation team (ET) with evaluation questions as stated in the Statement of Work (SOW; Annex 1) to assess the seasonal Rainfall Prediction Tool’s accuracy, effectiveness, and utility. The evaluation team has revised the original questions from the SOW to increase the coherence and alignment of the questions with the evaluation purposes. This approach will allow respondents to answer questions in more detail and perhaps provide richer qualitative data. The ET provided a brief description of the evaluation team’s understanding of the focus of each question on table below in the inception report that approved by USAID on June 21, 2022. EVALUATION QUESTION (EQ) COMMENTS 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.1. What are the components/variables used in the model to predict/forecast rainfall? Where do they come from? Hundreds of climate forecasting models are available, and each model has limitations and provides different levels of accuracy spatially and temporally. Different models may use different assumptions to address the limitations that affect the reliability and accuracy of the models. The availability of data inputs used in the development of the model will also affect the reliability and accuracy of the models. The evaluation team (ET) has added new sub-questions (in italics) to cover those aspects. 1.2. What types of rainfall characteristics are being forecasted (season onset, duration, and intensity for a season? What are the lead times of the forecast – how much in advance the forecast be made)? 1.3. What are the methods used for rainfall forecasting? Statistical, dynamic, or statistical-dynamic model? Ensemble models? 1.4. What are the assumptions used in the model? 1.5. What is the area covered by the forecast tool? Is it only for specific sites (districts)? What is the resolution? 1.6. What is the period of data used for developing the model, as well as calibration/validation and verification of the model? And the period used for updating the forecast? 1.7. What are the methods used to evaluate the accuracy of the forecast? 1.8. What criteria are used to define that the forecast is accurate 1.9. Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth-year? Is the accuracy of the forecast conducted on a point base or spatial base? 1.10.Are there other forecast models in use by BKMG that could be modified to produce similar forecasts? And if so, what is restricting their adoption? 2. What are the utilities of the Rainfall Prediction Model/Tool for farmers? 2.1. How does the climate affect farmers’ cropping system? (e.g., change in season onset/false rain, season break, extreme rainfall, etc.) The ways farmers address climate variability in their cropping system depends on their understanding of how the climate affects their crops and the availability of options for managing the variability. The ET has added these elements (in italics) in sub-questions 2.2. How are the forecast results turned into advice to farmers? What is the advice? 2.3. What percentage of farmers are using the tool? If it is low, why? Do they view it as a useful tool (provide examples), and why? 2.4. How is the advice used by the farmers for making planting decisions or other planting operations (provide the example and reason ~, e.g., adjust technology like delay/accelerate planting, change planting arrangement, irrigation scheduling, applying pesticide/herbicide; change commodity; etc.)? 2.5. For farmers who used the rainfall prediction advice, what difference did it make in that seasonal production? 2.6. For the farmers who adopt/follow the rainfall prediction, cropping pattern, and other guidance, what is the percentage increase in food crop production and/or income? 2.7. How to differentiate the increase in food production due to changes in or adopted improved farming practices and due to the adoption of forecasts? USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 54 EVALUATION QUESTION (EQ) COMMENTS 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 3.1. What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? The effectiveness of climate information used by farmers depends on the timeliness of the information received by farmers and ways to communicate the information to the farmers. ET has added a new sub-question (in italics) to cover this aspect. 3.2. What are the main sources of information that farmers use to guide decision-making each season? What is information that farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? 3.3. Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in￾person gatherings and those using the online application? 3.4. Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places with no app. support? 4. In what ways have the tool been utilized for Disaster Preparedness? 4.1. What is the awareness level of stakeholders on the tool? Availability of climate forecast and translation of the forecast into disaster risk/impact outlook are key elements in managing the disaster risk. ET has added a new sub-question (in italics) to cover this aspect. 4.2. Do they use the tool in disaster prediction? How is it used? What are the forms of disaster information provided based on the tool? 4.3. Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? 4.4. How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? By whom? 4.5. How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? 4.6. Do the results of rainfall predictions causing disasters to facilitate the coordination of related agencies to prepare for disaster management? 4.7. How is the disaster risk information disseminated and communicated broadly? 4.8. Who are the government agencies/stakeholders/ community groups that use it? What are the benefits of the disaster prediction for the community, government agencies/stakeholders? (future benefit vs. current benefit) 55 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV ANNEX III: DATA COLLECTION AND ANALYSIS TOOLS Interview template INTERVIEW TEMPLATE – WN Team; WN Local partners Date: Location: Name of organization: Name / Job title of respondent(s): Gender of the respondent(s): Name of the interviewer(s): Evaluation Questions Response EQ 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.1. What are the components/variables used in the model to predict/forecast rainfall? And where do they come from? 1.2. What types of rainfall characteristics are being forecasted (season onset, duration, and intensity for a season? What are the lead times of the forecast - how much in advance the forecast be made)? 1.3. What are the methods used for rainfall forecasting? Statistical, dynamic, or statistical-dynamic model? Ensemble models? 1.4. What are the assumptions used in the model? 1.5. What is the area covered by the forecast tool? Is it only for specific sites (districts)? What is the resolution? 1.6. What is the period of data used for developing the model, as well as calibration/validation and verification of the model? And the time period used for updating the forecast? 1.7. What are the methods used to evaluate the accuracy of the forecast? 1.8. What criteria are used to define that the forecast is accurate? 1.9. Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth- year? Is the accuracy of the forecast conducted on a point base or spatial base? EQ 2. What are the utilities of the Rainfall Prediction Model/Tool for farmers? 2.1. How the climate affects farmers’ cropping system? (e.g., change in season onset/false rain, season break, extreme rainfall, etc.) 2.2. How are the results of the forecast turned into advice to farmers? What is the advice? 2.3. What percentage of farmers are using the tool? If it is low, why? Do they view it as a useful tool (provide examples), and why? 2.4. How is the advice used by the farmers for making planting decisions or other planting operations (provide the example and reason ~ e.g. adjust technology like delay/accelerate planting, change planting arrangement, irrigation scheduling, applying pesticide/herbicide; change commodity; etc.)? 2.5. For farmers who used the rainfall prediction advice, what difference did it make in that seasonal production? 2.6. From the farmers who adopt/follow the rainfall prediction, cropping pattern, and other guidance, what is the percentage increase in food crop production and/or income? 2.7. How to differentiate the increase in food production due to changes in or adopted improved farming practices and due to the adoption of forecasts? EQ 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 1.1 What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? 1.2 What are the main sources of information that farmers use to guide decision- making each season? What is information that farmers use to guide decision- making each season? How do farmers make decisions in each season from the provided information? 1.3 Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 56 advice from in-person gatherings and those using the online application? 1.4 Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places where there is no app. support? EQ 4. In what ways have the tool been utilized for Disaster Preparedness? 4.1. What is the awareness level of stakeholders on the tool? 4.2. Do they use the tool in disaster prediction? And how is it used? What are the forms of disaster information provided based on the tool? 4.3 Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? 4.4 How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? And by whom? 4.5 How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? 4.6 Do the result of rainfall predictions causing disasters facilitate the coordination of related agencies to prepare for disaster management? 4.7 How is the disaster risk information disseminated and communicated broadly? 4.8 Who are the government agencies/stakeholders/community groups that use it? What are the benefits of the disaster prediction for the community, government agencies/stakeholders? (future benefit vs. current benefit) 4.9 What policy needs to be in place to increase the use of climate forecast information for disaster preparedness? 57 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV INTERVIEW TEMPLATE – EXPERTS Date: Location: Name of organization: Name / Job title of respondent(s): Gender of the respondent(s): Name of the interviewer(s): Evaluation Questions Response EQ 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.3 What are the methods used for rainfall forecasting? Statistical, dynamic, or statistical- dynamic model? Ensemble models? 1.4 What are the assumptions used in the model? 1.5 What is the area covered by the forecast tool? Is it only for specific sites (districts)? What is the resolution? 1.6 What is the period of data used for developing the model, as well as calibration/validation and verification of the model? And the time period used for updating the forecast? 1.7 What are the methods used to evaluate the accuracy of the forecast? 1.8 What criteria are used to define that the forecast is accurate? 1.9 Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? Is the accuracy of the forecast conducted on a point base or spatial base? 1.10 Are there other forecast models in use by BKMG that could be modified to produce similar forecasts? And if so, what is restricting their adoption? Note: the questions are not only referred to the tool, but also models developed by experts EQ 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 3.1. What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? 3.2. What are the main sources of information that farmers use to guide decision-making each season? What is information that farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? 3.3. Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in- person gatherings and those using the online application? 3.4. Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places where there is no app. support? Note: EQ3 is only asked to expert from BMKG and Kementan EQ 4. In what ways have the tool been utilized for Disaster Preparedness? 4.1 What policy needs to be in place to increase the use of climate forecast information for disaster preparedness? USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 58 INTERVIEW TEMPLATE – Regional BMKG Date: Location: Name of organization: Name / Job title of respondent(s): Gender of the respondent(s): Name of the interviewer(s): Evaluation Questions Response EQ 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.10 Are there other forecast models in use by BKMG that could be modified to produce similar forecasts? And if so, what is restricting their adoption? EQ 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 3.1 What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? 3.2 What are the main sources of information that farmers use to guide decision-making each season? What is information that farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? 3.3 Do farmers understand and adopt the provided in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in-person gatherings and those using the online application? 3.4 Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places where there is no app. support? EQ 4. In what ways have the tool been utilized for Disaster Preparedness? 4.1 What is the awareness level of stakeholders on the tool? 4.2 Do they use the tool in disaster prediction? And how is it used? What are the forms of disaster information provided based on the tool? 4.3 Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? 4.4 How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? And by whom? 4.5 How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? 4.6 Do the result of rainfall predictions causing disasters facilitate the coordination of related agencies to prepare for disaster management? 4.7 How is the disaster risk information disseminated and communicated broadly? 4.9 What policy needs to be in place to increase the use of climate forecast information for disaster preparedness? 59 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV INTERVIEW TEMPLATE – Local Government (Agriculture Division, Bappeda/planning division) Date: Location: Name of organization: Name / Job title of respondent(s): Gender of the respondent(s): Name of the interviewer(s): Evaluation Questions Response EQ 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.1 Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? Is the accuracy of the forecast conducted on a point base or spatial base? EQ 2. What are the utilities of the Rainfall Prediction Model/Tool for farmers? 2.1 How the climate affects farmers’ cropping system? (e.g. change in season onset/false rain, season break, extreme rainfall, etc.) 2.2 How are the results of the forecast turned into advice to farmers? What is the advice? 2.3 What percentage of farmers are using the tool? If it is low, why? Do they view it as a useful tool (provide examples), and why? 2.4 How is the advice used by the farmers for making planting decisions or other planting operations (provide the example and reason ~ e.g. adjust technology like delay/accelerate planting, change planting arrangement, irrigation scheduling, applying pesticide/herbicide; change commodity; etc.)? 2.5 For farmers who used the rainfall prediction advice, what difference did it make in that seasonal production? 2.6 From the farmers who adopt/follow the rainfall prediction, cropping pattern, and other guidance, what is the percentage increase in food crop production and/or income? 2.7 How to differentiate the increase in food production due to changes in or adopted improved farming practices and due to the adoption of forecasts? Note: EQ2 is only asked for the Agriculture division EQ 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 3.1 What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? 3.2 What are the main sources of information that farmers use to guide decision-making each season? What is information that farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? 3.3 Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in- person gatherings and those using the online application? 3.4 Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places where there is no app. support? EQ 4. In what ways have the tool been utilized for Disaster Preparedness? 4.1. What is the awareness level of stakeholders on the tool? 4.2 How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? 4.3 Do the result of rainfall predictions causing disasters facilitate the coordination of related agencies to prepare for disaster management? 4.4 What policy needs to be in place to increase the use of climate forecast information for disaster preparedness? USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 60 INTERVIEW TEMPLATE – BPBD, DRR forums Date: Location: Name of organization: Name / Job title of respondent(s): Gender of the respondent(s): Name of the interviewer(s): Evaluation Questions Response EQ 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.1 Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? Is the accuracy of the forecast conducted on a point base or spatial base? EQ 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 3.1 What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? 3.2 What are the main sources of information that farmers use to guide decision-making each season? What is information that farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? 3.3 Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in- person gatherings and those using the online application? 3.4 Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places where there is no app. support? EQ 4. In what ways have the tool been utilized for Disaster Preparedness? 4.1 What is the awareness level of stakeholders on the tool? 4.2 Do they use the tool in disaster prediction? And how is it used? What are the forms of disaster information provided based on the tool? 4.3 Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? 4.4 How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? And by whom? 4.5 How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? 4.6 Do the result of rainfall predictions causing disasters facilitate the coordination of related agencies to prepare for disaster management? 4.7 How is the disaster risk information disseminated and communicated broadly? 4.8 Who are the government agencies/stakeholders/community groups that use it? What are the benefits of the disaster prediction for the community, government agencies/stakeholders? (future benefit vs. current benefit) 4.9 What policy needs to be in place to increase the use of climate forecast information for disaster preparedness? 61 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV INTERVIEW TEMPLATE – Village governance, KMPB and Farmers Date: Location: Name of organization: Name / Job title of respondent(s): Gender of the respondent(s): Name of the interviewer(s): Evaluation Questions Response EQ 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.1 Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? Is the accuracy of the forecast conducted on a point base or spatial base? EQ 2. What are the utilities of the Rainfall Prediction Model/Tool for farmers? 2.1. How the climate affects farmers’ cropping system? (e.g. change in season onset/false rain, season break, extreme rainfall, etc.) 2.2. How are the results of the forecast turned into advice to farmers? What are the advices? 2.3. What percentage of farmers are using the tool? If it is low, why? Do they view it as a useful tool (provide examples), and why? 2.4. How is the advice used by the farmers for making planting decisions or other planting operations (provide the example and reason ~ e.g. adjust technology like delay/accelerate planting, change planting arrangement, irrigation scheduling, applying pesticide/herbicide; change commodity; etc.)? 2.5. For farmers who used the rainfall prediction advice, what difference did it make in that seasonal production? 2.6. From the farmers who adopt/follow the rainfall prediction, cropping pattern, and other guidance, what is the percentage increase in food crop production and/or income? 2.7. How to differentiate the increase in food production due to changes in or adopted improved farming practices and due to the adoption of forecasts? EQ 3. By comparing the traditional system and online application as a means of information dissemination, which system is more effective for farmers? And why? 3.1 What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? 3.2 What are the main sources of information that farmers use to guide decision-making each season? What is information that farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? 3.3 Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in- person gatherings and those using the online application? 3.4 Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places where there is no app. support? EQ 4. In what ways have the tool been utilized for Disaster Preparedness? 4.1 What is the awareness level of stakeholders on the tool? 4.2 Do they use the tool in disaster prediction? And how is it used? What are the forms of disaster information provided based on the tool? 4.3 Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? 4.4 How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? And by whom? 4.5 How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than the second, third, or fourth? Why? USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 62 LIST OF CLARIFICATION QUESTIONS TO TOOL DEVELOPER 1. The year period of climate data at each station used to make the basic rainfall prediction model using the FFT method (If it is too time consuming to fill in, it is enough to inform the general data year period used for prediction as well as the prediction year period and data year period to verify results predictions in each district) 2. The period of the year of climate data at each station used to update the basic rain prediction model with the FFT method (If it takes too long to fill it out, it is enough to inform the general data year period used to update predictions as well as the prediction year period and verification data year period for each district) 3. Based on the available documents, to make predictions of spatial rain is done by interpolation technique based on the kriging method. Is the prediction data used only data from the model results from the station above (in question No. 1)? YES NO a. If NO, is satellite data (such as CHIRPS) used to create predictive models? YES NO b. If YES, is the CHIRPS data used directly or is bias correction done first, or is it only used to map the shape of the rain distribution pattern? c. If the CHIRPS data is corrected for bias, what method is used and the references (such as books and scientific journals)? d. If the bias correction model is prepared by yourself (there is no reference), please explain the method used? e. If the bias correction is not done, and the CHIRPS satellite data is only used to get the rain distribution pattern, can you please explain the method or at least the reference? 4. Based on the verification and validation report documents, the method used is based on the R2, R-skill and RSME values at each station. Was checking the accuracy of the model spatially carried out referring to the CHIRPS data? YES NO a. If YES, is the spatial map of the model accuracy checking available? YES NO b. If YES, is documentation available? YES NO c. If YES, please provide the documents 5. For the purposes of further evaluation of the prediction model, can the data on the results of rain predictions be provided before and after being updated for the 2018-2022 period. Please provide predictive data for each grid or at least one point (grid) data for each village along with its coordinates. 6. In the preparation of the prediction map of the planting period, and the potential for climate￾related disasters (floods/landslides, droughts, distribution of pests, distribution of human diseases) are determined based on data on predictions of basic rainfall height (pattern/distribution and basic rainfall height). In the documentation obtained, the criteria used in determining the potential for disasters are still limited. What are the criteria or rain conditions used to determine the potential for disaster? a. For the types of plant pests and human diseases (DHF, Malaria, Filaria), do the criteria used above apply to all types of human pests and diseases? YES NO b. If NO, what other types of information are used to establish a potential disaster? c. For potential flood disasters, such as the example in the picture below for the case in Central Lombok (from the Update on Rainfall Prediction Modeling…), the baselines that have the potential for flooding are January 1, Dasarian 3 February, Dasarian 3 April, October 3 and Dasarian. December 3rd. This information shows that the level of rain that has the potential to cause flooding is not the same. Rain in February, April and October with predictions of high rainfall for the bases is much lower than Jan (Dasari 2 and 3) and December (Basic 1 and 2), potentially flooding, 63 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV while in the two bases, Jan and Dec have no potential. Can you please explain what other information or conditions are used besides the rain height in determining the potential for flooding? (note: this question does not need to be answered if it has been explained in the table above other criteria/indicators other than rain) Rainfall in Gunung Sari (2019). Red circle means the potential of landslide d. Are the criteria/indicators used for determining disaster potential described in the table of disaster criteria/indicators above (No. 6) applied to all locations in the related district? YES NO e. If NO, what other factors were used or considered in the area? (note: this question does not need to be answered if it has been explained in the table above other criteria/indicators other than rain) 7. In the documents, there is information related to rainfall condition, disaster and planting date, and harvest time in the last 5 years for each dasarian that gathered form interviews/FGD with farmers (example provided in Table below). Are the data from the interview being used to determine planting date and disaster potential as explained in the Table of criterie or disaster indicator above? YES/NO a. If YES, are these data, especially for disasters resulting from farmer interviews, verified using data from other available observations from other institutions (e.g., statistical data, BPBD, Department of Agriculture, etc.)? YES/NO b. If YES, what types and sources of data are used? 8. In the online application (http://smartclim.info/nagekeo/prediction?variable=6) which has been compiled for three districts (Nagekeo, Dompu and Loteng), is the rain data in the application used for making spatial maps different from the rain data in the application used for making spatial maps? which is used to display the basic rain graph? For the smartclim/nagekeo application, for example, the base rain height at a certain location on the map with that on the base rainfall graph is inconsistent, indicated by the mismatch USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 64 between the colours on the map and the rainfall height on the base rainfall graph. For example, the second basic rain prediction in December 2021 in the village of Tedamude, on the graph the rain prediction is 29 mm, while based on the colour on the map the prediction is higher (see the following picture). The data inconsistencies in the graphs and colours on the map are quite common at points in other villages. Which predictive data is the true value is it on the map or on the graph? 9. In the android/online application, weather prediction information (hourly and daily) is also provided, what method or model is used in weather prediction? If another model is used, please provide a reference. 65 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV ANNEX IV. EVALUATION DESIGN MATRIX Primary EQ Sub-questions Data Collection Methods Data Analysis Approach 1. To what extent has the Rainfall Prediction Model/Tool been applied? Is the model reliable and accurate? 1.1. What are the components/variables used in the model to predict/forecast rainfall? And where do they come from? Primary Data Collection: KIIs, Small Group, and FGDs with all relevant stakeholders (i.e., WN, Tool developer, experts, regional and village governance, field facilitator, local partners of WN, DRR forums) Secondary Data Collection: Project documents, rainfall data series 2006-2022 - The accuracy of prediction is evaluated using three indicators, namely percent of bias (PBIAS), root-mean square-error (RMSE), and coefficient of determination (R2) - Using Method Triangulation, Source triangulation, and Evaluator triangulation - Descriptive Analysis & Causal Attribution 1.2. What types of rainfall characteristics are being forecasted (season onset, duration, and intensity for a season? What are the lead times of the forecast - how much in advance the forecast be made)? 1.3. What are the methods used for rainfall forecasting? Statistical, dynamic, or statistical-dynamic model? Ensemble models? 1.4. What are the assumptions used in the model? 1.5. What is the area covered by the forecast tool? Is it only for specific sites (districts)? What is the resolution? 1.6. What is the period of data used for developing the model, as well as calibration/validation and verification of the model? And the time period used for updating the forecast? 1.7. What are methods used to evaluate the accuracy of the forecast? 1.8. What criteria are used to define that the forecast is accurate? 1.9. Compared to actual rainfall, have past forecasts been accurate? Is the rainfall prediction for the first year more accurate than for the second, third, and fourth year? Is the accuracy of the forecast conducted on a point base or spatial base? 1.10. Are there other forecast models in use by BKMG that could be modified to produce similar forecasts? And if so, what is restricting their adoption? 2. What are the utilities of the Rainfall Prediction Model/Tool for farmers? 2.1. How the climate affects farmers’ cropping system? (e.g. change in season onset/false rain, season break, extreme rainfall, etc.) Primary Data Collection: Primary Data Collection: KIIs, Small Group, and FGDs with all relevant stakeholders (i.e., WN, regional and village governance, field facilitator, local partners of WN, DRR forums, KMBP, farmer groups) Secondary Data Collection: Project documents - Using Method Triangulation, Source triangulation, and Evaluator triangulation - Descriptive Analysis & Causal Attribution 2.2. How are the results of the forecast turned into advice to farmers? What is the advice? 2.3. What percentage of farmers are using the tool? If it is low, why? Do they view it as a useful tool (provide examples), and why? 2.4. How is the advice used by the farmers for making planting decisions or other planting operations (provide the example and reason ~ e.g. adjust technology like delay/accelerate planting, change planting arrangement, irrigation scheduling, applying pesticide/herbicide; change commodity; etc.)? 2.5. For farmers who used the rainfall prediction advice, what difference did it make in that seasonal production? 2.6. From the farmers who adopt/follow the rainfall prediction, cropping pattern, and other guidance, what is the percentage increase in food crop production and/or income? 2.7. How to differentiate the increase in food production due to changes in or adopted improved farming practices and due to the adoption of forecasts? USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 66 Primary EQ Sub-questions Data Collection Methods Data Analysis Approach 3. By comparing the traditional system and online application as means of information dissemination, which system is more effective for farmers? And why? 3.1.1 What are the traditional and online systems used to disseminate the information to farmers? How do the farmers access the information on the rainfall prediction and cropping pattern guidance? Is the information regularly shared with farmers? When? Did all farmers access or receive the information? If not, why? Primary Data Collection: KIIs, Small Group, and FGDs with all relevant stakeholders (i.e., WN, regional and village governance, field facilitator, local partners of WN, DRR forums, KMBP, farmer groups) Secondary Data Collection: Project documents - Using Method Triangulation, Source triangulation, and Evaluator triangulation - Descriptive Analysis & Causal Attribution 3.1.2 What are the main sources of information that farmers use to guide decision-making each season? What is information that farmers use to guide decision-making each season? How do farmers make decisions in each season from the provided information? 3.1.3 Do farmers understand and adopt the provided advice (for both in-person and online advice)? If farmers do not understand and adopt the advice, what are the reasons? What are the differences between farmers receiving rainfall prediction advice from in- person gatherings and those using the online application? 3.1.4 Has the investment in developing a mobile app been worthwhile? What difference has that made compared to places with no app. support? 4. In what way has the tool been utilized for Disaster Preparedness ? 4.1 What is the awareness level of stakeholders on the tool? Primary Data Collection: KIIs, Small Group, and FGDs with all relevant stakeholders (i.e., WN, regional and village governance, field facilitator, local partners of WN, DRR forums, KMBP) Secondary Data Collection: Project documents - Using Method Triangulation, Source triangulation, and Evaluator triangulation - Descriptive Analysis & Causal Attribution 4.2 Do they use the tool in disaster prediction? And how is it used? What is the form of disaster information provided based on the tool? 4.3 Are there other sources of disaster information used in addition to the tool to state the level of emergency? If not, what are the reasons? 4.4 How is the rainfall forecast/projection translated to climate risk/disaster impact outlook? And by whom? 4.5 How accurate/precise is the disaster prediction with the reality on the ground? Is the disaster prediction for the first year more accurate than for the second, third, and fourth? Why? 4.6 Do the result of rainfall predictions causing disasters facilitate the coordination of related agencies to prepare for disaster management? 4.7 How is the disaster risk information disseminated and communicated broadly? 4.8 Who are the government agencies/stakeholders/community groups that use it? What are the benefits of the disaster prediction for the community, government agencies/stakeholders? (future benefit vs. current benefit) 4.9 What policy needs to be in place to increase the use of climate forecast information for disaster preparedness? 67 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV ANNEX V: SOURCES OF INFORMATION LIST OF DOCUMENTS AND DATA SOURCES The USAID, MEL-P, and WN have shared several documents to be reviewed by the evaluation team. These documents were used to develop the evaluation design, instruments, list of targets, and analysis. 1. WN-BHA Indonesia Final Report - Oct 2018-Dec 2021 2. WN Final Report - Aug 2014-Sept 2018 3. Multi-hazard Early Warning Systems: A Checklist - May 2017 4. A New Way to Read the Weather: Rainfall Prediction for Improved Agriculture and Disaster Preparedness - March 2019 5. Project evaluation report on strengthening the disaster risk reduction capacity of local governments and communities in Nusa Tenggara for 2018-2021world neighbors 6. Data of rainfall BMKG in Central and West Lombok 2006-2018 7. Rainfall forecast PCH maps Central and West Lombok 8. Field study data for verification of Central and West Lombok PCH modeling results 9. Verification and validation reports on the results of rainfall predictions 10. Website/application of forecast rainfall in three districts 11. List of Names of PCH Access Villages in Nusa Tenggara 12. Climate change adaptation and disaster risk reduction exercise module 13. Documents of Desa Tangguh Bencana (Disaster resilience village): Indicator matrix, village scores 14. Rainfall data from BMKG of NTB 15. Rainfall data from CHIRPS REFERENCES Risbey, J.S., Squire, D.T., Black, A.S. et al. Standard assessments of climate forecast skill can be misleading. Nat Commun 12, 4346 (2021). https://doi.org/10.1038/s41467-021-23771-z Aldrian, E., & Djamil, Y. S. (2008). Spatio-temporal climatic change of rainfall in East Java Indonesia. International Journal of Climatology, 28(4), 435–448. doi:10.1002/joc.1543 Susilokarti, D, SS Arif, S Susanto, L Sutiarso. 2015. Comparation Study on Precipitation Prediction Using Fast Fourier Transformation (FFT), Autoregressive Integrated Moving Average (ARIMA) and Artificial Neural Network (ANN). Agritech 35 (2):241-247. Moriasi, D. N., Gitau, M. W., Pai, N., & Daggupati, P. (2015). Hydrologic and water quality models: Performance measures and evaluation criteria. Transactions of the ASABE, 58(6), 1763-1785. Nerini, D., Besic, N., Sideris, I., Germann, U., and Foresti, L.: A non-stationary stochastic ensemble generator for radar rainfall fields based on the short-space Fourier transform, Hydrol. Earth Syst. Sci., 21, 2777–2797, https://doi.org/10.5194/hess-21-2777-2017, 2017 USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 68 LIST OF KEY INFORMANTS TYPE OF KI POSITION / INSTITUTION GENDER FEMALE MALE Tool Developer Developer, ITB - 1 Implementing Partner Program officer, World Neighbors - 2 Program Manager, World Neighbors - 1 National Expert Statistics expert, ITS - 1 Balitklimatologi, Kementan 1 - Diseminasi Informasi Iklim dan Kualitas Udara, BMKG - 1 Analisis Variabilitas Iklim, BMKG - 1 Geophysics and Meteorology, IPB - 1 BMKG - 1 National Research and Innovation Agency (BRIN) - 3 National Government BMKG - 5 Local Government (Provincial) Information Service Coordinator, NTT Climatology Station 1 - Assessment Institute for Agriculture Technology (BPTP) Naibonat, NTT - 1 Head of Food Security & Extension, Distan NTT - 1 Meteorology Station Ruteng, NTT - 1 Meterology Station II Zainuddin Abdul Madjid, Lombok, NTT - 1 Climatology Station, Lombok Barat, NTT - 1 Local Government (District) Head of Government & Human Development, Bappeda Central Lombok - 1 Young Functional Officer, Bappeda Central Lombok - 2 BPBD Central Lombok - 1 Secretary Distan, Central Lombok - 1 Kabid Perencanaan, Distan Central Lombok - 1 PPL Village of Tumpak and Waja Geseng, Distan Central Lombok - 2 Head of Distan, Dompu - 1 PPL Distan, Dompu 2 2 Bappeda, Dompu - 4 Head of DLH Dompu (ex-head of BPBD Dompu) - 1 Distan Nagekeo 2 4 BPBD Nagekeo 1 3 BPP Gerung & Kuripan sub-district, Distan West Lombok 1 1 Head of UPT Agriculture Kec. Gerung, Distan West Lombok - 1 Head of Prevention & Preparedness, BPBD West Lombok - 1 Secretary Distan, East Lombok - 1 PPL Coordinato,r Distan East Lombok - 1 PPL Distan East Lombok - 2 BPBD East Lombok - 1 Local Government (Village) Secretary Tiwugalih Village, Central Lombok - 1 Leader KMPB Tiwugalih Village, Central Lombok - 1 Member KMPB Tiwugalih Village, Central Lombok - 3 Head Nusa Jaya Village, Central Lombok - 1 69 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV TYPE OF KI POSITION / INSTITUTION GENDER FEMALE MALE Secretary Nusa Jaya Village, Central Lombok - 1 Staff Nusa Jaya Village, Central Lombok - 1 KMPB Nusa Jaya Village, Central Lombok - 2 Head Pekat Village,t Dompu - 1 Staff Pekat Village, Dompu - 1 Leader KMPB Pekat Village, Dompu - 1 Member KMPB Pekat Village, Dompu - 2 Head Wajomara Village, Nagekeo - 1 Secretary Wajomara Village, Nagekeo - 1 Staff Wajomara Village, Nagekeo - 1 Leader KMPB Wajomara Village, Nagekeo - 1 KMPB Wajomara Village, Nagekeo - 2 Secretary Pagomogo Village, Nagekeo - 1 Staff Pagomogo Village, Nagekeo 1 1 Coordinator KMPB Pagomogo Village, Nagekeo - 1 Head Cendi Manik Village, West Lombok - 1 Secretary Cendi Manik Village, West Lombok - 1 Leader KMPB Cendi Manik Village, West Lombok - 1 Member KMPB Cendi Manik Village, West Lombok - 1 Secretary Taman Baru Village, West Lombok - 1 Staff Taman Baru Village, West Lombok - 1 Leader KMPB Taman Baru Village, West Lombok - 1 Member KMPB Taman Baru Village, West Lombok - 1 Head Giri Sasak Village, West Lombok - 1 Secretary Giri Sasak Village, West Lombok - 1 Staff Giri Sasak Village, West Lombok - 1 Leader KMPB Giri Sasak Village, West Lombok - 1 Head Gunung Malang Village, East Lombok - 1 Secretary Gunung Malang Village, East Lombok - 1 KMPB Village of Gunung Malang East Lombok - 1 Secretary Puncak Jeringo Village, East Lombok - 1 KMPB Puncak Jeringo Village, East Lombok - 4 Head Padak Guar Village, East Lombok - 1 Secretary Padak Guar Village, East Lombok - 1 Leader KMPB Padak Guar Village, East Lombok - 1 Member KMPB Padak Guar Village, East Lombok - 1 Local Partner Director, Berugak Dese - 1 Treasurer, Berugak Dese 1 - Field Supervisor, Berugak Dese - 2 Field Staff, Berugak Dese - 2 Coordinator, Berugak Dese - 1 Director, LESPEL - 1 USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 70 TYPE OF KI POSITION / INSTITUTION GENDER FEMALE MALE Coordinator, LESPEL - 1 Field Staff, LESPEL - 2 Treasurer LESPEL 1 - Director, PSN - 1 Coordinator, PSN - 1 Supervisor, PSN - 1 Field staff, PSN - 3 Coordinator, YMTM - 1 Supervisor, YMTM - 1 Supervisor Field, YMTM - 1 Field Staff, YMTM - 2 Director, LPSDM 1 - Coordinator, LPSDM - 1 Field Staff, LPSDM 1 4 FPRB FPRB Central Lombok - 1 Leader FPRB Nagekeo - 1 Member FPRB Nagekeo 1 1 Leader FPRB West Lombok - 1 Member FPRB West Lombok - 1 Leader FPRB East Lombok - 1 Treasurer FPRB East Lombok 1 - Farmers Farmer Group Tiwugalih Village, Central Lombok - 4 Farmer Group Semayan Village, Central Lombok 1 2 Farmer Group Mekar Sari Village, Dompu - 6 Farmer Group Nowa Village, Dompu 2 3 Farmer Group Pekat Village, Dompu 2 3 Farmer Group Lembar Selatan Village, West Lombok - 2 Farmer Group Padak Guar Village, East Lombok - 2 Farmer Group Surabaya Village, East Lombok - 1 Farmer Group Mekar Baru, Wajomara Village, Nagekeo 2 - Farmer Group Pagomogo Village, Nagekeo 1 - Frmer Group Ngusokerja, Nangaroro Village, Nagekeo - 2 Farmer Group Cendi Manik Village, West Lombok - 3 Farmer Group Taman Baru Village, West Lombok - 2 Farmer Group Giri Sasak Village, West Lombok - 3 Farmer Group Gunung Malang Village, East Lombok - 10 Farmer Group Puncak Jeringo Village, East Lombok - 2 Farmer Group Padak Guar Village, East Lombok - 2 # KIs by Gender 23 180 Total Key Informants 203 71 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 72 73 | THE RAINFALL PREDICTION TOOLS EVALUATION REPORT USAID.GOV USAID.GOV THE RAINFALL PREDICTION TOOLS EVALUATION REPORT | 74 ANNEX VII: EVALUATION TEAM MEMBERS A three-person team carried out the evaluation activities. The MEL-P team provided additional support related to technical guidance, logistics, and quality oversight of data collection, analysis, and reporting. The specific qualifications and roles for each evaluation team member are listed below. Mr. Rizaldi Boer serves as the Team Leader. He is responsible for overall direction, writing, analysis, and interface with USAID and WN. Rizaldi is a Professor at the IPB University, Indonesia, and currently Executive Director of the Centre for Climate Risk and Opportunity Management in Southeast Asia and Pacific of IPB University. He received a doctoral degree from the University of Sydney, Australia. He has been working on climate change mitigation and adaptation, particularly on agriculture, forest, and other land use, for over 20 years. He is involved in many international scientific teams, national research missions, and several scientific projects of the United Nations and high-level global players (specifically on REDD, with the UNFCCC, SDSN, IDDRI, DANIDA, IGES, JICA, NIES, etc.). He had been appointed as Chairperson of the RA-V Region Agriculture Meteorology Working Group (2002-2009) for WMO (World Meteorological Organization) and a member of the Task Force Bureau for the IPCC Greenhouse Gas Inventory (2008-2015) and now serves as Chairman of the Expert Board of the Indonesian Agriculture Meteorology Society, as well as Member of the Advisory Board for the Asian Greenhouse Gas Inventory Working Group (WGIA), and the Asian Low Carbon Research Network (LoCARNet). He has extensive experience in climate modeling and evaluation of climate-related projects. Mr. Muh Taufik serves as the Agriculture and Climate Expert for the evaluation team. Equipped with a Ph.D. in Hydrology from Wageningen University, Netherlands, and over ten years of experience, Taufik has worked as a senior lecturer and researcher. He has participated in several research and studies in Indonesia, mainly in climate-related disasters. He has experience in modeling and worked with many climate-related hazard/disaster projects in Indonesia, including BRG and World Resource Institute (WRI), to develop an early warning system for peat fire based on a weather forecasting model. Miss Resti Salmayenti serves as the Research Assistant for the evaluation team. Equipped with an MSc in Environmental Sciences from Wageningen University and about 5 years of experience, Resti is a researcher & lecturer at the Geophysics and Meteorology Department, IPB University. Her interdisciplinary academic and applied work focuses on applied climatology, ecosystem services, environmental economics, and valuation. U.S. Agency for International Development / Indonesia Jl. Medan Merdeka Selatan 3-5 DKI Jakarta, Indonesia