*Ff Tonga Community Disaster Risk Management (TCDRM) Program End of Program Evaluation Report ` PREPARED BY PACIFIC RESEARCH & EVALUATION ASSOCIATION PTY LTD ATF PREA TRUST 90 PARK ROAD ELTHAM VIC 3095 AUSTRALIA www.prea.com.au ACN: 605 751 899 ABN: 138 349 971 00 CORRESPONDENCE RELATED TO THIS REPORT SHOULD BE ADDRESSED TO: NAME PHONE EMAIL Acknowledgements: PREA would like to thank the following people and organisations for their contributions to planning, supporting and informing this evaluation: Act For Peace (Yvette Crafti, Geoff Robinson and Meg Laufer), Tonga National Council of Churches (Siu'ivahamama'o Fangupo, Mele Moimoi, Asena Taufa, Collin Mulikiha’amea Siale, Ma’u Kakala Halaifonua), TNCC Board members, Town Officers, VEMC members, community members, representatives of the Tongan Government across various departments, and representatives from civil society organisations. Without their help this evaluation would not be possible. Special thanks to PREA associate Jodie Kane for peer reviewing the evaluation framework, data collection tools and draft report. Front cover photos: Improved Matamaka Evacuation Centre with VEMC members; Fotua Town Officer with VEMP in front of TCDRM supplied water tank; VEMC deputy chairperson from Matafonua. Photo credits: Martin Pritchard, PREA. VERSION CONTROL VERSION STATUS DATE 1.0 Initial draft 18/04/2019 1.1 Peer review from Jodie 19/04/2019 1.2 1.3 Added Exec summary and conclusion. Minor improvements Peer review from Jodie (Exec Summary, conclusion and recommendations 23/4/2019 26/4/2019 1.4 Integrate OFDA input and Jodie’s review of Exec Summary and recommendations. Integrated feedback from AFP Meg Laufer. 1/5/2019 1.5 Integrate feedback from AFP and TNCC. 26/5/2019 TABLE OF CONTENTS 1 EXECUTIVE SUMMARY................................................................................................................................... 1 2 INTRODUCTION ............................................................................................................................................. 7 3 EVALUATION METHODOLOGY....................................................................................................................... 8 3.1 THEORY OF CHANGE.................................................................................................................................. 10 4 FINDINGS..................................................................................................................................................... 11 4.1 EFFECTIVENESS - TO WHAT EXTENT WERE THE PROJECT OBJECTIVES ACHIEVED? ........................................................ 11 4.2 WHAT WERE THE MAJOR FACTORS INFLUENCING THE ACHIEVEMENT OR NON-ACHIEVEMENT OF THE OBJECTIVES? ............. 23 4.3 WHAT IS THE POTENTIAL FOR REPLICATION OF THE CBDRM APPROACH? ................................................................ 29 4.4 TO WHAT EXTENT HAS ACHIEVEMENT OF PROJECT OBJECTIVES BEEN EQUITABLE (CONSIDERING GENDER, DISABILITY AND MARGINALISATION/ REMOTENESS)? .......................................................................................................................... 31 4.5 WHAT UNEXPECTED POSITIVE OR NEGATIVE OUTCOMES HAVE BEEN IDENTIFIED? ...................................................... 34 4.6 HOW EFFECTIVE HAS THE TCDRM PROGRAM BEEN IN MEETING THE NEEDS OF COMMUNITIES? ........................... 35 4.7 WERE APPROPRIATE TECHNICAL STANDARDS USED AND ACHIEVED? ....................................................................... 36 4.8 TO WHAT EXTENT DID THE PROGRAM IMPLEMENT RECOMMENDATIONS FROM THE MID-TERM EVALUATION IN YEARS 4 AND 5 OF THE PROGRAM ................................................................................................................................................. 37 4.9 TO WHAT EXTENT DID THE PROGRAM VALIDATE THE FINDINGS FROM THE MID-TERM REVIEW? ..................................... 38 4.10 HOW LIKELY ARE PROJECT ACHIEVEMENTS TO BE SUSTAINED AFTER THE END OF THE PROGRAM? ................................... 38 5 CONCLUSION ............................................................................................................................................... 41 6 FULL RECOMMENDATIONS .......................................................................................................................... 42 7 ANNEX ......................................................................................................................................................... 47 7.1 PROGRAM ACTIVITIES ................................................................................................................................ 47 7.2 TERMS OF REFERENCE FOR EVALUATION ......................................................................................................... 48 7.3 SOURCES/BIBLIOGRAPHY ............................................................................................................................ 55 7.4 DATA COLLECTION TOOLS ........................................................................................................................... 56 7.5 TARGET COMMUNITIES AND POPULATIONS...................................................................................................... 57 7.6 SAMPLING METHODOLOGY ......................................................................................................................... 61 7.7 LIST OF EVALUATION ACTIVITIES IN COMMUNITIES ............................................................................................. 63 7.8 LIST OF EVALUATION INFORMANTS ................................................................................................................ 64 7.9 TIMETABLE ............................................................................................................................................. 72 7.10 THEORY OF CHANGE.................................................................................................................................. 75 7.11 WATER STORAGE VOLUME INCREASES............................................................................................................ 78 7.12 RELATIONSHIP STRENGTH ASSESSMENT .......................................................................................................... 80 7.13 ASSESSMENT OF CORE HUMANITARIAN STANDARD COMMITMENTS ...................................................................... 81 7.14 ASSESSMENT OF MTE RECOMMENDATIONS .................................................................................................... 84 7.15 VALIDITY OF MTE FINDINGS........................................................................................................................ 96 7.16 LADDER OF PARTICIPATION SUMMARY ......................................................................................................... 101 7.17 EVALUATION FRAMEWORK ....................................................................................................................... 103 7.18 COMMUNITY CONSULTATION SNAPSHOTS ..................................................................................................... 116 7.19 SAMPLE OF PHOTOGRAPHIC EVIDENCE ......................................................................................................... 120 Acronyms / Abbreviations AfP Act for Peace CbCERT Community based Emergency Response Training of Trainers CbDRM Community based Disaster Risk Management CbP Community based Protection CCA Climate Change Adaptation CHS Core Humanitarian Standards CSO Civil Society Organizations DO District Officer DPO Disability People’s Organisation DRM Disaster Risk Management DRR Disaster Risk Reduction EC Evacuation Centre GoT Government of Tonga JNAP Joint National Adaptation Plan on Climate Change Adaptation and DRM (Tonga) M A F F Ministry of Agriculture Food and Forests M&E Monitoring and Evaluation NEMO National Emergency Management Office (Tonga) NGO Non‐Government Organisation OFDA Office of Foreign Disaster Assistance PCCCRR Pacific Community Climate Change Risk Reduction Program PCIDRR Pacific Community Focused Integrated DRR Project PWD People with Disabilities SIDS Small Island Developing States TC Tropical Cyclone TCDRM Tonga Community Disaster Risk Management project TNCC Tonga National Council of Churches TNQAB Tonga National Quality Assurance Board TO Town Officer ToT Training of Trainers TRCS Tonga Red Cross Society USAID United States Agency for International Development VEMC Village Emergency Management Committees VEMP Village Emergency Management Plans 1 1 Executive Summary The five-year Tonga Community Disaster Risk Management (TCDRM) Program funded by USAID and implemented by the Tonga National Council of Churches (TNCC) in partnership with Act For Peace (AFP), aimed to reduce the impact of natural disasters on 51 vulnerable communities in Tonga. The Program worked with communities through Government endorsed structures and processes, to create Village Emergency Management Committees (VEMCs) and Village Emergency Management Plans (VEMPs). The VEMCs were trained in a number of fields including Disaster Risk Management (DRM), food and water security, protection from abuse, proposal writing and first aid. Simulation activities (SIMEX) and VEMC outreach helped share new DRM and first aid knowledge and skills with the broader community. Small community water security, food security or Evacuation Centre (EC) improvement projects were funded and implemented in most of the targeted communities. The Program also sought to build the capacity of national and sub-national Government actors and Civil Society Organisations (CSO) involved in DRM. The Program aimed to be inclusive in terms of its geographical community, targeting -gender and age of participants, and inclusive of people with disabilities. A summary of key program outputs is presented below before program outcomes are summarised. Program beneficiaries 15,472 community members (7,794 men, 7678 women) over 51 communities VEMCs established 51 VEMPs created 51 Capacity building workshops 83 DRM, WASH, & Food Security workshops 51 Simulations 6,441 participants (2,968 men, 3,473 women, 73 people with disabilities Food security 4 community fences 7 Livestock fences 18 chicken fences 2 Greenhouses 2 Sets of garden tools 53 Sheep^ Water security Additional water storage capacity 1,465 Kilolitres 2 Water pumps distributed Average additional days water supply in emergency, 21 days or drought, 43 days. Evacuation Centre improvements 5 Evacuation Centres improved This end of program evaluation finds that overall, the program made good progress in ultimately achieving its four high level objectives. Objective 1. To support the Government of Tonga to reduce the impact of natural disasters on communities, especially the vulnerable. The future impact of natural disasters on targeted vulnerable communities was reduced by the Program. The Program established or strengthened 51 VEMCs and developed or revised 51 VEMPs. VEMC capacity was built in Disaster Risk Management (DRM), First Aid, and knowledge of water and food security. The evaluation found strong evidence confirming new knowledge and skills had been retained. Reports from disaster simulation exercises and reflections on TC-Gita demonstrated both VEMC members and the general community applied knowledge and skills which enabled them to be better prepared for disaster events. VEMPs were actioned by VEMCs in response to Early Warning System (EWS) emergency notifications (SMS, phone, radio) from Government of Tonga (GoT) stakeholders, however, some Town 2 Officers relied on knowledge from their past lived experience in cyclone events, instead of following the VEMP. Preparedness measures taken by VEMCs in Tongatapu and ‘Eua (TC-Gita) and Vava’u (TC￾Winston) included notifying the community about the warning, encouraging people to take preparedness action and offering assistance to those who need it to prepare their homes and evacuate to the EC. Community members now take warnings more seriously and they now take early action to protect their homes, water sources, bush gardens, cultural artefacts, and prepare a 72 hour bag with food, water and other essential items.1 This improved behaviour reflects an increased understanding of EWS notifications and the importance of being prepared. The TCDRM program combined with improved DRM information communicated via radio are likely contributing factors for this change. Some community members demonstrated improved behaviour by evacuating to the EC before TC-Gita impacted the community and damaged homes. During TC-Gita, VEMC members performed head counts, helped to manage the EC and make sure people did not return home during the quiet eye of the cyclone. VEMC and community youth led search and recovery operations where it was deemed safe. After the emergency, VEMC and the broader community then set about cleaning up and offering first aid and assistance to those in need. The Town Officer supported by the VEMC, conducted the initial damage assessments and reported up to the District level. VEMCs demonstrated improved knowledge of the vertical hierarchy of communication between communities and GoT departments. Whilst community members already possessed a degree of DRM knowledge through past experience, they reported that the Program reinforced this existing knowledge and importance of taking warnings seriously and being better prepared. The simulations activities were a key component of knowledge sharing and creating new behaviours such as preparing a 72 hour bag and ensuring people with disabilities and other vulnerable groups received additional support to prepare and/or evacuate. The program strengthened the capacity of sub-national Government stakeholders such as district-level Government Representatives, District Officers and Town Officers. These stakeholders benefited from targeted training which made them more aware of their roles and responsibilities as outlined in the National Disaster Plan. CSOs also benefited through attending DRM related Training of Trainer (TOT) workshops. The Program provided GoT departments and CSOs the opportunity to give presentations to communities. This was highly valued by GoT and CSO stakeholders as they normally lack the funds to engage at the community level. This increased community knowledge on key stakeholder DRM roles and responsibilities. In 2015, the Program engaged community members, CSOs and the private sector to support NEMO in the drafting of a National Community based Disaster Risk Management (CbDRM) Framework. No stakeholders consulted recalled their involvement in the Framework which made it difficult for the evaluation to determine the extent to which their feedback was incorporated. The Framework is still in a draft status, however, AFP indicated it had still been useful in guiding the partnership between NEMO and TNCC. Whilst all these exchanges helped improve the short-term relationship between communities and GoT stakeholders, there was no evidence to support that these relationships had grown or continued beyond the engagement created by the Program. 1 72 hour bag contains important necessities (food, water, torch, important documents, food, clothes, etc.) to support 3 days of living in an emergency situation and to assist with recovery. 3 Objective 2. To strengthen the capacity of local communities to identify and assess their own protection risks and to develop strategies and action to prevent, mitigate and respond to these risks. Seven of the ten VEMCs engaged by the evaluation were found to have heightened awareness of personal protection issues as a direct result of TCDRM activities. The importance of keeping women, young girls and other vulnerable groups safe from abuse in a mixed gender environment at the EC was recognised by most VEMCs. VEMCs were able to recite protection response measures, however, these focused only EC management and did not include any measures to prevent abuse in the disaster response situation where relief supplies are being distributed by external assistance. VEMC members were assigned roles of male and female protection officers, however, no VEMCs consulted were able to identify who their protection officers were without referring to the VEMP. This indicates a lack of retention and awareness about the documented VEMC roles. The perceived relevance of protection issues in small communities was observed to be low in some instances. In many communities nearly everyone is related and they all know each other. This creates a sense of safety and trust which may lessen the perceived importance of protection. The evaluation was unable to make a conclusive finding on the capability of VEMCs to identify new protection risks independently without external assistance. VEMCs will likely need assistance to identify and respond to existing and potential emerging protection risks. Objective 3. To improve the community level food security of participating communities. Whilst baseline and end line food production data was not available, the evaluation found evidence that food security has been enhanced in target communities through food security projects, food security workshops and the inclusion of food security response measures in VEMPs. Seventeen communities implemented practical measures to improve food security through TCDRM funded community food security projects. Technical solutions implemented included greenhouses in four communities to raise vegetable and cash crop seedlings for distribution to the community. Twelve fencing projects kept livestock in paddocks for easy management or kept feral animals out of food gardens to reduce production loses. Fencing combined with food security training encouraged communities to plant more food gardens with climate and disaster resilient crops. AFP supplied additional funding to implement complementary food security measures. These funds supported the distribution of 53 climate adapted sheep to communities which were bred and increasing in numbers before distribution to households for ongoing breeding and consumption. Objective 4. To expand water coverage in targeted settlements to improve access to safe and sufficient water, both in the immediate and longer term. TCDRM community projects have directly increased short and long-term access to safe potable water for 35 communities. WASH training conducted in years 4 and 5 has also highlighted the importance of water conservation and good hygiene practices. An additional 1,465 Kilolitres (KL) of community water storage was added as a result of the program. The additional water storage capacity will extend community access to water during times of drought (43 days) and emergency (21 days). The effectiveness of the new rainwater tanks was enhanced through best practice design considerations and oversight provided by a WASH engineer. The positive findings regarding achievement of the Program’s four key objectives were supported by a number of enabling factors including the multiple engagement strategies, multiple touch points over 4 multiple years, and effective delivery at the community level by competent, knowledgeable, committed Program staff with excellent facilitation skills. TNCC’s capacity to manage and implement the Program was strengthened by AFP’s Partnership Approach which facilitated both one-on-one coaching and Training of Trainer workshops in core DRM modules for both Program staff and GoT and CSO stakeholders. The association of TNCC with key church groups in Tonga combined with best practices such as pre-program community consultations, assisted the Program team to build trust with communities. Support by national and sub-national Government departments and CSOs also helped bring credibility to the Program. The evaluation identified the Town Officer a key enabling or inhibiting factor to the effectiveness of the Program. The Town Officer has strong authority and respect in both community committees and the VEMC. If they are not active and supportive of the Program then community engagement in activities is not likely to be high and VEMCs are not likely to be active, effective or sustained. Conversely, an active and supportive Town Officer has the potential to engage and bring the entire community together to participate in the Program. They can lead the VEMC and also delegate roles and responsibilities to VEMC members as outlined in the VEMP. Replication The evaluation finds that there is strong potential for the replication of CbDRM in other communities and other countries. This finding is premised on the effectiveness of the TCDRM program and the need to provide similar programming to other remote and vulnerable communities in Tonga. Prior country and community needs/situation analysis is needed to inform the need CbDRM and customisation of the CbDRM approach. A number of characteristics such as the frequency and severity of natural disasters in new target communities were identified as having influence on the likely effectiveness of the CbDRM approach if replicated in other countries. Equity The program successfully engaged men, women, elderly and people with disabilities. Overall, there was a near equal number of men and women benefiting from the Program. VEMCs included fairly equal representation of men and women, however, women’s actual level of participation and power within the VEMC was relatively low and this reflects current Tongan gender power relations. People with disabilities participated in Program activities such as disaster simulations, however, there was no evidence found of their representation on VEMCs or the use of a strengths-based approach to their engagement in other program activities or community projects. Ambitions for high and deeper participation of people with disabilities need to be appropriately resourced to implement all the inclusive actions set out in the program design. Whilst youth members were documented as members in VEMCs and male youth were noted as being active in DRM preparedness activities, their overall level of engagement with the program was notably lower than other groups. Community projects demonstrated a commitment to equitably engage women, men, people with a disability and the elderly, however, highly gendered roles mostly saw men carry out the work and women prepare food, and people with disabilities likely observing or not taking an active role in the projects. Measures to improve inclusiveness are identified in the body of the report. 5 Relevance The Program, including outcomes achieved by community projects, were all regarded as highly relevant by community members and GoT stakeholders. Tonga’s high exposure to natural hazards combined with vulnerable remote communities makes risk reduction and improving resilience to disasters highly relevant. Water and food security measures implemented were judged highly relevant and appreciated in remote areas with large numbers of the population living subsistence lifestyles. The improvements made to ECs were important in proving a safe and accessible place for community members to evacuate to. The utilisation of the CbDRM approach to deliver program outcomes was also found to be highly relevant. The approach created or strengthened government legislated community-level governance structures (VEMC) and plans (VEMP), and used multiple activities to encourage positive DRM behaviours in communities. Technical standards The Program used and adhered to all relevant technical standards and this improved the quality of program implementation and outputs. The involvement of a WASH and construction engineer helped ensure water harvesting projects adhered to standards and best practices. The construction engineer ensured that ECs were structurally sound before improvements were authorised. The Program aimed to adhere to six commitments from the Core Humanitarian Standards (CHS). The evaluation found strong evidence of conformity to four of the six commitments and a medium degree of evidence to support implementation of the remaining two commitments. CHS commitments and supporting evidence are discussed in more detail in the body of the report. Continuous improvement The Program successfully implemented in full or in part two thirds of the 31 recommendations from the Mid-term Evaluation (MTE) and this commitment to reflection and continual improvement was observed throughout the program. Four recommendations not implemented fell outside the scope/control of TNCC to action and the evaluation deems that TNCC lacked the additional resources to implement at least one other recommendation. Validity of the MTE findings Overall this evaluation determined that the findings of the MTE to be valid and still mostly relevant at the stage of the end-of-program evaluation. Sustainability Benefits achieved from community WASH, food security and EC improvement projects demonstrate a medium to high degrees of ownership and sustainability. The strong community support for these improvements combined with the quality of the technical solutions implemented (e.g. low maintenance, community appropriate technology rainwater harvesting) should see the benefits sustained for up to 20 years. Whist DRR/DRM knowledge is likely to be retained into the future, the active functioning of VEMCs and their application and VEMPs is less assured unless there is ongoing support from NEMO. The competency and support of the Town Officer is a key determining factor in sustaining the new structures and behaviours. The absence of a hand-over and induction process for new Town Officers to better understand their DRM/VEMC role and responsibilities and be familiar with CbDRM and the VEMP, is a high risk to the sustainability of VEMCs the DRM benefits delivered by the Program. 6 The Program’s use of nationally endorsed structures processes (VEMCs, VEMPs) and close working relationship with NEMO are foundations that support the potential for Program outcomes to be sustained. The Program made an official handover of CbDRM to NEMO in 2019 and NEMO expressed support for the CbDRM approach, however, delays in the finalisation and endorsement of the CbDRM Framework do not support this commitment. Tentative plans for NEMO to lead the delivery of the CbDRM approach in Tonga’s most remote islands is a more positive sign of support and ownership. Overall, NEMO noted that it lacked the resources to reach out and engage with more communities and support existing beneficiary communities. NEMO expressed a strong desire for the TCDRM program to continue. Support and resourcing from MIA and NEMO will be required to encourage existing VEMCs to meet, share knowledge with the community, run simulations and update VEMPs on an annual basis. This support task could be built into the annual workplans for NEMO district officers in advance of cyclone season. High priority recommendations The evaluation makes the following five high priority recommendations to strengthen and improve the TCDRM Program. These recommendations and others are explained in further detail in the Full Recommendations section of the report. Priority recommendations Allocated stakeholder Use DRM focused drama performances to entertain and draw the bulk of community members to attend program activities. Performances would highlight key program messages. TNCC Build TNCC capacity in the use of behaviour change models, frameworks and tools to improve behaviour change outcomes at the community level. TNCC, AFP Embed additional behaviour change strategies, frameworks and tools into the program design to improve behaviour change outcomes. TNCC TCDRM field officers should engage other community stakeholders such as VEMC members and church leaders, in addition to the Town Officer, to increase awareness of planned activities in the community and in turn, increase attendance at activities. TNCC Create or enhance existing induction processes for new Town Officers to ensure proper hand-over and awareness of DRM/VEMC roles responsibilities and VEMC contents. NEMO/MIA Seek longer term (3 to 5 year) funding commitment from donor to enable better strategic planning of community engagement and help retain skilled TNCC staff over longer durations. TNCC, USAID 7 2 Introduction The Tonga Community Disaster Risk Management (TCDRM) Program (‘the Program’) has been funded by USAID/OFDA since 2013 and is scheduled to end in 2019. The Program was funded for five years, however, a no cost time extension was granted to take into account the impacts and delays to Program activities caused by Tropical Cyclone Gita (TC-Gita). The Program was implemented by the Tonga National Council of Churches (TNCC) in partnership with Act For Peace (AFP). Program Goal To support the Government of Tonga to reduce the impact of natural disasters on communities, especially the vulnerable. Objectives 1. To support the Government of Tonga to reduce the impact of natural disasters on communities, especially the vulnerable.2 2. To strengthen the capacity of local communities to identify and assess their own protection risks and to develop strategies and action to prevent, mitigate and respond to these risks. 3. To improve the community level food security of participating communities. 4. To expand water coverage in targeted settlements to improve access to safe and sufficient water, both in the immediate and longer term. The objectives were progressed through three main strategies including: 1. Capacity Building and Training 2. Community Awareness and Mobilization 3. Policy and Planning The main Program activities included:  Vulnerability and Capacity Assessments (VCA)  Stakeholder consultations (National and community-level, including the feeding back of evaluation findings)  Community-based training workshops (Disaster Risk Management (CbDRM), Emergency Response Training (CbCERT) Protection (CbP), disability inclusion, food security, Water, and Sanitation and Hygiene (WASH) training, First Aid training,)  Establishment of Village Emergency Management Committees (VEMC) and support to create or review Village Emergency Management Plans (VEMP)s,  Disaster simulation exercises  Proposal writing workshops  Food security, water supply and evacuation centre strengthening projects supported by community grants program Gender equality, disability inclusiveness and child protection safeguards are integrated across all components of the program. Additional information about Program activities can be found in Annex 1. Community capacity building was usually delivered over two separate one-week blocks. Some communities were engaged by the Program over multiple years and received refresher training and additional capacity building. Pacific Research & Evaluation Associates (PREA) was commissioned by AFP and TNCC to undertake an end of program evaluation in the April 2019 (See Annex 2 for Terms of Reference). The main purpose of 2 The first objective overlaps with the goal and this is a result of changes to the scope included AFP’s Year 5 TCDRM proposal to OFDA. 8 the evaluation was to assess the effectiveness of the TCDRM methodology and the extent to which Program objectives have been achieved. Program relevance and sustainability will also be explored as secondary focus areas. 3 Evaluation Methodology The evaluation was guided by an Evaluation Plan that included an Evaluation Framework (Annex 17). The evaluation was informed by mostly qualitative methods including a desktop review (see Annex 3 for document reviewed), Focus Group Discussions (FGD), Community interviews and Key Informant Interviews (KII) and observations of community projects. A visual ‘ladder of participation’ tool was used to assess the extent of participation of women in VEMCs. A list of data collection tools is documented in Annex 4 and all tools and templates have been submitted to AFP and TNCC as a separate attachment. Consultations were held in ten of the 51 targeted communities with representation from Tongatapu, Vava’u, Ha’apai and ‘Eua. See Annex 5 for list of communities and table 1 for a summary of consultations. A stratified purposive sampling approach was used to choose the communities consulted. See Annex 6 for more details on sampling. Two additional communities that did not benefit from program activities were also consulted as a ‘control group’ to inform if there were any noticeable differences in how the prepared for and responded to disaster events. Table 1. Summary Community consultation data Number of people consulted Number held Men Women Target Community FGD 6 6 44 Target Community KII 21 12 9 Target Community VEMC FGD 10 21 34 Control community FGD 2 2 25 Control community KII 4 2 2 Totals 43 43 114 At the community-level, the evaluation was informed by 157 community and VEMC members (36 men, 93 women, six young men and six young women less than 30 years of age, and including four people with disabilities. See Figure 1). Women (69%) were overrepresented due to their greater availability during community visits. For example, over 30 women in women’s groups were represented in Tefisi (Vava’u control community) and 21 women were represented at ‘Ohonua because the Town Officer incorrectly announced that the consultation was for women only. An additional 13 men and 11 women representing national, sub-national and Civil Society Organisations (CSO) informed the evaluation. A summary of community consultations is provided below and more details can be found in Annex 7 with the list of people consulted in Annex 8). A timeline of evaluation activities is documented in Annex 9. Figure 1. Community informant gender representation 9 In most communities, FGD was conducted with community members and VEMCs. In some communities these discussion were held as a mixed group. In four communities, only VEMC members were present and community input was reliant on interviews conducted with community members. Approximately three community members were interviewed from each community. The sampling for interviews was mostly intentional and aimed to make up for the lack of representation of one or more groups at the FGD. In most communities, interviews were conducted with one man, one women, a youth member and where possible, a person with disability. In many instances, available time in communities and access to specific groups of people (e.g. youth or people with disabilities), limited the number of interviews conducted and the ability to target specific groups. See Annex 18 for community consultation snapshot entries that summarise some key findings from each community. FGD were often conducted with one mixed gender group instead of the planned gender segregation. The views of women were deliberately targeted in FGD to ensure their voices were heard. KII conducted outside of the FGD also enabled women’s voices to be heard in the evaluation. Intervention v’s control group comparison Interviews and focus groups were held with men and women from the two control group communities. They were tested on their level of DRM knowledge and skills and their degree of preparedness through DRM-related committees and plans. The responses provided by the control group communities were compared to those of the intervention sample to identify differences and to better understand if the TCDRM Program was responsible for those differences. As noted in the constraints section below, one control community was less reliable due to other non-Program DRM activities that were implemented by other stakeholders. Constraints and limitations The evaluation acknowledges that there were a number of constraints that limited the effectiveness of the evaluation. A summary of constraints includes:  The evaluation timeline was very tight for both preparation (one week) and report drafting (one week for first draft). The tight timeline may have reduced the quality of the evaluation in some areas, specifically presentation of findings (infographics) and developing stories and case studies to compliment the main body of the report.  Time and cost limitations that resulted in sampling of only ten of the 51 communities. Whilst sufficient, this is not a representative sample.  Disproportionate high representation of women due to their availability and choices made by Town Officers. The views of women may be amplified by the evaluation. In most cases there were two or more men consulted in each community. There were no large disparities in responses provided by people from different genders and thus the evaluator is confident that the findings should be representative of the views of both men and women.  The evaluator mostly relied on translators to convey questions and interpret responses. Subtle differences in questioning and responses may not always have been conveyed accurately thus biasing the evaluation. The translators were all connected to the program in some way and in some instances they were known by the community members being consulted. This introduced the potential for bias in reporting. The use of ‘private stories’ shard by community members directly with the evaluator did not reveal any evidence of bias. 10  There was a low representation of people with disabilities in FGD (n=1) and despite deliberate efforts to seek out people with disabilities to participate in community interviews (n=3), representation was low (3%). Participation of young men and women under the age of 30 was also very low (n=12)  The Program sought to change DRM behaviours of community members, however, the only time these behaviours are reliably displayed is during a disaster situation. Instead of observing actual behaviour, the evaluation relied on knowledge tests and self-reporting of actioning new behaviours during simulations and in recent disaster events or disaster warnings.  Tefisi was selected as one of the two control group communities. Tefisi had benefited from another DRM related program led by another project. The project shared similar knowledge and skills with the community. The presence of the other DRM related program in Tefisi made its status of a control group partially invalid and this made the identification of differences between target and control group communities difficult to identify. The program in Tefisi highlights that other programs are sharing DRM knowledge with communities. 3.1 Theory of Change Each year of the program required AFP to write a detailed proposal to OFDA that described the core problem(s) the program sought to address. The proposal also included the recommended technical solution and activities to address the core problem(s). Despite this level of detail, the Program was not based on a guiding logic model or Theory of Change (ToC). A logic model or ToC can help communication the program design with external stakeholders. Through monitoring and evaluation, the ToC can also serve as a tool to explore if the theories and assumptions that guide the Program are valid or if changes are required to achieve the desired program outcomes. For these reasons, it is recommended that a ToC be created at during the design phase of any new development program. As part of the evaluation, a ToC was drafted and verified with TCDRM staff through a 1-hour workshop. The ToC was later refined, and on request, a simplified more generic ToC was created to accompany the more detailed ToC (See Annex 10 for the detailed and simplified ToC). 11 4 Findings 4.1 Effectiveness - To what extent were the project objectives achieved? Overall, the program made good progress in achieving its four high level objectives (See page 7 for objectives). Most notably, the targeted communities had improved Disaster Risk Reduction (DRR) and Disaster Risk Management (DRM) knowledge and skills, and have demonstrated the application of these new knowledge and skills in both simulations and real life disaster events to improve their level of safety and reduce the extent and damage suffered. Key findings for each of the four objectives are explored below. 4.1.1 Reducing the impact of natural disasters on vulnerable communities. The actual and likely future impact of natural disasters on targeted vulnerable communities was reduced by the Program. The Program supported 51 vulnerable villages spread across three of Tonga’s islands groups to create or strengthen Village Emergency Management Committees (VEMC). These VEMCs are structures that supported communities to be better prepared for natural disasters through the development and implementation of their Village Emergency Management Plans (VEMP), participation in training and simulations and improved access to information. The number of community members benefiting from the Program activities totalled 15,472 (7,794 men, 7678 women) spread across the 51 remote villages. DRM structures and Plans 51 VEMCs established 51 VEMPs created Capacity building workshops 83 DRM, WASH, & Food Security workshops 51 Simulations 6,441 participants (2,968 men, 3,473 women, 73 people with disabilities Evacuation Centres 5 Evacuation Centres improved The Program facilitated the creation of 51 VEMCs who had their capacity build through a range of community-based capacity building workshops (DRM, Protection, Emergency Response Training, First Aid, Food security and Water, Hygiene and Sanitation (WASH)). The Program team worked with VEMCs to create 51 VEMPs, one for each community. The VEMPs documented the roles and responsibilities of VEMC members and included a DRR Action Plan that outlined steps to follow before, during and after an emergency for identified hazards. These activities also helped the Town Officer, who fills the role of the VEMC chairperson, to better understand his/her DRM responsibilities as outlined in the National Disaster Act. "Before I know I am the Town Officer and I lead the people but I didn't know that I have a lot of responsibilities especially when it comes to disaster my role as a chairperson before/during and after a disaster." Town Office, Motafonua “I now have a clear idea of my responsibility as the town officer / chairperson of VEMC.” Town Officer, Muitoa, Ha'apai Both target and control communities consulted as part of the evaluation scored well in DRR/DRM knowledge tests with most citing multiple beneficial actions to perform before, during and after a disaster such as a cyclone. Whilst a cyclone was the main hazard discussed in consultations, community members also demonstrated knowledge of how to prepare for or respond to tsunami and earthquake events. The high level of knowledge in both target and control communities reflects the fact that much of the DRM knowledge has been learnt from past experience of disasters and also through other 12 information sources such as the radio which provides detailed guidance to listeners when an emergency is announced. “Before, people did not really care or aware of the importance of being prepared. Now I have witnessed that people start to be more aware and start preparing when there's a cyclone warning.” Ha’ano community member Evidence of target communities retaining TCDRM knowledge and skills was found through how target communities responded to category five TC-Gita which impacted ‘Eua and Tongatapu in February 2018, only several months after TCDRM training and simulation activities had been conducted in three ‘Eua communities. The activation of VEMCs and implementation of measures in VEMPs combined with the knowledge and skills reinforced by simulation activities were key measures that assisted target communities to be better prepared for the cyclone. Examples of better preparedness include:  Community members taking proper precautions to prepare their homes, including youth assisting other people in the community. For example, youth assisted a people with a disabilities by offering assistance to evacuate).  Down pipes being disconnected from water tanks to protect the safety of the water source  Cassava and banana gardens being cut to lessen the impacts on these productive assets.  The Town Officer and VEMC members encouraged vulnerable community members to evacuate.  Some community members evacuated to the Evacuation Centre (EC) or a neighbours stronger home early, before the cyclone hit, and there were reports that people came to the EC prepared with their 72-hour bag.  VEMC members were reported to be in attendance at the EC and they took action to actively manage the EC and implement some degree of protection measures. For example, VEMC members undertook head counts at the start and end of the emergency, and women and girls were kept in separate areas of the EC to reduce protection risks. “The program is very good and it really helped me to learn something that I don't know before, like to get ready for a cyclone and do first aid. I have also used the first aid with my grandchildren when they get cuts. When TC-Gita came, I took care of the elderly people. I helped them evacuate to the evacuation centre (community hall) and made cups of tea for them” VEMC member, Ohonua, Eua It was also reported that target communities in Vava’u and Ha’apai have implemented preparedness actions in response to news of tropical depressions and cyclone warnings. For example, Longomapu community in Vava’u cut their cassava and Banana trees in response to an early warning about a tropical cyclone depression. Act For Peace also reported that VEMPs activated and assisted in evacuating vulnerable community members when TC-Winston struck Vava’u in 2016. There was mixed evidence form national and sub-national sources about target communities being better prepared and capable. The ‘Eua Government representative indicated that Town Officers from communities were more organised with their post-disaster initial assessment reports and head counts, however, he was unable to isolate this improvement to the target communities. The ‘Eua NEMO officer indicated that Mata’aho community (TCDRM target community) was more active in comparison with 13 other communities, however, he did not report the same positive findings for other two target communities in ‘Eua. Only one target community, Matafonua, was visited in Tongatapu as part of the evaluation. This community was last engaged by the TCDRM in 2015. When TC-Gita warnings were issued, it was reported that a small meeting of VEMC and community members was held and that other community members were notified about the cyclone by the VEMC chairperson (Town Officer) using his hailer. There were then mixed reports about action taken. The VEMC indicated that they didn’t offer any assistance to households to prepare. They indicated that it was each individual household’s responsibility to prepare and get ready. It was even pointed out that a widow with a disabled son was responsible for making their own preparations without assistance from the VEMC. The Town Officer indicated that the paper copy of the VEMP was not used and he relied on knowledge from past lived experience of cyclone events. Distributing two copies of the VEMP to VEMC members (not just the Town Officer) may help these other VEMC members encourage the Town Officer to follow the plan. The additional copy could also be used by the VEMC when the Town Officer is absent during an emergency. Interviews with other community members in Matafonua indicated that the VEMC was not currently very active, however, the few VEMC members in attendance at the FGD appeared very active within the community and in lieu of a functioning Community Council, they had stepped in to provide a community governance structure. During TC-Gita they also indicated that some youth had offered assistance to households to prepare for the cyclone. It was unclear if these youth were directed by the Town Officer. One community member at the FGD shared her story that despite the warnings about TC-Gita, she did not prepare early because there was no signs of strong winds during the day. When the winds became stronger, she evacuated to the EC without a 72-hour bag. The Matafonua community demonstrates the difficulty in coming to strong conclusions about the effectiveness of VEMCs and usefulness of VEMPs several years after engagement with the Program. The three-year gap between TCDRM community engagement and TC-Gita may explain why some of the behaviour change encouraged by the program was not displayed in Matafonua. Community cohesions was thought to be relatively low in Matafonua, with residents coming from many different parts of Tonga as opposed to long-term stable family groupings. This population characteristic may have also reduced the effectiveness of the Program in this community. Both NEMO and the community members themselves provided evidence that target communities took proactive action after TC-Gita in ‘Eua and Tongatapu. With a better understanding of their roles and responsibilities, they did not sit and wait for the Government to clean-up or offer rebuilding assistance. With DRM measures implemented, target communities, and ‘Eua communities more broadly, had access to food and water sources. The evaluation was unable to determine if target communities recovered any faster than non-target communities in ‘Eua and Tongatapu after TC-Gita. 14 Another means of exploring the change made by the Program is comparing the differences noted between intervention and control communities. For the purpose of this comparison, only one of the control communities will be used as the second community, Tefisi, benefited from another DRM program, Coastal Community Adaptation Project (C-CAP) funded which partially invalidates the findings from this community.3 Intervention communities Foa - Control community Emergency Management Committee Dedicated VEMCs formed No separate committee responsible for DRM. DRM responsibilities were included in the Community Committee. Emergency Management Plan Specific and detailed VEMPs drafted and shared with NEMO. No separate Plan for DRM. This information was included in the Community Plan. Committee/ community DRM capacity Capacity building in CbDRM, CbCERT, First Aid, SIMEX (x2) held in each community. Refresher training provided to some communities No DRM capacity building received No SIMEX , no First Aid training DRM knowledge & skills DRM knowledge and skills built, reinforced and refreshed. VEMC targeted. VEMC shared knowledge with community members in some, but not all instances. DRM knowledge - what to do before, during and after a cyclone was generally high. Knowledge of response to earthquake and tsunami events also present. No noted First Aid skills. High degree of DRM knowledge in relation to cyclones, tsunami and earthquakes. DRM is discussed at fono (community meeting), but this is mainly limited to ensuring appropriate water and food security. Protection Awareness of protection issues and appropriate mitigation measures. VEMC members assist to manage the EC and implement protection measures. Protection roles within VEMC established, however, VEMC members are unaware of who is allocated these roles Not strong knowledge of protection issues. Main focus was on physical safety from the direct impacts of the cyclone, not physical or sexual abuse risks. No one was noted as being responsible for controlling or managing the EC. Inclusion People with disability and elderly people were involved in program activities (SIMEX). These vulnerable groups, including widows and pregnant women were noted as receiving additional assistance during an emergency to help prepare their homes or evacuate. Noted importance of assisting elderly and people with disabilities to evacuate to the EC. Relationship with GoT Initially improved through Program bringing GoT representatives to the community to better explain DRM roles and responsibilities. Improvements in relationship did not Understanding of communication between community and GoT was not clear with some confusion between MIA and NEMO. Mixed views on relationship between community and GoT. Women indicated weak 3 Tefisi was involved in C-CAP Program: https://www.usaid.gov/pacific-islands/environment-and-global￾climate-change/coastal-community-adaptation-project 15 Intervention communities Foa - Control community continue. relationship. Men, indicated strong relationship, but most interactions are through the Town Officer. Water/Food security Capacity building in food security, water security and community projects implemented to improve water and food security. No noted changes or improvements to water security or food security in recent years. Improved EC Five ECs were improved with stronger roofing, windows, doors, water tanks and other measures to provide a safer location for people to evacuate to. EC existed in the community (LDS and FWC buildings). However, their status was not assessed for integrity or safety. No reported EC maintenance projects were reported. The simple comparison provided above provides evidence that the TCDRM Program has built DRM capacity within communities to be better prepared and respond to disasters. Dedicated VEMCs with improved DRM knowledge and skills supported by VEMPs and improvements to water and food security are all noted benefits not experienced by the control community. The above benefits experienced by the target communities has assisted them to feel safer and more confident that they can survive disaster events when they do strike. For example, the VEMC members in Muitoa now felt safer because the training provided them with the skills to prepare for disasters. “When I grew up, most time we would just wait until last warning of the cyclone, then jump up and start preparing. When I realised after the training that that is too late and that's what happened in TC Ian and the damage was beyond. I think if we were prepared there would have been a different result. I was very fortunate to be a part of this program. Training is very very important and we share it with our children and community.” Town Officer, Muitoa, Ha'apai 16 4.1.2 Increased community capacity to identify protection risks and propose mitigation strategies Each VEMC established benefited from capacity building in Community based Protection (CbP) to highlight and reduce the risk of abuse of vulnerable groups during a disaster. Seven of the ten VEMCs engaged by the evaluation were found to have heightened awareness of protection issues as a direct result of TCDRM activities. The importance of keeping women and girls safe from abuse in a mixed gender environment at the EC was recognised by most VEMCs. Most VEMC members consulted were able to identify, women, girls, elderly and people with disabilities as being more at risk of protection issues. Widows and female headed households were also noted by informants as being more vulnerable. Protection mitigation measures identified by VEMCs included keeping men and women in separate areas of the EC. There was evidence that VEMCs demonstrated this during SIMEX activities and it reportedly happened at Motafonua EC in Tongatapu during TC Gita. Another strategy commonly mentioned was to ensure families stayed within their family groups within the EC. The TCDRM team confirmed that the grouping of families also assisted with initial head counts. The Ha'ano VEMC also noted the importance of having separate toilets for men and women. All of these response measures are valid and were discussed as part of the CbP training where community members worked with the TCDRM team to identify how they believed protection risks could be mitigated. The three communities where enhanced knowledge of protection issues was not found focused their responses more on the physical safety risks from the impact of disaster event, rather than risk of abuse from other community members. The Program placed more emphasis on CbP during years 4 and 5, however, the evaluation did not detect any reportable increase in knowledge or application of CbP between communities involved in Years 1-3 compared to Years 4-5. This may be due to the limited time for questioning in this area and being overly culturally sensitive not to probe too deeply into this sensitive topic during the evaluation. No VEMCs proactively made reference to the presence of Protection Officers in the committee and when questioned about this, several VEMCs appeared confused and most had to refer to the VEMP hardcopy that was just handed over to the Town Officer by Program staff. This lack of knowledge highlights that more work needs to be done in this areas to refresh knowledge on VEMC roles and responsibilities. Two VEMPs had been translated into English and these were inspected and both allocated committee members to the role of male and female Protection Officers, however, there was no other content in the plan that described the possible risk faced by vulnerable groups during (at the EC) or after a disaster (external third parties delivering assistance). 17 The Program was effective at motivating the ‘Eua Government Representative to take positive action to reduce protection risks at several ECs in ‘Eua. Taking positive action to increase protection “When I saw the VEMCs demonstrating protection measures during the simulation it gave me some new insights and got me thinking. I noticed that the EC did not have any generator and there was no facility available to plug in a generator to power the EC and provide lighting at night to help protect women and girls.” “I later called an electrician and asked them to quote for the work to install a secondary power inlet plug to power the EC via a generator. Using funding available to me I had these additional generator input plugs installed in approximately 3 ECs.” ‘Eua Government Representative The evaluation was unable to make a conclusive finding on the capability of VEMCs to identify new protection risks independently without external assistance. Based on past experience in Tonga and other Pacific Island Countries (PICs), the evaluation finds that the expectation that VEMCs will independently identify and respond to protection risks in a proactive manner, is overly ambitious given their level of exposure to the topic and the culturally sensitive nature of the topic. TCDRM helped communities take a first step forward in increasing understanding on the topic of protection, however, repeated exposure and broader community acceptance about the topic is needed for it to gain greater traction. The perceived relevance of protection issues in small communities was observed to be low in some instances. In many communities everyone is related and all community members know each other. These trusted family and friendship relationships provide a sense of safety even where real protection risks such as family violence exist in everyday life. No informants made any mention of possible protection risks after a disaster even though real risks involving external disaster relief providers exist. It is unclear if CbP did not successfully communicate this element of protection risk or if it failed to gain traction with VEMC members. The inclusion of real case studies of protection violations based on past experiences in Vava’u and Ha’apai may assist communicate the importance of CbP. Awareness about protection issues was not limited to TCDRM target communities. The Town Officer from the Tefisi control community was also aware of protection concerns and responses and this knowledge was likely acquired through Tefisi’s involvement in C-CAP program.4 The C-CAP program shared much of the same knowledge covered by TCDRM. “During cyclone we need to manage the EC. People should stay in family groups in EC to keep safe.” Tefisi Town Officer The Town Officer from Foa, the second control community, also provided some knowledge of protection issues, however, his response “men and women go to different EC” did not align with what reportedly actually happens in the community where there were ‘mixed groups’ at the EC. 4 C-CAP Program Overview: https://www.usaid.gov/pacific-islands/environment-and-global-climate￾change/coastal-community-adaptation-project 18 Perception of safety Most male and female community members reported feeling safe or very safe before, during and after a disaster. The most common reasons for feeling safe were that they stayed with their family, either at home as they feel their homes are strong enough, or they stay within family groups at the EC. Elderly and people with disabilities consulted also reported feeling safe as they noted that vulnerable groups are now a priority for the community during disasters. People who reported feeling unsafe made reference to their male partner or parent not being at home or being too frail to protect and care for the family. One young women from Tu’anuku reported feeling very safe because she stayed at home with her parents, however, she also mentioned that she would feel less safe if she needed to evacuate to the EC. 19 4.1.3 Improved community-level food security Food security has been enhanced in target communities through food security projects, food security workshops and the discussion of food security and response measures which were integrated into VEMPs. Seventeen communities implemented practical measures to improve food security through TCDRM funded community food security projects. Food security community project summary 17 beneficiary communities 4 community fences 7 Livestock fences 18 chicken fences 2 Greenhouses 2 Sets of garden tools 53 Climate adapted sheep^ ^ Sheep were provided through the Program, but funded by Act For Peace, not USAID. Technical solutions implemented and their food security benefits are documented in Table 2 below. Table 2. Food security solutions Food security measure Benefit Greenhouse construction. Vegetable & cash crop propagation Greenhouses create an environment safe from pigs, chickens, heavy rain and harsh sun to propagate seeds and raise seedlings to a mature age ready for planting in community and household vegetable gardens. Increased seedling distribution assisted increase food security and cash crops (kava and sandalwood) will assist with longer-term livelihoods. Animal enclosure fences Used to keep sheep within allocated paddock boundaries for easier feeding and management. Enclosures reduce risk of sheep roaming and being attached or killed by wild dogs. Garden protection fences Used to keep pigs and chickens out from community food gardens, thus reducing losses from crop damage and increasing harvested food volumes. Livestock Approximately 53 climate adapted sheep were distributed to communities.5 These sheep have been breed and numbers have more than tripled in some communities. Once sheep numbers have increased sufficiently, breeding sheep pairs will be given to community members for continued breeding and later slaughter for meat consumption. Sheep provide another diversified source of protein and iron to complement existing traditional diets and substitute the purchase of imported sheep meat. The original VEMPs were created before community food security projects were implemented and additional DRR measures may need to be included into VEMPs to devise actions to keep livestock safe during emergencies. These measures may include developing cyclone proof animal shelters or setting livestock free to escape a possible tsunami event. The desktop review found evidence that some food security baselines had been created through Ministry of Agriculture Food and Forests (MAFF)-led Food Security Assessments (Koloa community). 5 The sheep distributed as part of TCDRM activities were procured and funded separately by AFP and not USAID. 20 MAFF compiled food security prediction reports for target communities in Vava’u and these indicated that three communities would be 99% food secure and self-sufficient within 12 months (2018) and that an additional three communities would be 50% food secure. These predictions are useful indicator of future food security, however, there was no evidence of end-line assessments being conducted to verify if these planned increases occurred. Without the presence of baseline and end line food production data, the evaluation is not able to quantify the increase in food security. Anecdotally, some target communities reported their community members had started to plant more food crops under the encouragement of the Town Officer and MAFF in response to food security concerns and the need to be more prepared for disasters. These crops included traditional resilient root crops that have long shelf lives post-harvest. Interviews with target community members indicated the presence of canned food and root crops in the home that could be quickly prepared or packed in a 72 hour bag. Whilst food security training also encouraged the processing and preservation of food, no community members made any references to these practices in their cyclone season preparation activities. All communities (control and target) indicated that preparing food, usually undertaken by women, was an activity conducted to prepare for a cyclone. VEMCs also made reference to preparing food in the umu (underground earth oven) and the long storage time of this cooked food. 4.1.4 Increased access to safe and sufficient water TCDRM community projects have directly increased short and long-term access to safe potable water for 35 communities. Communities reported experiencing fewer water shortages as a result of the additional water storage. WASH training conducted in years 4 and 5 has also highlighted the importance of water conservation and good hygiene practices. Water security summary 35 communities benefiting from water￾related community grants Additional water storage capacity 1,465 Kilolitres 2 Water pumps Average additional days of water supply in emergency: 21 days & drought: 43 days Rainwater captured in cement and plastic tanks remain the primary water source of potable water for communities. Some communities are also connected to village water supply systems where ground water is pumped to header tanks and distributed to communities. Households in some communities directly benefited from the presence of a new TCDRM-funded water tank connected to their house. In these instances the location of their primary potable water source changed from either a neighbour’s tank or community tank to the one on their property, thus providing more convenient access to water.6 Approximately 144 new tanks were distributed to 35 communities. An additional 27 cement tanks were renovated to fix leaks and cracks. New and renovated tanks combined provided an additional 1,465 Kilolitres (KL) of community water storage. This additional water capacity provided communities with an average of 22 days of additional water supply during an emergency, or 43 days during drought 6 Tanks located at private premises were still accessible to the entire community. 21 situations.7 Additional details of water capacity increases and the variation between communities for additional days of water supply can be found in Annex 11. Water tanks were installed at community and church halls, schools and community households. All community members consulted indicated that anyone could access any of the TCDRM funded water tanks, even where they were installed at a private household or a church facility different from the denomination they followed. Each community, through the WASH committee or Town Officer, was responsible for developing water usage guidelines for the water from the rainwater tanks. Guidelines commonly consisted of rules to ensure universal access regardless of tank location and that water from the tanks was to be used only for drinking or food preparation. Community members reported that this rule was followed. Two pumps were distributed to communities to assist in the pumping of underground water. The increase in water storage capacity was greatly appreciated by community members. Improvements in water security was the most recalled benefit delivered by the program. Community level water storage baselines do not exist for all communities. Where baselines exist, increases varied greatly from modest increases of 17% (Koloa) to the tripling of storage (Holeva). “Previously our water was not good but now our water source is clean and safe. Our standard of living is developing. Dry period is now in Vava'u, but we are safe because of these tanks, thanks.” Taoa male community member There were minor delays to the implementation of some water projects. The delays were a result of transport and logistics challenges to deliver tanks and materials from Tongatapu to the remote islands of Ha’apai and Vava’u. These delays are not uncommon in Tonga and they should be factored into project work plans. The effectiveness of water tanks solutions was enhanced and assured by a number of factors including:  Use of proven rainwater harvesting and storage technology that is appropriate for remote islands.  Engagement of a WASH engineer to oversee tank system design, influence tank site selection and perform monitoring of implementation.  Use of quality tanks from an ISO 9001 provider using quality UV resistant plastic to provide at least 20 years (20 year guarantee) life.  Design including best practices such as First Flush Diverters (FFD), mosquito proof mesh on tank entry points, debris screen, metal cables to tie tanks down to foundation (for all tanks in years 4 and 5)  Solid tank foundations  Letters of agreement outlining the tank site land-owners obligation to grant all community members access to the tank were signed before the tanks were installed. Issues that may limit the effectiveness water projects include: 7 Using WHO guidelines of 15 Litres per person per day for emergencies and 7.5 litres during drought conditions. Assumes new water tanks are full at the start of the emergency or drought period. 22  Town Officer influence on tank site selection. In some instances the site selection was not optimal and there may have been some favouritism for friends and family. In other instances sites were not appropriate due to the presence of existing tanks or the poor quality of roofing and guttering. After an inspection of the proposed locations, the WASH engineer was able to persuade Town Officers to reconsider the site selection in most, but not all case. The oversight provided by the WASH engineer was vital to ensuring the citing of the tanks delivered optimum benefits. This best practice of engaging a WASH engineer should be continued in future programs.  Some communities made minor deviations from the tank design and procured lower quality taps (faucets) than those recommended by the WASH engineer. These lower quality taps are readily available and were selected in preference to waiting for order for higher quality parts to be shipped. The long term outcome of this quality issue may include early tap corrosion, higher likelihood of breakage and leaking. Drought was included a slow-onset hazard in VEMPs and simple mitigation and response measures were included in the plans. Response measures could be enhanced through the creation of separate WASH plans or integrating components of community WASH plans, specifically the triggering of water rationing measures once certain water storage thresholds are reached. For example, rationing could be introduced once water volumes reach 25% to limit each household to 15 litres per person per day. Such measures may prove difficult to implement and enforce, however, community members could discuss and agree on other rationing methods. Larger WASH community project budgets in years 3 and 4 combined with an earlier shift from building cement tanks to using plastic rotomould tanks, facilitated the delivery of more new water storage capacity in each community in the later years of the program. WASH training was also added in years 3 and 4 of the program and there was some evidence that this contributed to improved WASH behaviours, specifically improved handwashing and increased awareness of the importance of clean drinking water. The extent of this behaviour change was not adequately assessed by the evaluation and whilst only one or two stakeholders made reference to improved hygiene behaviours in informant interviews and FGD, the uptake of these behaviours may have been greater. 23 4.1.5 Private stories verify findings Individual private story (feedback) collected from Focus Group participants confirmed overwhelming positive community support for the Program. Ninety percent of the 71 stories collected praised the program or made specific reference to increased DRM knowledge or putting in practice DRM actions promoted by the program (n=34) (See Figure 2). Nine stories made positive comments about the impact of water security projects. The few critical stories (n=4) provided evidence that not all VEMCs were active and sharing information, and there was one report of the VEMC not being present at the EC to manage the EC during an emergency. Figure 2. Classification of private story feedback on TCDRM Program 4.2 What were the major factors influencing the achievement or non-achievement of the objectives? The evaluation identified a number of key factors that supported the success of the TCDRM program. These key factors are documented in Table 3 below. Table 3. Key success factors. Success factors Comments Competent, knowledgeable, committed staff  TCDRM field staff had prior community development experience and foundation knowledge in CbDRM and related content. The team demonstrated their competency through the successful delivery of outputs. Their competency was observed both in the office and through their organisation of the evaluation. Program design  The program design featured multiple engagements with communities, often spread over two or three years. This spread of engagement created continuity in the relationship between Program staff and community members. It also enabled the Program to continually reinforce prior learning and build new capacity on top of existing foundations.  Community projects were effective in gaining buy-in from the communities. Project outputs (VEMC, VEMP, water tanks, fencing) provided community members with tangible evidence that the Program was delivering on its commitments.  The design featured various modalities of engagement including knowledge building workshops featuring presentations, activities and multimedia. Simulation activities reinforced prior learning through the practical application of knowledge and skills. The VEMP could be viewed as a reminder prompt, even if not visible, for communities to take action.  The first simulation activity was led by the TCDRM team. The second simulation was led by VEMCs and this approach follows best practice learning strategies.  The repeated demonstration of desired behaviours helped to embed them into existing disaster response and recovery routines. For example the preparation of 72 hour bag was demonstrated at SIMEX and people were also exposed to the behaviour by Mormons from Latter-day Saints (LDS) church who practice this 24 behaviour as part of their church commitments. These practical demonstrations make the desired behaviours visible to a broad section of the population and thus encourage broader adoption of what is viewed as the new ‘normal’ behaviour being championed by the Town Officer (respected member of the community) and other respected members of the community (VEMC leaders). Program implementation  Consultation with national, district and community stakeholders occurred before the delivery of any community activities. This consultation, especially at the community level was noted as critically important to building trust and the successful delivery of activities. Consultations were useful in explaining the program intentions and roles and responsibilities of all parties.  Consultations enabled the TCDRM team to uncover the loss of trust community members had with external stakeholders as a result of prior broken promises from both CSO and GoT representatives. The power of the church  The churches and religious orders in Tonga are very influential and are highly respected by the general population. TNCC’s close links to major church denominations also helped the program gain the trust of the communities. The Partnership Approach  The partnership approach between AFP and TNCC was one of the programs key pillars of success. The support and capacity building led by AFP greatly strengthened TNCC’s ability to effectively implement the program. This included building capacity in finance, administration and procurement as well as specific TCDRM themed areas such as protection. Without a strong and capable implementation team, it is highly unlikely the program would have achieved the same level of success.  The TCDRM program is fortunate to have been underpinned by the Partnership Approach from its outset and this should be regarded as best practice. The evaluation notes the importance of the SIMEX in demonstrating and role modelling the desired DRM behaviours. Behavioural science predicts that knowledge alone is a poor predictor of people performing the desired behaviour (DRM actions).8 Additional behaviour change models, frameworks and tools such as commitments, prompts, incentivise and social norms could be applied to the TCDRM design and methodology to enhance behaviour change outcomes. A short case study below illustrates the power of combining different behaviour change interventions with capacity building. More specific behaviour change considerations are included the recommendations section of the report. 8 Ajzen, I, 2017, 'Why interventions fail: The reasoned action approach to changing human behaviour', University of Massachusetts, http://www.behaviourworksaustralia.org/wp￾content/uploads/2014/12/IcekAjzenpresentation.pdf 25 SODIS Champions in Kiribati The Secretariat of the Pacific Community Global Climate Change Alliance: Pacific Small Island States (SPC-GCCA:PSIS) program in Kiribati had as one of its objectives to improve environmental health in communities through the provision of improved potable water. Solar water disinfection (SODIS)9 was deemed a low-cost appropriate technology for community members, but changing their behaviour to use SODIS required multiple strategies including:  Communication from known and trusted sources. International scientific experts performed experiments in front of the public and demonstrated that SODIS killed 99.99% of dangerous bacteria in the local water making it safe to drink.  Prompts. SODIS posters were pinned up in key community locations and radio advertising promoted SODIS. Brochures and songs helped reinforce the behaviour.  Social norms. SODIS champions were paid to demonstrate the desired behaviour publicly and answer questions about SODIS on a weekly basis.  Trusted story narrative. Stories from local community member’s experience using SODIS were collected and shared.  Convenience. The program distributed PET water bottles to make the behaviour easy and convenient. Those households without a tin roof could use the Champion’s demonstration station to improve the effectiveness and convenience of SODIS. These behaviour change focused measures were undertaken in additional to community workshops that shared the knowledge and skills required to perform SODIS and collectively they helped to shift the behaviours of hundreds of people in the target communities and there was evidence that many community members continue to practice SODIS after the program completed. A number of more challenging factors made it difficult for the Program to achieve its outcomes and achieve its full potential. These challenges are explored in table 4 below. Table 4. Challenging factors. Challenging factors Comments Town Officer is a critical success/failure point  The Town Officer is a highly respected and powerful member in the community. The Town Officer has the role of chairperson in the VEMC.  Active and proactive Town Officers can help ensure VEMC meetings are scheduled and actions are progressed. If the Town Officer does not feel a meeting or any action is needed, then it is highly unlikely VEMCs will meet and take action, even in times of disaster.  The TCDRM team shared stories that in some VEMCs, there were active members outside of the Town Officer who were taking action even when the Town Officer’s commitment to the VEMC was low. However, given the power the Town Officer has and the need to respect his authority, taking action independently of the Town Officer could be viewed as offensive or undermining.  Many Town Officers were reported to be old and less active in chairing the VEMCs and making use of VEMC members. 9 SODIS: https://www.sodis.ch/methode/index_EN 26  The power of the Town Officer to select the VEMC members can be problematic. Despite membership criteria guidance provided by the Program, there were instances where the VEMCs were originally made up of people related or very close to the Town Officer. The Program intervened to help correct this bias, however, there were also reports that in in some communities, the VEMC members were trained and then played no further active role in the Committee. This lack of commitment highlights to importance of selecting community members who are already motivated and active within the community. Difficulty sharing DRM knowledge with the community  Most, but not all VEMCs shared DRM knowledge with the community through a number of different channels such as women’s groups, kava circles and church.  Community meetings (fono) were the most heavily relied upon forum to share DRM news and knowledge. Unfortunately, attendance at fono was reported to be poor in many communities, even with door to door notifications to try to boost attendance. Poor attendance at fono has made this efficient method of knowledge sharing ineffective. The ineffectiveness of sharing knowledge was observed in several communities where community members were unaware of the TCDRM Program and were unsure if their community had a VEMC and VEMP. For example, in 'Ohonua, only three of the 23 people in attendance at the evaluation FGD had any knowledge about the TCDRM program or the VEMC.  VEMCs may need to rely more on direct door-to-door forms of engagement to ensure DRM news and knowledge is shared with the entire community. Allocating one or a number of Village areas (‘blocks’ of houses) for each VEMC member to target could be one strategy to improve knowledge sharing.  Another response strategy would be to re-emphasise the importance of having a member for each key religious denomination included within the VEMC. These members could be provided with a short sermon and supporting religious DRM messages and resources to improve the likelihood of information being shared at church.  Engaging schools directly via the TCDRM team, or indirectly via VEMCs to participate in community or school specific simulation activities would also help increase the reach of the Program. School children would return home and inform their parents about what they had learnt/done. The Motafanua Secretary highlighted the difficulty of sharing CbDRM knowledge and skills. "The Town Officer, I feel sorry for him. He tried to work together with the community, but some people just don't listen. We call a meeting and not everybody comes, only a few. We really need everybody to attend so we can share our message.", VEMC Secretary, Motafanua, Tongatapu Most VEMCs indicated that Town Officers did not have a functioning hailer and that the committee members did not have high visibility vests to wear during a disaster event. A hailer and vests may help give VEMC members a sense of pride and authority in addition to making their presence obvious during disasters and SIMEX. 27 4.2.1 Feeling undervalued The evaluation uncovered a sentiment that the work of CSOs in DRM was undervalued by Government stakeholders and that there was a lack of recognition of CSOs in delivering community-level components of the JNAP. Several CSOs representatives felt that GoT stakeholders were taking credit for their achievements without acknowledging the work of their programs. A Memorandum of Understanding (MoU) between each CSO and GoT could help articulate the roles and expectations of both parties to improve the relationship and good-will through adhering to the guidance in the MoU. 4.2.2 Connectedness The evaluation was asked to explore if any changes in connectedness occurred and if these contributed towards the achievement of objectives. The following exploration expands upon the view that connectedness was enhanced, but was short-lived and not highly influential in the achievement of objectives. How has the program promoted horizontal connectedness? The program strengthened sub-national Government stakeholders such as district-level Government Representatives, District Officers and Town Officers. These stakeholders benefited from targeted training which made them more aware of their roles and responsibilities as outlined in the National Disaster Plan. It also built their capacity through attendance at Training of Trainer (TOT) CbDRM workshops and SIMEX activities which provided an opportunity for them to fulfil their DRM roles and apply skills learned. Cross-sector sharing at the national-level was also reported to have occurred during national consultations to inform the draft CbDRM Framework. The capacity of CSOs in DRM was built through their attendance at TOT workshops and national consultations. There was limited evidence that the program created greater connectedness between GoT and CSOs outside of the relationship between TNCC and GoT (NEMO, MIA). The program promoted greater connectedness between GoT departments and communities, however, the gains made appear to be short-lived. Most, but not all, community members recalled GoT representatives (NEMO, MIA, MAFF, Tonga Water Board, Police, Armed forces) and civil society (TNRC, TNYC, TNCC) attending TCDRM events to explain their roles and responsibilities and sometimes deliver training (First Aid, Food Security). NEMO also used the opportunity to meet with VEMCs and review VEMPs. Not all communities recalled visitations or the strengthening of knowledge about the roles of GoT stakeholders. For example, in Ha'ano, VEMC and community members could not recall what NEMO’s role was and they do not recall NEMO visiting their community. Prior to the evaluation consultation, community members in Ha'ano were unaware that they could ask for assistance from MAFF to provide extension services and advice on sheep farming. This demonstrates a lack of awareness of MAFF roles and responsibilities. GoT and CSOs were extremely appreciative of the community engagement opportunity facilitated by the Program. Both stakeholder groups acknowledged that resources and funding were barriers that prevented them from undertaking community engagement and education outside of other specific funded programs. Communities appreciated the opportunity to hear from various stakeholders that rarely or have never visited their community in the past. Whilst the interactions and increased awareness was useful, in most cases it marks only a very first small step towards a working relationship that will deliver benefits to 28 communities through deeper and ongoing dialogue. See Annex 12 for summary of community and VEMC ratings on the strength of relationships between communities and GoT. “MAFF only come when it is time for the agricultural show” Matamaka community member “We have never seen NEMO here” Fotua Community member VEMC consultations indicated that there had not been any notable shift in the relationship between communities and GoT departments. There was no evidence of GoT or CSO stakeholders (other than TNCC) visiting communities more frequently than before the TCDRM program. Most community members indicated that GoT stakeholders only came when there was an activity on. “The program really helped remind me about the challenges the people in the community face. We sometimes lose touch with that in our offices. ” Former Vava’u NEMO Officer. The former Vava’u NEMO representative highlighted the importance of the community visitations facilitated by the program. The visits helped remind Government about the challenges of daily life faced by people at the community level. The ‘Eua Government Representative also noted this benefit from his engagement with community members. How has the program promoted vertical connections (strengthening linkages from national to sub￾national and community level? Whilst ‘connectedness’ may not have been greatly improved in terms of relationships, there has been a positive improvement in knowledge and behaviour in relation to the vertical communication between communities and the GoT for post-disaster initial assessment survey reports. Nearly all Town Officers consulted were able to correctly identify the chain of communications between communities and GoT officials (NEMO, MIA). The TCDRM program team noted that in the past Town Officers may make reports and complains to the media, but they are now going through the proper communications channels to raise their issues. District-level NEMO staff noted a big change in how Town Officers work with NEMO. Town Officers were now viewed to be more cooperative and willing to work with NEMO. A greater understanding of roles and responsibilities of all parties is a likely contributing factor that explains this shift. 4.2.3 CSOs and GoT working on DRR/DRM The TCDRM program followed in the footsteps of TNCC’s and AFP’s prior partnership project Pacific Community Focused Integrated DRR Project (PCIDRR). Both programs worked with and within Government structures (NEMO, MIA, National Disaster Plan, District Officer, Town Officer) and processes (creating VEMC, VEMP) demonstrating good practice and increasing the likelihood of national ownership and sustainability. This approach taken by TNCC and AFP was observed to be different from that taken by other CSOs working in DRM where there was a tendency for CSOs to have their own independent structure and operational hierarchy or work more independently of GoT (Tonga Trust, MORDI). For example, the TRCS trains its own independent first responders in communities and these volunteers report up to the TRCS and may or may not be a member of the GoT endorsed VEMC. In a second example, one CSO reportedly delivered a 2-day CbDRM training followed by a SIMEX in a Vava’u 29 community. The community had already benefited from the TCDRM program activities including SIMEX. The duplication of work would appear to indicate that unlike TCDRM, the CSO did not consult with NEMO to identify the highest priority vulnerable communities in need of assistance and capacity building. No evidence was found to support that there has been a significant shift in how CSOs (other than TNCC) engage and work with GoT in the DRR/DRM sector. 4.2.4 Community influence on DRR decision making. The Program provided an opportunity for a small number of community representatives (TO, DO) and CSO stakeholders, to provide input into the content of the National CBDRM Framework. Consultations involved 99 people (59 male, 40 female) spread across 5 districts. Limited time and funding compounded by geographical and logistical challenges were all factors that limited the breadth and depth of consultation to inform the Framework. The Framework was drafted in 2015 and despite indicative approval dates noted in TCDRM reports, the Framework it still has not been finalised or endorsed by the GoT. 4.3 What is the potential for replication of the CbDRM approach? The complete package of existing TCDRM components are recommended for inclusion in future programs that seek to replicate the success of TCDRM. This finding is premised on the effectiveness of the TCDRM program and the need to provide similar programming to other remote and vulnerable communities in Tonga. Six VEMC consultations and one community member reported that the TCDRM program was valued as a complete package and were unable to specifically identify individual components that were more or less effective at building resilience. Community members had less exposure to Program activities and they indicated that water projects (n=4) and simulations (n=3) were very effective at building resilience. The TCDRM program team also viewed the program as a complete package. The only less effective component identified was the national consultations and this component was removed from program implementation during years four and five. The actual make up of future programs should be determined by a country and community needs/situation analysis. For example, some countries may have funded national CbDRM programs already in place that have reached the majority of vulnerable communities. Communities in other countries may have already benefited from WASH or first-aid training through prior programs which means the extent of focus on these components can be greatly reduced. In Tonga, the success of TCDRM is dependent on the support and action from the community Town Officers. Identifying key community leaders/actors who can enable or block community engagement will be a critical first step in successful replication. The identification of these roles within community could be achieved through a stakeholder power analysis led by the in-country CSO/partner organisations and verified through a small number of community consultations. The partnership approach underpinning the program should also be replicated in future CbDRM programs and similar programs that seek to educate and change behaviours at the community level. The partnership approach will ensure that capacity within the local delivery partner is built to effectively deliver the program and that there is an external or mature local partner involved that can assist the 30 local delivery partner to adhere to donor requirements such as risk management, procurement rules, fiduciary standards and reporting. The CbDRM approach has been used successfully in many parts of the world including in Nepal through another USAID funded program.10 Provided there is appropriate contextualisation of the approach, then it is likely to be relevant and effective in other contexts. Some factors that may influence the extent of program effectiveness in other countries include: Factors influence replication success Comments Frequency of natural disasters Frequent disasters would demonstrate the high level of relevance of CbDRM to the community and suggests there would be community support for the program. Conversely, frequent disasters may result in communities already being highly knowledgeable and skills in preparedness and response. This can become a barrier to engaging in program activities where community members already feel they already know what to do, even though their behaviours and practices could be improved through additional capacity building. Severity of natural disasters Buy-in, support and impact is likely to be higher in contexts where the severity of natural disasters is also high. Community expectations and attitudes and level of dependency on external assistance Community expectations and attitudes towards external assistance will also be a contributing factor to the likely success of the replication in other countries. For example, in Vanuatu, their equivalent of VEMCs were somewhat capable, however, they had an expectation that they should be paid by civil society or the Government to perform their roles and share knowledge with the community. In these instances the community grant projects will be an important element of the program to motivate community volunteerism and participation in the early phases. The sustainability of outcomes will be an ongoing problem in these contexts. Community cohesion Communities in conflict zones or those experiencing political or ethnic tensions are likely to be more challenging to operate within. In these situations, DRR/DRM may not be a high priority when compared with other needs. Creating effective and cohesive VEMCs that include representation from different community groups is also likely to be more challenging. National DRM context The presence of an overarching national JNAP or DRR/DRM plan should be a pre-condition of country selection for replication. The absence of a plan is an indicator of weak maturity in DRR/DRM and may indicate a lack of commitment to this area from government. Without a plan that outlines EWS, national response mechanisms and roles and responsibilities of national actors, then the content of CbDRM will be more difficult to develop with risks on inconsistency between messages shared by other stakeholders implementing CbDRM. Where a national plan does not yet exist, then the program focus can be changed to address this initial gap. In Tonga, examples were cited of other CSOs running CbDRM and creating VEMCs, however, it was noted that they often took short-cuts, did not cover CbDRM fundamentals and did not build the capacity required to create and maintain functioning and effective VEMCs with VEMPs. This lower quality CbDRM 10 CBDRM in Nepal with USAID https://www.usaid.gov/nepal/fact-sheets/community-based-disaster￾risk-management-cbdrm 31 delivered in communities can leave them more exposed to disaster risks because NEMO, based on the reports of CbDRM training, will believe these communities are capable of preparing and responding to emergencies through their VEMCs and VEMPs and thus these communities will not be prioritised for CbDRM capacity building. The different depth and quality of CbDRM training, VEMC formation and VEMP creation by different CSOs highlights the need for standardisation and consistency of approach led by NEMO. The finalised National CbDRM Framework may advance this cause, however, other measures should and can be considered in the interim by CSOs and NEMO. Firstly, NEMO and CSOs could agree on a core set of minimum DRM related knowledge and skills that must be covered if a CSO is going to engage a community to create a VEMC/VEMP. These requirements can then be documented in a Tonga National Quality Assurance Board (TNQAB) recognised Community Course. Course delivery could be supported by best practice resources developed by customising existing TCDRM resources. CSOs wishing to deliver the program would (or could) need to become registered Community Education Providers through TNQAB. This status of education provider and the Community Course model was created specifically for CSOs and individuals to increase the quality of non-certified course delivery at the community level. Whilst TNCC may fear losing their competitive advantage in CbDRM by sharing their resources, the sharing their best practice TCDRM program and resources has the potential to strengthen and enhance the delivery of CbDRM at the national level through improving the work of all CSOs. 4.4 To what extent has achievement of project objectives been equitable (considering gender, disability and marginalisation/ remoteness)? The program objectives were achieved in a matter that was equitable in considering gender, disability and geographical remoteness. Attendance in Program activities demonstrated the presence of men, women and people with disabilities and there were nearly equal numbers of male and female beneficiaries. Whilst VEMCs were led by Town Officers who are nearly always, men, other VEMC roles such as deputy-chairperson, secretary and treasurer were held by women. Overall, women were reasonably well represented within VEMCs, although most documentation and evaluation consultations found that there were more men than women elected (e.g. 'Ohonua, Ha'ano). There were other examples of equal representations (Fotua) and a few examples where there were more women than men (Faleloa, Toula). Rates of women’s participation was assessed through a ‘ladder of participation’ tool issued to 23 female VEMC members across seven communities. The results indicated that men hold power and make decisions within the VENC. Six female VEMC members indicated they felt they could only attend meetings and not speak up. Twelve women thought they could speak up at meetings, but that they were not listened to. Only five women ranked their extent of participation as ‘being listened to’ or higher on the ladder. See Annex 18 for ladder of participation details. Table 5. Number of beneficiaries from 51 communities Men Women Total Beneficiaries 7,794 7,678 15,472 Conclusive data on youth participation in activities was not readily available, however, the evaluation found youth had lower levels of representation in activities. The mid-term evaluation also made this finding regarding youth participation. When youth were Figure 3. Beneficiaries disaggregated by gender 32 questioned about strategies to increase their engagement, the most common response was that VEMC members should attend youth group meetings. It was noted that not many youth attended fono and therefore these meetings were not effective at engaging youth. The Ha’apai MIA sports officer proposed that engaging local youth in drama that they would perform for the enjoyment and education of all community members would increase both youth and community engagement. The TCDRM program manager agreed that drama is an effective community engagement and education tool that would likely increase youth participation. Program activities were implemented across three island groups with the majority (86%) of the 51 communities within the outer island groups of Ha’apai and Vava’u and eighteen very remote communities only accessible via smaller boats. See figure 4 for a summary of community locations. More detailed maps are provided in Annex 5). The deliberate selection of the most vulnerable communities by NEMO assisted this effective and equitable targeting. Figures 5 to 8 show the locations of target communities across the different island groups. Project resource constraints limited participation opportunities in some instances. For example, only a few VEMC members from each community were able to attend proposal writing training and most of the CbDRM, CbP and CbCERT training was targeted only at VEMC members, not the broader community. Figure 4. Location of participating communities (n=51)11 Community projects were initially designed during proposal writing training. Draft proposals were then taken back communities for broader discussion and finalising. Whilst this process may enable input from 11 ‘Eua belongs to the Tongatapu island group, however, as it is more remote, it has been reported separately. ‘Eua X 28 X 16 X 4 X 3 33 all stakeholders who attend community meetings, it is clear that decisions concerning the initial project focus area (water, food, EC upgrade) had already been determined firstly by AFP and TNCC during initial program design, and secondly that specific solutions (water tank, pump, fencing or nursery) had been decided by only a few VEMC members attending the proposal writing training. Additionally, marginalised groups who don’t attend community meetings would have had no opportunity to direct or provide input into the design. Whilst acknowledging this consultation deficit, the evaluation finds that in most cases, community projects have benefited both men, women, youth and the elderly and not added a great additional work burden on any one group. One strategy to improve the community project proposal template could be to ask applicants list the names of men, women, youth, elderly and people with disabilities that have been consulted to inform the community project design. The proposals should outline how the project’s planned outcomes will benefit each group and also seek to identify any addition work burdens or risks that any specific group may face as a result of the project. Disability inclusion Activity attendance sheets and interviews provide evidence that at least 73 people with disabilities participated in program activities. 12 The evaluation was unable to determine what percentage of people with disabilities were engaged. “VEMC share knowledge on what they were taught to do and people start seeing that and joined in doing the same thing. Community becomes more active and start to work together. Like for a person like me to join it is a change!” Person with disability, Fotua The engagement of people with disabilities in the Program was limited to attendance and participation in activities. The evaluation found no evidence of people with disabilities being represented on VEMCs. In Tu’anekivale, the carer of person with disability was included in the VEMC and they voiced their opinion that programs like TCDRM needed to be more inclusive of people with disabilities in reference not to just his son who experienced mental and physical impairments, but other people with disabilities in his community. Two communities that received funds to renovate their EC were visited. Both had wheelchair access into the building, however, there were no accessible toilets in either facility. Noting the cost of toilet facility upgrades can be high, an alternative that could be considered is the provision of portable toilets in ECs for people with disabilities. These portable toilets can be easily and cheaply constructed and they have been used in disaster response situations in Vanuatu (See figure 5). 13 Figure 5. Portable toilet Engagement of Naunau 'o e 'Alamaite Tonga Association (NATA) to provide a direct and first hand voice of people with disability in national level consultations was valuable improvement to the Program’s 12 This is an estimate based on the data readily available. Actual numbers are likely to be higher. 13 Also see Special Needs Toilets in ‘A practical guide to building and maintaining toilets in the Pacific’ https://livelearn.org/what/resources/clean-communities-practical-guide-building-and-maintaining￾toilets-pacific 34 strategy to be more disability inclusive. With additional funding this engagement could be more effective at the community level and NATA’s presence in the community may help further shift behaviours and attitudes towards people with disabilities, helping to transform the lives of people with disability. Community project proposals required applicants to demonstrate how community projects would equitably engage women, men, people with a disability and the elderly. Responses to this question reflected the gendered nature of work in communities where men would undertake most of the physical work and women would prepare food to feed the men. Elderly and people with disabilities were to be assigned supervisory roles. These responses were appropriate when considering the Tongan culture, however, they likely fail to reflect the desires of external actors who are expecting projects to result in transformation and shift in gender roles and taking a strengths-based approach to inclusion where people with disability are more actively involved in program activities. Exploration of the Fiji disaster￾inclusive CbDRM Toolkit (mid-term evaluation recommendation) could be undertaken to advance progress in this area.14 Programs with serious intentions of meaningful inclusion of people with disabilities need to ensure that these intentions are resourced with sufficient funding and skilled staff to provide the support required to facilitate engagement. An overarching inclusion strategy for the program and individual inclusion plan for each community that identifies people with disability, grades them appropriately using the Washington Group Short Set questions and then responds with reasonable accommodation measures to encourage and facilitate their inclusion in activities should be considered in future programming.15 Response measures may mean additional costs for aids and services and these need to be budgeted for. 4.5 What unexpected positive or negative outcomes have been identified? The evaluation uncovered a number of positive unexpected outcomes that were achieved as a result of program activities. No negative unexpected outcomes were found. Whilst the role of the VEMC is focused on DRR/DRM, it was noted that in Mataika in Vava’u, that the VEMC had also taken on the local Village Security Committee (VSC) role to prevent the sale and distribution of drugs in the local community. In Matafonua, the VEMC secretary along with the Town Officer and assistant chairperson (female) all stepped up to fill a community governance vacuum resulting from the absence of a Community Council being elected. Several other VEMCs reported that they were active in keeping the community clean. The Program team also shared stories of VEMC members starting or reinvigorating existing committees within communities. “People become more resilient, more prepared, more involved in community work, more active.” Fotua community member The program was also found to have increased community cohesion across religious denominations and between neighbours. The SIMEX and community projects were noted as facilitating better relationships and brining community members together. In some instances community members commented that 14 Fiji Disaster Inclusive Community Based Disaster Management Toolkit http://www.pacificdisability.org/getattachment/Resources/PDF-Resources/PDF-Fiji-Toolkit￾2013.pdf.aspx 15 Washington Group Short Set questions for disability assessment: http://www.washingtongroup￾disability.com/washington-group-question-sets/short-set-of-disability-questions/ 35 the VCA transect walks were the first time they had walked around certain parts of their own village even though they had lived there their entire life. 4.6 How effective has the TCDRM program been in meeting the needs of communities? The TCDRM program has contributed to meeting the priority needs of all target communities. TCDRM proposal documentation highlights Tonga’s exposure to frequent and devastating natural disasters. Remote and less developed rural villages are highly vulnerable to the impacts of natural disasters and their location can mean that national disaster response measures take days or weeks to reach them. These factors highlight the importance of local community capacity to act before, during and after disasters to reduce disaster risks and lessen the impact of natural disasters. The program clearly targets DRR through creating relevant government legislated structures (VEMCs), plans (VEMPs) and building the capacity of VEMCs and the broader community to action the VEMPs. Community feedback supported evidence collected in the desktop review that indicates community projects (water security, food security and evacuation centre improvements) all responded directly to the needs of the communities. For example, a Matamaka woman confirmed that the program’s renovation of their EC was the community’s number one priority. Water projects were very relevant to communities given the existing low levels of investment and infrastructure in this areas and recurring water shortages. The food security projects were also highly relevant, especially considering the subsistence nature of life in the remote communities. Whilst not conclusive in its application across all projects, there was evidence found that community projects addressed DRR actions outlined in VEMPs, thus demonstrating relevance. The year-by-year nature of the OFDA funding created a situation where the TCDRM program needed to decide upon the number and nature of small community project interventions prior to engaging with communities to ensure projects could be delivered within the nominated budget. Whilst the three thematic sectors (water, food, infrastructure - EC improvement) were all relevant and addressed community needs, this approach is not best practice and a program designed in this manner risks not being able to address priority needs. Future TCDRM programs should seek to secure multi-year funding windows to support timeframes which allow for initial community consultation to identify and address community’s priority needs. Community needs in each of the target areas have not been fully met by the small projects delivered. This finding reflects the existing low levels of investment in these areas and the high cost of goods and services provision in remote areas, set against the very limited funding allocated for each small community project (TOP 25,000). For example, whilst water storage capacity was tripled in some communities, community members still provided feedback that storage capacity was insufficient to provide water in periods of extended drought. It is not uncommon for communities to request additional assistance and only a thorough community-level WASH analysis can accurately determine what community water storage capacity needs are. With additional funding, the initial VCA could be deepened and extended to include community-level water-security assessments that would result in medium to long term plans to meet community needs. Actions from these plans could be included in overarching Community Development Plans as a means of incrementally making progress towards fulfilling community needs. Water security plans also serve as a reference point for future development programs. Implementing partners can refer to the plans to see 36 the current needs and then invest accordingly to address priority needs without spending time and funds conducing their own assessments. The plans also act as a means to track the contributions that have been made by past projects so that implementation partners have a history of what activities have preceded their investment and this information may influence where new funds are invested. National and district-level stakeholders also acknowledged the relevance and importance of the Program. The Program filled a gap in existing DRM capacity building at the community level and helped the GoT to implement components of the JNAP and National Disaster Management Plan. “TCDRM is the working arm to do this [establish VEMCs] .No other project was doing the smaller level on-ground work with communities.” NEMO CEO 4.7 Were appropriate technical standards used and achieved? The Program used and adhered to all of the relevant technical and Core Humanitarian Standards (CHS) it identified in its program design. The evaluation found the program implemented or achieved all six of its chosen CHS. Table 6 below summarises the standards and degree of supporting evidence found to confirm achievement with the standards. There was strong evidence found in achieving four standards and medium evidence found for the remaining two standards. More complete explanations of the standards and evidence found is documented in Annex 13. Table 6. Summary of achievement of selected CHS CHS # Core Humanitarian Standard Commitment Evidence score (High, Medium, Low, None) 4 Awareness of rights, program intentions and informed consent High 5 Confidential complaints Medium 6 Coordinated, complementary assistance Medium 7 Continual learning and improvement High 8 Competent and well-managed staff and volunteers High 9 Efficient use of resources, accountability and ethics High The program adhered to national construction standards. This was demonstrated by the use of a construction engineer from the Ministry of Infrastructure to assess EC improvement projects. This assessment helped ensure the ECs selected were structurally strong enough and appropriate to be classified as an EC that could withstand the impact of natural disasters. One EC assessed was deemed inappropriate and as project funds were insufficient to make the required structural improvements, thus the community project focus areas shifted from EC improvement to food security. WASH projects were well designed and the technical solutions implemented were appropriate for remote target communities. Key features of the design included a raised concrete foundation, 5,000 and 10,000 litre Rotomould rainwater catchments fitted with First Flush Diverter (FFD), debris and mosquito screen. The quality of the design and materials were deemed to be of high standard, including the Rotomould tank supplied by an ISO 9001 certified supplier with high quality materials guaranteed for 20 years and tanks that are certified to international quality standards (ISO Type 5: AS/NZS 4766). Water tanks foundations were designed and built to Tonga’s national construction guidelines as instructed by 37 the TCDRM WASH Engineer. The Tonga Water Board does not have any standards for rainwater catchments. 4.8 To what extent did the program implement recommendations from the mid-term evaluation in years 4 and 5 of the program The program has successfully implemented in full or in part the majority (two thirds) of the 31 recommendations from the Mid-term Evaluation (MTE). Figure 6 below shows the extent of evidence found that recommendations have been implemented. There was significant evidence that five recommendations had been implemented in full and a medium degree of evidence that an additional 14 recommendations were also implemented, but perhaps only partially. There was little evidence found to prove that 10 recommendations were implemented, however, four of these (orange bar) recommendations should have been allocated for other stakeholder to implement and thus TNCC should not be held accountable for their implementation. Additionally, some recommendations, whilst valid, would require additional resourcing to implement in full making them impossible for the TCDRM Program to implement during the funding period. The need for additional funding which was not available to the Program partially explains why several recommendation were likely not progressed and rated as ‘low’ in terms of supporting evidence of their implementation. See Annex 14 for a list of MTE recommendations and assessment of recommendation implementation. Figure 6. Evidence found to MTE recommendations were implemented 0 2 4 6 8 10 12 14 16 High Medium Low Unknown # recommendations Extent implemented by TCDRM in Year 4 and 5 TCDRM Other stakeholder 38 4.9 To what extent did the program validate the findings from the mid-term review? This evaluation determined that the findings of the MTE to be valid and still mostly relevant at the stage of the end-of-program evaluation. This evaluation found evidence to strongly support 31 (69%) of the MTE’s 45 findings. The evaluation supports an additional eleven findings and only one finding ‘Women and youth had some knowledge gaps’ was disputed by this evaluation. (See figure 7). Annex 15 contains a list of the MTE findings along with a rating of the extent to which this evaluation validates the findings with supporting comments and evidence to justify the rating provided. Figure 7. Support (validity) of MTE findings based on evidence from this evaluation. 4.10 How likely are project achievements to be sustained after the end of the program? Benefits achieved from community WASH, food security and EC improvement projects demonstrate a medium to high degrees of ownership and sustainability. Whist VEMC and community DRR/DRM knowledge and skills are likely to be retained, the active functioning of VEMCs and their application of VEMPs is less assured in the future unless there is ongoing support from NEMO. WASH infrastructure inspected (12 tanks over 4 communities) were all found to be in sound working order. The presence of FFD and debris screens greatly reduces maintenance helping to support safe and sustained use. Tank foundations showed no signs significant cracking or damage. Rotomould tanks were also found to be good quality with no signs of damage. Most tanks were connected to downpipes and observed to be full of water at the time of the evaluation. Water quality (transparency, smell, taste) was found to be high across all samples taken. The high value that the community members place upon the additional water storage capacity is a good predictor of ownership and with minimal risks of failure and minimal maintenance requirements, the evaluation finds the benefits of increased resilience to drought achieved are highly likely to be sustained into the medium to long-term (20 years). Improvements to food security through fencing in livestock and fencing out pigs and chickens were also observed to be continuing. No livestock were found to be outside of the fences and no feral and wild animals were found inside fenced plantations. Fences were found to be in good working order with no breaks in wire, wire being reasonably tensioned, adequately spaced and posts being reasonably sturdy and not rotten. The community at Ha’ano reported that there were several rotten posts that needed replacing and that these would be replaced in the future when the waterlogged soil had dried. Whilst no greenhouses created by the project were observed, the TCDRM team reported that all were still intact, maintained and being utilised to grow a range of vegetable seedlings, and cash crops such as kava and sandalwood. Sheep supplied through the TCDRM program were found to be well kept and breeding with numbers more than doubled in communities visited. The long term strategy to breed and distribute sheep to community members is highly likely to see these benefits sustained. Wild dogs that attack and Strongly support 69% Support 24% Not supported 2% Unsure 5% 39 kills sheep were noted as a threat to sustaining and growing sheep populations with some losses likely already experienced, but not confirmed. Two EC improvement projects were observed during the evaluation. Both structures appeared strong and solid with no signs of maintenance neglect inside or outside on roof and guttering. Water tank and toilets at Matamaka were all functioning. The water tap on the water tank at the Matafonua hall (EC) had a minor leak that needs attention. The gender segregated bathrooms at the Matafonua had no running water in the men’s toilet and no functioning wash basin. These issues could be addressed with minimal expenditure and local labour. A number of other community project outputs (water tanks and fences) were briefly inspected on-route to target communities and during weekend outings. All were found to be in good working order.16 VEMC status Knowledge tests conducted as part of the evaluation indicate that VEMC and community members retained a great deal of the core knowledge shared by the TCDRM program. The extent to which VEMCs were still operational varied between communities and it was difficult during the evaluation to uncover how active they were in reality. Some VEMCs reported to be having monthly meetings (Tu'anekivale), running community clean-up activities (Taoa) and running SIMEX independently of TNCC (Matamaka). Most VEMCs had not met since either the last cyclone (Matafonua, 'Ohonua) or cyclone warning (Muitoa). In Muitoa, some VEMC members said that the only time the VEMC works is when the TCDRM team visit the community. This indicates that there is a lack of leadership by the Town Officer, or that the Town Officer has too many other responsibilities to give the VEMC the necessary attention. This evidence calls into the question the long-term sustainability of VEMCs in some communities. The important role of the Town Officer in sustaining the VEMCs has already been noted and this is re￾emphasised here. The turnover of VEMC members in response to migration and travel is also an issue that impacts the effectiveness and sustainability of VEMCs. Without processes in place to train new VEMC members, the benefits of the Program will be reduced over time. Town Officer turnover presents serious risk to VEMC and VEMP sustainability Town Officer elections are held every four years in Tonga and whilst many remain in their positions for several terms, new Town Officers are eventually elected. Town Officers indicated that there is no current induction or handover process for new Officers. This lack of induction and handover presents a serious risk for the sustainability of VEMCs and maintenance and actioning of VEMPs. New Town Officers may not be aware of their DRR/DRM roles and responsibilities. They may also not be aware of the existing VEMP, nor possess the foundation knowledge and skills to lead VEMP revisions, especially specialty areas such as the identification of protection issues and protection mitigation actions. This issue needs to be raised with both MIA and NEMO to determine if any Town Officer induction/hand￾over policy or plans exist or need to be created to sustain DRM improvements made by the Program. Relationships between communities and GoT have not been sustained Any short-term improvements achieved in the relationship between communities and national and sub￾national Government have not been sustained. The advancement and introductions made through the 16 Community fence at Fotua, Ha'ateiho and Fangeale’ounga in Ha'apai. Cement tank restoration at 'Utangake, Vava’u. New water tanks in Ha’alaufull in Mataika in Vava’u. 40 program need to be viewed as the first step on a longer journey to develop mature trusting reciprocal relationships. These findings indicate the challenge of sustaining behaviour change beyond program interventions and a range of new and complimentary measures may be required to support sustainability. These behaviour change theories, frameworks and tools are explored in more detail in the Recommendations section of this report. National commitment to CbDRM The Program’s use of nationally endorsed structures processes (VEMCs, VEMPs) and close working relationship with NEMO are foundations that support the potential for Program outcomes to be sustained. Specific support from NEMO required to sustain Program outcomes includes encouraging VEMCs to meet, progress the DRR action plan, keep VEMPs updated, and to plan and action SIMEX on an annual or biennial basis. This support work could be built into the annual workplans for NEMO district officers in advance of cyclone season. The Program made an official handover of CbDRM to NEMO in 2019 and NEMO expressed support for the CbDRM approach, however, delays in the finalisation and endorsement of the CbDRM Framework do not support this commitment. NEMO indicated the Framework finalisation would be revisited after the current cyclone season had ended. NEMO indicated they had plans to deliver the CbDRM approach in Tonga’s most remote islands, the Niuas (Niuafoʻou and Niuatoputapu) in the next financial year. Actioning this commitment would be a strong indication of ownership and buy-in and provide hope for the continued implementation of CbDRM. NEMO also indicated that they had received requests from other Tongatapu communities for assistance to create VEMCs. Additional funding and resourcing, especially for district-level staff, will be required for NEMO to begin and continue the rollout of the CbDRM approach and support communities that have already been targeted by TCDRM. For example, the NEMO ‘Eua staff reported they did not have access to a vehicle and lacked funding for community engagement work. NEMO staff may also benefit from community facilitation training to ensure staff have the skills to work with communities and communicate on their level. The commitment from other CSO to support CbDRM was not included in the scope of the evaluation. Proposal writing There is evidence that the benefits from increased capacity in proposal writing have been sustained and will deliver future positive DRM and community development outcomes for communities. The TCDRM team provided examples of successful VEMC-led funding proposals being developed to apply for government grants such as the Climate Change Trust Fund, and international funds through organisations such as the World Bank. “The proposal writing training was very useful and helps us a lot. Thank you to TNCC for their assistance. It has made a huge impact” VEMC Secretary, Motafanua Tongatapu 41 5 Conclusion The TCDRM Program built resilience and reduced the impact of natural disasters in 51 vulnerable communities in Tonga. The Program worked with communities through Government endorsed structures and processes to create VEMCs and VEMPs. VEMC capacity building and simulation activities helped ensure communities were better prepared for disasters with increased knowledge of what to do before, during and after a disaster being demonstrated. Community responses to emergency warnings and TC-Gita provided evidence that VEMCs activated and implemented some measures in their VEMPs. Whilst community members are still fearful of natural disasters, the improved DRM behaviours such as preparing a 72 hour bag and evacuating early to the EC, which is managed by the VEMC to reduce protection risks, helped community members to feel safer. There was little evidence that the program resulted in changes to DRM gender roles, however, there were highly equitable levels of participation of men and women within VEMCs and program activities. People with disabilities were also represented and improved attitudes towards them and caring for them during emergencies was noted. The increased water storage capacity within communities combined with improved food security and safer evacuation centres all contributed to increased community resilience to fast and slow onset disasters. The competency, motivation and support from the Town Officer (VEMC Chairperson) has been noted as a key determining factor in achieving strong or weak outcomes related to the VEMCs and community engagement. The dependency on the Town Officer is a critical program risk that can undermine the effectiveness and sustainability of outcomes. The absence of induction and DRM related handover for new Town Officers further puts the sustainability of these outcomes at risk. NEMO’s tentative plans to deliver CbDRM to a small number of communities is a positive sign of national ownership of the CbDRM approach, however, these positive signs are not matched by past behaviour, including delays in finalising and endorsing the National CbDRM Framework. Both NEMO and MIA will need to step up and engage with Town Officers and VEMCs to encourage the sustained functioning of VEMCs and annual revisions of VEMPs if these benefits are to continue into the medium to long-term future. 42 6 Full recommendations In response to the evaluation findings and feedback from the TCDRM team during a findings presentation workshop, the evaluation makes the following recommendations for consideration. Recommendations are colour coded according to their priority (High – green; Medium – blue; Low – orange). Category Recommendation Assigned to Priority (High Medium, Low) Enhance reach and uptake of DRR/DRM knowledge and skill and awareness of VEMP/VEMC Create drama performances to entertain and draw the bulk of community members to attend program activities. Performances would highlight key program messages, of which performers would be involved in understanding and developing. Two delivery models could be piloted: a) Local youth group model. Local youth groups are provided with key messages and some suggested content. They then improvise and develop locally contextualised drama performances that are performed within the community. Whilst youth may volunteer their time in return for food at rehearsal and performance time, significant time will likely be required to build presentation skills and oversee content to ensure key messages and not conflicting messages are presented. b) Partner with TNYC to form and train a youth drama group in each island group. This main group would travel with the TCDRM project team to deliver performances after the delivery of training and events. Strategic thought around timing could see a small teaser performance before events to engage community and increase participation at key TCDRM events such as simulations targeted at the entire community level. A longer performance could follow TCDRM training to reinforce messages and engage a wider section of the community. Drama performances should not replace existing TCDRM modules. Drama has been used successfully by other programmes in Tonga and across the Pacific to engage and educate communities. This recommendation would also likely increase youth engagement in activities. TNCC High Improve VEMPs Review VEMPs to ensure that DRR/DRM measures are included to reduce the risk TNCC Low 43 Category Recommendation Assigned to Priority (High Medium, Low) of livestock losses during disasters. These measures may include developing cyclone proof animal shelters or setting livestock free to escape a possible tsunami event. Review VEMPs to assess if community level water monitoring and self-managed water rationing (15 Litres per person per day) could be included as a DRR action to extend the number of days of access to water in drought conditions. TNCC Low Build TNCC delivery capacity Continue best practice of AFP-TNCC partnership approach AFP/TNCC High Build TNCC capacity in the use of behaviour change models, frameworks and tools to improve behaviour change outcomes at the community level. TNCC, AFP High Continue best practice of refreshing and building facilitation skills on TNCC field officers TNCC Medium Continue to engage external expertise through ToT model to build TNCC capacity in new and emerging areas of DRR/DRM/Protection/SPHERE/Climate Change. TNCC Medium Build TNCC capacity in M&E. Options may include: a) National TOT session with partners followed by targeted workshop for TNCC. b) Review TNCC M&E framework and tools to ensure clear baseline of existing knowledge, water and food security is established to better measure and demonstrate improvement or gaps post-engagement or at the end of the program. AFP/TNCC Medium Seek and take advantage regional training opportunities to continually upskill and embed best practices into the program team and related activities. TNCC, AFP Low Improve program design TCDRM field officers to engage other community stakeholders such as VEMC members and church leaders, in addition to the Town Officer, to increase awareness of planned activities in the community and in turn, increase attendance at activities. TNCC High Embed additional behaviour change strategies, frameworks and tools into the program design to improve behaviour change outcomes. Theories and frameworks could include the Theory of planned behaviour, The Capability Opportunity TNCC High 44 Category Recommendation Assigned to Priority (High Medium, Low) Motivation Behaviour (COM-B) model using the Behaviour Change Wheel, and Community Based Social Marketing.17 Example tools could include:  Commitments: Use of verbal commitments and signed public declarations displayed in town halls made by the Town Officer and VEMCs to meet monthly and serve the community. Photos of VEMC member and their roles could also be posted to create a sense of pride and obligation amongst VEMC members.  Prompts: Automated SMS reminders sent to VEMC members to meet and take action. Frontline SMS could support this. 18  Social Norms: Include annual simulation activities in VEMPs to create public displays of the desired new behaviours. Ensure respected community members such as church leaders model and promote the new behaviours. VEMCs to ensure they publicly model the desired behaviours so they are visible for others to observe and copy. Add a new module to engage and work with schools to deliver DRR/DRM education to school teachers and children. SIMEX could be run independently or integrated with TCDRM community SIMEX. TNCC Medium Make better use of radio, TV and social media channels to promote planned work at the community level and share project achievements. Engagement with community specific Facebook pages may help provide advanced notice of program activities in addition to relying on the Town Officer. TNCC Medium Review and update TCDRM training resources to continually improve content, integrate new best practices. TNCC Medium Commitments to gender equity, disability inclusion and compliances with CHS standards need to appropriately resourced with additional time allocated to community engagement if they are to be achieved to the extent documented in the Program design. Specific detail strategies around gender, disability inclusion TNCC Medium 17 Theory of Planned Behaviour: COM-B: http://www.behaviourchangewheel.com/about-wheel Community Based Social Marketing: https://www.cbsm.com/ 18 Frontline SMS: https://www.frontlinesms.com/ 45 Category Recommendation Assigned to Priority (High Medium, Low) and specific measures within CHS, such as a confidential complaints mechanisms, need to be created to ensure actions are taken to deliver on the commitments. Encourage VEMC members to share DRM knowledge with community members on a household basis instead of relying on town meetings. Allocated a ‘block’ or area of the village for each VEMC member to focus their communications efforts and spread news about upcoming DRM related activities and pass on DRM related knowledge and skills. TNCC Medium Two hardcopy VEMPs should be distributed to VEMC. One copy for the chairperson and one for the secretary or other active member. TNCC Medium Encourage Town Officers to include members from each key religious denomination in VEMC to assist with DRM knowledge sharing. TNCC Medium Create small religious DRM pack with sermon and key talking points and related resources for distribution to church leaders or church members on the VEMC to increase DRM knowledge sharing through churches. TNCC Medium All VEMCs should be provided with a hailer and set of high visibility vests. The vests could be NEMO or VEMC branded to give VEMC members a sense of pride and authority. NEMO, TNCC Low CSOs should draft or revise existing MoU with GoT to ensure the roles and expectations of both parties are clearly outlined and agreed to CSO (TNCC), NEMO Low Sustainability Create or enhance existing induction and hand-over processes for new Town Officers to ensure:  Awareness of TO responsibilities in relation to DRR/DRM.  Introduction to VEMC roles, purpose and TO responsibilities and hosting of an initial meeting to activate the VEMC.  Awareness of VEMP contents, value and purpose and need to update VEMP and share with relevant authorities. MIA/NEMO High Follow-up with TO on an annual basis prior to cyclone season to remind them to:  Call a VEMC meeting  Fill committee vacancies  Refresh committee on roles and responsibilities  Update VEMP, including contact details MIA/NEMO Medium 46 Category Recommendation Assigned to Priority (High Medium, Low)  Share updated VEMP with MIA and NEMO and other relevant stakeholders (Cluster lead organisations) Including these tasks in district-level NEMO officer work plans may help advance this recommendation. The desired outcomes could also be partially achieved through automated scheduled SMS push notifications and confirmations using Frontline SMS. Replication Continue running (improved) TCDRM program to achieve 100% coverage of vulnerable communities in Tonga. TNCC, NEMO, USAID Medium Explore South Pacific region for opportunities to fill gaps in CbDRM coverage and increase resilience of vulnerable communities to natural disasters. USAID, AFP Medium Create a CbDRM community course recognised by the Tonga National Quality Assessment Board to standardise the TCDRM approach and ensure minimum quality standards. TNCC, NEMO Medium NEMO to require all CSOs delivering CbDRM and creating VEMCs and VEMPs to become recognised community training providers and deliver, as a minimum, the contents of the CbDRM community course. NEMO Low Administrative Seek longer term (3 to 5 year) funding commitment from donor to enable better strategic planning of community engagement and help retain skilled TNCC staff over longer durations thus avoiding issues with turn-over. TNCC, USAID High Develop Theory of Change at the program inception and use this to design a targeted M&E Framework that goes beyond collecting data required by the funding agency. Additional indicators and monitoring will assist provide confidence in the achievement of planned outcomes in addition to the delivery of outputs. TNCC, USAID Medium 47 7 Annex 7.1 Program activities Program components and modules were delivered in a logical order. Key components and modules and their primary benefits are outlined in table 7 below. Table 7. Key program components Components Core benefits Consultations Initial introductions, gain buy-in and build trust. VEMC formation Create/strengthen community-level institution to lead community in DRR/DRM. VCA Practical activity exploring community vulnerabilities to various hazards and identification of local resources and knowledge to minimise risk. VEMP creation Key process and Plan that documents critical information of who, what and how of DRR/DRM for the community. CbDRM CbCERT Core foundation module to enhance and reinforce existing DRR/DRM knowledge. CbP Important, as it is often the first formal open discussion and introduction of this sensitive topic in the community. WASH training Due to the remoteness of communities and limited contact with Government services, it is sometimes the first time that communities have benefited from targeted information on the importance of good hygiene and sanitation. Food security training Expands upon existing traditional knowledge of food production and encourages the planting of more diversified and climate resilient crop species. Food preservation techniques are also high relevant to both DRM and the disaster recovery phase. Seed saving knowledge helps sustain low-cost vegetable production. First Aid training Highly valued for its applicability to emergency and everyday life situations such as treating sports injuries or responding to household emergencies Disability inclusive CbDRM People with disabilities are highly vulnerable in times of disaster. This module helped VEMCs to recognise the importance of prioritising people with disabilities in DRR/DRM. SIMEX Vital component for VEMC to action VEMP and demonstrate practical elements of other modules (search & rescue, First Aid training). Vital for sharing VEMC knowledge and skills with the broader community as most community based capacity building was attended only by VEMC members. Proposal writing Increased capacity to create higher quality project proposals aids continual community development which may include the funding of longer-term DRR actions such as building EC or increasing community water storage. Blessing/opening ceremony Facilitate a degree of awareness to community members not engaged by any other component of the program. Another opportunity to build relationships between communities and GoT stakeholders. 48 7.2 Terms of reference for evaluation EVALUATION TITLE Evaluation of the ‘Tonga Community-Based Disaster Risk Management’ (TCDRM) Program COMMISSIONING AGENCY Act for Peace TIMELINE OF EVALUATION February to April 2019 EVALUATION MANAGER Yvette Crafti, Disasters and Emergencies Program Coordinator Background to the Evaluation Disasters present an ongoing and increasing threat to the people of the Pacific region, with Tonga remaining the second most at-risk country from disasters worldwide.1 Act for Peace and TNCC have jointly delivered community-based disaster risk management (CBDRM) programs over the past eight years across the country, which have mitigated the human and economic impacts of subsequent disasters, especially for the most vulnerable people. The CBDRM model, including establishment of local disaster management committees, design of emergency management plans and facilitation of simulation exercises, is broadly accepted to be “of paramount importance” in Pacific island communities, due to the isolation of many communities and level of exposure to disaster risk.2 The Tonga Community Disaster Risk Management (TCDRM) Program3 , funded by USAID/OFDA since 2013, has supported the Government of Tonga to use the CBDRM model in order to equip remote Tongan communities with the necessary tools to be self-sufficient, adequately prepared and resilient in times of disaster. Now in its fifth and final year of USAID/OFDA-funded programming, Act for Peace is seeking to evaluate the effectiveness of the five-year program. An evaluation was conducted at the end of the third year, and this evaluation will expand on knowledge acquired at that time. The benefits of CBDRM in Tonga, as an approach, were evidenced by the extraordinarily low rates of injury4 and high capacity for immediate relief and recovery efforts after Tropical Cyclone (TC) Gita impacted the Tongatapu and ‘Eua islands in February 2018, in contrast with losses experienced following and TCs Ian (Ha’apai, January 2014) and Winston (Niuas, February 2016). All communities in the most affected areas that had participated in the TCDRM program, had active Village Emergency Management Committees (VEMC), had reviewed their Village Emergency Management Plans (VEMP) and had undertaken disaster risk reduction initiatives through TCDRM projects reported being better prepared as a result of training received. In particular, disaster response simulation exercises – conducted as part of the TCDRM project activities to test and practise the new learning – have been recognised by targeted communities as highly valuable in achieving this enhanced level of preparedness and resilience.5 Natural disasters in Tonga and across the Pacific more broadly6 have highlighted the interconnectedness of disaster risk reduction and food security at the community level, with significant need for food relief distribution in response phases, and restoration of food gardens, agricultural crops and livestock in early recovery stages. Since 2016, Act for Peace has continued to improve community level food security, which was integrated into the CBDRM approach for TCDRM 4 and 5. The food security interventions incorporate household and community level assessment, training and skills development for communities and government extension staff, along with implementation of initiatives at the community level to improve food security such as fencing or community greenhouses. Equally, disasters in Tonga generally result in loss of access to safe and sufficient water supplies, in both the immediate and longer term. Post-cyclone contamination, isolation from shops, disruption to supply chains, and damage to coconut trees are some of the contributing factors impacting reliable water provision. TCDRM 4 and 5 expanded access to potable water in targeted settlements. Prolonged dry spells and subsequent water shortages in recent years have impacted agricultural outputs, household food security and nutrition for many countries in Tonga and further afield. More broadly, disasters can expose a range of vulnerable people within communities to increased protection risks. Such risks could include violence, neglect or exploitation, or may relate to loss of housing, land or property. Some individuals, including women, children and the elderly, may be disproportionately exposed to increased harm following disasters due to their status within the community and ability to defend or represent themselves. Disasters exacerbate the already high levels of violence against women and children in Tonga (predominantly domestic violence), an issue that is often not sufficiently addressed in DRM programming. In response, TCDRM 5 strengthened its emphasis on community-based protection for VEMC members participating in CBDRM training. Disability-inclusive CBDRM training was also amplified during TCDRM 5, and included a member of a Tongan Disabled People’s Organisation (DPO) travelling with the TNCC facilitation team and delivering training to the 11 VEMCs participating in CBDRM training. 49 In its fifth and final year of USAID/OFDA-funded activity, TCDRM 5 focuses on its exit strategy and handover to the Government of Tonga National Emergency Management Office (NEMO). It will build the capacity of local government officials to provide ongoing technical and organisational DRR support to communities through local government officials’ participation in the Project at local and national levels. For five years, TCDRM has supported NEMO to continue expanding its CBDRM approach, thereby improving disaster resilience for all Tongan communities not covered by TCDRM. Additionally, AfP and the TNCC have cultivated a strong relationship with NEMO over several years of co-facilitation and coordination, planned to actively advocate to the Government of Tonga to finalise the National Framework for Community-based Disaster Risk Management, formalising the Project methodology in national plans. Tonga’s Joint National Adaptation Plan on Climate Change Adaptation and Disaster Risk Management (JNAP 2) is aligned to the priorities of the Tonga Strategic Development Framework 2015-2025 and reflects the Government of Tonga’s commitment to “improved national and community resilience to the potential disruption and damage to wellbeing, growth and development from extreme natural events and climate change”. The TCDRM program methodology includes:  Vulnerability and capacity assessments conducted in each regional location on commencement of the Project;  National consultations with key stakeholders in each of regional locations to promote the Project, seek collaboration and address queries;  Community consultations at village level to promote the Project, seek collaboration and address queries;  Community-based training workshops of between five and seven days for residents and their VEMCs focusing on DRM, community-based protection (CBP) in emergencies, access and inclusion planning for people with disabilities, food security and First Aid training (provided by Tonga Red Cross Society);  Disaster simulation exercises conducted by VEMCs with support from the Project Team to reinforce knowledge and skills and to test and improve community emergency response capacity.  Establishment of VEMCs, which are supported to assess vulnerability and food security and develop VEMPs.  For communities participating in the project a second time, refresher training to review their VEMPs, assess food security risks and develop action plans to improve resilience.  WASH training workshops for communities assessed to be eligible for water security grants based on their need;  Proposal writing workshops for communities that are eligible for small grants (water supply or food security);  Water supply projects approved by the Grants Advisory Group for communities that propose priority WASH interventions based on their VEMPs;  Food security projects approved by the Grants Advisory Group for communities that propose priority food security interventions based on their VEMPs;  Grant agreements signed by community representatives and TNCC;  Official blessing ceremonies in accordance with custom.  Presentation of evaluation findings and handover to the Government of Tonga on completion of the Program. TCDRM prioritises gender equality, disability inclusiveness and child protection safeguards. The Core Humanitarian Standard (CHS) informs TCDRM’s design, implementation and management. TCDRM attempts to demonstrate accountability to communities as well as donors and the Government of Tonga. In particular, TCDRM 5 sought to incorporate those CHS commitments which relate to non-response phase humanitarian interventions. The project objectives are as follows: Project objective 1: To support the Government of Tonga to reduce the impact of natural disasters on communities, especially the vulnerable. Project objective 2: To strengthen the capacity of local communities to identify and assess their own protection risks and to develop strategies and action to prevent, mitigate and respond to these risks. Project objective 3: To improve the community level food security of participating communities. Project objective 4: To expand water coverage in targeted settlements to improve access to safe and sufficient water, both in the immediate and longer term. 50 Purpose of the Evaluation The evaluation focuses on project effectiveness, specifically the extent to which the project has met its objectives. This includes the extent to which the capacity of VEMCs and local communities in Tonga to prepare for, mitigate, and respond to, disaster-related risks has been strengthened. A key element to this will be to understand how communities perceive their level of safety, including people with disabilities and other vulnerabilities. Specifically, the evaluation will determine the extent to which the TCDRM methodology improves community-level disaster preparedness, response capacity and resilience. This evaluation is being undertaken to determine effectiveness of the methodology and to make recommendations to the Government of Tonga, USAID/OFDA, Act for Peace, TNCC, VEMC members, Town Officers, community members, DPO staff and the Tonga Red Cross Society in relation to longer term commitments both in Tonga and other similar countries. Objectives of the evaluation The evaluation will primarily focus on effectiveness, relevance and sustainability. The following key evaluation questions and sub-questions will be used: Effectiveness 1. To what extent were the project objectives achieved? o To what extent have project outcomes, especially in TCDRM communities affected by TC Gita, supported the Government of Tonga to reduce the impact of natural disasters on communities, especially the vulnerable? o To what extent has the project strengthened the capacity of local communities to identify and assess their own protection risks and to develop strategies and action to prevent, mitigate and respond to these risks? o To what extent and in what ways has the project improved the community level food security of participating communities? o To what extent has the project expanded water coverage in targeted settlements to improve access to safe and sufficient water? Has this differed in the immediate and longer term? (compare communities who participated in years 1-2 with years 3-4) 2. What were the major factors influencing the achievement or non-achievement of the objectives? o How effective was the TCDRM methodology, and which specific aspects of the approach, if any, should be replicated or adapted for future programming? o How has the program promoted connectedness, both horizontal (strengthening both government and civil society structures and systems and the relationships between them) and vertical connections (strengthening linkages from national to sub-national and community levels.) o To what extent and in what ways have communities been empowered to participate in government decision-making in DRR as a result of the program? o In what ways do communities, including people with disabilities and other vulnerabilities, perceive their level of safety (before, during, after disaster) to have improved as a result of the project? 3. To what extent has achievement of project objectives been equitable (considering gender, disability and marginalisation/ remoteness)? Relevance 4. How effective has the TCDRM program been in meeting the needs of communities and what unexpected positive or negative outcomes resulted? (CHS 3) 5. Are appropriate technical standards (WASH SPHERE Standards, Food and Livelihoods SPHERE standards, and relevant National Standards such as construction standards) used and achieved? (CHS 2) Sustainability 51 6. How likely are project achievements to be sustained after the end of the program? Which achievements more sustainable than others? o Have people from communities targeted in years 1 – 3 retained knowledge and behaviour changes? o Are village emergency management committees that were established in years 1 – 3 still functioning and maintaining plans and equipment? o Have relationships between communities and decision-makers been sustained? Scope of the evaluation In scope: TCDRM covers the period July 2013 to February 2018. During this time TCDRM 1-5 covered the annual periods 1 July to 30 June. During TCDRM 5, a project suspension and two no-cost extensions were approved by USAID/OFDA allowing TNCC to respond to TC Gita and resume implementation after a break, meaning TCDRM 5’s conclusion date is 30 April 2019. On completion of TCDRM 3, a year 1-3 evaluation was undertaken by an external evaluator who reviewed the program and made a series of recommendations, most of which were addressed in the design and implementation of years 4 and 5. Therefore, the scope of this evaluation is to affirm the initial evaluation findings, assess the level and efficacy to which the recommendations were integrated into years 4 and 5 and to evaluate the quality and effectiveness of years 4 and 5. The evaluation will focus on the knowledge, skills, attitudes and behaviours, in relation to disaster preparedness and response capacity, of:  Communities which participated in years 4-5;  Communities participating in years 1-3 to evaluate the sustainbility of the approach over a longer period;  Comparitive evaluation of a control group of communities which did not partipcate in the program, in order to understand the overall effectiveness of the project;  The extent to which the project has directly or indirectly strengthened government-led disaster management and coordination processes. Stakeholders who will be consulted during this evaluation:  Act for Peace and TNCC staff  VEMC members and Town Officers  Community members  NEMO staff including provincially-based representatives  Representatives of DPO NATA, including the trainer who accompanied TNCC on all 11 CBDRM trainings in year 5.  Tonga Red Cross Society, which provided first aid training to VEMC members. The Evaluator will visit Ha’apai, Vava’u, ‘Eua and Tongatapu islands accompanied by TNCC staff who will act as facilitators and translators. It is not expected that all 51 VEMCs and their communities will be interviewed; rather, the consultant will develop a schedule together with TNCC and Act for Peace to identify approximately 10-12 communities to be visited during the period, noting access, logistics and weather-related constraints of cyclone season. As people with disabilities are significant stakeholders in the TCDRM program (from year 5), special attention will be given to emphasise the voices of people with disabilities and the eldery through participatory feedback mechanisms. Where a VEMC has not responded yet to a disaster, the evaluation will be limited to evaluating knowledge and skills acquired, but not be able to take into account the full extent of behaviour change in support of the project objective of reducing the impact of natural disasters. Out of scope: Tongan Government policy and performance is not being assessed in this evaluation, nor are the external stakeholders which have supported the project, such as the DPO or Tonga Red Cross Society. 52 Approach and methodology of the evaluation It is expected that the evaluation will use a mixed methods approach and an evaluation process that engages the project team. The methodology will be finalised in conjunction with the Critical Reference Group and Evaluator, but as a minimum will be required to make use of quantitative and qualitative data collection and analysis methods, including participatory data collection techniques where possible. The evaluation methodology will include:  A review of documents and project materials including previous evaluation report of years 1-3 and monitoring data from partner reports and records  Development of a program logic that demonstrates the project’s theory of change  Quantitative and qualitative primary approaches and data collection techniques  Field visits to community project locations in four districts (Tongatapu, Ha’apai, ‘Eua, Vava’u)  Key informant interviews and case study collection (other participatory data collection techniques also encouraged) with communities and stakeholders including Tongan Government Evaluation management arrangements a. Evaluation Manager The evaluation manager is the Act for Peace staff member with overall responsibility for the management of the evaluation process including the establishment of a critical reference group, development of terms of reference for the evaluation, management of evaluator(s), stakeholders both in-country and in-Australia, and with responsibility for drawing up a response to the evaluation recommendations, and a suitable summary of the evaluation for internal and possible external communication. The evaluation consultant will communicate with, and submit reports to, the evaluation manager. Critical Reference Group Critical Reference Group members will provide advice and guidance on the planning, implementation and documentation of the program evaluation. This will include: providing relevant technical knowledge and expertise, identifying relevant documentation and literature to guide the evaluation, and providing feedback on draft tools/reports. Where appropriate, inclusion of Partner counterparts is encouraged. The Critical Reference Group members for this evaluation are:  Evaluation manager Yvette Crafti, Disasters and Emergencies Program Coordinator, Act for Peace. Identifying material for desk review, review draft report.  Meg Laufer, Performance Effectiveness Manager, Act for Peace, review draft methodology submitted by Lead Evaluator, review draft report.  USAID/OFDA – up to two M&E advisors based in Washington DC  Programs Director, TNCC Guiding Principles and Conditions The Evaluator will be required to adhere to, and be guided by, professional and ethical standards while contracted by Act for Peace. Evaluation managers are to ensure that evaluation team members and subcontractors have a copy of these standards. These standards include:  Act for Peace Code of Conduct  Act for Peace Child Safeguarding Policy and Code of Conduct  Australian Evaluation Society Guidelines for the Ethical Conduct of Evaluations.  Tolerance for and understanding of the role of culture and custom, including religious values adhered to by Act for Peace, TNCC and the people of Tonga.  Adherence to Core Humanitarian Standard commitments Conditions 53 The evaluation team members will be required to be aware of and compliant with Act for Peace Code of Conduct and Act for Peace Child Safe Code of Conduct. When in Tonga the evaluation team members will be required to adhere to security instructions and comply with the travel safety regulations. All documents and discussions relating to this evaluation are confidential and all products remain the property of Act for Peace and USAID/OFDA, to be shared only with their express approval. Evaluator An independent evaluator will be contracted to undertake the evaluation. Act for Peace is seeking to engage an experienced consultant with a minimum of 3 years or five published multi-year program evaluations, preferably in the field of disaster risk reduction, preparedness and response, and with experience in the Pacific. In consultation with the Evaluation Manager and the Critical Reference Group, the consultant will be engaged between mid-February and mid-April for a range of dates expected to be approximately 30 days, including travel to remote island locations in Tonga. The Evaluator will have experience working closely with local partners and have flexibility to adjust the program depending on factors such as weather conditions and ability to travel by boat/plane during the data collection period. The Evaluator will also have experience working with translators. Please note, flights, accommodation and local transport will be arranged and paid by Act for Peace and TNCC. The consultant will invoice Act for Peace for consulting fees, airport transport, travel insurance, meals in- country, and communications costs on arrival, such as Tongan sim card. Act for Peace will not pay for vaccinations or other pre￾departure medical costs, incidentals such as water, or minor pharmaceutical purchases. Evaluation location and schedule for deliverables The desk review component and report writing will be home-based and the field component of the evaluation, including data collection, field travel and presentation of initial findings will take place in Tonga (travel in second half of March). The consultancy is expected to commence early March and be completed towards the end of April 2019. Final dates for field work will be determined in agreement between the consultant, Act for Peace and our partner the Tonga National Council of Churches. 54 An indicative timeframe for key activities for the evaluation is below: Task Latest date for completion Estimate number of days Responsibility Terms of Reference out to tender 19 Feb Act for Peace Due date for Expressions of Interest 26 Feb Consultant Selection & interview (phone) 1 March Act for Peace / Consultant Contracting 1st week March Act for Peace / Consultant Commence document review 2nd week March 2 Consultant Evaluation methodology finalised, including development of program logic 2nd week March 3 Consultant In-country field visits: data collection7 and analysis 3rd week March- 1 st week April up to 158 Consultant / Project staff Program team reflection and analysis workshop 2 nd week April / following field data collection 1 Consultant / Project staff First draft report submitted to Act for Peace 21 April 5 Consultant Feedback provide to consultant 26 April Act for Peace Final evaluation reportsubmitted 29 April 2 Consultant Invoicing 30 April Consultant Dissemination of evaluation report 1 May Act for Peace and USAID/OFDA 28 Deliverables The evaluation methodology should be finalised by the consultant and submitted to Act for Peace for review prior to the field visit commencing. The format of the evaluation methodology for review can be decided by the consultant and should include development of a program logic that illustrates the theory of change. The proposal should include the identification and sampling methodology for selection of target communities. The evaluation report should adequately cover all areas specified in the terms of reference and additional factors that affected the performance of the intervention. The evaluation report should cater for the intended readership and users. It should use clear language, be succinct, be clearly formatted with appropriate data visualisation graphics. The total length of the evaluation report should ideally be no more than 30 pages, excluding annexes. 55 7.3 Sources/bibliography TCDRM documentation reviewed:  TCDRM Proposal (Year 1 and 5)  Year 1, Milestone Reports (Milestone 1, 2, 3, 4, 5)  Year 2, Milestone Reports (Milestone 1, 2, 3, 4, 5, Final)  Year 3, Milestone Reports (Milestone 1, 2, 3, 4, 5)  Year 4, Milestone Reports (Milestone 1, 2, 3, 4, 5, Final)  Year 5, Milestone Reports (Semi-annual, annual)  Community Project Proposals (Faleloa, Toula)  Community Project Grant Agreements (Koloa, Okoa)  Tonga Community Disaster Risk Management Program (TCDRM) Proposal (Years 1 and 5)  National and Community Consultations & Official Launching of the TCDRM 5 Report, 2017  TCDRM Monthly Indicator Reporting (June to December 2017)  TCDRM 5 - Staff Monthly Activity Report (October, November, December)  Village Emergency Management Plans (Vaimaloo, Nga'unoho)  Vulnerability and Capacity Assessments of six Communities in Vava’u  Food Security Projects - Monitoring And Evaluation Report, TNCC 2017  Water Projects - Monitoring And Evaluation Report, TNCC 2017  TCDRM Project - Phase 4 & 5 - Short Summary of Funded Projects  Update Report on the 6 Communities Water Project in Vava’u, 2017  Feedback from NATA - Rhema Misa  TCDRM Year 1 – 5 Database  TCDRM Population database Other documents reviewed: Fangupo, S., 2018, ‘Tonga Community Disaster Risk Management (TCDRM) 5, Impact of the TCDRM program in relation to Tropical Cyclone Gita, TNCC Handicap International, 2012, ‘Disability Inclusive Community Based Disaster Risk Management - A toolkit for practice in South Asia’, Handicap International MEIDECC, 2016, 'Tonga Climate Change Policy - A resilient Tonga by 2035', Government of Tonga MEIDECC, 2018, 'Joint National Action Plan 2 on Climate Change and Disaster Risk Management (JNAP 2) 2018 - 2028', Government of Tonga MEIDECC, 2018, ‘Official Handover of the Tonga Community Disaster Risk Management Project from TNCC to NEMO’, Government of Tonga, http://www.gov.to/press-release/official-handover-of-the-tongacommunity-disaster-risk￾management-project-from-tncc-to-nemo/ Ministry of Finance and Planning, 2015, 'Tonga Strategic Development Framework 2015 - 2025', Government of Tonga Sphere Association, 2018, ‘The Sphere Handbook: Humanitarian Charter and Minimum Standards in Humanitarian Response’, fourth edition, Geneva, Switzerland, 2018. www.spherestandards.org/handbook Whitfield, S. 2016, ‘Project Evaluation - Tonga Community Disaster Risk Management Program June 2016’, Act For Peace WHO, 2013, ‘Water sanitation hygiene: What is the minimum quantity of water needed?’, https://www.who.int/water_sanitation_health/emergencies/qa/emergencies_qa5/en/ 56 7.4 Data collection tools Primary and secondary data to inform the evaluation were collected using a number of tools and methods documented in table 8 below. Table 8: Data collection tools Data collection tools Description Desk review A desk review will be conducted of key documents related to the program - both internal (design documents, implementation plans, and reports) and external (government plans, policies and strategies, etc.) Key Informant Interview (KII) Guides The KII guides will be used to focus and probe specific issues relevant to the program implementation. Focus Group Discussion (FGD) Guide The FGDs will be used to confirm findings from KII and gather perspectives from a broader range of community stakeholders. Story collection/Personal Testimony To bring to life the experience of those participating in the program, personal stories of change will be gathered and used to inform the evaluation. Transect walk/village walk/observations Program site visits will be used to gain an overall sense of each community and the context in which the program operates, as well as view program outputs, outcomes. A community member may be asked to guide the evaluator around the community and to point out what changes the program has supported. It may also be possible to observe if program outputs are maintained & being used as anticipated. Ladder of participation Assess the extent of women’s participation (not just representation) within VEMCs. Private Story At the end of some group processes such as FGD, participants will be asked to anonymously write down a short story or feedback or draw a simple picture (happy/ unhappy) on paper. Everyone is required to place their piece of paper in a bag so that no one is singled out as providing feedback outside of the group process. This is called their ‘private or secret story’ This feedback is collected by the evaluator and not directly shared with the client. It provides a means for shy people to provide feedback outside of the very structured community power hierarchy. It also provides more comfort for people to provide critical feedback that is often withheld due to fear of offending the client or program advocates in the community. Feedback can be translated at a later date by a third party not related to the program. Sixteen data collection tools have been attached to this report and submitted to Act For Peace and TNCC for their records. 57 7.5 Target communities and populations The table below lists all the communities targeted by TCDRM over five years. Community population summaries are also provided. Four maps follow the table and outline the location of the communities. Name of Communities Male Female Total persons TCDRM 1 COMMUNITIES Hofoa 498 479 977 Puke 369 362 731 Matafonua 85 86 171 Fatai 151 161 312 Kapa 34 35 69 Falevai 55 58 113 'Otea 64 61 125 Lape 13 11 24 Matamaka 75 72 147 Nuapapu 66 69 135 Ovaka 42 43 85 Taunga 20 19 39 Hunga 125 104 229 Ofu island 86 75 161 Olo'ua 67 73 140 TOTAL 1750 1708 3458 TCDRM 2 COMMUNITIES 'Utungake 57 151 160 Talihau 109 109 218 Okoa 128 138 266 Koloa 103 94 197 Holeva 62 54 116 Vaimalo 49 46 95 TOTAL 508 592 1052 TCDRM 3 COMMUNITIES Pangai 642 569 1,211 Hihifo 434 410 844 Holopeka 65 76 141 Fakakai 93 83 176 Felemea 74 63 137 Mo'unga'one 37 55 92 Pukotala 42 49 91 Ha'ano 61 54 115 Muitoa 17 20 37 58 'Uiha 228 187 415 TOTAL 1693 1566 3,259 TCDRM 4 COMMUNITIES Mangia 49 55 104 Houma 69 67 136 Holonga 267 186 453 Toula 204 207 411 Utui 147 155 302 Tu'anuku 174 139 313 Faleloa 163 179 342 Ha'afakahenga 52 50 102 Koulo 93 121 214 TOTAL 1218 1159 2377 TCDRM 5 COMMUNITIES Ta'anea 348 358 706 Tu'anekivale 257 230 487 Mataika 251 239 490 Taoa 236 260 496 Ha'akio 65 74 139 Fangale'ounga 89 85 174 Fotua 113 113 226 Lotofoa 213 197 410 'Ohonua 768 760 1,528 Angaha 199 202 401 Mata'aho 134 135 269 TOTAL 2673 2653 5326 59 Figure 8. ‘Eua communities Figure 9. Tongatapu communities 60 Figure10. Ha’apai communities Figure 11. Vava’u communities 61 7.6 Sampling methodology A stratified purposive sampling approach was used to choose the proposed community locations for site visits. Available time, travel distance, budget and transport logistics impacted the selection process. Input from AFP, TNCC and OFDA was also considered in the selection of communities whilst still allowing the evaluator the independence to make the majority of selection decisions to avoid bias. The following guidelines were applied in the selection process:  Minimum 1 communities per island group, with more representation from island groups where there were more target communities.  Minimum 2 more remote communities only accessible via boat.  Minimum of 3 communities who participated in Years 1 – 3.  Minimum of 5 communities who participated in Years 4 and/or 5.  Minimum 2 communities who participated in the mid-term evaluation.  Minimum 2 communities nominated by TNCC for achieving positive outcomes.  Minimum 2 communities nominated by TNCC for being more challenging and achieving fewer positive outcomes.  Minimum 2 communities chosen completely randomly.  Minimum 2 communities implementing projects in each of three focus sub-sectors (food security/water security/evacuation centre renovation).  Two non-targeted communities to form the control group spread across two island groups with one community having experienced a recent disaster event and the other community not having experienced a recent disaster event. 62 Community sampling Annex 5 documents all the communities targeted by the program. Table 9 below lists the communities selected by the sampling approach. Field visits will be conducted to these communities. Table 9. Communities selected to visit on field trip. Island Group Community Name Year engaged by TCDRM year? (Year) Year refresher training delivered (year) Grant project focus area Included in mid￾term evaluation Outer, more remote sub-island Tongatapu (4) Matafonua 1 2 EC Renovation Yes No ‘Eua (3) ‘Ohonua 5 - None No No Mata’aho 5 - None No No Ha’apai (16) Muitoa 2 3 WASH Yes Yes Ha’ano 4 - Food Security No Yes Fotua 5 - Food Security No No Foa CONTROL Vava’u (28) Matamaka 1 2 EC Renovation No Yes Tu’anuku 3 4 WASH No No Tu’anekivale 5 - WASH No No Taoa 5 - WASH No No Tefisi CONTROL 63 7.7 List of evaluation activities in communities The following table lists the key evaluation activities that took place in communities. Table 10. Summary of community consultations. # Type Island Community Target Male Female Young male Young females PLWD 1 FGD Vava'u Tu'anuku VEMC 2 6 2 KII Vava'u Tu'anuku CM 1 1 1 1 3 FGD Vava'u Tefisi CCM 13 4 KII Vava'u Tefisi CCM 1 2 5 FGD Vava'u Taoa VEMC 1 4 6 FGD Vava'u Taoa CM 13 7 KII Vava'u Taoa CM 2 1 1 8 FGD Vava'u Tu'anekivale VEMC 4 7 9 KII Vava'u Tu'anekivale CM 2 1 1 10 FGD Vava'u Matamaka VEMC 3 2 11 FGD Ha'apai Fotua CM 1 2 1 12 FGD Ha'apai Fotua VEMC 2 13 KII Ha'apai Fotua CM 2 2 2 14 FGD Ha'apai Ha'ano VEMC 2 3 15 FGD Ha'apai Ha'ano CM 5 2 16 KII Ha'apai Ha'ano CM 1 2 17 FGD Ha'apai Muitoa VEMC 2 4 1 18 FGD Ha'apai Muitoa CM 1 19 FGD Ha'apai Foa CCM 2 10 2 20 KII Ha'apai Foa CCM 1 21 FGD 'Eua Mata'aho VEMC 4 3 22 KII 'Eua Mata'aho CM 1 1 23 FGD 'Eua 'Ohonua VEMC 2 24 FGD 'Eua 'Ohonua CM 21 25 FGD Tongatapu Motafonua CM 5 26 FGD Tongatapu Motafonua VEMC 1 2 27 KII Tongatapu Motafonua CM 1 1 Community consultations totals 37 108 6 6 5 Legend: KII Key Informant Interview FGD Focus Group Discussion CM Community member CCM Control Group community member VEMC Village Emergency Management Committee members PLWD People Living With Disability 64 7.8 List of evaluation informants Table 11. List of community members consulted # Consultation Island Community Name Role Classification Gender Age Person with Disability Impairment 1 FGD Vava'u Tefisi Palasa Fainu CCM F 50 2 FGD Vava'u Tefisi Sailolo Tauondati CCM F 57 3 FGD Vava'u Tefisi Lasina Faiuie CCM F 41 4 FGD Vava'u Tefisi Nehusita Malaefo'ou CCM F 42 5 FGD Vava'u Tefisi Ofa Mahiue CCM F 48 6 FGD Vava'u Tefisi Susisa Halalo CCM F 57 7 FGD Vava'u Tefisi Mele Katoa CCM F 44 8 FGD Vava'u Tefisi Uheina Fainu CCM F 48 9 FGD Vava'u Tefisi Puol Ra CCM F 48 10 FGD Vava'u Tefisi Via Vea CCM F 44 11 FGD Vava'u Tefisi Nita Lawnlau CCM F 63 12 FGD Vava'u Tefisi Pala Vea CCM F 22 13 FGD Vava'u Tefisi Kalisi Poliaki CCM F 43 14 KII Vava'u Tefisi Sesilina CM F 40 15 KII Vava'u Tefisi Ola Fehi Fuina Town Officer M 16 KII Vava'u Tefisi Tokilupe Afenui weaving F 35 17 FGD Vava'u Tu'anekivale Ailini Teumohenpa VEMC F 41 18 FGD Vava'u Tu'anekivale Lihai Taihia VEMC M 53 19 FGD Vava'u Tu'anekivale Paea Noje VEMC F 46 20 FGD Vava'u Tu'anekivale Aotoni Tifita Chairperson VEMC M 62 21 FGD Vava'u Tu'anekivale Kelepi Nafe VEMC M 50 22 KII Vava'u Tu'anekivale Esafe CM M 27 Yes Right eye closed, Moderate developmental disability 65 23 KII Vava'u Tu'anekivale Ema Sanft CM F 33 24 KII Vava'u Tu'anekivale Toni Fotu' CM M 19 25 FGD Vava'u Tu'anuku Polosapina Veekala VEMC F 47 26 FGD Vava'u Tu'anuku Alemoi Funou VEMC F 72 27 FGD Vava'u Tu'anuku Manunu Mafustaimi VEMC F 45 28 FGD Vava'u Tu'anuku Emeli Ngalo VEMC F 49 29 FGD Vava'u Tu'anuku Heamon Finau Chairperson VEMC M 75 30 FGD Vava'u Tu'anuku Lausiva Taukei VEMC F 53 31 FGD Vava'u Tu'anuku Fe'ofa'aki Nimo VEMC F 35 32 FGD Vava'u Tu'anuku Suli Ma VEMC M 38 33 KII Vava'u Tu'anuku Tevita Lolohoa CM M 90 Yes Cane to assist walking 34 KII Vava'u Tu'anuku Anaseini Halaieafu CM F 30 35 KII Vava'u Tu'anuku Mele Taukei CM F 20 36 KII Vava'u Taoa Viliami Tuitupu CM M 58 37 KII Vava'u Taoa Makeleta Mamata CM F 22 38 KII Vava'u Taoa Sione Misale Latu CM M 65 Yes Damaged foot, difficulty walking, Needs crutcher 39 FGD Vava'u Taoa Kilipi Kiokata Chairperson VEMC M 66 40 FGD Vava'u Taoa Vasil 'Ali CM F 69 41 FGD Vava'u Taoa Keiti 'Ali CM F 39 42 FGD Vava'u Taoa Ofa Sonipata CM F 42 43 FGD Vava'u Taoa Kalisia Finou CM F 36 44 FGD Vava'u Taoa Lpe Sali CM F 50 45 FGD Vava'u Taoa Eiela Sinipata VEMC F 55 46 FGD Vava'u Taoa Vai Mamafa VEMC F 44 47 FGD Vava'u Taoa Etimifaua Asikia CM F 57 48 FGD Vava'u Taoa Silivia Rea'kia VEMC F 43 49 FGD Vava'u Taoa Mele Funaki CM F 56 50 FGD Vava'u Taoa Maali Latu CM F 69 66 51 FGD Vava'u Taoa Emiline Funaki VEMC F 76 52 FGD Vava'u Taoa Esefi Lau CM F 65 53 FGD Vava'u Taoa Overema 'Tima CM F 35 54 FGD Vava'u Taoa Ani Vaumelan CM F 35 55 FGD Vava'u Taoa Mele Latu CM F 46 56 FGD Vava'u Taoa Pua 'Ali CM F 64 57 FGD Vava'u Matamaka Latu Tauf Chairperson VEMC M 58 FGD Vava'u Matamaka Siiu VEMC F 59 FGD Vava'u Matamaka Susiana Feaomoeata VEMC F 52 60 FGD Vava'u Matamaka Palia Haven VEMC F 61 FGD Vava'u Matamaka Tukula Inamo CM M 64 62 FGD Ha'apai Fotua 'Alifeleti Lounoa CM M 43 63 FGD Ha'apai Fotua Makasini Latu CM F 53 64 FGD Ha'apai Fotua Haisaane Lup CM F 49 65 FGD Ha'apai Fotua Sita Mohetku Treasurer VEMC M 60 66 FGD Ha'apai Fotua Lami Mothetan Secretary VEMC M 40 67 KII Ha'apai Fotua Mangisi Kalafi Tupou CM M 55 Yes Difficulty walking & a bit slow cognitavely 68 KII Ha'apai Fotua Sione Kualouku CM M 30 69 KII Ha'apai Fotua Lioni Kualouku CM F 30 70 KII Ha'apai Fotua Sharon Tu'ineau CM F 33 71 KII Ha'apai Fotua Sione Lavaka Takelo CM M 18 72 KII Ha'apai Fotua Mosese Langi CM F 18 73 FGD Ha'apai Ha'ano Fine Tu'akoi CM M 33 74 FGD Ha'apai Ha'ano 'Aisoa Muli Deputy Chair VEMC M 32 75 FGD Ha'apai Ha'ano Suhieti Muli CM F 40 76 FGD Ha'apai Ha'ano Heleane Lepolo CM F 61 77 FGD Ha'apai Ha'ano Mele Havea CM F 41 67 78 FGD Ha'apai Ha'ano Nuna Veatupu CM F 41 79 FGD Ha'apai Ha'ano Ana Tolu CM F 43 80 FGD Ha'apai Ha'ano Kololina M CM F 42 81 FGD Ha'apai Ha'ano Latai CM F 34 82 FGD Ha'apai Ha'ano Halamehi Langi CM F 30 83 FGD Ha'apai Ha'ano Sione Lifila CM M 69 84 FGD Ha'apai Ha'ano Ofa Sopauvalu CM M 43 85 FGD Ha'apai Ha'ano Lisali Laugi Chairperson VEMC M 44 86 KII Ha'apai Ha'ano Mahina Tua Loi CM F 33 87 KII Ha'apai Ha'ano Semisi Tuitavaki CM M 25 88 KII Ha'apai Ha'ano Tautali Fuapau CM M 24 89 FGD Ha'apai Muitoa Lesieli Latu Women's leader VEMC F 54 90 FGD Ha'apai Muitoa Tangikina VEMC F 26 91 FGD Ha'apai Muitoa Valinli CM F 36 92 FGD Ha'apai Muitoa Tanita Langi VEMC F 42 93 FGD Ha'apai Muitoa Saia Laug Chair VEMC M 44 94 FGD Ha'apai Muitoa Falefo'ou Langi Deputy Secretary VEMC F 38 95 FGD Ha'apai Muitoa Sepiuta Luninaoaku VEMC F 48 96 FGD Ha'apai Muitoa Anamaki Tamiup Secretary VEMC F 46 97 FGD Ha'apai Foa Foa Tapa women ** CCM F 40 98 FGD Ha'apai Foa Foa Tapa women CCM F 40 99 FGD Ha'apai Foa Foa Tapa women CCM F 40 100 FGD Ha'apai Foa Foa Tapa women CCM F 40 101 FGD Ha'apai Foa Foa Tapa women CCM F 40 102 FGD Ha'apai Foa Foa Tapa women CCM F 40 103 FGD Ha'apai Foa Foa Tapa women CCM F 40 104 FGD Ha'apai Foa Foa Tapa women CCM F 40 105 FGD Ha'apai Foa Foa Tapa women CCM F 40 106 FGD Ha'apai Foa Foa Tapa women CCM F 40 68 107 FGD Ha'apai Foa Foa Tapa women CCM F 30 108 FGD Ha'apai Foa Foa Tapa women CCM F 30 109 FGD 'Eua Mata'aho Kaneisini Lu'au Secretary VEMC F 49 110 FGD 'Eua Mata'aho Eseta Lu'au Treasurer VEMC F 60 111 FGD 'Eua Mata'aho Lalesita 'Uluheua Deputy treasurer VEMC F 42 112 FGD 'Eua Mata'aho Apai Lu'au Chair VEMC M 60 113 FGD 'Eua Mata'aho Manu Lau VEMC M 49 114 FGD 'Eua Mata'aho Alipate Sailosi VEMC M 67 115 KII 'Eua Mata'aho Fuifui Luau CM M 50 Yes Difficulty walking, needs a walking aid 116 KII 'Eua 'Ohonua Melani Mahe VEMC F 58 117 FGD 'Eua 'Ohonua Losi Lalah CM F 52 118 FGD 'Eua 'Ohonua Palanoa CM F 70 119 FGD 'Eua 'Ohonua Lup Lakau CM F 49 120 FGD 'Eua 'Ohonua Hheka'a Lapas'o CM F 35 121 FGD 'Eua 'Ohonua Paila Havea CM F 42 122 FGD 'Eua 'Ohonua Naomi Kaufis CM F 34 123 FGD 'Eua 'Ohonua Sela Katoa CM F 30 124 FGD 'Eua 'Ohonua Kalo Loni CM F 52 125 FGD 'Eua 'Ohonua Lolohea Vaka VEMC F 63 126 FGD 'Eua 'Ohonua Simaima Veituna CM F 65 127 FGD 'Eua 'Ohonua Ketilani Fatai CM F 33 128 FGD 'Eua 'Ohonua Suini Matai CM F 65 129 FGD 'Eua 'Ohonua Tilikitonga CM F 31 130 FGD 'Eua 'Ohonua Lupa Palu CM F 34 131 FGD 'Eua 'Ohonua Supeli Vea CM F 40 132 FGD 'Eua 'Ohonua Melevasa Tu'angalu CM F 33 133 FGD 'Eua 'Ohonua Pulonga Loumoli CM F 52 134 FGD 'Eua 'Ohonua Moana Sikaleli CM F 40 69 135 FGD 'Eua 'Ohonua Naomi Vailea CM F 50 136 FGD 'Eua 'Ohonua Talita Tavalea CM F 43 137 FGD 'Eua 'Ohonua Kilisita Loli CM F 40 138 FGD 'Eua 'Ohonua Lolina Kamalon CM F 59 139 FGD Tongatapu Matafonua Senolita Sekoana CM F 50 140 FGD Tongatapu Matafonua 'Ofa k Finau CM F 44 141 FGD Tongatapu Matafonua Niuola Hifanga CM F 35 142 FGD Tongatapu Matafonua Tonga Mieller CM F 47 143 FGD Tongatapu Matafonua Nifo Fale CM F 58 144 FGD Tongatapu Matafonua Lepolo Luaefe Secretary VEMC F 39 145 FGD Tongatapu Matafonua Talena Hausia Assistant Secretary VEMC F 57 146 FGD Tongatapu Matafonua Paulo H Muller Chairperson VEMC M 49 147 KII Tongatapu Matafonua Lelini Moala Youth committee secretary CM F 29 148 KII Tongatapu Matafonua Nito Moala CM F 58 ** Women in Foa FGD were very busy joining tapa cloth together and they were not inconvenienced to provide their names. Photo of the women is available on request. Legend: KII Key Informant Interview FGD Focus Group Discussion CM Community member CCM Control Group community member VEMC Village Emergency Management Committee members Nine community members are missing from the list when reconciled against the number of community members taking part in activities. Many names were illegible and the evaluator has taken a ‘best guess’ approach to interpreting the names. 70 Representatives from CSO, National and sub-national organisations consulted are documented below. # Island Organisation Name Role Gender 1 Act For Peace Yvette Crafti F 2 Act For Peace Geoff Robinson M 3 Tongatapu TNCC Siu'ivahamama'o Fangupo M 4 Tongatapu TNCC Mele Moimoi F 5 Tongatapu TNCC Asena Taufa F 6 Tongatapu TNCC Collin Mulikiha’amea Siale M 7 Tongatapu MEIDECC (NEMO is under MEIDECC) Mr. Paula Ma’u CEO M 8 Tongatapu Pacific Timber and Hardware (Main supplier of projects materials) Mr. Sifa Langi Mr. Naikano Paletu’a Estimator Store Manager M 9 Tongatapu Rotomould Pacific Ltd (Main supplier of plastic water tanks) Mr. John Rass Sales Manager M 10 Tongatapu NEMO Ms. Moana Kioa Senior Assistant Secretary (Officer in Charge) F 11 Tongatapu TNCC Rev. ‘Ikani Tolu Fr. ‘Atolo Tu’inukuafe Ms. Katea Lutui Ms. Latu Patolo TNCC Board M M F F 12 Tongatapu Private Worker Mr. Samiuela ‘Ofa Technical Engineer M 13 Tongatapu MAFF Ms. Moana Mafi Agriculture Officer F 14 Tongatapu Disability Organisation/Groups Mr. Rhema Misa Training Facilitator M 15 Vava’u NEMO Mr. ‘Inoke Taufa M 16 Vava’u Tonga Red Cross Society - Vava’u Branch Ms. Norie Lavaki F 17 Vava’u Vava’u Youth Congress Ms. Akosita Tu’a F 18 Vava’u TNCC Ma’u Kakala Halaifonua TNCC Country coordinator M 19 Ha’apai Ha’apai Youth Congress Ms. Teisa Fakatene F 20 Ha’apai Ministry of Internal Affairs Mr. Soakai Taungakava M 21 ‘Eua Tonga Red Cross Society - ‘Eua Ms. Meleane Mahe F 71 Branch 22 ‘Eua Government of Tonga Mr. Sunia Havea M 23 ‘Eua NEMO Mr Sealae Taufe ‘Eua NEMO officer M 72 7.9 Timetable The evaluation timeline is presented in the following tables. Task Date 1. Inception meeting 15th March 2. Create Evaluation Plan 22nd March 3. Document review & create program logic 22nd March 4. Develop data collection tools 25th March 5. Field trip (data collection: KII, FGD, observations, story collection). See detailed timeline below. 26th March – 13th April 6. Reflection workshop & Theory of Change validation 12th April 7. Draft evaluation report 23rd April 8. Finalise evaluation report 30th April Detailed field tripe timeline Date Time Activity Thursday 28th March, 2019 9am-4pm Meeting with key stakeholders in Tongatapu (venues and times to be confirmed for each stakeholder) Friday 29th March, 2019 9am-4pm Continue meeting with key stakeholders in Tongatapu Saturday 30th March, 2019 8am-9am Travel by plane from Tongatapu to Vava’u Leaves to the accommodations by rental cars and prepare for community interviews Sunday 31st March, 2019 Spend it at church and planning meeting if required Monday 1st April, 2019 9am Visit Tu’anuku by rental cars and conduct interviews Lunch 2pm Visit Tefisi by rental cars and conduct interviews then return to Neiafu Tuesday 2nd April, 2019 9am Visit Taoa by rental cars and conduct interviews Lunch 2pm Visit Tu’anekivale by rental cars and conduct interviews then return to Neiafu Wednesday 3rd April, 2019 8:30am Leaves by rental cars from each accommodation to the main wharf in Neiafu 9am Leaves on a boat from Neiafu’s wharf to Matamaka (outer island) approx. 1-2 hours depending on the weather 73 Conduct interviews in Matamaka Return back by boat from Matamaka to Neiafu 8pm Evening catch up meeting after Vava’u communities evaluation Thursday 4th April, 2019 9am-10am Travel by plane from Vava’u to Ha’apai 11pm-12:00pm Leaves from the airport to the accommodations by rental cars 2pm-4:00pm Visit Fotua by rental cars and conduct interviews. Friday 5th April, 2019 7:30am Leaves from the accommodations by rental cars to Faleloa (last village where a wharf is located) 8am Leaves on a boat from Faleloa’s wharf to Ha’ano (outer island) approx. 40mins-1hour depending on the weather 9am Conduct interviews in Ha’ano Lunch 2pm Visit Muitoa (next village to Ha’ano) by a motor vehicle and conduct interviews then return back to Ha’ano Return back by boat from Ha’ano to Faleloa, and by rental cars to respective accommodations Saturday 6th April, 2019 9am Visit Fotua by rental cars and conduct interviews Lunch Do other stuff as people will be busy with their Saturday routines in preparing for Sunday 8pm Evening catch up meeting after Ha’apai communities’ evaluation Sunday 7th April, 2019 All day Spend it at church and do other stuff Monday 8th April, 2019 9am Visit Foa by rental cars and conduct interviews Lunch 4:10pm-5pm Travel back by plane from Ha’apai to Tongatapu Spend one night in Tongatapu Tuesday 9th April, 2019 9:10am-9:25am Travel by plane from Tongatapu to ‘Eua Lunch 2pm Visit ‘Ohonua by rental cars and conduct interviews then return to accommodations Wednesday 10th April, 2019 am Visit Mata’aho by rental cars and conduct interviews Lunch 3pm-3:15pm Return back by plane from ‘Eua to Tongatapu Thursday 11th April, 2019 9am Visit Matafonua by motor vehicles and conduct interviews 74 Lunch 2pm Preparation for the presentation of the evaluation findings to all the stakeholders Friday 12th April, 2019 TBC Presenting of the Evaluation findings to all the stakeholders Refreshment for the participants 8pm Dinner with TNCC Board members and Project staffs 75 7.10 Theory of Change Figure 12. Theory of Change developed with TNCC input 76 The theory of change contains a number of assumptions about how the Program believes change will happen. An analysis of the theory of change finds that the following change pathway elements and assumptions did not hold true or occur to the extent expected. ToC pathway elements that have not held true. Comments VEMC s share DRR/DRM/First Aid/WASH/Food security/Inclusion knowledge & skills with community The assumption that Town Officers and VEMCs would share knowledge with community members has not held true in some communities. This was more evident in larger communities and communities with less active Town Officers and low attendance rates at community meetings. This is explored more in section 4.2 of the report. VEMP actioned The distribution of hardcopy VEMPs were delayed until they were handed over to NEMO in some instances which meant that VEMCs could not action the plan. Some Town Officers appeared to rely on past experience and not refer to the VEMP. Improved relationship and communication between GoT and communities Without TCDRM facilitation of GoT visitations, no continued GoT visitations or increased communication between GoT and communities was noted. National CbDRM Framework endorsed by GoT & includes community input and responds to Gap Analysis After three years, the draft Framework is yet to be endorsed. This is explored more in section 4.10 of the report. GoT & other partners fund and implement best practice CbDRM approaches The absence of a finalised and endorsed Framework has reduced the achievement of this outcome. Whilst some CSOs are delivering what they claim to CbDRM, examples provided by the TCDRM team indicate there are numerous quality issues that could end up creating more harm than good in communities through other CSOs implementing insufficient CbDRM programs. 77 Figure 13. Simplified Theory of Change 78 7.11 Water storage volume increases The following table provides a summary of water storage volume increases achieved across thirty two communities, plus one community that benefited from a pump. Table 12. Water storate capacity increases Additional days water supply Village Additional capacity (Kilolitres) Village population Emergency situation (15 Litres/person/day) Drought situation (7.5 Litres/person/day) MATAMAKA 5 147 2 5 FAAKAKAI Water pump 176 MATAFONUA 10 171 4 8 FALEVAI 25 113 15 29 FELEMEA 55 137 27 54 HA'AKIO 35 139 17 34 HA'ALAUFULI 40 unknown HOFOA Water pump 977 HOLEVA 55 116 32 63 HOLONGA 35 453 5 10 HOLOPEKA 190 141 90 180 HOUMA 45 136 22 44 HUNGA 35 229 10 20 KAPA 20 69 19 39 KOLOA 55 197 19 37 MANGIA 35 104 22 45 MATAIKA 50 490 7 14 MUITOA 50 37 90 180 NGA'UNOHO 10 218 3 6 NUAPAPU 35 135 17 35 OKOA 60 266 15 30 OLO'UA 30 140 14 29 'OTEA 30 125 16 32 OVAKA 25 85 20 39 PANGAI 70 1211 4 8 PUKE 30 731 3 5 TAOA 50 496 7 13 TAUNGA 25 39 43 85 TONGOLELEKA 55 unknown TOULA 45 411 7 15 TU'ANEKIVALE 45 487 6 12 TU'ANUKU 50 313 11 21 79 UTUI 35 302 8 15 'UTUNGAKE 70 160 29 58 VAIMALOO 60 & pump 95 42 84 Total 1,465 Kilolitres Minimum 3 5 Maximum 90 180 Average 21 43 80 7.12 Relationship strength assessment The following tables provide an assessment of the strength of the relationship that community members and VEMCs felt between their community and the various stakeholders in each column. Table 13. Strength of relationship between community and GoT stakeholders: Community member rating Relationship strength District Officer NEMO MIA MAFF Total Strong 7 5 7 5 24 OK 6 2 2 6 16 Weak 0 1 1 0 2 Unsure 1 6 3 3 13 Table 14. Strength of relationship between community and GoT stakeholders: VEMC rating Relationship strength District Officer NEMO MIA MAFF Total Strong 2 2 3 3 10 OK 2 1 1 2 6 Weak 3 5 4 3 15 Unsure 1 1 1 1 4 81 7.13 Assessment of Core Humanitarian Standard commitments Table 15 below documents TCDRM’s commitments to a number of the Core Humanitarian Standards that were outlined in the TCDRM Year 5 Proposal document. The right hand side columns contain the evaluations assessment of the use of these standards in the TCDRM program. Table 15. Core Humanitarian Standard Commitments. CHS Commitment Evidence of inclusion in the Project Evidence score (High, Medium, Low, None) Evidence and comments #4. Communities and people affected by crisis know their rights and entitlements, have access to information and participate in decisions that affect them. Project staff will work closely with communities and VEMCs to consult, share information and make transparent decisions. Communities will be aware of their rights, and be able to provide or withhold consent throughout the Project. High  Initial community consultations were transparent and provided information for community leadership to accept or decline participation in program.  This degree of consent may or may not have been made clear to other community members. There was no evidence found of forced or coercive behavior to encourage participation.  Community engagement and consultation was ongoing throughout the program. #5. Communities and people affected by crisis have access to safe and responsive mechanisms to handle complaints. Confidential complaints mechanisms will be established which will include procedures to follow up and remedy where needed. Medium  No evidence was found of a confidential complaints mechanism.  Evidence was found that issues and complaints raised by community members were addressed by the program. For example in one community, a complaint was raised that the Town Officer was not keeping the community informed about the community project. The TCDRM team visited the community to explain and clarify the status of the project.  The creation of a low-cost anonymous website feedback form may be an initial step to fulfill this commitment, however, this solution would not be considered accessible for all community members. 82 CHS Commitment Evidence of inclusion in the Project Evidence score (High, Medium, Low, None) Evidence and comments  A genuine effective complaints mechanism would require additional resourcing, multiple accessible complaint lodgment channels (face￾to-face, online, phone) and engagement of a third part to ensure privacy and independence. .  Future programs could explore if the Office of the Ombudsman was willing to expand its scope of operations to receive and pass on complaints related to CSO programs. The Ombudsman and complaints mechanism is accessible via phone, website and offices in Vava’u, Ha’apai, Tongatapu and ‘Eua. #6. Communities and people affected by crisis receive coordinated, complementary assistance. The Project will coordinate with relevant partners, government departments and local authorities, and share information and findings to encourage a stronger community of practice. Medium  High levels of coordination with national and sub￾national Government departments and CSO was found. These stakeholders reported to have been kept informed of planned and completed program activities.  Opportunity to more proactively share findings exists, specifically through the sharing of community profiles, VCA and VEMPs (electronic and paper versions).  National stakeholders also have a role to play in facilitating the sharing of best practice and evidence was found that the TCDRCM team was not always invited to forums where this knowledge could be shared. #7. Communities and people affected by crisis can expect delivery of improved assistance as Act for Peace and TNCC engage in continuous quality improvement cycles. They have and High  Evidence found to confirm continuous improvement in program design (addition of WASH workshops and first aid training accessible to entire community) and 83 CHS Commitment Evidence of inclusion in the Project Evidence score (High, Medium, Low, None) Evidence and comments organisations learn from experience and reflection. continue to monitor, evaluate and improve activities based on evidence and feedback. implementation (M&E systems improved).  M&E plan used to monitor activities and collect feedback from participants.  Continuous improvement cycles were observed by the MTE. Many MTE recommendations were implemented. #8. Communities and people affected by crisis receive the assistance they require from competent and well￾managed staff and volunteers Act for Peace and TNCC staff have been recruited with high levels of skill and appropriate experience. They will receive ongoing professional development during the Project to continue to build their skills and knowledge High  Program staff recruited had knowledge experience in many relevant TCDRM program components (CbDRM, CbCERT, First aid). Some staff had preexisting community development and facilitation skills.  Through TOT workshops led by AFP and other national and regional experts, program staff gained knowledge and skills in key program areas (CbDRM, CbCERT, First Aid). Subject areas that were relatively new for TNCC such as Community Based Protection were given additional attention. Capacity building in operational areas such as finance, acquittals, audit procedures and procurement were also provided to relevant staff.  Opportunities for continuous capacity building through regional and international training was also observed (SPHERE training in New Zealand) #9. Communities and people affected by crisis can expect that the organisations assisting them are managing resources effectively, efficiently and ethically. Act for Peace and TNCC have strong, ethical and accountable financial and human resource managementsystems and include external audits and third party evaluations. High  Evidence of fraud detection measures were provided and reports of both internal and external audits were noted.  TNCC staff sign Child Safeguarding Code of Conduct and receive Child Safety training.  An external MTE was conducted in 2016 and an external end of program evaluation is being undertaken in 2019. 84 7.14 Assessment of MTE recommendations Table 16 below lists all the of MTE recommendations. The right hand side columns note the extent that the evaluation believes these recommendations have been implemented, and how realistic the recommendation is given the local context. In many instances the recommendation should have been directed towards other stakeholders and TNCC should not be accountable for their implementation. Table 16. Assessment of MTE recommendations Mid-term evaluation recommendation categories Specific recommendations Extent of evidence indicating recommendations implemented (High, Medium, Low, Unsure) Realistic and actionable (High, Medium, Low, Unsure) Should be directed at other stakeholder Comments Capacity Building Build knowledge and strengthen capacities through direct training in communities. Half-day or full-day TCDRM refresher training delivered within communities L L NEMO Ability to implement constrained by limited resources and cost-benefits of delivering to new vulnerable communities as opposed to delivering refresher training to past targeted communities. First aid training at the community level (or TOT) H H Audience for Frist Aid training was expanded to include the entire community. SIMEX included First Aid demonstrations for those who did not attend First Aid training Strengthen understanding of in early warning systems in communities and work with government to improve EWS for enhanced Further targeted capacity building in EWS and climate forecasting, particularly in relation to drought and perhaps cyclone tracking would be beneficial, along with support to government to enhance EWS messaging for communities. M M TCDRM & MET & NEMO Nearly all community members reported the ability to access climate forecast information from MET via radio that would inform them of potential drought. This enhanced knowledge may or may have come from improvements to TCDRM training to place more emphasis on slow onset disasters. 85 practical relevance within communities. Only a small percentage of community members had previously seen a cyclone tracking map and most were unable to accurately read the map. Community Awareness and Mobilisation Revise and strengthen VEMPs for enhanced community resilience Develop Response Arrangements for other key hazards, including slow onset hazards such as drought. H H Drought identified as a hazard and response measures included in DRR Action Plan in VEMP. In the absence of comprehensive community water/WASH plans, the scope of drought response actions could be expanded to introduce rationing of supply from community tanks once storage reserves reach 25% or some other threshold. The use of other measures such as SODIS to treat ground water and the procurement of Solar Water Purifiers for deployment at schools and health facilities could also be explored. Combine Hazard and Resource maps if possible to better visualize impacts on assets and livelihoods. H H TCDRM reported that VEMPs in Years 4 and 5 included maps that integrated hazards and resources. Verify the accuracy of traditional early warnings noted in the Plans. L L NEMO There was little evidence found that a scientific process had been followed to verify traditional knowledge. To action this recommendation would require the set-up of an additional research project supported by resources and expertise. Identify and analyse gender- U M TCDRM team reported improvements 86 differentiated vulnerabilities, capacities, and coping mechanisms using participatory approaches and tools and integrate these findings into VCAs and DRR Action Plans. in this area under the guidance of AFP. VCAs and gender differentiated vulnerabilities and actions were not observed. Integrate gender-differentiated livelihoods assets and activities into DRR Action Plans. U M Unsure Endorse new and revised VEMPs by the District Officer and/or NEMO, rather than the Program to support linkages between communities and government and increase understanding among different levels of Government of the specific risk and resilience factors within different communities. M M NEMO VEMPS are currently still authorised by TO and TCDRM program manager, not NEMO. After the handover of VEMPs to NEMO they are to take control and ownership for future amendments and approvals. Support widespread community engagement, awareness and ownership of the VEMP. Engage the wider community in the development and/or revision of the VEMP, specifically women, PWD and the marginalised. L M No evidence found to demonstrate change in engagement of women or PWD and marginalised in VEMC development. VEMP development is still restricted to VEMC. As VEMC is comprised of men and women, women’s voice should be already captured. VEMP is supposed to be shared with the broader community via community meeting, however, poor attendance at these meetings may limit sharing and awareness within the broader community. A foundation level of DRR/DRM 87 knowledge is likely required to understand the VEMP and the program is not resourced to provide this foundation knowledge to all community member which makes broader community involvement difficult. The practical implementation of this recommendation would require at least two or three additional days in communities to build foundation knowledge and deliberately seek out PWD and marginalised groups. Collecting and collating data and views from a larger diverse range of stakeholder would also extend and further complicate the process. Posting the VEMP in publicly visible and accessible locations L M Whilst noted as a ‘good idea’ the public posting of the plan was thought to be ineffective by the TCDRM team in the Tongan context. Whilst the evaluator respects the cultural knowledge of the TCDRM team, the evaluator believes that a finalised bound copy of the plan should be present in each EC. It should be attached via small chain to the wall of the EC to prevent its removal. The location of the plan should be marked with a simple A4 laminated sign that indicates the presence of the Village Emergency Management Plan. Noting that there is only one TO and the TO has the only copy of the VEMP, 88 additional copies of the VEMP at ECs may be beneficial, especially if the TO is away on business and VEMCs fail to activate. regular monitoring of the implementation of the DRR Action Plan by VEMC and non￾VEMC members M L VEMC, Community council MIA TCDRM successfully monitored the implementation of the TCDRM community project which was aligned to one action in the DRR plan. Additional monitoring of implementation of DRR actions is important, however, such responsibility sits beyond the scope of the TCDRM program and responsibility must rest with the existing community management structures. Additional capacity building in project management and governance (beyond that provided to VEMC chairperson and secretary) may assist in the implementation of these actions. Sub-national MIA staff could be alerted to the need follow-up with communities regarding the implementation of DRR actions. Policy and Planning Support national CBDRM standardization in Tonga through partnerships and collaboration and systematic application of TNCC through TCDRM should continue to work in collaboration with NEMO and government ministries and cluster partners to standardize CBDRM messaging, training packages and terminology M M NEMO Red Cross Tonga Trust MORDI TNQAB The CBDRM Framework once endorsed would be a strong response to this recommendation. A separate project would need to be established to formalise a recognised training package (or community course) that would standardise the delivery of CBDRM. 89 lessons learned and good practices from the Program. It is beyond the scope of the program to influence what and how other CSO deliver CBDRM. GoT needs to step up and lead or initiate a standardisation process to bring CSO together. Support widespread community awareness and engagement in government policy-making about CBDRM. Community involvement in the development of public policy, strategies and plans should not be a one-off process and there should still be scope for the team to collect direct feedback from communities about standardized CBDRM processes and tools and forward recommendations to further strengthen the draft Framework. L L NEMO/MIA Beyond the draft CBDRM Framework, little evidence was found to support increased community influence of public policy. The implementation of this recommendation would require an additional significant program strategy and two or more years of programming with additional resourcing. Foundation knowledge and skills in policy and advocacy and application of a ‘rights based approach’ would likely be needed to establish the foundations from which to fulfil this recommendation. Project Implementation Develop a robust M & E framework incorporating both qualitative and quantitative outcome indicators to measure increased resilience among Additional capacity building and/or technical support for monitoring, evaluation and learning M H AFP worked with TNCC to strengthen MEL Framework and MEL capacity within the team. Improvements in collecting baseline data and outcomes through stories were observed. 90 women and men in the communities. M & E for the program should be strengthened with the collection of baseline and end line data M H Baseline and end line data was captured in some, but not all instances for WASH projects. M & E for the program should be strengthened with identification of qualitative and quantitative outcome and impact indicators and targets M H AFP worked with TNCC to improve the MEL Framework and include additional outcome indicators. Opportunity exists to continually improve upon M&E, however this will require additional resourcing to implement. Data should be collected and analysed in relation to each of the program’s components or sub-sectors in order to effectively measure results across all areas. M M Some general reporting in increased knowledge and skills in relation to CbDRM was identified, however specific reporting across all separate components was less evident. Such data may exist in activity reports, but was beyond the scope of the desktop research conducted. Measuring outcomes in terms of behaviour change is noted as being more challenging, especially when the desired behaviours are visible only in the instances where a disaster is announced. Consolidate the evidence-base and share lessons learned. Challenges, lessons learned and good practices in CBDRM should be consolidated and disseminated at community, District, national and regional levels using a range of tools and M H Oral sharing of lessons and findings occurs at the community level. Clusters (WASH, Protection, shelter) are also forums utilised for reaching stakeholders at the national level. An opportunity still exists to create a 91 methods (and used to inform national CBDRM standardization) best practices and lessons learnt guide. This activity could be considered if there is sufficient additional content not captured in the CbDRM Framework, or if progress on finalising the CbDRM Framework is further delayed. Collate, quantify where relevant, and analyze existing data and information in order to capture additional insights. M M There is always room for future improvement in MEL processes. Existing processes were deemed basic, yet acceptable. There was evidence that additional qualitative data had been collected (Post TC -Gita stories and interviews) and that this data had been turned into a knowledge and promotional product by USAID Gender Equality, Disability Inclusion and Protection Develop a clear framework to support gender, disability inclusion and protection and monitor results. Implement a systematic approach to address these cross - cutting areas. Development of a framework or action plan with outcome objectives and indicators and clear reporting of results. M M TCDRM Year 5 proposal outlines specific actions to increase participation of women and promote gender equality. Many of these measures were implemented (gender balance at training and VEMC make - up, male and female project staff, an element of gender analysis in project proposals). There was less evidence of implementation of other measures such as affirmative action with regard to prioritising activities targeted towards women and proposed by women’s groups; Engagement with 92 partners will include an assessment of gender issues at the organisational level; Empower women to use their knowledge, skills and leadership at the household and community level. There were some gender participation targets, however, fewer indicators and targets at the outcome level to track progress in terms of disability inclusion and protection. These areas are challenging to monitor beyond the output level. Disaggregate population and household data in the VEMPs to address additional risk factors and vulnerabilities. Considerations such as martial status, household composition or socio-economic status that should inform vulnerability status (in addition to age & disability) M H Program team reported that marital status (Widows, female headed households) were integrated into community profile data. Socioeconomic status classification is a more challenging task in subsistence based communities and there was little evidence found that this was considered or mapped. Update all population and household data annually M H MIA/NEMO Population data was updated in VEMCs during the years that that program team had engagement with communities. No evidence was found that VEMPs population data had been updated independently of the TCDRM intervention. Responsibility for prompting and ensuring population data is updated should rest with MIA or NEMO. Add the following to community L M TCDRM team felt that this 93 profiles - pregnant and breastfeeding women, female￾headed households and the age, as well as the number of children under 5 years, the age and gender and specific impairment of people with disability. categorisation was too detailed for VEMPs. Whilst the data may prove useful in ensuring appropriate community and national response measures, the recommendation needs to be applied against existing community capacity to collect and update data. Presently, TO and VEMC feel it challenging enough to collect basic demographics data within their communities (gender, age, disability status). Provide training for staff on basic principles and approaches of disaster-inclusive CBDRM to strengthen the focus on disaster￾inclusive CBDRM. Measure if and how people with disabilities specifically benefited from TCDRM support and this should be captured through strengthened monitoring and evaluation tools. M M Stories captured by the TCDRM team included references to how PWD benefited from the program in terms of being a priority for VEMC for evacuation and EC management. Specific interaction with PWD to collect their experiences and feedback first hand was limited. Review existing tools on disability-inclusive CBDRM and adapt and apply suitable approaches. 19 L M This recommendation was not implemented. Explore collaborative opportunities with Naunau ‘o e ‘Alamaite Tonga Association (NATA) to strengthen the program’s focus on disability￾inclusive CBDRM and strengthen H M NATA representative presented workshops at national-level training, however, funding resources limited NATA’s involvement in community￾level training and activities such as SIMEX. 19 e.g. see Fiji Disaster Inclusive Community Based Disaster Management Toolkit http://www.pacificdisability.org/getattachment/Resources/PDF￾Resources/PDF-Fiji-Toolkit-2013.pdf.aspx 94 cross-learning Support youth￾focused activities to increase the meaningful engagement and participation of young women and young men in CBDRM Increase the overall engagement of young people in CBDRM beyond labour contributions. L H Current VEMC requirements include the membership or two or more youth members. Youth members were listed as committee members in VEMPs, however, interviews with youth indicated a lack of awareness about the existence of the VEMCs which may indicate that youth VEMC members have not shared information with their fellow youth members. There was no observed change in youth engagement in program activities during Years 4 and 5. Sustainability Strengthen leadership and work planning skills of Town Officers and VEMCs to build and sustain self￾reliant and resilient communities Include greater support for work planning will assist all VEMCs to remain active and focused on continuing actions to reduce risk L M MIA/NEMO No improvement in this areas was noted. Project funding limitations restrict the reach of proposal writing and project management training and thus existing work planning responsibility mainly rests with the VEMC Chairperson and Secretary. The Town Officer is the driver of the VEMC. No amount of training will assist VEMCs to remain active without either leadership from the Town Officer or Town Officer’s delegation of authority for the Deputy Chair or Secretary to lead to VEMC. Annual workplans should include simulations and First Aid refreshers and steps to meet broader long-term objectives M H The TCDRM team noted that simulations and First Aid refresher training were included in DRR Action Plans. 95 outlined in DRR Action Plans Whilst present, they will require a proactive VEMC led by an active Town Officer to action these plans. Develop an exit strategy together with VEMCs and communities that supports ongoing DRR action and resilient community development H M NEMO Exist strategy at the national-level implemented, however, actual ownership of CbDRM and ongoing support for communities appears weak. 96 7.15 Validity of MTE findings Table 17 below lists all the of MTE findings and the extent to which this evaluation agrees with (validates) each finding. Table 17. Validity of MTE findings. Mid-term evaluation findings Confirmed (High, Medium, Low, No) Comments Overall rating - very successful in reducing risk in communities in Tonga H Agee with MTE and supported by evidence below. Relevance Overall- Highly relevant H Agreed as per evidence in main body of the report. National-level. Highly relevant to Joint National Action Plan on Climate Change Adaptation and Disaster Risk Management (2010- 2015) Community-based approach – Highly relevant given limited NEMO resources H Aligned to new JNAP NEMO continues to have limited resources to progress CbDRM, thus TNCC’s investment is critical. Community-level – Highly relevant for both knowledge & skills capacity building H Whilst general DRR knowledge predates the program, best￾practices and Government endorsed approaches were previously missing at the community level. Effectiveness A) Capacity Building VEMC & Town Officer (TO) capacity building was effective H Relevant DRR/DRM Knowledge and skills noted. TO leadership role noted as important to mobile community H Key finding from this evaluation. Roles & responsibilities of stakeholders in a disaster were made clearer and shared with community via various channels. M Noted, however, less evidence and confidence in this finding Women and youth had some knowledge gaps L This evaluation did not identify this outcome. Short timeframes for capacity building may have reduced effectiveness, specifically regarding learning new knowledge & skills & invoking changes in attitude needed for community resilience. M Perhaps a common finding across most time-limited programs. Other approaches and tactics may assist in changing attitudes and shifting behaviour that could be explored using behaviour change frameworks. 97 Wash Training – overall satisfaction, targeting of attendees could be improved. Content could be less-technical M Agree with MTE. Overall less feedback obtained on WASH training. Staff training capacity - (ToT in CBDRM and in-house training in gender, protection and monitoring and evaluation). Noted as effective and improved over time with reflection exercises. H Aligned with the Partnership Approach, TOT was one key element of building capacity of the program team and broader network of stakeholders. TNCC & delivery team noted as an asset to TCDRM. Their effective facilitation of training was noted as a key success factor. H Noted as a key enabling success factor. B) Community Awareness and Mobilization Effective in helping communities establish VEMCs and build their capacity (prepare for disaster, evacuation) H Agreed as per evidence in main body of report. The needs of vulnerable people were taken more into account with commitments from community & VEMC to assist vulnerable people. H Agreed with examples and supporting evidence. Disaster simulations were effective – reinforce – practice. H Another key enabling success factor and key component of delivery. VEMPS were well-structured, well written, visually appealing. H The two English VEMPs inspected were deemed to meet MTE description. VEMP ownership by VEMC/communities varied. Some VEMP included names next to roles & responsibilities. H This evaluation questions the extent to which VEMPs are used by the Town Officer. There was evidence they used knowledge in their head from past experience and did not refer to the VEMP. Some VEMP integrated into community development plans U The evaluation did uncover any evidence to verify or disprove this claim. Community-based resilience projects Practical and useful H Agreed. Some Community members indicated need for follow-up support to apply grant writing skills H Common request made VEMC Secretaries had ownership of grant reporting to increase ownership. H TCDRM team reported that the weekly reporting process was effective. Implementation of the resilience projects has been variably effective. Communities were satisfied. There were some M Outside of normal delays, this evaluation found overall very high degrees of community support for community projects 98 delays, but nothing unusual. and evidence that all had increased resilience either directly (increased water storage volumes ) or indirectly (future food security improvements through increased production and increased production of food crops with long storage periods. C) Policy and Planning National CBDRM Framework designed (NEMO & SPC) with input from TNCC through Gap Analysis & consultations. H Agree, however, commitment to the Framework is not yet confirmed, noting the draft Framework is yet to be finalised. Limited awareness of Framework within communities H No community member made any reference to the Framework, however, this evaluation believes that the value of the Framework is at the National-level to guide GoT and CSOs to implement a consistent best practice CbDRM approach. Framework highlighted need for NGOs to work with GoT ministries. H Consistent with findings from discussions with NEMO. Broader awareness of VEMC, national disaster management structures, communication & reporting lines. M NEMO cited examples of TO using official communication lines instead of going to the media to report issues/status. D) Project Implementation effective management systems and planning processes & M&E Processes with detailed reports documenting lessons. M This was the focus of this evaluation, but reporting and MEL plans and reports support this finding Program responded to community feedback H Noted continuous improvement of delivery to reflect lessons learnt and TCDRM intervention to address community concerns when raised. M&E could be improved to measure outcomes / changes in community resilience, gender equality and disability inclusion M Qualitative story collection has assisted assess outcomes, however, a more structured approach with some quantitative data (small community survey) would strengthen this area. Strong relationships between communities and TNCC noted as a key factor in the success of the program. H Community sentiment towards TNCC was very positive. Gaining trust and building relationships was another key enabling success factor. Good communications & follow-up with communities. H Strong local TNCC team with strong communication and facilitation skills who developed close relationships with communities. As above, another key enabling success factor 99 E) Equity: Gender, Disability Inclusion and Protection Staff received capacity building within the context of protection training to strengthen the focus on gender & vulnerable groups H Confirmed AFP have strengthened capacity in this area. Program imparted a new understanding and appreciation among community members of the importance of caring for people with disabilities, the elderly and children during disasters H Confirmed through VEMC and community interviews No references were made in FGDs to protection risks for women. M Prompting questions provided evidence of knowledge about the importance of protection, mainly in relation to EC management. Some, but not all program strategies to promote gender & disability inclusion were implemented. H Agreed. For example, TCDRM Year 5 proposal included a number of Gender specific actions and many, but not all were progressed. Little evidence was found of more proactive activities to reach and engage PWD and their carers. VEMC have women representatives. Men and women noted as Group Leaders & Protection Officers. H Agreed, however, this evaluation believes that the role of Protection Officers is limited role assignment within the VEMP with some evidence of demonstration within SIMEX, but little recollection of this role and responsibility noted during questioning of VEMCs during the evaluation. Community resilience projects lacked gender analysis M Proposals attempted to explore gender in terms of participation in planned activities. This could be improved to explore the benefits and potential negative outcomes on different genders and sections of the community (PWD, elderly, youth) DRR Action Plans do not incorporate any gender dimensions M English VEMPs inspected were old and it is unclear if VEMPs created in Years 4 and 5 improved in this area set against MTE recommendations. Broad agreement at community level that both men & women should share leadership roles for building resilience. U This was not explored as part of the evaluation. The evaluation finds that the TO, usually a male is responsible for leading most DRR/DRM activities and he instructs others on what to do. Women and other men may have more autonomy within their individual men’s/women’s groups, 100 but the TO rules the VEMC. Less evidence to document how PWD benefited. PWD presented positive views about their involvement in activities and the focus given to PWD during disaster. H Confirmed. Noting the additional time to conduct effective monitoring of benefits of PWD. E) Sustainability Mixed likelihood of sustained benefits H Community project outputs and outcomes likely to be sustained. VEMC/VEMPs have more mixed prospects which mainly hinge on the quality of and how active the TO is. Strong sense of cooperation and unity in communities about disaster preparedness and response M Not the focus of this evaluation, but some evidence to suggest greater community cohesion supports this. Positive indication that communities would continue to maintain water/food/emergency centre improvements H As stated above and in main body. In some communities there was ownership of VEMC. VEMC meeting regularly and discussing DRR/DRM. First aid kits maintained. H Agreed, however, no communities reported having first aid kits as this output was phased out due to cost considerations. In some communities there was less ownership of VEMC. VEMC had not met outside of program activities & first aid kit supplies diminished. Extra support for these communities noted. H Agreed. Noting the difficulty of assessing VEMC ownership and activity, especially when they generally only take action in response to an emergency. 101 7.16 Ladder of participation summary The ladder of participation was used in seven communities to determine the self-reported degree of participation of women in VEMCs. The ladder goes beyond measuring representation (number of men and women in VEMC) and instead looks at how much women do, or are allowed to participate in the VEMC. Figure 14. Ladder of participation20 Legend for position on ladder: 1. Allowed to join; 2. Attend meetings; 3. Speak up; 4. Listened to; 5. Influence decisions; 6. Make the decisions; 7. Set own rules 20 Four scores were removed from one community as the three female VEMC members indicated not just their position on the ladder, but they guessed where the other four female VEMC members would be on the ladder. 102 Figure 15. Sample ladder of participation tool (Tongan version) 103 7.17 Evaluation Framework The evaluation framework below was used to guide the development of data collection tools to ensure that the required data would be collected in respond to the sub-questions and indicators which in turn assist responding to the key evaluation questions. 104 Evaluation question Sub-question Indicators Source & Method (See legend at end of table) Effectiveness KEQ1. To what extent were the project objectives achieved? For context only – program objectives as per TOR: 1. To support the Government of Tonga to reduce the impact of natural disasters on communities, especially the vulnerable. 2. To strengthen the capacity of local communities to identify and assess their own protection risks and to develop strategies and action to prevent, mitigate and respond to these risks. 3. To improve the community level food security of participating communities. 4. To expand water coverage in targeted settlements to improve access to safe and sufficient water, both in the immediate and longer term. See a, b, c, d below a) To what extent have project outcomes, especially in TCDRM communities affected by TC Gita, supported the Government of Tonga to reduce the impact of natural disasters on communities, especially the vulnerable? (To what extent did the program reduce the impact of natural disasters on target communities?) Did target communities implement DRR and DRM measures as per TCDRM training when faced with real-life disaster situations (or simulations where real-life disaster have not been experienced since program inception)? # and % of communities demonstrating implementation of DRR/DRM processes and actions FGD VEMC KII Community members (M,W,Y,PWD) Post-TC Gita reports KII NEMO Village observations/transect walk # VEMCs created/supported by the program Program reports % of target communities with functioning VEMCs (some actual data, some data reliant on perception/view of TNCC) KII Community leader/member KII TCDRM PM 105 Examples of VEMC members fulfilling their DRM roles and responsibilities during emergency (Oversee implementation of DRR actions in VEMC, Communication with district/national level (pre & post disaster), Early warning to community when disaster approaching – Mobilise the community, issue first aid, Post disaster rapid assessment & reporting. Putting plans in action, showing leadership.) FGD VEMC Reflection TCDRM team # VEMPs created/supported by the program Program reports What was the most significant change experienced by community members as a result of the TCDRM? Story collection % VEMPs that are current or been reviewed in the past 18 months. FGD VEMC Program reports KII TCDRM PM Were target communities more prepared for disaster events such as TC-Ian and TC￾Gita when compared with communities that did not benefit from the program? DRR/DRM knowledge & skills in target and non-target communities KII NEMO KII DO Source of knowledge & skills (TCDRM, other programs, existing local knowledge) KII Community member Numbers of and examples of DRR measures implemented in each category of community before the disaster event. KII Community FGD VEMC Numbers of and examples of DRM measures implemented in the lead-up to and after a disaster event. FGD Community FGD VEMC Presence of VEMC and VEMP or similar structures/plans in target and not-target communities KII TCDRM PM Sample of VEMPs KII NEMO Clarity of roles and responsibilities during disaster events FGD Community FGD VEMP Understanding of cyclone specific disaster information (strength (category) , tracking map) within target and non-target communities FGD Community KII Community 106 Effectiveness of Early Warning systems within target and non￾target communities. (Did community member receive cyclone notification & updates from NEMO/MET & were they shared with community & did they trigger a response to put emergency preparation plans into action. KII Community FGD VEMC Did target communities experience less damage to assets (buildings/crops/water sources) and experience few injuries and loss of life compared to communities that did not benefit from the program? Disaster (cyclone) impacts for target and non-target communities Community assessment reports KII DO NEMO KII Compared to non-target communities, did target communities require less government/external assistance in the immediate disaster response phase as a result of the DRR and DRM measures undertaken at the community level? Level of external assistance requested/needed DO KII NEMO KII Did target communities recover faster than non-target communities as a result of the DRR and DRM measures undertaken at the community level? Time taken for life to return to what community members identify as ‘normal’ Cyclone recovery cycle diagram (Community FGD & KII) KII DO KII NEMO a) To what extent has the project strengthened the capacity of local communities to identify and assess their own protection risks and to develop strategies and action to prevent, mitigate and respond to these risks? What capacity exists at the community-level to identify and assess protection risks and to develop strategies to respond to those risks? Capacity building activities delivered at the community-level included protection risk content (See sub-questions below) CbP capacity building activity reports What is the degree of understanding amongst community members about what protection risks are (in simple local context)? Community member ability to simply define what protect risk is via a description or listing some examples CbP post-capacity building activity reports KII community member FGD VEMC Changes in knowledge as a result of CbP training Pre & post knowledge testing in CbP capacity building activity reports 107 KII Community member How many target communities have undertaken a process to identify protection risks within their community? # VEMPs that document protection risks VEMP review FGD VEMC KII TCDRM PM Quality of protection risks identified (good coverage, missing risks, specific re: specific vulnerable groups or individuals referred to) VEMP review KII TCDRM PM How many target communities have developed strategies to respond to protection risks? # VEMPs that document responses to protection risks VEMP review KII TNCC Are protection risk proposed responses realistic and actionable by the community? Community capacity to implement responses (roles & responsibilities clear) VEMP review FGD VEMC What is the community capacity to identify protection risks and create responses independently of TCDRM/TNCC? VEMC confidence to undertake CbP analysis & document responses. VGD VEMC Is there evidence that responses to protection risk have been actioned in SIMEX or a real disaster event? Examples of protection responses (as per VEMP or otherwise) put in action Community SIMEX reports/reflection KII TNCC PM b) To what extent and in what ways has the project improved the community level food security of participating communities? What changes to food planning, production, storage and distribution have occurred as a result of program activities? Food security discussed at community meetings or in community plans. KII Community leader FGD VEMC Anecdotal changes in land area under production in response to plans/future needs (e.g. emergency supply) KII Community leader FGD VEMC Changes in crop/animal varieties farmed – more resilient to climate and disaster impacts KII Community leader FGD VEMC Program reports KII Community member Changes in farming techniques/practices to increase yield/productivity/increase resilience to CC/disaster events KII Community leader FGD VEMC Program reports KII Community member Changes in farming tools/equipment used KII Community leader FGD VEMC 108 Program reports KII Community member Anecdotal changes in productivity (crop/animal yield) KII Community leader KII Community member FGD VEMC Program reports KII Community member Changes in access to required inputs such as water and other inputs (natural inputs such as compost or synthetic inputs such as fertiliser or animal feed via the private sector supply chain) KII Community leader KII Community member FGD VEMC KII Community member Changes in food processing/storage (e.g. drying, preserving) /distribution KII Community leader KII Community member FGD VEMC Program reports Measures in place to facilitate the supply of food to the elderly, people with disabilities and other vulnerable/marginalised groups during emergency response KII elderly KII people with disability FGD VEMC In relation to control communities, do target communities have higher levels of food reserves which can be drawn upon if a disaster strikes? Examples of food reserves at the community level * It will be difficult to quantify kg/tonnes etc KII community leader KII VEMC Resilience of food reserves in relation to disaster events (stored safely / survive a disaster event, shelf-life) Observation Perceived number days emergency food supply in target and non-target communities KII Community leader KII Community member Without household-level data (baseline & post-program), it will not be possible to accurately identify changes in food security re: in terms of # hh reporting ‘sufficient’ food for emergency / kg per person, # day supply. c) To what extent has the project expanded water coverage in targeted settlements to improve access to What changes to water supply planning, storage, water quality and distribution have occurred as a result of program activities? # communities benefiting from WASH related grant projects Project reports 109 21 WHO standards www.who.int/water_sanitation_health/emergencies/qa/emergencies_qa5/en/ safe and sufficient water? Has this differed in the immediate and longer term? (compare communities who participated in years 1-2 with years 3-4) Additional KL water storage of potable water added to communities Project reports TNCC Procurement Add footnote www.who.int/water_sanitation_health/emergencies/qa/emergencies_qa5/en/ Estimated additional # days water supply added by the project (in disaster situation with rationing at recommended WHO levels of 15 litres per person per day)21 TNCC Procurement Project reports (re: # community members ) Community perception of having access to sufficient water to cope with 14 days in an emergency response scenario. KII Community member KII Community leader Perceived changes in water quality from existing/new/improved sources KII Community member Measures in place to facilitate the supply of water to the elderly, people with disabilities and other vulnerable/marginalised groups. KII Community member FGD VEMC Observation # VEMPs/communities that have plans to manage water in slow and fast onset disaster events FGD VEMC What are the current main sources of water currently and what were they before the project? Current / past primary water source Are you happy with the water access you have currently? 110 Without household-level data (baseline & post-program), it will not be possible to accurately identify expanded water coverage in terms of # hh reporting ‘sufficient’ water. How have changes in community water security outcomes varied between communities based on which year (Years 1 & 2 v’s Year 3 & 4) communities were engaged by the project in WASH training and WASH projects ? TBC – water tank volume or number of tanks increased per community in Y 3 & 4. May be in response to shift from cement to Rota-mold design Community perception of changes in water security Analysis of data from existing questions (KII Community, FGD VEMC) Changes to WASH project design / implementation between the two time periods. KII TCDRM PM KEQ 2. What were the major factors influencing the achievement or non￾achievement of the objectives? What were the key factors of TCDRM success in achieving planned objectives? Key success factors Reflection TCDRM team KII AFP KII NEMO KII Partners/stakeholders (NATA/Red Cross) FGD VEMC How influential was the AFP partnership approach (delivery model) to the achievement or non-achievement of planned objectives? Degree of influence Reflection TCDRM team KII AFP Key success factors/inhibiting factors of the partnership Reflection TCDRM team KII AFP What were the factors that inhibited the success of TCDRM in achieving its objectives? Key inhibiting factors Reflection TCDRM team KII AFP a) How effective was the TCDRM methodology, and which specific aspects of the approach, if any, should be replicated or adapted for future programming? Which specific measures of the TCDRM methodology were found to be more or less effective at building community resilience to disaster events? Rating of TCDRM components in increasing community resilience (DRR/DRM) Reflection TCDRM team KII AFP FGD Community FGD VEMC How can the existing TCDRM methodology be improved? Reflection TCDRM team KII AFP FGD VEMC KII NEMO What components of the TCDRM methodology should be considered for replication in future programming? What contextual factors may need to be considered when replicating the program components? (What works, where, when and for whom?) Are there any pre-requisites for replication? Any local contextual factors that promote / impede replication. Reflect upon what was found to be most effective (what worked well) and why above 111 b) How has the program promoted connectedness, both horizontal (strengthening both government and civil society structures and systems and the relationships between them) and vertical connections (strengthening linkages from national to sub-national and community levels.) What changes have occurred in GoT and civil society structures and systems in relation to DRR and DRM as a result of the program? Clarity of roles & responsibilities KII NEMO KII Red cross/ NATA KII DO KII TO Reflection TCDRM team What changes have occurred in the way that GoT and civil society work together on DRR and DRM? Extent of communication, coordination & collaboration in DRR/DRM (capacity building/policy & strategy work/decision making/on-ground implementation) KII NEMO KII Red cross/ NATA Reflection TCDRM team What change has there been in the relationship, engagement and communication between the different levels of the DRR/DRM system in Tonga (National-level (NEMO/MIA), Sub-national - Island Group (DO), Community (TO) & VEMCs) KII NEMO FGD VEMC Reflection TCDRM team c) To what extent and in what ways have communities been empowered to participate in government decision-making in DRR as a result of the program? What opportunities have existed and been taken up for community members to participate in decision making in relation to DRR at the national, sub-national and community level as a result of the program? KII NEMO KII TNCC PM KII Community leader KII Community member (women, people with disability) What, if any, are the barriers that prevent participation in DRR related decision making at the national, sub-national and community level? Barriers KII Community members KII NEMO KII DO Were any identified consultation processes genuine with meaningful outcomes that considered community input? Decisions / policies / plans reflect stakeholder input. d) In what ways do communities, including people with disabilities and other vulnerabilities, perceive their level of safety (before, during, after disaster) to have improved as a result of the project? How safe do vulnerable community members feel in relation to before during and after a recent disaster event? (only applicable for communities that have recently experienced a disaster event) KII women KII People with disability or caregiver KII Elderly Has there been any positive or negative change in the perceived level of safety experienced by vulnerable people in relation to DRM and DR response? KII women KII People with disability or caregiver KII Elderly What are the reasons for any changes in perceived safety? KII women 112 KII People with disability or caregiver KII Elderly KEQ 3. To what extent has achievement of project objectives been equitable (considering gender, disability and marginalisation/ remoteness)? Have remote island groups and communities received similar program inputs and activities and achieved similar beneficial outcomes in comparison to more urban and central areas/communities? TCDRM proposal Program reports Level of attendance and participation by [women, youth, elderly, people with disabilities] in program activities (CbDRM, CbP, WASH training, SIMEX, grant project, VEMC, VCA activity, VEMP creation) Activity reports Ladder of participation with women, youth, elderly and people with disabilities. What reasonable accommodation was made to enable people with disabilities to participate in program activities? KII people with disability TNCC reflection Were the views/needs of women, elderly, youth and people with disabilities consulted and used to inform DRR/DRM/VEMP and grant project design? VEMP & grant proposals KEQ 4. What unexpected positive or negative outcomes have been identified? What unintended program outcomes (negative or positive) were identified? Unintended outcomes Reflection TCDRM team KII AFP Analysis of all data collection If negative outcomes were identified, what could be done to avoid them in future KII TNCC PM KII AFP Relevance KEQ 5. How effective has the TCDRM program been in meeting the needs of communities? Did TCDRM proposals accurately document community needs and did the program respond to these needs? If Yes, how many needs were addressed and how thoroughly were they addressed? Community needs KII Community leader TCDRM proposal Program reports Have community needs been met? Community perception of TCDRM meeting community needs KII Community leader What priority do community members place on the needs that were addressed (problems targeted) by the program in relation to other needs/problems? Needs/problem priority KII Community leader Community perception of TCDRM meeting community needs FGD Community Were WASH & food security projects appropriate? Technical solution appropriate for remote communities (re: low￾tech, maintenance, replacement parts & costs) Cost-effective in relation to other solutions Projects addressed the core problem identified 113 KEQ 6. Are appropriate technical standards (WASH SPHERE Standards, Food and Livelihoods SPHERE standards, and relevant National Standards such as construction standards) used and achieved? Have standards (WASH SPHERE Standards, Food and Livelihoods SPHERE standards, and relevant National Standards such as construction standards) been used to inform relevant aspects of program/activity design? SPHERE Core Humanitarian standards are listed in Annex 7. TCDRM proposal KII TCDRM PM KII AFP Have standards (WASH SPHERE Standards, Food and Livelihoods SPHERE standards, and relevant National Standards such as construction standards) been followed during implementation of relevant program components? KII TCDRM PM Program activity reports To what extent did the program implement recommendations from the mid-term evaluation implemented in years 4 and 5 of the program? How many MTE recommendations have been adopted by the program in Years 4 and 5? MTE recommendations are documented in Annex 6 Adopted recommendations MTE report TCDRM proposal Year 4 and 5 KII AFP KII TCDRM PM Sustainability KEQ 7. How likely are project achievements to be sustained after the end of the program? Which achievements are more sustainable than others? a) Have people from communities targeted in years 1 – 3 retained knowledge and behaviour changes? # and % of communities demonstrating recall of DRR/DRM processes and actions FGD VMEC KII Community members Reflection TCDRM team Refresher training knowledge baseline data? # and % of communities demonstrating implementation of DRR/DRM processes and actions KII Community member FGD VMEC b) Are village emergency management committees that were established in years 1 – 3 still functioning and maintaining plans and equipment? # and % VEMC still functioning (meeting, planning, acting) Up to date VEMPs KII VEMC Reflection with TCDRM team First-aid kit supply levels Observation Hailer & flash light operational Observation Radio operational Observation High visibility vests and other clothing present Observation % VEMPs that are current or been reviewed in the past 18 months FGD VEMC Reflection TCDRM team c) Have relationships between communities and decision-makers been Frequency of visits by decision makers KII NEMO KII DO 114 sustained? KII Community leader Community perception on relationship with decision makers KII Community leader FGD VEMP KII Community member Community perception of support from decision makers KII Community leader KII Community member Are the benefits from water and food security project likely to be sustained beyond the end the program? Maintenance budget / funding mechanisms to support maintenance/repairs. KII Community leader Level of community ownership of assets Collection of surrounding indicators Clarity of roles and responsibilities for maintenance KII Community leader Asset maintenance / management in included in VEMP or community plans. VEMP/Community plan Has WASH and food security infrastructure been maintained? Community trained in maintenance KII Community leader WASH infrastructure still functioning effectively Observation Have evacuation centres that were improved by the program been maintained? Visible building strength and functionality Maintenance plans/budget Observation What is the likelihood that NEMO will continue implementing the CbDRM approach in additional communities and continue to support existing TCDRM communities? NEMO capacity (staffing) NEMO budget allocation for CbDRM / related activities Level of commitment from NEMO for CbDRM (is it valued) KII NEMO Reflection TCDRM team KII AFP What policies, plans, measures are in place to support the sustainability of TCDRM achievements? Do existing national-level DRR/DRM plans support the use of CbDRM approach? Policy/plan/strategy review (TSDF,JNAP, climate change policy CbDRM Framework) Has the National CbDRM Framework been finalised and endorsed by GoT? KII NEMO Does the National CbDRM Framework include the views of TCDRM communities, civil society and other national stakeholders? National CbDRM Framework ADDITIONAL QUESTIONS FOR CONSIDERATION/INTEGRATION ABOVE To what extent did the program How many MTE findings were validated by this evaluation? Validated findings MTE report 115 validate the findings from the mid-term review? MTE findings are documented in Annex 6 Final evaluation report 116 7.18 Community consultation snapshots The following table provides a very brief summary of some key points noted from each community consulted. Community Key points and observations Tu'anuku Only VEMC were present at FGD. Good understanding of VEMC DRM roles & responsibilities. No strong evidence of past or future VEMC activity Weak relationships with GoT. SIMEX useful to bring GoT to community, but not relationship not strong. VEMC know what to do before, during and after disaster and this was attributed to the Program capacity building. Knowledge of 72-hour bag. EWS working - Radio and SMS notification. Town officer alerts community. Additional water tanks very highly valued. Every home now has a tank. Reduced water shortages. Taoa Community recalled Water projects and SIMEX Evidence of increased DRM knowledge & skills. All 3 community members understood TC category and achieved medium scores on knowledge of what to do before a cyclone. 3 community members not aware of 72-hour bag, but VEMC know about it. Maybe knowledge not well shared with all. All noted that additional assistance should be provided to elderly, people with disabilities and widows. “Previously our water was not good but now our water source is clean and safe. Our standard of living is developing. Dry period is now in Vava'u , but we are safe because of these tanks, thanks.” Reported feeling saver and stronger. Improved EWS and proactive DRM behavior before cyclone. VEMC DRM knowledge high. Most not sure how to read TC-tracking map. Feeling of safety depends of many factors, including cyclone strength. VEMC not aware of their roles. Community member engagement in business and farming, were barriers to greater community participation in activities. Tu'anekivale OK DRM knowledge, before, during and after emergency. VEMC set example for community at SIMEX. Community feel safer because there is VEMC in the community to assist them prepare and manage the EC. EWS OK. TO and VEMC notify the community. Community aware of VEMC and role. DRM information shared at fono. One youth member was unaware of the VEMC. Program helped break down barriers between different groups in the community Matamaka High levels of DRM knowledge and skills, but only VEMC present at FGD, so could not assess if this was shared with other community members. No other community members available to inform evaluation. EC improvement was a priority for the community Relationship with GoT noted as strong, but no justification given to justify this. MAFF noted as only coming with the agricultural show was approaching. Appreciate TCDRM brining GoT to visit their community. VEMC aware of protection issues, but in reality, the EC is too small to keep women and girls in separate areas. Indicated they ran a SIMEX as part of the last TC-Warning (unsure if this is accurate). 117 Reported VEMC met annually. EC, toilets and one water tank still in good condition. VEMC indicate they have DRM knowledge to conduct own trainings, however, no evidence that this was or would occur. Fotua Reported program increased their level of DRM knowledge and skills. Learnt from multiple sources, including the Program. VEMC ran their own SIMEX for community Not all community members are aware of VEMC. Not all go to fono which might explain. Church champions suggested to increase reach. VEMC aware of need to give special attention to PWD and elderly. Ha'ano VEMC last met in 2018 and indicated they only met when there was an approaching cyclone. TO leads VEMC and others have no clear understanding of their roles within the VEMC. TO indicated he will use VEMP at meetings and tell other people what to do. Medium level awareness of protection issues and mitigation measures. Livestock fence still OK, but few rotten posts need replacing. Noted absence of vests, but TO had a hailer. VEMC rated relationship with DO was OK, but with other GoT, it was noted as weak. Community members rated relationships as strong or OK. Only just learn they can contact MAFF to ask for extension support for improving sheep management. One of three community members was unaware of the VEMC or if the community had a VEMP. One community member indicated that the VEMC did not go around to household to alert them of approaching cyclones. Reported increased planting of food crops in response to TO & MAFF encouragement to plan for disaster & food security. Muitoa VEMC not clear on roles. VEMP has not been updated for a few years. TNCC handed over updated VEMP. “The only time the committee works is when TNCC come to visit. Need more motivation.” DRM knowledge OK. Understood TC category and 4 could read the TC tracking map and they reported TCDRM as source of improved knowledge. TO aware of role to conduct initial assessment and report to NEMO. Simulations and First Aid noted as very beneficial. Protection was not well understood. No knowledge of protection officers. Food security measures included in VEMP. Relationships with GoT noted as weak. They knew NEMO and NEMO roles. Program noted as a starting point for improved relationships. All water tanks inspected in good condition and community very appreciative. Noted that there was insufficient pressure from header tank in Ha’ano to supply water to households. Issue with neighbouring dogs eating community sheep. Mata'aho Increased DRM knowledge and skills and program acknowledged for this increase. E.g. 72-hour bag. This is a new practice that has been adopted. VEMC activity depends on the TO. Last time VEMC met was before TC Gita. No meetings since then. No future SIMEX planned. There was increased focus on looking after PWD and elderly. No noted change in relationship between community and GoT. Community appreciate GoT coming to present. 118 EWS OK. VEMC were active in EC management. Rain water tanks installed by another CSO led program had no FFD. Ohonua TO was not highly engaged in Program or VEMC. He was not strong during SIMEX and other VEMC members (TRCS) stepped up to lead the activity. Only 3 of 23 FGD knew about the program. This is a big community and it was noted as difficult to spread the knowledge and news, especially because fono attendance is weak. Half of the VEMC members live in Tongatapu. Need to updated VEMC members. Low awareness of 72-hour bag in community. LDS church members and few VEMC members were the only ones aware. Evidence of PWD being checked on during Gita. SIMEX noted as useful. VEMC noted increased preparedness and some VEMC activity (but perhaps not so much involving the TO). Matafonua VEMC not officially meeting, but VEMC members are active and having one-on-one discussions to progress issues (mainly community development). Community does not have an elected community committee, so VEMC members have stepped up to informally take on this role. EC still in good condition, water tank leak. No water in toiles, no wash basin. First Aid, Proposal writing and SIMEX highly valued. No recollection of GoT coming to visit during program, but this was many years ago now. VEMP exists, but TO indicated he acted on past experience, not the plan. TO indicated that there is no handover or induction for new TO. This was noted as a difficult community b/c there were people from many different communities and family groups living here – less cohesion. Very low attendance at fono. Community have DRM knowledge and skills, but do not always act accordingly. Tefisi Control community Community benefited from C-CAP project which covered similar TCDRM knowledge & skills. This included CbDRM and SIMEX. Met with two groups of women, plus the Town Officer. Community knew what to do before, during and after disaster and this was attributed to past experience. Only some recalled C-CAP and stated it helped build capacity. Knowledge of 72-hour bag. Town Officer knew of his DRM responsibilities and reporting hierarchy up to GoT. Protection: Two different mitigations measures recalled (stay in family groupings, allocate one father to look after all the women & children). Two of the three community members were aware of the community emergency management committee (not called VEMC). They understood the committee would inform the community of incoming hazard, would assist vulnerable people, could do first aid and also led village beautification project. The committee shared DRM information with the community at fono. The Town Officer was aware about the communities emergency management plan, another community member was unsure if a plan exists. Town Officer indicated he sometimes looked at the emergency plan, however, mostly acted from knowledge from past experience. (same as most target communities) In recent years the community has benefited from external assistance in the form of pig fences and 15 x 5000L water tanks. Relationships with GoT rated as OK by TO and community. 119 Foa Control community No other DRM interventions reported in the community in recent years. There was no separate committee responsible for DRM. This role was included in the Community Committee. There was no separate Plan for DRM. This information was included in the Community Plan. No one was noted as being responsible for controlling or managing the EC. No first aid knowledge noted. No simulations have been conducted. Understanding of communication between community and GoT was not clear with some confusion between MIA and NEMO. Not strong knowledge of protection issues. 120 7.19 Sample of photographic evidence A small sample of photos are included in the report to provide further evidence of the evaluation, and delivery of Program outputs and outcome. The following photos were taken by Martin Pritchard and Mele Moimoi. TCDRM Program Coordinator, translator and Tu’anuku Town Officer with VEMP Livestock fence at Fotua 121 Taoa FGD line up indicating that most (left of centre) do not know how to read the TC tracking map Livestock fence at Ha’ano. 122 Community food plantation fence at Fangale’ounga Tu’anuku TCDRM water tank 123 Improved EC at Matafonua Evaluation process: Person with disability from Fotua who participated in SIMEX and evaluation FGD is being assisted by TNCC’s Mele Moimoi to provide a private story to inform the evaluation. 124 Evaluation process: Simple flip chart drawings of various village settings were used to prompt evaluation informants on what they do in the community to prepare for a cyclone. Evaluation process: Female VEMC member completing ladder of participation at Taoa Evaluation process: Theory of change workshop held with TNCC