1 April 2021 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Charlie Kabanga, Dieudonne Mahinanda, Emmanuel Makwe and Bertin Nyumbakwingo Final Evaluation of the Tanganyika Emergency Food Assistance Project (TEFA) in Moba and Kalemie, Democratic Republic of Congo Final Evaluation of the Tanganyika Emergency Food Assistance Project (TEFA) in Moba and Kalemie, Democratic Republic of Congo April 19th , 2021 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Table of Contents ACRONYMS .................................................................................................................................................................1 EXECUTIVE SUMMARY ...............................................................................................................................................2 I. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS .........................................................8 1.1. Evaluation Objectives and Questions ...........................................................................................................8 1.2. Evaluation methods...........................................................................................................................................9 1.3. Enablers and limitations..................................................................................................................................11 II. FINDINGS AND CONCLUSIONS ....................................................................................................................12 2.1. Comparison of Baseline and End line Data at Final Evaluation.............................................................12 2.2. Un-conditional Food Distribution (UCFD)...............................................................................................16 2.3. Food for Asset (FFA)......................................................................................................................................19 2.4. Non Food Items (NFI)....................................................................................................................................26 2.5. Provision of agricultural inputs and training ..............................................................................................29 2.6. Health and Nutrition.......................................................................................................................................38 2.7. Conflict Management......................................................................................................................................40 2.8. Cross Cutting Issues.......................................................................................................................................42 2.9. Project Management........................................................................................................................................45 III. CONCLUSION..............................................................................................................................................49 IV. ANNEXES ......................................................................................................................................................52 1 ACRONYMS ARR Annual Results Report BMI Body Mass Index CEPROCAL Centro de Producción Caprina Del Altiplano DFAP Development Food Assistance Project DIP Detailed Implementation Plan DHS Demographic and Health Survey DPCCE Daily Per Capita Consumption Expenditures EBF Exclusive Breast Feeding EQ Evaluation Question FDP Field Distribution Point FFP Office of Food for Peace FFW/A Food For Work/Asset FGD Focus Group Discussion FY Fiscal Year GDP Gross Domestic Product GMP Growth Monitoring and Promotion HDDS Household Dietary Diversity Score IDI In-Depth Interview IDPs Internally Displaced People IP Implementing Partner IPTT Indicator Performance Tracking Table KII Key Informant Interview LOA Life Of Activity SGBV Sexual and Gender Based Violence UCFD Un-Conditional Food Distribution 2 EXECUTIVE SUMMARY INTRODUCTION The Tanganyika Emergency Food Assistance (TEFA) project was implemented in Kalemie and Moba territories in D.R. Congo to support restoration of food security and resilience for 19,727 households affected by conflict. The project strategy included conditional and unconditional food distributions, agricultural inputs distributions, restoration of community assets, and health and nutrition. In addition, the project implemented activities aiming at mitigating gender-based violence and conflict. The TEFA project was implemented by Food for the Hungry (FH) for 18 months, from August 2019 to February 2021. This endline evaluation was designed to determine whether expected changes were achieved and to what extent the TEFA project contributed to observed changes. Four specific objectives were: (i) To ascertain final values and changes over time in the performance indicators from the baseline values; (ii) Evaluate the effectiveness and efficiency of the project interventions on targeted groups of interest, with particular emphasis on cross-cutting themes; (iii) Evaluate the effectiveness and applicability of complementary interventions; (iv) Identify promising practices, lessons learned, strengths, weaknesses and challenges in project design and implementation. Methodology: The evaluation used a mixed methods approach, with a combination of a household survey and a qualitative evaluation. Six criteria were used: relevance, effectiveness, efficiency , impact, sustainability and coherence. A quantitative survey of 508 households was organized to evaluate the TEFA project’s achievements in relation to 6 indicators: Percentage of households with poor, borderline, and adequate Food Consumption Score (FCS); Reduced Coping Strategies Index (rCSI); Prevalence of households with moderate or severe Household Hunger Scale (HHS) score; Percentage of women aware of where women affected by domestic violence can access the required support; Percentage of men who consider that wife beating is acceptable; Proportion of children 6-23 months of age who receive foods from 4 or more food groups. The qualitative elements of the evaluation (Key Informant Interviews; Focus Group Discussions and Observation) were used to collect perspectives from project beneficiaries, staff and other key stakeholders on the quality of the assistance, as well as to understand the context within which the project was implemented. In addition the evaluation team conducted a desk review of key project documents including the proposal and results framework, the Detailed Implementation Plan (DIP); the baseline report, project progress reports and monitoring and evaluation reports (post distribution reports and conflict sensitivity reports). Limitations: The quantitative component covered more or less the whole project area, but the qualitative assessment was constrained by poor road infrastructure and insecurity. In addition, as a result of the COVID-19 pandemic, the indicator on prevalence of global acute malnutrition could not be measured because anthropometric measures could not be taken due to social distancing. Lastly the evaluation contract was awarded 2 weeks after the end of the project, which restricted interaction between the evaluation team and project staff. 3 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The results from the household survey show that overall the food security situation improved among beneficiaries of the TEFA project. When comparing the final endline values of key indicators to the baseline situation, the following changes have been observed: EFSP3. Reduced Coping strategy (rCSI): The mean rCSI at endline was 21.37, not as low as the project’s target of 13.61. However there was still a statistically significant improvement from the baseline situation (mean rCSI = 23.92) to the endline (mean rCSI = 21.37), given that a household with a lower rCSI figure has a better food security situation. EFSP4. Prevalence of household with moderate or severe Household Hunger Scale (HHS) score: The percentage of households with a moderate to severe HHS score decreased by 6 percentage points, from 98% at baseline to 92% at endline. The difference between the baseline and endline values was statistically significant. EFSP2. Percentage of households with poor, borderline, and adequate Food Consumption Score (FCS): The percentage of households with poor FCS (0-21) had decreased by 28 percentage points (baseline poor FCS: 34% versus endline poor FCS: 6%). The percentage of households with borderline FCS (21.5-35) had also decreased by 10 percentage points (baseline borderline FCS: 52% versus endline borderline FCS: 42%). The percentage of households with acceptable FCS (>35) had meanwhile increased by 39 percentage points (baseline acceptable FCS: 13% versus endline acceptable FCS: 5 %). The difference between the endline and baseline was again statistically significant. OFDA 4. Proportion of children 6-23 months of age who receive foods from 4 or more food groups: The proportion of children 6-23 months of age receiving foods from 4 or more food groups increased by 33 percentage points (baseline 13% versus endline 46%). The difference between the endline and baseline was again statistically significant. Custom 19. Percentage of men who consider that wife beating is acceptable: At the baseline 61% of men believed that beating their wives was acceptable, compared to 64% at endline. Although these figures suggest that the situation in this regard had slightly worsened the difference between the baseline and endline was not statistically significant. Custom 20. Percentage of women aware of where to women affected by domestic violence can access the required support: There was a small increase in the percentage of women who said that they were aware of where women affected by violence could access the required support (baseline 5% against endline 8%). However, the difference between the baseline and endline was not statistically significant. Relevance and effectiveness: The TEFA project was very relevant to the needs of Internally Displaced People (IDPs), as well as host communities who were experiencing hunger and could not resume productive activities due to lack of assets. The targeting of communities, villages and households was in line with vulnerability criteria defined in the project proposal, while sensitization sessions for target communities were found to have been organized. The involvement of community members through community leaders and committees was also found to have been done at each key stage of the project. 4 Overall the TEFA project’s package of activities responded well to the various needs of beneficiaries. The unconditional food distribution (UCFD) was addressed the hunger situation. The rehabilitation of roads and water sources through Food for Assets (FFA) not only allowed communities to move easily and access their fields but it also addressed the hunger situation. The provision of Non-Food Items (NFI) to the most vulnerable community members addressed hygiene and broader needs. The provision of agricultural inputs and training allowed beneficiaries to resume their agricultural activities. Health and Nutrition through vegetable gardens and training of mothers on good health and nutritional practices was appreciated by women as they reported being better able to learn how to look after themselves and their children. Training and sensitization on conflict management and SGBV activities were also reported to be relevant to the needs of local communities as they had been affected by conflict and violence against women. It was noted however that while both the Bantu and Twa community members were targeted for the full range of activities the Twa communities were generally not accustomed to practicing agriculture. All the project’s activities took place broadly according to plans. However, the provision of assistance started 6 months after the beginning of the project for various external and internal reasons. As a result, some beneficiaries missed the best planting seasons, and some of the planned road infrastructure was not completed by the end of the project. Efficiency: The project was efficient in some aspects and inefficient in other aspects. In terms of efficiency, it completed activities with a lean staff structure and limited acquisition of new equipment. However, the project was less efficient in terms of time management as the provision of assistance started 6 months after the start date, while there were some challenges with late delivery and at times poor reported quality of project food and agriculture inputs (meaning that some could not be used). Sustainability and Coherence: The TEFA project had an exit strategy which was built into the design of the project, which was also coherent given the complementarity of activities in contributing to food security. The combination of food distribution with support to agricultural production and rehabilitation of roads was supposed to respond to short term food needs, while in parallel supporting the start of food production and facilitating access to markets. In addition, using a Care Group model in addressing issues related to maternal and child health and nutrition was intended to equip communities with knowledge and practices that could improve nutrition and food security in the longer term. The inclusion of conflict management and awareness raising on SGBV additionally contributed to the restoration of peace at household and community levels, key to sustainability of any intervention. There were however some aspects that could have improved the sustainability of the TEFA project had FH better taken them into account in implementation, such as collaborating more fully with government health and agriculture structures at local level which could carry on activities such as training or provision of technical advice. Impact and attribution: As the TEFA project was an emergency project a positive impact could be felt already in period immediately following distributions, with more food available at household level. The key quantitative indicators that were measured by comparing the baseline and the endline show that there was a significant improvement in the food security situation of beneficiaries. For instance the proportion of households with acceptable Food Consumption Scores (>35) increased by 39 percentage points and thereby quadrupled (baseline acceptable FCS: 13% vs endline acceptable FCS: 52%). In addition, households reporting that their children received food from 4 or more types of foods increased from 5 13% at baseline to 46% at endline. Qualitative data also confirmed the impact of the project for each sector, for instance Care Group mothers sharing their positive experiences of learning how to feed their children a more balanced diet and to take them to health centers, key informants from communities mentioning about the availability of vegetables in the market that they had not seen for a long time, community members speaking about being able to access their fields again thanks to road rehabilitation, and both FH staff and community members sharing success stories about Bantu and Twa community members being able to build trust and interact peacefully. The observed changes seemed mainly related to the interventions of FH, although other actors had also intervened in the TEFA project area (WFP, World Relief, CRS and Concern). Except in few cases with WFP, however, other organizations that intervened in the area either worked in different communities or implemented different types of activities. Enabling factors and challenges: There were many factors that enabled the implementation such as the relevance of the interventions; strong communication regarding the intervention; a conflict and gender sensitive approach; the involvement of community members especially local leaders; and collaboration with government structures especially at provincial level. There were also were some challenges in the implementation of project activities such as poor reported quality of food and agriculture inputs in some cases; limited staff to undertake supervision affecting mainly the health and nutrition components; and some beneficiaries of road rehabilitation reporting not receiving full tools and equipment. Cross Cutting Issues: In terms of strategies, policies and actual implementation approaches the TEFA project was found to be sensitive to key cross cutting issues such as: the existence of a complaint mechanism; beneficiary and staff protection; inclusivity and equitability; conflict management; and gender mainstreaming. Most evaluation participants agreed that FH had tried to address key issues raised via complaint mechanisms. However, there was also some issues that were found in this regard as some beneficiaries did not fully understand how the complaint mechanisms worked; and some complaints were not considered to have been addressed (e.g. on material for some trainings or food and agriculture inputs quality; and on non-payment of certain FFA workers). In addition community members were not considered to have been adequately engaged at the final stage of beneficiary selection, and overall the project staff were gender imbalanced. Project Management: The project was found to have a clear structure in place, with a dedicated team that was recruited to implement the project. The project management mechanisms and resources (human and financial) for planning (DIP) were appreciated by staff. The Monitoring and Evaluation (M&E) system was well resourced; and most M&E related activities were implemented according to plan. The finance department was able to provide financial resources and support when needed. Internal and external coordination mechanisms were in place and were perceived as key to the effectiveness of the implementation of project activities. There were however some issues that were raised during the evaluation: the late start of activities; the project being seen as understaffed in terms of field level staff and logistics staff; the project being seen as overambitious in terms of geographic targeting; and some budget lines having been significantly underbudgeted. 6 Key recommendations - Put in place a system that will allow FH to move fast in the implementation of emergency projects. This could include increasing availability of surge staff, or getting enough staff who are assigned to an emergency project exclusively, or more flexible procurement procedures. - A more effective food and agricultural input quality assurance system should be put in place, for purchasing, storage and distribution, to respond to complaints on food and agricultural input quality and to mitigate any related health risks to communities. - Adopt a payment in cash approach for the rehabilitation of infrastructure, instead of using the FFA approach, especially when the unconditional food distribution approach has already been made in the same environment. - Improve the level of awareness on processes for the management of complaints at community level, given that there is a good level of understanding of the fact that complaints mechanisms exist, but a limited level of understanding of how these mechanisms operate. - Improve the exit strategy: FH should plan to collaborate more extensively with government structures at local levels (health centers or local agronomists) in order to ensure sustainability of results, especially for behavior change related activities. 7 EVALUATION PURPOSE AND EVALUATION QUESTIONS PROJECT BACKGROUND Kalemie and Moba territories are located in Tanganyika Province, in the Democratic Republic of Congo. Food for the Hungry (FH) has been present in these two territories for several years, implementing food security projects. In 2016 FH started implementing Tuendelee Pamoja II, a Development Food Security Assistance (DFSA) project. In 2018 however, due to a surge of inter-ethnic conflicts and an army operation, which caused the displacement of more than 415,250 people, FH was unable to continue the implementation of the DFSA project. The insecurity additionally put pressure on limited resources available for Internally Displaced People (IDPs) and host communities, which already had limited access to basic services such as water and sanitation, agricultural inputs and primary health services. In order to meet the above needs FH was funded by USAID to implement the Tanganyika Emergency Food Assistance project (TEFA), an 18-month long project which started in August 2019. TEFA’s main objective was to assist 19,727 households affected by insecurity in Tanganyika province (Moba and Kalemie territories), to restore their food security and self-resilience. The project strategy includes the distribution of agricultural tools and seeds and support in transition to traditional livelihoods. The project also implements activities relating to conflict reduction and protection against Sexual and Gender-Based Violence (SGBV). The 5 main project objectives were: - Monthly unconditional food assistance for 4 months for 17,273 vulnerable households (IDPs, returnees and host communities), and for 6 months for the 1,727 most vulnerable households. - Provision of agricultural inputs (seeds and tools) and training to support 9,000 vulnerable households, including returnees, to re-engage in agricultural production. - Food for Work/Asset (FFA/A) programming for 18,000 vulnerable households in communities, in order to promote renovation of community assets. - Health and nutrition training for all households, targeted at improved household and community case-management of maternal and child health conditions. - Sensitization for all households on conflict and SGBV mitigations. TEFA was implemented in 3 health zones of Kalemie and Moba territories (Kalemie, Nyemba and Kansimba). TEFA mainly targeted host communities but IDPs and returnees were also included among beneficiaries. 7 selection criteria were used to identify households which would receive assistance: - Length of displacement of household (where applicable) - Loss of household assets - Households with pregnant and lactating mothers - Elderly and child-headed households - Households with chronically ill member(s) and people living with disabilities - Households hosting orphaned children - Single adult headed households, and not benefiting from other food assistance. The planned distribution of beneficiaries per location and per group is included in table.1 below: 8 Table .1 Distribution of beneficiaries per location and per group People living in the areas Target Location # of IDPs # of Returnees # Host Communities (Individuals) # of Villages FH targeted number of people FH targeted # of households Coverage Kalemie 54,346 34,348 86,185 76 57,108 9,518 33% Nyemba 2,852 1,769 8,740 49 6,018 1,003 45% Kansimba 57,310 23,330 113,190 96 55,302 9,217 29% Grand Total 114,508 59,447 208,115 221 118,428 19,727 30% Various strategies were put in place to ensure that only people who met the eligibility criteria were selected to receive assistance, resulting in the coverage percentages above (which show the targeted beneficiaries as a percentage of total households in the locations). A baseline was conducted at the start of the project to establish initial values for the selected indicators, which this report compares against the endline findings. I. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS 1.1. Evaluation Objectives and Questions The main evaluation questions have been outlined below: Objective I: To ascertain final values and changes over time in the performance indicators from the baseline values Effectiveness: - What are the final values for each of the key performance indicators? - What was the progress made against the key performance indicators? Relevance and Effectiveness: - To what extent has the project achieved its goals with regards to each respective sector? - Were the correct groups targeted by the project? How was the participant identification process conducted? - Were interventions rendered relevant to the targeted participants considering their initial vulnerabilities? - How well did the implementation processes adhere to underlying principles and activity protocols? - Did activities go in accordance to plans? - What were the movers and inhibitors to realizing set targets the effectiveness and efficiency? - Evaluate the effectiveness and efficiency of the project interventions on targeted groups of interest with particular emphasis on cross-cutting themes 9 Objective 2: Evaluate the effectiveness and efficiency of the project’s interventions on targeted groups of interest with particular emphasis on cross-cutting themes - To what extent did the project make considerations with regard to existing gender dynamics and protection? - To what degree did the project alleviate risks associated with participation in some FFW activities? - Did the project activities bolster or deter equity in participation? - How are monitored complaints addressed? - What was the level of efficiency of service delivery of associated goods? Objective 3: Evaluate the effectiveness and applicability of complementary interventions - Were complementary services offered relevant to the communal context? - What was the level of adoption of complementary activities? Objective 4: Identify promising practices, lessons learned, strengths, weaknesses and challenges in project design and implementation - Identify significant lessons or conclusions that can be drawn from the project in terms of effectiveness, efficiency, sustainability and networking. - What would be top five recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - What are some of the topics to celebrate with FH DRC? 1.2. Evaluation methods To answer the evaluation questions this evaluation used a mixed-methods approach (the combination of quantitative and qualitative methods). This approach allowed evaluators to draw on the benefits of each method to collect rich, precise and measurable data, and ensure the validity of the results by triangulation. Quantitative data collection: The sampling universe was made up of all the direct beneficiaries of the project. Based on beneficiary registers made available by FH the evaluation team carried out simple random sampling to select 525 households included in the survey. The basis of household selection was the presence of children between 6-23 months. The sample size for all of the indicators was calculated according to the Food for Peace (FFP) guidelines. 10 Table.2 : Sampling Key indicators Description Sample size EFSP2 The food consumption score (SCA / FCS) for different categories of households 525 EFSP3 The Reduced Dietary Adaptation Strategies Index (rCSI) EFSP4 The Household Hunger Scale (HHS) Custom 20 The percentage of women who know where to go for help with domestic violence Custom 19 The percentage of men who find it okay to beat their wives Table.3: Summary of data collection Site Sample size Achieved Kalemie 306 305 Moba 219 203 Total 525 508 A Training was organized for supervisors and interviewers. The training included a focus on the meaning and importance of the questions, on how to complete the questionnaire, and on relevant ethical issues. A day was allotted to pre-test the questionnaire in the neighboring village(s). After this pre-test there were adjustments to the data collection tools. The data was collected from the selected sample of households benefiting from the project. Quantitative data was used to provide an overall description and measurement of the variables within the framework of monitoring of indicators (e.g. number of beneficiaries and communities reached, type and quantity of assistance received). The other quantitative information that was gathered was as follows: - The food consumption score (FCA) for different categories of households (see ToR); 11 - The Reduced Dietary Adaptation Strategies Index (rCSI); - The Household Hunger Scale (HHS); - The percentage of women who knew where to seek help regarding domestic violence; - The percentage of men who considered it acceptable to beat their wives. The prevalence of global acute malnutrition (GAM) was not measured because anthropometric measurements of children under 5 years old were not taken during data collection, due to the health context (COVID-19). The same questionnaire from the baseline study was used for the final evaluation, but it was first translated into Swahili in order to ensure consistency in the way questions were asked in the local language, and adaptations were made where necessary. Data collection was carried out using smartphones and ODK software. This method had the advantages of speeding up the data collection and analysis process, reducing the cost of entering data, and reducing data entry errors. Qualitative data collection: Qualitative data concerned the context in which the program was implemented and allowed the evaluation team to explore subjective questions linked to the experience of beneficiaries and staff in relation to the program (quality of the assistance, participation and complaint mechanisms), and in relation to the security situation. Qualitative data collection helped to explain why program interventions had been more or less consistent with expected results. Table.4 Qualitative Data collection Site Types of participants Comments Men Women Focus group discussions 94 212 Key informant interviews Community 37 8 Key informant interviews Staff 15 0 Observation Several sites were visited, which involved observing road rehabilitation sites, vegetable gardens, water sources, etc. The evaluation followed Congolese guidelines on COVID-19 during the qualitative and quantitative data collection, including social distancing. 1.3. Enablers and limitations Enablers: The evaluation work was facilitated by factors such as: village chiefs being highly respected and helping to mobilize participants to join discussions; the security situation being relatively calm during fieldwork, thereby avoiding disruptions; and using several survey staff with previous experience from the baseline. Limitations: Challenges during data collection were created by distances to field sites and related transportation issues linked to poor road infrastructure and the rainy season, and 12 beneficiary lists missing certain details. More specific methodological limitations for the evaluation were as follows: - The final evaluation had been due in December 2020 but took place in in February 2021. While the difference in timing is limited, it may have had an impact on food security related findings, as the project took place in an area where food security is not stable. - Linked to this the evaluation contract was awarded 2 weeks before the end of the project, so the evaluation team had a limited opportunity to interact with staff who were directly involved in the implementation of the project including the Programme Manager and it was not possible to hold validation workshop for the project staff. - The baseline was done only for quantitative indicators. In order to evaluate the project from the qualitative perspective at end line the evaluation team therefore had to rely mainly on what participants could recall, which could be affected by recall bias (while in addition the reach of the qualitative assessment was limited by accessibility challenges, in particular limiting Twa participation in FGDs to 15% of the total participants). - It was not possible for the evaluation team to have access to the FH server for household survey data, limiting the ability of the team to monitor data quality during the data collections and to make timely corrections. - Due to the COVID-19 situation some indicators such as EFSP9 / Prevalence of global acute malnutrition (GAM) could not be measured, as social distancing would have been impossible if taking anthropometric measurements. II. FINDINGS AND CONCLUSIONS 2.1. Comparison of Baseline and End line Data at Final Evaluation Effectiveness and Relevance What are the final values for each of the key performance indicators? EFSP3. Reduced Coping Strategies Index (rCSI) When comparing the baseline and the end line the proportion of households which did not have food or money to buy food had gone down, with 95% of households responding that they lacked food or money to buy it at baseline, compared to 76% of households at end line. The target for the rCSI was 13.61, but the endline value was 21.37, which means that the target was not achieved. The Two Sample t-test, testing the difference in coping strategy for the two evaluations, suggests that the effect is negative, statistically significant and small (difference = -2.55, 95% CI [1.29, 3.80], t(1225) = 3.99, p < .001; Cohen's d = 0.25, 95% CI [0.12, 0.37]). A household with a smaller rCSI, has a better food security situation than a household with a bigger rCSI, meaning that the food security situation at the end line was better than at the baseline (mean rCSI in group Baseline = 23.92, mean rCSI in group End line = 21.37), even if not to the full extent targeted. 13 EFSP4. Prevalence of households with moderate or severe Household Hunger Scale (HHS) score The findings show that the percentage of households with a moderate to severe HHS Score decreased by 6%, from 98% at baseline to 92% at end line. The Pearson's Chi-squared test gives a p-value of 0.0000, meaning that these differences are significant, confirming a finding that the proportion of households experiencing hunger in the project area had decreased. EFSP2. Percentage of households with poor, borderline, and adequate Food Consumption Score (FCS) When comparing findings from the baseline and endline the percentage of households with poor FCS (0-21) had decreased by 28 percentage points (baseline poor FCS: 34% versus endline poor FCS: 6%). The percentage of households with borderline FCS (21.5-35) had also decreased by 10 percentage points (baseline borderline FCS: 52% versus endline borderline FCS: 42%). The percentage of households with acceptable FCS (>35) had meanwhile increased by 39 percentage points (baseline acceptable FCS: 13% versus endline acceptable FCS: 52 %). The Pearson's Chi-squared test gives a p-value of 0.0000, meaning that these differences are significant. Effect sizes for the mean FCS were labelled following Cohen's (1988) recommendations. The Two Sample t-test, testing the difference of FCS by evaluation (mean in group baseline = 25.81, mean in group endline = 37.86) suggests that the effect is positive, statistically significant and large (difference = 12.05, 95% CI [-13.21, - 10.88], t(1394) = -20.29, p < .001; Cohen's d = -1.13, 95% CI [-1.25, -1.01]). These results confirm a finding that household diets in the project areas had improved. OFDA 4. Proportion of children 6-23 months of age who receive foods from 4 or more food groups Findings show that the proportion of children 6-23 months of age who receive foods from 4 or more food groups had increased by 33 percentage points. At the baseline only 13% of households reported that their children received food from 4 or more food groups, compared to 46% at endline. Even though anthropometrics measures were not taken due to the COVID-19 situation the fact that children were reported to be being given 4 or more types of food suggests that feeding habit had improved due to greater availability of food, and also due to increased knowledge among mothers about the need to diversify food given to children. Custom 19. Percentage of men who consider that wife beating is acceptable The percentage of men who considered that wife beating was acceptable had increased by 3.2 percentage points. At the baseline 61% of men believed that beating their wives was acceptable, compared to 64% at end line. While this finding shows no increase in rejection of SGBV the Pearson's Chi-squared test at 95 Confidence Interval, p>.57, suggests that the difference between the baseline and endline is not statistically significant. It would therefore not be possible to consider it proven that men's views on acceptability of SGBV had worsened. Custom 20. Percentage of women aware of where women affected by domestic violence can access the required support There was a small increase in the percentage of women who said that they were aware of where women affected by domestic violence could access the required support (baseline 5% against endline 8%). However again the difference between the baseline and the endline is not statistically significant, as shown by the Pearson's Chi-squared test, at 95% CI, the p￾value >0.05. It would therefore not be possible to consider it proven that there was an improvement in women’s’ awareness about SGBV support services. 14 Table.5 Final values for each of the key performance indicators Indicator Disaggregates Value Statistical test Baseline Endline Sample Total 888 508 Female 675 307 Male 213 201 888 508 FOOD SECURITY INDICATORS EFSP3. Reduced Coping Strategies Index (rCSI) Households that did not have enough food or money to buy food No 45 (0.5%) 123 (24%) Pearson's Chi-squared test with Yates' continuity correction1 data: Evaluation and Insufficient food X-squared = 109.9, df = 1, p-value < 2.2e-16 Two Sample t-test data: Reduced Strategy by Evaluation t = 3.9872, df = 1225, p-value = 7.082e-05 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: 1.292835, 3.797593 sample estimates: Mean in group Baseline Mean in group End line 23.91924 21.37403 Yes 842(95%) 385(76%) Min. 0 0 1st Qu. 17 14 Median 23 20 Mean 23.92 21.37 3rd Qu. 30 27 Max. 56 56 Standard deviation 10.28283 10.57662 EFSP4. Prevalence of households with moderate or severe Household Hunger Scale (HHS) score Percentage of households with moderate to severe HHS Score 98% 92% Pearson's Chi-squared test data: Evaluation and Hunger Score Low(0-1) 17(2%) 34(8%) X-squared = 237.68, df = 2, p-value < 2.2e-16 1 Fienberg, S. E. (2007). The analysis of cross-classified categorical data. Springer Science & Business Media. https://www.sciencedirect.com/science/article/pii/B9780128170847000140?via%3Dihub 15 Moderate(1.5-3) 304(35%) 326(73%) Severe(3.5-6) 555(63%) 84(19%) HEALTH AND NUTRITION INDICATORS EFSP2. Percentage of households with poor, borderline, and adequate food consumption score (FCS) Percentage of Hhs with FCS of 0-21 (Poor) 305(34%) 31(6%) Pearson's Chi-squared test data: Evaluation and Consumption Score X-squared = 287.79, df = 2, p-value < 2.2e-16 Two Sample t-test data: Consumption Score by Evaluation t = -20.292, df = 1394, p-value < 2.2e-16 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -13.20994 -10.88104 sample estimates: mean in group Baseline mean in group End line 25.81081 37.85630 Percentage of Hhs with FCS of 21.5-35 (Borderline) 464(52%) 214 (42%) Percentage of Hhs with FCS of > 35 (Acceptable) 119(13%) 263 (52%) Min. 3 10.5 1st Qu. 20 29 Median 25 35.5 Mean 25.81 37.86 3rd Qu. 30.5 45 Max. 112 88 Standard deviation 9.47 12.50 OFDA 4. Proportion of children 6-23 months of age who receive foods from 4 or more food groups 0 to 3 food types 335(87%) 137(54%) Pearson's Chi-squared test with Yates' continuity correction data: Evaluation and Food type 4 to 7 food types 50(13%) 115(46%) X-squared = 82.891, df = 1, p-value < 2.2e-16 SEXUAL AND GENDER BASED VIOLENCE Custom 19. Percentage of men who say that wife beating is acceptable No 83(39%) 72 (35.8%) Pearson's Chi-squared test with Yates' continuity correction data: Evaluation and Male Acceptability of GBV Yes X-squared = 0.31304, df = 1, p-value = 0.5758 130(61%) 129 (64.2%) No 640(95%) 283(92%) Pearson's Chi-squared test with Yates' continuity correction data: Evaluation and Knowledge of GBV organization Yes X-squared = 2.1443, df = 1, p-value = 0.1431 35(5%) 24(8%) 2.2. Un-conditional Food Distribution (UCFD) To what extent has the project achieved its goals with regards to each respective sector? Unconditional Food Distribution activities were effective in all project areas and provided target communities with sufficient food. These were a real and priority response to the needs of communities identified during the initial assessment. The project therefore achieved its objectives related to general food distributions. Were the correct groups targeted by the project? How was the participant identification process conducted? The targeting of communities and villages was in line with vulnerability criteria defined in the project proposal. Vulnerability surveys in target communities were organized and the registration process followed, including sensitization sessions for target communities on the project in general and the activities in particular. All the target households identified and present in the village were validated via ration cards. All households benefiting from other project interventions should receive in-kind food assistance, with a beneficiary validation process having been carried out to create the project database. TEFA project implementation successfully targeted 7 categories of beneficiaries as in section 1.2 above. According to the October 2020 annual report the project reached 19,067 beneficiary households, made up of 7,203 households in Moba territory and 11,864 households in Kalemie territory. Selection of FDPs (Field Distribution Points) was done based on proximity to villages, to avoid long travel distances for beneficiaries during distributions. Were interventions relevant to targeted participants considering their initial vulnerabilities? Food distribution activities were relevant and effective vis-à-vis the target groups, and the level of vulnerability of the targeted households was successfully reduced compared to the situation before the start-up of project activities. How well did implementation processes adhere to underlying principles and activity protocols? Project staff had been trained on the fundamentals of the project and had knowledge of activity protocols. Trainings in dealing with COVID-19, conflict sensitivity, the environment, sensitivity to identification criteria and handling of complaints had been carried out. Staff also testified to having had activity specific trainings on food management and distribution activities, while food management procedures manuals were found to be available. The participation of communities and beneficiaries was effective in the implementation of unconditional food distribution activities, which were carried out in line with the project plans, with target communities involved in the distribution through local committees. Post-distribution verification surveys were additionally carried out and target groups were sensitized on the complaints handling mechanism, with community complaints management committees set up and trained. 84.7% of complaints were evidenced as having been successfully handled, i.e. 409 out of a total of 483 complaints received, including: - 188 (38.8%) relating to the distribution of food. - 178 (36.9%) relating to the distribution of seeds and NFI kits. - 117 (24.2%) relating to the identification of beneficiaries who were absent during the identification and validation. Linked to this suggestion/complaint boxes were found at all FDPs visited during the assessment. However, some beneficiaries felt that FH resolutions or responses to some complaints were not satisfactory e.g. in relation to complaints about the quality of food. Beneficiaries who participated in the focus groups talked about challenges with hard beans and the poor quality of flour and palm oil. Overall community awareness on the complaints mechanism appears to have been insufficiently deep. The majority of beneficiaries admitted having been informed from the start of the project of the existence of the complaints mechanism, but some were unable to explain how the complaints mechanism worked. Did activities go in accordance to plans? The project started in September 2019 but distribution activities started in March 2020 with awareness raising, and the establishment of general food distribution monitoring committees took place in April 2020. According to the monitoring and evaluation reports of the project, as well as the statements of FH staff, food distribution activities started late, compared to the planned October to December 2019. This delay was attributed to a combination of factors, with many of the project staff having started work 6 months after the contractual start of the project (in February 2020); with poor road conditions having hampered the delivery of food to the field, and with there having been delays in the delivery of food by suppliers. What were the drivers and obstacles to the achievement of the targeted objectives? As food distribution activities were carried out in a context of emergency and extreme vulnerability, especially in terms of access to food, the target communities were receptive to food distribution activities. This facilitated the achievement of the project objectives relating to the unconditional distribution of food. Broader factors however proved obstacles to the achievement of project goals over time such as the COVID-19 pandemic, the rainy season, the poor condition of the roads, the delay in the supply of food by the suppliers, and insecurity linked to the conflicts in some project areas. Efficiency What was the level of efficiency of service delivery of associated goods? The distribution of un-conditional food seemed be efficient. According to staff local purchase was a good approach as the food that was purchased suited beneficiaries’ preferences. However, the transportation to distribution sites seemed not be very efficient, because of the poor quality of road. In addition, beneficiaries raised issues about flour and oil that had gone bad meaning that they could not use it for consumption. Impact and Attribution What significant changes (positive/negative, planned/unintended) have been observed in the life and environment of people and groups directly or indirectly causally related to the project? According to the beneficiaries met in FGDs the availability of food in their environments, resulting mainly from the distribution of food made by FH and provision of inputs for production, had significantly contributed to reduction in thefts of food produced in the fields. This had a positive effect via the reduction of conflicts locally, since theft in the fields was one of the causes of the conflict. Group leader Mazonde shared that because of hunger people even used to steal cassava during the processing phase of retting or fermentation in water (this transformation practice is generally done in rivers which are not far from the field). According to the group leader, the theft of cassava had however decreased during the project. The observed changes seem mainly related to the interventions of FH, although other actors have also intervened in the TEFA project area. WFP during the second quarter distributed food on the Kalemie-Kabwela axis, including the Lwanika Health Area in Kalemie, while organizations such as World Relief, CRS and Concern have also intervened in the area, but either in different communities, or with different activities. In addition, some communities stated that World Vision and AIDES had also made distributions in certain villages which had been targeted by the project, but in small quantities. Sustainability Are the net benefits of the intervention likely to last? Food distribution usually provides quick benefits in terms of improvement of nutrition status, but these benefits will last only if beneficiaries are able to do their own food production. In this light FH has used a sound approach for TEFA, which combines food distribution with support to agricultural production through the distribution of seeds and tools and technical support to farmers. If the security situation remains stable, and the food production picks up, then the nutritional status that had been achieved through food distribution should last. Coherence To what extent is the intervention compatible with other interventions carried out by FH? The general food distribution was compatible with other TEFA project activities, as all were a response to the problem of food insecurity. In addition to the general distribution of food FH organized road rehabilitation activities (via FFA) and water points, the distribution of seeds and agricultural tools, distribution of NFI, and training in the framework of the TEFA project. Recommendations What would be the top recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - A more effective food quality assurance system should be put in place at all levels, for purchasing, storage and distribution, to respond to complaints on food quality or condition and to mitigate any health risks to communities when consuming food supplied. - Position stocks in advance at sub-warehouses in implementation areas, to mitigate risks of delays to distribution caused by road conditions during the rainy season or difficulties in river crossings (e.g. at Nyemba to circumvent the risk of delay in crossing the Lukuga River by boat). - In order to strengthen commodity tracking it would be advisable to put in place summary reports on the use of food, including the quantities planned, ordered, received, damaged, and balances remaining. - Improve the level of awareness on processes for the management of complaints at community level, given that there is a good level of understanding of the fact that complaints mechanisms exist, but a limited level of understanding of how these mechanisms operate. Taking into account barriers to understanding and/or poor retention capacity of beneficiaries it is recommended to increase the tools used to raise community awareness of how complaints mechanisms work, and have more regular and sustained communication on this. 2.3. Food for Asset (FFA) Effectiveness and Relevance To what extent has the project achieved its goals with regards to each respective sector? The rehabilitation of roads was a real and priority response to the needs of the communities identified during the initial assessment, due to allowing both access to the fields and access for incoming food. The rehabilitation of agricultural feeder roads took place in all project areas, while there were also a number of cases of additional roads which were not initially targeted, but which were rehabilitated. A total of 111.2 km of roads were rehabilitated, compared to 162.5 km planned). The distribution by section is shown in the table below. Table.6 : Data on rehabilitation by road section Health area Road Road Mileage (in Km) Comment Planned Completed Remainder % TERRITOIRE DE KALEMIE Tundwa Mwingiji -Kanketchou 13,0 10,0 3,0 76,9% Tundwa Emilingombe - Kyanza 7,0 7,0 0,0 100,0% Emilingombe - Kibugu 10,0 3,8 6,2 38,0% Mushaba II - Mandela 9,0 3,0 6,0 33,3% Lukumbe Kyambo - Matambwe 30,0 8,0 22,0 26,7% Lwanika Pontien - Kisuba 5,0 5,0 0,0 100,0% Lwanika Lwanika - Kimbinje 15,0 5,0 10,0 33,3% Lukombe Mulenda Kazadi - Kasenge 12,0 7,0 5,0 58,3% = Mugabo￾Njala Sub-Total Kalemie 101,0 48,8 52,2 48,3% MOBA TERRITORY Mazonde/ Mwindi Pont Mulobozi – Moket 20,0 20,0 0,0 100,0% Mazonde Mwele- Mpama 8,5 8,5 0,0 100,0% Kansabala Katolola Mitonga 10,0 10,5 -0,5 105,0% Mwindi/ Mazonde Moket meteka 10,0 10,0 0,0 100,0% Kansabala /Lungulun gu Kala- Kalolo 13,0 13,4 -0,4 103,1% Sub-Total Moba 61,5 62,4 -0,9 101,5% TOTAL GENERAL 162,5 111,2 51,3 68,4% Ref: Statistics of rehabilitated road sections, Detailed Implementation Plan, Annual Report for October 2020 and information collected in the field. Rehabilitation activities were not finalized as planned on some roads, especially in Kalemie territory, for reasons such as: - Insufficient populations over certain stretches of road. - Lack of adequate materials for the execution of the works by the road menders (especially for roads on which there were large invasive trees to clear). - Delays in distribution of food (FFA) to road menders for tasks completed, due to late delivery of food by suppliers. - Confinement relating to the COVID-19 pandemic. - Impassability of roads in certain areas during the rainy season. Were the correct groups targeted by the project? How was the participant identification process conducted? The targeting of communities was based on criteria such as: - Inaccessibility of villages or fields. - Transport routes for produce from the fields to major consumption centers. - Willingness of beneficiary communities to work on a FFA basis. - Presence of sufficient able-bodied persons in the target households. The beneficiary identification process was the subject of community sensitization, followed by an identification campaign. 18,000 households were initially expected to participate in FFA activities but given the configuration of the villages, the majority of which were near main roads, it proved difficult to find enough households that could participate on the roads to achieve the goal. In FGDs and KII with community leaders the majority felt that the right groups were targeted according to the criteria. It seems however that the the needs assessment for rehabilitation of agricultural feeder roads was not done as comprehensively as it could have been, as there are roads that were selected which did not seem to bring tangible benefits to communities, e.g. roads that were far from villages. Were interventions relevant to targeted participants considering their initial vulnerabilities? FFA activities relevant at 2 levels: in relation to target groups, and in relation to the need to rehabilitate infrastructure. The households that participated in FFA activities were found to be those in vulnerable groups, and FFA food was reported to have been used by households largely for consumption, with a part sold in order to raise funds to purchase other necessities. The road rehabilitation activities had also made it possible for communities to access their fields by road. The activities therefore constituted a real and priority response to the needs of the beneficiary communities, who were delighted to find themselves with improved roads. How well did implementation processes adhere to underlying principles and activity protocols? FFA activities were found to be guided by the underlying principles and activity protocols: - The project was multisectoral considering the package of activities provided to target communities (unconditional distribution of food, rehabilitation of agricultural feeder roads via FFA as conditional distribution of food, distribution of essential household items/NFIs, distribution of seeds and agricultural tools, and practical training for resuming rural activities, etc.) - The project targeted the most vulnerable households, with vulnerability mainly linked to the movements of internal populations following inter-community conflicts, which caused material losses including agricultural inputs, essential household items, etc; - The project focused on improving access to communities and fields via road rehabilitation activities. - Project staff had been trained on the fundamentals of the project and had knowledge of activity protocols. Trainings in dealing with COVID-19, conflict sensitivity, the environment, sensitivity to identification criteria and handling of complaints had been carried out. The rehabilitation activities were implemented according to project plans, with target communities involved through local committees. Target groups had been sensitized on the complaints management mechanism and local complaints management committees set up. Did activities go in accordance to plans? The rehabilitation activities had not progressed fully in accordance with plans, with an overrun of around 5 months and with not all planned work completed on certain sections (see detailed table of road completion and context on constraints above), although there was a good level of community satisfaction with the rehabilitation completed. What were the drivers and obstacles to the achievement of the targeted objectives? The drivers which contributed to the achievement of objectives relating to the rehabilitation of roads were: the will of the communities to carry out the rehabilitation works and the targeting and availability of people able to do the physical work. The food received by the beneficiary road menders as payment for carrying out the work also constituted a motivation and an encouragement at the time of distributions. However, there were also challenges that slowed down the completion of the work, including: Care Group activity at Sylvano, Kalemie territory Complaints box at Sylvano, Kalemie territory Complaints box at Sylvano, Kalemie territory - The COVID-19 pandemic in D.R. Congo from March 2020. This had repercussions on the organization of awareness-raising and identification activities. - The rainy season caused a lot of delay in the execution of the works, especially on the sections without homes. - The insufficiency and/or lack of materials for road works caused delays. - Delays in the payment in FFA food for completed tasks and or non-payment, due to the delay in the delivery of the food by the suppliers (and/or the incomplete constitution or quality of food rations at the time of distribution) proved a discouragement. Some beneficiaries who worked on road rehabilitation activities reported they did not receive beans or maize. - Long sections where the project did not provide for transport for the road menders. - The addition of new road sections to be rehabilitated in the last quarter of the project. Efficiency What was the level of efficiency of service delivery of associated goods? Overall the project was efficient, as it used labour from the local community at limited cost in FFA terms, as well as simple tools that were procured in bulk. However, it appears that the choice of certain road sections was not well thought through. For instance, some staff and beneficiaries mentioned that some roads that were chosen for rehabilitation did not have any nearby villages. This resulted in a situation where it was difficult to get workers to some road sections due to lack of transport being provided. Such roads without villages are also less likely to be used by communities, meaning that some rehabilitation resources may not have been used most efficiently. In addition, the close out project report shows a lot of material that was not used. Impact and Attribution What significant changes (positive/negative, planned/unintended) have observed in the life and environment of people and groups directly or indirectly causally related to the project? The project was able to improve access to the fields by improving harmony between two ethnic groups in conflict, thanks to community-building activities and the sensitization of a large part of the population on conflict management. In addition, the rehabilitation of agricultural roads by the project has contributed to improving the flow of agricultural products and the movement of people and goods; including humanitarian interventions by other projects and other actors. The Mazonde group leader testified that following the rehabilitation of the roads by FH now even buses were arriving in different villages of his group for the sale of agricultural products. This result can be directly attributed to FH, because it is the only organization that organized rehabilitation of agricultural feeder roads locally. Sustainability Are the net benefits of the intervention likely to last? Community members generally reported believing that they would continue to benefit from positive effects relating to the rehabilitation works carried out, in the sense that the rehabilitated roads would remain in use, and members would arrange for periodic maintenance to keep the roads in good condition to remain useful. However, in some sections which had been partially rehabilitated, and/or where rehabilitation materials had not been distributed to roadmenders, there is a risk that the changes will not last due to lack of materials and discouragement. Some beneficiaries are also claiming still to be owed food for the completed tasks carried out (Lubunda and Mugabo-Njala section in Kalemie territory). Road Mubunda Mugabo-Njala, Kalemie territory Road Katolola-Mitonga, Moba territory Coherence To what extent is the intervention compatible with other interventions carried out by FH? The rehabilitation of road infrastructure using the FFA approach was complementary to other activities carried out under the TEFA project, as for example these roads can contribute to the easier flow of production which will be achieved through the provision of agricultural production materials (distribution of seeds and tools). Recommendations What would be the top recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - Adopt a payment in cash approach for the rehabilitation of infrastructure instead of using the FFA approach, especially when the unconditional food distribution approach has already been made in the same environment. This will have the advantages of: Increasing the motivation of individuals to participate in the rehabilitation works. Reducing logistical challenges related to the purchase and distribution of food. Allowing beneficiary households to meet other non-food needs more easily, as opposed to selling part of the food received in order to buy other essentials. Reducing the impact of food distributions on food prices in the local market, while indirectly supporting sellers in such markets. Go beyond the approach of payment by completed tasks (in food or cash) for road infrastructure rehabilitation activities, so as to plan in more detail during project design the tasks expected and the resulting cost, and so be able to compare planned vs actual efficiency. Road rehabilitation: Sambala bridge, Mwele, Moba territory Strengthen and/or revitalize the capacities of local committees in the management of rehabilitated infrastructure, through enhanced trainings. Donate rehabilitation equipment for agricultural feeder roads to local road section management committees at the end of project works, to facilitate ongoing maintenance and further works beyond the lifetime of the project. 2.4. Non Food Items (NFI) Effectiveness and relevance To what extent has the project achieved its goals with regards to each respective sector? NFI distribution activities were carried out in the project intervention areas to meet the needs identified within the most vulnerable households. The October 2020 annual report indicates that 1,715 of the most vulnerable households out of 1,727 planned households were identified, validated and served with NFI kits. The majority of beneficiary households interviewed through FGDs declared that they benefited from the full planned NFI kits (comprising 2 blankets, 1 cooking pot, 1 basin, 3 empty containers, 3 plates, 3 plastic cups, 3 spoons, 1 metal ladle and 3 bars of soap); but some households reporting not receiving certain items as part of their kits (e.g. the soap, the blankets or the empty containers), with the figures from distribution records given in the table below. Table.7 Final distribution values of NFI kits and materials by sex Description Quantity per household Target Achievement Balance Male Femal e Total Targeted Households 1 1727 692 1023 1715 12 NFI kits 1 1727 692 1023 1715 12 Blanket (piece) 2 3454 1384 2046 3430 24 Cooking pot (piece) 1 1727 692 1023 1715 12 Plastic basin (piece) 1 1727 692 1023 1715 12 15L empty container (piece) 3 5181 2076 3069 5145 36 Plate (piece) 3 5181 2076 3069 5145 36 Plastic cup (piece) 3 5181 2076 3069 5145 36 Spoon (piece) 3 5181 2076 3069 5145 36 Metal ladle (piece) 1 1727 692 1023 1715 12 Soap (bar) 3 5181 2076 3069 5145 36 NFI kit distribution activities were reported to have been effective in all project areas, enabling target communities to have essential NFI materials after returning to their respective villages. This was a real and priority response to the needs of the communities as identified during the baseline assessment. The project almost wholly achieved NFI kit objectives. Were the correct groups targeted by the project? How was the participant identification process conducted? The selection criteria for beneficiary households were based on the level of vulnerability of certain households defined in the project proposal (the target population including the elderly, physically impaired, pregnant and lactating mothers and the terminally ill will receive NFIs), and targeting was found to be in line with plans. Sensitization sessions were carried out to inform community members about the intervention in general and the identification criteria. The number of households planned for this activity was 1,727. A total of 1,715 households were reached with NFI kits (99.3% of the target), of which 692 were male headed and 1,023 were female headed. All households identified and present in a village during the validation of lists were validated via ration cards. Were interventions relevant to targeted participants considering their initial vulnerabilities? The distribution activities of the NFI kits were relevant for the target groups given the level of vulnerability of the households. The selection criteria were very strict, since plans had been made and budgets set based on distributing to 1,727 households only. The beneficiaries appeared to have been in significant need, as in the case for example of the elderly who lacked kitchen utensils, body protection, etc within their households. However there seemed to be some issues with beneficiaries not understanding the use or the importance of some of the items they had received, suggesting some gaps in the sensitization process, e.g. according to a staff member, it was found that some Twa community members had kept the soap that they received for 3 months without using them. How well did implementation processes adhere to underlying principles and activity protocols? NFI kit distribution activities were found to be implemented according to the project plans, with target communities involved in the distribution through their local committees. The vulnerability criteria provided for in the project were met to select the beneficiaries. According to staff FH also participated in coordination meetings in which the activities were harmonized with other actors, to avoid duplication. Did activities go in accordance to plans? The project plans did not specify planned start and finish dates for the NFI kit distributions, but the annual report of October 2020 states that NFI kit distributions started in the second quarter of 2020 and intensified in the third quarter, with 1,715 kits distributed in total as in the table below. The majority of beneficiaries interviewed confirmed NFI kits were distributed in full, although with certain items reported to have been missing in some instances as above. What were the drivers and obstacles to the achievement of the targeted objectives? The relatively straightforward nature of the assistance facilitated the implementation of the activity, especially given the urgency and extreme vulnerability of targeted households, and the fact that a minority of total beneficiaries were being targeted through this activity. However, similar to other TEFA project activities, the distribution of NFIs also faced challenges, due to the COVID-19 pandemic, the poor condition of the roads, the period during which the distributions took place (rainy season), and the insecurity linked to the ongoing conflicts in certain areas of project implementation. Impact and Attribution What significant changes (positive/negative, planned/unintended) have been observed in the life and environment of people and groups directly or indirectly causally related to the project? The beneficiary households consulted during the focus groups declared that the NFI kits received enabled them to cook, fetch water, be clean, and also to protect themselves against the cold. Positive changes relating to NFIs can be attributed entirely to FH since no other organization distributed these items in the area. Sustainability Are the net benefits of the intervention likely to last? Apart from soap the items received in the framework of the NFI distribution are non￾perishable, so if households store them well, they can use them for a long time. Beneficiary households expected that they would be able to continue to use items for many years to come if at least security continued to stabilize. Factors that may limit the sustainability of NFI kits distributed, however, include the vulnerability of recipient households, who may be tempted to sell items received if they need the money; and the quality of these articles. Some households consulted said that some items had started to deteriorate (e.g. pots), but it could not be determined whether this was due to poor maintenance or poor quality of items. Coherence To what extent is the intervention compatible with other interventions carried out by FH? The distribution of the kits was compatible with the other activities of the TEFA project because the items received such as kitchen utensils or soap can be used to prepare meals, or for cooking hygiene, therefore forming part of the response to the problem of food insecurity. Recommendations What would be the top recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - Reinforce Monitoring and Evaluation team visits to project activities with more frequent visits be responsible managers at the provincial level, to make it possible to respond more quickly to questions or difficulties in implementation. - Assign a specific logistics team or focal point for emergency focused projects to enable stronger engagement and quicker responses to project issues requiring a quick turnaround, lessening inefficiencies involved in activities needing to be placed on hold pending solutions. 2.5. Provision of agricultural inputs and training Effectiveness and Relevance To what extent has the project achieved its goals with regards to each respective sector? Due to the delayed start of activities and the security situation the agriculture components of the project did not achieve their targets in full. Overall, out of 9,000 beneficiaries targeted 8,135 received cassava cuttings and sweet potato vines but only 1,167 received maize and bean seeds (according to the annual report, however field staff reported that a total of 8,687 beneficiaries had received maize and bean seeds). Project staff explained that they had to change distribution plans for agricultural inputs, as they had already missed the first main planting season (Season A, September 2019), and were almost missing the second season as the first distribution of agriculture inputs took place in March 2020. While TEFA’s Outcome 1.1 states that “Farmers engage in food production activities”, and the indicator is “100% of farmers receiving seeds who planted crops during the agricultural season”, the change in the distribution plan undermined the possibility for planting all agricultural inputs. Most beneficiary interviews, however, confirmed that they were able to engage in agricultural activities as planned. Table.8 : Quantity of seeds received by beneficiaries (based on annual report) Activity Target Achieved Male Female Total Remainder Distribution of legume and cereal seeds (bean seed and corn grain) 9,000 490 677 1,167 7,833 Distribution of tuber and root food seeds (sweet potato and cassava cuttings) 9,000 3,063 5,072 8,135 865 Table.9: Distribution of maize and been seeds (based on verbal information from field staff) Overall the project enabled a large majority of targeted households to rehabilitate their basic means of production by providing tillage tools and food seeds. In particular the Sawasawa variety of cassava that the project distributed to 8,135 beneficiary households is tolerant of an epidemic disease known as the African cassava mosaic, as reflected in FGDs and observation, which should increase the production of this crop in the area and improve the availability of food and household income, cassava flour being a staple food in the areas. - Agricultural calendar Cassava and sweet potato cuttings were distributed in May 2020, which according to beneficiary farming households was not appropriate because this month corresponds to the end of season B and the start of the dry season. This distribution coincided with the start of the dry season, with a negative effect on the regrowth rate of planted cassava cuttings. According to beneficiaries consulted during FGD, and based on the observation of some of the beneficiaries’ fields, the regrowth of the cassava cutting seemed to be high. The situation was different for sweet potatoes vines as most beneficiaries reported that they had not plant them because they were already rotten when they received them. However due to the lack of a systematic post distribution assessment, it was difficult for the evaluator to give a precise estimate. - Quantity of tools and seeds distributed The quantities of maize and bean seeds distributed to beneficiaries appear not to have been standard. Beneficiaries consulted during FGDs spoke of different quantities because the distribution was made by the local scoop called "Kifaru" which can be subjective. Most said that the amount of seeds and tools received was insufficient for the acreage that they own. Activity Area Target Achieved Distribution of legume and cereal seeds (bean seed and corn grain) Kalemie 5,116 5,116 Moba 3,870 3,870 Total 8,986 8,986 Table.10 : Inputs received based on farmer feedback Agriculture Inputs Types Variety Benefit Quantity received Beneficiaries’ view Seeds Maize Babungo Local variety 1,5 Kifaru (local scoop) Good variety Quantity not enough Beans D6 Kenya 0,5 Kifaru (local scoop) Good variety Quantity not enough Cassava cuttings Sawasawa, Tengu, Ulangala Tolerant to African cassava mosaic One bag of 80 to 100 LM Good variety Quantity not enough Sweet potato vines Rotten Tools Hoe Coq Useful 1 piece Rake Less useful 1 piece Poor quality - Quality of inputs (seeds and tools) Among the agricultural tools received the hoes were the most appreciated by beneficiaries based on FGDs, because they were commonly used for agricultural activities and because of the good quality. The rake was the least appreciated because it was seen as unsuitable for food crops and because it was seen as poor quality in terms of the solidity of the materials. The quality of the seeds varied depending on the types of seeds distributed, with the bean and maize seeds highly appreciated, the Sawasawa cassava seeds highly appreciated though with some local varieties also being included in small proportions, but the sweet potato cuttings that were distributed were reported to not have been rotten and so unplantable. Were the correct groups targeted by the project? How was the participant identification process conducted? The targeting of beneficiaries for agricultural inputs was good in that the planned selection criteria provided were respected. In FGDs there was repeated feedback that FH had registered the beneficiaries on the basis of criteria (such as the number of people in the households, their accommodation, their food situation, their children, the number of people in the households, or the goods that the household had). However it appears that FH did not take into account certain context specific issues in the choice of agricultural activities per areas. In some KIIs it was reported that many households in the Twa communities are not farmers, but they also received the same generic agricultural inputs planned for the project. For the Twa, it took a lot of effort to sensitize them to practice agricultural activities, rather than the hunting and gathering activities that they were accustomed to. Linked to this the average area of land used by a Twa household was small, meaning they were likely to get less food from their harvest than beneficiaries from Bantu communities. Were interventions relevant to target participants considering their initial vulnerabilities? Agricultural input interventions were relevant because they enabled beneficiary farmers to return to their villages, to access their fields and together with necessary inputs (seeds and agricultural tools) for the revival of their agricultural activities. All the beneficiary farmers encountered in FGDs explained that during conflict it was difficult for a farmer to exploit his distant fields, but that now they exploit their distant fields without problems with the aid of support provided. The relevance of the project in this area is also confirmed by the needs analysis, as the rapid conflict analysis report produced by FH reports that 22% of the population of Tanganyika Province were displaced following the inter-ethnic conflict in 2017, and that this displacement had the impact of causing significant disruption to agricultural production and livelihoods. Cassava and maize met the needs of the population well in terms of the choice of seeds, given that these crops play an important role in the daily diet of the populations of Kalemie and Moba, and also in terms of boosting household income given the significant demand for these crops at the local level and even in the export market for other territories and provinces. How well did implementation processes adhere to underlying principles and activity protocols? There was a mixed picture in terms of protocols related to seeds and tools distribution being followed. On the positive side the distribution of seeds and tools correctly took place after food distributions, to enable beneficiaries to have enough food to eat while they were waiting to get their own harvest and to ensure that seeds would not be consumed. There were also however some protocols that were not followed. Firstly the distribution was meant to take place in the main planting season (Season A, 2020) but actually took place in Season B 2020 (March-May 2020), or in Season A, 2021. Secondly the project was due to train beneficiaries on agricultural techniques before they received the agricultural inputs, but beneficiaries reported that in FGDs they were trained on the distribution day or after. Training topics particularly appreciated by beneficiaries were seed conservation, respect for spacings and semi density, but topics that were less valued were crop rotation and row sowing, and more focus was desired on pest/disease control of crops and soil fertilization. Did activities go in accordance to plans? As above there was a generally positive picture but within this there were challenges with delays in the distribution of cassava and sweet potato cuttings, which were distributed in May 2020 instead of in February 2020 as planned. This was attributed to the impact of COVID-19 on economic activity but a constraint also seems to have been that the TEFA project had not recruited a Logistics Manager and support was from the logistician of the DFSA project. What were the drivers and the obstacles to the achievement of the targeted objectives? There were factors external to the project which either contributed to or hindered the success of agricultural activities. Supportive factors included beneficiaries being given access to land for farming after returning from displacement, on which to use the seeds and tools provided through the project; while hindrances included COVID-19 restrictions which contributed to distribution delays and limited awareness-raising activities due to meetings of more than 10 people being prohibited. Despite these challenges however support from FH staff and monitoring of activities was reported to have been strong. Impact and Attribution What significant changes (positive/negative, planned/unintended) have been observed in the life and environment of people and groups directly or indirectly causally related to the project? The most significant change caused by the distribution of seeds and tools was that food production activities could be started again by community members who had not been able to grow their own food because of the effects of conflict. It was difficult to assess in full the Lead Farmer member field in Kabulo village, serving as training model Lead Farmer member field in Kabulo village, serving as training model effectiveness of restarted food production at the time of the evaluation (February 2021) due to the long life cycle of certain crops like cassava (1 years ) or maize (up to 5 months) but there were already changes visible as a result of the TEFA as for instance beans and maize had been already been successfully harvested in Kalemie. In addition, beneficiaries had been equipped via trainings to be more resilient to various hazards that can affect food security. Sustainability Are the net benefits of the intervention likely to last? The majority of beneficiary farmers who have already harvested have shown that agricultural activities are sustainable because after harvesting they are still able to reseed from their produce. In addition, beneficiaries appreciated the training on seed conservation which will significantly help them to sustain their agricultural activities. The main factor that could affect the sustainability of agriculture related activities is any resurgence of conflict. Coherence To what extent is the intervention compatible with other interventions carried out by FH? TEFA was designed to respond to urgent food security needs due to ethnic conflicts between the Bantu and the Twa in the territories of Kalemie and Moba where the DFSA project was already operational, to prevent this development project from being very unbalanced. However, certain activities included in these two projects are coordinated, such as sharing the same office, joint capacity building, and certain logistical aspects. TEFA was executed in the same operational area as the DFSA project. TEFA through its emergency activities enabled many households who moved following the Bantu and Twa conflicts to return to their villages, with knock-on benefits for the continuation of DFSA activities. Recommendations What are the top recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - It is recommended for the technical team of project agronomists to be involved in the process of seed quality testing and selection at SENASEM or from other suppliers and it is recommended that FH put in place a system of seed testing by FH agronomists, in order to cross-check the tests that are done elsewhere. - Taking into account the agro-ecological and cultural diversification of the territories of Kalemie and Moba, it is important to diversify the crop species to be distributed to the beneficiaries and to diversify the varieties of seeds for more production security (e.g. of peanuts and rice). - Agro-ecological and climate-resilient techniques should be reinforced such as mulching, soil water protection and conservation techniques, crop rotation, diversification of species and varieties, in addition to reinforcing trainings on pest and disease control of crops. - The project although an emergency intervention could also develop the seed multiplication activity in the community for greater impact and sustainability, as well as coordinating more closely with the longer-term DFSA project and grassroots organizations. - Ensure that food production activities are well supported and planned so that harvests take place before the last rations of food provided will have been consumed, to avoid shortages. 4.2.4 Vegetable Gardens Effectiveness and Relevance To what extent has the project achieved its goals with regards to each respective sector? The main indicator that was measured on child nutrition was “OFDA 4. Proportion of children 6- 23 months of age who receive foods from 4 or more food groups”. Due to the current COVID-19 situation, anthropometric measurement were not taken during the end line survey to ascertain any change in the key indicators “EFSP9(Prevalence of global acute malnutrition (GAM)”. The comparison between the baseline and endline shows the following situation: There was an improvement in the consumption of high-quality nutritious food and the health status of households and CU 5 due to the establishment of vegetable gardens in the 137 villages targeted by the TEFA project, according to the mothers’ Care Groups (CG) and leaders in Moba and Kalemie: “Since we had the training with FH on nutrition, we start to eat the vegetables but before that was not the case, only gave either the Ndakala (small fish) or a small piece of meat without vegetables, and from then on this project we changed the way of doing things.” This situation does not however apply equally to all the women or all the 134 CG because there were certain places where the seeds did not produce. Interventions were delivered through CG to facilitate the adoption of essential practices in the establishment of home vegetable gardens. There was provision of inputs or seeds with tillage tools, followed by training of beneficiary members on innovative methods in food security with an emphasis on gardening. In terms of activities, it was reported that all CG members received vegetable seeds for the establishment of gardens, with a demonstration garden also set up in each CG. Trainings focused on the importance of the vegetable garden, improving cultivation techniques, making beds, sowing rows, mulching, making a shade tree, watering techniques, and transplantation. Regular follow-ups were carried out by the TEFA project teams, made up of agricultural and health promoters. Awareness campaigns were also run by CG members to their peers on good nutritional and health practices, as well as on the use of diverse foods. Were the correct groups targeted by the project? How was the participant identification process conducted? The criteria for selection of beneficiaries for the vegetable garden activities were to be a woman of childbearing age, pregnant and breastfeeding. The inclusion criterion was well defined at the start of the project but interventions were not always applied as a package within the same village, with all of the 134 CGs being functional in Bantu villages and so with Twa beneficiaries who meet the criteria coming to meetings in Bantu villages. Were interventions relevant to target participants considering their initial vulnerabilities? The relevance of the project vis-à-vis the targeting of beneficiaries is evident from the fact that it targets the areas most affected by malnutrition and food insecurity as well as the most exposed and vulnerable groups. For market gardening activities the needs were the same for host families and the displaced. There was also the possibility for all beneficiaries to have access to a space or piece of land to establish a family vegetable garden. Participants in the vegetable garden activity declared during a focus group that “even if there are some speculations that have not turned out well but it has helped us a lot for our cooking and for the improvement of the health of our children, especially for us pregnant and breastfeeding women and even for the whole family”, and that “Thanks to these activities, we will no longer be hungry, and our children will be immunized.” To what extent are the implementation processes aligned with the fundamental principles of the project and with the activity protocols? CG activities were implemented according to the principles and protocols of the project, with active participation of stakeholders (village chiefs and community leaders) in the selection of beneficiaries, and awareness-raising activities (including awareness sessions on COVID-19 through a partnership contract with community radio). Did the activities go according to the plans? According to FH all the women members of the CG had been trained on how to set up portion gardens, receiving vegetable seeds and agricultural tools kit consisting of a jerry can, a sprinkler, a hoe and a rake for setting up vegetable gardens. The beneficiaries and members of care groups confirmed to have received support from health and agricultural promoters in setting up their allotment gardens. As with other activities of the TEFA project the start of activities relating to CG vegetable gardens was delayed, with the seeds distributed in February 2020 instead of October 2019, due to recruitment delays and knock on delays in CG set-up. What were the drivers and the obstacles to the achievement of the targeted objectives? There were enabling factors that made the implementation of the vegetable garden activities easier to implement e.g. beneficiaries showed an active interest in the activities, which could be also seen in some places where men were attending training sessions with their wives. The choice of activity was also another factor as it was addressing existing needs among the targeted beneficiaries. There were also some challenges that were identified, such as around quality/germination of certain seeds, the effects of insect pests, cases of a lack of full equipment, low literacy levels which led to a change in training approach to use of songs, and a limited number of health promotion staff to cover a large area (1 promoter to 40 CGs, instead of 1 promoter to 10 CGs that is usually recommended). Impact and Attribution What significant changes (positive/negative, planned/unintended) have been observed in the life and environment of people and groups having a direct or indirect causal link with the project? A particular positive change brought up in FGDs with mothers, besides the direct nutritional and health benefits, was the fact that they had started to sell the surplus of their production, giving them a further source of income. Beneficiaries also recognized that WFP, IRC, Solidarité and others shared a focus on health and nutrition, with WFP for instance distributing food in certain communities. Sustainability Are the net benefits of the intervention likely to last? The training provided helped build local knowledge, experience and skills and the creation of CGs and establishment of vegetable gardens constitute the basis of sustainable activities; although behavior change is a long process and so to consolidate the achievements of the project, FH staff decided to collaborate with local agronomists, who could then continue to train mothers even after the end of the project. Factors that could limit the sustainability of home gardens include: No demonstration field being installed in the Twa villages, which could limit the learning opportunities for mothers in this community. Limited possibilities of ongoing support from government health centers to CGs. Coherence To what extent is the intervention compatible with other interventions carried out by FH? The vegetable garden activities are consistent with other activities of the TEFA project such as food distributions and activities to revive agricultural production, given that all of these contribute to the same overall objective of improving food security. Recommendations What would be the top recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? For better cooperation between the different communities, demonstration sites should be established in Bantu villages as well as in Twa villages, to allow all parties to feel valued. Select the seed supplier carefully, choose seeds resistant to diseases and suitable for the environment, and organize the distribution of tillage tools and seeds on time. Select sites taking into account the accessibility of water for gardening maintenance. Set up a seed bank system or device that can be done through government structures to allow beneficiaries to continue to access seeds even after the end of the project. Collaborate with the agricultural inspectorate through agricultural monitors for the continuity and sustainability of activities, and with the health zone for the continuity and monitoring of malnutrition prevention activities. 2.6. Health and Nutrition Effectiveness and Relevance To what extent has the project achieved its goals with regards to each respective sector? 134 CGs were formed and members benefited from training from health promoters who were trained by the Provincial PRONANUT (National Nutrition Programme representative). Leading Mothers received behavior change training to improve infant and young child feeding practices, taking ownership of health and nutrition activities and sensitizing other pregnant and breastfeeding women to use relevant services. CG members had received training on essential nutritional actions (ENA) during pregnancy, childbirth and early childhood, on water, hygiene and sanitation (WASH) and orientation of women for the integrated management of childhood illnesses (IMCI), on feeding pregnant and lactating women, feeding infants and young children, key moments for hand washing and the use of hygienic latrines, The majority of CGs had received training modules and image boxes in an accessible format for awareness and education, but the quality of training was not uniform according to FGDs, as there were participants who reported that did not receive full training materials. How well did the implementation processes adhere to underlying principles and activity protocols CG activities were implemented according to the principles and protocols of the project, with active participation of stakeholders (village chiefs and community leaders) in the selection of beneficiaries, and awareness-raising activities. To identify beneficiaries FH used a door-to-door approach to identify women to become CG members using predefined selection criteria. Did the activities go according to the plans? The 134 CGs planned by the project were formed, made up of 2,000 women who met the selection criteria, although as with other activities this took place around 3 months later than originally planned. Each CG was headed by a mother leader elected by her peers. Impact and Attribution What significant changes (positive/negative, planned/unintended) have been observed in the life and environment of people and groups having a direct or indirect causal link with the project? Given the duration of the project it is difficult to estimate the impact at this time as many health and nutrition activities are linked to behavior change. However, some women who participated in FGDs mentioned that thanks to the knowledge acquired, they had started to attend health facilities and apply the advice on infant and young child feeding, as well as apply the advice on the 5 key moments of hand washing for hygiene. In addition, the activities of CGs contributed to improving social cohesion, especially among mothers who came from ethnic groups that were previously in conflict. FH played a large role in the changes observed for nutritional aspects, but the contribution of other humanitarian actors who intervened in the region (WFP, IRC, CRS, etc.) should also be taken into account. Sustainability Are the net benefits of the intervention likely to last? While trainings appeared effective while project support continued it appears that the exit strategy relating to health and nutrition activities was not well designed, as in the context of poverty it is likely to be challenging for Leading Mothers to continue convening groups without support, and as it would have been best to have had fuller integration of health and nutrition activities into the health system at the local level (by working with health centers more directly, as opposed to on a referral basis). Recommendations What would be the top recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - Increase availability of staff with relevant experience to monitor activities, to ensure that targeted support is provided and issues like on training materials are addressed in good time. - Consider CGs mixed by gender, to strengthen the engagement of men as well as women in the search for solutions to health and nutrition problems, and to build upon the interest of men in participating in these TEFA project activities. - Work more directly with government health and nutrition structures (community relays, agricultural monitors), to ensure the sustainability of health and nutrition actions 2.7. Conflict Management Effectiveness and Relevance To what extent has the project achieved its goals with regards to each respective sector? At the start of the project a rapid conflict analysis study was conducted to understand local conflicts, in order to ensure the conflict sensitivity of the project and respect for the principle of "Do no harm". As confirmed in FGDs the main activities that were able to open space for collaboration included sensitization and training of key leaders from the Bantu and Twa communities on conflict management and SGBV; training of FH staff; and the organization of food distribution, agriculture and FFA activities that included messaging on conflict management and brought together the two competing communities to reduce conflict and promote peaceful coexistence, while reducing resource pressures contributing to conflict. Were interventions relevant to targeted participants considering their initial vulnerabilities? This intervention was aimed at all households benefitting from the project in addition to more targeted responses to specific vulnerabilities, given that inter-ethnic conflict is a challenge facing all communities in the project area in the current context. How well did the implementation processes adhere to underlying principles and activity protocols The implementation approach adopted by FH, such as the selection of common distribution sites for food and agricultural inputs where all the communities could meet and interact, appears to have been a good strategy which made it possible to bring the communities together through practical activities. Did activities go in accordance to plans? As with all other activities there were initial delays due to project start-up challenges linked to recruitment, logistics and COVID-19, but once underway the activities were delivered in a timely and effective manner, except for in regards to feedback on trainings which it was reported could have been facilitated more effectively. What were the drivers and obstacles to the achievement of the targeted objectives? A key driver in regards to the reach of this activity was the broad reach and goodwill that the project built up through parallel activities, which created space and a platform for conflict management messaging over a large area. An obstacle to this messaging was the COVID-19 related regulation restricting mass outreach activities, as it was prohibited to meet more than 10 people in one location, but this was addressed through the approach of training community leaders who could then cascade the trainings. Impact and Attribution What significant changes (positive/negative, planned/unintended) have been observed in the life and environment of people and groups directly or indirectly causally related to the project? The impact of conflict reduction interventions cannot be fully appreciated in a short period but evaluation participants affirmed that there were some changes that were already visible, such as the participation of Twa village chiefs in project activities and cases of Bantu village chiefs who had granted land to members of Twa communities, which allowed them to carry out agricultural activities using project inputs. In addition, it was reported that attacks on Bantu fields by Twa communities had decreased significantly after both communities had benefitted from food distributions and agricultural inputs under the project, as scarcity had been a trigger for conflict. The changes observed in the context of conflict management can be attributed to the TEFA project but also to the DFSA project, as well as to other actors such as Search for Common Ground, OCHA and MONUSCO, which had already established certain local structures such as the Local Peace Committee (CLP). Sustainability Are the net benefits of the intervention likely to last? The project collaborated with grassroots peace structures by empowering them in different strategies and trainings related to conflict resolution. This will increase the capacity of these grassroots structure but there are also wider security aspects that depend directly on the government and the communities will not be able to manage, hence there is a need to also get involved in advocacy actions with the authorities at the provincial and national level. Coherence To what extent is the intervention compatible with other interventions carried out by FH? The TEFA project was executed in an area that used to be part of the DFSA project. So if the security remains stable, shall FH decided to resume DFSA activities in the same area, benefits from conflict management activities should also facilitate the implementation longer-term through lessening the triggers for conflict which would affect DFSA project activities. Recommendations What would be the top recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - FH should consider providing greater ongoing support to grassroots peace structures such as the CLP through the ongoing DFSA project, to enable more sustainability. - FH should where possible work with like-minded organisations to carry out higher-level advocacy actions with authorities at the provincial and national level for the restoration of peace and respect for women's rights, to reinforce project messaging on these. 2.8. Cross Cutting Issues To what extent did the project make considerations with regard to existing gender dynamics and protection? Gender Gender mainstreaming appears to be an important part of how FH works in general and how the TEFA project was designed, implemented and evaluated. FH has a very detailed gender strategy that states clearly how gender issues should be taken into account, and so the project proposal laid out clearly how the conflict and food insecurity had affected women and men differently. This differentiation then flowed through into having certain activities that were meant for the whole household, like food and agriculture inputs distributions, but complementing these with CG activities around health and nutrition specifically targeted at women, and by a focus on SGBV awareness and prevention across the project as a whole. The October 2020 annual report also indicates that female headed households were registered in greater numbers than men, i.e. as 66.8% of beneficiary households, although beneficiaries participating in FGDs stated that more women should have benefited from hygiene kits distributed by FH. Overall the importance of gender mainstreaming for the TEFA project could be summarized by what a FH staff member shared: “The constitution of groups like Lead Farmers (LF), Care Groups, we wanted men and women to be empowered and women to be represented on group committees”. (FH TEFA Staff) In terms of project Monitoring and Evaluation data collection tools were designed to get disaggregated information regarding men and women, which allowed project team to know how the assistance was provided to men and women. In most reports and M&E tools information was disaggregated by gender, except for the Detailed Implementation Plan (DIP). Although the project design was gender sensitive the project staff was gender imbalanced. Both at management and field level most project staff were male, even though FH’s gender policy states that the organization will ensure that “staff and governance structure are gender balanced”. The list of project staff indicates only 5 female staff out of 43 staff. For a project that was promoting women’s rights and focusing on health and nutrition for mothers it would have been helpful to have had a larger proportion of female staff, to enable more connection between female beneficiaries and FH staff who shared similar experiences as women. Protection Conflict sensitivity assessment Conducting conflict sensitivity was one of the steps taken by FH in order so apply the “Do no harm“ principle. According to one staff member understanding the conflict dynamics was useful in deciding to include people from the Bantu and Twa villages among beneficiaries: “We were registering beneficiaries who were near the road, but we received messages from those communities that were still in the bush, who threatened to burn and destroy any assistance if they were not included. This why the local chief from the Bantu communities sent messengers to ask Twa communities to gather in the Bantu villages where they could be registered in order to receive assistance.” (FH Staff Member) Point distribution While the project proposal states clearly that protection issues will be taken into account when deciding the distance between villages and distribution points (maximum 5 km), it seems that there was not a strong system in place to ensure that those standards were followed during the first distribution. According to the post distribution monitoring report some distribution points were at 11 km from certain targeted villages, which may have put beneficiaries at risk. The same report also mentioned that there were some distribution sites where commodities had arrived late, leading some beneficiaries to sell part of their food in order to get home before it was too late. It is encouraging however to see that the post distribution evaluation had identified these issues, and had made recommendations to address them going forwards. According to FH staff measures were put in place in the next distribution to ensure that all distribution points were within 5km of the targeted villages. COVID-19. All FH staff who were interviewed agreed that COVID-19 was a major disturbance in the implementation of the project, but they also acknowledged that FH had put in place a strategy to protect staff and beneficiaries (e.g. including training and sensitization of staff and beneficiaries on COVID-19 protection measures like social distancing and hand washing). Abuse/exploitation The TEFA project seems to have put in place a good system for safeguarding beneficiaries from abuse and exploitation, as multiple FH staff reported that they had been trained on safeguarding in line with the organizational Safeguarding Policy and that they had signed a code of conduct. The evaluation team was not able to ascertain that other stakeholders such as suppliers had also been trained on the same, but a complaint management system had been put in place in order to help beneficiaries to report any abuse or exploitation. Security/Contingency plan FH has a contingency plan which helps the organization to anticipate and respond to security related issues, with a staff member being dedicated to monitoring the security situation in all FH implementation areas. The latest TEFA annual report mentioned a specific situation where FH responded well by evacuating staff from a distribution point when there was a security incident, although this situation also suggests that more could have been done in this case in terms of anticipating security issues in advance. How are monitored complaints addressed? A complaints management committee and a toll-free contact number, as well as a complaints management box, was installed in each village. The complaints management box is managed jointly by the committee and a focal point from FH who has the second key to the box. In addition, a post distribution monitoring process was organized to pick up issues that were not identified through the formal complaint system. Although the complaints system had resolved many questions or complaints raised by the community, it was also seen to have some weaknesses in terms of approach and feedback. According to FH staff complaints were received much more through the suggestion box, and relatively less through the use of the toll-free number. In a context where the level of illiteracy is very high, and of pronounced poverty, the suggestion box and the toll-free number may cause certain groups such as women (who are predominantly illiterate, and may have less access to the telephone), to be unable to submit their complaint confidentially, and so it would be best to put in place additional mechanisms or options in light of such situations. The criteria for selecting complaints (major or minor) that require follow-up is also not clear, as it seems to be up to the committee to decide which complaints to share with the FH office. More focus appears to have been on complaints about food distributions than about other activities such as Care Groups or agricultural inputs (e.g. issues were cited about some Care Groups not receiving their full training materials before the end of the project, or about a complaint about the quality of sweet potato seeds which had not been addressed). Linked to this there appear to have been no post distribution monitoring activities organized for agricultural input related activities, which could have enabled relevant issues to be addressed. Did the project activities bolster or deter equity in participation? Targeting There were clear targeting criteria and a system in place to avoid fraud (e.g. the use of technology via Last Mile Mobile Solution, working with other actors such as local leaders, verification of lists and triangulation of information with other sources). However, many community members were identified and registered in villages where they were not residing either as host community members or IDP, creating potential loopholes. According to a FH staff member the challenges in accessing some areas meant that many community members were called to a distribution point away from where they were living in order to be registered , making it more difficult to cross-check that they fitted the eligibility criteria. The targeting and the selection criteria were shared with community members, and the local community were involved in registration of beneficiaries, but some participants in FGDs felt that there was lack of transparency in the final decision about who should be assisted (e.g. there was a perception that FH staff were registering many people to start with, but that after going back to their office they decided to drop some of the names). The TEFA project included Twa community members in agricultural activities which most of them were not used to, by raising awareness and distributing agricultural inputs, thereby broadening participation in agricultural production. Recommendation: The final selection of beneficiaries should be done together with representatives from the community, to reassure community members that those who have been kept on the list are those who really meet the selection criteria. 2.9. Project Management Planning TEFA being an emergency project, and an agricultural project that depended on the planting season, planning was essential in order to achieve the expected impact of restoring livelihoods, but also saving lives. A challenge in this regard was the delay in the starting of activities due to a combination of recruitment, logistics and COVID-19 related challenges, which pushed planting of some seeds into less favorable conditions in the secondary planting season and meant that there was a rush to procure other seeds. While there were significant external factors at play in terms of COVID-19 it is possible that stronger HR planning for recruitment and logistics planning could have lessened the challenges experienced. Monitoring and Evaluation According to FH staff, and the Detailed Implementation Plan document, most Monitoring and Evaluation (M&E) related activities took place as planned, although not all targets were achieved. Regular post distribution monitoring reportedly took place after each distribution process to assess beneficiary satisfaction, except for after the distribution of sweet potato vines. The evaluation team was however able to see only one post distribution monitoring report, which had focused only on food distribution. In this case feedback, positive or negative, was taken into consideration to improve the distribution process e.g. according to a FH staff member, recommendations helped the organization to better plan distribution points to be within 5km of target villages, and led to additional checks that the food distribution packs were complete. Most FH staff who were interviewed found that M&E tools such as Log Frames, Indicator Performance Tracking Tables and Detailed Implementation Plans were useful in tracking project progress and for planning. While these tools were able to provide general information on how the project performed the evaluation team was not able to access detailed information on some aspects of commodity management (quantity ordered, received, used, spoiled, and balance), or to disaggregate the quantity of food per type of activity (conditional and un-conditional distributions). The M&E department seemed to be well resourced in terms of the number of staff: 1 M&E Manager, 2 M&E Officers, 2 M&E Assistants, and 1 Research Coordinator. Budget Management The management of the TEFA project budget was done by the Project Manager. The full budget allocated was spent by the end of the project, an indicator that project activities had been implemented broadly to plan and other project resources (human and logistical) used. There were however some budget lines that were overspent such as commodity procurement (14% overspent) and transportation and warehousing (26% overspent), which raised questions about the efficiency of the project and/or the accuracy of the initial budgeting. The main line that was underspent was for staff salaries (41% underspent), which is to be expected due to recruitment delays at the start of the project. According to most FH staff who were interviewed the funding needed for the implementation of project activities was supplied in a timely manner by the FH DRC Country Office. This was reported to be mainly due to well-planned cashflow projections and cash requesting processes between programmes, logistics, finance and administration teams. The situation in Moba was however more challenging, especially when COVID-19 related travelling restrictions were put in place, making it difficult for the office to receive cash due to limited financial institutions locally that could provide cash transfer service, which had caused a delay of up to 3 months in the payment of suppliers. Financial reports were also reportedly not always shared on time, causing some delays in decision making, and the evaluation team noticed that Finance Officers did not have information on budget vs actual expenditure for their respective sites, which according to them made it difficult to advise on local expenses. Recommendation: Put in place a system to enable information on the budget vs actual expenditure position to be shared in a timelier manner, including at field level. Recommendation: Create different codes for different projects to facilitate the classification, reporting and analysis of financial transactions for different Cost Centers and sites. Logistics Logistics was an area where pain points were reported, suggesting challenges in planning and resourcing of logistics support to the project given its emergency focus, broad geographical coverage and multisectoral nature. A particular challenge was that there was not a logistics staff member allocated specifically to the TEFA project, with logistics support provided by staff from the parallel DFSA project, leading to staff at times being overloaded and this had an impact on the time it took to buy and transport inputs, leading to programmatic challenges (e.g. in Moba beneficiaries working on road rehabilitation had to start working without tools). Additionally, transport equipment was underbudgeted e.g. purchase of motorbikes was not included in the budget in spite of long distances and wide areas that the project was covering. Recommendation: For future emergency related projects like TEFA it would be best to have one or more logistics staff dedicated solely to the project and engaged in budgeting, in order to expedite the purchase and transport of time-critical inputs and ensure adequate budget. Internal coordination Coordination between FH department was effective and appreciated by staff. Most staff reported that weekly and monthly reports gave them the opportunity to share information and to know what other departments were planning and doing, which in turn provided to each department the opportunity to identify challenges and solve them in a timely manner. External Coordination During the implementation of the project FH took part in various coordination mechanisms in Tanganyika province and appears to have coordinated effectively with other relevant actors. FH is part of the Food Security Cluster, and used information from United Nations Office for the Coordination of Humanitarian Affairs (OCHA). For the implementation of TEFA activities FH also had meetings with World Vision and the World Food Program in order to coordinate food distribution. FH likewise collaborated with various government services, especially at provincial level including such as the National Nutrition Program (PRONANUT ) for the training of health promoters or Direction Voie de Desserte Agricole (DVDA) to plan road rehabilitation activities. Timeframe An 18-month period was planned for the implementation of the project activities but the project was in practice implemented in 12 months. According to staff this timeframe led to working under significant pressure and stress. There is no evidence that this situation impacted the quality of assistance although the shift in the activity plan for activities related to agriculture seemed to have had an impact on the quality of the harvest. In addition, most staff who were interviewed felt that they had not been given enough time to close the project. Human Resources Except for the Programme Manager most project staff were recruited 6 months after the start of the project (in February or March 2020), which impacted negatively on the project’s implementation plan. On a positive note however FH recruited experienced staff to manage and implement the project. Most staff (finance, programme, M&E and field staff) reported that they had worked in similar organization and roles previously, which together with a good orientation process helped them to hit the ground running when they joined FH. As a result, the project was completed on time, in spite of a late start. The evaluation team noticed however that there were staff on the technical side who had general experience but not a specific qualification required to perform their role most effectively (e.g. Health Promoters had not been trained in agriculture or health and nutrition). Overall, the project appears to have been understaffed given the large areas covered in both Kalemie and Moba; and the number of beneficiaries, making it difficult to monitor and support activities effectively in areas that were e.g. 100km to 170km away from FH’s office. For instance, for Care Groups where there were only 3 Health promotors for 134 Care Groups which is a more than 1 to 40 ratio, compared to the recommended ratio of 1 to 10. Geographical targeting As an emergency project, it is a positive thing that the project was implemented in areas that were difficult to reach, and where many other NGOs did not usually go. This allowed FH to assist a large number of people who had been affected by conflict, especially those who were usually left out. However, the geographical coverage of the project was very ambitious. Most staff interviewed felt that it was very difficult to implement the project in such a big area given the available resources, and given the local context of poor road infrastructure and security. Support from National, Region and HQ offices While the FH DRC National Office was perceived to provide enough support for the implementation of the project, it was felt by staff that support from the Region and HQ should ideally have been greater. For instance it was felt that the initial delay that the project had experience due to the late recruitment of staff could have been avoided if FH had surge staff who could have been deployed while waiting for ongoing project staff to be recruited. Inclusion The TEFA project appears to have had a strong focus on being inclusive, due to trainings having been organized on vulnerability and marginalized groups such as people living with disability, on gender mainstreaming and on safeguarding. In addition, the project was designed to assist the most vulnerable people such as IDPs, people living with disabilities, etc. Sustainability and exit strategy The project design shows clearly how food distributions followed by provision of agricultural inputs would enable long-term benefits of the project for participants, but the late start of the project may have limited the effectiveness and thus the sustainability of some activities. The project proposal stated that FH would seek funding for integrated resilience programming that will include interventions such as: Food Security and Livelihoods, Nutrition, Disaster Risk Reduction-DRR and WASH. However according to project staff, there has not in practice been funding available to transition to resilience-based interventions. According to staff and beneficiaries the implementation of all activities was done in collaboration with local committees, which enabled the project to take into account beneficiaries’ perspectives, but concerns were expressed that limited engagement with government structures at local level could undermine the sustainability of the project. III. CONCLUSION Relevance Overall the project seemed very relevant to the needs of beneficiaries, as the conflict situation had led to the loss of livelihood sand assets. Food distributions were well received by all beneficiaries regardless of their ethnic groups but there were some differences in terms of needs for different ethnic groups that were assisted with complementary activities. For instance, the Bantu communities are agriculturists, so the distribution of seed and tools for agriculture was very relevant to their needs, but Twa communities were not all interested in tools and seeds due to traditionally being hunters. Effectiveness The project was assessed as effective on the basis of completion of activities and beneficiaries were able to see the results within a short time, especially for food distributions. Efficiency The project does not appear to have been as efficient as it could have been in relation to time management and use of resources, as most project activities started 6 months after the start of the project, leading to some wasting of inputs such as agricultural inputs that were not planted as they were distributed after the planting season. Gaps in the quality assurance of food and agricultural inputs also led the project to buy some food that was spoiled and some poor￾quality agricultural inputs (e.g. some vegetable seeds for Care Groups that did not germinate, and rotten sweet potato vines as agricultural inputs). Sustainability Although an emergency project TEFA’s design included activities that could last beyond the life of the project. The combination of food distribution, food production, asset restoration, health and nutrition, and training on conflict and gender sensitivity provided to beneficiaries a solid starting point for returning to their normal life. However, the project did not integrate as effectively as it could have done with local government structures (agriculture, health and nutrition), which could have enabled a better exit strategy. For example, for the health and nutrition component TEFA did not collaborate with health centers; and for the agriculture production component government agriculture extensionists were not involved. In addition, there were factors such as climate change and security issues that were beyond the control of the project. Impact The key quantitative indicators that were measured by comparing the baseline and the end line show that there was a significant improvement in the food security situation of beneficiaries who received assistance. For instance the proportion of households with acceptable Food Consumption Scores (>35) increased by 39 percentage points and thereby quadrupled (baseline acceptable FCS: 13% vs endline acceptable FCS: 52%). In addition, households reporting that their children received food from 4 or more types of foods, increased from 13% at baseline to 46% at endline. Qualitative data also provided a lot of information on the impact of the project for each sector, for instance Care Group mothers sharing their positive experiences of learning how to feed their children a more balanced diet and to take them to health center, key informants from communities mentioning about the availability of vegetable in the market that they had not seen for a long time, community members have speaking about being able to access their fields again thanks to road rehabilitation, and both FH staff and community members having shared success stories about Bantu and Twa community members being able to build trust and interact peacefully. What would be top five recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? - Put in place a system that will allow FH to move fast in the implementation of emergency related projects. This could include surge staff, or getting enough staff who are assigned to the emergency project only, or more flexible procurement procedures. - A more effective food and agricultural input quality assurance system should be put in place, for purchasing, storage and distribution, to respond to complaints on food and agricultural input quality and to mitigate any related health risks to communities. - Adopt a payment in cash approach for the rehabilitation of infrastructure, instead of using the FFA approach, especially when the unconditional food distribution approach has already been made in the same environment. - Improve the level of awareness on processes for the management of complaints at community level, given that there is a good level of understanding of the fact that complaints mechanisms exist, but a limited level of understanding of how these mechanisms operate. - Improve the exit strategy: FH should plan to collaborate more extensively with government structures at local levels (health centers or local agronomists) in order to ensure sustainability of results, especially for behavior change related activities. What are some of the topics to celebrate with FH DRC? - Acknowledgment by community members of the key role that FH played in helping them resume their livelihoods, thanks to the level and diversity of assistance provided. - Multiple positive changes (in food availability, access to agricultural inputs, road opening, health and nutrition, conflict management, SGBV related awareness) were shared by community members and attributed mainly to effects of FH support. - Mothers who were trained on vegetable gardens have been able to harvest enough crops that have allowed them both to diversify diets for children and generate income. - TEFA project’s approach and activities have been successful in bringing together communities (Twa and Bantu) that had been experiencing intercommunity conflict. - A collaborative approach between different FH departments (programme, logistics, finance, HR and M&E) was praised by most staff as the reason for completing the project on time in spite of the initial delays. IV. ANNEXES ANNEX.1 RESULT FRAMEWORK Narrative Indicators (Targets) Data Source 2 Risks and Assumptions Goal: Increased food security and self￾resiliency of conflict affected communities in the targeted territories of Tanganyika province, in the Democratic Republic of Congo (DRC). • EFSP3. Reduced Coping Strategies Index (rCSI) (13.61) BL/FE Purpose 1 Households have food and income security • EFSP4. Prevalence of households with moderate or severe Household Hunger Scale (HHS) score (48%) BL/FE Stable rainfall, no pests and crop diseases, Outcome 1.1 Farmers engage in food production activities • Custom 1. Percentage of farmers receiving seeds who planted crops during the agricultural season (100%) RM Enabling conditions sufficiently conducive for uptake of food production Output 1.1.1 Farmers receive seeds and tools for agricultural production • OFDA 1. Number of people directly benefiting from seed systems/agricultural input activities (9,000) • Custom 3: Number of farmers receiving tools for farming activities (9,000) RM RM Output 1.1.2 • Custom 4. Number RM 2 BL/FE – Baseline/Final Evaluation AS – Annual survey RM – Routine monitoring PDM – Post Distribution Monitoring Narrative Indicators (Targets) Data Source 2 Risks and Assumptions Farmers trained in production of basic food crops of farmers who received short-term agricultural sector productivity or food security training (9,000) Purpose 2 Improved consumption of high-quality nutritious foods and health status of households and CU5 • EFSP2. Percentage of households with poor, borderline, and adequate food consumption score (FCS) (Baseline - 50%) • OFDA 4. Proportion of children 6-23 months of age who receive foods from 4 or more food groups (18%) • EFSP9. Prevalence of global acute malnutrition (GAM) (16.3%) BL/FE BL/FE BL/FE Year round availability, accessibility and affordability to nutritious food Outcome 2.1 Care group members produce nutritious food for home use • Custom 7. Number of Care Group Members maintaining home gardens (2,000) RM Change in household behaviors and practices in relation to food and nutrition for WRA, children under-5, and adolescent girls Output 2.1.1 Care Group members receive vegetable and fruit seeds for establishing home gardens • OFDA 3. Number of people receiving behavior change RM Narrative Indicators (Targets) Data Source 2 Risks and Assumptions interventions to improve infant and young child feeding practices (2,000) Output 2.1.2 Care Group members trained • EFSP8. Number of individuals receiving nutrition-related professional training through USG supported programs (2,000) RM Purpose 3 Reduced incidence of SGBV • Custom 19. Percentage of men who say that wife beating is acceptable (20%) BL/FE Changed social behaviors and attitudes on gender issues Outcome 3.1 SGBV Awareness strengthened • Custom 20. Percentage of women aware of where women affected by domestic violence can access the required support (50%) BL/FE Risk: Backlash occurs from SGBV programming (i.e. girls report GBV but are stigmatized by family members, friends, and/ local authorities for reporting Output 3.1.1 Project participants trained on SGBV • Custom 21. Number of participants trained on SGBV (17,273) RM Narrative Indicators (Targets) Data Source 2 Risks and Assumptions Reach and coverage • EFSP 1. Number of individuals participating in USG food security programs (103,638) • EFSP5. Number of children under five (0- 59 months) reached with nutrition-specific interventions through USG-supported programs (20,000) • EFSP6. Number of pregnant women reached with nutrition-specific interventions through USG-supported programs (3,500) • EFSP7. Number of children under two (0-23 months) reached with community-level nutrition interventions through USG supported programs (3,000) • Custom 26. Number of households receiving food assistance (17,273) RM RM RM RM RM Narrative Indicators (Targets) Data Source 2 Risks and Assumptions Quality Management • Custom 22. Percentage of affected population that is aware of the complaint and response mechanism and knows how to use it (90%) • Custom 23. Percentage of complaints received through the complaint and response mechanism responded to in line with the required procedures (90%) RM (PDM) RM Local and regional procurement • Custom 28 Actual quantity delivered (MT) (9,037 MT) • Custom 29 Quantity distributed (MT) (9,037 MT) • Custom 30 Planned and actual participants disaggregated by sex (103,638) RM RM RM ANNEX II: SOURCES OF INFORMATION A. Key Informant Interviews (KII) 1. HQ-Senior Director, Public Resources 2. Country MEL Manager 3. Research Coordinator / TEFA 4. Finance Officer / Kalemie 5. Finance Officer /Moba 6. Manager Assistant TEFA/ Moba 7. Zone Supervisor / Kalemie 8. M&E Officer / Moba 9. Health Center Staff (Malenga, Kabulo, Ndalama et Shebele) 10. Representatives of a Community Based Organization and Community Health Committee Local leaders (Malenge and Ndalama) 11. Beneficiaries (3) / Ndalama, Mitonga and Shebele 12. Manager Assistant TEFA/ Moba 13. Zone Supervisor / Moba 14. Agriculture Promotor / Moba 15. M&E Officer / Moba 16. Zone supervisor / Kalemie 17. Agriculture Promotor Kalemie 18. Field Monitor Kalemie 19. Commodity Officer / Kalemie 20. Commodity Officer / Moba 21. Warehouse Supervisor / Moba 22. CTS/ LMMS Assistant / Moba 23. Field Monitor / Moba 24. Local chief, Benze village / Kalemie 25. Local chair of the CLP (Local Peace Committee) Benze /Kalemie 26. KII with a local chief Mazonde / Moba 27. Local Inspector and secretary / Moba 28. Local Agronomist, Nkasabala /Moba 29. Local Administrator / Moba 30. Local chief, Sylvano and 3 of his team members 31. Local chief, Malenge Village /Kalemie: 32. Representative of the local chief, Kabulo Village 33. Local chief, Kayobwe Village, with his assistant / Moba 34. Local chief Shebele village and his team (3 people) 35. Representative of the local chief Mitong village Moba territory 36. Representative of the local chief, Ndalama Village B. Focus Group Discussions (FGD) 1. FGD with CG mothers (5) 2. FGD with Lead Mother (4) 3. FGD with women on SGBV (4) 4. FGD on agriculture inputs and training (3 Kalemie and 4 in Moba) 5. Focus group discussion on Conflict Management (3 in Kalemie and 4 in Moba) 6. Focus Group Discussion on road rehabilitation activities (01 in Kalemie and 02 in Moba); 7. FGD on conditional and unconditional food distribution (03 in Kalemie and 03 in Moba); 8. FGD on NFI distribution (03 in Kalemie et 03 in Moba) C. Observations 1. Road rehabilitation sites 2. Vegetable gardens 3. Field and gardens belonging to beneficiaries who receive agriculture inputs 4. Suggestion boxes D. Documents reviewed 1. Food for the Hungry, 2020 Tanganyika Emergency Food Assistance International Emergency Food Assistance Project Tentative Statement of Work for Final Evaluation Food for the Hungry/DRC 2. Food for the Hungry, 2020, FH DRC, Annual Results Report, Emergency Food Security Program – EFSP 3. Copy of DRC_TF_TP002_Consolidated Worksheet 4. FH, Indicator Estimates 5. Food for the Hungry, FH DRC TEFA Logframe 6. Food for the Hungry, FH DRC , Organization Chart TEFA 7. Food for the Hungry, FH DRC , Cartographies des activités TEFA 8. Food for the Hungry, FH DRC , Personnes ressources TEFA 02.13.2021 9. Food for the Hungry, FH DRC , Staffs TEFA DRC_02.12.2021 10. Food for the Hungry, FH DRC , Stratégie Genre FH DRC_Final 11. Food for the Hungry, FH DRC , Tronçons routiers 12. Food for the Hungry, FH DRC , villages couverts TEFA, liste des villages, points de distribution Food for the Hungry, FDP, kilomettrage de la base au aux villages pour le site de Moba 13. Food for the Hungry, 2020, FH DRC, Rapid Conflict Scan analysis report 14. Food for the Hungry, 2020 FH DRC, Market Survey report 15. Food for the Hungry, 2020, FH DRC, EFSP Project Baseline Survey Report Narrative 16. Food for the Hungry, 2020, FH DRC, EFSP Monitoring and Evaluation Plan 17. Food for the Hungry, 2020, FH DRC, EFSP - Technical Narrative Revised Final 09-12- 2019 FINAL 18. Food for the Hungry, 2020, FH DRC EFSP plan de travail de réhabilitation des routes de desserte agricole au Tanganyika 19. Food for the Hungry, 2020, Post Distribution Monitoring Report, Emergency Food Security Program - EFSP 20. Food for the Hungry, 2021 Tanganyika Contingency plan 21. Food for the Hungry, 2021, TEFA Financial Report Final March 2021 22. Food for the Hungry, FH DRC IEFA Activity DIP Reviewed_02.15.2021 23. Food for the Hungry, Presentation on overview of project activities 24. Food for the Hungry,2019, Baseline Data Collection Tool 25. John Mugunga, Pronanut-Tanganyika, Les neurones pendant les 1000 premiers jours de vie 26. Ministry of Agriculture, Agriculture Calendar 27. OCHA, feb 2011, Rapport de RÉUNION D’INFORMATIONS GÉNÉRALES, 28. Programme alimentaire mondial des Nations Unies (PAM) , Rapport sur l’évaluation de la sécurité alimentaire en situation d’urgence (EFSA deuxieme round COVID-19) dans les provinces du Tanganyika et du Haut-Katanga (territoires de Pweto et Mitwaba) 29. PRONANUT, formation, La Communication pour le Changement de Comportement ANNEX III: EVALUATION STATEMENT OF WORK Tanganyika Emergency Food Assistance International Emergency Food Assistance Project Tentative Statement of Work for Final Evaluation Food for the Hungry/DRC Agreement Number 72DFFP19GR00073 Date Submitted to Food for Peace: November 24, 2020 November 2020 Bukavu DRC Table of Contents Introduction ................................................................................................................................................................................................3 Project Background...................................................................................................................................................................................3 Evaluation Plan............................................................................................................................................................................................4 A. Scope and Focus.................................................................................................................................................4 B. Specific Evaluation Objectives.........................................................................................................................4 D. Evaluation Framework..........................................................................................................................................4 D. Evaluation Limitations .................................................................................................................................................................7 E. Evaluation Design ........................................................................................................................................................................7 Qualitative study.....................................................................................................................................................7 Quantitative End line Surveys..............................................................................................................................8 Survey Team Training and Field Testing .........................................................................................................12 Comparison of Baseline and End line Data at Final Evaluation..................................................................13 F. Schedule.............................................................................................................................................................................................14 G. Estimated Budget..................................................................................................................Error! Bookmark not defined. Intellectual Property................................................................................................................................................................................16 Ethical Guidelines.....................................................................................................................................................................................16 Annex1: TEFA Baseline/Endline Survey Tool....................................................................................................................................17 INTRODUCTION This scope of work is prepared to perform the final evaluation of the USAID funded International Emergency Food Assistance (IEFA) project dubbed Tanganyika Emergency Food Assistance (TEFA). The final evaluation survey will be conducted in December 2020. This Statement of Work (SoW) will serve as a guidance document for the survey team to properly and uniformly conduct the survey. PROJECT BACKGROUND Food for the Hungry / Democratic Republic of Congo (FH/DRC) has been implementing a Development Food Security Assistance (DFSA) – Tuendelee Pamoja II (TPII) since September 2016. However, owing to the protracted insecurity in Tanganyika Province and in Kalemie territory in particular during the 2018 period, to-date, a number of communities have been displaced leading to loss of livelihoods and assets as well as increase in pressure for resources in the host communities. Although the situation has continued to improve since December 2018 after the general elections, with relative improvement in the security situation allowing for some population to return, the returnees lost everything, the IDPs continue to live in severe hunger and the resulting malnutrition as well as lack of agricultural inputs to restart farming which is their main livelihood activity. FH/DRC approach under the DFSA was to support farmers who had something, including; housing, some capital, tools and some seeds, however, these communities virtually lost everything and have to start from zero to build their lives. In December 2019, FH conducted the baseline survey to get benchmarks for performance guidance and monitoring. Overall Program Objective: The overall goal of the program was to support 19,727 households affected by the protracted insecurity in Tanganyika province rebuild food security and self-resiliency through provision of the support in tools, and seeds for agricultural production as well as facilitating transition to traditional livelihoods activities. The program will also include conflict mitigation and protection (SGBV) because households that have been affected by displacement have high possibility of abuse. Specifically, the closing interventions intended to: • Support at least 17,273 vulnerable households (IDP, Returnee and Host community) with monthly unconditional food assistance for the initial 4 months, and 1,727 most vulnerable households with food to improve household level food insecurity for a period of 6 months in addition to the above unconditional food transfer. • Support 9,000 vulnerable households including returnees to re-engage in agricultural production through provision of agricultural inputs (seeds and tools) and training; • Promote renovation of community assets through food assistance for productive assets or Food for Work/Asset (FFW/A) programming that will target the 18,000 most vulnerable households in the communities; • All households targeted receive Health and Nutrition training on improved household and community case-management of maternal and child health conditions of diseases. • All households are sensitized on conflict as well as Sexual and Gender Violence mitigation in communities EVALUATION PLAN The evaluation methodology ultimately dictates the methods to be used, including data sources, data collection tools, sampling strategies, data management, analysis techniques, reporting techniques, and how these different pieces of methods will be woven together into a coherent way that meets all the requirements of the survey. Each of the above patterns involves important management decisions on issues such as which data sources to rely on and how to gain access to them, what sites should be included in the evaluation (all project sites or a sample communities, and if just a sample, how the sample should be drawn), how much depth of explanation is needed, which stakeholders’ views are most relevant, and so on. A. Scope and Focus While ongoing monitoring was essential for project management and improvement of this project, periodic evaluations is necessary to draw conclusions about the effects of the project on target populations. This evaluation will serve two important functions: determine (a) the extent to which desired changes have occurred, and (b) whether the project is responsible for such changes. This information, in turn, would allow the Program Team who plan and implement and the BHA who fund the program to learn, and to make sound judgments on the future of the project. This evaluation will appraise data and information that inform strategic decisions, thus improving the program in the future. B. Specific Evaluation Objectives The purposes of the final evaluation study are: i. To ascertain final values and changes over time in the performance indicators from the baseline values ii. Evaluate the effectiveness and efficiency of the project interventions on targeted groups of interest with particular emphasis on cross-cutting themes iii. Evaluate the effectiveness and applicability of complementary interventions iv. Identify promising practices, lessons learned, strengths, weaknesses and challenges in project design and implementation C. Evaluation Framework This evaluation following the OECD criteria will help to draw conclusions about five main aspects of the project: • relevance • effectiveness • efficiency • impact • sustainability Information gathered in relation to these aspects during the monitoring process will also provide the basis for the evaluative analysis. As an effective program evaluation, this evaluation will have a systematic way to improve and account for actions by involving procedures that are useful, feasible, ethical, and accurate. The framework will then become a practical, non-prescriptive tool, designed to summarize and organize essential elements of this program evaluation. Building on the specific objectives of the evaluation, the evaluation framework breaks them into questions that the evaluation will answer, define the tool to collect data to answer the question and define the source, i.e. from whom the data will be collected. Adhering to these steps and standards allows the evaluation team to conceive and conduct the evaluation. The evaluation will be participatory in nature. The detailed evaluation framework used for this evaluation is presented in table 1. Table 1: Evaluation Framework Objective 1: To ascertain final values and changes over time in the performance indicators from the baseline values Key Questions/Issues Method Sources of Data Effectiveness: What are the final values for each of the key performance indicators? Descriptive Statistical analysis Household survey (baseline and final evaluation) Effectiveness: What was the progress made against the key performance indicators? Descriptive Statistical analysis Household survey (baseline and final evaluation) Relevance and Effectiveness: To what extent has the project achieved its goals with regards to each respective sector? Document Review Progress Reports Were the correct groups targeted by the project? How was the participant identification process conducted? Document Review Progress Reports Were interventions rendered relevant to the targeted participants considering their initial vulnerabilities? Focus group, KII, PRA project Staff, participants, Local Leaders How well did the implementation processes adhere to underlying principles and activity protocols KIIs, Focus group Staff, Project management, project participants, Did activities go in accordance to plans? Document Review Progress Reports What were the movers and inhibitors to realizing set targets Focus Group Staff Objective 2: Evaluate the effectiveness and efficiency of the project interventions on targeted groups of interest with particular emphasis on cross-cutting themes Key Questions/Issues Method Sources of Data To what extent did the project make considerations with regard to existing gender dynamics and protection? Focus Group Staff To what degree did the project alleviate risks associated with participation in some FFW activities? KIIs, FGD Project participant Did the project activities bolster or deter equity in participation? FGD, KIIs, PRA Project participants How are monitored complaints addressed? Focus Groups Staff What was the level of efficiency of service delivery of associated goods? Focus Group Staff Objective 3: Evaluate the effectiveness and applicability of complementary interventions Key Questions/Issues Method Sources of Data Were complementary services offered relevant to the communal context? Focus Group Staff What was the level of adoption of complementary activities? Focus Group Staff Objective 4: Identify promising practices, lessons learned, strengths, weaknesses and challenges in project design and implementation Key Questions/Issues Method Sources of Data Identify significant lessons or conclusions that can be drawn from the project in terms of effectiveness, efficiency, sustainability and networking. Data Synthesis from the Field Visits Lessons Learned Workshop Field Visit debriefing and Reports What would be top five recommendations that would help FH DRC to enhance the effectiveness, efficiency, sustainability and impact of their programming? Reflection Session Nominal Group Technique Lessons Learned Workshop Evaluation Reflection and Debriefing What are some of the topics to celebrate with FH DRC? Data Synthesis from the Field Visits and guided workshop with the projects staff Evaluation Team and FH DRC Program Staff D. Evaluation Limitations Validity and reliability of self-reported data. Some of the data collected for the household survey will be self-reported, which has limitations, such as: the possibility of exaggeration or omission of information; inaccurate recollection of experiences or events; social desirability bias or reporting of untruthful information; and reduced validity when respondents do not fully understand a question. E. Evaluation Design In FH’s Evaluation we recommend using a combination of qualitative and quantitative data that will improve quality of an evaluation by ensuring that the limitations of one type of data are balanced by the strengths of another. This will ensure that program understanding is improved by integrating different ways of knowing. Using mixed methods provides rich insights and triangulates the information. FH will employ a parallel data collection methodology where qualitative and quantitative data will be collected at the same time, but would require lot of coordination in terms of logistics and evaluation skills. An evaluation team comprising of FH global and National Office will conduct the final evaluation, as they are not directly involved in the implementation of the project. The final evaluation will be split into two parts; the quantitative survey will be conducted ideally 3 months before the close of the project so that the results may be comparable. The qualitative survey will occur concurrently with the quantitative survey. F. Qualitative study Qualitative methods will be implored to directly answer the evaluation questions. Qualitative study results will also support the interpretation of findings from the quantitative survey. FH will utilize interviews such as Focus Group Discussions (FGDs), Key Informant Interviews (KII) also will use Interactive tools such as Venn diagrams, Ranking, transect walk and seasonal calendars to gather qualitative data on the perceptions, judgements, and opinions of target populations. The qualitative study will use a purposive sample of individuals and groups who are directly involved in project implementation. The sample size will depend on the data saturation point1 where no new information is collected. Sample selection of the Focus Groups will be purposive in nature and will keep in mind the objectives of the evaluation, considering the best source of information. Factors such as geographic location, age, gender and participation characteristics will be used. There are four main types of focus group questions that will be used during an interview. For KII knowledgeable people from the community, community leaders, and local government staff will be targeted. 1 Collection of data until no new information emerges G. Analysis The goal of the analytic process in qualitative research is to tease out themes, patterns, and connections among ideas embedded in the data. Preparing the data: The goal at this point is to convert the messy, raw data into words presented in an intelligible format that can be read, edited, checked for accuracy, coded, and analyzed. This may take the form of verbatim transcription (interviews and focus groups), detailed narrative (field notes), or reduced and simplified summaries of these two. Conceptualizing data: For each question asked recurring ideas, language and patterns of belief; will be listed down as a way to organize ideas and concepts. Once ideas and concepts have been identified they will be organized into categories. Each response category will have one or more associated themes that give deeper meaning to the data thus different categories may be collapsed under one over-arching theme. Principally objectives will guide the analysis process As themes and patterns emerge from the data, it is important to go through the data, carefully searching for negative instances of the patterns (outliers). Outliers will be carefully examined and possible explanations for these outliers will be thought out. Data Synthesis: Once the themes are in place reflection will be made on the evaluation’s goals and classify the data in accordance to each objective. Quantitative data is used for generalization of qualitative findings. Qualitative data can put flesh on the bones of quantitative results, bringing results to life through in-depth case elaboration. The prevalent use of quantitative data is to focus inquiry on a discrete set of variables to test specific hypothesis or research question. In contrast, the prevalent use of qualitative data is to open the study through presenting the large, interconnected complexities of a situation. Credibility assurance: To ensure validity and credibility of data the evaluation team will implore two methods: Triangulation: cross checking data from multiple data sources and using the mixed method approach to explore and understand inconsistencies Participant feedback: checking with participants concerning the accuracy of the data and interpretations H. Quantitative End line Surveys Sampling strategy A robust sampling frame will be established by registration of all targeted households in the intervention areas through a census. Collected data will then be uploaded and stored in the project database (WL3). Data to be collected during the census will include: • Geographic location of households • All household members data (sex, age, contact details) • Household target criteria met (Households to be grouped by target criteria) A one stage simple random sampling technique will be used where each household has an equal chance of selection. Sample size calculation The sample for the endline study will be a random sample of targeted project participants drawn from the census list. As two distinct sampling frames exist, two surveys will be carried out i.e. one for indicators EFSP2, EFSP3 and EFSP4 and one for EFSP9. For the first group of indicators we will calculate the sample size for EFSP2 and EFSP4 as these will be adequate to estimate EFSP3. The highest sample size will be considered as the final sample size for the survey. The formula for calculating sample size survey is: A 10% non-response adjustment is implored to the initial sample size so as to get the final sample size. Table 1: Sample size calculation for required end line indicators EFSP2 EFSP4 EFSP9 P1,est 50% (0.5) 93% (0.93) 23% (0.23) P2,est 40% (0.4) 88% (0.88) 18% (0.18) Z1-α 95% (1.64) 95% (1.64) 95% (1.64) Z1-β 80% (0.84) 80% (0.84) 80% (0.84) Dest 1 1 1 ninitial 305 425 806 Non-response adjustments 10% 10% 10% nfinal 339 Hhs 472 Hhs 896 children Table 2: Sample size calculation for other end line indicators Custom 20 Custom 19 OFDA 4 P1,est 50% (0.5) 50% (0.5) 10% (0.1) P2,est 70% (0.7) 65% (0.65) 20% (0.2) Z1-α 95% (1.64) 95% (1.64) 95% (1.64) Z1-β 80% (0.84) 80% (0.84) 80% (0.84) Dest 1 1 1 ninitial 220 399 469 Non-response adjustments 10% 10% 10% nfinal 244 individuals 433 Individuals 521 children The final survey will cover indicators EFSP2, EFSP3, EFSP4, Custom 20, Custom 19. A sample size of 5252 households will be drawn from the registered households. The basis of household selection is the presence of children 6-23 months. For indicator EFSP9 a sample of 896 children under the age of 5 will be considered. 2 A minimum sample size of 525 is implored as the minimum after taking into account the three adjustments to the initial sample size according to: Participant-Based Survey Sampling Guide for Feed the Future Annual Monitoring Indicators By Diana Maria Stukel, PhD September 2018 Sampling Frame Seven indicators will be considered for the end line study as shown below. Table 3: Required Indicators considered for quantitative end line study Indicator Number Indicator name Sampling Frame EFSP2 Percentage of households with poor, borderline, and adequate Food Consumption Score (FCS) All participant Households in target areas EFSP3 Reduced Coping Strategies Index (rCSI) All participant Households in target areas EFSP4 Prevalence of households with moderate or severe Household Hunger Scale (HHS) score All participant Households in target areas EFSP9 Prevalence of global acute malnutrition (GAM) Children under 5 years of age Table 4: Other indicators considered for end line study Indicator Number Indicator name Sampling Frame Custom 20 Percentage of women aware of where women affected by domestic violence can access the required support Women in targeted areas Custom 19 Percentage of men who say that wife beating is acceptable All men in target areas OFDA 4 Proportion of children 6-23 months of age who receive foods from 4 or more food groups Mothers and caregivers of children 6-23 months old (Hhs with Children 6-23 months) Data collection In this end line evaluation survey, the casual enumerators and PRONANUT staff will fully participate in the process while the leading role will be on the M&E team at country and regional level. The M&E manager in DRC will be responsible to prepare the data collection tool in consultation with the regional M&E manager as well as calculate the sample size for the survey. Each cluster M&E officer will be responsible to prepare the sample framework and communicate with the country M&E manager. The data collection will be done using smart phones and the Open Data Kit (ODK) platform. I. Roles and responsibilities of data collectors Enumerators: The enumerators will collect data from household respondents as dictated by the survey design. Prior training will be given to each enumerator on how to conduct the interviews as well as probing techniques. Each enumerator will be required to go through the entire questionnaire and have the same understanding with regards to how to interpret the questionnaire to the targeted respondents. Supervisors Some TPII staff will have a supervisory role. Supervisors will also receive the same training with the enumerators. Moreover, supervisors will also receive training on how to effectively manage their assigned teams as they will be responsible for closely following up the data collection process. Supervisors will also ensure that all necessary materials are available before the team leaves for the field. Finally supervisors will resolve any field related conflicts or challenges to ensure smooth flow of the data collection process. Anthropometric team These will be responsible for the second survey for recording GAM. The team will be comprised of casual enumerators under a handful of government health department workers3 . Survey Team Training and Field Testing Training for Promoters and Supervisors will be conducted across the two project sites (Moba and Kalemie). Each training will be conducted over a 4 day period. The initial training will focus more on the ethics and norms of data collection. The next phase of the training will involve review of the hard copy version of the data collection tools as to identify any inconsistencies in the flow of the survey. The next phase will involve training on the use of ODK in collecting data. The final phase of the training will involve a field practice where field testing will be conducted to ensure that the tool’s questions are clear and easy to answer. The field testing will also identify any gaps that may exist in the ODK whether it be technical or content wise. After the field testing recommendations will be then incorporated into the final questionnaire. The Anthropometric team will have a separate 2-days training on how to conduct the required measurements. Data Analysis and management The primary tool for data collection will be ODK collect which is a data collection software that will be used with compatible smart phones. The supervisor is required to validate data before synchronization. Validated data from the field is synchronized with an online central server. Once 3 For such surveys it is imperative to involve the health ministry to ensure that results are valid data is synchronized only the M&E team will have access to consolidated data from all project sites. Data will then be imported to EPI Info for analysis. Data Quality Assurance The M&E team will follow an interactive and participatory process so that program staff are aware of the information gathered by the process monitoring. At the field level supervisors are the pinnacle of monitoring data collection and control. Supervisors will review data collected by each promoter within no more than two days from the initial data collection. Using the smart phone, the supervisor will be required to re-interview at least one household per each given day with special attention to critical questions. In the instance of non-response households, the supervisor will be required to validate that the households truly opted out of the survey. If the inconsistencies are major the enumerator will be required to receive further training from the M&E officer before continuing with the surveys. Validated data is then uploaded through a secure transmission to the cloud server Once validated data has been uploaded, the M&E officers will ensure correctness and completeness of the data uploaded from the field. Any changes made to the raw data will be made online and will be documented so as to justify the need for the changes implored. For field related changes the interview will be pushed back to the responsible enumerator with an issues letter will be sent with the supervisor copied for validation. The M&E team will also remotely monitor data consistency throughout the ongoing data collection process using data periodically downloaded from the cloud server. Comparison of Baseline and End line Data at Final Evaluation As baseline and end line data will be collected at about the same time during the year; they will be comparable. FH will use EPI-INFO for data analysis of the quantitative data. As the survey will be a cross-sectional study a test of difference (z-test) will be conducted. A Kolmogorov￾Smirnov test will be conducted to ascertain and validate that the two arbitrary distributions (baseline, final evaluation) are the same. A table of comparison will be filled out as depicted in Table 2. Table 5: Comparison of indicators in intervention areas from Baseline and End line survey Indicator Values p value Baseline Endline (Kolmogorov) Percentage of households with poor, borderline, and adequate Food Consumption Score (FCS) FCS of 0-21: 32.76 (Poor) FCS of 21.5-35: 55.17 (Borderline ) FCS of > 35: 12.07 (Acceptable) Reduced Coping Strategies Index (rCSI) 22 Prevalence of households with moderate or severe Household Hunger Scale (HHS) score 96.73 Prevalence of global acute malnutrition (GAM) 20.4% Percentage of women aware of where women affected by domestic violence can access the required support 5.2 Percentage of men who say that wife beating is acceptable 55.4 Indicator Values p value Baseline Endline (Kolmogorov) Proportion of children 6-23 months of age who receive foods from 4 or more food groups 13.0 Schedule The following detail timeline is tentative and subject to modification depending on the accessibility of intervention sites. The time frame provided is illustrative and subject to adjustment until the end line survey work plan is finalized and approved by USAID. Table 6: Timelines for key end line Survey activities Key Activities Responsible Timeframe I. Pre-Planning Submission of SOW to USAID M&E Manager/Focal Point person 25 November, 2020 Develop a list of all implementation villages/communities and classify them by types of program activities, number of beneficiaries, distance from the center contents covered M&E Manager 26 November, 2020 Prepare list of beneficiaries … etc by cluster/ community (Done after registration) M&E Officers 27 November, 2020 II. Planning Develop endline survey work plan and logistical requirements M&E Manager 27 November, 2020 Inform selected communities and individuals of upcoming survey Emergency Program manager,Zonal supervisors 30 November, 2020 III. Implementation Training of Enumerators, Anthropometry specialists and Supervisors by M&E team and Health team on tools and questionnaire Revision of tools / questionnaires if required, and updating enumerators on changes M&E Team and Health and Nutrition Coordinators 2-5 December, 2020 Conduct data collection and validation (December 7- 20; and 9-27) respectively Kalemie and Moba Promoters and Supervisors 8 to 27 December, 2020 Sectoral data analysis and synthesis M&ETeam 10 -20 January 2021 Present the preliminary observations and results to GSC, FH office and the GSC, then to partners to validate findings and implications. (Validation workshop) M&E manager 27 – 29 January, 2021 IV. Reporting and dissemination Key Activities Responsible Timeframe Send first draft of endline Report to USAID M&E Manger 3 February, 2021 Intellectual Property In general terms, it is the FH’s, right to intellectual property produced under the evaluation surveys. However, FH may retain the rights, title, and interest to data that are first acquired or produced under the evaluation surveys. Regarding USAID, the following statement holds. “USAID reserves a royalty-free, worldwide, nonexclusive, and irrevocable right to use, disclose, reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, in any manner and for any purpose, and to have or permit others to do so.” Ethical Guidelines The following ethical guidelines are for every member of the evaluation survey team to strictly adhere to as outlined in the American Evaluation Association’s Guiding Principles for Evaluators. A summary of these guidelines is provided below. • Systematic inquiry: Evaluators conduct systematic, data-based inquiries. • Competence: The evaluation team possesses the education, abilities, skills, and experience appropriate to undertake the tasks proposed in the evaluation. Evaluators practice within the limits of their professional training and competence, and decline to conduct evaluations that fall substantially outside those limits. The evaluation team collectively demonstrates cultural competence. • Integrity/honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process. • Respect for people: Evaluators respect the security, dignity, and self-worth of respondents, activity participants, clients, and other evaluation stakeholders. Evaluators regard informed consent for participation in evaluation and inform participants and clients about the scope and limits of confidentiality. • Responsibilities for general and public welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation. ANNEX IV: DATA COLLECTION INSTRUMENTS A. Quantitative data collection tools Health Zone Village GPS Coordinates Date of Interview Kalemie Nyemba Moba Kansimba Name of Enumerator :…………………………………………………….. Name of Supervisor :…………………………………………………… Respondent’s Full Name :…………………………………………. Household Composition (Complete the table by Sex and Age Group) Please write ‘0’ if there are none 0-5 months 6-23 months 2-5 years 6-14 years 15- 29 years 30-49 years Above 50 years Total Male Female Section 1: Food Consumption score 1a. Over the past seven days which foods do you recollect eating (Instruction to enumerator: indicate category where each food group falls under and probe for each remaining food group) 1b. How often did you eat each of the mentioned foods over the past seven days? Select applicable food groups and frequencies: Food Group Food group weight Specific foods in category Frequency Food Consumption Score (Frequency * weight) Main Staples 2 Maize, Rice, Sorghum, other cereals, tubers, plantains Pulses 3 Beans, peas, groundnuts and cashew nuts Vegetables 1 Vegetables and leaves Fruit 1 Fruits Meat/fish 4 Beef, goat, poultry, pork, eggs and fish Milk 4 Milk, Yogurt and other dairy Sugar 0.5 Sugar and sugar products, honey Oil 0.5 Oils, fats and butter Condiments 0 Spices, tea, coffee, salt, fish power, small amounts of milk for tea TOTAL CONSUMPTION SCORE Section 2: Reduced Coping Strategy Index 2a. In the past 7 days, have there been times when you did not have enough food or money to buy food? Yes/No (Skip rule: If “no” in 2a skip to next section, if response is “yes” proceed to 2b) 2b. How did you cope with the lack of food? (Instruction: Probe for coping strategies) 2c. During the past seven days how often did you employ the mentioned coping strategy? Coping Strategy For how many days in the past 7 days did you adopt this coping strategy Severity Weight Weighted Score (Calculate: frequency*seve rity weight) Rely on less preferred and less expensive foods; Instruction: This refers to both staple foods and other food items that accompany staple foods (e.g. sauces, vegetables). In some contexts, the term "food" refers only to staple foods, so enumerators should ensure that respondents understand this to mean anything that the household would prefer to eat as part of their usual meals. 1 Borrow food or rely on help from a friend or relative, Instruction: This does NOT include any borrowing behavior that is typical for the household (e.g. regularly exchanging food and/or resources with a family member or neighbor). 2 Limit portion size at mealtimes, Instruction: This refers to eating smaller amounts of food during meals (i.e. a smaller portion size of a staple food or a smaller serving of sauce or meat) because there was not enough to go around. This does not refer to intentional restriction of food (e.g. for religious fasting or dieting). 1 Restrict consumption by adults in order for small children to eat, Instruction: This refers to adults intentionally eating less when children are present so that the children have enough to eat. 3 Reduce number of meals eaten in a day. Instruction: This refers to a reduction in the number of meals typically consumed by the household; e.g. if a household typically eats three meals per day and in the last week they ate only two meals per day some of the days, 1 they would answer, "yes" to this question. This does not include intentional restriction of food (e.g. for religious fasting or dieting). TOTAL HOUSEHOLD SCORE (Calculate: Sum of weighted scores) Section 3: Hunger Score (This section’s questions are to be asked to the person in the household in charge of food preparation) 3a. During the past for weeks have you experienced any of the following incidents, if so how often? Event Frequency Score No food at all in the house Never 0 Rarely or sometimes 1 Often 2 Went to bed hungry Never 0 Rarely or sometimes 1 Often 2 Went all day and night without eating Never 0 Rarely or sometimes 1 Often 2 Total Score Section 4: Global acute malnutrition If child between 0-5 years exists in the household measure their weight and height Name of Child Sex Weight (kg) Height (cm) Weight to Height ration (weight/height) B. Qualitative data collection tools EVALUATION FINALE/ TANGANYIKA EMERGENCY FOOD AID PROJECT GROUPE DE DISCUSSION DISTRIBUTION DES VIVRES (GD, SPP et FFW) Type de participants (ce que les participants ont en commun) :……..…………………………………. Territoire : ……………………………………………. Zone de santé………………………………….. Village/Quartier:……….…………………………. Details démographiques (nombre): Homme …………. Femme………………..Mixte………………….. ___________________________________________________________________________ Introduction : Se présenter en donnant le nom et la provenance. • Présentation : Nous sommes des évaluateurs, nous ne sommes pas membre du personnel de FH, mais nous aidons l’organisation dans l’évaluation finale du projet d’assistance aux personnes et communautés récemment affectées par le conflit dans cette communauté, qui a été mis en œuvre par FH. • Attente vis-à-vis des participants : Vos connaissances et votre expérience en rapport avec votre participation dans les activités de FH nous seront très utiles. • Objectifs : Cette évaluation est une opportunité d’apprentissage sur la manière d’apporter une assistance adaptée aux besoins des communautés. Nous avons besoin de votre perspective sur ce qui a bien marché et sur ce qui nécessite une amélioration dans le futur, au cas où FH décidait d’apporter la même assistance à d’autres personnes dans une situation similaire à la vôtre. • Anonymat : nous dirons à FH ce qui a été dit pour améliorer le travail, mais nous ne dirons pas qui a dit quoi. Nous ne prendrons pas vos noms. En cas de besoin, on mentionnera seulement votre fonction, par Example, une femme agricultrice nous a dit ceci… ; ou un leader a demandé ceci…mais on fera tout pour que on ne soit pas en mesure de connaitre l’identité exacte de la personne qui nous a donné l’information. • Procédure : Nous allons poser des questions, et vous répondrez. Pour ne pas oublier les choses importantes en écrivant le rapport pour FH, nous allons noter ce que vous dites (En termes de conclusion). • Durée : la discussion prendre au plus 1h 15 minutes. • Droit des participants : Vous êtes libre de participer ou de refuser de participer. Il n’y aura aucune conséquence négative pour vous. • Gestion des attentes : Votre participation à cette réunion ne veut pas dire que vous avez été sélectionnés pour recevoir une assistance de FH, mais bien pour participer et contribuer à cette évaluation dont les résultats permettront à FH d’améliorer le projet d’urgence dans l’avenir et dans d’autres zones d’intervention. • Consentement éclairé : Êtes-vous d’accord de participer ? I. Information au sujet du projet et critère d’inclusion 1. Est-ce que ça fait longtemps que vous vivez dans ce village ? 2. Comment avez-vous entendu parler du projet de FH ? 3. Comment est-ce que vous vous êtes retrouvés impliqués dans les activités de distribution des vivres organisées par FH ? a. Pourquoi vous et pas les autres ? b. Pensez-vous qu’il y a d’autres qui méritaient et qui n’ont pas été sélectionnés par FH ? c. Si OUI, citez quelques deux (02) exemples/ cas. 4. A quel type de distribution des vivres dont vous avez été impliqués et ou bénéficiaires ? a. Distribution Non Conditionnelle (Distribution Générale, Paquet de Protection des Semences) ? Pourquoi ? b. Distribution Conditionnelle ou FFW ? Pourquoi ? II. Effectivité 5. Est-ce qu’il y a dans cette communauté des activités pour prévenir ou répondre au problème d’Insécurité Alimentaire à travers les distributions des vivres ? a. Quoi par Exemple ? b. Qui les fait et comment il les fait? c. Comment cela a-t-il été initié ? 6. Avez-vous été formés et/ou sensibilisés par FH sur les modalités de Distribution Non Conditionnelle des vivres et/ou le FFW et comment en bénéficier? a. Par qui exactement ? b. Quand avez-vous été formés ou sensibilisés? c. Quels a été le thème de formation et/ou sensibilisation ? Dans quelle langue ? i. Que pensez-vous de ces thèmes ? Pourquoi ? d. Qui étaient les participants ? e. Comment les participants à la formation et/ou à la sensibilisation étaient choisis ? f. Avez-vous été satisfaits de ce choix ? pourquoi ? g. Quelles étaient les plaintes de la communauté par rapport au choix des participants ? 7. Qu’est-ce que vous avez aimé le plus dans la formation et/ou sensibilisation de FH sur les activités de FFW ? Pourquoi ? 8. Qu’est-ce que vous aimeriez changer par rapport à la formation/ sensibilisation de FH sur les activités de distribution des vivres (GD, SPP) et/ou le FFW ? a. Pourquoi ? b. A part FH, Y a-t -il quelqu’un d’autre ou une autre ONG qui vous a formé/ sensibilisé sur le la distribution des vivres (Non Conditionnelle et FFW) ? Qui ou quelles organisations ? c. Quel était le thème ? 9. Quelles activités de distribution des vivres ont été réalisées dans le cadre du projet TEFA ? 10. Quels tronçons routiers avaient été réhabilités par FH moyennant le FFW ? a. Citez quelques troquons que vous connaissez b. Sur quels tronçons routiers avez-vous travaillé pour le FFW ? Pourquoi ? 11. Pensez-vous que les taches étaient convenablement déterminées/ quantifiées et/ou proportionnelles à la quantité de nourriture reçue en contrepartie ? Si OUI ou NON, Pourquoi ? 12. Quelle était la tache (Quantité de travail) attribuée par Homme/ Jour (H/J) selon les types de travaux (Défrichement, Remblayage ou rechargement des nids d’oiseaux, canalisation, Aide￾maçon, etc)? 13. Quelle valeur monétaire avait la quantité des vivres reçue par jour en compensation du travail effectue/ accompli ? III. Pertinence 14. A quoi est-ce que les activités de distribution des vivres (GD, SPP) et/ou de réhabilitation d’infrastructures agricoles moyennant le FFW vous ont servi ? Expliquez un peu plus ? 15. Pensez-vous que ces activités relatives à la distribution des vivres ont constitué la réponse réelle aux besoins de votre communauté en matière de sécurité alimentaire ? Si OUI ou NON Comment expliquez-vous cela ? 16. Pensez-vous que les activités de distribution non conditionnelle des vivres (GD, SPP) ont été prioritaires par rapport à la problématique de l’insécurité alimentaire observée avant ou au début du projet ? Si OUI ou NON Pourquoi ? 17. Pensez-vous que les activités de réhabilitation des routes moyennant le FFW ont été prioritaires par rapport à la problématique de l’insécurité alimentaire observée avant ou au début du projet ? Si OUI ou NON Pourquoi ? 18. Comment avez-vous procédé pour être ciblés parmi les personnes vulnérables en général et les plus vulnérables en particulier pour les activités de FFW ? 19. Quelles interventions du projet que votre communauté a bénéficiées ? 20. Sur base de quoi votre communauté été sélectionnée pour en bénéficier ? 21. Les interventions du projet ont-elles répondu aux besoins de votre communauté ? Si OUI ou NON, comment ? 22. Pensez-vous qu’il y ait de ménages hors critères qui ont été servis ? Comment jugez-vous ces interventions du point de vue Pertinence et l’efficacité ? 23. Que reprocherez-vous en matière de ciblage des bénéficiaires pour la distribution conditionnelle des vivres (FFW) ? IV. Impact ! 1. S’il y a un risque d’Insécurité Alimentaire dans cette communauté, a. Qu’est-ce qui doit être fait pour le prévenir ? b. Comment le sauriez-vous ? 2. S’il y a déjà une Insécurité Alimentaire dans votre communauté qu’est-ce qui doit être fait pour le gérer selon vous ? a. Comment le sauriez-vous ? 3. Quels changements significatifs positifs avez-vous observés en matière de Sécurité Alimentaire au sein de votre communauté pendant et/ou après le projet ? Donnez au moins un exemple de changement. 4. Avez-vous observé aussi des changements négatifs au sein de la communauté par suite des activités de distribution des vivres ? Si OUI lesquels et comment expliquer cela ? Selon vous, quelle serait la situation d’aujourd’hui si du moins les activités de distribution des vivres et de réhabilitation des routes n’étaient pas organisées par FH ? Pourquoi ? 5. Savez-vous d’où vient l’assistance que vous recevez de FH ? V. Pérennité/ Durabilité 6. Pensez-vous que les changements observés par suite des activités de distribution des vivres en général et de FFW en particulier s’observeront pendant très longtemps encore ? Si OUI ou NON, pourquoi ? 7. Pour combien de temps estimez-vous que la communauté continuera à bénéficier des effets positifs dus aux activités de distribution des vivres en général et de FFW en particulier du projet TEFA ? VI. Attribution 1. A quoi attribuerez-vous les changements observés sur l’etat des routes et des ponts de desserte agricole ? Comment est-ce possible ? 2. A quoi attribuerez-vous les changements observés dans vos ménages en matière de sécurité alimentaire ? Comment est-ce possible ? 3. Savez-vous d’où vient l’assistance que vous recevez de FH ? VII. Mécanisme de plainte 4. Quand il y avait un problème en termes de plainte en rapport avec les activités de distribution des vivres en général et de FFW en particulier organisées par FH, a. Auprès de qui est-ce que ces problèmes étaient posées ? b. Avez-vous reçu des réponses de la part de FH en termes de feed-back et/ou des solutions ? c. Quel était le mécanisme d’expression du problème ou de la plainte par la communauté ? d. 5. Donnez deux exemples de problème que vous aviez soulevés en rapport avec les activités de distribution des vivres en général et le FFW en particulier (ciblages de villages, des ménages bénéficiaires, distribution, etc). a. Quelles étaient les réponses ou solutions reçues de FH ? Après combien de temps aviez￾vous reçu de réponse ? b. Etiez-vous satisfait de la solution que vous aviez reçue ? Pourquoi ? 6. Quelles seraient vos critiques sur le mécanisme de gestion des plaintes qui a été mis en place pour le projet ? En étiez-vous d’accord au départ ? VIII. Coordination avec autres activités FH et/ou d’autres intervenants 7. En plus des activités de distribution des vivres en général et de FFW en particulier, FH avait-il fait ou organisé quelque chose d’autre pour vous ou avec vous au sein de la communauté ? ✓Si Oui ? Quoi exactement? ✓Si Non : pourquoi ? 8. Avez-vous reçu une assistance en vivres d’une autre ONG ? ✓Quoi exactement ? ✓De quelle ONG ? IX. Recommandation 9. Qu’est-ce que le projet peut faire encore pour que tous les membres de la communauté accèdent facilement aux vivres de qualité et en quantités suffisantes de façon permanente ? sachent comment prévenir et gérer les conflits ? 10. Supposons que vous étiez responsables des activités de distribution des vivres et du FFW en particulier, que feriez-vous différemment de ce que FH a fait ? 11. Quels sont les conseils que vous donneriez à FH pour améliorer son programme d’urgence en termes de distribution des vivres ? Nous vous remercions pour votre temps et pour toutes les réponses EVALUATION FINALE/ TANGANYIKA EMERGENCY FOOD AID PROJECT GROUPE DE DISCUSSION : AGRICULTURE Type de participants (ce que les participants ont en commun) :…..…………………………………. Territoire : ……………………………………………. Zone de santé…………………………………. Village/Quartier:……….…………………………. Details démographiques (nombre): Homme …………. Femme………………..Mixte………………….. ___________________________________________________________________________ Introduction : Se présenter en donnant le nom et la provenance. • Présentation : Nous sommes des évaluateurs, nous ne sommes pas membre du personnel de FH, mais nous aidons l’organisation dans l’évaluation finale du projet d’assistance aux personnes et communautés récemment affectées par le conflit dans cette communauté, qui a été mis en œuvre par FH. • Attente vis-à-vis des participants : Vos connaissances et votre expérience en rapport avec votre participation dans les activités de FH nous seront très utiles. • Objectifs : Cette évaluation est une opportunité d’apprentissage sur la manière d’apporter une assistance adaptée aux besoins des communautés. Nous avons besoin de votre perspective sur ce qui a bien marché et sur ce qui nécessite une amélioration dans le futur, au cas où FH décidait d’apporter la même assistance à d’autres personnes dans une situation similaire à la vôtre. • Anonymat : nous dirons à FH ce qui a été dit pour améliorer le travail, mais nous ne dirons pas qui a dit quoi. Nous ne prendrons pas vos noms. En cas de besoin, on mentionnera seulement votre fonction, par Example, une femme agricultrice nous a dit ceci… ; ou un leader a demandé ceci…mais on fera tout pour que on ne soit pas en mesure de connaitre l’identité exacte de la personne qui nous a donné l’information. • Procédure : Nous allons poser des questions, et vous répondrez. Pour ne pas oublier les choses importantes en écrivant le rapport pour FH, nous allons noter ce que vous dites. • Durée : la discussion prendre au plus 1h 15 minutes. • Droit des participants : Vous êtes libre de participer ou de refuser de participer. Il n’y aura aucune conséquence négative pour vous. • Gestion des attentes : Votre participation à cette réunion ne veut pas dire que vous avez été sélectionnés pour recevoir une assistance de FH. • Consentement éclairé : Êtes-vous d’accord de participer ? I. Information sur au sujet du projet et critère d’inclusion 1. Est-ce que ça fait longtemps que vous vivez dans ce village ? 2. Comment avez-vous entendu parler du projet d’urgence de FH ? 3. Comment est-ce que vous avez été impliqué dans les activités de FH ? a. Pourquoi vous et pas les autres ? II. Effectivité et Pertinence 1. Est-ce qu’il y a dans cette communauté des activités de production agricole ? a. Quoi par Example ? b. Qui le fait ? 2. Avez-vous reçu de FH une assistance dans le cadre de production agricole ? Quand ? a. Qu’est-ce que vous avez reçu ? b. Qu’en est-il des outils c. Qu’en est-il des semences ? d. Qu’en est-il des vivres ? e. Qu’en est-il des formations agricoles ? 3. Qu’est-ce que vous avez aimé le plus dans l’assistance reçu de FH ? Pourquoi ? 4. Qu’est-ce que vous avez aimé le moins dans l’assistance reçu de FH ? Pourquoi ? 5. Est-ce que tout le monde dans cette communauté pense que les critères de sélection pour l’assistance étaient juste ? Pourquoi ? Pourquoi ? 6. Est-ce que tout le monde qui était dans le besoin a été servi ? Sinon, qui n’a pas été servi ? Pourquoi pas ? 7. Connaissez-vous d’où vient l’assistance que vous recevez de FH ? III. Impact et attribution 8. Vous avez dit avoir reçu ………………………………. Dans le cadre de la production agricole (Citez l’assistance reçus plus haut). Est-ce que cette assistance vous a servi à quelque chose ? Quoi ? 9. Selon vous, quel changement positif peut être observé parmi ceux qui ont reçu l’assistance relatif à la production agricole de FH dans le territoire de Moba/Kalemie ? 10. A part FH, Y a-t -il quelqu’un d’autre ou une autre ONG qui vous a formé sur la production agricole ? Qui ? Quand ? Et sur quoi avez-vous été formés ? 11. Selon vous quelle est la part de FH et celle des autres acteurs dans les résultats observés ? IV. Pérennité 12. Vous avez dit avoir reçu ………………………………. (Citez certains éléments mentionnés plus haut) dans le cadre de la production agricole. Est-ce que ce que vous avez reçu va vous servir pendant longtemps, même après le départ de FH ? a. Si oui pourquoi ? b. Si non, pourquoi pas ? V. Mécanisme de plainte 13. Quand il y avait un problème en rapport avec les activités relatives à la production agricole (distribution semences, outils, vivres et formation), a. Auprès de qui est-ce que ces problèmes étaient posés ? b. Est-ce qu’il y a un mécanisme de plainte ? Comment fonctionne-t-il ? est-ce que tous le monde est au courant de ce mécanisme ? Qui sont le plus au courant ? Qui sont le moins au courant ? Pourquoi ? c. Donnez un exemple de problème en rapport avec l’assistance sur la production agricole que vous aviez soulevé et pour laquelle une solution a été trouvée. De l’identification du problème jusqu’à la solution. Etiez-vous satisfaites de la solution que vous aviez reçue ? Pourquoi ? Pourquoi pas ? d. Donnez un exemple de problème en rapport avec l’assistance sur la production agricole que vous aviez soulevé et pour laquelle une solution n’a été pas été trouvée ? i. Pourquoi la solution n’a pas été trouvée ? VI. Coordination avec autres activités FH 14. A part les activités relatives à la production agricole, est ce que FH a fait quelque chose d’autre pour vous ou avec vous ? Si Oui ? Quoi ? Si Non : pourquoi ? 15. Avez-vous reçu une assistance d’une autre ONG ? Quoi ? La quelles ? VII. Recommandation 16. Qu’est-ce que le projet d’urgence aurai pu faire pour que la production agricole et la sécurité alimentaire soient meilleurs pour toute la communauté ? 17. Supposons que vous étiez responsables des activités de production agricole et de la sécurité alimentaire, que feriez-vous différemment de ce que FH a fait ? Nous vous remercions pour votre temps et pour toutes les réponses EVALUATION FINALE/ TANGANYIKA EMERGENCY FOOD AID PROJECT GROUPE DE DISCUSSION : MERES _CARE GROUPE Type de participants (ce que les participants ont en commun) :…..…………………………………. Territoire : ……………………………………………. Zone de santé…………………………………. Village/Quartier:……….…………………………. Details démographiques (nombre): Homme …………. Femme………………..Mixte………………….. ___________________________________________________________________________ Introduction : Se présenter en donnant le nom et la provenance. • Présentation : Nous sommes des évaluateurs, nous ne sommes pas membre du personnel de FH, mais nous aidons l’organisation dans l’évaluation finale du projet d’assistance aux personnes et communautés récemment affectées par le conflit dans cette communauté, qui a été mis en œuvre par FH. • Attente vis-à-vis des participants : Vos connaissances et votre expérience en rapport avec votre participation dans les activités de FH nous seront très utiles. • Objectifs : Cette évaluation est une opportunité d’apprentissage sur la manière d’apporter une assistance adaptée aux besoins des communautés. Nous avons besoin de votre perspective sur ce qui a bien marché et sur ce qui nécessite une amélioration dans le futur, au cas où FH décidait d’apporter la même assistance à d’autres personnes dans une situation similaire à la vôtre. • Anonymat : nous dirons à FH ce qui a été dit pour améliorer le travail, mais nous ne dirons pas qui a dit quoi. Nous ne prendrons pas vos noms. En cas de besoin, on mentionnera seulement votre fonction, par Example, une femme agricultrice nous a dit ceci… ; ou un leader a demandé ceci…mais on fera tout pour que on ne soit pas en mesure de connaitre l’identité exacte de la personne qui nous a donné l’information. • Procédure : Nous allons poser des questions, et vous répondrez. Pour ne pas oublier les choses importantes en écrivant le rapport pour FH, nous allons noter ce que vous dites. • Durée : la discussion prendre au plus 1h 15 minutes. • Droit des participants : Vous êtes libre de participer ou de refuser de participer. Il n’y aura aucune conséquence négative pour vous. • Gestion des attentes : Votre participation à cette réunion ne veut pas dire que vous avez été sélectionnés pour recevoir une assistance de FH. • Consentement éclairé : Êtes-vous d’accord de participer ? I. Information sur au sujet du projet et critère d’inclusion 1. Est-ce que ça fait longtemps que vous vivez dans ce village ? 2. Comment avez-vous entendu parler du projet de FH ? 3. Comment est-ce que vous été trouvé impliqué dans les activités de FH ? a. Pourquoi vous et pas les autres ? II. Effectivité 4. Est-ce qu’il y a dans cette communauté des activités sur la santé et la nutrition des enfants et des mères (allaitante et enceinte) ? a. Lesquels ? par Example ? b. Qui fait ces activités ? 5. Avez-vous été impliquée dans les activités sur la santé et la nutrition des enfants et des mères de FH? Comment ? i. Parlez-moi des Mère Leaders ? ii. Il y en a combien dans votre air de santé ? 6. Travaillez-vous avec les autres femmes ? Combien ? Comment ? 7. Avez-vous été formée par les mère leaders ? a. Sur quels thèmes ? b. Comment les participantes à la formation étaient choisies ? III.Pertinence 8. Qu’est-ce que vous avez aimé le plus dans la formation des Mère Leaders sur la santé et nutrition (mère et enfant) ? Pourquoi ? 9. Qu’est-ce que vous aimeriez changer par rapport à la formation des Mère Leaders sur la santé et nutrition (mère et enfant) ? Pourquoi ? 10. Y a-t-il d’autres formations qui vous seraient utiles ? Lesquels ? Pourquoi ? 11. Que pensez-vous du travail des Mères Leaders ? 12. Qu’est-ce qui peut être amélioré par rapport au travail et rôles des Mères Leaders ? Pourquoi ? IV.Impact et attribution 13. Vous avez dit avoir été formées sur………………………………. (Citez certains thèmes mentionnés plus haut). Est￾ce que ces connaissances vous ont servi à quelque chose ? Quoi ? 14. Selon vous qu’est-ce qui peut être identifié en ce moment comme changement positif provenant ces activités d’assistance aux personnes affectées par le conflit de FH dans le territoire de Moba/Kalemie dans le volet santé et nutrition (mère et enfant) ? 15. Selon vous quelle est la part de FH et celle des autres acteurs dans les résultats observés ? 16. A part FH, Y a-t -il quelqu’un d’autre ou une autre ONG qui vous a formé sur la santé et la nutrition (mère et enfant) ? Qui ? Quand ? 17. Savez-vous d’où vient l’assistance que vous recevez de FH ? V. Pérennité 18. Vous avez dit avoir formé les femmes sur………………………………. (Citez certains thèmes mentionnés plus haut). Est-ce que ces connaissances vont leur servir pendant longtemps, même après le départ de FH ? c. Si oui pourquoi ? d. Si non, pourquoi pas ? VI.Mécanisme de plainte 19. Quand il y avait un problème en rapport avec les activités de formations ou de sensibilisation organisée par FH sur la santé et la nutrition (mère et enfant), a. Auprès de qui est-ce que ces problèmes étaient posées ? b. Donnez un exemple de problème en rapport avec la formation ou sensibilisation sur la santé et la nutrition (mère et enfant) que vous aviez soulevé et pour laquelle une solution a été trouvé i. Etiez-vous satisfaites de la solution que vous aviez reçue ? Pourquoi ? Pourquoi pas ? c. Donnez un exemple de problème en rapport avec la formation ou sensibilisation sur la santé et la nutrition (mère et enfant) que vous aviez soulevé et pour laquelle une solution n’a pas été trouvé i. Pourquoi la solution n’a pas été trouvé ? VII. Coordination avec autres activités FH 20. A part la formation ou sensibilisation sur la santé et la nutrition (mère et enfant) , est ce que FH a fait quelque chose d’autre pour vous ou avec vous ? Si Oui ? Quoi ? Si Non : pourquoi ? 21. Avez-vous reçu une assistance d’une autre ONG ? Quoi ? La quelles ? VIII. Recommandation 22. Qu’est-ce que le projet peut faire pour que la santé et la nutrition (mère et enfant) soit meilleure pour toutes les femmes de la communauté ? 23. Supposons que vous étiez responsables des activités sur la santé et la nutrition (mère et enfant) , que feriez-vous différemment de ce que FH a fait ? EVALUATION FINALE/ TANGANYIKA EMERGENCY FOOD AID PROJECT INFORMATEURS CLES RESPONSABLE DU PROJET D’URGENCE Poste/Fonction :…..…………………………………. Territoire : ……………………………………………. Zone de santé………………………………….. Village/Quartier:……….…………………………. Details démographiques (nombre): Homme …………. Femme……………….. ___________________________________________________________________________ Introduction : Se présenter en donnant le nom et la provenance. • Présentation : Nous sommes des évaluateurs, nous ne sommes pas membre du personnel de FH, mais nous aidons l’organisation dans l’évaluation finale du projet d’assistance aux personnes et communautés récemment affectées par le conflit dans cette communauté, qui a été mis en œuvre par FH. • Attente vis-à-vis des participants : Vos connaissances et votre expérience en rapport avec votre participation dans les activités de FH nous seront très utiles. • Objectifs : Cette évaluation est une opportunité d’apprentissage sur la manière d’apporter une assistance adaptée aux besoins des communautés. Nous avons besoin de votre perspective sur ce qui a bien marché et sur ce qui nécessite une amélioration dans le futur, au cas où FH décidait d’apporter la même assistance à d’autres personnes dans une situation similaire • Anonymat : dans notre rapport nous mentionnerons ce qui a marché et ce qui doit être amélioré par rapport au projet. Mais nous ne citerons pas les noms. En cas de besoin, on mentionnera seulement votre fonction, par Example, selon un staff de terrain, ; ou un leader a demandé ceci…essayera dans la mesure du possible qu’on ne soit pas en mesure de connaitre l’identité exacte de la personne qui nous a donné l’information. • Procédure : Nous allons poser des questions, et vous répondrez. Pour ne pas oublier les choses importantes en écrivant le rapport pour FH, nous allons noter ce que vous dites. • Durée : la discussion prendre au plus 1h. • Droit des participants : Vous êtes libre de participer ou de refuser de participer. Il n’y aura aucune conséquence négative pour vous. • Gestion des attentes : vous n’aurez aucun gain matériel comme résultat de votre participation à cet entretien, cependant vous pouvez être satisfaits du fait que vous aurez contribué à l’amélioration du programme assistance des personnes vulnérables. • Consentement éclairé : Êtes-vous d’accord de participer ? I. Introduction 1. Est-ce que ça fait longtemps que vous travaillez avec FH sur le projet d’urgence ? 2. Avez-vous été impliqué depuis l’évaluation des besoins jusqu’à présent ? 3. Parlez-moi de votre rôle en rapport avec le projet d’urgence ? II. Effectivité, pertinence et efficience 4. Parlez-moi un peu du projet d’urgence TEFA, ses activités et des résultats attendus ? 5. Quelle était la stratégie de mise en œuvre ? 6. A votre avis, est-ce que toutes les activités prévues dans le projet ont été réalisées dans le délai conformément aux plans d’exécution ? i. Pourquoi ? Pourquoi pas ? ii. Quelle est votre estimation de ce qui a été réalisés pour chaque thématique iii. QU’est-ce qui est prévu pour ce qui reste. 7. Est-ce que les activités ont mis en œuvre à temps conformément aux attentes des communautés/bénéficiaires ? 8. Selon vous quelles sont les succès de la mise en œuvre du projet d’urgence TEFA Moba, et de Kalemie ? 9. Quelles sont les difficultés dans chaque territoire ? Pourquoi ? 10. Est-ce que selon vous, ces activités correspondez aux besoins des communautés - Comment le savez-vous ? Donnez un exemple ? III. Défis 11. Selon vous quels ont été les grands défis relatifs aux activités du projet d’Urgence TEFA dans les territoires de Moba et Kalemie ? 12. Comment les avez-vous contournés ? IV. Impact et attribution 13. Selon vous qu’est-ce qui peut être identifié en ce moment comme changement provenant des activités relatives au projet d’Urgence TEFA dans les territoires de Moba et Kalemie ? 14. Selon vous quelle est la part de FH et celle des autres acteurs dans les résultats observés ? 15. Selon vous est-ce qu’il y a des conséquences négatives provenant des activités du projet d’urgence ? a.Donnez un exemple 16. Est-ce qu’il y a des résultats ou changements qui vous ont surpris ? C’est-à-dire inattendu ? V. Pérennité 17. Selon vous est-ce que les résultats vont durer ? Pourquoi ? Pourquoi pas ? VI. Mécanisme de plainte 1. Parlez-moi de votre système de redevabilité. 2. Pouvez-vous donner deux exemples de problème résolu qui a été soulevé par les communautés dans le cadre du projet d’urgence. a. Le problème était résolu rapidement ? Pourquoi/Pourquoi pas b. Qu’est-ce qui a contribué à la résolution du problème ? 3. Pouvez-vous donner deux exemples de problème non - résolu qui a été soulevé par les communautés dans le cadre du projet d’urgence. c. Qu’est-ce qui a contribué à la non résolution du problème ? VII. Aspects transversaux 18. Selon vous pensez-vous que tout au long de ce projet d’urgence, les aspects suivants ont toujours été pris en compte: 19. Avez-vous des stratégies pour chacune de thèmes transversaux ? a. Est-ce tous vos staffs sont équipes pour intégrer les aspects transversaux ? comment ? VIII.Services d’appui HR, Logistique , Finance, M&E , A. Ressources humaines et administration 20. Selon vous, avez-vous été préparé et équipé par FH pour mieux assumer votre rôle ? 21. Selon vous, est-ce que tout le staff impliqué dans le projet a été bien préparé et équipé par FH pour mieux assumer leur rôle ? 22. Est-ce qu’il y a quelque chose qui aurait pu être amélioré pour vous aider à mieux assumer cette responsabilité et faire votre travail sur ce projet ? 23. Qu’est-ce qui a été mis sur place dans le cadre de ce projet pour protéger des staffs, compte tenu du contexte sécuritaire ? b. Qu’est-ce qui a fait qu’on n’ait pas anticipé l’incident qui a causé l’évacuation du staff lors de la distribution? A. FINANCE 24. Quelle est la situation actuelle du budget du projet d’urgence TEFA ? Est-ce que toutes les lignes budgétaires ont été dépensés ? d. Quelles lignes ont été dépensés conformément au budget et dans les délais ? e. Quelles lignes ont eu des dépassements, plus de 110 % ? Pourquoi ? f. Quelles lignes ont été sous utilisés, moins de 70 % ? 25. Quelle est votre perspective en rapport avec la situation du déboursement des fonds par les services des finances pour répondre aux besoins du projets ? 26. Est-ce que vous aviez toujours l’information dont vous aviez besoin par rapport a la situation des dépenses du projet ? g. Était-ce à temps ? Pourquoi ? B. LOGISTIQUE 27. Quelle est votre perspective par rapport à l’approvisionnement en intrants par les services de logistiques pour répondre aux besoins du projets ? h. Aviez-vous un plan d’approvisionnement pour le projet ? i. Conforme à la demande ? j. Était-ce à temps ? k. Est-ce que le stockage des intrants étaient fait selon les normes de stockages de ces intrants (vivres, NFI, semences, outils) l. La livraison était-elle fait conformément au calendrier soumis, et conformément aux intrants attendus ? i. Pourquoi ? Pourquoi pas ? 28. Quels sont les points positifs que le projet a connu en rapport avec la logistique ? 29. Quels sont les autres défis que le projet a connu en rapport avec la logistique ? A. VIVRES 30. Ou est-ce que les vivres ont été achetées 31. Comment est-ce que la gestion des vivres étaient organisées ? 32. Quels sont les défis que vous avez rencontré par rapport aux vivres : approvisionnement, stockage, transport, livraison, et distribution 33. Quel a été l’impact des distributions des vivres sur les marches locales à Moba et à Kalemie ? quelle a été la stratégie de mitigations ? B. M&E 34. Parlez-moi de la stratégie de suivi et évaluation du projet d’urgence TEFA ? a. Est-ce que toutes les activités prévues dans le plan de suivi et évaluation ont été mis en œuvre ? i. Était-ce dans le délai ? Pourquoi / Pourquoi pas 35. Quelles sont les défis que vous avez rencontré en rapport avec la mise en œuvre de la stratégie de M&E ? IX. Coordination avec FH, autres intervenants et communautés 36. Quelle est le niveau et la qualité de la coordination du projet d’urgence avec les autres programmes de FH ? m. Y a-t-il une cohérence dans tout le programme pays de FH ? 37. Quelle est le niveau et la qualité de la collaboration de FH dans le cadre du projet d’urgence avec les autres acteurs humanitaires travaillant dans le même secteur ? n. Expliquez ou donnez un exemple (la réponse donnée) 38. Quelle est le niveau et la qualité collaboration du projet d’urgence avec bénéficiaires ? o. Expliquez ou donnez un exemple (la réponse donnée) X. Recommandations Nous sommes presque arrivés à la fin de notre entretien. Mais nous avons besoins de vos recommandations sur certains points spécifiques si vous permettez. 39. Que recommanderiez-vous à FH RDC ou Siège pour améliorer l’assistance aux personnes affectées par le conflit dans le contexte des territoires de Moba et de Kalemie ? 40. Quelles sont vos recommandations en rapport avec les services d’appui ? 41. Que recommandez-vous en rapport avec les aspects transversaux : genre, protection, redevabilité, inclusion, coordination ? Nous vous remercions pour votre temps et pour toutes les réponses ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Charlie Kabanga Title Consultant Organization Independent Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID Previous work experience with the organization that has been evaluated. X X X operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date April 19th, 2021 ANNEX V: FINAL EVALUATION DATASET ANNEX VI: BASELINE EVALUATION DATASET ANNEX VII: TEFA DETAILED IMPLEMENTATION PLAN (DIP) U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523