Consortium for Rehabilitation and Development (CORAD) Development Relief Program (DRP) Impact Assessment Report Impact Assessment Coordinators: Ms. Meghan Audette Mr. Fred Bavoray Goba Mr. Morlai Moses Conteh Ms. Veronica Smith Mr. Gibril Bangura Mr. Joseph Benedictus Juana Mr. Patrick Borbor March 2007 DRP Impact Assessment Report 2 January-March, 2007 Table of Contents Table of Contents..................................................................................................2 List of Acronyms and Abbreviations......................................................................3 Executive Summary..............................................................................................4 Introduction and Background of DRP....................................................................7 Evaluation Methodology........................................................................................8 Preparation and Team Composition..................................................................8 The Most Significant Change Approach............................................................8 Main Assessment Activities...............................................................................9 Sampling ...........................................................................................................9 Data Analysis ....................................................................................................9 Limitations...........................................................................................................10 Main Findings and Recommendations................................................................11 Intermediate and Long-term Health Impacts ...................................................11 Intermediate and Long-term Nutrition Impacts ................................................12 Intermediate and Long-term Agricultural and Economic Impacts ....................13 Food for Work..................................................................................................14 Gender Impacts...............................................................................................16 Peacebuilding and CORAD.............................................................................15 Perceptions of Government Officials ...............................................................16 Conclusions and Implications for LEAD..............................................................18 Annexes..............................................................................................................19 Annex 1: Impact Assessment Guide ..............................................................19 DRP Impact Assessment Report 3 January-March, 2007 List of Acronyms and Abbreviations CHC: Community Health Committees COP: Chief of Party CORAD: Consortium for Rehabilitation and Development CRS: Catholic Relief Services DRP: Development Relief Program FFP: Food for Peace FFS: Farmer Field Schools FFW: Food for Work GoSL: Government of Sierra Leone IR: Intermediate Result LEAD: Livelihood Enhancement and Asset Development M&E: Monitoring and Evaluation IPTT: Indicators of Performance Tracking Table PHC: Primary Health Care PO: Program Objective TBAs: Traditional Birth Attendants USAID: United States Agency for International Development USG: United States Government VGF: Vulnerable Group Feeding VHWs: Village Health Workers WV: World Vision DRP Impact Assessment Report 4 January-March, 2007 Executive Summary From January-March 2007, the partners of the Consortium for Reconstruction and Development (CORAD) conducted an Impact Assessment of the Developmental Relief Program (DRP). This grant, agreement number FFP-A-00-04-00020-00, was awarded to CORAD by USAID/FFP in March 2004, and ran for three years until March 2007. The total amount of funding obligated for this grant is $24,154,809. Of this funding $6,079,381 is 202(e) cash, $5,987,626 is monetization proceeds and $2,731,608 ITSH. The goal of this program was to support Sierra Leone’s recovery from war on restoring livelihoods for rural households in 30 focal chiefdoms in Bonthe, Kailahun, Koinadugu, Kono and Tonkolili districts. To achieve this goal, the DRP program aimed to: improve the health status of 15,800 rural food insecure households (Program Objective One) and; improve supply and access to food for more than 37,000 beneficiaries over its LOA (Program Objective Two). To achieve these objectives, the DRP implemented a series of community￾focused interventions aimed at achieving 4 critical results: (1) strengthening the capacity of existing government Primary Health Care (PHC) systems to improve beneficiary access to PHC services; (2) increasing household knowledge related to improved health and nutrition practices; (3) building the capacity of village-based health/nutrition groups to address health issues and capitalize on opportunities to improve the health status of their community members and; (4) linking strengthened community health/nutrition groups to decision-makers existing at chiefdom and district levels to leverage resources to implement health-related activities in their communities. To increase food supply and access for beneficiaries, the DRP was designed to reestablish livelihoods and do so in ways that will facilitate the transition of target beneficiaries from relief to development. To achieve this result, the program implemented household-focused interventions that focused on six critical areas: (1) maintaining a safety net to address the immediate food and housing needs of target beneficiaries; (2) restoring productivity of inland valley lowland rice farms, tree farms and upland farms; (3) rebuilding storage and processing capacities; (4) reestablishing access to local markets through rehabilitation of farm-to-market roads and market facilities; (5) enhancing the capacity of village-based agricultural groups to address agricultural problems capitalizing on existing opportunities and; (6) linking these strengthened agricultural groups to government institutions existing at chiefdom and district levels to increase their participation decision-making and policy formulation processes. The Impact Assessment was designed as a follow on to the more comprehensive DRP Final Evaluation, which was conducted from January-May 2006. Although this exercise provided valuable data to CORAD and informed the planning for subsequent programming, it was felt that it would nevertheless be valuable to obtain additional information on the opinions and perceptions community members and government officials in the Districts and Chiefdoms served. The Impact Assessment utilized the Most Significant Change approach to ascertain the perceptions and opinions about significant changes that had taken plans, and conducted 128 interviews and focus groups. The Impact Assessment was done internally by CORAD partners staff, and was overseen by the M&E Unit. Data was analyzed by developing frequency charts for common responses, and then discussing the implications of the most frequent responses and notable outliers. Impact Assessment participants were able to provide specific information about how particular DRP activities had impacted their health and economic situation. Participants noted that health trainings had not only contributed to a decrease in disease outbreaks and deaths through raising knowledge and awareness, but also had increased the confidence of trainees (most notably TBAs and VHWs). Health trainings had also resulted in improved DRP Impact Assessment Report 5 January-March, 2007 behaviours in practice by women in targeted communities. The DRP’s objectives related to health related to establishing linkages with the formal health sector, and so it is not only important to ascertain the most effective methodology for teaching health topics, but also to look at how the groups tie into the Village Development Committee and the formal health sector. From the perspective of the communities, the CHCs have been particularly visible in health activities, including training. All health groups (i.e. CARE’s CHCs, CRS’s Village Development Health Committees, WVSL’s Village Health Committees) were viewed by health authorities as positively impacting the health situation in the village, which indicates positive relationships with the formal health sector (a point that would perhaps not have been readily evident to community members not directly engaged with formal health activities). Some organizations were forming the CHCs with greater success than others; it is recommended to share best practices in this area across the four CORAD partners. Growth promotions weighings and FFW together resulted in the perception from the communities that the rates of malnutrition were decreasing. The HEARTH model employed by some agencies was particularly effective in that women in focus groups could clearly articulate how HEARTH assisted them in knowing when their children were malnourished. Vulnerable Group Feeding was not mentioned often by community members as a high impact activity—although this might seem at face value to be a weakness, it might also suggest the success of the very specific targeting of the VGF. This also may be a semantic issue, as the forms and procedures for food for work and VGF were extremely similar, and many communities used the term FFW in place of VGF. Participation in the FFS had also increased farmers’ knowledge on agricultural methods, and had improved several practices. Community participants and governmental officials alike affirmed that CORAD had contributed to an increase in agricultural production and crop yields, which would logically result in an increase in income. Some communities in which CORAD had been working in agriculture were able to clearly articulate the way in which additional income that they had received as a result of CORAD interventions was being used. Others affirmed that they had received additional income, but did not provide specific answers to how they were utilizing this money. Introduction to improved methods of agro￾processing was another contributing factor to enhanced income. The FFS also provided an opportunity for farmers to teach other farmers, which encouraged cooperation and working together on common issues. A number of FFW activities were conducted, which makes it somewhat difficult doing this type of Impact Assessment to ascertain the impact of such interventions. Nevertheless, it is clear that FFW activities contributed to a widespread perception among community members and government officials that people are more organized to think about and act upon their development needs, and that there is a greater degree of peace and unity in communities. It may be worthwhile under the LEAD to further define the priorities that CORAD has for FFW, and to document further the contributions made to reducing conflict in post-war Sierra Leone. Women involved in the DRP were not only empowered with enhanced knowledge through participation in health and agricultural activities, but also used their greater economic potential to play a greater role in financial decision-making in their homes. There are; however, still questions remaining whether the greater role in decision making translates into an ability to prioritize children’s nutritional needs in times of food scarcity. On the whole, government officials at the District, Chiefdom, and local levels were very satisfied with the work of the DRP, although some District level officials felt that they did not have sufficient information about the project to comment. DRP Impact Assessment Report 6 January-March, 2007 On the whole, the Impact Assessment confirmed the positive findings from the Final Evaluation from 2006, and provided the CORAD agencies with an opportunity to review their experiences to date as lessons learned to be applied to the upcoming LEAD program. DRP Impact Assessment Report 7 January-March, 2007 Introduction and Background of DRP Since 2002, Sierra Leone has made an impressive transition from war to peace. The unprecedented destruction resulting from the war led the government of Sierra Leone, civil society, and international donors to deploy ever-greater resources to accelerate the recovery process in all socio-economic sectors. From March 2004-March 2007, a Developmental Relief Program (DRP) was implemented in 5 districts (Koinadugu, Kailahun, Kono, Bonthe and Tonkolili ) of Sierra Leone. The program is managed by the Consortium for Relief and Development (CORAD), a partnership between Africare, CARE, Catholic Relief Service and World Vision International. The Consortium is a recipient of USAID funding under the Food for Peace Office, P. L. 480, Title II Program. Using monetization proceeds, other US Government Funds, and Government of Sierra Leone contributions plus CORAD member contributions, the four consortium members undertook activities to restore agricultural livelihoods, reestablish health services, and build capacities of groups and individuals at the village level. Key target groups for the program were the extremely vulnerable and food insecure, such as women-headed households, pregnant mothers and children. Over the life of the program, approximately 37,400 food insecure households and 131,200 FFW/VGF recipients benefited. The program had two objectives: (1) improving community health status and; (2) restoring community livelihoods by reestablishing agriculture. In line with the above objectives, the following Intermediate Results (IRs) were planned: 1. Improving access to health services 2. Increasing knowledge and skills related to improved health and nutrition practices 3. Building village-level capacities to address health issues and capitalize on opportunities to improve health status at the village level 4. Linking village-based health groups to wider decision-making structures 5. Restoring productivity of inland valley lowland rice farms, tree farms and upland farms 6. Rebuilding storage and processing capacities 7. Reestablishing access to local markets 8. Building village-level capacities to address agricultural problems and capitalize on agricultural opportunities 9. Linking village-based agricultural groups to wider decision-making structures 10. Maintaining a safety net From January-May 2006, the CORAD partners conducted a Final Evaluation. The reason why this survey was done well in advance of the end of the DRP was to measure seasonal indicators in the same season in which the Baseline Survey was conducted; however, CORAD agreed to do a follow up exercise closer to the program’s completion. For this reason, in December 2006, CORAD undertook a qualitative Impact Assessment that sought to gather additional data regarding the opinions and perceptions of project beneficiaries and governmental officials at the local, chiefdom, and district levels. This was designed to be a solely qualitative exercise, and mainly served to further validate the qualitative and quantitative results of the Final Evaluation. This was not designed to be a large or elaborate assessment, bur rather an exercise to focus CORAD leadership on ensuring that lessons learned from the DRP would be used to inform the planning for LEAD. DRP Impact Assessment Report 8 January-March, 2007 Evaluation Methodology Preparation and Team Composition The Impact Assessment Team was made up of staff of the four CORAD DRP partner agencies, whose efforts were coordinated by the CORAD M&E Unit. The M&E Unit drafted an Impact Assessment Toolkit that provided information on sampling, question guides, and recording. A member of each agency team volunteered to act as the main contact person for that agency, to ensure that activities were implemented in a quality way and according to the schedule set by the M&E Unit. Key agency representatives were: M&E Unit Ms. Meghan Audette Mr. Fred Bavoray Goba Mr. Morlai Moses Conteh Africare Ms. Veronica Smith CARE Mr. Gibril Bangura CRS Mr. Joseph Benedictus Juana WV Mr. Patrick Borbor Agencies were asked to nominate staff members to act as facilitators, who were then trained by a facilitation team led by the M&E Unit and made up of representatives of each agency. This one day training was followed by a field test done by all four agencies, after which the question guide and translations of key terms were agreed. The Most Significant Change Approach To gather impact data, the Impact Assessment Team used a variation of the dialogic, "Most Significant Change" approach (Dart et. al). Although a full analysis including working with community groups to establish the domains of change was not conducted, participants in interviews and focus groups were asked to respond to a series of questions: Question One: What do you feel is the most important thing that the DRP [or ORGANIZATION NAME] Program has done here? Question Two: What about this has been particularly important for you and your family? Question Three: And what has this [change] meant for you? Question Four: How is the situation now different than it was before? Question Five: Why do you think that this change [from before the DRP to now] has been so significant for you? Although these questions might seem repetitive, asking the same question in slightly different ways did lead focus groups to reflect more deeply upon their development reality, and offer up longer-term impacts. For example, answers given to question one dealt mainly with project activities themselves (ex: drying floors, the provision of health training), while answers to subsequent questions dealt more with outcomes and impacts (ex: now we know when our children are malnourished, or increased income to spend on other things). It is important to note that this methodology measures the perceptions and opinions of community members, not actual and demonstrable change. Many things can influence perceptions: the way in which an activity was conducted and even how long ago the activity finished could all contribute to beneficiaries vocalizing satisfaction less. For this DRP Impact Assessment Report 9 January-March, 2007 reason, it is important to thoroughly analyse the raw data, and to cross check responses against what activities occurred most recently. Main Assessment Activities The activities of the assessment were: ¾ 27 focus groups with community men ¾ 27 focus groups with community women ¾ 27 interviews with local chiefs ¾ 20 interviews with District officials ¾ 27 interviews with Chiefdom officials The responsibility for implementing these activities was spread across the four partner agencies. Sampling The M&E Unit provided agencies with a purposive sample of chiefdoms that took into consideration the areas of work of the four partners: Kailahun: • Dea (Africare) • Njaluahun (Africare) • Peje West (CRS) Koinadugu • Diang (CRS) • Folosaba Dembelia (CARE) Tonkolili • Gbonkolenken (CARE) Bonthe • Jong (World Vision) Kono • Mafindor (World Vision) • Toli (World Vision) Agencies were asked to do their own sampling of communities within the selected chiefdoms, three communities per chiefdom. Agencies were also asked to select the 8- 12 individuals to participate in the focus group, based on the following criteria: Men • 1-2 Village Development Committee members • Youth leader • Farmers who have participated in the agriculture portion of the project • Husbands whose wives have participated in the health portion of the project • Heads of household whose household has participated in the VGF • At least 2 men below the age of 35 Women • Women’s Group leader • 1-2 Village Development Committee members • Women who have participated in the health portion of the project • Heads of household whose household has participated in the VGF • Women whose husbands have participated in the agriculture portion of the project • At least 2-3 women below the age of 35 Interviews were conducted based on positions held. A final list of all participants was submitted to the M&E Unit. Data Analysis The M&E Unit had the main responsibility for analysing the data from the Impact Assessment; however, partner staff requested the opportunity to participate in the analysis sessions as a capacity building opportunity. Data was analysed by reading CORAD-DRP Impact Assessment 10 For use by CORAD partners only over all responses given at the agency level and developing frequency charts (documenting the frequency of responses made by men, women, and key informants). The frequency charts from each agency were then compared to determine the overall findings. Where appropriate, this report highlights outliers or areas where one or more agencies significantly varied from the others. Limitations Although CORAD stands by the quality of the data gathered, there are some limitations that are worthy of note: • None of the Impact Assessment activities sought to engage non-beneficiaries, although opinions regarding the DRP’s impact at the community-level were sought through interviews with Village Chiefs. • There were no activities to assess the opinions and perceptions of national government officials • The Impact Assessment was not a participatory exercise—during this exercise the main role for community members was as data sources. • The Impact Assessment was a purely internal exercise. No external stakeholders were involved in the data gathering or analysis. These decisions were made as there was limited budget to conduct the Impact Assessment. It was somewhat challenging for agencies to find women within their organization that had the necessary language skills to conduct the assessment (i.e. recorders, facilitators, observers, translators). In some instances, it was necessary to have men acting as recorders. Partners did not face any significant obstacles in implementing the Impact Assessment, although in certain cases it was not possible to interview the actual official—in certain cases it was instead an assistant familiar with the program. CORAD-DRP Impact Assessment 11 For use by CORAD partners only Main Findings and Recommendations1 Intermediate and Long-term Health Impacts Health activities were linked to a number of intermediate and long-term impacts by community members. Women stated that trainings had served to increase their knowledge and understanding of health issues, particularly their own reproductive health and the health of their children. Men noticed a similar change, but focused more on the decreased need for spending money on health care due to improved preventative health care. In the areas covered by some CORAD partners (most notably CARE), enhanced awareness of family planning and HIV and AIDS was also noted as a benefit from trainings. Communities served by CARE also clearly articulated the benefits of the formation of Community Health Committees (CHCs), a specific health education methodology used by CARE. A number of Traditional Birth Attendants (TBAs) and Village Health Workers (VHWs) articulated that they feel more confident in their work due to the training that they have received. In particular, they feel more able to deal with certain types of birth complications, and are empowered to seek additional medical advice if a complication is beyond their abilities or facilities. Consistent with government policy, CORAD agencies engaged in TBA training sponsored the District Health Management Team to conduct training. The CORAD agencies involved provided supervision and support. Indeed, it appears that the demand for the services of TBAs and VHWs is growing—communities frequently requested that more people be trained! Both genders also noted improved access to health services, and a higher number of referrals of more complex health issues to nearby clinics. Consistent with the DRP final evaluation, households consistently ranked distance and cost as the main factors for not visiting health clinics when a household member had diarrhea, malaria, or cough.2 A number of community members incorrectly attributed to the DRP activities to construct hygiene facilities, while others noted that not constructing such facilities had been a gap in the DRP. The Assessment Team was heartened to find out that the construction of such facilities had already been integrated into the LEAD by some of the partners. Women in communities were able to link the training that they had received to improved health and sanitation practices, such as using clotheslines from drying clothes, rather than drying them on the ground. Women linked this practice to a decrease in skin infections among their children. The impact of these intermediate results is improved health in the community. Assessment participants articulated that they perceive that births are safer, and that maternal, infant, and child mortality has declined. They also credit the efforts of the DRP in reducing disease outbreaks, with one community member stating, “Before, malaria was a big problem for us. We still have cases, but now the rates have gone down.” It is 1 As this is a final Impact Assessment, recommendations will look mainly ahead to the LEAD program that will follow on from the activities of the DRP. 2 DRP Final Evaluation Report, May 2006, page 17. CORAD-DRP Impact Assessment 12 For use by CORAD partners only not possible to verify all such results mentioned, but it is clear that from the perspective of the community, the DRP has played a significant role in the improvement of health conditions. As one focus group participant noted, “Health really is wealth.” Recommendations: ⇒ Continue to support the District Health Management Team to train the TBAs and continue to train VHWs under LEAD, both in providing refresher/additional training to existing TBAs/VHWs, and training new women to serve in these roles. ⇒ As a part of the LEAD start-up, CARE should share with other CORAD agencies its methodology for starting the CHCs, and its approach to working with communities on topics such as family planning and HIV and AIDS awareness. Intermediate and Long-term Nutrition Impacts At the time of the DRP Final Evaluation, one of the key indicators of nutrition, “percentage of children under five years of age who are underweight” had not changed significantly during the DRP.3 Reasons given for this included questions raised regarding the methodology employed for gathering the initial baseline measurement, as well as the fact that measures of underweight (weight for age) also encompass past stunting. In addition, national nutritional surveys conducted during the DRP implementation period registered an increase in the prevalence of underweight. Given these results, it was heartening that the Impact Assessment found that from the perspective of community members, the DRP has assisted in addressing this problem. In the words of one community member, “Food is happiness.” From the perspective of communities, the DRP has successfully combated malnutrition through the combination of several activities. For example, both women and men articulated that growth monitoring and promotion sessions contributed to an enhanced understanding among women about children’s nutritional needs. Women stated that they were better able to plan for the feeding of their children, and that rather than feeding them “bush yams,” a low-protein food often relied on during the hungry season, they fed their children a greater variety of food. Women directly linked this change in behavior to improved nutrition, and both genders linked this to decreased child deaths. Indeed, some communities articulated that before, many children would die from diseases related to poor nutrition, but now they did not have such problems. FFW activities have also resulted in increased availability food, which has enabled women to plan more effectively for their children’s nutritional needs. Although this can be counted as a short-term success of the DRP, it remains to be seen if, under LEAD, communities will be able to replace the food donations made through FFW with food provided through community-led safety nets. With the notable exception of Africare, Vulnerable Group Feeding (VGF) was not mentioned with any frequency during the Impact Assessment, and appeared to be something of an afterthought among participants. Field staff gave feedback that they felt this was because too few people in each community were targeted, but this might also be seen as a positive sign, given that the purpose of the VGF was to target only the most vulnerable in each community. This may also have been due to the fact that many communities referred to any kind of food distribution as food for work, as the 3 DRP Final Evaluation Report, page 11. CORAD-DRP Impact Assessment 13 For use by CORAD partners only commodities were the same and the paperwork and associated administration was also similar. The results on this point are inconclusive, which may link to a lack of clarity regarding the desired impact from the VGF. The Impact Assessment team was heartened to note that the strategy for VGF will be different under the LEAD program, with a more thorough understanding about how VGF will be utilized to meet the LEAD program goals. There was also a great degree of variety in the ability of women in targeted communities to explain the methods for identifying malnutrition. Although women were able to mention other methods as well, the color-coded growth monitoring and promotion model used by CRS was the easiest for women to articulate: “The provision of color coded beads (green, orange, and red) for growth monitoring has given mothers more knowledge in the nutritional status of our children.” While further study would be needed to ascertain if this ability to articulate had translated into decreased malnutrition in the areas in which it was in use, it is nonetheless evident that the color-coded growth monitoring and promotion method is well understood and actively used by women. Recommendations: ⇒ As per the LEAD design, utilize FFW and VGF so that food levels gradually decrease over time and are replaced with food from community-led safety nets. ⇒ The color-coded GMP methodology will be the main method for growth promotion for the LEAD program. CRS should take the lead in training other CORAD member staff in the model, and should utilize the Health Working Group to disseminate information and feedback to partners regarding color-coded GMP. Intermediate and Long-term Agricultural and Economic Impacts Community members, especially men, were able to link DRP activities to enhanced agricultural productivity and crop yields. This, in turn, meant that many people linked the DRP to increased income and enhanced livelihoods. Some responses to this effect were somewhat vague, and did not link the income to any tangible benefit. These responses were thus considered somewhat suspect. In the cases of CORAD partners World Vision (WV) and CARE, the increased income was linked by community members to an improved ability to pay school fees for children. It is interesting that this comment only came up rarely in focus groups with the other organizations, suggesting that WV and CARE may have prioritized this with the communities in which they were working, either directly through their efforts with the DRP, or in conjunction with other projects with overlapping areas of operation. Other comments related to agricultural production noted that the DRP had been successful in sharing with farmers more efficient and effective means of agricultural processing, especially the processing of cassava into garri, as the following quote explains, “Before, farmers used the rudimentary method of processing cassava into garri which is back breaking and time consuming. Today the hand operated machine is being used and it is very efficient in terms of time and labor.” Based on responses in the Impact Assessment, it appears that communities are beginning to experience benefit from the drying floors that the DRP has constructed. Specifically, these were linked to a decrease in post-harvest losses, as well as improved sanitation during the drying process: “When we use the drying floors, we do not have CORAD-DRP Impact Assessment 14 For use by CORAD partners only sand or small stones like when we dry using the ground.” Improved harvest methods piloted through the FFS were also articulated to have contributed to decreased losses. The FFS were described by participating community members as serving to enhance farmers’ skills and knowledge in the area of agricultural production and agro-processing. Comments made by farmers were often very specific, citing such improvements to their farming as crop spacing and the number of seedlings to be planted per hole.4 Farmers also shared with the Assessment Team the benefits they had received from the agricultural inputs, although it was noted that not everyone in the community benefited from such activities. This may suggest some level of dissatisfaction with the distribution of inputs, although the lack of any widespread critique to this effect suggests that this was not the case. Farmers also experienced enhanced confidence through being able to train other farmers in the techniques they had learned. Another benefit of this approach was that farmers were more willing to test different agricultural methods when explained by a fellow farmer as opposed to an unknown outsider. This collective feeling by farmers was also highlighted by farmers who expressed that they were appreciative that the FFS has provided them with enough seed that they could share with one another, and help out farmers who were less fortunate with their crops, “We have enough seeds reserved that we can share with our brothers and sisters in the community.” Recommendations: ⇒ CORAD should continue to look for opportunities to improve agro-processing practices in communities served by LEAD. Food for Work That it was challenging for the Impact Assessment Team to assess the impact of Food for Work activities was telling in and of itself. The main reason for these difficulties was that there was a wide variety of activities done as FFW, with varying levels of impact. For example, CORAD partner CARE constructed several fish ponds. Although participants in the Impact Assessment identified the ponds as an activity that had been significant for them, the cases were too isolated to conclusively say in this type of assessment what the result had been. On one hand, it may be seen as a strength that the DRP was so able to contextualize FFW activities based on the needs of the communities served. Yet on the other hand, there may be a need for a greater degree of standardization, to ensure that CORAD’s FFW achieves the desired impacts. As a number of markets had been recently completed, the markets were frequently mentioned by community members. They were not often linked to the generation of additional income, which is logical given that they have not been in use for very long. It is encouraging; however, that the markets were described as having an impact in “bringing together farmers and traders.” One outlier that is worthy of note also mentioned the possibility of using markets as venues for meetings, although the Assessment Team was not sure how this would be possible as the markets are usually in use. 4 It should be noted that site visits were not included in the Impact Assessment. Thus, it is not possible to determine conclusively whether farmers are indeed using improved methods on their farms. CORAD-DRP Impact Assessment 15 For use by CORAD partners only It also appeared that from the perspective of the community, there was some confusion on the part of the community regarding FFW activities and Food for Agriculture activities that were conducted early on in the DRP; from their perspective they often mis-identified Food for Agriculture activities as FFW. Further confusing this issue is that FFS sometimes conduct similar activities, without providing food for work. For example, if an FFS creates a demonstration plot, they do not receive food for the work that they do; however, if a community farm (which may appear similar from the perspective of the community) is done in the same community as a Food for Agriculture activity, they do. Although there was no widespread dissatisfaction about this stated by participations in the assessment, this overlap may create confusion, and should be addressed. One noteworthy exception is that regardless of the actual nature of the construction, FFW activities were frequently linked to an increase in focus on development and community work. This is discussed below in the section on “Peacebuilding and DRP.” Recommendations: ⇒ Consider the possibilities of limiting FFW under LEAD to a few categories of activities that CORAD members feel, based on their DRP experiences, have been high￾impact. Another possibility for activities would those that related to past FFW activities, to maximize the impact from the structures themselves. ⇒ As a part of LEAD start up, establish policies regarding FFW activities that take into consideration similar activities that might be done by community stakeholders affiliated with CORAD without the provision of food. Peacebuilding and DRP One of the most remarkable and unanticipated findings of the DRP was that community members perceive strongly that the project has contributed to a decrease in conflict. Such impacts as “unity,” “peacefulness,” and “more communal work” were all mentioned regularly. Community members not only attributed such impacts to the relationship between themselves and the CORAD partners, but also pointed specifically to FFS activities and FFW projects. These projects have provided a mechanism for community leaders and other concerned individuals to organize their neighbors around development activities—something that they stated they had not had significant opportunities to do in the past. In the words of one man, “We are now able to organize and mobilize our groups to undertake community work.” Government officials also note the change in the areas they cover, stating, “DRP has created awareness in the development of the chiefdom.” It was not possible to link unity or peacefulness to specific activities, although one possible rationale would be that the DRP has resulted in improved livelihoods, which has decreased tension. One participant noted, “There is now unity and peace amongst community members”—a sentiment shared in a number of discussions. One notable comment made by a female community member stated that the peacefulness was also felt at the household level, with a decrease in domestic violence. Although it is not possible to verify such a statement, there may be exceptional situations where this is in fact the case. Recommendations: ⇒ As the LEAD project has a component dealing with democracy and good governance, consider applying the lessons learned from the DRP with regards to CORAD-DRP Impact Assessment 16 For use by CORAD partners only peacebuilding towards building peaceful relationships between organizations and governmental bodies and those they serve. Gender Impacts All stakeholder groups included in the Impact Assessment noted that the DRP has enhanced the role of women in their communities. Women noted benefits such as enhanced confidence, and improved involvement in decision making in the home, with one woman stating, “We are able to plan for the future together as a family.” Men also noted that the role of women has changed, but focused more on wage-earning, stating the women were more able to raise money for expenses in their homes. A few men also noted that women’s participation in health trainings had led to a decrease in the need for spending on health, which they considered positive. It was not clear during the Impact Assessment whether the improved decision making ability that women possessed included decisions made regarding the allocation of food. Although women have been trained in child nutrition, do they usually have the ability to provide additional food to malnourished children, especially in times when food for the family is not readily available? Although mothers clearly linked the trainings conducted to improved nutrition of their children, decisions about food allocation are profoundly related to cultural practices and social norms regarding women and children’s status in Sierra Leonean society. Changes in this area will require sensitive, consistent emphasis, and this remains a point upon which the LEAD program will endeavor to address. Recommendations: ⇒ Under LEAD, continue to engage women in both health and agricultural activities, for the purpose of empowering women within their homes and communities. ⇒ As a part of the start-up for LEAD, gather additional information on who makes household level decisions regarding the feeding of children (both content and quantity). Perceptions of Government Officials Overall, the communities served expressed favorable impressions of the CORAD agencies and the work they had done through the DRP. Indeed, the small number of negative responses was seen by the analysis team to reflect not only a positive impression of the program, but also that individuals did feel comfortable in expressing certain areas where they were less satisfied. One particularly telling comment summarizes this sentiment: “We are now better off than before. Before we were standing upright wishing to lie down. [CORAD’s] assistance has helped us to sit down. We still want [CORAD] to help us so that we can lie down, which has been our dream. We are looking forward to that.” There were some negative perceptions from government officials, particularly at the District level. The main reason stated by these key stakeholders was a lack of coordination by partners. Indeed, one District Council Chairman insisted that he had no heard of the DRP, although CORAD partner records show otherwise. When questioned on this point, CORAD partner staff also stated that it can be difficult to engage government officials, who sometimes request seating fees to participate in meetings. CORAD-DRP Impact Assessment 17 For use by CORAD partners only While there may be a number of reasons why it proved difficult to engage with key governmental partners, significant successes, most notably the relationship between CORAD health personnel and the Ministry of Health, indicate that it is possible to overcome these obstacles. Given the greater focus under LEAD towards relationships with governmental officials, it is a point on which CORAD should consider spending additional time and energy. Interestingly, a Ministry of Health official in one District made a statement that the analysis team found noteworthy, stating, “The burden of training has been lifted from the Ministry.” While this comment shows that the DHMT considers CORAD health training activities helpful, it seems to imply a perception of shifting the burden, rather than sharing it. It will be important for CORAD agencies to emphasize the complementary nature of training and other activities. Recommendations: ⇒ As a part of LEAD planning activities, CORAD partners should review their strategies for ensuring the engagement of key government officials and agencies, and make adjustments based on their experiences in each District under the DRP. ⇒ CORAD trainings conducted under LEAD should require a contribution from the relevant line ministry in terms of personnel, supplies, or other assistance. CORAD-DRP Impact Assessment 18 For use by CORAD partners only Conclusions and Implications for LEAD The DRP has clearly achieved some very significant impacts during its three years of implementation, as is evident from the large number of intermediate and long-term impacts noted by communities and key decision makers. While significant obstacles to long-term, sustained development still exist, DRP participant communities made substantial inroads in food security and community unity, particularly noteworthy in a country transitioning to peace after a protracted civil war. Given this context, the unanticipated impact of enhancing peace and unity in targeted communities is extremely significant. Indeed, given the implications of this finding on the work of CORAD agencies and FFP in other post-conflict environments, this finding should be properly researched and documented. Although significant challenges remain, the DRP has successfully aided in that transition, and in so doing has created a platform for the LEAD program, through which CORAD and its governmental and non￾governmental partners can support communities in continuing along the path to long￾term development. In this process, it is crucial that CORAD look for ways to document and standardize best practices from its partner agencies, such as the significant successes CRS has seen with the color-coded GMP model. Part of this process may also be looking at the targeting for VGF and the types of activities considered for FFW. CORAD should also continue to engage women in targeted communities in both agricultural and health programming, not only to benefit their economic and health situation directly, but also to empower women to be more active in development and on economic issues in their communities. CORAD-DRP Impact Assessment 19 For use by CORAD partners only Annexes Annex 1: Impact Assessment Guide Focus Group Planning and Sampling Guide Details Time Frame Staffing Facilitation Teams: • Facilitator: Asks questions, decides when to move on to next question • Recorders (x2): Writes down answers, as specifically as possible, and meet afterwards to compare their notes and come to a common understanding regarding what they have heard. • Time keeper/Observer: Makes sure that facilitator is aware of time, observes dynamics of focus group and shares any observations with facilitator after focus group (should NOT interrupt focus group for any reason) There should be at 2 teams, one all women and one all men. All members of the Facilitation Team must be able to speak the local language. Staff should be identified as soon as possible. List to be sent to M&E Unit no later than 10 January. Anticipated Timeframe FGs should be conducted in January or February, with data sent to M&E Unit for analysis no later than 28 February, 2007. Dates should be set no later than 2 weeks prior to focus group. Training A training for Facilitators will be conducted on 18 January in Bo (exact location TBA). This will be a one day training, and all facilitators should attend (please note that non-attendees cannot facilitate FGs). There will be no CORAD-wide training for Recorders, however, agencies should plan to test their skill in note-taking, to make sure that they can fill this role. 18 January (facilitators) Prior to FGs for Recorders. Field Testing To be decided, based on 18 January training (as may depend on skill/experience of facilitators). Community Sampling The following purposive sample of chiefdoms was taken to provide significant data with regards to the District, and the partners. The responsible organization should conduct FGs in 3 villages per chiefdom. Kailahun: • Dea (Africare) • Njaluahun (Africare) • Peje West (CRS) Koinadugu List of selected villages to be sent to M&E Unit prior to 18 January. CORAD-DRP Impact Assessment 20 For use by CORAD partners only • Diang (CRS) • Folosaba Dembelia (CARE) Tonkolili • Gbonkolenken (CARE) Bonthe • Jong (World Vision) Kono • Mafindor (World Vision) • Toli (World Vision) There should be 2 FGs in each communities: one for women and one for men (see criteria below) # of Participants/ Sampling 8-12 participants in each group, as per the following criteria. Men • 1-2 Village Development Committee members • Youth leader • Farmers who have participated in the agriculture portion of the project • Husbands whose wives have participated in the health portion of the project • Heads of household whose household has participated in the VGF • At least 2 men below the age of 35 Women • Women’s Group leader • 1-2 Village Development Committee members • Women who have participated in the health portion of the project • Heads of household whose household has participated in the VGF • Women whose husbands have participated in the agriculture portion of the project • At least 2-3 women below the age of 35 No later than 2 weeks before FG Location Common meeting location in the village, that is comfortably accessible for women and men. Ideally, FGs would be done in a location where the facilitator can control people observing/listening/making comments FGs should be done in the morning, so that people can go to their farms. 2 weeks before FG, including getting permissions where necessary Time Needed Allow for 1.5-2 hours. The focus group should Part of invitation CORAD-DRP Impact Assessment 21 For use by CORAD partners only only last for around 1-1.5 hours, but this way, if you start late, you will be able to finish! (see below) Invitations 1. Go first to the Paramount and Village Chief, to let them know that the meeting is taking place, and that they will be separately interviewed. Make sure that they are aware that they should not be present. 2. After sampling of participants, project staff should go to houses of participants, invite them to the meeting. Invitations must include a clear understanding of the purpose of the FG, and how the information will be used—it is unethical for us to get participants if they do not understand these things. Invitations should be done approximately 3-4 days before FG. Chiefs to be informed before this. CORAD-DRP Impact Assessment 22 For use by CORAD partners only Interview Planning and Sampling Guide Details Time Frame Staffing Facilitation Teams: • Facilitator: Asks questions, decides when to move on to next question • Recorder (x1): Writes down answers, as specifically as possible, and meet afterwards to compare their notes and come to a common understanding regarding what they have heard. May be same as the FG team. All should speak the local language Staff should be identified as soon as possible. List to be sent to M&E Unit no later than 10 January. Anticipated Timeframe FGs should be conducted in January or February, with data sent to M&E Unit for analysis no later than 28 February, 2007. Dates should be set no later than 2 weeks prior to focus group. Training A training for Facilitators will be conducted on 18 January in Bo (exact location TBA). This will be a one day training, and all facilitators should attend (please note that non-attendees cannot facilitate FGs). There will be no CORAD-wide training for Recorders, however, agencies should plan to test their skill in note-taking, to make sure that they can fill this role. 18 January (facilitators) Prior to FGs for Recorders. Field Testing To be decided, based on 18 January training (as may depend on skill/experience of facilitators). Community Sampling The following purposive sample of chiefdoms was taken to provide significant data with regards to the District, and the partners. The responsible organization should conduct interviews with chiefs in 3 villages per district. Kailahun: • Dea (Africare) • Njaluahun (Africare) • Peje West (CRS) Koinadugu • Diang (CRS) • Folosaba Dembelia (CARE) Tonkolili • Gbonkolenken (CARE) Bonthe • Jong (World Vision) Kono • Mafindor (World Vision) • Toli (World Vision) List of selected villages to be sent to M&E Unit prior to 18 January. Interviews to District Council Chairman: Provide names/ CORAD-DRP Impact Assessment 23 For use by CORAD partners only be Conducted • Kono: World Vision • Bonthe: World Vision • Tonkolili: CARE • Koinadugu: CRS • Kailahun: Africare District Medical Officer • Kono: World Vision • Bonthe: World Vision • Tonkolili: CARE • Koinadugu: CRS • Kailahun: Africare Mayors (some areas): to be decided by agencies Agriculture District Director • Kono: World Vision • Bonthe: World Vision • Tonkolili: CARE • Koinadugu: CRS • Kailahun: Africare Chairman of Ag Development Committee: • Kono: World Vision • Bonthe: World Vision • Tonkolili: CARE • Koinadugu: CRS • Kailahun: Africare Chiefdom Speaker: to be done in selected chiefdoms where FGs will be held, as per FG guide. Chiefdom Counselor: to be done in selected chiefdoms where FGs will be held, as per FG guide. Paramount Chief: to be done in selected chiefdoms where FGs will be held, as per FG guide. Village chiefs: to be done as per FG guide in the villages included in FG sample. titles to the M&E Unit no later than 2 weeks before FG Location Offices of members mentioned above. Time Needed Allow for 1 hour Part of invitation (see below) Invitations Make appointment to see these people. Appointments set between 1 week CORAD-DRP Impact Assessment 24 For use by CORAD partners only Invitations must include a clear understanding of the purpose of the FG, and how the information will be used—it is unethical for us to get participants if they do not understand these things. and 1 day before, depending on individual and relationship with agency. CORAD-DRP Impact Assessment 25 For use by CORAD partners only Focus Group/Interview Question Guide, English5 Introduction: Good morning/afternoon. My name is ……………………, and I am working with ……………………. on the DRP Project. We are here to get your opinions today regarding what aspects of the DRP project have been the most important here in ….COMMUNITY NAME…. I will be acting as the facilitator today, and would like to ask my colleagues to introduce themselves. [Wait for the Recorder and the Time Keeper to introduce themselves.] Thank you. I will be asking you some questions today, and would like to encourage you to be honest in your responses. We will not be recording any names, but we will be using your comments in a report regarding your opinions on the work of the DRP. This will not affect the chances of your village to receive funding in the future. Is everyone comfortable to participate? [Wait for people to respond affirmatively. If people have questions, answer them. If people choose not to participate, tell them they are free to go.] I would like you all to introduce yourselves—again, we will not use your names, but we would like to get to know you a bit. [Wait for participants to introduce themselves.] Thank you. Let’s begin. Focus Group/Interview Questions: 1. What do you feel is the most important thing that the DRP [or ORGANIZATION NAME] Program has done here? 2. What about this has been particularly important for you and your family? 3. And what has this [change] meant for you? 4. How is the situation now different than it was before? 5. Why do you think that this change [from before the DRP to now] has been so significant for you? In a FG, once you feel you have the answer from one participant, ask the same question to others. If you find you are interrupted, feel free to go back and ask additional questions to a particular respondent. 5 This will be the same for both Focus Groups and Interviews CORAD-DRP Impact Assessment 26 For use by CORAD partners only Conducting Focus Groups/Interviews, Instructions and Tips 1. The introduction is a mandatory part of the interview/focus group—it is important ethically, and it helps to set people at ease. 2. As a part of the introduction, make sure that participants clearly understand the purpose, and how their information will be recorded and used. You MUST wait for participants to give oral consent. 3. The facilitator will be walking a fine line between acknowledging authority figures or older men/women, and making sure that people can freely express their opinions. 4. The trick with this is to KEEP Digging!!!! Don’t stop until you have reach the “ultimate change.” 5. If chiefs do try to attend focus groups, politely but firmly let them know that they will be separately interviewed on the same points. 6. Recorders must submit their raw notes, in case points in the final data provided are unclear. 7. Because of the nature of this exercise, in focus groups you will probably ask the same set of questions more than once, to different people. This is normal—be patient and record answers carefully. 8. Don’t worry if your focus group respondents take more “steps” to get to their “ultimate change,” or take more time discussing the past situation. Simply record all of the rich data they are give you, and take your time. 9. This technique does NOT work well with larger focus groups—make sure that you invite people as per the specifications above. 10. Remember that Most Significant Change only measures impact from the perspective of beneficiaries. This is their qualitative perspective on what has made a difference. The DRP evaluation provides a more objective statistical measure of impact. 11. Take your time. You do not have many questions to ask. The main thing is to follow the chain all the way to the “ultimate change.” 12. Do not force people to answer, and do not keep other people from chiming in if they agree with another person’s idea. Just record all results. 13. Never disagree with focus group respondents. If they are sharing their opinions, how can they be wrong? 14. There is almost no way to have a “standard” focus group or interview. You may find that you have to ask questions slightly different ways to get respondents to “dig deeper.” Do not worry about varying slightly the way that you ask the questions, but do try to stick to the guide. CORAD-DRP Impact Assessment 27 For use by CORAD partners only Examples of MSC done incorrectly Facilitator: What do you feel is the most important thing that the DRP Program has done here? Respondent (Female, mother): I think that the most important thing is the improvements in the health monitoring for our children. F: Thank you. Next? Problem: Facilitator stopped too early. Keep digging! Facilitator: What do you feel is the most important thing that the DRP Program has done here? Respondent (Female, mother): I think that the most important thing is the improvements in the health monitoring for our children. F: Actually, though, our studies show that rates of nutrition are still poor, so this is not a change. Who has a better example? Problem: Facilitator contradicted respondent. Remember, this is about getting their perspective! Example of MSC done correctly Facilitator: What do you feel is the most important thing that the DRP Program has done here? Respondent (Female, mother): I think that the most important thing is the improvements in the health monitoring for our children. F: Okay, good. What about this has been particularly important for you and your family? R: Now we can see very clearly the health status of our children, with the new colour system. F: Yes, I see. And what has this meant for you? R: For me, I have four children. Now I can show you very clearly what is the status of each child, and I make sure that the children with poorer nutrition receive more food. F: And how is this different than the situation before? R: Before, I would just look at my child, and try to estimate, but there were many things I did not know. Now, with the help of the VHW’s in our village, I have a clear role in assessing my child, and I tell all of the mothers in our village that they should participate in the program. F: Why do you think that this has been so significant for you? R: For me, I feel empowered to help with my children’s health. For my children, I think that now they will have a better future, with a better nutrition. Good nutrition is the start of a better life, where they may have a good education and a bright future. CORAD-DRP Impact Assessment 28 For use by CORAD partners only List of Proposed Participants Focus Group: Type of Group: Men / Women (Circle One) Community Name: ____________ Chiefdom: ____________ District: __________ Organization: __________ No. Name of Participant Name of Interviewee Title of Interviewee Agency/Ministry Please submit FG list for each FG and Interview list to M&E Unit as per the schedule given above. Documentation and Analysis The main analysis with the technique is to trace the chain of the respondent’s logic, until you reach the “ultimate change.” This guide, using the example above, should provide some help. (You do not have to use this form, but you might find it useful.) Community Name: ___________________________________ Respondent Demographic: __Female, mother______________ Recorders are also required to submit their original notes to the M&E Unit. Once the data gathering has been done, the remainder of the analysis will be done by the M&E Unit. I think that the most important thing is the improvements in the health monitoring for our children. Now we can see very clearly the health status of our children, with the new colour system. For me, I have four children. Now I can show you very clearly what is the status of each child, and I make sure that the children with poorer nutrition receive more food. Situation Before DRP vs. Now: Before, I would just look at my child, and try to estimate, but there were many things I did not know. Now, with the help of the VHW’s in our village, I have a clear role in assessing my child, and I tell all of the mothers in our village that they should participate in the program. For me, I feel empowered to help with my children’s health. For my children, I think that now they will have a better future, with a better nutrition. Good nutrition is the start of a better life, where they may have a good education and a bright future.