Decentralized Evaluation Decentralized evaluation for evidence-based decision making Prepared by Bert Fret, Team Leader Fernanda M Leite Villamarin, International Consultant Raul Mendes Fernandes, National Consultant Final evaluation report March 2021 WFP Guinea-Bissau Country Office Evaluation Manager: Elber Nosolini Final Evaluation of the McGovern-Dole International Food for Education and Child Nutrition Program in Guinea-Bissau From 2016 to 2019 Evaluation Report i | P a g e Acknowledgements The Evaluation Team (ET) wishes to thank the World Food Programme Country Office (WFP CO) in Bissau for all assistance received during the evaluation mission. We are also grateful to all Ministry of Education (MoE) staff, both in Bissau and in the regional capital, and in the various schools that were visited, who took the time and were available to help us along with all the information we needed. We thank the volunteer cooks in the school kitchens, and the leaders of the School Management Committees (SMCs) and Parents’ Associations (PAs), and the local NGOs with whom we could discuss, for helping us to understand their perspective. And finally, our thanks go also to the pupils, boys and girls, whom we could interview, for volunteering, and for readily answering our questions. Disclaimer The opinions expressed in this report are those of the Evaluation Team, and do not necessarily reflect those of the World Food Programme (WFP). Responsibility for the opinions expressed in this report rest solely with the authors. Publication of this document does not imply endorsement by WFP of the opinions expressed. The designation employed and the presentation of material in maps do no imply the expression of any opinion whatsoever on the part of the WFP concerning the legal or constitutional status of any country, territory or sea area, or concerning the delimitation of frontiers. DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third-party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this document can be made available by contacting fas.monitoring.evaluation@usda.gov ii | P a g e Table of contents Acknowledgements........................................................................................i Disclaimer .....................................................................................................i Executive Summary...................................................................................... 1 Context ................................................................................................................................................................1 Methodology........................................................................................................................................................1 Key Findings: ..................................................................................................................................................... 2 1. Relevance ....................................................................................................................................................... 2 2. Effectiveness................................................................................................................................................... 2 3. Efficiency ........................................................................................................................................................ 3 4. Impact ............................................................................................................................................................ 3 5. Sustainability.................................................................................................................................................. 4 Conclusions and Recommendations.................................................................................................................. 4 1. Introduction ...........................................................................................8 1.1. Overview of the Evaluation Subject.......................................................................................................8 1.2. Context................................................................................................................................................... 9 1.3. Evaluation Methodology and Limitations:........................................................................................... 11 1.4. Limitations:.......................................................................................................................................... 13 1.5. Ethics:................................................................................................................................................... 13 2. Evaluation Findings .............................................................................. 14 2.1. Evaluation Question 1.1: Is the project’s strategy relevant to the beneficiaries’ needs? .................... 14 2.2. Evaluation Question 1.2: Is the project aligned with the national government’s education and school feeding policies and strategies? ........................................................................................................................17 2.3. Evaluation Question 1.3: Does the project complement other donor-funded and government initiatives? ........................................................................................................................................................ 18 2.4. Evaluation Question 2.1. The extent of project implementation – Did the project carry out all activities as planned? (Effectiveness).............................................................................................................. 19 2.5. Evaluation Question 2.2: To what degree have the interventions resulted, or not, in the expected results and outcomes? In particular, to what extent did providing THRs result in increased attendance and enrolment of girl pupils? (Effectiveness).........................................................................................................23 2.6. Evaluation Question 2.3. Is short term hunger reduced? (Effectiveness)..........................................24 2.7. Evaluation Question 2.4. Did assistance reach the right beneficiaries in the right quantity and quality at the right time? (Effectiveness).........................................................................................................24 2.8. Evaluation Question 3.1. Has the programme been implemented in an efficient way? Was there efficient use of resources and efficient methods of work? (Efficiency)...........................................................26 2.9. Evaluation Question 4.1. To what degree has the project achieved the results that were foreseen in the project level framework? (Impact) ...........................................................................................................28 2.10. Evaluation Question 4.2. Have there been any unintended outcomes, either positive or negative? (Impact)............................................................................................................................................................34 2.11. Evaluation Question 4.3. What internal and external factors affected the project’s achievement of intended results? (Impact).............................................................................................................................. 35 2.12. Evaluation Question 5.1: Is the school meal programme sustainable, including a strategy for: sustainability; sound policy; stable funding; quality programme design; institutional arrangements; local iii | P a g e production and sourcing; partnership and coordination; and community participation and ownership? (Sustainability).................................................................................................................................................36 2.13. Evaluation Question 5.2 What substantive progress has the Government made toward developing a nationally owned school feeding programme? (Sustainability)..................................................................... 37 2.14. Evaluation Question 5.3. How are local communities involved in and contributing to school feeding? (Sustainability) .................................................................................................................................39 2.15. Evaluation Question 5.4. What needs remain in order to achieve a full handover and nationally owned school feeding programme? (Sustainability).......................................................................................40 3. Conclusions and Recommendations......................................................42 Overall assessment...........................................................................................................................................42 Lessons learned..........................................................................................46 Annexes......................................................................................................47 Annex 1 - McGovern-Dole FFE Framework .................................................................................................... 47 Annex 2 - Revised version of the reconstructed Logical framework of the WFP/McGovern Dole FFE Programme in Guinea Bissau (2016-19)..........................................................................................................49 Annex 3 – Evaluation Matrix............................................................................................................................51 Annex 4 - Interviewed Stakeholders................................................................................................................ 61 Annex 5 - Number of schools assisted by the McGovern-Dole FFE programme and visited during the baseline, Mid-Term and Final evaluations......................................................................................................62 Annex 7 - Project Performance Indicators ......................................................................................................63 Annex 8 – Total amount of food distributed to schools, meals prepared and served.................................... 67 Annex 9 – Approximate estimates of financial reporting ...............................................................................68 Annex 10 - Nutritional content of one school meal.........................................................................................69 Annex 11 – Results of the SABER exercise ...................................................................................................... 70 Annex 12 - To what extent have the recommendations of the baseline survey and of the mid-term evaluation been implemented by WFP............................................................................................................ 72 Annex 13 - Terms of Reference ........................................................................................................................ 74 Annex 14 - Documents Reviewed.....................................................................................................................92 Annex 15 – Documents gathered.....................................................................................................................96 Annex 16 - Data Collection Tools....................................................................................................................101 Annex 17 - Bibliography..................................................................................................................................113 iv | P a g e List of Tables Table 1.1 - Overview of the McGovern-Dole FFE-WFP project in Guinea-Bissau (2016-2019)....... 8 Table 2.1 - Application of international criteria and the corresponding evaluation questions .......14 Table 2.2 - Take-home rations for boys and girls by region and income quintile (2019) ...............16 Table 2.3 - Number of girls from 4th to 6th grade receiving THR by year ........................................20 Table 2.4 - Perceptions of students (boys and girls) at WFP-supported schools who reported on quantity and quality of the meals provided (2019).......................................................................... 25 Table 2.5 - Main programme outcome values per year, against the baseline and the target values ............................................................................................................................................................28 Table 2.6 - Median enrolment for all applicable school levels, according to school years (2014-15, 2017-18 and 2018-19) and gender. ................................................................................................... 29 Table 2.7- Profile of households according to the Food Consumption Score by year (2016, 2018, 2019)...................................................................................................................................................31 Table 2.8- Profile of households according to wealth quintiles by year (2016, 2018, 2019)...........31 Table 2.9- Percentage of schools with improved source of water and toilet facilities, handwashing practices and separate toilets (2016, 2018, 2019).............................................................................31 Table 2.10- Percentage of SMC members who received training from WFP (2016, 2018, 2019).. 32 Table 2.11- Analysis of Results of SF End-line Survey (2019): Means values in percentage and significant differences between WFP assisted and non-WFP assisted schools .............................. 33 Table 2.12- Median of numbers of students in WFP and Non-WFP schools (2019)...................... 34 Table 2.13- Local communities’ participation (2016, 2018, 2019)..................................................40 List of figures Figure 2.1 - Attendance rates of grade 4 students in WFP and non-WFP Schools (baseline, mid￾term and final evaluation)................................................................................................................. 29 Figure 2.2. - Percentage of students who ate a meal before and after school (2016, 2018, 2019).30 v | P a g e List of Acronyms ABC Brazilian Cooperation Agency CIMCE Inter-ministerial School-Feeding Committee CO Country Office CRS Catholic Relief Service DEQAS Decentralized Evaluation Quality Assurance System DGASCE General Directorate for School Meals and Social Affairs DGPASE Evaluation and Planning Department of the Ministry of Education ESP Education Sector Plan ET Evaluation Team FAO Food and Agriculture Organization of the United Nations FSNMS Food Security and Nutrition Monitoring System GB Guinea-Bissau GEEW Gender Equality and Emancipation of Women GoGB Government of Guinea-Bissau HGSF Home Grown School Feeding HIV Human Immunodeficiency Virus ICSP Interim Country Strategic Plan INE National Statistics Institute IPDH International Programme for Human Development LIC Low-income country MADR Ministry of Agriculture and Rural development MAM Moderate Acute Malnutrition M&E Monitoring and Evaluation MNECJD Ministry of National Education, Culture, Youth and Sports (former name) MENES Ministry of National Education and Higher Education (current name) MGD McGovern-Dole MGD FFE McGovern-Dole Food for Education and Child Nutrition Programme MoE Ministry of Education NAEC Nutritional assessment, education and counselling NDSF National Directorate of School Feeding OEV Office of Evaluations PA Parents’ Association SABER Systems Approach for Better Education Results SDG Sustainable Development Goals SFP School Feeding Programme(s) SISSAN Food and Nutrition Security Monitoring System SMC School Management Committees THR Take-home ration(s) ToR Terms of Reference UNDP United Nations Development Programme UNFPA United Nations Population Fund UNICEF United Nations Children's Fund US United States USDA United States Department of Agriculture VAM Vulnerability Analysis & Mapping WFP World Food Programme RB Regional Bureau WHO World Health Organization 1 | P a g e Executive Summary The final evaluation of the McGovern-Dole International Food for Education and Child Nutrition Program (McGovern-Dole FFE) project (FFE-657-2015/019-00), implemented by the World Food Programme (WFP) in Guinea-Bissau (GB), from March 2016 to July 2019 (extended to April 2020), had two main objectives: (1) Accountability: To account for the activities carried out by WFP as well as the outputs and outcomes reached; (2) Learning: To draw lessons for the main actors. Baseline (2016) and mid-term (2018-2019) evaluations preceded this end-line evaluation. The baseline study conducted before the start of the project provided a situational analysis and allowed WFP to establish indicator baseline information. It also verified the targets established in the Project Agreement. The mid-term evaluation covered the period from August 2017 (preparation phase) to July 2019 (final evaluation report). This evaluation aimed at allowing the WFP to monitor the progress of the established indicators. The present end-line evaluation comprises the entirety of activities covered by the McGovern-Dole funded WFP school feeding project in GB (2016-2018). The final evaluation, in line with the completed mid-term evaluation includes: (1) a review of the project’s relevance, effectiveness, efficiency, impact, and sustainability; (2) the collection of performance indicator data; more specifically it (3) assesses whether or not the project achieved its expected results; (4) identifies lessons learned; (5) assesses project replicability; and (6) investigates whether or not the mid-term evaluation recommendations were implemented. Main users of the evaluation are the WFP Country Office (CO), the WFP Regional Bureau (WFP RB), the Office of Evaluation (OEV), the Ministry of National Education and Higher Education (Ministério da Educação Nacional e Ensino Superior, MENES)1 along with its General Directorate for School Meals and Social Affairs (DGASCE), USDA, and Japan. The study covered 100 schools, 50 of which were WFP schools and 50 of which were non￾WFP schools, as a control group. WFP schools covered a total of 19,323 students. Context GB is a low-income country (LIC), with 70% of the population living below the poverty line of USD 1.9 per day. Women are most affected by poverty due to inadequate healthcare, low levels of education, poor literacy rates, and low-income rates. Roughly half of the population 15 years of age or older are illiterate, and malnutrition among children under 5 years old is staggering: in 2016, up to 4.2% and 6.1% of children under five had severe and moderate acute malnutrition, respectively. The main activities of the McGovern-Dole FFE project, which cost USD 20 million, were the provision of school meals to pupils in 758 primary schools, and of take-home rations (THRs) of rice to girls in grades 4, 5, and 6 with sufficient attendance (80%). It also included activities related to capacity building and equipment provision. WFP’s main partner was MENES and its DGASCE. Methodology A mixed-methods approach was implemented for data collection. Primary data was collected from stakeholders using inquiry techniques such as questionnaires, interviews, and on-site observation. The quantitative survey used a non-experimental method. Data were collected by applying the same questionnaire as the base-line and mid-term surveys in 100 schools, distributed in eight regions of GB. The final sample consisted of 50 WFP￾schools and 50 non-WFP schools. Interviews were conducted with school directors (n=100, one in each school visited), male (n=500) and female (n=500) students (five per sex per school) and male (n=500) and female (n=500) students’ parents and guardians (n=1,000). 1 Previously named Ministry of National Education, Culture and Youth and Sports (MNEJCD) 2 | P a g e Existing WFP reports were used for triangulation along with other sources of data2. Both gender equality and human rights were mainstreamed throughout the evaluation process. The qualitative survey was based on semi-structured individual and group interviews during the fieldwork phase of the evaluation. These interviews were held in 30 WFP￾schools located in six regions of GB: Oio, Bafata, Cacheu, Biombo, Quinara and Gabu. In Bissau, interviews were carried out with the WFP staff involved in SF project management, as well as stakeholders from the Ministry of Education (MoE), the Ministry of Agriculture, and other government institution; national non-governmental organizations (NGOs); and the United Nations Children's Fund (UNICEF). Additionally, the ET gathered secondary data from databases, reports, surveys, web resources, and other documented sources for triangulation. Results from the mid-term school feeding survey were also considered for data comparison. Some limitations were encountered during fieldwork, such as unreliable road or sea access and diverse language use. Nevertheless, measures were taken to mitigate these barriers as much as possible, i.e., by using local interpreters; by inviting a key actor from the Bolama￾Bijagós region (which was not visited) for an interview in Bissau. Key Findings: 1. Relevance The school feeding program (SFP) is very relevant to the needs of pupils, their families and local communities, and is aligned with educational policies and strategies of the Government of Guinea Bissau (GoGB). It alleviates short-term hunger and supplements household food income. The THRs given to girls from 4th to 6th grade have motivated parents to send their daughters to schools. The strategy to evolve towards a programme based on locally purchased foods, or Home Grown School Feeding (HGSF), further increases the relevance of the programme to local communities. This shift should provide a local selling opportunity for farmers - increasing their income - and could thus have a positive effect on the local economy, thereby reducing poverty. Currently, two pilot initiatives are testing different modalities of local food purchases. One is funded by the GoGB, and the other is funded by Japan. 2. Effectiveness All activities linked to the handling of school meals at schools were generally well performed and short-term hunger has been reduced. Overall, the targeted numbers of pupils (boys and girls from grades 1 to 6) have been served the agreed upon number of quality school meals. Meals were 7.67 % in excess of the predicted number. Girls from 4th to 6th grade with an attendance of 80% or more have received a monthly THR of rice for their families in higher numbers than foreseen (1.4% more3). Delays were negligible. The planned number of kitchens, storerooms and firewood saving stoves have all been renovated or constructed. There is still space for improvements for some of the accompanying activities that have been performed to a lesser degree, e.g., the training of various locals and MENES personnel (30,36% of target, see activity 8, table A.7.1, Annexe 7 p. 69), and the number of timely school feeding reports produced (50,45% of target, see activity 9, same table, p. 70). The high turnover of the MoE’s staff and recurrent strikes have been among the reasons for this insufficient level of performance. The DGASCE benefitted from capacity strengthening and has been involved in the local purchase pilot projects. However, further improvements should be made to increase government ownership. The local purchase pilot project extension in the region of Biombo 2 Sources: I) UNICEF Annual Reports; II) Guinea-Bissau Interim Country Strategic Plan 2018-2019; III) Rethinking School Feeding: Social Safety Nets, Child Development, and the Education Sector. World Bank (2009). Available at: https://elibrary.worldbank.org/doi/abs/10.1596/978-0-8213-7974-5, access in 8/12/2019; IV) World Food Programme/Partnership for Child Development /World Bank: Workshop for Assessing National Capacities in School Feeding in Guinea-Bissau - SABER Action Plan; V) Terms of Reference, Mid-term and Final Evaluations of McGovern-Dole funded School Feeding project in Guinea-Bissau (2016-2018). 3 See Annex 7, p.67. 3 | P a g e will be implemented in 2020 by MENES so that it can gain management experience and improve ownership. The WFP provided important technical assistance to MENES and to the GoGB in the areas of policy formulation and development of a legal framework. This has led to the promulgation of the National School Feeding Law. Outcome results for the specific subgroup of girls from grades 4 to 6 were not defined as indicators, hence no quantitative data exist for this subgroup, and no proof of the effect of the THR could be measured. However, qualitative data gathered through interviews with several stakeholders reflect that THRs is correlated to an increase in enrolment, the maintenance of school attendance, lower school dropout rates, and a narrow gender gap in the target schools. Quantitative data show a stagnation during the years the project was underway for the overall retention rate of all pupils from grades 1 to 6, and no reduction of the drop-out rates for WFP assisted schools, which remained at 5%. Nevertheless, this rate is much lower than the most recent national dropout average4. This includes all non￾supported primary schools. While WFP did reach poor children, the programme did not reach the poorest children from the poorest and most vulnerable communities. The unequipped and makeshift schools (including rudimentary kitchens) in these more vulnerable communities did not meet WFP technical selection criteria. The WFP CO justified this by stating the need to safeguard children from unsafe meals by ensuring proper food storage, proper hygiene conditions in kitchens, and acceptable school infrastructure for teaching and learning. Also, WFP has shifted its focus exclusively to rural schools with local food production, stopping its support of urban schools in an effort to reach more of the poorest children. 3. Efficiency WFP has used very efficient methods, work procedures and monitoring systems to ensure the correct flow of food from the Port of Bissau to beneficiary schools. Control over the food stock and its use in the schools was more efficient when there was an active School Management Committee (SMC) and Parent Association (PA). The toll-free anonymous complaint and denunciation phone number (106) is well known and regularly used by local communities. WFP has leak-proof systems to record the messages and deal with the issues accordingly. Interviews with stakeholders from local NGOs revealed that the quality and sustainability of their work are frequently jeopardized by the lack of budget and inadequate infrastructure provided by WFP for them to conduct their activities, e.g., lack of vehicles. Because the NGOs involved in the two HGSF pilots about local food purchases are only offered short￾term contracts, planning for them is difficult. According to the WFP CO, this mainly happens because the pilots are funded annually so they cannot set-up any agreements longer than that. 4. Impact Qualitative data indicate that the SF programme had a positive impact on student attendance and retention, and on enrolment and attendance rates of girls. There is an assumption that the provision of daily school meals to students helps them to achieve their nutritional needs and that they have the potential to improve attentiveness during classes. An increase of knowledge on safe food preparation and storage was not fully attained because sufficient numbers of local stakeholders were not trained, particularly women (cooks) and SMC members. This was mostly observed during the 2018-19 school year. Nevertheless, WFP’s training of SMC and schools increased awareness about diet diversification. Targets related to the treatment of pupils with deworming medication were only achieved in 2018, with insufficient actions related to this during the 2016-17 and 2017- 18 school years. 4 The analysis of the school carreer of a generation of children in GB shows that 23% of a given age groups does not enter primary school,and that 18% of those who enter, drop out before grade six. Plano Nacional da Educação, Ministry of Education 2017, p.9. 4 | P a g e Both pupils and cooks reported insufficient equipment and utensils in visited schools due to losses or breakage. Follow-up actions by the WFP to tackle this issue were highly recommended by the ET. The WFP played a key role in assisting the MoE and the GoGB to formulate the new National School Feeding Law and to advocate for its promulgation. The GoGB funded HGSF project has exceeded its initial target number of schools. However, the WFP CO has been the principal implementer so far, instead of the DGASCE. Targets to promote local knowledge on management of the SFP were not reached, as insufficient numbers of education staff (only 30.4%) were trained at the regional and local levels. Female staff in particular were trained in much smaller numbers than had been planned. A major unintended effect was the migration of pupils from non-beneficiary schools to assisted ones. The role played by school canteens in several schools’ decision not to adhere to the frequent teacher strikes was also a positive and unexpected event. To a minor extent, certain children who took a school meal were being given a smaller than normal portion at the evening family meal that same day, based on the quantitative results survey. 5. Sustainability The WFP was the main entity responsible for implementing the programme, but has supported the DGASCE on a growth path towards increased capacity and involvement. Advances were made towards establishing a National School Feeding Law, and the HGSF pilot project is well appreciated locally. The Inter-Ministerial School Feeding Committee (CIMCE) is being relaunched by MENES. The DGASCE staff have strengthened their management capacities and will now implement a part of the pilot project in the Biombo region. In most schools, SMC and PA performed an operational and supportive role. Despite this, most of members saw their participation as essential for controlling the quantity and quality of foods delivered to schools and guaranteeing community ownership and programme sustainability. There is currently no sufficient and sustainable funding within the GoGB to implement the SF programme with only national resources. Additionally, political instability is a handicap for progress towards increased government ownership. The DGASCE has not yet created and implemented its own M&E system, quality-control procedures, and modus operandi procedures for the SF programme. Conclusions and Recommendations Overall assessment: The McGovern-Dole FFE programme was well implemented between 2016 and 2019, particularly in relation to the management of the flow of US food commodities from the port of Bissau to the pupils in the 758 assisted primary schools in GB. Other aspects of the programme that need further improvements include training of cooks and of SMC and PA leaders in the management and monitoring of the food at school level, the construction of fuel wood saving stoves and the training of school directors and inspectors in the reporting procedures about the use of the food in the school kitchen. The McGovern-Dole FFE programme had an important positive impact on alleviating short-term hunger of children and their families. It also contributed to the increase of enrolment, attendance, and retention rates, and to the stabilisation of the dropout rates for both boys and girls. Although the project framework includes (SO1) “Improved Quality of Literacy Instruction”, no planned activity was set to achieve this goal (see Annex 1). Therefore, the progress of the project's outcomes related to this objective as a result of USDA assistance could not be evaluated. The (SO2) “Improved health and dietary practices” and the foundational result “Capacity building in the Education Ministry” were both partly achieved. Conclusion 1 (Relevance): The SFP is very relevant to the needs of pupils, families and communities, as it reduces short-term hunger and supplements household food income. Key stakeholders, including at the school level, perceive the THRs for girls from 4th to 6th 5 | P a g e grade with an 80% attendance or more as having encouraged parents to send their daughters to school longer. The strategy to evolve towards a HGSF approach further enhances its relevance for the local communities as it increases local producer incomes. Conclusion 2 (Relevance): Certain relevant indicators are absent from the list of performance and result indicators in the Financing Agreement, and consequently also from the WFP M&E system. These are: (1) retention rates, or alternatively, dropout rates of girls from grade 4th to 6th grade with 80% attendance or higher who receive THRs of 4 kg of dry rice at the end of each month; (2) the frequency of school meals, expressed as the percentage of the school days with lessons, which could show the relative importance of delays in WFP’s deliveries to schools or of organizational problems at the school level (See recommendation 1). Conclusion 3 (Relevance): The WFP SF programme reached most of the poorer, vulnerable pupils of the country because the programme focused on rural areas, and covered about 60% of all existing schools. However, the WFP’s criteria for admission of primary schools into the SF programme actually exclude most of the poorest children livng in the most vulnerable communities, as their unequipped and makeshift community schools do not meet WFP selection criteria, which include a minimum level of infrastructure. Conclusion 4 (Effectiveness): Two different pilot projects are currently being conducted by WFP. Both pilots have reached a size that exceeds what is needed for an experiment. However, no clear trajectory has been defined that would lead to some kind of a decision-making process. With its Japanese funded pilot, the WFP is increasingly rolling out a model that has not explicitly been chosen by the main actors, particularly not by the MoE. This is becoming an untenable situation (See recommendation 4). Conclusion 5 (Effectiveness): The WFP has in some cases accepted proposals from local NGOs it wishes to subcontract, that have insufficient budgets. This implied a risk for insufficient provisions, undermining their work and consequently, their project deliverables. On the other hand, the support-NGOs in the local purchase pilot projects are only offered the perspective to be hired for their services for a series of consecutive shorter periods, instead of for the full duration of the pilot project. This makes planning and management of staff, resources, and means of transport much more complicated (See recommendation 7). Conclusion 6 (Effectiveness and sustainability): WFP has been the dominant actor in the SF programme in GB over the past three to four years, partly thanks to the McGovern-Dole FFE programme. The MoE and its DGASCE, have slowly developed a certain level of capacity, via their collaboration with WFP and their monitoring of operations at the school level. Notwithstanding a formal discourse that reaffirms the ownership of the school canteen programme by the DGASCE, in practice WFP has been responsible for putting in place operating systems, norms and procedures, as well as its monitoring and evaluation system. Efforts to develop and strengthen capacity in the DGASCE have been limited so far, for various reasons (See recommendations 9 and 10). Conclusion 7 (Effectiveness): School meals have generally been regular and timely, of sufficient size and satisfactory quality. In those schools that participate in the local purchase pilot projects, variations in the diet have been appreciated because they broke the monotony of rice-and-beans meals. Conclusion 8 (Gender): The recommendation from the baseline survey (2016) for WFP to conduct a more in-depth study on parents’ motivations for keeping their daughters at home, rather than attending school classes has not been implemented. The end-line evaluation demonstrated that other delicate factors beyond food shortages could be contributing to this outcome, with THRs being considered just one of the factors that could promote a change in parents’ mindsets (See recommendation 5). 6 | P a g e Conclusion 9 (Gender): The poor infrastructure of school toilets was pointed out by school directors, teachers and the PA as one of the reasons for pre-adolescent and adolescent girls to abandon school. Another possible reason was the walking distance to school, as it increases the risk of girls being harassed (See recommendation 6). Recommendations Recommendation 1: Directed to WFP (Priority 1, within the next 6 months): The WFP should, during the current school year of 2019-20, start monitoring the output indicators that refer specifically to girl pupils in grades 4, 5 and 6, even if its contract with USDA for the McGovern-Dole FFE programme will come to an end in April 2020. The cancellation of the monthly THRs of dry rice for the girls’ families offers an opportunity to measure the impact of this practice in GB, by measuring the impact of its sudden absence on attendance, retention and dropout rates. Recommendation 2: Directed to WFP (Priority 2, within 6 months – 1 year): The WFP should try to further integrate its SF programme with local efforts, e.g., UNICEF and INDE, to improve teachers’ skills and provide didactic materials as well as more efficient access to deworming medication for school children. Active partnerships with actors and initiatives that work on improved teaching pilots would add value to reaching this target, as well as appropriate indicators to be closely tracked over time. Recommendation 3: Directed to WFP-GB (Priority 2, within 1 year): The GB CO of WFP should, in relation to the SF programme, continue to adopt a more developmentalist5 approach regarding the SF programme with a focus on sustainability, particularly in relation to the following two evolutions: (a) the shift to an HGSF programme, in which food is purchased from local producers, because it requires a lot of developmental and organisational work on the production side, and to consolidate the link between school and community. This requires specific new skills. (b) the preparation and step by step implementation of the hand-over process of the management of the SF programme to the MoE and its DGASCE requires a specific set of new skills and attitudes, quite different from those required for the effective management of food supply. The pilot project in the Biombo region is a good opportunity for improvements in relation to this. (c) the management of the WFP CO should take the initiative for such a shift in attitudes and skills which may imply a retraining of staff. Recommendation 4: Directed to WFP (Priority 1, within the next 6 months): WFP should assist the DGASCE of the MoE in its reflection about the future HGSF model and modalities, and the process to make a fundamental choice on the basis of the various pilot projects that have been or are being conducted now. WFP could seek funding and offer organisational assistance for: (1) a jointly commissioned external evaluation by qualified consultants to assist in the participatory analysis of the accumulated experience of both WFP and the MoE; (2) an externally facilitated workshop with representation by all categories of stakeholders to determine which model or which modalities are best suited to be rolled out in the local context. Such an evaluation would be an opportunity to consult the communities, and to explore ways to set up a structure in which a more permanent dialogue or consultation between the WFP and the DGASCE along with the various SMCs and PAs can take place. Examples of this include the latter participating in self-evaluations, and a channel being created for an upward flow of complaints, suggestions, opinions, and so forth. Recommendation 5: Directed to WFP (Priority 2, within 1 year): The WFP should jointly organize with the DGASCE a more in-depth study into the motivations of parents to 5 Development work pursues sustainable post-intervention goals. From the very beginning the end of the assistance is envisaged and planned, even if at medium or long term. 7 | P a g e keep their daughters at home, and discontinue their school education. All factors that play a role in girls’ education should be identified and analysed to allow for a more balanced and complementary set of actions. Recommendation 6: Directed to WFP (Priority 1, within the next 6 months): The WFP CO should endeavour to urgently implement improvements in the availability and conditions of the toilet facilities for pupils in the assisted primary schools, in order to avoid the possible drop-out of pre-adolescent/adolescent girls. The WFP should proactively seek forms of collaboration with partners that are active in the field of water and sanitation. Recommendation 7: Directed to WFP (Priority 2, by the next round of contracts). The WFP should review the action and budget proposals of the local NGOs it wishes to subcontract for specific, critical support activities and add a sustainability criterion to their analysis. The NGOs should have enough means and funds to implement their tasks in optimal circumstances, and the sustainability of their outputs and outcomes should not be jeopardized by insufficient budgets. Being too economical can lead to a loss of efficiency in the long run. Recommendation 8: Directed to DGASCE (Priority 1, within the next 3-6 months): In the interest of better SF results for the children, the DGASCE should work with the WFP to generously share its experience and know-how with the CRS, the new McGovern-Dole operator, for the next 3 years. WFP and the DGASCE should negotiate the modalities for the continued existence and use of the toll free complaint and denunciation phone number (106) with CRS so that it can be used by both WFP and CRS beneficiaries, and later by the MoE and its partners. Recommendation 9: Directed to the MoE (Priority 1, within the next 3-6 months): The DGASCE should be more proactive in the process of determining inclusion and exclusion criteria for school/community selection. This should take into account criteria that cover the areas of nutrition status, food security, and education, respectively managed by the WFP (VAM and SISSAN) and by the MoE. The decision concerning the inclusion or exclusion of schools and communities is a political decision based on criteria that reflect the minimally required physical conditions that belongs to the MoE. Recommendation 10: Directed to the DGASCE of the MoE of GB (Priority 2, within 1 year): The DGASCE should start to develop and define its own management, monitoring and evaluation (M&E) systems for the SF programme, probably in a gradual process, and in close dialogue with the WFP, integrating all the good elements from the WFP’s experience and from former (International Programme for Human Development, IPHD) and future McGovern-Dole operators as much as possible. As the Ministry’s own management and M&E systems take shape and used regularly, the MoE should ask every successive McGovern-Dole operator to adopt and follow these national management and M&E systems. 8 | P a g e 1. Introduction The primary aim of the final evaluation of the McGovern-Dole FFE project (FFE-657- 2015/019-00), implemented by the WFP in GB, was to assess the extent to which the project has succeeded in achieving its goals at the end of the contract period. An extension period, lasting until April 2020 was agreed upon between the USDA and the WFP, in order to allow the distribution and regular use of the significant6 remaining stock of food commodities at the time of the nominal closure of the project in August 2019. The objectives of this final, external evaluation were the following: (1) accountability - to account for the activities carried out by WFP and the outputs and outcomes reached; (2) learning - to analyse internal and external factors that have positively or negatively influenced the achievement of the expected outputs and outcomes which should allow main actors to draw lessons for the future. The main users of this report include both internal and external programme stakeholders, such as the WFP CO, the WFP RB and the OEV, the MoE (MENES) and its DGASCE, USDA and Japan. Its main findings and recommendations should allow them to correct, improve or maintain actions and strategies in the GB SF programme. This evaluation covers all activities that were considered in the agreement between the FAS/USDA and the WFP for the period between 2016 and 2019, in eight regions7 of GB. The present extension, until April 2020, has also been considered. According to the Terms of Reference (ToR), the scope of the final evaluation is the entirety of activities covered by the McGovern-Dole funded WFP school feeding project in GB (2016- 2018). The evaluation was carried out with samples from all eight targeted geographic regions. The final evaluation, in line with the completed mid-term evaluation: (1) reviews the project’s relevance, effectiveness, efficiency, impact and sustainability and (2) collects performance indicator data; more specifically it (3) assesses whether or not the project achieved its expected results; (4) identifies lessons learned; (5) assesses project replicability; and (6) whether or not the mid-term evaluation recommendations were implemented. Gender equality and human rights were mainstreamed throughout the evaluation questions and sub-questions in order to capture different perspectives of men, women, boys and girls. The end-line evaluation relies on the Baseline Study and Mid-term results for the baseline data and situational analysis necessary to evaluate the project at interim and at the final stage as well as the mid-term evaluation, which was conducted halfway through project implementation, from January to November 2018. 1.1. Overview of the Evaluation Subject The Financing Agreement FFE-657-205/019-00 established the allocation of USD 20 million over three financial years (2016 - 2018), as shown in Table 1.1. below. Table 1.1 - Overview of the McGovern-Dole FFE-WFP project in Guinea-Bissau (2016-2019) Type of Intervention: Operation Dates Approval date: December 2015. Start date: March 2016. End date: July 2019. Prolonged until April 2020 Amendments Requested by the WFP in August 2017: Coverage extended from 638 to 758 schools. Updated target for number of kitchens and storerooms to be constructed. Addendum in 2019 to continue food distribution activities until April 2020 Duration Three years and four months, but now extended to four years and one month Beneficiary Numbers Planned: FY 2016 145,000 students; FY 2017 160,000 students; FY 2018 173,000 students. During school year 2019-2020: 173,000 pupils. Planned: 638 schools, increased in 2017 to 758 schools Donors MGD-USDA: USD 20 million (FY 2016: USD 6,217,100; FY 2017: USD 6,891,400; and FY 2018: USD 6,891,500) 6 1.103.75 MT of rice, 222.35 MT of beans and 110.86 MT of vegetable oil. Together 1,436.96 MT. 7 These are: Biombo, Cacheu, Oio, Gabú, Bafatá, Tombalí, Quinará, and Bolama-Bijagós. 9 | P a g e Type of Intervention: Operation Main Activities School meals, THR, improving school feeding structures (kitchen and storage) trainings, M&E system, supporting SF pilot Amount Transfered Planned: In-kind food: 1,956 MTs of beans, 9,894 MTs of rice and 737 MTs of vegetable oil. Of these 12,587 MT, only 11,103.69 MT have been received by WFP. USD Requirements Initial: USD 20 million (USD 6,217,100, USD 6,891,400 and USD 6,891,500) Past Evaluations MGD-WFP SFP baseline survey 2016 MGD-WFP Mid-line survey and evaluation 2018 Source: own elaboration, WFP While the financing agreement defined the terms of the implementation and set target values for a list of indicators related to activities, outputs and outcomes, the McGovern-Dole programme’s own logical framework (Annex 1) remained the overarching global reference framework to which the project in GB had to correspond. In an effort to harmonize both frameworks of reference, the Mid-term ET merged them into a single, reconstructed logframe for this McGovern-Dole FFE programme project in GB. The present end-line evaluation has tried to further improve this reference instrument. See the updated version of this reconstructed logframe in Annex 2. The McGovern-Dole FFE programme has two strategic objectives (SO) and one foundational result. SO1 is the improved literacy of school-aged children in beneficiary schools, and SO2 is the increased use of better health and dietary practices of school-aged children (see Annex 1). The foundational results are the increased capacity of government institutions, improved policy and regulatory framework, increased government support, and increased engagement of local organisations and community groups. Three outcomes should lead to the achievement of SO1: (1) Improved quality of literacy instruction; (2) Improved attentiveness; (3) Improved student attendance. The activities of the McGovern-Dole FFE programme supported outcomes 2 and 3 only. No activity was foreseen for improved literacy instruction, the first outcome. This specific domain of quality of instruction in primary schools is indeed outside WFP’s mandate and field of expertise. The MENES, more particularly the DGASCE, has been WFP’s main partner in the implementation of this project. The DGASCE collaborated with WFP in various areas, such as the selection of schools to be included, the monitoring of the implementation process, and the regular internal evaluations. The WFP also collaborated with the Ministry of Agriculture and Rural Development (MARD) and the Ministry of Health (MH). Throughout the implementation period of this project, WFP has been contracting local NGOs because of their specific know-how and knowledge of the local conditions in specific regions of the country. They have supported the creation and strengthening of SMC, the construction and maintenance of school kitchens and firewood saving cooking stoves, the monthly collection of reports on the use of food in schools, and the sensitization of communities to participate in school activities, especially the school canteen. Local NGOs also played a crucial role in the HGSF pilot project that started in 2015, with funding8 from the Government of Guinea-Bissau (GoGB) as well as in the new second pilot project funded by the Japanese Cooperation that started in the 2018-2019 school year. In both pilots, the NGOs mostly support and advise female farming groups in their production efforts. They purchase food products, deliver the products to schools, and ensure, by different mechanisms, the payment of producers. They also work on strengthening the relationship between the local communities and schools. 1.2. Context GB is an LIC with 70% of its population living below the poverty line ($1.90 per day) 9, ranking 177 out of 189 countries on the 2018 Human Development Index10. The country has significant potential in both agriculture and fishing. Despite this, the gross domestic product 8 During the first school year 2015-2016 there was also an important financial contribution by the European Union. 9 WFP. Guinea-Bissau Interim Country Strategic Plan 2018-2019. 10 Human Development Index. Available at: http://hdr.undp.org/en/countries/profiles/GNB. Access 28th Nov. 2019. 10 | P a g e (GDP) grew only 0.4% between 2000 and 2014, underperforming the 1.9% average of Sub￾Saharan African countries during the same period11 . Poverty is more prevalent in the rural areas (75.6%) than in urban Bissau (51.2%)12. Most families pursue self-sufficient farming, but turn increasingly to cashew nut production, the country’s main cash and export crop. Food Security and Nutrition Monitoring System (FSNMS) data from October 2017 show that on average, 20% of rural families were food insecure, with 18% being moderately insecure and 2% being severely insecure13. In November 2016, a crisis year, 26.3% of families were moderately food insecure, and 4.3% were severely food insecure. This resulted in chronic malnutrition among children under the age of 5. A total of 1.4% of all children were found to be with moderate acute malnourishment, and 0.3% with severe acute malnourishment in 2017. The same indicators for the difficult year of 2016 resulted in 6.1% having moderate acute malnourishment, and 4.2% having severe acute malnourishment. According to WHO standards, this was a high degree of malnutrition. In good years, with a normal harvest and a favourable price of cashew nuts, most families have a sufficient level of food consumption, e.g., in October 2017 86.5% of all rural families had an acceptable food consumption score14. Families consume, on average, 2.5 meals per day, with children consuming three meals. During the lean periods however, from July to October, many families struggle to make ends meet and to feed themselves properly. They then start to have one meal per day, mostly in the late afternoon. During those periods, children often go to school without having breakfast and lunch. The education system of the country is in a critical situation15, partly as a result of many years of political instability. Additionally, resources allocated to education only represent 6% of GB’s GDP, which is far below the African average. Almost half of school-aged children are absent from school. This is often due to the scarcity of schools offering a full curriculum, which leads to extensive dropouts. The primary school completion rate is 62%, reflecting delayed enrolment, a 20% repetition rate, and high numbers of dropouts between grades 4 and 5, especially among rural girls16. Roughly half of the population older than 15 years old is illiterate, with large disparities between men (45%) and women (71%). This is because girls face huge difficulties in relation to schooling. Until recently, parents did not attach much importance to school education, particularly for girls. Political instability has reduced economic growth, increased poverty and unemployment (especially among women and youth), constrained household access to food, and limited the availability of health, education and other basic social services. This has greatly affected the economy of the country. Other recent shocks that contribute to poverty and chronic food insecurity in GB include irregular rainfall, the volatility of prices for imported rice and cashew nuts for export, and seasonal floods in certain areas17. Although all forms of discrimination based on sex are prohibited according to articles 24 and 25 of GB’s Constitution, a lack of gender equality persists, and results in an important hunger and poverty gap. Men are dominant in the family hierarchy and in the social sphere. Higher rates of poverty and unemployment among women limit access to food, health and education. Early marriage (17.2 %) and pregnancy (17%) are the most common reasons for girls to drop out of secondary school, despite the legal age of marriage for women in GB being 1818. Work is the most common reason for boys (38.6%). Among women aged 20-49 years, 11 World Bank. 2016. Guinea-Bissau Turning challenges into opportunities for poverty reduction and inclusive growth: systematic country diagnostic (SCD). Available at: http://documents.worldbank.org/curated/en/100721467968248103/pdf/106725-CSD-P155168-IDA￾SecM2016-0127-IFC-SecM2016-0078-MIGA-SecM2016-0076-Box396273B-PUBLIC-disclosed-7-5-16.pdf. Access 28th Nov. 2019. 12 Idem Footnote 5 13 Source: FSNMS Enquête oct. 2017, SISSAN bulletin 14 Idem footnote 7 15 Source: UNESCO, November 2016 – Country Note 26 16 UNESCO, A major overhaul of the Guinea-Bissau education system is well overdue (November 2016). 17 Source: WFP Guinea-Bissau Interim Country Strategic Plan 2018-2019 (January 2019.) 18 UN General Assembly, Report of the Working Group on the Universal Periodic Review: Guinea, 2015, p.20, (accessed June 2020) 11 | P a g e 37% were married before the age of 18, and 7% were married before 1519. Long distances to schools and a lack of proper sanitary facilities are other critical barriers to educational participation, especially for girls. Only 6% of secondary school teachers are female20. There is a higher rate of food insecurity among woman led-households21 (34.7%), than among households headed by men (28%). The Education Sector Plan 2017-2025 (ESP) recognizes food insecurity as a major factor affecting completion rates and it acknowledges the lack of facilities and staff to provide meals. This ESP establishes the guidelines for school canteens. A School Canteen Law, first drafted in 2015, was promulgated in June 2019. The government’s Zero Hunger Strategy of 2018 also includes the school feeding system. WFP implemented two programmes during the past four years. One centred on nutrition support to treat moderate acute malnutrition, to prevent stunting and to assist people living with HIV, and the other was dedicated to school feeding. In 2016, five local NGOs were contracted by WFP to support SF activities in the more distant regions of Cacheu, Tombali and Bolama. They supported the creation of FMC’s, built and maintained stoves and kitchens, improved school infrastructure, and sensitized communities to participate in the SF programme. According to the World Bank22, primary enrolment rates in GB increased from 45% in 2000 to 67% in 2010, but only 64% of children complete primary education. While there is relative gender parity in primary education, the gap increases in secondary education, mostly due to early pregnancy and marriage. The male-female gap widens where health is concerned, due to one of the highest maternal mortality rates (900 deaths per 100,000 births in the world, a high prevalence of HIV (58.6% of adults with HIV are women), the adolescent pregnancy rate estimated at 28%, and the practice of genital mutilation affecting 44.9% of women. Finally, poverty and extreme poverty rates increased between 2002 and 2010, from 3.7% to 11.5%, respectively. This put 69.4% of the population in poverty and 33% in extreme poverty. Both adults and children also face a malnutrition burden with a prevalence of national under-five stunting at 27.6% while 43.8% of women of reproductive health have anaemia23. 1.3. Evaluation Methodology and Limitations: The final evaluation followed the methodology as laid out in the inception report, but with a few minor differences. It consisted of two complementary parts, a quantitative survey in 100 schools, 50 assisted by WFP and 50 not assisted, and a qualitative survey in 30 WFP￾assisted schools. Due to rainy an election seasons, the quantitative survey only took place between November and early December 2019. It was implemented by RESSAN (Rede de Soberania e Segurança Alimentar e Nutricional da Guiné-Bissau). The conclusions of the survey should have been an input for the qualitative survey, but they became available only after the completion of the field work for the qualitative research. However, a mixed methods approach was used to triangulate the information. The analysis started with the review of the qualitative information from interviews and desk review, and it was complemented by the descriptive and inference statistics performed with data from the quantitative survey. Although the evaluation was not a quasi-experiment, the comparison between baseline, mid￾line and end-line surveys allowed for estimating statistical significance among some variables means. A quantitative survey similar to the baseline study and the mid-line evaluation was undertaken. It applied survey instruments designed to collect key project data from schools, 19 Ministério da Economia e Finanças, Direcção Geral do Plano/Instituto Nacional de Estatística (INE). 2014. Inquérito aos Indicadores Múltiplos (MICS5) 2014, Relatório Final. Bissau, Guiné-Bissau: Ministério da Economia e Finanças e Direcção Geral do Plano/Instituto Nacional de Estatística (INE). 20World Bank. 2016. Guinea-Bissau Turning challenges into opportunities for poverty reduction and inclusive growth: systematic country diagnostic (SCD). 21 FSNMS data from December 2016 22 The World Bank. 2017, Guinea Bissau Country Partnership Framework. 23 Global Nutrition Report. 2018. Guinea Bissau Nutrition Profile 12 | P a g e students and local households in the school community. The survey was administered according to the design stipulated during the baseline study. The analysis of the collected data is mainly statistical, to capture key trends (cross tables, simple frequencies, etc.). Quantitative techniques were used with data from project framework, baseline, mid-term and final surveys, including econometric analysis to compare the variation of several parameters between WFP and non-WFP beneficiary schools, and using econometric methods to assess the impact of the programme implementation (using the STATA software). For the quantitative sampling, the Evaluation and Planning Department of the Ministry of Education (DGPASE) and WFP first sampled 50 WFP schools using the probability￾proportional-to-size technique. DGPASE then selected a comparison group of 50 schools that shared similar education and socioeconomic indicators, but which were not supported by WFP. In most cases, the comparison schools sampled were from the same sector as the WFP schools and usually were its nearest neighbor. No other program was giving support to these comparison schools. From each school, enumerators randomly sampled ten students from the Grade 4 enrollment roster who were asked to answer the student-level questionnaire. Enumerators then travelled to these students’ homes to administer the household-level questionnaire. For consistency purposes, the end-line survey was conducted in the same regions of both the baseline and mid-term surveys where WFP was supporting the Government of Guinea-Bissau with SF operations in June 2016: Oio; Bafata; Cacheu; Biombo; Quinara; and Gabu. The qualitative evaluation was conducted between 21 October and 12 November 2019. It used the 15 evaluation questions and sub-questions and the five international evaluation criteria24 presented in the Evaluation Matrix as a reference (see Annex 3). The five evaluation criteria applied were relevance, effectiveness, efficiency, impact and sustainability, selected with the aim to enable the determination of the merit, worth and significance of the project. It consisted of a qualitative non-experimental evaluation that used a mix of various methods: document analysis; semi-structured interviews; visits to schools with observation; and individual and focus group meetings with all actors and stakeholders. The qualitative information was revised using the research questions as a reference and, as explained before, the findings were complemented with the quantitative estimations from the survey. In total, about 60 interviews were held in Bissau and in the various regional capitals that were visited (Annex 4). Altogether, 32 schools were visited in 7 regions (Annex 5, Table 5.1). Bolama-Bijagós had to be left out for practical reasons, as travelling to the islands is somewhat complicated, it would have been too time-consuming25. In turn, the ET visited extra schools, to make up for the missed schools in Bolama-Bijagós. In the end, the number of schools visited exceeded the proposed number of 30. In each school, meetings were held with the directors or vice-directors, teachers, SMC, the PA members (if they could be reached), cooks, beneficiary pupils and girls in grades 4, 5 and 6. Often, during morning visits, no girls between grades 4 to 6 were found because they only had lessons during the afternoon. The visited schools were generally chosen during meetings with regional education directors and regional SF focal points. Practical considerations, like distance, were combined with a search for as wide a variety of school types and school feeding experiences as possible. Gender analysis was done throughout the evaluation process. The factors that affect Gender Equality and Empowerment of Women (GEEW) were evaluated by cross checking primary and secondary information. Data disaggregated by gender and age were used. Specific questions addressing GEEW were included in the topic lists for the interviews with key stakeholders and focus groups (See Annex 3). During the interviews in particular, valuable information was collected on and from both men and women participants in WFP activities, 24 For more detail see: http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm and http://www.alnap.org/what-we-do/evaluation/eha. 25 Particularly in the situation where the ET could only use two cars, which forced it to work with two working teams only, instead of the three consultants working separately, as was initially foreseen in the IR. 13 | P a g e applying mixed method approach. In each school, the ET ensured that all stakeholders (boys, girls, men and women) were met and interviewed. In most schools, group discussions were held separately with cooks, pupils, boys and girls, and girl pupils from grades 4 to 6. Information was systematically triangulated for analysis. For instance, the ET has checked whether the operation’s strategies were based on a sound gender analysis that considered the distinct needs and participation of boys and girls. It also measured whether the operation has made any difference to gender relations at any level and/or if any such change likely to be sustained after the programme is completed. The norms of the WFP Decentralized Evaluation Quality Assurance System (DEQAS) have been applied to this evaluation. Econometría Consultores was responsible for the internal quality control. The reliability and validity of data was ensured through the constant communication with the local firm responsible for the data collection, from which Econometria received periodical reports. Also, all information was digitalised by Econometría’s staff, which allowed for validating the information and running data quality exercises in Excel to identify duplicates, missing values, etc. 1.4. Limitations: The delay of the quantitative survey by RESSAN (finalised in December 2019), in relation to the qualitative assessment carried out by the ET in October 2019 was an important limitation to this evaluation. Survey-data and results were only incorporated into this report at the final stages. For the visits to the schools only two vehicles were made available by the WFP CO instead of the planned three (since WFP had to use the third car for other purposes) and for a shorter period. The ET had to adapt its working plans (two teams instead of three) and reduced the distances between the visited schools, which limited the coverage of the supported regions during school visits. Consequently, it was also not possible to visit Bolama-Bijagos. one of the most vulnerable regions, given the resulting lack of fieldwork time. The ET met key stakeholders and NGOs from that region to address this gap. Interviews were conducted with non-Portuguese speaking stakeholders. This barrier was overcome by using three NGO interpreters in Cacheu and Oio, and various ad-hoc interpreters in the regions of Gabú, Bafatá, Tombali and Quínara. 1.5. Ethics: The ET adopted a careful and thorough approach to the ethics of the evaluation, complying with standard 3.2 of the United Nations Evaluation Group (UNEG) Norms and Standards26. Econometría recognizes direct and indirect beneficiaries as key stakeholders. Hence their participation in the evaluation was not only to answer questions but to receive their feedback during the implementation of the project at this phase. Equally-weighted participation of females and males, both children and adults was considered. Quantitative and qualitative data and survey results were reviewed and cleaned. Quality and reliance on primary data was conducted; for example, during data collection, with parcial information, STATA was used for identifying outliers and each of them were clarified with RESSAN. Both data was triangulated by the ET team to evaluate data reliability and consistency, by comparing primary data with views of governments, civil society, beneficiaries, donors, girls vs boys, women vs men. The team systematically checked the accuracy, consistency and validity of collected data and information and acknowledged any limitations/caveats in drawing conclusions using the data. The evaluation team conducted consent and interviewing procedures with sensitivity to children’s specific needs because the data collection exercises included the direct participation of children. Children’s consent to participate, parental awareness of their children’s participation, and parent’s approval were ensured. Full confidentiality was respected so that any statement could not be attributed to a named individual or traced to its source. The field teams followed a standardized procedure in which the participants were 26 (UNEG, 2016). Norms and Standards for Evaluation 14 | P a g e informed of the scope of the evaluation. They were asked for their approval to participate in the study and to be recorded. 2. Evaluation Findings The evaluation findings and the evidence to substantiate them are presented below, structured per evaluation criterion and as a response to each evaluation question in turn. The evaluation applies international criteria and the corresponding evaluation questions from the evaluation matrix are presented in Table 2.1. below: Table 2.1 - Application of international criteria and the corresponding evaluation questions Criterion Evaluation Questions Relevance ▪ Is the project’s strategy relevant to the beneficiaries’ needs? ▪ Is the project aligned with the national government’s education and school feeding policies and strategies? ▪ Does the project complement other donor-funded and government initiatives? Effectiveness ▪ The extent of project implementation – Did the project carry out all activities as planned? ▪ To what degree have (or have not) the interventions resulted in the expected results and outcomes? In particular, to what extent did THR provide results in increased attendance and enrolment of girl students? ▪ Is short-term hunger reduced? ▪ Did assistance reach the right beneficiaries in the right quantity and quality at the right time? Efficiency ▪ Has the programme been implemented in an efficient way? Was there efficient use of resources and efficient methods of work? Impact ▪ To what degree has the project achieved the results that were foreseen in the project-level framework? ▪ Have there been any unintended outcomes, either positive or negative? ▪ What internal and external factors affect the project’s achievement of intended results? Sustainability ▪ Is the school meals programme sustainable, including a strategy for: sustainability; sound policy; stable funding; quality programme design; institutional arrangements; local production and sourcing; partnership and coordination; and community participation and ownership? ▪ What substantive progress has the government made toward developing a nationally owned school feeding programme? ▪ How are local communities involved in and contributing to school feeding? ▪ What needs remain in order to achieve a full handover and nationally-owned school feeding programme? 2.1. Evaluation Question 1.1: Is the project’s strategy relevant to the beneficiaries’ needs? The beneficiaries of the WFP SF programme in GB are generally pupils from poor communities. The programme focused on rural schools, as rural poverty27 is on average higher than in urban areas. The overall nutritional status of boys and girls in the beneficiary schools is therefore, on average, fairly poor. A study conducted in 2016 in Cacheu and Oio revealed that 42% of all pupils had anaemia, 21.5% had vitamin A deficiency, and 61.1% of all children had one or more indicators of undernutrition28. At the time of the evaluation, interviewed stakeholders reported that, because of the “lean period”, most rural families consumed only one real meal per day, mostly in the late afternoon. Children therefore came hungry29 to school, which reduces their energy level and capacity to be attentive. Evidence shows that chronic hunger increases the chance of a student dropping out while poor nutrition impairs their learning capacity30. Inclusion in the WFP SF programme depended on two other decisive elements that unfortunately resulted in the exclusion of the most vulnerable children living in the poorest communities. These were: (1) a sufficient capacity at the WFP level to supply food commodities to a larger number of schools; (2) infrastructure and organisation standards had 27 Poverty rates, meaning living on less than 2 USD/day, stood at 75,6% in rural Guinea-Bissau and at 51,2% in the capital in 2010. Source: World Bank Systematic Country Diagnostic 2016, p. 16-17 28. Source: E. Saltzman et al. Nutrition status of primary school students in two rural regions of Guinea-Bissau, Food and Nutrition Journal 2017. 29 Information from various school directors during visits to schools, specially in the regions of Cacheu, Oio and Gabu. 30 Bundy et al. Rethink School Feeding. Social Safety Nets, Child Development and Education Sector. The World Bank (2009) http://siteresources.worldbank.org/EDUCATION/Resources/278200-1099079877269/547664- 1099080042112/DID_School_Feeding.pdf Access in November 22 2019 15 | P a g e to be met in the schools that allowed for normal functionality at all school canteen linked operations. Because the WFP only had a few sources of food for its SF programme (mainly the McGovern-Dole and Japanese programmes) during the period 2016-2019, it could not expand much beyond the number of McGovern-Dole programme schools. WFP gave geographical preference to the sectors31 where food insecurity was highest when choosing the 758 schools for its programme in 2016, based on a composite food insecurity index. Within the prioritized sectors, WFP set seven practical criteria for the admission of schools into its SF programme: • at least 50 pupils • the existence of a storage place with acceptable conditions • the existence of a secure source of potable water • the existence of a real school building, not just a self-made shed (barraca) for shadow • the existence of a kitchen, even if rudimentary • community will/manifestation to accept and accommodate the SF Programme • the existence of toilets or latrines Over the past few years however, the rural population in GB has become increasingly aware of the importance of schools for the future of their children. Before, in many parts of the country, that was not the case. Instead of waiting for external initiatives, most villages have started their own local community schools, securing the most educated young people in the village as informal teachers, with each family contributing monthly to these teachers’ salaries. These community schools cannot comply with the WFP criteria for admission in the SF programme, despite high levels of food insecurity. Therefore, the WFP’s admission criteria create a certain exclusion32 bias. According to the WFP CO, there is a need to safeguard the children from unsafe meals by ensuring good storage, acceptable hygiene conditions of kitchens and acceptable school infrastructure for teaching and learning. In order to address this, WFP has been handing out building materials to allow some community schools to build a kitchen that complies with the criteria. Also, the WFP has been shifting from urban schools to rural ones with local food production in an effort to reach more of the poorest children. Other actors, like PLAN International, have been regularly constructing school buildings and toilet blocks for these new community schools, and UNICEF has been implementing a water programme for schools. Organizations have trained and upgraded the local community’s teachers so that they can qualify to be given a teaching contract by the MNEJCD. With external support, community schools can then be converted into public schools, and also, qualify for possible future admission to the WFP SF programme. Pupils in general were found to appreciate school meals, though some mentioned the monotony33 in the school meal diet, which is a combination of rice, beans, and oil, as these are the only commodities available for the programme. Over the past four years however, WFP has, first with financial support from the European Union (EU) and the GoGB, and then with funding from Japan, set up two pilot projects to try out different modalities of local food purchase for school canteens. These pilot projects are touching an increasing number of schools (respectively 100 and 214 in the 2018-2019 school year) and are providing a more diversified school meal, including tubers (e.g. cassava, sweet potatoes and yams), local beans, and occasionally vegetables. 31 A sector is an administrative subdivision of a Region. 32 The baseline survey found that on average non-WFP assisted primary schools have more children from the poorest quintiles as pupils than the schools that receive WFP school feeding support: 48 % against 37% (Baseline report. 2016, p.19) 33 This was also one of the findings of the mid-term evaluation of the programme in 2018. 16 | P a g e In order to encourage parents to keep their daughters in school, at least until the completion of the basic school and beyond, the McGovern-Dole FFE programme provided 4 kgs of rice per month of THRs 34 to every girl from 4th to 6th grade with an 80% attendance or higher, from 2016 to 201935. According to interviewees, these very popular rations have achieved their objective, with girls staying proportionally longer in primary school. Although such perceptions were stated by all stakeholders, it could not be proven by quantitative data given that no specific monitoring was done, as no indicators had been previously set. A slight increase in retention rate was noticed for girls (from 95.8% to 96%) but not for boys. However, this figure is for all girl pupils, not just for those in grades 4 to 6 (See Section 2.6, p.21.). Table 2.2 presents the percentage of boys and girls that received THRs by region and by quintile of income. A total of 82% of girls and 24% of boys benefited from these rations in 2019. Though the THRs were not specifically meant to be handed out to boys, this seems to have happened more or less systematically in some regions (Gabu, Bafatá, Quinara). The degree of poverty of the beneficiary family does not seem to have been the main reason for this exception. Table 2.2 - Take-home rations for boys and girls by region and income quintile (2019) 36 Source: Evaluation survey Teachers, headmasters, cooks, inspectors and SMC members have been trained in large numbers, though the numbers of trained cooks and SMC members were significantly less37 than was anticipated, in the management and handling of school meals, including daily and monthly reporting. At the local level, people are satisfied with the SF programme. In several cases, the existing food stocks in the school warehouses have been used by the community as a successful argument to convince teachers not to join their unions’ appeals to strike so that children could continue to receive school meals, particularly during the 2016-17 and 2018-19 school years. The pilot projects have been purchasing food locally, from farmers, mostly women’s groups, and the communities were very satisfied in general with this new opportunity to sell part of their harvest locally, as most rural areas suffer from a structural lack of access to markets. There are, however, complaints about the duration of the payment procedures in one pilot. Despite actions taken by WFP and local NGOs to reduce those delays, there were still complaints from local producers regarding this issue. The school meal programme is thus very relevant, and it is very appreciated by all local stakeholders, i.e., the children themselves, their parents, the local communities, the school staff and head masters. The strategy to evolve towards the local purchase of food for the school 34 Locally known as «ração seca para levar à casa» 35 These THR had been recommended by the Baseline Study of 2016. These have however now been stopped during the current school year 2019-20. The McGovern-Dole FFE programme came to an end in August 2019, and the remaining food stock in WFP’s warehouses is now being distributed to all 758 beneficiary schools, following an addendum to the contract between USDA and WFP. The stock is however insufficient to cover the whole school year 2019-20, therefore, in an effort to supply meals to pupils as long as possible, it was decided to stop the provision of the take-home rations as from October 2019 onwards. 36 Table 2.1 represents the percentage of WFP students receiving take home rations of rice at some point during the school year, according to gender, region, and wealth quintiles). 37 See also p.21, Activity 3 Region Male THR Female THR yes no Missing yes no Missing Bafata 29.4% 58.8% 11.8% 85.3% 8.8% 5.9% Biombo 0% 100% 0% 100% 0% 0% Bolama -Bijasos 0% 100% 0% 30% 70% 0% Cacheu 2.2% 97.8% 0% 91.3% 6.5% 2.2% Gabu 96.7% 0% 3.3% 100% 0% 0% Oio 3.4% 96.6% 0% 93.3% 6.7% 0% Quimara 28.6% 71.4% 0% 42.9% 57.1% 0% Tombali 0% 100% 0% 20% 80% 0% Total 23.2% 73.2% 3.6% 81% 17.1% 2% Wealth Male THR Female THR yes no Missing yes no Missing Poorest 2.4% 95.2% 2.4% 77.8% 17.8% 4.4% Second 16.7% 81.3% 2.1% 75.5% 20.4% 4.1% Middle 31.4% 60% 8.6% 81% 19% 0% Fourth 50% 43.3% 6.7% 91.9% 8.1% 0% Richest 27% 73% 0% 80.6% 19.4% 0% Total 23.4% 72.9% 3.6% 81% 17.2% 2% 17 | P a g e kitchen is a new element that considerably increases the relevance of the programme for the local communities. Key findings and conclusions – Sub-question 1.1. • The SF programme is very relevant to the needs of pupils, their families, and local communities, as it alleviates short-term hunger and supplements household food income. • The THRs of rice distributed to girl pupils from 4th to 6th grade with an attendance of 80% or higher are considered by most stakeholders to have contributed to more parents sending their daughters to school for longer, instead of keeping them at home for domestic work or for early marriage. Survey data show that 82% of girls reported to have received THRs in 2018-19. • The strategy, supported by both WFP and GoGB to gradually evolve towards an HGSF programme increases the relevance of the programme for the local communities. Over time, the change from imported food aid to locally produced foods will have a positive effect on the local economy, reducing the poverty which is partly at the root of the need for school meals. • The McGovern-Dole programme has been the core element of the WFP SF programme in GB during the period between 2016 and 2019. 2.2. Evaluation Question 1.2: Is the project aligned with the national government’s education and school feeding policies and strategies? The McGovern-Dole FFE programme has been a very valuable central component in the national SF programme, which is part of the GoGB’s strategy to improve the quality of its education system. However, the newly promulgated School Feeding Law, anticipates a different approach for the future. This law envisages a HGSF model based on local food. There is a discrepancy between the older model, based on the distribution of imported US food products, and the MoE’s new vision to offer a more balanced and more diversified school meal based on local food. Since 201538, the WFP together with the MNEJCD have been conducting several pilot projects in the poorer regions (Cacheu and Oio) that involved various modalities of local purchases of food for the school canteens. Because various difficulties were encountered over time, the operational models have been evolving, incorporating lessons learned. At present, two models exist side by side. One pilot project that is funded by the GoGB envisages the strengthening of farmers’ capacities to produce sufficient food supplies for the school canteens, and is based on payment to farmers through a mobile money39 system. The second, more recent pilot, originated from the change of part of the value of a Japanese donation of canned mackerel40 fish for school feeding. This turned into an experiment to purchase local food, mainly tubers (cassava, sweet potatoes, yam (Dioscorea spp) and beans (Vigna spp), in order to diversify and provide a more balanced school meal. Both models use local NGOs for various mediation functions, including advice and support to producers and their emerging organizations, as well as assistance to communities in order to consolidate the link between schools and local communities. WFP-GB has made, on its own initiative, certain efforts to coordinate with actors from the education sector to improve the quality of the education system in GB. It participated in various coordination activities, including the Local Education Group and the framework of the Essential Learning Package approach created by UNICEF. Key findings and conclusions – Sub-question 1.2 38 The very first pilot project on local purchases was funded by both the EU and the Government of Guinea-Bissau. 39 The MTN mobile money system is used. 40 Locally referred to as «sardinha», notwithstanding the clear labels on the tins. 18 | P a g e • The McGovern-Dole FFE programme (2016-2019) was the most important component of the national SF programme in GB, which is a very important part of the national education policy. School canteens are an essential component of the national efforts to upgrade the quality of the national education system. • The vision of both the MENES and of the WFP is to evolve towards HGSF models that are based on locally produced foods. As such, there is a certain discrepancy between the older, import￾based model of the McGovern-Dole FFE programme, and the newer models that are pursued by the MENES and the WFP for the immediate future. 2.3. Evaluation Question 1.3: Does the project complement other donor-funded and government initiatives? The McGovern-Dole FFE programme foresees the supply and distribution of US rice and beans, complemented with vegetable oil and salt. Operational costs for the distribution and the management of the programme are foreseen, but remain limited. Since 2016, the McGovern-Dole FFE programme has been complemented with pilot projects that experiment with different modalities of local food purchase for school canteens. These projects were initially funded by the GoGB and the EU in the Bafatá region, and later, solely by the GoGB in the regions of Cacheu and Oio. During the 2019-20 school year, this pilot project is to be extended from 60 to 100 schools and communities. In practice, schools participating in this pilot project remain part of the McGovern-Dole group of beneficiary schools, but now receive only half of the original quantity of US rice and beans, and the other half in the form of locally purchased tubers and beans. This allows for a much-appreciated variation in the composition of the daily school meals. Over the past few years, the McGovern-Dole FFE’s contribution to the SF programme has also been complemented by a multi-annual donation of canned mackerel fish from Japan. This proved to be an appreciated extra food item in the otherwise monotonous rice and bean meals supplied by the US. WFP agreed with the Japanese Cooperation that part of this in-kind food support be converted into funds to buy local food, in order to strengthen the local economy and support local communities with this new source of income. This second pilot project on local purchases has been benefiting 214 primary schools in six regions during the 2018-19 school year. These new developments are clearly complementary to the much more important McGovern-Dole programme activities. Key findings and conclusions – Question 1.3 • There is a clear complementarity between the McGovern-Dole FFE programme component of the national SFP, and the in-kind donation of canned mackerel fish from Japan that covers part of the country’s sourcing needs. This canned fish has been used as an extra ingredient in the rice￾with-beans meals of the McGovern-Dole FFE programme. • Recent pilot projects that seek to test out modalities of local food purchases, one of which is funded by the GoGB and the other by Japan, have been «grafted» on the existing McGovern-Dole FFE programme component of the school meal programme. Beneficiary schools are part of the McGovern-Dole FFE programme (n=758), but in the pilot project they currently receive only half of their original consignments of US food. The other half is local food, mostly tubers and dry beans. They have thus two complementary supply lines. • The resulting variation in meal composition and taste, is generally very well appreciated by the pupils. The local communities are particularly happy with this new opportunity to sell part of their harvest locally, for the benefit of their children, and allowing them to earn additional income in a context where the difficulties of access to markets are a major problem for rural producers. 19 | P a g e 2.4. Evaluation Question 2.1. The extent of project implementation – Did the project carry out all activities as planned? (Effectiveness) The project implementation focused predominantly on the: (1) management of the flow of imported US food products, from the Port of Bissau to the beneficiary schools; (2) preparation of the food into daily school meals, and their consumption by the beneficiary pupils; (3) provision of THRs of dry rice to girls from 4th to 6th grade with an attendance of 80% or higher; (4) accompanying capacity building, training, provision of kitchens, fuel efficient stoves, storage capacity, kitchen equipment, eating utensils and deworming medicine; and (5) establishment of an efficient and functional monitoring and evaluation system for the SF activities and their outputs. The performance of the WFP in these various interlinked areas is reflected in Table A7.141 (see Annex 7), which presents most indicators mentioned in the Project Agreement between the FAS/USDA and WFP. Target values refer to US fiscal years, while WFP achieved values for each of these indicators refer generally to school years42. However, this difference is not an obstacle to the interpretation of these figures. The analysis of these indicators shows that the activities linked to food reception, storage, distribution and its use (preparation, consumption, THR) were generally well performed. On the other hand, some accompanying activities were performed to a much lesser degree, at times even insufficiently. In the following analysis of activity performance, the sequence and numbering of the activities of Table A7.1 (Annex 7) are used. Activity 1: Provide school meals Number of school meals provided: Between 2016 and 2019, the WFP served a total of around 63,681,970 school meals to pupils in the beneficiary schools (See Annex 7). It is expected that the remaining stock will allow for a further 11,037,555 school meals, so that in total about 74,719,470 will have been served by the end of April 2020. This compares favourably to the total quantity of food received by WFP, which was 11,103.6943 MT. Calculated at 160 gr/meal and rounded off this corresponds to about 69,398,063 school meals. The total number of meals that will have been served is expected to be 7.67% more than the readjusted target. The reasons for this excess number are not directly clear, but this difference is probably acceptable. Numbers of school-aged children (boys and girls) receiving daily school meals: The programme has reached more pupils every year, both boys and girls, than was originally planned (119.4% in 2016-17; 108.5% in 2017-18; and 100.3% in 2018-19). This is essentially because the schools had more pupils enrolled each year. In the 2016-17 school year the numbers of boys and girls were equal. In the 2017-18 and 2018-19 school years there were about 12% more boys than girls that received school meals. Numbers of students (boys and girls) regularly attending (80%) USDA supported schools: These figures, always somewhat lower than the number of children that received a daily meal, remained stable during the first two school years. However, they dropped by 12.6% in the third school year (2018-19). This reduction was less significant for girls (only 7%). No explanation could be found for this reduction during the last school year. These figures are also being used to measure the attendance rate, see the discussion on page 28, paragraph 103. Activity 2: Provide take-home rations Number of girls from 4th to 6th grade receiving THRs: The average target numbers of beneficiary girls were reached every school year (Table 2.3). For the additional 2019-20 41 This table is an updated and condensed version of three tables from Annex 7 of the mid-term evaluation report: table A7.1, (targets), A7.2 (results 1st school year) and A7.3 (results 2nd school year). See pages 62-72 of the mid-term evaluation report. 42 School years in Guinea-Bissau start in October (private schools already in September) and finish at the end of June. A school year is 165 teaching days. 43 Source: calculation by WFP’s supply chain staff, 11 Nov. 2019. 20 | P a g e school year however, the THRs have been cancelled44, because, at the end of the three-year project, WFP had an insufficient stock of food available to cover another whole school year. This stock will thus, in agreement with the USDA, be distributed to all 758 schools only for meal preparation. In order to be able to supply school meals to all schools for as long as possible, the portion of rice per meal has been reduced45 this school year (2019-20), and the THRs of rice had to be stopped altogether. Even then, the stock will run out in April 2020, leaving the last months of the school year without meals. Table 2.3 - Number of girls from 4th to 6th grade receiving THRs by year Year 2016 2017 2018 Target numbers of beneficiaries 15,414 16,230 16,623 Numbers of beneficiaries reached 18,087 16,323 16,484 Rate of completion 108.8% 100.57% 99.16% Source: own elaboration based on Table 7.1, Annex 7 Number of take-home rations provided: The ratio between WFP’s numbers of THRs provided and the number of girls having received them every year is exactly 110, which is precisely 2/3 of 165, the official number of days with lessons in a school year (See Activity 2 in Table 7.1, annex 7, p.68). This may suggest a calculated reconstruction of data was used here, rather than field monitoring. During its visits to schools, the ET came across a certain degree of variability in the amount of rice actually given to the girls to take home in some of the visited schools. Variation46 was between 2.5 kg/trimester and 14kg/month. This is possibly because some schools calculated the ration47 in function of the days on which effectively lessons were given, leaving out, e.g., strike days. In Gabu and, to a lesser extent, also in Quinara and Bafatá, boys have also received regular THRs of dry rice (see Table 2.2. p.8). This finding from the 2019 survey is not reflected in any of the WFP reports. Most schools however have practised a regular distribution of the correct quantity of 4 kg of rice per girl pupil in grades 4, 5 and 6, per month. Activity 3: Number of SMC and PA members as well as cooks trained in food storage and preparation: The global number of individuals (both men and women) from these categories that have been trained during the three school years (2016-17, 2017-18 and 2018-19) was below what was planned (4,802 real; 6,650 planned, see p.70). This is insufficient and may have indirectly been contributing to the failure of the firewood saving cooking stoves (Activity 5), as insufficient numbers of local SMC and PA members and cooks were trained in their use. Activity 4: Number of teachers, directors and inspectors trained on food management and storage practices: Over the past three years, WFP was able to to reach this target. The number of trained male and female teachers, directors and inspectors over the whole period exceeded the initial targets (Male: 425 vs. 807, 189.8% of initial target; Female: 425 vs. 603, 141,8% of initial target) Although targets were achieved and sometimes exceeded, the ET during school visits observed difficulties faced by headmasters when managing two different systems of food distribution at the same time. The portions that should be served per pupil with McGovern Dole food were generally very clear but this was not so for the quantities of local food per school meal that should be distributed. Making it even more complicated was the calculation when cassava, brought fresh to the school, had to be dried by the cooks and the teachers to 44 School year 2019-2020 is beyond the end of the 3-yr project. Because of frequent strikes during the past school years, there was still an important leftover stock of food in July 2019. This leftover stock allows to ensure the continuation of the SF programme during this extra 4th schoolyear 2019-2020, while the new USDA MGD contractee CRS prepares itself for the task. This continuation will however have to be with a reduced food supply to the schools: the rations of rice are reduced and the THRs are cancelled for this special transition year. 45 Reduction to 100 g of rice per meal instead of 120 gr before. The quantities of beans and oil remained unchanged. 46 Some examples of the apparent variability of the size of the THR: EB Gambasse, Bafatá: 2,4 kg of rice/girl/last trimester of 2018-2019 (source: headmaster), other information from one girl: «2kg of rice, then once 5kg, often 3kg/month»; EBU Algodão, Gabú: 4kg rice once or twice a year per girl; EB Madina Sara: last year, the THR was 10-14 kg of rice/month. 47 At the foreseen rate of 208 gr of rice per day 21 | P a g e avoid its rapid deterioration. This was seen as a limitation of the project in guaranteeing a more diversified meal, since there were various occasions in which the school meals distributed to children consisted of only local beans or of only rice. Activity 5: Number of kitchens and storerooms constructed and/or rehabilitated: The WFP has been providing cement, nails and galvanized corrugated iron roofing sheets to SMCs to allow them to construct the school kitchens themselves, based on a WFP design. In total, 450 school kitchens have been built this way, mostly in correct fashion. There were only 400 planned for. In a few cases, however, the kitchen was rather low and smoky, due to insufficient ventilation. Number of fuel-efficient stoves provided and restored: A local NGO contracted by the WFP, built the 400 fuel-efficient stoves48 that were foreseen, in 400 school kitchens. However, the ET noted that in the vast majority of school kitchens visited, these fuel saving stoves were not being used49. See more information under Efficiency (Section 2.8) for the full analysis on this finding. Activity 6: Provide storage and food preparation equipment, tools and eating utensils: The WFP equipped all of its 758 schools with the necessary quality storage capacity and provided the necessary kitchen equipment, tools and eating utensils for all schools. However, in the course of the three years of project implementation, a certain number of items have either broken down or disappeared. In several kitchens visited, there was only one big cooking pot left of the initial two. Many pupils were not provided a plate and a spoon to eat their meal. They had to systematically bring a bowl and a spoon from home. Many were observed eating with their unwashed hands, mostly in their classrooms, but on some occasions on the ground in the school. Activity 7: Distribute deworming medication Number of pupils having received deworming medication: The number of pupils that received mebendazole tablets was insufficient during the first and second school years, when UNICEF oversaw the medicine supply (only 63.6% and 62.3% of the target). The WFP’s role was limited to distributing the medication to the beneficiary schools. During the third school year, the Ministry of Health organized a general deworming campaign in several regions, administering mebendazole to all children in the villages, and not only via schools. However, some areas as well as some of the visited schools were not touched by this «national» campaign, because the supply was insufficient to cover the whole country at once. Activity 8: Train government staff on the management of an SF programme Numbers of government staff trained at national, regional and local levels on this type of management, and number of refresher courses organized at the regional level to follow up on the implementation of the action plan: Over the past three school years, WFP trained more MENES staff at the national level than foreseen (25 people instead of 19). However, it trained only 30% of the anticipated number of staff at the Regional Directorates of Education. This is a serious shortfall. According to the WFP CO, this activity should have been conducted by DGASCE. On the other hand, WFP organized 10 refresher courses during the last school year, 2018-19, which is much more than the 3 courses that were planned. WFP also turned to training local inspectors in year three, for them to be able to train new incoming school directors. Activity 9: Establish a monitoring and evaluation system, and train Ministry of Education staff in its use 48 Palmeirinha itself provided slightly different numbers: it said to have built 174 stoves the first year, instead of the required 150, 150 the second year, and only 87 the third year. For the remaining 13 stoves it still has to finish in Biombo the NGO complains of lack of collaboration by the local school management committees that show considerable resistance to gather the required building materials (termite hill clay, cow dung, and rice straw). 49 The lack of budget for follow-up visits by NGO Palmeirinha after the dried stove had been fired for the first time, negatively impacted the use of the improved tool by the cooks. This has reduced the potential benefits related to these fuel saving stoves. 22 | P a g e A monitoring and evaluation (M&E) system established and functioning: WFP used its own M&E system, and trained MoE staff at various levels (central, regional, sector, school) in its use. Information from the schools was gathered, collected, verified and transmitted to the WFP by different MENES staff: headmasters; teachers; inspectors; school feeding focal points; and DGASCE-staff. It might have been better to have assisted the Ministry to develop its own M&E system, and to train its staff at the various levels in its use, in the perspective of a gradual handover of the management of the SFP to the MENES. Most probably, MoE's DGASCE was, in 2016, too weak in terms of staff and systems to be able to propose its own M&E system as an alternative to the fully fledged WFP M&E system. Ideally, the WFP’s system and the MoE’s system could coincide and overlap a great deal, so that duplication of efforts in monitoring and reporting is avoided. Number of government staff trained in M&E system operations: The WFP trained more government staff at the central level than was anticipated. Thirty-two people were trained against the nineteen forecasted to be trained, as well as directors and inspectors at regional levels. The WFP trained inspectors for them to be able to train new school directors, as these directors are frequently reappointed to other schools. Because inspectors are seen as technical staff, they are much more stable in politically volatile situations. Besides the formal training that is included in Table 7.1 (Annex 7, p.70), there is also the more informal, on the job training of regional education directorate staff during frequent monitoring visits conducted by the WFP field monitoring assistants to schools. These visits, carried out to correct mistakes in the monthly reports and explain how the reporting forms should be filled out, are always performed together with the local education inspectors as a learning experience. Number of timely school feeding reports produced by the school directors: The WFP’s M&E data show that a consistent half or more of all reports were produced in a timely fashion. This could be related, among other things, to the insufficient number of training (see Activity 4), the high turnover of the MoE’s staff, literacy levels of school directors and teachers responsible for filling out these reports, and a lack of vehicles for inspectors to collect the reports in schools. Another issue was related to the quality of these reports, which was not included as an indicator and therefore was not systematically measured. Activity 10. Support government to develop a school feeding pilot project: The WFP has supported the school canteen unit DGASCE of the MENES with the development of an HGSF pilot project. The GoGB has, on two occasions50, made important financial transfers to the WFP to conduct a pilot project in Cacheu and Oio to experiment with modalities for the local purchase of locally produced food products with payment to the producers via the mobile money system, MTN. This pilot was conducted entirely by the WFP, with DGASCE participation limited to the selection of schools and the monitoring of activities. Only during the 2019-2020 school year has the process made an important step forward. The DGASCE will start to actively manage a new extension of this pilot project in Biombo. This will involve the handling of funds and the control over the activities of the local intermediary NGO. Therefore, this activity corresponds fully to what was expected, though with some delay. The WFP also manages another HGSF pilot project, which is an extension of the Japanese donation of canned mackerel fish to the SF programme. Part of the value of this contract is now being used to experiment with the local purchase of food with local NGOs as intermediaries, without the use of mobile money technology. The school canteen unit of the MENES is much less involved in this second pilot project. Numbers of schools covered by the GoGB pilot project: The target number of schools covered by the GoGB pilot was 14. In reality, this pilot project has reached 60 schools in the 2018-19 school year, and a 100 schools are predicited to be reached in the 2019-20 school year. This number greatly exceeds the appropriate size of a pilot experience, as both 50 200.000 USD in 2017 and 264.000 USD in 2018. This information was provided orally to the evaluation team by the WFP CO team during one of the meetings held on 22 October 2019. 23 | P a g e the WFP and the MNEJCD seem to have limited capacity to ensure a close follow-up and monitoring of the pilot. The WFP also gave significant technical and analytic assistance to MENES and to the GoGB in the areas of policy formulation, development of a legal framework, and management of a national school meal programme. The WFP promoted the trilateral cooperation between the WFP, Brazil and GB through the WFP Centre of Excellence Against Hunger and the Brazilian Cooperation Agency (ABC). The former contributed significantly to the drafting of the new school feeding law. The WFP has been very active in explaining the proposed law to other line ministries, including the Council of Ministers, prior to its approval early in 2019. Key findings and conclusions – Question 2.1. • All activities that are directly linked to the supply, storage and distribution of food to schools and its use (preparation, consumption, THRs) were generally well performed. As a result, throughout the whole project period, the planned number of pupils, both boys and girls, from grades 1 to 6, have been served the agreed upon number of daily hot and nutritious meals. Girls from 4th to 6th grade with an attendance of 80% or higher, have been provided with the correct monthly THRs of rice for their families. • Delays in the delivery of food to schools have occurred, often at the beginning of the school year, affecting particularly private schools that tend to start their lessons earlier than the public schools. These delays were limited to some days or weeks. • Some of the accompanying activities have been performed to a lesser degree, which may have impacted the project’s results. Examples of this low performance include the training of MENES staff at the regional level on the management of an SF programme, and the delayed distribution of deworming medication. • The WFP has involved the MENES’ DGASCE in the monitoring and also increasingly in the management of the HGSF pilot project that it has been implementing on behalf of the GoGB for the last two years. The DGASCE will manage part of this pilot project in the region of Biombo during the 2019-20 school year. • The WFP has implemented a second pilot project on the local purchase of food, based on Japanese funding, in which the DGASCE is much less involved. • The WFP gave significant technical assistance to MENES and to other line ministries, and mobilized south-south cooperation via its Centre of Excellence against Hunger in Brazil. This was to support processes like the drafting of the new school feeding law, its discussion and approval by the National Assembly and the Council of Ministers, and its ratification as a new law. 2.5. Evaluation Question 2.2: To what degree have the interventions resulted, or not, in the expected results and outcomes? In particular, to what extent did providing THRs result in increased attendance and enrolment of girl pupils? (Effectiveness) This EQ entirely overlaps EQ 4.1. under the chapter of Impact. Therefore, in this section, the discussion is limited to the second part of the EQ. The remaining results and outcomes are analysed in-depth under EQ 4.1. (p. 25-29). The most important output indicators among those set for the targets that were in the Project Agreement FFE-657-2015/019-00 are the increased enrolment and attendance rates of pupils, both boys and girls, and the corresponding retention rates and drop-out rates, particularly for girls. Nevertheless, the specific drop-out rate of girls from 4th to 6th grade was not considered as an indicator, and no target value was set for it, in spite of being one of the priority activities (the handing out of THRs of rice). This indicator has not been monitored s by either WFP’s M&E system or by the MoE’s monitoring system. However, many informants at various levels (MENES, regional, local schools) were convinced that THRs effectively encouraged parents to keep their daughters in school longer. In some cases, girls from 6th 24 | P a g e grade stated that their mothers wished them to study, regardless, as they themselves had not had that opportunity when they were a child or a young teenager. Key findings and conclusions – Question 2.2. • Both the WFP’s and the MoE’s M&E systems do not directly monitor the outcome indicators for the specific subgroup of girls in grades 4 to 6 separately. This was not an indicator. Hence, there is no specific data for this subgroup, and the second part of evaluation question 2.2. cannot be answered with support of quantitative data. However, data collected during key informant interviews, showed that generally people believe THRs resulted in increased enrolment and attendance rates of girls (4th to 6th grades), and also in an increased retention rate. 2.6. Evaluation Question 2.3. Is short term hunger reduced? (Effectiveness) Short-term hunger is often experienced by most pupils in rural primary schools because the main or only family meal is served mostly late in the afternoon. In the lean season, as was the case at the time of this evaluation, with food stocks depleted at home and the new harvest still growing and maturing in the fields, children often come to school hungry, both those attending the morning and afternoon lessons51 . Most kitchens in beneficiary schools have been cooking two meals per day in sufficient quantities to serve each shift of pupils with freshly prepared food. In this way, the morning shift pupils would generally eat a meal during their 10:30am recreation break, and the pupils who attend class in the afternoon ate a meal on average somewhere around 1 or 2pm, in principle, just before entering classes. According to interviews conducted with students, school directors, teachers, PAs, etc., the provision of these meals alleviated short term hunger among beneficiary pupils. Based on the interviews with girl pupils that were beneficiaries of THRs of rice between 2016 and 2019, the ET tends to conclude that parents generally appreciated this welcome contribution to the household economy, where it is very difficult to make ends meet and to feed all family members sufficiently most of the time. Key findings and conclusions – Question 2.3. • The daily school meals have been an appreciated short-term nutritional comfort for all pupils, because many of them used to come to school without having had breakfast or lunch at home. • The THRs have been an appreciated contribution to the household economies of the families of the beneficiary girl pupils, where being able to feed all family members is a constant challenge most of the time. 2.7. Evaluation Question 2.4. Did assistance reach the right beneficiaries in the right quantity and quality at the right time? (Effectiveness) 1. The right beneficiaries? The WFP’s SF programme, through the McGovern-Dole FFE food commodities supply, has reached a great majority of poor primary school pupils in the country. Irregularities and diversions to unplanned beneficiaries remained low. The programme supported about 60% of all schools and focused on rural areas, where poverty is worse than in urban or peri-urban settings. Contrary to the previous McGovern-Dole operator in 2012-2015, the IPHD, the WFP left out all schools in the urban area of Bissau (86 schools), in an effort to prioritize the poorer groups of pupils. The 'right beneficiaries' have thus been receiving SF, but a whole cohort of poor primary school children were not reached, as their community schools did not meet WFP’s technical selection criteria. 51 Pupils have lessons only half of the day, either in the morning (mostly the younger children of the lower classes) or in the afternoon (4th to 6th grade mainly). 25 | P a g e 2. In the right quantity and quality? Between 2016 and 2019, all beneficiary schools regularly received the required quantities of rice, beans and vegetable oil for each trimester (See Table 8.1, Annex 8). The first supply of each new school year was often insufficient because it could only be calculated by the previous year’s number of pupils. That was then adjusted for the next deliveries. As a rule, there were three deliveries per year, one per trimester. During teacher strikes, the deliveries were delayed, the quantities reduced or both. Only occasionally have school directors mentioned problems related to the quality of the food supply, such as insect infestation. Pupils generally liked the school meals a lot, and never mentioned any quality problem. They only regretted the absence of a tasty sauce, like their mothers make at home from local ingredients such as badjiqui 52, onions and other ingredients. Because of this, there is a frequent use of Maggi cubes in the preparation of meals by the cooks, or the cubes are directly crumbled by the students onto the food, despite these products containing excessive amounts of salt and monosodium glutamate. Table 2.4 presents students’ perceptions of the quantity and quality of meals provided by the SF programme in 2019. Overall, 9% and 20% of girls vs. 14% and 21% of boys, found the quantity insufficient or acceptable, respectively. As for the quality, only 2% and 19% of girls found it bad or reasonable vs. 3% and 17% of boys, respectively. These figures demonstrate that most students appreciated the quality of meals. Table 2.4 - Perceptions of students (boys and girls) at WFP-supported schools who reported on quantity and quality of the meals provided (2019) Quantity of Meal Quality of Meal Insufficient Acceptable Enough Very good A lot Bad Reasonable Good Very Good Excellent Female 9% 20% 52% 19% 0% 2% 19% 58% 21% 1% Male 14% 21% 44% 21% 1% 3% 17% 56% 24% 0% Total 12% 20% 48% 20% 0% 3% 18% 57% 22% 0% Source: Evaluation survey 3. At the right time? Limited delays in the delivery of food commodities to the schools have been reported by some of the school directors, often during the first deliveries of each new school year. This was mostly the case in private schools that start lessons weeks earlier than government schools. Overall, beneficiary schools have received the required supplies of rice, beans and vegetable oil for each trimester on time. Also, the supplies from local food purchases have normally been delivered on time. 4. Take-home rations The THRs of rice have been given out regularly to girls from 4th to 6th grade, with some minor exceptions. The THRs have been cancelled during the 2019-20 school year because the WFP will be using the remaining stock of US food commodities at the end of the project for school meals only. The positive impact of three years of school meals and THRs on enrolment, attendance, retention and drop-out rates of girls from grades 4 to 6 could possibly disappear completely53 in a short period as a result of this decision. This situation offers an interesting opportunity for the WFP and the MNEJCD to start monitoring the effect of the absence of THRs during the present school year (2019-20), and thus in an indirect way, to measure the impact of the THRs. At the central, regional and local levels, all stakeholders were convinced of the positive contribution of THRs to girls from 4th to 6th grade, some of whom are already adolescents as many join primary school not at the age of six, but when much older. However, no effort has been made so far to measure the 52 Leaves of Hibiscus sabdariffa, the red flower bush known in Senegal as Bissap, that is also used a tea. 53 Experience from other West-African countries, as related by Mamadou S. Bah, UNICEF Bissau 26 | P a g e retention rates among this group, either in the presence or in the absence of THRs during a full school year. Key findings and conclusions – Question 2.4. • The WFP SF programme reached most of the poorer, vulnerable pupils of the country, as the programme focused on rural areas, and covered about 60% of all existing schools. However, the WFP’s criteria for the admission of schools into the SF programme excluded most of the poorest children in the most vulnerable communities, as unequipped and makeshift community schools do not meet WFP selection criteria. • Food supplies to the beneficiary schools were generally on time. Only private schools that start two weeks earlier than public schools complained of systematic delays in the first delivery of each school year. • School meals have generally been of sufficient size and of good quality. In those schools that participate in the local purchase pilot projects, variations in the diet were appreciated because it broke the earlier monotony of meals containing only rice and beans. • The monotony of rice-and-bean meals and the lack of a tasty sauce, i.e., badjiqui or others, to accompany it, led to a high use of Maggi cubes in a number of schools, known as «gusto», crumbled directly onto the pupils’ food. This practice is strongly discouraged by the WFP CO, which constantly emphasises the need to avoid using this type of product during the training of cooks, school headmasters, SMCs and PAs. 2.8. Evaluation Question 3.1. Has the programme been implemented in an efficient way? Was there efficient use of resources and efficient methods of work? (Efficiency) Methods of work and systems have been functional and efficient. The flow of food products from the Port of Bissau to the school canteens has been well managed, monitored and supervised. Efficient management and monitoring systems were used that capitalised on decades of experience the WFP had with this type of activity. Theft and misuse of food were kept to a minimal level. Overall, the implementation followed the predicted timing, though periods of school closures due to teacher strikes, particularly during the 2016-17 and 2018-19 school years have resulted in an important delay, and in a significant remaining stock of food (See Annex 8, table A8.1) at the end of the project’s implementation period. The WFP CO team proved sufficiently staffed and skilled and was able to use sufficient and appropriate means of work. The sum of the yearly distributed amounts of food and the remaining stock at the end of August 2019 corresponds to the total quantity of food commodities, rice, beans and oil, that was actually received in the Port of Bissau over the three school years (See Table 8.1, Annex 8). The total number of meals served in all schools corresponds also to this total quantity of food received. The toll-free anonymous complaint and denunciation telephone number (106) has been successfully publicised in all schools and is now «institutionalised» to a certain extent. Its existence, as well as its regular use by the local population, including pupils, has had a positive and dissuading effect. The WFP receives on average 20 relevant phone calls per year on this number. Based on the nature of the complaints, the WFP either channels the calls to the MNEJCD, scrupulously respecting the anonymity of the caller, or it handles the issue itself, if it concerns internal WFP issues. Control at the local level seems to be much more performant when there is a well-functioning SMC and PA, responsible for monitoring the food warehouse activities (food delivery, quantities taken out of the storeroom and given to the cooks, meals distributed to pupils, THRs given to the girls, etc.). The subcontracted local NGOs were sometimes contracted by the WFP with insufficient budgets, which endangered the quality of their work and the sustainability of their results. 27 | P a g e This might be because the WFP normally chooses the proposal with the lowest cost. A typical example of an NGO which built 400 firewood saving stoves for an equal number of school canteen kitchens. Because of some inexperience within this NGO, it had submitted an insufficient budget, that was accepted as such by WFP. This is an aspect the WFP should have taken into account when evaluating the proposal. The insufficient budget for follow-up visits after the dried stove had been fired for the first time, negatively impacted the use of the improved tool by the cooks. School visits during the fieldwork revealed that most stoves fell into disrepair quickly54, and were in most cases not used any more. This activity was exceptionally inefficient, due mostly to the insufficient level of expenditure. In the framework of the HGSF pilot projects, the local NGOs that provide intermediary services purchasing food from the local producers and giving technical advice to farmers’ associations and communities, are being given yearly renewable short-term contracts, though the contract with the Japanese Cooperation runs to 2024. This makes their planning difficult and impedes certain investments in equipment, like means of transport that can only be included in budgets covering a longer period. According to the WFP CO, they were only allowed to set up short-term contracts under this project. The use of the funds that accompanied the in-kind donation of US food is difficult to interpret without a comparison to the original budget55. The WFP’s successive yearly financial reports show expenditures, but not in relation to an initial budget. At the end of 2018, about USD 13.2 million had been used and reported, including the value of the in-kind food donation. The remaining balance will be reported in the 2019 and 2020 financial reports. The provisional figures at the end of 2018 suggest an insufficient level of expenditures for training costs (1.86% of the total), and probably too high of a level of indirect support costs (provisional figures from WFP reports until 2019 indicated a total of USD 1,983,400 for ISC. Projected against the total of USD 20,000,000, this would be 9.917% of the total). (See also table 9.1 in Annex 9) Key findings and conclusions – Question 3.1. • The WFP has used very efficient methods, work procedures and monitoring systems to organise the regular, timely, and theft-free flow of food from the Port of Bissau to the beneficiary schools. • At the school level, supervision and control over the food stock and its use are more present and more efficient when there is an active SMC and PA. • The toll-free anonymous complaint and denunciation telephone number 106 has beenpublicized a lot over the past two to three years and is now well known and regularly used. The WFP has satisfactory systems to guarantee confidentiality from receiving the messages up to handling the issues as soon as possible, either by the WFP itself or handed over to the school canteen unit of the MoE. • The WFP seems to be economizing too much on its contracts with some subcontracted local NGOs. Insufficent budgets impede follow-up visits, which in one particular case could have prevented the firewood saving stoves from falling into disrepair in the school kitchens. At the same time, subcontracted local NGOs in the local purchase pilot projects are only being given short-term contracts, which makes planning difficult for them, also impeding the inclusion of means of transport into their budgets. 54 Generally, the cooks, being volunteer mothers, are used to cook at home on open fires, with the cooking pot resting on three stones. 5 to 10 pieces of firewood can be used in such a fire, as most of the heat escapes easily besides the cooking pots. It is difficult for these women to accept that the fuel saving stove has such a small opening on the front side, that can only accept about 3 pieces of firewood, which should indeed be sufficient. Impatient with such a small fire, the women tend to widen the front opening of the oven somewhat, so as to be able to insert more pieces of firewood. That makes the fire hotter, which causes the expansion of the building material of the oven to be much stronger than foreseen. Thus the oven starts developing deeper cracks than normal, and the decay process begins. If there would have been a budget for regular follow-up visits by Palmeirinha that would have allowed to re-explain the fuel wood saving function of the stoves to the cooks, to make the stove inlet again smaller, and to repair the deeper cracks timely, a much higher longevity of the wood saving stoves would have been ensured. 55 This paragraph is based on an analysis of the successive financial sections in the WFP year reports, and on discussions with WFP CO financial staff. It was not possible to obtain the original budget for the McGovern-Dole project. 28 | P a g e • The use of funds is overall regular, though the interpretation of the figures is somewhat difficult. Provisionally, funds for capacity building appear to have been underused, while those used for indirect support costs seem rather high. 2.9. Evaluation Question 4.1. To what degree has the project achieved the results that were foreseen in the project level framework? (Impact) In order to measure the impact achieved by the McGovern-Dole FFE programme in GB between 2016 and 2019, baseline indicators were compared with data collected during the Mid-Term (2018) and Final (2019) evaluation surveys, using the same instruments and methodologies. A sample composed of WFP-assisted (n=50) and non-assisted schools (n=50) were proportionally selected from eight regions. Quantitative data provided by WFP reports and qualitative data from fieldwork were used for triangulation. SO1: Improved literacy of school-age children (boys and girls): Improved quality of literacy: An implicit assumption of the WFP-MGD FFE programme is that it can promote better literacy rates among school-age children through the delivery of school meals to boys and girls, and THRs to female pupils from 4th to 6th grade. Other activities related to “Improved Quality of Literacy Instruction (MDG 1.2)” were considered out of the scope of the WFP’s work and therefore were not planned. SO1 was thus not monitored. Table 2.5 shows that the retention rates of boys and girls were over 95% for the three school years evaluated, although a slight decrease of 0.6% was observed in 2017-2018 and 2018-2019 when compared to the baseline value. Additionally, drop-out rates remained below 6% during the project’s timeline (2016-2019). Table 2.5 - Main programme outcome values per year, against the baseline and the target values Indicator Baseline value 2016 Value 2017 (SPR/ACR) Value 2018 (ACR) Target end of project Attendance rate (total) 83 % 83% 84.55 % > 85% Attendance rate (girls) 83 % 83 % 85 % > 85% Attendance rate (boys) 83% 83% 84.1 % > 85% Dropout rate (all) 6 % 6 % 6 % < 5 % Dropout rate (girls all grades) 6 % 6% 6 % < 5 % Retention rate (all) 96.40 % 95.9% 95.9 % 85 % Retention rate (girls) 95.8% 96% 96 % 85% Retention rate (boys) 97.9% 95.8% 95.8 % 85% Retention rate girls 4th, 5th, 6th grades Not measured Not monitored Not monitored Not defined Enrolment (all) 75 % 75% 75 % >75% Enrolment rate change (all) 1.73 % 6 % Enrolment (girls) 75% 75% 75% >75% Enrolment rate of change (girls) 0.28 % 6 % Enrolment (boys) 75% 75% 75% >75% Note: at the time of the evaluation the data referring to the school year 2018-19 were not yet available. Improved attentiveness: Progress on the project result, MGD 1.2 Improved Attentiveness, was not directly measured. However, there is an assumption that the provision of daily school meals to students helps them to achieve their nutritional needs and, consequently, has the potential to improve attentiveness during classes. As mentioned under EQ 2.1. (page 15), the programme has reached more pupils every year, both boys and girls, 29 | P a g e than were originally planned for but in the last two school-years (2017-18 and 2018-19), 12% more boys than girls received school meals. Improved student attendance: Figure 2.1 shows the attendance rates of school children in grade 4 from WFP-assisted and non-WFP schools, comparing the percentages reported in the baseline, midline and end-line surveys. Although the attendance rates between WFP and non-WFP schools were similar, they have increased during the last two school years. Also, the percentage of girls attending classes during the 2018-2019 school year was slightly higher than boys in WFP-schools during part of that school year. This was not observed in non￾beneficiary schools. Overall, the impact of school meals on student attendance appears to be limited over the evaluated period and shows a tendency of higher girl attendance. Figure 2.1 - Attendance rates of grade 4 students in WFP and non-WFP Schools (baseline, mid-term and final evaluation) Source: Evaluation surveys, and School Feeding Baseline Survey Guinea-Bissau 2016 According to data from the qualitative survey, opinions obtained from several interviews with school headmasters, teachers, SMCs and Pas, the SF programme has positively impacted student attentiveness, retention of students, and enrolment and attendance rates of girls in WFP-assisted schools. Families now encourage their daughters to enrol and regularly attend classes, serving as examples to other girls. Enrolment rates for all applicable school levels have varied throughout the years and did not differ significantly between WFP-assisted and non-WFP schools (Table 2.6). However, interviewed stakeholders, including school headmasters, teachers, SMC members and Pas, reaffirmed their opinion that enrolment rates have been gnerally increasing over the years as a consequence of the SF programme. Quantitative data however do not confirm these opinions. In several schools visited by the ET, school headmasters and teachers reported the lack of infrastructure, e.g., tables and chairs and didactic materials associated with the increased numbers of students. Table 2.6 also demonstrates that enrolment rates tended to be higher for boys than girls. Table 2.6 - Median enrolment for all applicable school levels, according to school years (2015- 16, 2017-18 and 2018-19) and gender. WFP Assisted Schools Non-assisted Schools 30 | P a g e Baseline 2016 Mid-term 2018 Final Report 2019 Baseline 2016 Mid-term 2018 Final Report 2019 Grade boys girls boys girls boys girls boys girls boys girls boys girls 1st 1.10 1.12 0.99 1 1.01 1.09 1.08 1.02 0.99 1.03 1 0.997 2nd 1.09 1.63 1 0.99 1 1 1.25 0.96 1 1.06 0.95 1 3rd 1.60 2.22 1 1 1 1 1.25 1.03 1 1.05 1.03 1 4th 1.45 1.02 1 1 0.99 0.95 0.81 1.02 1 1.01 1 0.99 5th 0.86 1.09 1 1 0.99 1.00 0.97 1.10 1 1 1 1 6th 0.96 1.39 0.97 1.01 0.94 0.99 1.16 1.04 0.97 1.00 1 1 Source: Evaluation surveys SO2: Increased use of health and dietary practices: School meals are an important contribution to pupils’ nutrition and health, particularly those living in food insecure conditions. The main contributions of the McGovern-Dole’s project to children’s nutrition was through the daily delivery of school meals. Food baskets comprised of 120g of rice, 20g of pulses, 10g of fortified vegetable oil, 20g of canned fish, and 3g of salt provide key macro and micronutrients (580 Kcal, 12.6g of protein, and 2.3mg of iron, among other nutrients). The addition of canned fish to the school meal is the result of the contribution by the Japanese Government56 to the WFP SFP in GB and was really appreciated by school children, according to interviews. However, this activity was discontinued in 2018 (Annex 10, Table A10.1 and A10.2). Figure 2.2 shows that the percentage of students who ate a meal before and after school have increased since 2016, particularly in WFP-supported schools. Only in 2018 was the % of those who ate a meal at home after school highest in non-WFP supported schools . In 2019, 77.4% of pupils from WFP-assisted schools ate a meal before going to school and 94.3% after coming back from school, compared to 71% and 86% in 2016, respectively. However, the percentage of boys having a meal before and after school was higher than that of girls for most of the evaluated years and are aligned with evidence demonstrating that girls are more at risk for food insecurity Figure 2.2. - Percentage of students who ate a meal before and after school (2016, 2018, 2019) Source: Evaluation survey Quality of household food consumption: The quality of household diets was evaluated during the SF baseline, mid-term and end-line surveys, using the Food Consumption Score (FCS) as a reference57. Table 2.7 shows that the percentage of households with acceptable scores has increased from 91.6% in 2016 to 97.5% in 2019 in WFP-assisted schools and from 89.4% to 94.3% over the same period in non-WFP schools. At the same time, Table 2.8 shows that the percentage of families classified as poor or borderline poor seem to be higher in 56 The in-kind contribution of canned fish from the Government of Japan has complemented school meals since 2016, benefiting over 173,000 school children in 2017. Source: WFP/ Semi-annual Report for April to September 2017 Guinea-Bissau 57 The food consumption score (FCS) combines food diversity, food frequency (the number of days each food group is consumed) and the relative nutritional importance of each food group. For each food group the frequency represents the number of days an item was consumed, with a range from 0 (never) to 7 (every day). A weight is assigned to each food group, representing its relative nutritional importance. 31 | P a g e schools without WFP assistance. The fact that families assisted by the SF programme tend to have better quality of diets suggests that parents whose children receive meals in schools can spend the corresponding money to acquire more diverse foods to be consumed at home. Table 2.7- Profile of WFP assisted households according to the Food Consumption Score by year (2016, 2018, 2019) Profiles of household according to the Food Consumption Score Year 2016 2018 2019 2016 2018 2019 2016 2018 2019 Poor Borderline Acceptable WFP school 1.7% 1.5% 60.0% 6.7% 7.3% 1.9% 91.6% 91.2% 97.5% Non-WFP school 2.1% 1.3% 0.9% 8.5% 3.8% 4.8% 89.4% 95.0% 94.3% Note: The Food Consumption Groups (FCG) uses standardised thresholds that typically divide households into three groups. The thresholds that define the three groups are: Poor Consumption: < 21 FCS; Borderline Consumption: >= 21 FCS & < 35 FCS; Adequate Consumption: >= 35 FCS. Source: Evaluation survey Table 2.8- Profile of non-WFP assisted households included according to wealth quintiles by year (2016, 2018, 2019) Profiles of household according to the Food Consumption Score Wealth Quintile 2016 2018 2019 2016 2018 2019 2016 2018 2019 Poor Borderline Acceptable Poorest 3.4% 5.3% 0.5% 9.2% 13.0% 6.7% 87.4% 81.7% 92.8% Second 3.1% 0.8% 0.5% 12.9% 5.3% 6.1% 84.0% 93.9% 93.4% Third 1.7% 0.0% 1.5% 8.2% 5.0% 2.0% 90.1% 95.0% 96.5% Fourth 1.2% 1.0% 0.5% 3.6% 3.1% 1.5% 95.2% 95.9% 98.0% Richest 0.0% 0.0% 0.0% 3.3% 0.8% 0.5% 96.7% 99.2% 99.5% Total 1.7% 1.4% 0.6% 6.7% 5.5% 3.4% 91.6% 93.1% 96.0% Source: Evaluation survey Improved access to clean water and sanitation services: For this outcome of the MGD project framework, no corresponding activity was planned. Table 2.9 shows that, according to the quantitative surveys, the percentage of schools with improved sources of water and toilet facilities, as well as handwashing practices increased over time. Qualitative data however tend to contradict such results. Part of the schools visited by the ET did not have regular access to clean water nor hand washing facilities, even when pupils had to eat with their hands. This adds a great burden to cooks (mostly women) and students and poses challenges to food safety and hygiene conditions. Table 2.9 also demonstrates that the percentage of schools with separate toilets for boys and girls decreased from 74% in 2016 and 87% in 2018, to 64% in 2019. This was also confirmed by the ET during school visits. Safe and sufficient toilet facilities for girls are an important element in the family’s decision on whether to keep theirgirls in school longer. Table 2.9- Percentage of schools with improved source of water and toilet facilities, handwashing practices and separate toilets (2016, 2018, 2019) Year Baseline Mid-term End-line Improved source of water 70% 100% 100% Improved toilet facilities 74% 93% 100% Handwashing 68% 87% 100% Separate toilets for boys and girls 74% 87% 64% Source: Evaluation survey 32 | P a g e Increased knowledge of health and hygiene practices/safe food preparation and storage practices: No targets were set to measure how knowledge of stakeholders in health and hygiene practices have evolved as a result of the different training activities. Table 2.10- Percentage of SMC members who received training from WFP (2016, 2018, 2019) Baseline Mid￾term End-line SMC members who received training from WFP 42% 63.3% 83.7% Source: Evaluation survey Increased knowledge of nutrition to stimulate healthier eating behaviours among students and local communities is also an expected result without a planned corresponding activity (Annex 1). Nonetheless, the WFP GB has worked in partnership with the National Institute for Educational Development (Instituto Nacional de Desenvolvimento da Educação, INDE) to integrate the teaching of food, nutrition and healthy eating in several learning areas in the new primary school curriculum. The WFP training of SMC members and cooks also helped to increase awareness about the benefits of diet diversification. During school visits conducted by the ET, many cooks and SMC members emphasised the importance of including locally produced food products in school meals given their role in stimulating dietary diversity and improving pupils’ nutritional status. Nutritional contribution of school meals and THRs: The food basket supplied by the MGD grant (approximately 580 kcal and 12.6g of protein) covers on average, 22.8% to 46.0% of the recommended daily caloric intake for boys from 3 to 13 years old, respectively. Due to different nutritional requirements, the caloric contribution supplied by the MGD FFE programme ends up being higher for girls from 2 to 13 years (580 kcal covers 25.5% to 50.2% of the recommended calories needed). The combination of rice and beans in the food basket is a particularly relevant source of protein (12.6g) and covers around 37.1% to 66.3% of the recommended daily amount of protein for both girls and boys from 4 to 13 years old. The addition of 20g of mackerel fish per meal into school canteens increased the amount of high￾quality protein. For both the 2016-17 and 2017-18 school years, the WFP attained targets for THRs, reaching 16,232 (105,3%) and 16,323 (100%) girls in 4th to 6th grade, respectively. These rations supplied 745 kcal, 15g of protein, and 1mg of iron along with other nutrients on average and, according to qualitative data, had a significant impact on the nutritional status of girls and their family members. However, during the 2018-2019 school year the expected number of rations of rice given out to girls (4th to 6th grades) was not achieved (99.2%), mainly due to recurring teacher strikes. Additionally, qualitative data collected in the final evaluation revealed that the amount of rice distributed to girls occasionally varied widely among schools, and even within the same school. Increased access to preventative health interventions: See the information related to deworming medication, activity 7 under EQ 2.1. page 18. Increased access to requisites for food preparation and storage tools and equipment: Although WFP has equipped all schools covered by the MGD FFE programme during the three evaluated school years, observations in schools demonstrated that there was no conjoint plan for monitoring the number and quality of equipment and storage tools. Many pupils had to bring bowls and spoons from home or eat with their hands. Interviewed cooks also reported they needed to bring their own equipment in order to cover the lack of adequate utensils to cook. The fact that the WFP did not develop follow up actions to tackle these issues posed additional health risks to students and cooks, mostly volunteer women working under poor conditions, and reduced potential benefits that could be achieved during the period covered 33 | P a g e by the MGD grant. This is especially true in relation to the increased risks of food-borne diseases among students and other work-related diseases among cooks. Differences between WFP assisted and Non-WFP assisted schools: The analysis of mean differences presented in Table 2.11 shows that the percentage of public schools was significantly higher in the WFP assisted schools than in non-WFP assisted schools. Furthermore, a higher percentage of WFP assisted schools offering 5th and 6th grade, had a higher number of both boys and girls enrolled and better school infrastructure as compared to non-WFP supported schools. There were no statistically significant differences in the number of students who had meals before or after school nor in the availability of a library and SMC at the school. The teacher’s gender also demonstrated no statsitcal difference. Table 2.11- Analysis of Results of SF End-line Survey (2019): Mean values in percentage and significant differences between WFP assisted and non-WFP assisted schools Variable WFP Non-WFP Test p￾value School category Public 87.5 65 0.0280 Statistically significant Community initiative 8.2 30 Other 3 2.5 School grade Grades 1, 2, 3 and 4 32.7 62.5 0.0064 Grades 5 and 6 67.3 37.5 Mean number of boys 205.9 110.7 0.0005 Mean number of girls 185.2 109.2 0.0062 School has kitchen 81.6 50 0.0080 School has storage room 79.6 47.5 0.0058 Ate meal at school 95.3 4.7 0.000 Meal provided at school 97.9 2.1 0.000 School has library 28.6 25 0.5169 Non￾statistically significant School has CGE 93.9 82.5 0.2022 Female Teacher 24.6 30.9 0.293 Ate meal before going to school 77.1 71.3 0.161 Ate meal after going to school 92.8 90.8 0.556 Source: Evaluation survey Foundational Result: Strengthening the capacity of the Government of GB on developing and implementing a nationally owned and sustainable school feeding programme: Improved policy and regulatory framework: The central MGD outcome (MGD 2.7.2) has no specific activity corresponding to it. After the 2015 SABER (Systems Approach for Better Education Results) exercise however, the WFP CO played an important role, together with the Centre of Excellence Against Hunger and the ABC, in assisting the MoE to formulate the new National School Feeding Law, and to have it promulgated in June 2019. Increased Government Support: The main activity related to this outcome is “Support Government to Develop School Feeding Pilot Project”. HGSF projects that WFP set up in partnership with the GoGB have exceeded the initial target of supplying locally produced foods to 14 WFP-assisted schools. See main achievements and challenges under EQ 2.1. (p.18) for further information. Building the capacity of the Ministry of Agriculture is an activity that has been planned as part of the future South-South cooperation with the Brazilian Government and the WFP Centre of Excellence Against Hunger. Increased Capacity of Government Institutions: See the discussion on Activity 8, Effectiveness, EQ 2.1. page 18. 34 | P a g e Establish and Train Government Staff on Monitoring and Evaluation System: See the discussion on Activities 9 and 10, pages 18-19, under EQ 2.1. Effectiveness Key findings and conclusions – Question 4.1. • Although the McGovern-Dole framework includes “Improved Quality of Literacy Instruction (MDG 1.2)” as one of its main outcomes, this activity was considered out of the scope of this project. • The SF programme has positively impacted student attendance, retention and enrolment rates of girls (4th to 6th grade) in WFP-assisted schools. • A higher percentage of WFP schools offered 5th and 6th grade, and had a higher number of both boys and girls enrolled and better school infrastructure when compared to non-WFP schools. • Despite the close association between inadequate access to clean water and sanitation services and malnutrition among children, no activity was specifically designed by WFP to further improve infrastructure related to those services in beneficiary schools. • Most of the targets to promote increased knowledge of health and hygiene practices, particularly safe food preparation and storage practices were not reached, especially for females over the 2018-2019 school year. However, the number of SMC members who received training from the WFP increased from 42% in 2016 to 83.7% in 2019. • Targets related to the distribution of deworming medication to children were only achieved at the end of the 2018-2019 school year. Furthermore, at the time of the evaluation, many schools did not have enough utensils and equipment for cooking and eating. • The WFP CO played a key role in assisting the MoE to formulate and promulgate the new National School Feeding Law. The implementation of HGSF projects in partnership with the GoGB has also exceeded initial targets. • The target, “number of government staff trained at the regional and local levels on the management of a school feeding programme” was not reached in any of the three school￾years. 2.10. Evaluation Question 4.2. Have there been any unintended outcomes, either positive or negative? (Impact) Intended and unintended long-term effects on institutional capacities: An unintended effect observed during the Mid-Term evaluation report and confirmed by the Final evaluation survey was the migration of pupils from non-beneficiary schools to those assisted by the WFP SF programme. Interviews with key stakeholders indicated that the number of students had been increasing every year and, many times, schools do not have the capacity to accommodate them in a short period of time. This was mainly reported in the regions where locally produced foods have been included in school menus, e.g., Cacheu and Oio, with pupils walking 7 km per day on their way to and from schools. The existence of a nutritious meal in school supplied by school canteens has certainly been a positive contribution to pupils’ immediate well-being, increasing their motivation to come to school and follow the lessons. Consequently, beneficiary schools tend to attract more new pupils each year than non-beneficiary schools, as presented in Table 2.12. For instance, schools where the SF programme is implemented have on average 114 boys and 91 girls in grades 1 to 4, whereas in non-beneficiary schools the average is 50 for both boys and girls for the same classes. This flow to WFP supported schools is particularly strong for the highest grades (5 and 6), which are not offered by most smaller schools. Table 2.12- Median number of students in WFP and Non-WFP schools (2019) Grades WFP NOT WFP Median of Boys Median of Girls Median of Boys Median of Girls Grades 1, 2, 3 and 4 114 91 50 50 Grades 5 and 6 232 211 162 155 Source: Evaluation survey 35 | P a g e Unintended positive or negative effects for recipients and non-recipients of assistance, specifically for women and girls: Negative effects: The number of girls benefiting from school meals was overall lower than the number of boys, and capacity building though training mostly reached male staff in the MoE. Cooks generally do volunteer work, often in rather difficult conditions. They usually do not have a health screening done. Such facts demonstrate that gender inequality continues to be real challenge in GB and further actions should be carefully considered in order to address this. Another unintended outcome was the fact that a certain proportion of children that took a school meal were given a smaller portion than normal at the family evening meal that same day. Positive effects: The SF programme encouraged school stakeholders to improve school infrastructures, e.g., improving access to clean water and adequate latrines separated by sex and to build stronger relationships with local communities through PAs. School meals also served as a protective factor that helped convince teachers not to adhere to recurrent teacher strikes. Parents decided to contribute to teachers’ salaries because they did not want classes to be stopped and food items to deteriorate and be lost because of the strikes. Finally, the programme has helped to make policymakers and the population itself more accountable in regards to the provision of school meals, especially after the promulgation of the National School Feeding Law. Key findings and conclusions – Question 4.2. • The main unintended effect observed was the migration of pupils from non-beneficiary schools to those assisted by the WFP SF programme, leading to all kinds of capacity problems. • One of the main weakness was that part of the intended results presented in the project framework were not accompanied by planned activities that would have helped WFP and its partners to achieve these results. • The SF programme encouraged school stakeholders to improve schools’ infrastructures and the engagement of local communities through PAs. • The programme has helped making policymakers and the population itself more accountable in regard to the provision of school meals 2.11.Evaluation Question 4.3. What internal and external factors affected the project’s achievement of intended results? (Impact) The project was implemented according to the plan. However, as discussed in previous sections, the ET found that the implementation of training was delayed during the first year, and less than planned in the two latter years, particularly for cooks and SMC members. Some school staff members reported difficulties in running the SF programme and said they had not been sufficiently trained. According to the WFP CO, training activities were conducted every year, but due to the high turnover of staff at the MoE, the impact is lower than foreseen. To mitigate this, the WFP began training more inspectors for them to be able to function as local trainers because inspectors are a more stable category than school directors. The recurring teacher strikes, and the reported migration of students to assisted schools negatively impacted planning and the implementation of the school feeding activities. The increased number of students was not planned for or considered during the WFP’s first food delivery each school year, causing unexpected adjustments such as a reduction in the portion size of the school meals and the THRs. 36 | P a g e Frequent delays and poor quality of many monitoring reports issued by schools caused delays and problems in the reports sent by the MoE to the WFP. As a response, in 2018, the WFP took ownership of monitoring the project by directly collecting and processing the school feeding reports sent by schools58. Key findings and conclusions – Question 4.3. • Some school directors and teachers reported difficulties in running the SF programme and felt they had not been sufficiently trained. • The systematic delays and insufficient quality of monitoring reports led to the adjustment of the process in 2018 with WFP assuming increased responsibility. • The recurring teacher strikes, and the reported migration of students to assisted schools negatively impacted planning and the implementation of the school feeding activities. 2.12. Evaluation Question 5.1: Is the school meal programme sustainable, including a strategy for: sustainability; sound policy; stable funding; quality programme design; institutional arrangements; local production and sourcing; partnership and coordination; and community participation and ownership? (Sustainability) A sustainable SF programme requires multiple elements including a strategy for sustainability, a sound policy, regular funding, institutional arrangements, partnerships, coordination, and community participation and ownership59. There is clear evidence of a growing interest from the GoGB and from MENES in establishing a sustainable national SF programme in the country. However, the WFP CO is still the main entity responsible for implementing and managing the programme. The lack of financial resources and of internal capacity from MENES to manage and monitor the SF programme is seen by several stakeholders as the main challenge for a full handover. At the institutional level, MENES is increasingly committed to taking up a leading role in the management and implementation of the SFP as its relevance is increasingly recognized. During the implementation of the McGovern-Dole programme between 2016 and 2019, the degree of ownership of the SF programme by the GoGB, through MENES has increased significantly. A School Feeding Law60 has been approved and published, and the Government has started making important financial contributions61 to the WFP-managed local purchase pilot projects. The school canteen unit of MENES taking part in the planning and monitoring stages is now also taking full responsibility formanaging the programme in the region of Biombo from the 2019-20 school year onwards, including the monitoring of the intermediary local NGOs and the handling of the cash flow to producers supplying 20 schools. These new developments follow a dynamic that is based on a vision, shared between the WFP and the MoE, that the SF programme should gradually evolve towards a nationally owned and managed programme, based as much as possible on the provision of locally purchased foods. In 2015, a SABER exercise62 was conducted in the country with different stakeholders (See findings in Table 11.1 and Status 2015, Annex 11). Since then, SF school meals have been explicitly included in prominent education policy papers, such as the Decennial Education 58 Source: Interviews with headmasters, and WFP staff responsible for monitoring activities 59 Sources: Rethinking School Feeding: Social Safety Nets, Child Development, and the Education Sector. World Bank (2009) https://elibrary.worldbank.org/doi/abs/10.1596/978-0-8213-7974-5. Access in 8/12/2018; Terms of Reference, Mid-term and Final Evaluations of McGovern-Dole funded School Feeding project in Guinea-Bissau (2016-2018) 60 Lei das Cantinas Escolares, published in June 2019. 61 Government contributions to the WFP managed local purchase pilot project amounted to 200.000 USD in 2017 and 264.000 USD in 2018. Additional funding by the government is foreseen for the current school year 2019-20. 62 Source: Report of SABER 2015. World Food Programme /Partnership for Child Development /World Bank: Workshop for Assessing National Capacities in School Feeding in Guinea-Bissau (23-24 July 2015) 37 | P a g e Plan for 2015-202563, and the WFP has been supporting the DGASCE on a growth path towards increased capacity and involvement, by creating and improving, to a certain extent, conditions for future sustainability. Both actors collaborate in pilot HGSF experiments in the Cacheu, Oio and Biombo regions. This latter part will now be managed and monitored exclusively by the DGASCE, with support from the WFP. However, USDA’s decision to award the new three-year McGovern Dole contract to a totally new operator, CRS, can pose new challenges for coordination among partners and for the hand-over process. Although the SABER exercise has not been officially repeated in the country since 2015, the ET identified the following situation in 2019: (1) Policy Framework (Established); (2) Financial Capacity (Latent); (3) Institutional Capacity and Coordination (Emerging); (4) Programme Design and Implementation (Emerging); and (5) Community Participation and Ownership (Emerging). This overview takes into account the improvements made between 2015 and 2019 in terms of establishing and promulgating a National SF Policy (Lei das Cantinas Escolares) which outlines the objectives, rationale, scope, design, funding and sustainability of the programme, and sets out a strategy for MENES to design and implement the pilot project in the Biombo region (Table A11.1, Annex 11, See Status 2019). Local communities’ views on the SF programme: In general, headmasters, cooks, SMCs, PAs, pupils and local communities perceive the provision of school meals as an essential social and safety net policy that helps not only improve learning capacities and nutritional status of students, but also reduces gender disparities. Observations in the field have elucidated the huge impact of local food purchases on the diversification of school meals while providing local farmers and producers access to a new market for their produce. However, there is still room for improvement. Interviewed cooks were very satisfied with the programme, though there were several complaints about lacking equipment and poor working conditions. They all acknowledged the importance of school canteens in the lives of their children and grandchildren. Most cooks work as volunteers, which seems to create irregularities in their attendance at the schools. The hours spent daily in the school kitchen compete with economic activities that could help their families to improve their living conditions. Rotating shifts of volunteers managed by the schools was an observed good practice. Key findings and conclusions – Question 5.1. • The WFP is still the main organization responsible for implementing and managing the programme. • The WFP has been supporting the DGASCE on a growth path towards increased capacity and involvement by creating and improving conditions for future sustainability to a certain extent. However, the fact that USDA has chosen to award the new 3-year McGovern Dole contract to a new operator, CRS, can pose new challenges to coordination among partners and to the hand￾over process • Improvements were made between 2016 and 2019 towards establishing a national SF Policy and setting out a strategy for the MoE to design and implement a part of the local purchase pilot in the Biombo region. • Local communities perceive the SF programme as an essential social and safety net policy that helps improve learning capacities and the nutritional status of pupils, as well as reduces gender disparities. 2.13. Evaluation Question 5.2 What substantive progress has the Government made toward developing a nationally owned school feeding programme? (Sustainability) 63 A pilot on local purchase for supplying school meals was previously funded and run by WFP. Source: WFP SPR 2017; Semiannual Report Narrative, October 1st 2016 to March 31st 2017. 38 | P a g e An important step taken since the Mid-term Evaluation was the promulgation of the National Law on School Canteens (Lei das Cantinas Escolares) in June 2019. This law recognizes the right to being fed at school for all pupils enrolled in public schools from grades 1 to 6, and clearly opts for the local procurement of food as the chosen approach for the national SF programme. Regrettably, this final evaluation found a clear lack of monitoring, reporting and evaluation activities related to both pilot projects. Those are the one with funding from the GoGB, conducted in Cacheu and Oio, as well as the Japanese funded pilot project in Gabú, Bafatá, Quinara and Tombali. This absence of close monitoring and critical reporting, as would be expected with pilot projects, not only made it very difficult to document good practices, but makes it difficult to evaluate the experience and thus to allow policy makers to make justified choices about the future model and modalities that should be chosen for rolling out when expanding the approach as a national model. Local producers who benefited from the project financed by the GoGB were said to be strongly complaining about the delays in payment for their produce because of the nature of the payment procedure. Though there has been a significant improvement since the first pilot project, this important issue is still causing some mistrust and dissatisfaction among suppliers. They would prefer faster or even immediate payment. This was reported specifically by SMC members and regional directors during the school visits in the regions of Cacheu and Oio. It was also emphasised by local NGOs who are in charge of organising local food purchases from farmers to school canteens. Due to the multisectoral nature of SF programmes, various sectors of government and civil society should be involved in developing capacity for a sustainable model. The DGASCE is making efforts to gather key stakeholders and to revitalise the CIMCE64. This has proved challenging, mainly in relation to finding representatives from other ministries willing to participate and to invest in the SF programme. Additionally, a lack of coordination between the MoE and other development agencies in the country (WHO, UNICEF, PLAN International, the Food and Agriculture Organization of the United Nations (FAO), among others) has been impeding efforts to improving educational indicators, particularly in regards with improving literacy rates and quality of teaching. Technical capacity at the national and regional levels to implement SF programme: WFP has trained teachers, school directors and inspectors in food management and storage while SMCs, PAs, school directors and cooks received training on food preparation and storage, though not in sufficient numbers. However, the results related to this target were further undermined by the numerous changes in MNEJCD’s central staff and by the frequent turnover of previously trained personnel, mainly headmasters, but also regional directors. The WFP CO has adopted a “training of trainers” approach directed to school inspectors in order to build capacity of local school canteens focal points, given that they are rarely replaced. The WFP also seems to continuously make use of its own technical and administrative procedures during capacity development. It thus capacitates trainees to work with WFP systems only. This may have been impeding65 progress on the creation of a common approach with other actors, mainly the DGASCE. Up to 2015, the McGovern Dole FFE programme was managed and administered by the International Programme for Human Development (IPHD), independently from the WFP. The former applied its own technical and administrative procedures. For the coming 3-year period, 2019-22, a new operator, CRS will likely introduce other, slightly different technical and administrative procedures. This constant shift of technical and administrative procedures is not contributing to sustainability in terms of management and reporting capacities at the various levels (schools, regions, and ministries). There is need for the DGASCE to develop its 64 Comissão Interministerial para as Cantinas Escolares (CIMCE) 65 Before WFP, the IPHD did the same, and from next school year onwards, CRS will do the same, each McGovern-Dole implementer with its own systems. In the meantime, the DGASCE has its own monitoring system. This profusion of systems and procedures it not propitious to the emergence of a common management system. 39 | P a g e own technical and administrative procedures and formats, to which each new McGovern-Dole operator should then strictly adhere. That would contribute to organizational sustainability and to strengthen the capacity of school feeding players at the central and decentralized levels. Key findings and conclusions – Question 5.2. • The promulgation of the National Law on School Canteens (Lei das Cantinas Escolares) in June 2019 was one of the main steps taken since the Mid-Term evaluation. • The HGSF pilot project, tested out in partnership with the GoGB, has exceeded the initial target of beneficiary schools and is well perceived by school and local communities. Swift payment to the producers continues to be a challenge. • DGASCE is currently making efforts to gather key stakeholders to revitalize the (CIMCE which has proved challenging. • The lack of coordination between MENES and other development agencies in the country has been lagging behind efforts to improve educational and nutritional indicators. • The WFP has developed activities for strengthening technical capacity at the national and regional levels to implement the SFP but the numbers of trainings and trainees remained under the targets. 2.14. Evaluation Question 5.3. How are local communities involved in and contributing to school feeding? (Sustainability) Community participation and local ownership is one of the pillars of any school feeding programme. In the case of the McGovern-Dole FFE programme, community participation was mostly performed through SMC and PA and had an operational nature only (work, supply of vegetables, dried fish, salt, firewood, etc.). Female community members generally support school canteens by volunteering as cooks or donating food items like salt, moringa) and utensils for cooking. Men, on the other hand, mostly occupy leadership roles. The provision of food produced by smallholder women’s associations to school canteens seems to reduce this gender imbalance somewhat, stimulating women to play a more explicit participatory role in schools while at the same time increasing their income. Fieldwork interviews showed that SMCs are generally composed of five members, a president, a teacher (school canteen), two members of the PA (one of each gender) and one cook (always a women). This committee is responsible for unloading the food delivered to the school, storing it, controlling the inventory and movement from the storage place, controlling the attendance of the students in the classrooms, and sometimes supervising the cooks and attending to the distribution of meals. According to findings from both the mid-term and end-line evaluations, SMCs collaborate with the SF programme mostly because they see this activity as a unique opportunity for controlling the quantity and quality of food delivered to schools and guaranteeing that their own children will benefit from them. SMC members meet regularly, at least once a month in most of the schools visited. However, findings from the end-line evaluation show that there is room for creating a more participatory platform so that the community can become more involved in programme design, monitoring and evaluation processes (See recommendation 6, 2nd part, p.39). Local NGOs that work directly with smallholder farmers emphasised the importance of local food purchases as a means to ensure access to markets for producers, especially to smallholder women’s associations. Finally, Table 2.13 shows an increase in the percentage of schools that have PAs, growing from 80% in 2016 to 96% in 2019. In contrast, the percentage of schools with SMCs decreased from 100% in 2018 to 94% in 2019. According to the WFP CO, this decline is mostly explained 40 | P a g e by low performance of SMCs in a few assisted schools and not necessarily by the absence of an SMC. Table 2.13- Local communities’ participation (2016, 2018, 2019) Baseline Mid-term End-line School has an SMC 94% 100% 94% School has a PA 80% 97% 96% Source: Evaluation survey Key findings and conclusions – Question 5.3. • Community participation is mostly performed through SMCs and PAs and has an operational and supportive function. • SMC members saw their participation as key for controlling the quantity and quality of foods delivered to schools and guaranteeing that their own children and grandchildren will benefit from them. • The purchase of locally produced food creates local income. It means access to a new market for agricultural production and stimulates the local farming economy. As such, it contributes to reducing poverty, which is the root need for school feeding. • The provision of foods produced by smallholder women’s associations to school canteens helps to reduce gender imbalances, stimulating women to play a more participative role in schools, i.e., being part of local purchase committee while increasing their income. 2.15. Evaluation Question 5.4. What needs remain in order to achieve a full handover and nationally owned school feeding programme? (Sustainability) Despite the promulgation of the National Law on School Canteens in June 2019, there is currently no sustainable funding within the MNEJCD and other Ministries that would allow for the implementation of the SF programme with only national resources. The DGASCE did not manage to implement its own M&E system, quality-control or modus operandi procedures. The WFP has the responsibility to improve support and capacity building, as envisioned in the McGovern Dole framework, in order to achieve a full handover. This will require increased investments from the GoGB and its partners to invest in more qualified human resources, improved management and communication tools, well-defined work procedures, and an efficient and well-functioning M&E system. This final evaluation came to the conclusion that the necessary conditions for the government to be more in charge of the programme in the short and mid-term are not in place yet, despite the progress already achieved between 2015 and 2019. Key findings and conclusions – Question 5.4. • There is currently no sustainable funding within the MoE and other Ministries to implement the SF programme with national resources only. • The lack of a specific SF budget line and political instability are handicaps for progress towards increased government ownership. • So far, the DGASCE has not managed to create and implement its own M&E system, quality￾control or modus operandi procedures for the SF programme. • The necessary conditions for the government to be in charge of the programme in the short and mid-term are not yet in place. 2.16. Baseline study and Mid-term evaluation recommendations and how these were implemented: See also Annex 12 41 | P a g e WFP’s response to recommendations from the baseline study Short term SR1: The MoE and WFP should consider appropriate strategies for addressing the “leakage” found in the THR component in most regions. The WFP has not implemented this short-term recommendation, and a significant degree of liberty and variation continues to exist in the way schools handle the THR component. The promotion of the tollfree denunciation phone number 106 is an important positive element against “leakage”, i.e., theft. SR2: The WFP and MoE should consider reviewing and standardizing the role of CGEs and the community in the SFP and incorporating these into official training packages, as the current lack of clarity and support on these issues raises the likelihood of corruption and reduces accountability throughout the system. The WFP’s training of SMC members has most likely responded to a great deal of this recommendation. Also, a Terms of Reference was created and distributed to schools. SR3: The WFP should consider partnering with current initiatives in GB aimed at improving the types and quality of data collected through the national education system. This will ensure that SFP indicators, and the administrative data upon which they are built, are given appropriate value in such processes and begin putting standardized indicators in place for SFP monitoring in the event of a handover. WFP response: It is not clear which initiatives of that period this recommendation was referring. Correct and timely data collection is still a challenge to both the WFP and MENES, but there has been improvements since, especially for the 2018-2019 school-year. Training by the WFP of school staff and MENES personnel has been less performant than planned. MENES’s DGASCE has not yet developed its own monitoring and evaluation system. A pilot project in the region of Biombo will be implemented in 2020 so MENES can gain experience with this system and improve ownership. Medium term MR1: The WFP and MoE should consider commissioning a more detailed investigation into the factors which predict student attendance in GB to better understand whether the current incentives are appropriately designed and targeted. This would ideally be with a research institute or university with experience in such exercises (particularly in the education sector). The baseline survey findings suggest that other factors beyond gender may play a strong role in determining attendance. The outcomes of such an investigation could facilitate an updated school feeding policy so that the process of allocating resources is standardized and guided by more comprehensive data on student vulnerability. WFP response: WFP has not taken the initiative to launch such a study into the factors which influence student attendance in GB. It has kept to its standard formulas, namely the provision of school meals and take home rations for girls from 4th to 6th grade. MR2: The MoE and WFP are encouraged to explore the feasibility and implications of implementing a cost-sharing system, i.e., between Government and parents and communities for the school feeding programme in light of its future commitments under the handover scenario. Cost-sharing might take any number of forms, including increased dependence on local production. The benefits of the SFP are quite visible from the baseline survey findings yet donor funding will never match what is needed by GoGB to scale SFP to all primary-aged students. WFP response: The WFP and MENES have not explored the implementation of a cost-sharing system between the Government and parents and their communities. On the contrary, the HGSF approach seeks to support the communities, which are generally considered as vulnerable. In community schools there seems to be a high degree of local ownership and community contribution. In more established public primary schools increasing contributions from parents might be more difficult and politically sensitive. Mid-term evaluation recommendations: 42 | P a g e 1. Diversify school meals, and promote nutrition and health education. WFP response: The two pilot projects on the local purchase of food are a clear response to this recommendation. Nutrition was also integrated into the new primary school curriculum developed by INDE in partnership with the WFP. 2. Improve kitchen infrastructures and working standards of school cooks. Only 84% of all WFP supported schools have kitchens in good conditions according to the end-line survey. The corresponding values in previous surveys were 82% in baseline and 93% in midline. The working standards of school cooks have not been substantially improved since the mid-term evaluation and survey. 3. Expand school meals to kindergartens. This does not fit into the MoE’s policy, and has not been implemented. 4. Provide health workers trained in nutrition to assist schools. As far as we are aware, this recommendation has not been implemented by the WFP nor by the MoE or the Ministry of Health. 5. Rethink the system for delivering monthly monitoring reports. Efforts are underway to try out digital reporting using smart phones during the 2019-2020 school year. The training effort of MoE personnel on reporting has been insufficient. 6. Map the development partners and other organisations working in partnership with the education system of GB for better coordination. There are only a few organisations working on the improvement of the education system in GB. The WFP is in contact with them, but there has not been active coordination. 7. Expand the involvement of civil society in the project to other regions. Local NGOs are now active in all regions in support of the regular food distribution and of local purchase experiments. 8. Clearly state the role of the project on promoting the quality of teaching. This recommendation has not been implemented by WFP. 9. Create multi-staffed school feeding units in the regions. This recommendation has not been implemented by WFP. 10. Closely follow-up on the numbers of students enrolled in schools for better planned food distribution. The WFP continued to use old numbers for the first food delivery of each year, but then adapted the quantities for the second and third delivery. The WFP has not responded positively towards most of the recommendations made by the mid-term evaluation mission of 2018. In a very similar way, practically all of the recommendations made by the baseline survey team in 2016 have not been effectively implemented. The ET of the final evaluation had no access to the WFP management response to these evaluations and recommendations. This finding suggests that the WFP in GB has had limited capacity to take evaluation missions’ recommendations to heart. 3. Conclusions and Recommendations Overall assessment The McGovern-Dole FFE programme was well implemented between 2016 and 2019, particularly in relation to the management of the flow of US food commodities from the Port of Bissau to the pupils in the 758 assisted primary schools in GB. Other aspects of the programme that need further improvements include the training of cooks, and of SMC and PA leaders in the management and monitoring of the food at the school level, the construction of fuel wood saving cooking stoves, and the training of school directors and inspectors in the reporting procedures about the use of the food in school kitchens. 43 | P a g e The McGovern-Dole FFE programme had an important positive impact on the immediate well-being of pupils and their families, alleviating short term hunger of children and their families. It also contributed to slight increases of enrolment and attendance, and to the stabilisation of the dropout rates, for both boys and girls. The retention rate for girls increased by 2% but stagnated for boys. As the SO1 «Improved Quality of Literacy Instruction» (see Annex 1) was not an objective of the project, its outcomes related to this objective could not be evaluated. The second strategic objective SO2 «Improved health and dietary practices» and the foundational result «Capacity building in the Ministry of Education» were both partialiy achieved (See recommendation 2). Conclusion 1 (Relevance): The SFP is very relevant to the needs of pupils, families and communities because it reduces short term hunger and supplements household food income. Key stakeholders, including at the school level, perceive the THRs of rice for the girls from 4th to 6th grade with an 80% attendance or more as having encouraged parents to send their daughters to school for longer periods of time. The strategy to evolve towards a HGSF approach further enhances its relevance for the local communities as it increases local producer incomes. Conclusion 2 (Relevance): Certain relevant indicators are absent from the list of performance and result indicators in the Financing Agreement, and consequently from the WFP M&E system. These are: (1) retention rates, or alternatively the dropout rates of girls from grades 4 to 6 with an attendance rate of 80% or more who receive THRs of 4 kg of dry rice at the end of each month; (2) the frequency of school meals, expressed as a percentage of the school days with lessons, which could show the relative importance of delays in WFP’s deliveries to schools or of organizational problems at the school level (See recommendation 1). Conclusion 3 (Relevance): The WFP SF programme reached most of the poorer, vulnerable pupils of the country, as the programme focused on rural areas, and covered about 60% of all existing schools. However, the WFP’s criteria for admission of primary schools into the SF programme actually exclude most of the poorest children in the most vulnerable communities, as their unequipped and makeshift community schools do not meet WFP selection criteria which include a minimum level of infrastructure. Conclusion 4 (Effectiveness): Two different pilot projects are currently being conducted by the WFP. Both pilots have reached a size that exceeds what is needed for an experiment. In the meantime, no clear trajectory has been defined that would lead to some kind of a decision-making process. With its Japanese funded pilot, the WFP is increasingly rolling out a model that has not explicitly been chosen by the main actors, particularly not by the MoE. This is becoming an untenable situation (See recommendation 4). Conclusion 5 (Effectiveness): The WFP has been offering contracts with insufficient budgets to some of its subcontracted local NGOs. This implies the risk of insufficient provisions, putting their work on risk and consequently, their project deliverables. The support NGOs in the local purchase pilot projects are only offered the perspective to be hired for their services for a series of consecutive shorter periods, instead of for the full duration of the pilot project. This makes planning and the management of staff, resources, and means of transport much more complicated (See recommendation 7). Conclusion 6 (Effectiveness and Sustainability): The WFP has been the dominant actor in the SF programme in GB over the past three to four years, partly thanks to the McGovern-Dole FFE programme. MENES and its DGASCE have slowly developed a certain level of capacity via their collaboration with the WFP and their monitoring of operations at the school level. Notwithstanding a formal discourse that reaffirms the ownership of the DGASCE over the SF programme, the WFP has been responsible for putting operating systems, norms and procedures, as well as its monitoring and evaluation system in place. Efforts to develop and strengthen capacity in DGASCE have been limited for various reasons so far (See recommendations 9 and 10). 44 | P a g e Conclusion 7 (Effectiveness): School meals have generally been regular and timely, of sufficient size and quality. In schools that participate in the local purchase pilot projects, variations in the diet have been appreciated because they broke the monotony of rice-and￾beans meals. This monotony, and the lack of a tasty sauce, like the very popular badjiqui to accompany it, leads many schools to the excessive use of Maggi cubes by pupils. Conclusion 8 (Gender): The recommendation from the baseline survey (2016) for the WFP to conduct a more in-depth study of parents’ motivations for keeping their daughters at home, rather than attending school classes, has not been implemented. The end-line evaluation demonstrated that other factors beyond a food shortage are also intervening in this much more delicate and complicated issue, and that THRs should probably not be the sole activity to promote a change in parents’ minds (See recommendation 5) Conclusion 9 (Gender): The poor infrastructure of school toilets was pointed out by school directors, teachers and PAs as one of the reasons for pre-adolescent and adolescent girls to abandon school. Another possible reason is the walking distance to school, as this increases the risks for girls being harassed (See recommendation 6). Recommendations Recommendations to the WFP CO Recommendation 1: Directed to the WFP (Priority 1, within the next 6 months): The WFP should, throughout the course of the current school year (2019-20), start monitoring the output indicators that refer specifically to girl pupils in grades 4, 5 and 6 separately, even if its contract with USDA for the McGovern-Dole FFE programme will come to an end in April 2020. The cancellation of the monthly THRs of dried rice for the girls’ families offers an opportunity to measure the impact of this practice in GB by measuring the impact of its sudden absence on the rate of presence in school, on their retention rate and on their dropout rate). Recommendation 2: Directed to WFP (Priority 2, within 6 months – 1 year): The WFP should try to further integrate its SF programme with local efforts, (e.g., UNICEF and INDE to improve teachers’ skills and provide didactic materials, and for more efficient access to deworming medication for school children. Active partnerships with actors and initiatives that work on improved teaching pilots would add value to achieving this target, as well as appropriate indicators to be closely tracked over the years. Recommendation 3: Directed to WFP-GB (Priority 2, within 1 year): the WFP GB CO should continue to adopt a more developmentalist66 profile regarding the SF programme, in relation to the SF programme, with a perspective of sustainability, particularly in relation to the following two evolutions: (a) the shift to a HGSF programme, in which food is purchased from local producers, because it requires a lot of developmental and organisational work on the production side, and to consolidate the link between school and community. This requires specific new skills. (b) the preparation and step by step implementation of the hand-over process of the management of the SF programme to MENES and its DGASCE, as it requires a specific set of new skills and attitudes, quite different from those required for the effective management of food supply. The pilot project in the Biombo region is a good opportunity for improvements in relation to this. (c) The management of the WFP CO should take the initiative for such a shift in attitudes and skills which may imply a retraining of staff. Recommendation 4: Directed to WFP (Priority 1, within the next 6 months): The WFP should assist the DGASCE of the MoE in its reflection about the future HGSF model and modalities, and the process to make a fundamental choice on the basis of the various pilot 66 Development work pursues sustainable post-intervention goals. From the very beginning the end of the assistance is envisaged and planned, even if at medium or long term. 45 | P a g e projects that have been or are being conducted now. The WFP could seek funding and offer organisational assistance for: (1) a jointly commissioned external evaluation by qualified consultants to assist in the participatory analysis of the accumulated experience by both the WFP and the MoE; (2) an externally facilitated workshop with representation by all categories of stakeholders to determine which model or which modalities are best suited to be rolled out in the local context. Such an evaluation would be an opportunity to consult the communities, and to explore ways to set up a structure in which a more permanent dialogue or consultation between the WFP and the DGASCE, along with the the various SMCs and PAs can take place. The latter could potentially participate in self-evaluations, and a channel could be created for an upward flow of complaints, suggestions, opinions, etc. Recommendation 5: directed at WFP CO: (Priority 2, within 1 year): The WFP should organize a more in-depth study into the motivations of parents to keep their daughters at home, and to stop their school education with the DGASCE,. All factors that are playing a role in girls’ education should be identified and analysed, so as to allow for a more balanced and complementary set of actions. Recommendation 6: directed at WFP CO: (Priority 1, within the next 6 months): The WFP CO should endeavour to urgently implement improvements in the presence and conditions of the toilet facilities for pupils in the primary schools it assists with SF, in order to avoid the possible drop-out of pre-adolescent/adolescent girls. The WFP should proactively seek forms of collaboration and partners that are active in the field of water and sanitation. Recommendation 7: directed at WFP CO (Priority 2, for the next round of contracts). The WFP should review the action and budget proposals of the local NGOs it wishes to subcontract for specific support activities much more critically and add a sustainability criterion to their analysis of the proposals. The NGOs should have enough means and funds to implement their tasks in optimal circumstances, and the sustainability of their outputs and outcomes should not be jeopardized by insufficient budgets. Being too economical can lead to a loss of efficiency in the long run. Recommendations to the DGASCE of the MoE: Recommendation 8: Directed to DGASCE (Urgent, to be started immediately): In the interest of better SF results for the children, the DGASCE should work with the WFP to generously share its experience and know-how with the CRS, the new McGovern-Dole operator for the next 3 years. The WFP and the DGASCE should negotiate the modalities for the continued existence and use of the toll-free complaint and denunciation phone number 106 with CRS so that it can continue to be used by both WFP and CRS beneficiaries and later by the MoE and its partners. Recommendation 9: Directed to the Ministry of Education (Priority 1, within the next 3-6 months): The DGASCE should be more proactive in the process of determining inclusion and exclusion criteria for schools and community selection in future SF programmes, like the one CRS is establishing now. This process should take into account criteria that cover the areas of nutrition status, food security, and education, respectively managed by the WFP (VAM and SISSAN) and by the MoE. The decision about the inclusion or exclusion of schools and communities is a political decision that belongs to the Ministry of Education, based on criteria that reflect the minimally required physical conditions. The MoE should ensure the application of the National Quality Standards, together with its partners and the communities. Recommendation 10: Directed at DGASCE (Priority 2, within 1 year): The DGASCE should start to develop and define its own management,M&E systems for the SF programme, probably in a gradual process, and in close dialogue with the WFP, integrating all the good elements from WFP’s experience and from earlier (International Programme for Human 46 | P a g e Development, IPHD) and future McGovern-Dole operators as much as possible. As the Ministry’s own management and M&E systems take shape and are being used regularly, the Ministry of Education should ask every successive McGovern-Dole operator to adopt and follow these national management and M&E systems in the future. Lessons learned • One of the main factors that impeded the McGovern-Dole FFE programme in achieving its SO1, “Improved Literacy”, was the absence of planned activities that would have helped the WFP and its partners to reach this intended objective. For example, improving the quality of teaching, and learning about health and diet should have been considered as a direct result of specific project activities. The Financing Agreement between the FAS/USDA and the WFP did not explicitly recognize the need for collaboration with other actors in the domain of pedagogical improvement. As such, the McGovern-Dole programme seemed somewhat out of rhythm with the national efforts to improve the quality of primary education that is slowly taking shape. As its attention was limited to the distribution of imported US food commodities, it pursued only the outcomes67 of improved attendance rates, (assuming these results in improved attentiveness and learning), leaving the other outcomes and strategic objectives of its own logical framework unattended; firstly, the improved literacy rates of school-aged children. • The SF programme in GB played an unexpected role in several schools’ decisions to not adhere to the frequent teacher strikes, especially during the 2018-2019school￾year. The role of the local SMCs was determinant in convincing their teachers. • The WFP worked very close with national NGOs to scale up local food purchases, exceeding the number of schools that was foreseen between 2016-2019. This may help the WFP to address its selection criteria bias and to expand the programme to more deprived rural areas and communities. 67 See also the McGovern-Dole FFE Framework, that is reproduced in Annex 1. E.g. with regard to SO1: Improved literacy, only the outcomes MGD 1.2 (improved attentiveness) and MGD 1.3 (improved student attendance) are pursued by various activities, while for outcome MGD 1.1 (improved quality of literacy instruction) no activities have been foreseen. 47 | P a g e Annexes Annex 1 - McGovern-Dole FFE Framework 48 | P a g e 49 | P a g e Annex 2 - Revised version of the reconstructed Logical framework of the WFP/McGovern Dole FFE Programme in Guinea Bissau (2016-19) Incorporating68 the relevant elements of the McGovern-Dole FFE Framework and the terms of the Project Financing Agreement FFE-657-2015/019-00 Outcome 1: School age children in Guinea-Bissau have adequate access to safe and nutritious food all year-round Output 1 - Primary school children in targeted schools receive timely, sufficient and nutritionally adequate food transfers to meet their basic food and nutrition requirements Activity 1 - Provide school meals to primary school children, Indicator 1 - Number of daily school meals provided to school-age children; Indicator 2 - Number of school-aged children receiving daily school meals (male) Indicator 3 - Number of school-aged children receiving daily school meals (women) Indicator 4 - Number of school-aged children receiving daily school meals (2+3) Indicator 5 - Number of school students enrolled in schools receiving assistance (women) Indicator 6 - Number of school students enrolled in schools receiving assistance (male) Indicator 7 - Number of school students enrolled in schools receiving assistance (5+6) Indicator 8 - Number of individuals receiving directly from funded intervention (new) Indicator 9 - Number of individuals receiving directly from funded intervention (continuing) Indicator 10 - Number of students regularly (80%) attending supported classrooms/school (women) Indicator 11 - Number of students regularly (80%) attending supported classrooms/school (male) Indicator 12 - Number of students regularly (80%) attending supported classrooms/school (total) Activity 2 - Provide take-home-rations for girls from 4th to 6th grade. Indicator 13 - Number of take-home ration provided as a result of the assistance indicator 14 - Number of students receiving take-home rations as a result of the assistance (women) Indicator 15 - Number of students receiving take-home rations as a result of the assistance (total) Outcome 2: Pupils in assisted schools benefit from increased use of health and dietary practices. Output 2: Better health and dietary practices are increasingly used in the school canteens by the local actors. Activity 3 - Provide training to school management committees (SMC), parent associations, Headmasters and inspectors on food preparation and storage. Indicator 16 -Number of SMC members and cooks trained in food preparation and storage (total) Activity 4 - Provide training to teachers, directors and inspectors in food management and storage Indicator 17 - Number of teachers, directors and inspectors trained in food management and storage (women) Indicator 18 - Number of teachers, directors and inspectors trained in food management and storage (male) Indicator 19 - Number of teachers, directors and inspectors trained in food management and storage (total) Activity 5 - Building/rehabilitation of kitchens and storerooms Indicator 20 - Number of kitchens constructed/rehabilitated Indicator 21 - Number of fuel-efficient stoves provided/rehabilitated Activity 6 - Provide storage and Food Preparation Equipment, Tools and eating utensils 68 Created by the Mid-term evaluation team in 2018, revised and amended by the End-line evaluation team in November 2019. 50 | P a g e Indicator 22 - Number of schools with improved storage equipment, Food Preparation Equipment, Tools and eating utensils as a result of the assistance Activity 7 - Distribute deworming medication and training Indicator 23 - Number of students receiving deworming medication Outcome 3: National institutions have enhanced capacity to efficiently plan and implement programmes in the areas of food security and nutrition and disaster mitigation. Output 3 - Food-insecure people in targeted areas benefit from improved institutionalization of monitoring capacity in Government ministries of Agriculture and Finance to increase and protect their access to food Activity 8 - Capacity building at local, regional and National level Indicator 24- Number of government staff trained at national level on management of schools feeding programme Indicator 25- Number of government staff trained at regional level on management of schools feeding programme Indicator 26- Number of Government staff trained on M&E system Indicator 29- Number of timely schools feeding reports produced Activity 9 – Support Government to develop school feeding pilot project Indicator 27- Number of school feeding government pilot projects conducted Indicator 28- Number of schools covered by the government pilot projects Activity 10 - Support monitoring and evaluation systems Indicator 30 - Monitoring and evaluation system established and functional Evaluation Report Version March 2021 51 | Page Annex 3 – Evaluation Matrix 1 Relevance Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information Question 1.1. Is the project’s strategy relevant to the beneficiaries’ needs? To what extent did the project respond to the needs of the students (boys and girls), especially girls from 4th to 6th grade? Degree of correspondence of the project with the needs, as expressed by all stakeholders including students, especially by the girls, as well as by external soci0-economic assess-ments and analysis Primary data collected from students (specifically girls from the 4th to 6th grade) in the sample of schools of the 8 regions, and their parents (women and men) Secondary data: programme monitoring database, survey, programme reports Group interview (mixed groups of boys and girls). Girls (4th to 6 th grade) semi direct interviews Group interview with parents of students Database and document analysis Content analysis of interviews (specific attention to gender aspects) Systematization of information with excel, cross-checking between information from interviews and data primary data collected in the interviews Expected shyness of the children may interfere with the collection process Illiterate interviewees may not speak Portuguese, requiring interpreter during interviews To what extent did the project respond to the needs of teachers, cooks, inspectors, School Management Committees (SMC) and communities (women and men) Degree of correspondence of the project with needs as expressed by teachers, cooks, inspectors, SMC, and com￾munity leaders. Analysis of primary data collected from the SMC, teachers, cooks (women, men)during on-site visits Secondary data: programme monitoring database (women and men) Focus groups Document analysis In-depth observation Photo collection Content analysis of interviews (specific attention to gender aspects) Systematization of information with excel, cross-checking between information from interviews, observation and secondary data collected in the documents Labour discontentment and tense work relations may dominate the interviews. Question 1.2. Is the project aligned with national government’s education and school feeding policies and strategies? To what extent is the intervention aligned with the education policies and strategies of the government? Degree of correspondence of the programme with the government education policies and the national school feeding law. Primary data collected from Ministry of Education officials Secondary data collected from Government Edu-cation policy documents, the national school fee-ding law (aligned with policies on education, food security and nutrition and gender, including gender elements of sector policies). Interview with Ministry of Education officials Government Documents analysis Interviews Content analysis of interviews Systematization of infor-mation using excel, cross-checking between information from interviews and secondary data collected in the documents Public services in reorganization due to recent appointment of government Question 1.3. Does the project complement other donor-funded and government initiatives? To what extent are other donor-funded initiatives complementary? Degree of complementary of the programme with other donor-funded initiatives. Primary data collected from other stakeholders/ donors Primary data collected during In￾site visiting Focus group with other stakeholders/ donors Analysis of documents and web resources Content analysis of interviews and of existing documents/web resources. Systematization of information using excel, cross-checking Reliable information exists Public services in reorganization due to recent appointment of government Evaluation Report Version March 2021 52 | Page 1 Relevance Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information To what extent is there complementarity with the initiatives of the MED ? Degree of complementary of the programme with government initiatives. Secondary data existing in project literature and web resources In-depth observation, photo collection Interview with Ministry of Education officials between information from interviews and secondary data collected in the documents 2.Effectiveness Question 2.1. Extent of project implementation – did the project carry out all activities as planned? Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information To what extent were school meals provided regularly to primary school children, including take-home-rations to girls from 4th to 6th grade (Activity 1+2)? Number of children (girls vs. boys) receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance; Number of days when school meals were delivered to the students, in relation to the number of school days. Number (and %) of girls from 4th to 6th grade receiving THR Number of schools reached (and % of total number of schools in the different regions and globally) WFP monitoring database WFP Guinea-Bissau Interim country strategic Plan (ICSP) WFP Reports Survey by RESSAN Database analysis Document analysis Systematization of information using Excel Excel, cross-checking between information from database and information collected in the documents Survey report analysis Reliable information exists To what extent did primary school children in targeted schools receive sufficient and nutritionally adequate food transfers in a timely and regular way (Output 1.1)? Number of meals delivered to pupils in relation to the number of school days (percentage of coverage) as a result of USDA assistance Planned food in relation to delivered food (quantities of each food item of the school menu) as a result of USDA assistance Percentage of students who ate meal at school the previous day and source of the meal in WFP and Not-WFP assisted schools(percentage) Nutritional contents of the school meal and daily take home ration (energy, protein, vitamin A, Iron) WFP reports and database Survey by RESSAN Database analysis Document analysis Systematization of information using Excel, cross-checking between information from database and reports Survey report analysis. Reliable information exists Evaluation Report Version March 2021 53 | Page 2.Effectiveness Question 2.1. Extent of project implementation – did the project carry out all activities as planned? Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information To what extent have kitchens and store-rooms been built/ rehabilitated? And were storage and food preparation equipment, tools and eating utensils provided? Number of kitchens constructed/rehabilitated Number of schools where food is prepared in the kitchen Number of schools with improved storage equipment (store-rooms, pallets) and food preparation tools (plates, pots, etc.) Number of fuel-efficient stoves provided/ rehabilitated WFP monitoring database Observation Survey by RESSAN Database analysis Observation in the schools Systematization of information using Excel Excel, cross-checking between information from database and information collected by observation Reliable information exists To what extent was training provided to school management committees (SMC), teachers, and inspectors on management of school meals and complementary activities ? Number of School Management Committee members and cooks (male vs. female) trained in food preparation and storage Number of teachers, directors, and inspectors (male vs. female) trained in food management and storage WFP monitoring databases WFP’s Guinea-Bissau Interim country strategic Plan (ICSP) WFP Reports Database analysis Document analysis Systematization of information using Excel Excel, cross-checking between information from database and information collected in the documents. Specific attention to gender aspects. Labour discontentment and tense work relations may dominate the interviews. To what extent has deworming medication been distributed? Number of students (girls vs. boys) receiving deworming medication WFP monitoring database Observation Teachers Database analysis Observation in the schools Interviews Content analysis Systematization of information using Excel, cross-checking between information from database and information collected by observation Labour discontentment and tense work relations may dominate the interviews. To what extent was capacity building delivered at local, regional and national level? And has technical and analytical assistance been provided to the Government in policy formulation, legal frame-work and management of a national school meals programme? Number of government staff (m/f) trained at national level on management of schools feeding program Number of government staff (m/f) trained at regional level on management of schools feeding program Number of Government staff (m/f) trained on M&E system Number (and %) of school feeding reports that were produced timely and accurately Implementation/progress on five SABER policy goals WFP monitoring database Interviews with Ministry of Education Officials at National and Regional level Reports of Agência de Cooperação Brasileira (ABC) Reports of other stake￾holders and donors Database analysis Interview content analysis; Interviews with ABC Officials Focus group with Educa-tion sector actors, NGOs, local communities, ABC, African Union, WFP, and other UN agencies Analysis of reports Content analysis of interviews Systematization of infor￾mation using Excel, cross￾checking between infor￾mation from database and interviews Reliable information exists Some of the Stakeholders may not be currently present in Guinea-Bissau Evaluation Report Version March 2021 54 | Page 2.Effectiveness Question 2.1. Extent of project implementation – did the project carry out all activities as planned? Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information Implementation of local institutional purchase component of school feed programme To what extent was technical and analytical assistance provided to the Ministry of Education to institutionalize Monitoring and Evaluation the System (M&E) for the school feeding with appropriate budget ? Number of national assessments Number of workshops and trainings Number of M&E tools developed MEN and District Education, field monitors WFP and UNICEF reports Interview Report analysis Systematization of information using Excel, cross-checking between information from database and reports with information collected in interview. Reliable information exists 2.2. To what degree have (and have not) the interventions resulted in the expected results and outcomes? In particular, to what extent did providing THR result in increased attendance and enrolment of girl students? Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information To what extent do school age children in Guinea￾Bissau have adequate access to safe and nutritious food all year-round? Percentage of students (boys and girls) who ate a daily school meal provided by the WFP, as result of USDA assistance Degree of satisfaction: opinion of pupils (boys vs. girls) and teachers in the quality and quantity of food delivered by the school meals WFP reports Mid-term survey On site data collection (survey and ET school visits) Girls from 4th to 6th grade Interviews with stake￾holders Database analysis Document analysis Content analysis Survey Systematization of information using Excel, cross-checking between information from database and reports with information collected in interview. Specific attention to gender aspects. Expected shyness of the girls may interfere with the collection process To what extent did providing take-home￾rations (THR) result in Increase in attendance and enrolment of girl pupils, as compared to non THR assisted schools. Database Survey results Database analysis Interviews with all stake￾holders, and particularly female pupils of grades 4-6 Comparison of data on enrolment and attendance between comparable assisted and non assisted schools Reliable data exist. Interviews with parents may need assistance for Evaluation Report Version March 2021 55 | Page increased attendance and enrolment of girl students? Interviews in schools (focus groups with girl pupils) and (in some cases) with their parents, Verification of hypotheses with teachers, girl pupils and their parents. Specific attention to gender aspects. translation and for intercultural mediation. To what extent have national institutions enhanced capacity to efficiently plan and implement programmes in the areas of food security and nutrition and disaster mitigation? Number of government staff trained at national level on management of schools feeding program Number of government staff trained at regional level on management of schools feeding program Number of Government staff trained on M&E system For all trained government staff: numbers disaggre￾gated by sex if possible. Number of timely schools feeding reports produced WFP Reports WFP database Interview with Ministry of education Officials Document analysis Database analysis Content interview Systematization of information using Excel, cross-checking between information from database and reports Reliable information exists 2.3. Is short term hunger reduced? Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information To what extent has hunger been reduced within the assisted families (THR’s) and in the local communi-ties? Level of vulnerability of local households (as measured by the RESSAN survey using the Food Insecurity Experience Scale (FIES), as a proxy.) Level of vulnerability of women and girls within the household, compared to men and boys Interviews with local community representatives, parent association leaders, local MoA and MinSA staff Baseline and Mid-term Survey School visits Interviews, also with SMC and PA members, pupils, cooks, girl pupils. Database analysis Interview content analysis Specific attention to gender aspects. Systematization of information using Excel, cross-checking between information from interviews, reports and databases. Reliable information exists 2.4. Did assistance reach the right beneficiaries in the right quantity and quality at the right time? Evaluation Report Version March 2021 56 | Page Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information Is the food perceived as sufficient in terms of quantity and quality? Is the food reaching the right beneficiaries? Local evaluation of Food quantity and quality Students Parent Associations SMC, Teachers, Local communities representative Interviews with boys and girls, men and women Interview content analysis (including specific atten-tion to gender aspects) Expected shyness of the children may interfere with the collection process Parents may not speak Portuguese, requiring interpretation during interviews Which improvements can be envisaged to increase the effectiveness and the efficiency of the school feeding programme ? No targets set WFP and Education Ministry officials SMC members (men and women) Local community representatives (men and women) Head masters and teachers, cooks, pupils Essentially open inter-views, both individually and in focus groups. Interview content analysis Interviewees may not speak Portuguese, requiring interpretation during interviews 3.Efficiency 3.1. Has the programme been implemented in an efficient way? Sub-question Measure / Indicator of progress or success Main source of data / information Methods of data collection Methods of Analysis Quality of Information Was there efficient use of resources and efficient methods of work ? Degree to which resources have been used in an efficient and cost-effective way, as compared to realistic alternative ways to implement the project. Degree to which work set up and methods were efficient, as compared to possible alternatives. WFP reports Mid-term survey Interviews with all stakeholders and with WFP staff On site data collection (survey and ET school visits) Interviews with stake￾holders Database analysis Document analysis Content analysis Survey Systematization of information using Excel, analysis of the financial reports, cross-checking between information from database and reports with information collected in interview. Information of reliable quality exists, but might be difficult to obtain. Evaluation Report Version March 2021 57 | Page 4. Impact 4.1. To what degree has the project achieved the results that were foreseen in the project-level framework? Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information What were the results (outcomes) of the project on improving the quality of education and literacy of school children (boys and girls)? Measurement of project achievements against its targets (enrollment, attendance, completion and dropout rates) and of its impact on the promotion of literacy Project agreement: FFE-657-2015/019- 00 Monitoring database Secondary data on Education that is used by the WFP to evaluate outcomes on education SPRs Semiannual Report Narratives Baseline, mid-term and final surveys Project partners reports On-site data collection (school visits) Documental review Interviews and focus group with WFP, their partners (including GoGB officers), headmasters, teachers, school committees, school cooks, parents and students (boys and girls): open questions on results of the project Systematization of information using Excel, cross-checking between information from database, reports and interviews. Specifc attention to gender aspects. Reliable information exists in part of the collected information. Expected shyness of the children may interfere with the collection process. Parents and SMC members may not speak Portu-guese, requiring interpretation during interview. What were the results of the project on promoting health and nutrition of school children (boys and girls) and local communities? Measurement of project achievements against its targets: planned outputs and outcomes of the project on health and nutrition. Project agreement: FFE-657-2015/019- 00 Monitoring database SPRs Semiannual Report Narratives Baseline, mid-term and final surveys Project partners’ reports On-site data col-lection (school visits) Documental review Interviews and focus group with WFP, their partners (including GoGB officers), headmasters, teachers, school committees, school cooks, parents and students (boys and girls) also using open questions. Systematization of information using Excel, cross-checking between information from database and reports and interviews. Specifc attention to gender aspects. Reliable information exists. Expected shyness of the children may interfere with the collection process. Parents may not speak Portuguese, requiring interpreter during interview. What were the progresses of the project on strengthening capacity of the GoGB on developing and implementing a nationally owned and sustainable school feeding? Progress on NCI: National School Feeding Capacity Index (estimated by the WFP using SABER Exercise results) SABER assessments Standard Project Reports In-site data collection Documental review Interviews and focus group with WFP, Government officers (national, regional and local levels), teachers and school committees, using also open questions. Systematization of information using Excel, cross-checking between information from database and reports and interviews Reliable information exists 4.2. Have there been any unintended outcomes, either positives or negatives? What were the intended and unintended long-term effects on institutional capacities? No targets set, no indicators available. Project agreement: FFE-657-2015/019- 00 SPR Documental review Interviews and focus group with WFP and their project Systematization of information using Excel, cross-checking between Reliable information exists. Evaluation Report Version March 2021 58 | Page Semiannual Report Narratives In-site data collection (fieldwork) partners (UNICEF, FAO, WHO, NGOs, GoGB) school members (headmasters, teachers), school committees, school cooks, parents and pupils (boys and girls), also using open questions. information from database and reports and interviews Expected shyness of the children may interfere with the collection process. Parents may not speak Portuguese, requiring interpretation during interviews. Were there unintended (positive and negative) effects for recipients and non-recipients of assistance, also for women and girls? Open issue, no targets or indicators set. SPR Semiannual Report Narratives In-site data collection (fieldwork) Documental review Interviews and focus group with WFP, GoGB, school members, school committees, Parents Associations, pupils, using open questions. Systematization of information using Excel, cross-checking between information from database and reports and interviews. Specifc attention to gender aspects. Reliable information exists. Parents may not speak Portuguese, requiring interpretation during interview. 4.3. What internal and external factors affected the project's achievement of intended results? Was the project implemented according to the plan? If not, how this affected the implement￾tation of the project and in what manner? Open issue, no targets or indicators set. WFP officials SPR Semiannual Report Narratives Project partners reports On-site data collection (fieldwork) Regional structures of Minister of Education Documental review Interviews and focus group with WFP and their project partners (UNI-CEF, FAO, WHO, NGOs, GoGB), school members (headmasters, teachers), school commit￾tees, school cooks, parents and pupils, (boys and girls) using open questions. Systematization of information using Excel, cross-checking between information from database and reports and interviews Reliable information exists 5. Sustainability 5.1. Is the school meals program sustainable, including a strategy for sustainability; sound policy; stable funding; quality program design; institutional arrangements; local production and sourcing; partnership and coordination; community participation and ownership? Sub-question Measure / Indicator of progress Main source of data / information Methods of data collection Methods of Analysis Quality of Information How is the school feeding system perceived by local communities (man and Level of appreciation of the SFP (by men and women) Local community representatives, Parent Association Focus groups including men and women Interview content analysis. Specifc attention to gender aspects. Reliable information exists Evaluation Report Version March 2021 59 | Page women)? What is not good? What could be improved? Are schools exploring the full potential in terms of local institutional purchase of food? Level of coverage of the school food needs by local products Project database, NGO, Local community representatives, FAO Reports on the pilot projects with local purchases Project database, Reports, interviews, Focus groups including men and women Visits to some of the pilot project schools Interview Content analysis Systematization of information using Excel, cross-checking between information from the reports and interviews. Specifc attention to gender aspects. Reliable information exists 5.2. What substantive progress has the government made toward developing a nationally owned school feeding program? Is there a plan for nationally owned SFP? Laws, policies, pilot projects, plans Ministry of Education Officials Reports, legal documents, Plans Interviews Document and report analysis Interview Content analysis Systematization of information using Excel, cross-checking between information from the reports and legal documents and interviews Reliable information exists What progress has been made towards local purchase of food for school feeding ? Pilot experimentation of methods and systems Pilot projet reports, inter￾nal evaluation reports, visits to these schools Document analysis, inter￾views with stakeholders and actors, including local NGOs that participated in the pilot projects Systematization of information, cross-checking between infor-mation from the reports and the interviews Reliable information exists 5.3. How are local communities involved in and contributing toward school feeding? To what extent local communities participate in the school feeding systems (men and women)? To what extent communities contribute to local SFS? Number of women and men participating in the school committees Number of women in decision making positions in the school committees (chair and deputy) WFP reports School Management Committees, Parent Associations, teachers and headmasters Community leaders Document analysis Individual interviews and focus group interview Open questions on type and level of involvement of the community in process of decision taking in aspects related to the local school feeding system. Systematization of information using Excel, Interview content analysis, including specific attention to gender aspects. Parents may not speak Portuguese, requiring interpreter during interview. 5.4. What needs remain in order to achieve a full handover and nationally-owned school feeding program? Is the legal and political environment favourable to the handover of the school feeding systems to the government? If not, what is missing? Existing legal frameworks Policies Interviews with Ministry of Education official Reports, laws, national policies Interview Document review Interview content analysis Systematization of information using Excel, cross-checking between information from database and interviews Reliable information exists Evaluation Report Version March 2021 60 | Page Is there technical capacity at national and regional level to implement school feeding programs? And at school level? If not, what kind of training is needed? Number of government staff (men and women) trained at national level on management of schools feeding program Number of government staff (men and women) trained at regional level on management of schools feeding program Number of Government staff trained on M&E system Interviews with Ministry of Agriculture and other Government Officials (men and women) WFP database Interview Data analysis Contents analysis, including specific attention to gender aspects. Systematization of information using Excel, cross-checking between information from database and interviews Reliable information exists Evaluation Report Version March 2021 61 | P a g e Annex 4 - Interviewed Stakeholders Institutions Stakeholders COAJOQ 1 representative from management DGASCE 3 representatives, including management, school feeding and statistics Evangelical Church 1 representative ECAS-D (Estrutura Comunitária de Animação e Sensibilização para o Desenvolvimento) 2 representatives, including management INED 6 representatives, including management NGO KAFO 5 representatives, including management, finance, and local purchases RESSAN 11 representatives, including management and data collectors Regional Directors 7 representatives Regional SF focal points 7 representatives Tininguena 3 representatives from local purchases UNICEF 2 representatives, including management WFP Center of Excellence Against Hunger (Brasil) 1 representative from South-South Cooperation WFP Guinea-Bissau 12 representatives, including programme management, school feeding, nutrition, finance, VAM and M&E, supply chain and IT WFP Regional Bureau Dakar 1 representative from programme Evaluation Report Version March 2021 62 | P a g e Annex 5 - Number of schools assisted by the McGovern-Dole FFE programme and visited during the baseline, Mid￾Term and Final evaluations Table A5.1. Number of schools assisted by the WFP programme in 2019 and the size of the samples for the End-line survey (July￾October 2019) and the End-line Evaluation fieldwork (October-November 2019) Number of Schools Number of Schools in Baseline Number of schools in the Mid￾term Number of Schools surveyed/visited during the End-line evaluation Region (2019) Baseline Survey Sample Mid-term survey sample End-line Quantitative Survey Sample ET School Visit Sample Biombo 46 2 2 2 2 Oio 150 8 7 8 6 Gabu 109 10 5 8 4 Bafata 179 14 8 12 8 Quinara 60 3 3 3 3 Cacheu 117 13 5 12 7 Bolama-Bijagós 63 0 0 2 0 Tombali 34 0 0 3 2 Total 758 50 30 50 32 Source: own elaboration Evaluation Report Version March 2021 63 | P a g e Annex 7 - Project Performance Indicators Table A7.1 Project Results by Activity and Target for school year 2016-2017/2017-2018/2018-2019 2016-2017 2017-2018 2018-2019 Over the whole period Activity Indicator Target 2016 Result % Target 2017 Result % Target 2018 Result % Sum of targets Sum of achieved results Total % 1.Provide School Meals Number of daily school meals provided as a result of USDA assistance 23,986,380 19,095,230 79.6% 26,400,00 0 19,095,340 72.3% 28,610,175 23.489.32569 89.1% 78.996.555 61.679.895 78.80 % Number of school￾aged children receiving daily school meals as a result of USDA assistance 145,372 173,593 119.4% 160,000 173,593 108,5% 173,395 173,995 100.3% Number of school￾age children (male) receiving daily school meals/Number of students (male) enrolled in schools receiving USDA assistance 75,572 92,278 122% 81,600 91,978 127,2% 83,230 92,202 110.8% Number of school￾age children (female) receiving daily school meals/ /Number of students (female) enrolled in schools receiving USDA assistance 69,800 92,279 132% 78,400 81,616 104,1% 90,165 81,793 90.7% Number of students (male) regularly (80%) attending USDA supported classrooms/schools 60,458 87,379 144.5% 65,720 85,540 130.2% 66,584 73,854 110.9% 69 Source: ACR 2018, output indicators (p.21). Calculation mode: average number of schooldays/month on which foods or at least 4 food groups were provided x N. of project participants (male + female) = 15 dias/mês (15 x 9) x (92.202 + 81793) = 135 x 173.995 = 23.489.325 Evaluation Report Version March 2021 64 | P a g e 2016-2017 2017-2018 2018-2019 Over the whole period Activity Indicator Target 2016 Result % Target 2017 Result % Target 2018 Result % Sum of targets Sum of achieved results Total % Number of students (female) regularly (80%) attending USDA supported classrooms/schools 55,840 75,902 135.9% 62,720 75,086 119.7% 72,132 69,524 96.5% Number of students regularly (80%) attending USDA supported classrooms/schools70 116,298 163,177 140.3% 128,000 164,913 128,8% 138,716 143,378 103.4% 2.Provide Take-Home Rations Number of individuals receiving take-home rations as a result of USDA assistance 15,414 16,232 105.3% 16,320 16,323 100% 16,623 16,484 99.2% 48,357 49,039 101.41% Number of take￾home rations provided as a result of USDA assistance 2,543,310 1,785,520 70.2% 2,692,800 1,795,530 66.7% 2,742,795 1,813,240 66.1% 3 Train School Management Committees and Cooks on Food Prep and Storage Number of School Management Committee members and cooks trained in food preparation and storage 4,466 1710 38.2% 4466+784 = 5,250 1710+2,420 =4,130 78.9% 5,250+1,400 = 6,650 1710+2420 +672=4,802 72.2% 6,650 4,802 72.2% 4. Train Teachers, Directors and Inspectors on Food Manage￾ment and Storage Practices Number of teachers, directors, and inspectors (male) trained in food management and storage 350 141 40.2% 350+25= 375 141+61271= 753 200.8% 375+ 50 = 425 141+612+54 = 807 189.8 % 425 807 189.8% Number of teachers, directors, and inspectors (female) 352 370 105.1% 352+23= 375 370+205= 575 153.3% 375+50= 425 370+205+28 = 603 141.8% 425 603 141.8% 70 This output indicator is also used to assess 1.3: Improved Student Attendance 71 According to PAM, more men than planned because there are less women in rural areas working as teachers. The number exceeded the 2017 target to fill the gap observed in 2016 Evaluation Report Version March 2021 65 | P a g e 2016-2017 2017-2018 2018-2019 Over the whole period Activity Indicator Target 2016 Result % Target 2017 Result % Target 2018 Result % Sum of targets Sum of achieved results Total % trained in food management and storage Number of teachers, directors, and inspectors trained in food management and storage 702 511 72.7% 702+48= 750 511+ 817=1328 177.0% 750+200= 950 511+817+ 82 =1410 148.2 950 511+817+ 82=1410 148.4% 5. Build/Reha￾bilitate Kitchens and Storerooms Number of kitchens constructed / rehabilitated 150 150 100 % 150 150 100% 100 150 150% 400 450 112.5 % Number of fuel￾efficient stoves provided and rehabilitated 150 150 100 % 150 150 100% 100 100 100% 6 Provide Storage and Food Prep Equipment, Tools and Eating Utensils Number of schools with improved storage equipment, food preparation tools and eating utensils as a result of USDA assistance 638 758 118.8 % 750 758 101.1 950 150 15.8% 7. Distribute Deworming Medication Number of students receiving deworming medication(s) 145,372 92,523 63.6% 160,000 99,657 62.3% 173,395 173,995 100.3 % 173,395 173,995 100.3% 8 Train Government Staff on Management of a School Feeding Programme Number of Government staff trained at national level on management of a school feeding programme 19 0 0 % 0 25 131.6%72 0 0 0% 19 25 131.6 % Number of Government staff 224 0 0 % 224 0 0% 0 136 (82 Male; 54 Female) 30.4% 73 448 136 30.36 % 72 In relation to 2016 target (n=19) 73 In relation to 2016/17 targets (n=224+224=448) Evaluation Report Version March 2021 66 | P a g e 2016-2017 2017-2018 2018-2019 Over the whole period Activity Indicator Target 2016 Result % Target 2017 Result % Target 2018 Result % Sum of targets Sum of achieved results Total % trained at regional and local level on management of a school feeding programme Number of refresher trainings organized at the regional level as a follow up to the implementation of the action plan 0 0 0% 3 0 0% 3 10 166.6% 74 9 Establish and Train Government Staff on Monitoring and Evaluation System Number of monitoring and evaluation (M&E) systems established and functional 1 0 0% 0 1 100% N/A N/A N/A Number of Government staff trained on M&E system 19 0 0 % 19 32 168,4% 0 0 0%75 Number of timely school feeding reports produced 5,742 3,138 54.6% 6,750 3,790 56.1% 8,550 4,548 53.2% 21.042 11.476 54.45 % 10 Support Government to Develop School Feeding Pilot Project Number of school feeding Government pilot projects conducted 0 0 0% 1 1 100% 0 1 100% Number of schools covered by the Government pilot Project 0 0 0% 14 0 0% 14 60 428% 14 60 428.0% Note: FE: Final Evaluation/ (o): original/ (r): revised Source: Own elaboration based on MGD Project Results Framework (SO2) and Agreement FFE-657-2015/2015/019-00/ Attachment F: Performance Monitoring Plan and Project Results 2016 and 2017 74 In relation to 2017/18 targets (n=3+3=6) 75 Target (n= 19) achieved in 2018 Evaluation Report Version March 2021 67 | P a g e * the result only includes part of planned school days Annex 8 – Total amount of food distributed to schools, meals prepared and served Table A8.1 Amount of food distributed to schools and number of meals served. Year Tonnage distributed to schools (MT) # of meals served in schools Sources and observations Planned Distr’d Planned Served 2016 5,805 1,650 23,986,380 19,095,230 SPR 2016 2017 3,260 4,716 26,400,000 19,095,230 SPR 2017 2018 5,508 3,323 28,610,175 25,491,510 ACR 2018 p4 Subtotals 14,573 9,689 78,996,555 63,681,970 2019 1,437(stock of Aug 201) 1,437 (foreseen) 11,037,500 11,037.500 WFP CO Supply Chain + projection + own calculation Grand totals 16,010 11,126 90,034,055 74,719,470 Actually received quantity: 11,103,69 T 11,126 Readjusted target* 69,398,063 74,719,470 x 160 gr/meal = 11,955.11 T Own calculation Difference: 22.31 T or 2‰ in excess 5,321,407 meals too much or 7,67 % Acceptable differences * 11,103.69 T / 160 gr (one meal) = 69,398,062.5 meals (Source of the data on quantity received: calculation by WFP’s supply chain staff, 11 Nov. 2019) Table A8. 2. Calculation of the estimated number of meals to be prepared from the existing stock of August 2019: Item and quantity (T) Required quantity for 1 meal Possible # of meals Rice: 1,103.75 T 100 gr (reduced ration) 11,037,500 Beans: 222.35 T 20 gr (reduced ration) 11,117,500 Oil: 110.86 Tons 10 gr 11,086,000 Total: 1,436.96 Tons of food Conclusion: the then existing quantity of rice allows for 11,037,500 meals only. Source: calculation by WFP’s supply chain staff, 11 Nov. 2019. Evaluation Report Version March 2021 68 | P a g e Annex 9 – Approximate estimates of financial reporting Table 9.1: Approximate summary of the financial reporting. Considered : USA-C-01203-01 USA-C-01203-02 USA-C-01203-03 Approved budget US$ 2015 US$ SPR 2016 US$ SPR 2017 US$* ACR 2018 US$ Total expenditures US$ % Confirmed contributions In kind 2,246,950 4,343,820 In cash 3,635,000 14,097,139 Subtotals: 5,881,950 18,440,959 Expenditures Food and related costs 6,780,368 3,165,098 9,945,466 75% Cash/vouchers and related Capacity strengthening 196,458 113,921 310,379 2.3% Implementation costs 246,360 246,360 1.86% Direct support costs 67,441 553,110 620,551 4.68% Adjusted support costs 130,367 130,367 0.985% Indirect support costs ICS 384,800 1,232,276 366,324 1,983,400 ** 14.98% Total expenditures 452,241 0 8,762,212 4,022,070 13,236,523 99.80% * This column for 2017 reflects the totals of expenditures under 3 contracts, with references: USA-C-01203-01 (10026275), USA-C-01203-03 (10027343), and USA-C-01203-02 (1002765). **The current total of ICS spread over the total value of the project is: 1,983,400 / 20,000,000 = 9,17% Sources: own elaboration based on the SPR’s for 2015, 2016, 2017 and on ACR 2018. Evaluation Report Version March 2021 69 | P a g e Annex 10 - Nutritional content of one school meal The nutritional contents of the school meal and daily take home ration in Energy, Protein, Vitamin A and Iron, were estimated using the Tabela Brasileira de Composicão de Alimentos/ TACO - 4th edition, 2011. Table A 10.1 - Nutritional content of one school meal only considering food items acquired by the MGD funded school feeding project Food type Amount g Energy Kcal Protein g Iron mg Vegetable oil 10 90 0 0 Beans 20 66 4 1.6 Rice 120 424 8.6 0.7 Salt 3g 0 0 0 Total 150 580 12.6 2.3 Table A10.2 - Nutritional content of one school meal including all WFP SF food items distributed at schools Food type Amount g Energy Kcal Protein g Iron mg Vegetable oil 10 90 0 0 Beans 20 66 4 1.6 Rice 120 424 8.6 0.7 Sardine 20 57 3.2 0.7 Total 170 637 15.8 3.0 Nutritional content in 30 g beans ● Energy: 98.7 Kcal ● Protein: 6 g ● Iron: 2.4 mg Nutritional content of take home ration - Rice 208 g/ day ● Energy: 745 Kcal ● Protein: 15 g ● Iron: 1.2 mg Evaluation Report Version March 2021 70 | P a g e Annex 11 – Results of the SABER exercise Table A11.1 - Results of SABER Guinea-Bissau 2015 and main improvements identified during the End-line Evaluation (2019) Policy Goal Status 2015 (SABER exercise) Status 2019 (final evaluation)* Policy Framework Latent A School Feeding law was approved by the National Assembly approved in 2010 but it wasn’t promulgated by the President of the Republic. The National Strategy Paper on Poverty Reduction (DENARP II) does not specifically address SF SF is not included in the National Agricultural Investment Programme (NAIP) as a food safety net. Established A School Feeding law was approved by the National Assembly approved in 2010 and promulgated in June 2019 SF is included in the Sectorial Education Plan (2017-2025) as strategy for improving learning capacity and nutritional status of pupils Financial Capacity Latent There is not a national budget line nor regular funds allocated for school feeding Latent There is not a national budget line nor regular funds allocated for school feeding Institutional Capacity and Coordination Emerging Multisectoral Committee emerging, not yet formalized its intervention There are structures, but no coordination between the central structures of the MEN with the deconcentrated structures to plan, implement and monitor the annual programme. At the regional level, a staff member appointed by WFP (Focal Point) is responsible for canteens managed by the Management Committee at the school level. Also, there are no formal bodies including the Community and representatives of Ministries at regional level involved in school feeding. Canteen management committees are in place in schools but there are insufficient human resources and materials; they do not manage resources or the supply of food. Emerging In process of officializing a Multisectoral Committee Coordination between the central structures of the MEN with the deconcentrated structures to plan, implement and monitor the annual programme is still weak No formal bodies including the Community and representatives of Ministries at regional level involved in school feeding. School management committees are in place in schools but they mostly manage food supply. In the regions where the GoGB finance local food purchases, the SMC helps to manage resources. Evaluation Report Version March 2021 71 | P a g e Programme Design and Implementation Latent The importance of monitoring and evaluation is recognized by the government, but there is no national policy on school feeding; as a result, there is no monitoring plan and tools. The reliability of the data collected by the partners and the quality of the reports are problematic There is potential for development through focal groups that are recognized by regional and sectoral education authorities, and by local groups in areas where WFP and IPHD operate. But there is no policy or programme. As there is no policy, no programme plan and no action plan, there can be no distribution modality. No mention of National Standards defined for the food basket Emerging School feeding policy supported by national laws, although definition of different actors and responsibilities are still unclear Institutionalization of the foreseen fiscal hurdles and definitions of the responsibilities of each structure Testing mechanisms for M&E led by the ME in the Biombo region with the assistance of WFP Piloting HGSF projects to diversity school menus based on the local availability recipients' needs and habits Support to small producers for the provision of school canteens based on the surplus production of certain crops that may be part of the school system (mostly coordinated by local NGOs) Community Participation and Ownership Emerging The community through the School Canteen Management Committee has a rather operational contribution (labor, supply of vegetables, dried fish, salt, etc.) It participates in the management of the stocks allocated to the school but not in the design of the Programme (targeting, food basket) or monitoring and evaluation Emerging Training of school feeding management committee Progressive sensitization and awareness of communities at the local, regional and national levels * The ET applied the same tool to evaluate achievements since the last SABER exercise in 2015 Evaluation Report Version March 2021 72 | P a g e Annex 12 - To what extent have the recommendations of the baseline survey and of the mid-term evaluation been implemented by WFP. 1. Baseline survey (2016) recommendations. Recommendation (short and medium term) How WFP has dealt with it SR1: MEN and WFP should consider appropriate strategies for addressing the “leakage” found in the THR component in most regions. E.g., the introduction of a short random, student- or parent-level questionnaire into the existing monthly monitoring system which tracks leakage and operation breaks. WFP has not implemented this short term recommendation, and a significant degree of liberty and variation continues to exist in the way schools handle the THR component. The promotion of the toll-free denunciation phone number 106 is however an important positive element against «leakage» in general. SR2: WFP and MEN should consider reviewing and standardizing the role of CGEs and the community in the SFP and incorporating these into official training packages. Currently the lack of clarity and support on these issues raises the likelihood of corruption and reduces accountability throughout the system WFP’s training of SMC members has probably responded to a great deal of this recommendation. The ET could not verify this in detail. SR3: WFP should consider partnering with current initiatives in Guinea-Bissau aimed at improving the types and quality of data collected through the national education system. This will ensure that SFP indicators (and the administrative data upon which they are built) are given appropriate value in such processes and begin putting in place standardized indicators for SFP monitoring in the event of a handover. It is not clear to which activities of that period this recommendation was referring. Correct and timely data collection is still a challenge to both WFP and MNEJCD. Training by WFP of school staff and MNEJCD personnel has been less performant than planned. The MNEJCD’s DGASCE has not yet developed its own monitoring and evaluation system. MR1: WFP and MEN should consider commissioning a more detailed investigation into the factors which predict student attendance in Guinea-Bissau to better understand whether the current incentives are appropriately designed and targeted. This would ideally be with a research institute or university with experience in such exercises (particularly in the education sector). The baseline survey findings suggest that other factors beyond gender may play strong roles in determining attendance. The outcomes of such an investigation could then feed into an updated school feeding policy so that the process of allocating resources is standardized and guided by more comprehensive data on student vulnerability. WFP has not taken the initiative to launch such a study into the factors which influence student attendance in Guinea Bissau. It has kept to its standard formulas, namely the provision of school meals and take home rations for girls from 4th to 6th grade. MR2: MEN and WFP are encouraged to explore the feasibility and implications of implementing a cost-sharing system (i.e. between Government and parents/communities) for the school feeding programme in light of its future commitments under the handover scenario. Cost-sharing might take any number of forms, including increased dependence on local production. The benefits of the SFP are quite visible from the baseline survey findings yet donor funding will never match that needed by GoGB to scale SFP to all primary-aged students. WFP and the MNEJCD have not explored the implementation of a cost-sharing system between Government and parents and their communities. On the contrary, the HGSF approach seeks to support the communities, which are generally considered as vulnerable. In community schools there seems to be a high degree of local ownership and community contribution. In more established public primary schools increasing contributions from parents might be more difficult and politically sensitive. Evaluation Report Version March 2021 73 | P a g e 2. Mid-term evaluation (2018) recommendations. Recommendation How WFP has implemented it 1. Diversify school meals, and promote nutrition and health education. The two pilot projects on local purchase of food are a clear response to this recommendation. Nutrition was also integrated into the new primary school curriculum developed by INDE. 2. Improve kitchen infrastructures and working standards of school cooks. Only 84% of all supported schools have kitchens in good conditions, according to the end-line survey. The corresponding values in previous surveys were 82 and 93%. The working stan-dards of school cooks have not been substan-tially improved since the mid-term evaluation and survey. 3. Expand school meals to kindergartens This does not fit into the policy of the MNEJCD, and has thus not been implemented. 4. Provide health workers trained in nutrition for assisting schools As far as we are aware, this recommendation has not been implemented by WFP nor by MNEJCD or the Ministry of Health. 5. Rethink the system for delivering monthly monitoring reports Efforts are under way to try out digital reporting using smart phones during the school year 2019- 2020. The training effort of MNEJCD personnel on reporting has been insufficient. 6. Map the development partners and other organisations working in partnership with the education system of GB for better coordination. There are only a few organisations working on the improvement of the education system in GB. WFP has not a very active coordination or collaboration with them. 7. Expand the involvement of civil society in the project to other regions Local NGOs are now active in all regions, in support of the regular food distribution and of local purchase experiments. 8. Clearly state the role of the project on promoting the quality of teaching This recommendation has not been implemented by WFP. 9. Create multi-staffed school feeding units This recommendation has not been implemented by WFP 10. Closely follow-up the numbers of students enrolled in schools for better planned food distribution. WFP continued to use old numbers for the first food delivery of each year, but then adapted the quantities for the second and third delivery. Evaluation Report Version March 2021 74 | P a g e Annex 13 - Terms of Reference Terms of Reference Mid-term and Final Evaluations of the McGovern-Dole International Food for Education and Child Nutrition Program project in Guinea￾Bissau (2016 - 2018) WFP Guinea-Bissau Country Office Fighting Hunger Worldwide Evaluation Report Version March 2021 75 | P a g e Table of Contents 1. Introduction 3 2. Reasons for the Evaluation 3 2.1. Rationale 3 2.2. Objectives 4 2.3. Stakeholders and Users 4 3. Context and subject of the Evaluation 7 3.1. Context 7 3.2. Subject of the evaluation 8 4. Evaluation Approach 9 4.1. Scope 9 4.2. Evaluation Criteria and Questions 9 4.3. Data Availability 10 4.4. Methodology 11 4.5. Quality Assurance and Quality Assessment 13 5. Phases and Deliverables 14 6. Organization of the Evaluation 18 6.1. Evaluation Conduct 18 6.2. Team composition and competencies 18 6.3. Security Considerations 19 7. Roles and Responsibilities of Stakeholders 19 8. Communication and budget 21 8.1. Communication 21 8.2. Budget 21 Annex 1 Map Annex 2 Evaluation Schedule . Annex 3 Membership of the Evaluation Committee . Annex 4 Membership of the Evaluation Reference Group Annex 5 Acronyms ¡. Annex 6 Communication Plan . Annex 7 Project results framework Evaluation Report Version March 2021 76 | P a g e 1. Introduction 1. These Terms of Reference (TOR) are for the mid-term and final evaluations of the McGovern-Dole project in Guinea-Bissau. This evaluation is commissioned by WFP Guinea-Bissau Country Office and will cover the period from August 2017 (preparation phase) to July 2019 (final evaluation report). 2. These TOR were prepared by the Monitoring and Evaluation unit of WFP Guinea-Bissau Country Office based upon an initial document review and consultation with stakeholders and following a standard template. The purpose of the TOR is twofold. Firstly, it provides key information to the evaluation team and helps guide them throughout the evaluation process; and secondly, it provides key information to stakeholders about the proposed evaluation. Thirdly, since the McGovern-Dole agreement (USD $20 million) covers the period from March 2016 to July 2019, the mid-term evaluation results will allow comparison with baseline survey results to mesure the progress/ achievement in the proposed indicators. 3. The mid-term and final evaluation will meet the criteria in the project’s Evaluation Plan and USDA’s Monitoring and Evaluation Policy76 . 4. Currently, 758 schools receive school meals and a total of 173,593 children are fed every school day. The evaluation will be based on a reperesentative sample of schools selected from 8 regions of WFP intervention (Oio, Bafata, Gabu, Cacheu, Quinara , Bolama, Tombali and Biombo). 2. Reasons for the Evaluation 3. The reasons for the evaluation being commissioned are presented below. 2.1. Rationale 4. The evaluations are being commissioned for the following reasons: Since 2016, WFP and the Government of Guinea-Bissau have been implementing a three-year McGovern-DoleMcGovern-Dole project in Guinea-Bissau. A baseline study conducted before the start of the project provided a situational analysis and allowed WFP to establish indicator baseline information and to verify the targets established in the Project Agreement. These evaluations will allow WFP to monitor the progress of the indicators established based on the results of the baseline study. 5. WFP and its project partners will use the mid-term evaluation results to adjust course as necessary for the remainder of the project term. 6. WFP will also use the evaluations findings as a platform for an evidence-based policy dialogue and to inform engagement with the Government of Guinea-Bissau on the development of the national school feeding program and Monitoring and Evaluation system. Following SABER undertaken in 2015, WFP and MoE effort has been oriented to creation of a National School Feeding Programme adopted with an approved School Feeding Law, national budget line, institutional and conceptional capacity from Ministry of Education staff to design and implement the School Feeding Programme with comuinity participation. The evaluations will look into this aspect to come up with information on progress achieved and underline new strategies adapted to political context to proceed with creation of National School Feeding programme. 7. Furthermore, WFP will use the mid-term and final evaluations’ findings to create awareness among key school feeding stakeholders about project activities that could be incorporated into Guinea￾Bissau’s national school meals program for nationwide implementation. 76 https://www.fas.usda.gov/sites/default/files/2014-03/evalpol.pdf Evaluation Report Version March 2021 77 | P a g e 8. Findings and recommendations from the mid-term evaluation would inform and feed into the implementation of the WFP Guinea-Bissau transitional interim Country Strategic Plan (TI-CSP) (January 2018 - June 2019) and the design of the fully fledged Country Strategic Plan (CSP), which is planned to start in July 2019; subsequently, findings and recommendations from the final evaluation would inform the implementation of the CSP during its first years. 2.2. Objectives 9. Evaluations in WFP serve the dual and mutually reinforcing objectives of accountability and learning. ● Accountability – The evaluation will assess and report on the performance and results of the McGovern-Dole school feeding project. ● Learning – The evaluation will determine the reasons why certain results occurred or not to draw lessons, derive good practices and pointers for learning. It will provide evidence-based findings to inform operational and strategic decision-making. Findings will be actively disseminated and lessons will be incorporated into relevant lesson sharing systems. 2.3. Stakeholders and Users 10. A number of stakeholders both inside and outside of WFP have interests in the results of the evaluation and some of these will be asked to play a role in the evaluation process. Table 1 below provides a preliminary stakeholder analysis, which should be deepened by the evaluation team as part of the Inception phase. The Stakeholders organized around the “Essential Learning Package”, to improve learning condition: ensure potable water, latrines, training of teachers, improved school infrastructure, didactic materials and curricula revision, will be informed on the progress achieved through present evaluation. 11. Accountability to affected populations is tied to WFP’s commitments to include beneficiaries as key stakeholders in WFP’s work. As such, WFP is committed to ensuring gender equality and women’s empowerment (GEEW) in the evaluation process, with participation and consultation in the evaluation by women, men, boys and girls from different groups. Gender equity and women’s empowerment envisaged sinse the beginning/elaboration of present the project, will be confirmed in the present study namely: the increase of girls enroment in assisted schools, participation of women in food management committees, the impact of training for cooks in the use of local food and diet diversifitation, organization of local food purchase through women’s associations for provision to schools and their empowerment in literacy and income generation. Table 1: Preliminary Stakeholders’ analysis Stakeholders Interest in the evaluation and likely uses of evaluation report to this stakeholder INTERNAL STAKEHOLDERS Country Office (CO) [Guinea-Bissau] Responsible for the country level planning and operations implementation, It has a direct stake in the evaluation and an interest in learning from experience to inform decision-making. It is also called upon to account internally as well as to its beneficiaries and partners for performance and results of its operation. The Stakeholders organized around the “Essential Learning Pakage”, to improve learning condition: ensure potable water, latrines, training of teachers, improved schools infrastructures, didactic materials and curricula revision, will be informed on the progress achieved through present evaluation. Regional Bureau (RB) [Dakar] Responsible for both oversight of COs and technical guidance and support, the RB management has an interest in an independent/impartial account of the operational performance as well as in learning from the evaluation findings to apply this learning to other country offices. The Regional Evaluation Officers supports CO/RB management to ensure quality, credible and useful decentralized evaluations. WFP HQ [technical units] WFP HQ technical units are responsible for issuing and overseeing the rollout of normative guidance on corporate programme themes, activities and modalities, as well as of overarching corporate policies and strategies. They also have an interest in the lessons that emerge from evaluations, as many may have relevance beyond the geographical area of focus. Relevant HQ units should be consulted from the planning phase to ensure that key policy, strategic and programmatic considerations are understood from the onset of the evaluation. Evaluation Report Version March 2021 78 | P a g e Office of Evaluation (OEV) OEV has a stake in ensuring that decentralized evaluations deliver quality, credible and useful evaluations respecting provisions for impartiality as well as roles and accountabilities of various decentralised evaluation stakeholders as identified in the evaluation policy. WFP Executive Board (EB) The WFP governing body has an interest in being informed about the effectiveness of WFP operations. This evaluation will not be presented to the EB but its findings may feed into annual syntheses and into corporate learning processes. EXTERNAL STAKEHOLDERS Beneficiaries As the ultimate recipients of food assistance, beneficiaries have a stake in WFP determining whether its assistance is appropriate and effective. As such, the level of participation in the evaluation of women, men, boys and girls from different groups will be determined and their respective perspectives will be sought. The information will be collected from direct and indirect beneficiaries of project: girls and boys, women, men, teachers, Food Management Committees and cooks through individual and focus groups interview aiming to get their point of view for better decision making in the project implementation. Government The Government has a direct interest in knowing whether WFP activities in the country are aligned with its priorities, harmonised with the action of other partners and meet the expected results. Issues related to capacity development, handover and sustainability will be of particular interest. Various Ministries are partners in the design and implementation of WFP activities, including, for this specific project, the Ministry of National Education. UN Country team The UNCT’s harmonized action should contribute to the realisation of the government developmental objectives. It has therefore an interest in ensuring that WFP operation is effective in contributing to the UN concerted efforts. Various agencies are also direct partners of WFP at policy and activity leve, including UNICEF and FAO. NGOs NGOs are WFP’s partners for the implementation of some activities while at the same time having their own interventions. The results of the evaluation might affect future implementation modalities, strategic orientations and partnerships. Donors: USDA WFP operations are voluntarily funded by a number of donors. They have an interest in knowing whether their funds have been spent efficiently and if WFP’s work has been effective and contributed to their own strategies and programmes For this evaluation, the main stakeholder is USDA. USDA is the funder of the evaluation. Its role is to review and comment on TORs, participate in a key informant interview with the selected evaluator prior to field data collection, and to review and approve evaluation reports. Civil society Community leaders, School Management Committees, Parent Association members, teachers, and cooks are all active stakeholders and will have a direct interest in the results of this evaluation. 12. The primary users of this evaluation will be: ● The WFP Guinea-Bissau Country Office and its partners in decision-making, notably related to programme implementation and/or design, Country Strategy and partnerships. The evaluation should provide an evidence-based, independent assessment of performance of the school feeding project so that WFP and its project partners can adjust course as necessary for the remainder of the project term. ● Given the core functions of the Regional Bureau (RB), the RB is expected to use the evaluation findings to provide strategic guidance, programme support, and oversight. ● WFP HQ may use evaluations for wider organizational learning and accountability. ● OEV may use the evaluation findings, as appropriate, to feed into evaluation syntheses as well as for annual reporting to the Executive Board. ● USDA will use evaluation findings to inform planning and implementation of other McGovern-Dole projects. 3. Context and subject of the Evaluation 3.1. Context77 13. Guinea-Bissau is a low-income country with a population of 1.8 million people (50.3 percent of women and 49.7 percent of men) and national territory of 36,125 square kilometres, located on the West African coast. Three-fifths of Bissau-Guineans are under 25 and the annual population growth rate is 2.4 percent.1 Due to persistent political instablity, no elected president has successfully served a full 77 Source: WFP Guinea-Bissau transitional interim Country Strategic Plan (January 2018 – June 2019) Evaluation Report Version March 2021 79 | P a g e five-year term since independence from Portugal in 1973. It is ranked 178 out of 188 countries in the 2016 Human Development Index.2 14. Despite significant potential in agriculture and fisheries, gross domestic product (GDP) grew only 0.4 percent between 2000 and 2014, underperforming the 1.9 percent average of Sub-Saharan African countries during the same period.3 Forty years of political instability have deeply constrained socio￾economic and human development. Since democratic elections in 2014, five Prime Ministers have been nominated, four formed new governments, and three were subsequently dismissed. Each government has brought new ministerial appointments and changes in the cadre of technical policy makers, necessitating reestablishment of working relationships. 15. Women are more likely to be unemployed and have more difficulty in accessing social services than men. In some ethnic groups, customary laws deny women access to land or other resources. Women’s access to bank loans and property other than land is restricted because men have authority over most family decision-making. More than two-thirds of the population live below the poverty line.4 Half the population age 15 and above are illiterate, with large disparities between men (45 percent) and women (71 percent). Illiteracy among women is associated with lack of parental interest in education, poverty, distance to schools, forced marriage and early pregnancy. Due to the gender bias in access to resources, poverty impacts women more than men. Women are also vulnerable to forced marriage, early pregnancy, and maternal mortality 16. The primary school completion rate is 62 percent,6 reflecting delayed enrolment, a 20 percent repetition rate, and high numbers of drop-outs between years 4 and 5, especially among rural girls. This leads to gender disparity from 1.0 in primary schools – with regional variations – to 0.81 in secondary schools. Among children of school age, 45 percent are out of school (27 percent boys and 51 percent girls). Oio, Bafata and Gabu regions have the lowest education indicators. Net attendance in urban areas is 76 percent in primary schools and 74 percent in secondary schools, in contrast to rural areas where net attendance is 54 percent in primary and secondary schools. Disparities in attendance are also incomerelated. According to UNESCO, despite progress made in increasing access and reducing gender disparity in primary schools, poor retention rates contribute to completion rates reaching only 62 percent countrywide and even lower in the most vulnerable regions targeted by WFP operations. 3.2. Subject of the evaluation 17. The McGovern-Dole project in Guinea-Bissau (January 2016-December 2018) is using USDA commodities and cash funding, in the total amount of USD 20,000,000. WFP used this contribution to carry out the following activities: provide school meals; provide take home rations; train school management committees, parent associations, Headmasters, and Inspectors; Training: food preparation and storage practices; build/rehabiltate kitchens and storerooms; provide storage and food preparation equipment, tools & eating utensils; distribute deworming medication(s); capacity building: local, regional, and national level; and support monitoring and evaluation system. 18. WFP aimed to incorporate a strong focus on capacity building and long-term sustainability by targeting two of McGovern-Dole’s four Foundational Results: Increased Capacity of Government Institutions and Increased Government Support. Activities that aim to contribute to these Foundational Results include: Train Government Staff on Management of a School Feeding Programme in particular women school directors; Establish and Train Government Staff on Monitoring and Evaluation System; and Support Government to Develop School Feeding Pilot Project. The full project results framework in provided in Annex 7. 19. WFP has developed a nutritionally-balanced school meals program. For 2016, daily hot school meals consisted of 120g of rice, 20g of pulses, 10g of fortified vegetable oil, 20g of canned fish and 3g of salt. Evaluation Report Version March 2021 80 | P a g e The ration provided about 35 percent of the daily nutritional food requirements to school children.78 In all targeted schools, take-home rations of rice (4kg/month) had been provided to girls in grades 4-6 who maintained 80% attendance. Studies conducted in Guinea-Bissau showed that 70% of girls and women are illiterate and the drop out level among girls is higher than among boys. Simultaneously, WFP worked to build the capacity of the Government of Guinea-Bissau and local communities to manage and operate a nationally-owned school feeding program. WFP leveraged its close partnership with the Government of Guinea-Bissau and local communities to ensure successful project implementation. WFP anticipated assisting approximately 145,000 student beneficiaries in FY16, 160,000 in FY17, and 173,000 in FY18. The project is operating in eight regions of the country: Cacheu, Biombo, Oio, Bafata, Gabu, Tombali, Quinara, and Boloma-Bijagos. 20.Currently, the McGovern-Dole funded School feeding project is embedded in the WFP Guinea-Bissau Country Programme 2016-2018 and T-ICSP January 2018 – June 2019. Additional activities covered by the Country Programme are stunting prevention, treatment of moderate acute malnutrition (MAM), HIV/TB care&treatment and Food Assistance for Assets (FFA). Original project document, resource situation updates and the 2016 standard project report (SPR) of the Country Programme can be consulted here. 21. As of January 2018, the project will be transitioned to the WFP Gunea-Bissau Transitional Interim Counry Strategic Plan (TI-CSP) (January 2018-June 2019). The approved TI-CSP document can be consulted here. 4. Evaluation Approach 4.1. Scope and Purpose 22.The scope of the mid-term and final evaluations is the entirety of activities covered by the McGovern￾Dole project in Guinea-Bissau (2016-2018). The evaluations will be carried out with sample from all eight targeted geographic regions. 23. Specifically, the mid-term evaluation will (1) review the project’s relevance, effectiveness and efficiency, impact and sustainability; (2) collect performance indicator data; (3) assess whether the project is on track to meet results and targets; (4) review the results frameworks and theory of change; and (5) identify any necessary mid-course corrections. 24.The final evaluation will, in line with the mid-term evaluation, (1) review the project’s relevance, effectiveness and efficiency, impact and sustainability and (2) collect performance indicator data. More specifically it will (3) assess whether or not the project achieved its expected results; (4) identify lessons learned; (5) assess project replicability; and (6) assess whether or not mid-term evaluation recommendations were implemented. 25. The evaluations will rely on the Baseline Study for baseline data and situational analysis necessary to evaluate the project at interim and at the final stage. WFP envisions that the mid-term evaluation will be conducted approximately halfway through project implementation, from January to June 2018, whereas the final evaluation will be conducted during the first half of 2019. 78 WFP has mobilized sufficient canned fish from Japan to ensure its inclusion in the daily ration throughout 2016. While WFP will continue to appeal for the provision of fish for 2017 and 2018, at this time it is not guaranteed. Therefore, starting from 2017 the ration will include a higher quantity of pulses (30g) instead of fish. Evaluation Report Version March 2021 81 | P a g e 4.2. Evaluation Criteria and Questions 26.Evaluation Criteria The evaluation will apply the international evaluation criteria of relevance, effectiveness and efficiency, impact and sustainability.79 Gender Equality and empowerment of women should be mainstreamed throughout. 27. Evaluation Questions Allied to the evaluation criteria, the evaluation will address the following key questions, which will be further developed by the evaluation team during the inception phase. Collectively, the questions aim at highlighting the key lessons and performance of the McGovern-Dole funded school feeding project, which could inform future strategic and operational decisions. 28.Gender equality and women’s empowerment will be mainstreamed throughout the evaluation questions and sub-questions with consideration of how the perspectives of men, women, boys and girls will be sought in the evaluation process. Data collected will require disaggregation by gender as relevant. 29.Key criteria and questions are outlined in Table 2 below. Key evaluation questions may need to be re￾visited for the final evaluation. Table 2: Criteria and evaluation questions Criteria Evaluation Questions Relevance ▪ Is the project’s strategy relevant to the beneficiaries’ needs? ▪ Is the project aligned with national government’s education and school feeding policies and strategies? ▪ Does the project complement other donor-funded and government initiatives? Effectiveness and Efficiency ▪ What is the progress of project implementation – is the project on track to carry out all and activities as planned? ▪ To what degree have (and have not) the interventions resulted in the expected results and outcomes? In particular, to what extent did providing THR result in increased attendance and enrollment of girl students? ▪ Is hunger reduced? ▪ How can the theory of change be altered to increase efficiency and effectiveness?Did assistance reach the right beneficiaries in the right quantity and quality at the right time? Impact ▪ To what degree has the project made progress toward the results in the project-level framework? ▪ Have there been any unintended outcomes, either positive or negative? ▪ What internal and external factors affect the project’s achievement of intended results? Sustainability ▪ Is the school meals program sustainable, including a strategy for sustainability; sound policy; stable funding; quality program design; institutional arrangements; local production and sourcing; partnership and coordination; community participation and ownership? ▪ What substantive progress has the government made toward developing a nationally owned school feeding program? ▪ How are local communities involved in and contributing toward school feeding? ▪ What needs remain in order to achieve a full handover and nationally-owned school feeding program? 4.3. Data Availability 30.During the process, the evaluation team may rely on the following specific sources of information about the project: ✓ semiannual project reports; ✓ Baseline survey; ✓ Project databases inserted monthly into the COMET and ANDS systems; ✓ The project baseline survey; WFP annual Standard Project Reports (SPR) and other data collected periodically by the project team, including partners. 31. These documents contain quantitative and qualitative information that will assist the evaluators in the analysis of the evolution of the project during this half-period of implementation. Some data and/or information can also be obtained through the decentralized services (Regional Directors) of the Ministry of Education, which contains data on schools that are not assisted by the WFP and which can be used for the comparison of some indicators. 79 For more detail see: http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm and http://www.alnap.org/what-we-do/evaluation/eha Evaluation Report Version March 2021 82 | P a g e 32. The project was initially developed with its corporate indicators from WFP results framework, but with McGovern-Dole funding, other specific indicators were incorporated. So at this time, we have two sets of indicators (corporate and specific, developed after McGovern-Dole funding). Most of these indicators are being collected periodically, except for one or two (corporate) ones, but can be easily raised in an evaluation process, by designing questionnaires with this objective. 33.During the inception phase of the md-term and final evaluations, the evaluation team will determine whether gaps exist in data availability. 34.Despite frequent rotation of the M&E staff, most of the data is collected by the specific Project team in English, except for some in Portuguese. 35. All of this would involve a combination of skills and experience on the part of the assessment team, which could provide solutions to these adjacent situations. 36. The school feeding baseline survey design was based on a quasi-experimental approach to measuring programme impact. This design was necessary as the current school feeding programme is not randomly assigned to schools and students throughout Guinea-Bissau. Such a design identifies an intervention group (in this case, schools in which WFP supports a school feeding programme) and a comparison group which theoretically serves to demonstrate the outcomes where the school feeding programme is not implemented. 37. The baseline survey was a representative, two-stage cluster survey (with structured questionnaires). The quantitative survey collected key data from schools, students, and local households in the school community. Questions developed for the baseline survey will be used at the end of program implementation as a follow up to provide evidence of change from program inception to program conclusion. 38.Sampling: DGIPASE and WFP first sampled 50 WFP schools using the probability-proportional-to-size technique (see Annex II). DGIPASE then selected a comparison group of 50 schools that shared similar education and socioeconomic indicators but which were not supported by WFP. In most cases, the comparison schools sampled were from the same sector as the WFP school (and usually were its nearest neighbor). From each school, enumerators also randomly sampled ten students from the Grade 4 enrollment roster; these children were administered the student-level questionnaire. The enumerators then travelled to these students’ home to administer the household-level questionnaire. For consistency purposes, the baseline survey was conducted in the six regions where WFP was supporting the Government of Guinea-Bissau with school feeding operations in June 2016: Oio, Bafata, Cacheu, Biombo, Quinara, and Gabu. 39.Concerning the quality of data and information, the evaluation team should: a. assess data availability and reliability as part of the inception phase expanding on the information provided in section 4.3. This assessment will inform the data collection. b. systematically check accuracy, consistency and validity of collected data and information and acknowledge any limitations/caveats in drawing conclusions using the data. 4.4. Methodology 40.The methodology will be designed by the evaluation team during the inception phase. It should: ● Employ the relevant evaluation criteria listed in section 4.2. ● Demonstrate impartiality and lack of biases by relying on a cross-section of information sources (stakeholder groups, including beneficiaries, etc.) The selection of field visit sites will also need to demonstrate impartiality. ● Using mixed methods (quantitative, qualitative, participatory etc.) to ensure triangulation of information through a variety of means. Evaluation Report Version March 2021 83 | P a g e ● Contain a sampling strategy, including the sampling method, sample size calculations, and power calculations. ● Ensure comparability to the baseline evaluation ● Apply an evaluation matrix geared towards addressing the key evaluation questions taking into account the data availability challenges, the budget and timing constraints; ● Ensure through the use of mixed methods that women, girls, men and boys from different stakeholders groups participate and that their different voices are heard and used; ● Mainstream gender equality and women’s empowerment, as above; 41. The evaluation team must assess the quality of the baseline data and design during inception, to see whether it can be used to design and implement a high quality impact evaluation for the final evaluation. This would include ensuring that the midline is (i) not conducted during Ramadan, carnival, or cashew harvest seasons, and (ii) ensuring that questionnaires on take home rations make it clear as to what take home rations are, so respondents can answer accurately.80 42. For the mid-line evaluation, all the evaluation criteria must be used to answer the key evaluation questions, but a full impact evaluation design will not be needed. If an impact evaluation design for the final evaluation is not feasible, another high-quality evaluation design must be proposed by the evaluation team. 43.In particular, the mid-term evaluation will draw on the existing body of documented data, including the McGovern-Dole baseline and, as much as possible, regular program implementation assessments. A quantitative survey similar to the baseline study will be conducted. It will utilize survey instruments designed to collect key project data from schools, students, and local households in the school community. Ideally, the survey will be administered according to the design stipulated during the baseline study. The analysis of the collected data will be mainly descriptive, to capture key trends (cross tables, simple frequencies, etc.). In addition – at a minimum – t-tests will be performed to compare the treatment and comparison groups based on the criteria provided for selecting controls. 44.The qualitative data collection methods will include key informant interviews with relevant stakeholders, including: USDA (both the regional Agricultural Attaché, the Washington-based program analyst, and the Washington-based Monitoring and Evaluation staff), Representatives, Regional Directors and inspectors of the Ministry of National Education (MEN), General Direction for Information, Planification and Assessment of the Education System (DGIPASE), UNICEF, and FAO. Additionally, community leaders, School Management Committees, Parent Association members, teachers, and cooks will be targeted for focus group discussions. 45. The following mechanisms for independence and impartiality will be employed: an external service provider will be hired to conduct the evaluation; WFP has appointed a dedicated evaluation manager to manage the evaluation process internally; an internal WFP Evaluation Committee (EC), led by CO management, will make key decisions on the evaluation; an Evaluation Reference Group (ERG) (including WFP and external stakeholders) will be set up to steer the evaluation process and further strengthen the independence of the evaluation. All feedback generated by these groups will be shared with the service provider. The service provider will be required to critically review the submissions and provide feedback on actions taken/or not taken as well as the associated rationale. The compositions of the EC and the ERG are provided in the Annexes section. 46.One of the risks associated to the methodology includes a potential difference in the methodological approach used by the service provider for the mid-term evaluation and the one used for the baseline exercise. To mitigate this risk, an in-depth review of the methodological approach for the baseline study will be needed during the inception phase. The inception report will be carefully reviewed by WFP and stakeholders to ensure methodology and approach are sound. 80 Lessons learned from the conduct of the baseline study. Evaluation Report Version March 2021 84 | P a g e 4.5. Quality Assurance and Quality Assessment 47. WFP’s Decentralized Evaluation Quality Assurance System (DEQAS) defines the quality standards expected from this evaluation and sets out processes with in-built steps for Quality Assurance, Templates for evaluation products and Checklists for their review. DEQAS is closely aligned to the WFP’s evaluation quality assurance system (EQAS) and is based on the UNEG norms and standards and good practice of the international evaluation community and aims to ensure that the evaluation process and products conform to best practice. 48.DEQAS will be systematically applied to this evaluation. The WFP Evaluation Manager will be responsible for ensuring that the evaluation progresses as per the DEQAS Process Guide and for conducting a rigorous quality control of the evaluation products ahead of their finalization. 49.In particular, the DEQAS is also consistent with the principles and criteria outlined in the USDA’s Food Assistance Division’s Monitoring & Evaluation Policy. The evaluation team will make arrangements to ensure data used in the evaluation report is checked for accuracy and reliability, and the report will clearly indicate limitations to the conclusions that can be drawn from the evidence. 50.WFP has developed a set of Quality Assurance Checklistsfor its decentralized evaluations. This includes Checklists for feedback on quality for each of the evaluation products. The relevant Checklist will be applied at each stage, to ensure the quality of the evaluation process and outputs. 51. To enhance the quality and credibility of this evaluation, an outsourced quality support (QS) service directly managed by WFP’s Office of Evaluation in Headquarters provides review of the draft inception and evaluation report (in addition to the same provided on draft TOR), and provide: a. systematic feedback from an evaluation perspective, on the quality of the draft inception and evaluation report; b. recommendations on how to improve the quality of the final inception/evaluation report. 52. The evaluation manager will review the feedback and recommendations from QS and share with the team leader, who is expected to use them to finalise the inception/ evaluation report. To ensure transparency and credibility of the process in line with the UNEG norms and standards81[1], a rationale should be provided for any recommendations that the team does not consider when finalising the report. 53. This quality assurance process as outline above does not interfere with the views and independence of the evaluation team, but ensures the report provides the necessary evidence in a clear and convincing way and draws its conclusions on that basis. 54. The evaluation team will be required to ensure the quality of data (validity, consistency and accuracy) throughout the analytical and reporting phases. The evaluation team should be assured of the accessibility of all relevant documentation within the provisions of the directive on disclosure of information. This is available in WFP’s Directive CP2010/001 on Information Disclosure. 55. All final evaluation reports will be subjected to a post hoc quality assessment by an independent entity through a process that is managed by OEV. The overall rating category of the reports will be made public alongside the evaluation reports. 5. Phases and Deliverables 56. The evaluation will proceed through the following phases. The deliverables and deadlines for each phase are as follows: Figure 1: Summary Process Map 81[1] UNEG Norm #7 states “that transparency is an essential element that establishes trust and builds confidence, enhances stakeholder ownership and increases public accountability” Evaluation Report Version March 2021 85 | P a g e 57. The evaluation process (combined for mid-term and final evaluations) will proceed through nine phases. Annex 2 provides details of the activities and the related timeline of activities and deliverables. The timeline for fieldwork and reporting will be confirmed during inception phases. 58. Preparation phase (August-January 2017): The CO Evaluation Manager will conduct background research and consultation to frame the evaluation; prepare the TOR; select the evaluation team and contract the company for the management and conduct of the evaluation. 59. Mid-term evaluation Inception phase (January-March 2018): This phase aims to prepare the evaluation team for the evaluation phase by ensuring that it has a good grasp of the expectations for the evaluation and a clear plan for conducting it. The inception phase will include a desk review of secondary data and initial interaction with the main stakeholders. Deliverable: Inception Report (IR). The Inception Report details how the team intends to conduct the evaluation with an emphasis on methodological and planning aspects. It will present an analysis of the context and of the operation, the evaluation methodology articulated around a deepened evaluability and gender-sensitive stakeholders’ analysis; an evaluation matrix; and the sampling technique and data collection tools. It will also present the division of tasks amongst team members as well as a detailed schedule for stakeholders’ consultation. The draft IR will be submitted to the QS service for comments; a revised version will then be shared with the Evaluation Reference Group for comments before being submitted to the Evaluation Committee for approval. Stakeholders’ comments will be recorded in a matrix by the evaluation manager and provided to the evaluation team for their consideration before finalisation of the IR. For more details, refer to the content guide for the IR. 60. Mid-term Evaluation Data Collection phase (April-May 2018): The fieldwork will span over one month and will include visits to schools and primary and secondary data collection from local stakeholders. Two debriefing sessions will be held upon completion of the field work. The first one will involve the Country Office (relevant RB and HQ colleagues will be invited to participate through a teleconference) and the second one will be held with external stakeholders. Data collection needs to be completed before Ramadan starts (May 15, 2018). Deliverable: Exit debriefing presentation. An exit debriefing presentation of preliminary findings and conclusions (PowerPoint presentation) will be prepared to support the de-briefings. 61. Mid-term Evaluation Reporting phase (May-July 2018): The evaluation team will analyse the data collected during the desk review and the field work, conduct additional consultations with stakeholders, as required, and draft the evaluation report. It will be submitted to the evaluation manager for quality assurance. Deliverable: Evaluation report (ER). The evaluation report will present the findings, conclusions and recommendations of the evaluation in a concise report of 40 pages maximum, not including annexes. Findings should be evidence-based and relevant to the evaluation questions. Data will be disaggregated by sex and the evaluation findings and conclusions will highlight differences in performance and results of the operation for different beneficiary groups as appropriate. There should be a logical flow from findings to conclusions and from conclusions to recommendations. 1. Prepare 2. Inception Inception Report 3.Collect data Aide memoire / debriefing PPT 4. Analyze data and Report Evaluation Report 5.Disseminate and follow-up Evaluation Report Version March 2021 86 | P a g e Recommendations will be limited in number, actionable and targeted to the relevant users. These will form the basis of the WFP management response to the evaluation. The draft ER will be submitted to the QS service for comments; a revised version will then be shared with the Evaluation Reference Group for comments before being submitted to the Evaluation Committee for approval. Stakeholders’ comments will be recorded in a matrix by the evaluation manager and provided to the evaluation team for their consideration before finalisation of the ER. The draft ER must be submitted to USDA within 60 days of fieldwork completion. For more details, refer to the content guide for the ER. 62. Mid-term Evaluation Follow-up and dissemination phase (from August 2018): The CO management will respond to the evaluation recommendations by providing actions that will be taken to address each recommendation and estimated timelines for taking those actions. The RB will support WFP’s management response to the evaluation as appropriate, including following up with country offices on status of implementation of the actions. OEV will also subject the evaluation report to an external post￾hoc quality assessment to report independently on the quality, credibility and utility of the evaluation in line with evaluation norms and standards. The final evaluation report will be published on the WFP public website. Findings will be disseminated, and lessons will be incorporated into other relevant lesson sharing systems. 63. Final evaluation Inception phase (January-March, 2019): This phase aims to prepare the evaluation team for the evaluation phase by ensuring that it has a good grasp of the expectations for the evaluation and a clear plan for conducting it. The inception phase will include a desk review of secondary data and initial interaction with the main stakeholders. Deliverable: Inception Report (IR). The Inception Report details how the team intends to conduct the evaluation with an emphasis on methodological and planning aspects. It will present an analysis of the context and of the operation, the evaluation methodology articulated around a deepened evaluability and gender-sensitive stakeholders’ analysis; an evaluation matrix; and the sampling technique and data collection tools. It will also present the division of tasks amongst team members as well as a detailed schedule for stakeholders’ consultation. The draft IR will be submitted to the QS service for comments; a revised version will then be shared with the Evaluation Reference Group for comments before being submitted to the Evaluation Committee for approval. Stakeholders’ comments will be recorded in a matrix by the evaluation manager and provided to the evaluation team for their consideration before finalisation of the IR. For more details, refer to the content guide for the IR. 64.Final Evaluation Data Collection phase (March-April 2019): The fieldwork will span over one month and will include visits to project sites (schools) and primary and secondary data collection from local stakeholders. Two debriefing sessions will be held upon completion of the field work. The first one will involve the Country Office (relevant RB and HQ colleagues will be invited to participate through a teleconference) and the second one will be held with external stakeholders. Data collection needs to be completed before Ramadan starts (May 5, 2019). Deliverable: Exit debriefing presentation. An exit debriefing presentation of preliminary findings and conclusions (PowerPoint presentation) will be prepared to support the de-briefings. 65. Final Evaluation Reporting phase (May-July 2019): The evaluation team will analyse the data collected during the desk review and the field work, conduct additional consultations with stakeholders, as required, and draft the evaluation report. It will be submitted to the evaluation manager for quality assurance. Deliverable: Evaluation report (ER). The evaluation report will present the findings, conclusions and recommendations of the evaluation in a concise report of 40 pages maximum. Findings should be evidence-based and relevant to the evaluation questions. Data will be disaggregated by sex and the evaluation findings and conclusions will highlight differences in performance and results of the Evaluation Report Version March 2021 87 | P a g e operation for different beneficiary groups as appropriate. There should be a logical flow from findings to conclusions and from conclusions to recommendations. Recommendations will be limited in number, actionable and targeted to the relevant users. These will form the basis of the WFP management response to the evaluation. The draft ER will be submitted to the QS service for comments; a revised version will then be shared with the Evaluation Reference Group for comments before being submitted to the Evaluation Committee for approval. Stakeholders’ comments will be recorded in a matrix by the evaluation manager and provided to the evaluation team for their consideration before finalisation of the ER. For more details, refer to the content guide for the ER. 66. Final Evaluation Follow-up and dissemination phase (from August 2019): The CO management will respond to the evaluation recommendations by providing actions that will be taken to address each recommendation and estimated timelines for taking those actions. The RB will support WFP’s management response to the evaluation as appropriate, including following up with country offices on status of implementation of the actions. OEV will also subject the evaluation report to an external post￾hoc quality assessment to report independently on the quality, credibility and utility of the evaluation in line with evaluation norms and standards. The final evaluation report will be published on the WFP public website. Findings will be disseminated and lessons will be incorporated into other relevant lesson sharing systems. Notes on deliverables (mid-term and final evaluations): 67. A full list of expected deliverables is provided below: a. Inception, draft, and final evaluation reports (mid-term and final) b. Quality Assurance Plan c. Raw and clean data sets d. Suggested table of contents for evaluation reports: o Executive Summary o Introduction o Background (Program description and purpose of evaluation) o Methodology and Implementation o Results and Findings o Conclusions o Recommendations o Lessons Learned o Annexes I. Table of McGovern-Dole performance indicators with updated values in comparison to baseline values II. List of meetings III. Survey instruments IV. TOR V. Project-Level Results Framework e. A final evaluation summary brief, not to exceed 4 pages, that summarizes the main findings of the report. It should include charts, graphs, etc. to visualize the data in a clear, easy to read format, accessible to stakeholders from the community level to the government level. The final results and summary reports will be shared with project stakeholders. (final evaluation only) 68.The inception and evaluation reports shall be written in English and follow the DEQAS templates. The evaluation team is expected to produce written work that is of very high standard, evidence-based, and free of errors. The evaluation company is ultimately responsible for the timeliness and quality of Evaluation Report Version March 2021 88 | P a g e the evaluation products. If the expected standards are not met the evaluation company will, at its own expense, make the necessary amendments to bring the evaluation products to required quality level. 69.The evaluation TOR, evaluation reports and management responses will be public and posted on the WFP External Website (wfp.org/evaluation). The other evaluation products will be kept internal. The CO will translate final evaluation products in Portuguese as relevant, for broader dissemination at country level. 6. Organization of the Evaluation 6.1. Evaluation Conduct 70. The evaluation team will conduct the evaluation under the direction of its team leader and in close communication with the WFP evaluation manager. The team will be hired following agreement with WFP on its composition. 71. The evaluation team will not have been involved in the design or implementation of the subject of evaluation or have any other conflicts of interest. Further, they will act impartially and respect the code of conduct of the evaluation profession. 6.2. Team composition and competencies 72. The evaluation team is expected to include three to four members, including the team leader and at least one national consultant. To the extent possible, the evaluation will be conducted by a gender￾balanced, geographically and culturally diverse team with appropriate skills to assess gender dimensions of the subject as specified in the scope, approach and methodology sections of the ToR. At least one team member should have WFP experience. 73. The team will be multi-disciplinary and include members who together include an appropriate balance of expertise and practical knowledge in the following areas: ● School Feeding programmes ● Food and Nutrition Security ● Institutional capacity development ● Gender expertise / good knowledge of gender issues ● Familiarity with the USDA M&E policy ● All team members should have strong analytical and communication skills, evaluation experience and, to the extent possible, familiarity with Guinea-Bissau and/or western Africa development context. ● Oral and written language requirements include proficiency in English and Portuguese among team members. The inception and evaluation reports will be delivered in English. 74. The Team leader will have technical expertise in one of the technical areas listed above as well as expertise in designing methodology and data collection tools and demonstrated experience in leading similar evaluations. She/he will also have leadership, analytical and communication skills, including a track record of excellent English and Portuguese writing and presentation skills. 75. Her/his primary responsibilities will be: i) defining the evaluation approach and methodology; ii) guiding and managing the team; iii) leading the evaluation mission and representing the evaluation team; iv) drafting and revising, as required, the inception report, the end of field work (i.e. exit) debriefing presentation and evaluation report in line with DEQAS. 76. The team members will bring together a complementary combination of the technical expertise required and have a track record of written work on similar assignments. Evaluation Report Version March 2021 89 | P a g e 77. Team members will: i) contribute to the methodology in their area of expertise based on a document review; ii) conduct field work; iii) participate in team meetings and meetings with stakeholders; iv) contribute to the drafting and revision of the evaluation products in their technical area(s). 6.3. Security Considerations 78. Security clearance where required is to be obtained from WFP Guinea-Bissau CO. ● As an ‘independent supplier’ of evaluation services to WFP, the evaluation company is responsible for ensuring the security of all persons contracted, including adequate arrangements for evacuation for medical or situational reasons. The consultants contracted by the evaluation company do not fall under the UN Department of Safety & Security (UNDSS) system for UN personnel. ● Consultants hired independently are covered by the UN Department of Safety & Security (UNDSS) system for UN personnel which cover WFP staff and consultants contracted directly by WFP. Independent consultants must obtain UNDSS security clearance for travelling to be obtained from designated duty station and complete the UN system’s Basic and Advance Security in the Field courses in advance, print out their certificates and take them with them.82 79. However, to avoid any security incidents, the Evaluation Manager is requested to ensure that: ● The WFP CO registers the team members with the Security Officer on arrival in country and arranges a security briefing for them to gain an understanding of the security situation on the ground. ● The team members observe applicable UN security rules and regulations – e.g. curfews etc. 7. Roles and Responsibilities of Stakeholders 80.The WFP Guinea-Bissau CO: a- The WFP Guinea-Bissau CO: Management (Director or Deputy Director) will take responsibility to: o Assign an Evaluation Manager for the evaluation: Elber Nosolini, National Programme Officer. o Compose the internal evaluation committee and the evaluation reference group (see below). o Internally approve the final Tor, inception and evaluation reports. o Ensure the independence and impartiality of the evaluation at all stages, including establishment of an Evaluation Committee and of a Reference Group (see below and TN on Independence and Impartiality). o Participate in discussions with the evaluation team on the evaluation design and the evaluation subject, its performance and results with the Evaluation Manager and the evaluation team o Organise and participate in two separate debriefings, one internal and one with external stakeholders o Oversee dissemination and follow-up processes, including the preparation of a Management Response to the evaluation recommendations b- The Evaluation Manager: o Manages the evaluation process through all phases including drafting this TOR o Ensures quality assurance mechanisms are operational o Consolidates and shares comments on draft TOR, inception and evaluation reports with the evaluation team o Ensures expected use of quality assurance mechanisms (checklists, quality support o Ensures that the team has access to all documentation and information necessary to the evaluation; facilitates the team’s contacts with local stakeholders; sets up meetings, field visits; provides logistic support during the fieldwork; and arranges for interpretation, if required. o Organises security briefings for the evaluation team and provides any materials as required 82 Field Courses: Basic; Advanced Evaluation Report Version March 2021 90 | P a g e c- An internal Evaluation Committee has been formed as part of ensuring the independence and impartiality of the evaluation. This committee will be composed by Chair-Kiyomi Kawaguchi CD; deputy ChairBob Barad DCD; Secretary-Elber Nosolini NPO; Member-Jose Cabral School Meals Focal Point, Filippo Pompili, Regional Evaluation Officer. The members of the committee will provide inputs to the evaluation process and comment on evaluation products and make key decisions such as internal approval of evaluation deliverables (Refer to Annex 3 for the list of members). 81. USDA will be involved in the evaluation at the following stages: Appropriate members of USDA (Programme analyst and M&E lead) will be consulted for comment and approval of the TOR; serve as a member of the ERG; participate in key informant interviews with selected evaluators prior to field data collection; and participate in stakeholder meetings and presentation of the evaluation findings, as appropriate. As per agreement between USDA and WFP in the context of the McGovern-Dole grant, the final approval of the evaluation main products will be at USDA level. 82.An Evaluation Reference Group has been formed, as appropriate, with representation from DGASE, DGPASE, UNICEF and USDA. The ERG members will review and comment on the draft evaluation products and act as key informants in order to further safeguard against bias and influence. 83.The Regional Bureau: will take responsibility to: o Advise the Evaluation Manager and provide support to the evaluation process where appropriate. o Participate in discussions with the evaluation team on the evaluation design and on the evaluation subject as relevant, as required. o Provide comments on the draft TOR, Inception and Evaluation reports o Support the Management Response to the evaluation and track the implementation of the recommendations. While the Regional Evaluation Officer, Filippo Pompili, will perform most of the above responsibilities, other RB relevant technical staff may participate in the evaluation reference group and/or comment on evaluation products as appropriate. 84.Relevant WFP Headquarters divisions will take responsibility to: o Discuss WFP strategies, policies or systems in their area of responsibility and subject of evaluation. o Comment on the evaluation TOR, inception and evaluation reports, as required. 85.The Office of Evaluation (OEV). OEV, through the Regional Evaluation Officer, will advise the Evaluation Manager and provide support to the evaluation process when required. It is responsible for providing access to the outsourced quality support service reviewing draft ToR, inception and evaluation reports from an evaluation perspective. It also ensures a help desk function upon request. 8. Communication and budget 8.1. Communication 86.To ensure a smooth and efficient process and enhance the learning from this evaluation, the evaluation team should place emphasis on transparent and open communication with key stakeholders. These will be achieved by ensuring a clear agreement on channels and frequency of communication with and between key stakeholders. 87.As part of the international standards for evaluation, WFP requires that all evaluations are made publicly available. Following the approval of the final evaluation report, The CO will translate the final TOR and report in Portuguese. Final evaluation products of the evaluation will be disseminated or made available to partners in electronic and print form. 8.2. Budget 88.For the purpose of this evaluation, the budget will be based on pre-agreed rates with long-term agreement evaluation firms. Firm engagement for the final evaluation is dependent upon satisfactory Evaluation Report Version March 2021 91 | P a g e completion of the mid-term evaluation. The evaluation budget is planned under Mc-Govern-Dole contribution. 89.The evaluation budget should include costs associated with international travel and daily subsistence. Local travel will be supported by the Country Office. Please send any queries to Elber Nosolini , NPO, at elber.nosolini@wfp.org, +245 95 565 17 29]. Evaluation Report Version March 2021 92 | P a g e Annex 14 - Documents Reviewed Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Appraisal mission report N Project document (including Logical Framework in Annex) Project agreement between The Foreign Agricultural Dervice and the World Food Programme For the donation of agricultural commodities and related assistance under the MCgovern-Dole international food for education and child nutrition programme. (2015) Y Standard Project Reports Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Y Budget Revisions N Note for the record (NFR) from Programme Review Committee meeting (for original operation and budget revisions if any) N Approved Excel budget (for original intervention and budget revisions if any) Annex C in : Project agreement between The Foreign Agricultural Dervice and the World Food Programme For the donation of agricultural commodities and related assistance under the MCgovern-Dole international food for education and child nutrition programme. (2015) Y Intervention/Project Plan (breakdown of beneficiary figures and food requirements by region/activity/month and partners) N Other Country Office Strategic Documents (if applicable) Country Strategy Document Country Programme Guinea-Bissau 200846 (2016–2020) Feb (2016) Y Other Guinea-Bissau INTERIM Country Strategic Plan 2018-2019 The WFP Guinea-Bissau PRRO 200526(2013-2016) Transitional ICSP and project Budget revision for T-ICSP for approval by the executive Director (2017) Guinea-Bissau Country Strategic Plan - GW01 Logframe Assessment Reports Evaluation Report Version March 2021 93 | P a g e Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Comprehensive Food Security and Vulnerability Assessments NA Crop and Food Security Assessments (FAO/WFP) WFP: resume executif resultats de l’enquete approfondie sur la securite alimentaire en milieu rural 2011 N Emergency Food Security Assessments Guinée Bissau - Enquête de suivi de la Sécurité Alimentaire et de la Nutrition, 2017 Guinée Bissau - Enquête de suivi de la Sécurité Alimentaire et de la Nutrition, 2016 N Food Security Monitoring System Bulletins West Africa and the Sahel - Food Security and Humanitarian Implications, 2015 N Market Assessments and Bulletins West Africa - Regional Supply and Market Outlook, December 2017 N Joint Assessment Missions (UNHCR/WFP) N/A Rapid needs assessments Guinea Bissau - Rapid Food Security Assessment, June 2012 N Monitoring & Reporting (if applicable) M&E Plan Mentioned in annex 7 of project agreement. Y Country Situation Report (SITREP) N Country Executive Brief WFP Guinea-Bissau Country Brief 2017 N Food Distribution and Post-distribution Monitoring Reports N Monthly Monitoring Reports N Beneficiary Verification Reports N Donor specific reports Follow Up on Status of McGovern-Dole Guinea-Bissau Project Activities Amendment Request WFP Guinea-Bissau McGovern-Dole (FFE-657-2015/019- 00) (2017) Evaluation Report Version March 2021 94 | P a g e Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Implementation of USDA McGovern Dole International Food for Education and Child Nutrition Programme in Guinea-Bissau (25 August 2016) Output monitoring reports (if applicable) Actual and Planned beneficiaries by activity and district/ location by year Standard Project Report WFP Country Project 200825 (2017) Standard Project Report WFP Country Project 200825 (2016) Y Male vs. women beneficiaries by activity and district/ location by year Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Beneficiaries by age group Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Actual and Planned tonnage distributed by activity by year Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Commodity type by activity Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Actual and Planned cash/voucher requirements (US$) by activity by year Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Operational documents Organization chart for main office and sub-offices Terms of Reference Mid-term and Final Evaluations of the McGovern-Dole International Food for Education and Child Nutrition Programme project in Guinea-Bissau (2016 - 2018) Activity Guidelines Project agreement between The Foreign Agricultural Service and the World Food Programme For the donation of agricultural commodities and related assistance under the MCgovern-Dole international food for education and child nutrition programme. (2015) Y Mission Reports Semiannual Report Narrative April 1, 2016 – September 30, 2016 Semiannual Report Narrative October 1, 2016 – March 31, 2017 Semi-annual Report for April to September 2017Guinea-Bissau Guinea-Bissau Semiannual Report Narrative October 2017 – March 2018 Pipeline overview for the period covered by the evaluation Implementation of USDA McGovern Dole’s International Food for Education and Child Nutrition Programme in Guinea-Bissau Evaluation Report Version March 2021 95 | P a g e Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Partners (if applicable) Annual reports from cooperating partners Semiannual Report Narrative April 1, 2016 – September 30, 2016 Semiannual Report Narrative October 1, 2016 – March 31, 2017 Semi-annual Report for April to September 2017Guinea-Bissau Guinea-Bissau Semiannual Report Narrative October 2017 – March 2018 Y Cluster/ Coordination meetings Logistics/Food Security/nutrition cluster documents NA NFRs of coordination meetings N Other Evaluations/ Reviews Evaluations/ reviews of past or on-going operation School feeding baseline survey, Guinea-Nissau 2016 Resource mobilisation Resource Situation Resource Situation Project No 200846 (31-12-2017) Y Resource mobilization strategy Implementation of USDA McGovern Dole’s International Food for Education and Child NUTRITION PROGRAMME IN GUINEA-BISSAU Y Maps (if applicable) Food Security Map Guinée Bissau - Enquête de suivi de la Sécurité Alimentaire et de la Nutrition, 2016 N Other documents collected by the team (including external ones) National Programme of education Programme Sectoriel de l’Education de la Guinée Bissau (2017 -2025) N Evaluation Report Version March 2021 96 | P a g e Annex 15 – Documents gathered Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Appraisal mission report N Project document (including Logical Framework in Annex) Project agreement between The Foreign Agricultural Dervice and the World Food Program For the donation of agricultural commodities and related assistance under the MCgovern-Dole international food for education and child nutrition program. (2015) Y Standard Project Reports Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Y Budget Revisions N Note for the record (NFR) from Programme Review Committee meeting (for original operation and budget revisions if any) N Approved Excel budget (for original intervention and budget revisions if any) Annex C in : Project agreement between The Foreign Agricultural Dervice and the World Food Program For the donation of agricultural commodities and related assistance under the MCgovern-Dole international food for education and child nutrition program. (2015) Y Intervention/Project Plan (breakdown of beneficiary figures and food requirements by region/activity/month and partners) N Other Country Office Strategic Documents (if applicable) Country Strategy Document Country Programme Guinea-Bissau 200846 (2016–2020) Feb (2016) Y Other Guinea-Bissau INTERIM Country Strategic Plan 2018-2019The WFP Guinea-Bissau PRRO 200526(2013-2016) Transitional ICSP and project Budget revision for T-ICSP for approval by the executive Director (2017) Guinea-Bissau Country Strategic Plan - GW01 Logframe Y Assessment Reports Evaluation Report Version March 2021 97 | P a g e Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Comprehensive Food Security and Vulnerability Assessments NA Crop and Food Security Assessments (FAO/WFP) WFP: resume executif resultats de l’enquete approfondie sur la securite alimentaire en milieu rural 2011 N Emergency Food Security Assessments Guinée Bissau - Enquête de suivi de la Sécurité Alimentaire et de la Nutrition, 2017 Guinée Bissau - Enquête de suivi de la Sécurité Alimentaire et de la Nutrition, 2016 N Food Security Monitoring System Bulletins West Africa and the Sahel - Food Security and Humanitarian Implications, 2015 N Market Assessments and Bulletins West Africa - Regional Supply and Market Outlook, December 2017 N Joint Assessment Missions (UNHCR/WFP) N/A Rapid needs assessments Guinea Bissau - Rapid Food Security Assessment, June 2012 N Monitoring & Reporting (if applicable) M&E Plan Mentioned in annex 7 of project agreement. N Country Situation Report (SITREP) N Country Executive Brief WFP Guinea-Bissau Country Brief 2017 N Food Distribution and Post-distribution Monitoring Reports N Monthly Monitoring Reports N Beneficiary Verification Reports N Donor specific reports Follow Up on Status of McGovern-Dole Guinea-Bissau Project Activities Amendment Request WFP Guinea-Bissau McGovern-Dole (FFE-657- 2015/019-00) (2017) Implementation of USDA McGovern Dole International Food for Education and Child Nutrition Program in Guinea-Bissau (25 August 2016) Y Evaluation Report Version March 2021 98 | P a g e Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Output monitoring reports (if applicable) Actual and Planned beneficiaries by activity and district/ location by year Standard Project Report WFP Country Project 200825 (2017) Standard Project Report WFP Country Project 200825 (2016) Y Male vs. women beneficiaries by activity and district/ location by year Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Beneficiaries by age group Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Y Actual and Planned tonnage distributed by activity by year Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Y Commodity type by activity Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Y Actual and Planned cash/voucher requirements (US$) by activity by year Standard Project Report WFP Country Project 200825 (2016) Standard Project Report WFP Country Project 200825 (2016) Y Operational documents Organization chart for main office and sub-offices Terms of Reference Mid-term and Final Evaluations of the McGovern￾Dole International Food for Education and Child Nutrition Program project in Guinea-Bissau (2016 - 2018) Y Activity Guidelines Project agreement between The Foreign Agricultural Service and the World Food Program For the donation of agricultural commodities and related assistance under the MCgovern-Dole international food for education and child nutrition program. (2015) Y Mission Reports Semiannual Report Narrative April 1, 2016 – September 30, 2016 Semiannual Report Narrative October 1, 2016 – March 31, 2017 Semi-annual Report for April to September 2017Guinea-Bissau Guinea-Bissau Semiannual Report Narrative October 2017 – March 2018 Y Pipeline overview for the period covered by the evaluation Implementation of USDA McGovern Dole’s International Food for Education and Child Nutrition Program in Guinea-Bissau Y Logistics capacity assessment N Partners (if applicable) Evaluation Report Version March 2021 99 | P a g e Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Annual reports from cooperating partners Semiannual Report Narrative April 1, 2016 – September 30, 2016 Semiannual Report Narrative October 1, 2016 – March 31, 2017 Semi-annual Report for April to September 2017Guinea-Bissau Guinea-Bissau Semiannual Report Narrative October 2017 – March 2018 Y List of partners (Government, NGOs, UN agencies) by location/ activity/ role/ tonnage handled N Field level agreements (FLAs), Memorandum of Understanding (MOUs) N Cluster/ Coordination meetings Logistics/Food Security/nutrition cluster documents NA NFRs of coordination meetings N Other Evaluations/ Reviews Evaluations/ reviews of past or on-going operation School feeding baseline survey Guinea-Bissau 2016 Y Resource mobilisation Resource Situation Resource Situation Project No 200846 (31-12-2017) Y Contribution statistics by month N Resource mobilization strategy Implementation of USDA McGovern Dole’s International Food for Education and Child NUTRITION PROGRAM IN GUINEA-BISSAU Y NFRs Donor meetings Maps (if applicable) Operational Map N Logistics Map N Food/Cash/voucher Distribution Location Map N Evaluation Report Version March 2021 100 | P a g e Document Type Comment / Titles & dates of documents received Received - Y/N (N/A) Link to Evaluation matrix Project related documents Food Security Map Guinée Bissau - Enquête de suivi de la Sécurité Alimentaire et de la Nutrition, 2016 N Other documents collected by the team (including external ones) National Programme of education Programme Sectoriel de l’Education de la Guinée Bissau (2017 -2025) N Specify Evaluation Report Version March 2021 101 | P a g e Annex 16 - Data Collection Tools 12.1 List of topics to be used during school visits (Qualitative Survey) INTERVIEWS AT NATIONAL LEVEL / ENTREVISTAS AO NÍVEL NA￾CIONAL 1. How has this project been implemented since its beginning? /Como esse projeto tem sido implementado desde o início? 2. What were the main successes and difficulties?/Quais seus principais sucessos e dificuldades? 3. Who is responsible for monitoring the various Project’s activities? Quem é responsável pelo monitoramento das diversas atividades do Projeto? 4. Directed to WFP: How does the WFP follow up/monitores activities that are implemented by partners of the project (Ministry of Education, UNICEF, FAO, WHO, NGOs)? Dirigido ao PAM: Como é que o PMA acompanha/monitora as atividades do Projeto que são implementadas por parceiros de Projeto (Ministério da Educação, UNICEF, FAO, OMS e ONGs)? 5. How often are the Project performance indicators collected? By whom (WFP, DGPASE, others)? Com que frequência os indicadores de performance do projeto são recolhidos? por quem (WFP, DGPASE, outros)? 6. Does this Project incorporate guidelines/elements of Guinea-Bissau’s school feeding law? And to the Education policy/plan? How? / Esse projeto está alinhado aos princípios da Lei das Cantinas Escolares de Guiné Bissau? E à Política/ao Plano de Educação? Como? 7. Is this Project integrated to other programs/projects of your institution? Which projects/programmes? How? Esse projeto está alinhado/integrado a outros projetos/programas de sua instituição? Quais projetos/programas? Como? 8. Are there pilot initiatives on local procurement from smallholder farmers/farmers cooperatives for supplying the school feeding programme? If so, how do these work: how many schools are targeted and where, how is the collaboration (roles, responsibilities and cooperation) with MoE, MoA , FAO and other partners on this initiative? What are the mechanisms in place to control the local procurement (vouchers?) What are the results of these experiences?/ Há experiências piloto de compra de alimentos produzidos localmente por pequenos agricultores/cooperativas ? Se sim, como é que esta atividade funciona: quantas escolas participam, onde essas escolas se localizam, como é a cooperação com os parceiros (papéis, responsabilidades, e coordenação)? Quais são os mecanismos utilizados para controlar as compras locais («vouchers» ?) Quais são os resultados destas experiências ? 9. What was the contribution of the McGovern Dole-FFE programme to these local procurement intiatives ? Qual foi a contribuição do programa McGovern Dole-FFE a estas iniciativas de compra local de alimentos ? 10. Regarding to the following activities, please tell us about your successes and difficulties. Is the activity implemented according to the plan (target, approach)? Are the expected results being achieved ? Have these activities been implemented in the most efficient way? Em relação às atividades a seguir, indique quais foram os seus sucessos e dificuldades. As atividades estão sendo implementadas de acordo com o planeamento (meta, abordagem)? O resultado desejado está sendo alcançado? Estas atividades foram implementadas da maneira mais eficiente? Evaluation Report Version March 2021 102 | P a g e a) Provision of daily school meals/ Distribuição de refeições escolares b) Provision of take-home rations (to female pupils) / Distribuições de «rações secas» de gêneros alimentícios para levar à casa (para raparigas) c) Trainings on food management and storage for School Management Committees, Parent Associations, Headmasters, Inspectors / Treinamentos em gestão da alimentação escolar (em nível escolar) para Comitês de Gestão Escolar, Associações de Pais de Alunos, Diretores e Inspetores escolares d) Trainings on food preparation and storage practices for school cooks/ Treinamentos sobre práticas de armazenamento e preparo de alimentos, para cozinheiras das escolas e) Building and rehabilitating kitchens and storerooms/ Construção e reabilitação de cozinhas e armazéns/dispensas escolares f) Providing storage, food preparation equipment and eating utensils / Fornecimento de equipamentos para o armazenamento e o preparo de alimentos e utensílios para consumo de refeições escolares (talheres e pratos) g) Distributing deworming medication to students/Distribuição de vermífugos/desparasitantes aos alunos h) Capacity building / Desenvolvimento de capacidade a) at national level/ao nível nacional b) b) at regional level/ao nível regional c) at local level (district)/ao nível local 11. Erros de inclusão / de exclusão? Os critérios de seleção excluem as crianças mais vulneráveis das escolas mais pobres?(o Baseline Report diz isto). Os critérios de seleção das escolas não melhoraram desde a definição das 758 escolas (o Very First semi-annual report diz isto). Como correu a transição do IPHD para o PAM? Melhorou a questão da inclusão/exclusão? 11. If there was an opportunity to adjust future programmes regarding its various activities and targets, what changes would you recommend? Why ? / Se houvesse uma oportunidade de ajustar programas futuros em relação às diversas atividades previstas e às suas metas, qual seriam as suas recomendações ? Por quê ? 12. In your opinion were the targets that were established (for each planned activity) realistic? Na sua opinião, as metas estabelecidas para cada atividade foram realistas? 13. Regarding to the challenges faced, tell us about the underlying causes. How could these difficulties be solved?/ Em relação aos desafios enfrentados, conte-nos sobre suas causas subjacentes. Como poderiam ser superadas essas dificuldades? a) For the SF operations based on WFP (procurement and delivery of school meals and take-home rations)/ Para a implementação das atividades de alimentação escolar pelo PMA (aquisição de alimentos para o preparo de refeições escolares e para as rações secas para levar à casa . b) For the capacity building activities (at national, regional and local level)/Para o desenvolvimento/ fortalecimento da capacidade do governo (aos níveis nacional, regional e local) c) For monitoring and evaluating/Para o monitoramento/monitoria e avaliação Evaluation Report Version March 2021 103 | P a g e 14. Regarding the WFP partnership with the GoGB, what is the role/participation in the implementation and monitoring and evaluating of the Project activities / Em relação à parceria entre o PMA e o Governo de Guiné Bissau, qual o papel/responsabilidade: a) The Ministry of Education/National Directorate of School Feeding (NDSF)/ Ministério da Educação / DGPASE b) The Ministry of Agriculture/ Ministério da Agricultura c) The Ministry of Health (deworming activities)/Ministério da Saúde (distribuição de vermífugos/desparasitantes) d) District governments/Governos distritais 15. Please explain us the financial aspects of this project/ Por favor, nos explique os aspectos financeiros deste projeto: a) Successes and difficulties on financing the activities? Was the budget realistic? Is there any balance left? How used? / Sucessos e dificuldades (no financiamento das atividades)? O orçamento era realista? Ficou um saldo? Como foi utilizado? b) Outlook to the immediate future/Perspectivas para o future imediato ? 16. How is the collaboration between WFP and project partners, such as/Como tem sido a colaboração entre o PAM e os principais parceiros de projeto, tais como a) Ministry of Education/Ministério da Educação b) Ministry of Health /school health and nutrition services/Ministério da Saúde/serviço de nutrição e saúde escolar c) UNICEF: provision of didactic materials, teachers’ trainings, potable water and sanitation improvements and improvements of school structures/UNICEF: fornecimento de materiais didáticos, treinamentos de professores, melhorias nas condições sanitárias e de água potável em escolas, e melhorias nas estruturas escolares em geral d) FAO: support for school gardens and partner in the implementation of the Home Grown School Feeding approach through the Country Programme/FAO: assistência em hortas escolares, e parceria na implementação/abordagem sustentável do programa alimentação escolar (Home Grown School Feeding) no Programa-País? e) WHO: providing deworming medication/ OMS: fornecimento de vermífugos/desparasitantes f) NGOs (partners) that are responsible for sensitizing communities and parents’ associations to build school kitchens / ONGs (parceiras) que são responsáveis para sensibilizar comunidades e associações de pais de alunos para a construção de cozinhas escolares Extra: O programa logrou êxitos no seu grande objetivo de melhorar o nível de literacía dos alunos, ou falhou no alcançe deste objetivo? (Como se mede a literacía?) Porque teve sucesso/falhou? 17. What are the contributions of the McGovern-Dole FFE programme towards the development of a national school feeding programme (HGSF)?/Quais as contribuições desse projeto de alimentação escolar (MGD) para o desenvolvimento de um programa nacional de alimentação de Guiné-Bissau? Any perspectives for a future programme ? Perspetivas para um programa futuro ? Evaluation Report Version March 2021 104 | P a g e 18. How do you evaluate the perspectives of GB towards/Como você avalia as perspectivas da GB em relação a: a) Funding SF in Guinea-Bissau after the completion of this project?/ Finan-ciamento da alimentação escolar em Guiné Bissau após o término deste projeto? b) The GoGB establishing and running their own Home-Grown School Feeding Programme? O governo de Guiné Bissau estabelecendo e gerindo seu programa nacional de alimentação escolar sustentável? c) The GoGB regularly monitoring and evaluating the SF activities, and timely issuing monitoring reports? O governo de Guiné Bissau monitorando e avaliando regularmente as atividades de alimentação escolar, e emitindo os relatórios de avaliação como regularidade. 19. How is the institutional coordination for implementing the MGD/WFP school feeding Project?/Como é a coordenação institucional para a implementação do projeto de alimentação escolar PMA/MGD? Which improvements do you see possible? Quais melhorias acha possível? a) Coordination among Guinea-Bissau WFP CO and its sub-offices/ Coordenação entre do Escritório do País, do PMA, e seus escritórios regionais. b) Coordination among WFP and its partners (DGPASE, DGASCE, UNICEF, FAO, WHO, NGOs)/ Coordenação entre PMA e seus parceiros (DGPASE, DGASCE, UNICEF, FAO, OMS e ONGs) c) Coordination among Education Sector (national, regional, local) and school level (committees and community) d) Coordenação entre os níveis do Setor da Educação (nacional, regional e local) e nível escolar (comitês e comunidade) e) Coordination inter-sectorial (Education, Agriculture, Health, Finance and Planning)/Coordenação intersetorial (Educação, Agricultura, Saúde, Finanças e Planeamento) 20. What are the Programme’s outcomes in terms of gender equality and women’s empowerment? Could it have been more ambitious? How? Quais são os resultados do Programa em relação a igualdade de gêneros e empoderamento das mulheres? Poderia o programa ter sido mais ambicioso? Como? 21. Which improvements would you wish to see in future SF programmes in Guinea￾Bissau? /Quais melhorias você gostaria de ver em futuros programas de cantinas escolares na Guiné Bissau? 22. How is the WFP/MGD school feeding project monitored and evaluated at central level? Are the results of the project reported to other government (beyond the Ministry of Education) institutions and project partners? How often?/ Como o projeto de alimentação escolar do PMA/MGD é monitorado e avaliado a nível central? Os resultados do projeto são relatados e divulgados para outras instituições (além do Ministério da Educação) e para parceiros do projeto? Com que frequência? 23. In your opinion, is the existing WFP M&E of school feeding programme efficient (performance indicators are timely monitored and reported)? And for monitoring other activities planned on the MGD/WFP SF project? How the M&E of the McGovern-Dole FFE programme could be improved? / Em sua opinião o sistema de Monitoramento e Avaliação utilizado no programa de alimentação escolar do PMA é eficiente (os indicadores de desempenho do projeto são regularmente monitorados)? E para o monitoramento das Evaluation Report Version March 2021 105 | P a g e outras atividades planeadas no projeto de alimentação escolar do MGD/PMA? Como o monitoramento e a avaliação do projeto de alimentação escolar do MGD/PMA poderiam ser melhorados? 2. List of specific topics / Tópicos específicos (nível nacional) 2.1. INTERVIEWS WITH MNEJCD AND WITH THE DGASCE 1. Are you satisfied with the way that the McGovern Dole FFE programme was implemented? Está satisfeito com o programa McGovern Dole FFE, tal como foi implementado? 2. What difficulties did you experience during the implementation of this programme? Quais as dificuldades que vocês encontraram durante a implementação deste programa? 3. Was strengthening of the MNEJCD’s capacities in relation to SF sufficiently present in this programme? O reforço das capacidades do MNEJCD estava suficientemente presente no programa que agora terminou? 4. In what ways could partners like the WFP contribute to the improvement of the pedagogical quality of primary education in GB, with SF as a tool? Has the MGD FFE programme failed in its objective to improve the literacy level of the pupils? Why? How can one measure literacía? De que maneira parceiros como o PMA poderiam contribuir na melhoria da qualidade (pedagógica) do ensino primário na GB, em paralelo à alimentação escolar ? O programa FFE falhou no melhoramento dos níveis de literacía dos alunos? Porquê? Como se pode medir o nível de literaçía? 5. What improvements would you propose in relation to possible future similar programmes? Quais melhorias gostaria de propor em relação a possíveis futuros programas do mesmo tipo? 6. Existem planos para extender as cantinas escolares para todas as escolas do país? Qual é o grau de cobertura agora? Quantas escolas primárias existem no total? 2.2. INTERVIEWS WITH DONORS (ACTUAL AND/OR POTENTIAL) 1. Are you satisfied with the results obtained by the school feeding programme (to which you contributed financially)? Would you be interested in financially contributing to the programme in the future? Why? In what form? Está satisfeito com os resultados obtidos pelo programa de cantinas escolares (ao qual contribuiu financeiramente)? Estaria interessado em continuar a vossa contribuição (financeira) no futuro? Por quê? De que forma? 2. For future contributions to school feeding in GB, would you wish to see changes in the way of operating of WFP, MNEJCD and its DGASCE? Em relação a futuras contribuições à alimentação escolar em GB, gostaria de ver alterações no modo de funcionar do PMA, do MNEJCD e a sua DGASCE? 2.3. INTERVIEWS WITH OTHER UN ORGANIZATIONS: UNICEF, FAO, UNESCO, etc. 1. How is the coordination/collaboration between the different UN organizations functioning? Como funciona a coordenação/colaboração entre as diferentes organizações das NNUU? 2. Com UNICEF: como funciona o vosso programa de child-friendly schools? A combinação deste programa com cantinas escolares (do PAM) resulta numa sinergia que se pode constatar e provar estatísticamente ¿ Evaluation Report Version March 2021 106 | P a g e 3. Com a FAO: a importação de gêneros alimentícios pelo PAM não leva a distorções do mercado interno? Prejudicando os produtores locais? 2. How would you describe your collaboration/coordination with the WFP and with the MNEJCD (in relation to the school feeding programme)? What were the weak points? Any improvements you could suggest? Como pode caraterizar a vossa colaboração/coordenação com o PMA e com o MNEJCD (em relação à alimentação escolar)? Quais os pontos fracos que existiam? Poderia fazer sugestões para os melhorar? 2.4. Interviews with local and international NGOs and organizations in partnership with WFP / Entrevistas com ONGS LOCAIS E INTERNACIONAIS PARCEIRAS do PMA. 1. Are you satisfied about your work in this partnership and the partnership itself? What improvements would be posible? Estão satisfeitos do vosso trabalho no âmbito da parceria e da vossa parceria com o PMA? Que melhorias seriam possíveis? 2. What is your opinion about the role of school feeding in the reduction of rural poverty? Would you have any recommendation as to strategy? Qual a vossa opinião sobre o papel da alimentação escolar no alívio da pobreza rural? Teria alguma recomendação acerca da estratégia a seguir? 3. O programa falhou do reforço da literacía? Teria elementos para contribuir à análise e à explicação deste fenómeno? O que é realmente necessário fazer para realizar um aumento da literaçía dos alunos? I – INTERVIEWS AT LOCAL LEVEL / ENTREVISTAS AO NÍVEL LOCAL A. ESCRITÓRIO REGIONAL DO PAM 1. Qual a participação do escritório local do PAM no planejamento, implementação e monitoramento das atividades do WFP/MGD? 2. Qual o tipo de colaboração dos diferentes parceiros institucionais, incluindo governo? 3. Como é feita a articulação entre o PMA e o governo local, a UNICEF, a FAO, a OMS, e as NGOs parceiras, para coordenação das actividades desenvolvidas no âmbito desse projeto? Há outros intervenientes? (treinamentos, construções e reformas nas estruturas escolares, distribuição de vermífugos/desparasitantes, sensitização/sensibilização e formação de camponeses/agricultores, etc..) 4. Como é feito o acompanhamento/monitoria das actividades desenvolvidas pelos parceiros do projeto nas escolas? E com os agricultores? 5. Como funciona, ao nível local, o sistema de financiamento das compras locais (distribuição de vouchers às escolas, para a aquisição de alimentos produzidos pelos agricultores locais?) Em sua avaliação, quais os resultados dessa atividade (sucessos e dificuldades)? 6. Quais as experiências bem sucedidas? E as dificuldades e/ou fracassos na implementação desse projeto (desde 2016)? 7. O que está contribuindo para o sucesso do programa na região? E o que está causando dificuldades ou levando ao fracasso do mesmo? Evaluation Report Version March 2021 107 | P a g e 7bis: Porque é que o projeto falhou no aumento da literacía dos alunos? Como melhorar o programa a este respeito? 8. Como você avalia a participação/engajamento da comunidade local nas atividades do WFP/MGD? E do governo local? 9. Como você considera os resultados/contribuição desse programa para a qualidade de vida da população que se beneficia do mesmo (homens e mulheres, meninos e meninas)? 10. Caso fosse possível ajustar as atividades e a forma operacional de projeto para um melhor desempenho, o que você recomendaria? B. MEMBROS DOS GOVERNOS REGIONAL E DISTRITAL (EDUCAÇÃO E OUTROS SETORES ENVOLVIDOS) 1. Como você percebe o programa de cantinas escolares PMA/MGD, isto é, quais as atividades desenvolvidas e quem são os responsáveis pelas mesmas? 2. Qual o papel do governo regional/distrital no atual programa das cantinas escolares (WFP/MGD)? 3. De que forma o programa cantinas escolares está inserido na (ou coordenado com) política/estratégia/ plano regional/distrital do governo local? E no plano da educação? 4. Há alguma linha de financiamento dentro do orçamento da região/distrito para participar no custo das atividades desse programa? 5. Como é feita a articulação entre os sectores da educação, agricultura e saúde para o desenvolvimento das atividades como distribuição de medicação anti-verme, assistência técnica a agricultores, e a outras atividades intersetoriais? Existe algum tipo de coordenação entre o governo local para esse fim? 6. Quais os resultados do programa das cantinas escolares nas escolas da região/distrito (experiências bem sucedidas e dificuldades/fracassos)? (distinguir a parte do MGD FFE do resto do PAM e do programa realizado pela própria DGASCE) 6bis: A estatística mostra que não houve melhoria no nível de literacía dos alunos. Porquê é que o programa falhou em atingir esse objetivo? 7. Caso haja dificuldades ou até fracassos, quais foram as causas ? O que deveria ser feito para evitar que esses problemas se repitam, ou que novos problemas surjam no futuro? 8. Os membros do governo receberam algum tipo de treinamento para desenvolver as atividades relacionadas ao programa de cantinas escolares? Se recebeu, sente necessidade de treinamento adicional? Se sim, em que aspectos? 9. O que poderia ser feito para melhorar o desempenho desse projeto na região/distrito? 10. Existem planos regionais para o alargamento das cantinas escolares a todas as escolas da região ? Quais escolas são excluídas agora? Porquê? C. INSPETORES ESCOLARES 1. Qual é a vossa função? Quais as atividade desenvolvidas pelos inspetores da educação em relação às cantinas escolares? E em relação ao programa WFP/MGD FFE ? Evaluation Report Version March 2021 108 | P a g e 2. Com que frequência cada escola é visitada pelos inspetores? O que é observado na visita? Com que frequência os relatórios de visita são emitidos e divulgados para outros setores dentro do departamento/ministério de educação? 3. Houve/há algum tipo de assistência técnica/treinamento/outro tipo de apoio por parte do PMA para a participação dos inspetores da educação nas atividades desenvolvidas pelo Projeto? Se sim, que tipo de assistência/treinamento? 4. Como você percebe os resultados desse programa/projeto, ou seja sucessos e dificuldades em relação ao bom funcionamento das cantinas escolares? Inclusive a falha do programa em melhorar o nível de literaçía dos alunos? Como se poderia melhorar esta literacía? 5. E em relação às rações secas, há algum tipo de participação/acompanhamento dos inspetores escolares nessa atividade? Se sim, qual? Vossa opinião sobre a efetividade destas rações secas? 6. Em sua opinião, qual o resultado da alimentação escolar para a qualidade da educação e condições de vida dos alunos das escolas beneficiárias? 7. O que poderia ser feito para tornar mais eficiente a participação dos inspetores escolares nesse projeto? 8. Como é controlada (pelos inspetores) a utilização dos vouchers para a aquisição de alimentos? 9. O que poderia ser feito para melhorar o programa das cantinas escolares e os resultados destas cantinas? D. PROFESSORES E DIRETORES 1. Como tem sido a experiência da vossa escola com o projeto de Cantinas Escolares do PMA/MGD (desde 2016)? Quais os sucessos, dificuldades ou fracassos ocorridos no mesmo? 2. Desde o início desse programa de cantinas escolares (Janeiro 2016), houve algum dia ou período quando não foram distribuídas refeições aos alunos? Se sim, qual foi o motivo? 3. Se houve interrupção na oferta de refeições, foi tomada alguma medida pela escola para evitar que isso se repita? Se sim, o quê? 4. Qual a rotina de recebimento e armazenamento de alimentos, controle de estoque, preparo e distribuição de refeições? (Passo por passo?) Tem havido alguma iregularidade, algum conflito, na gestão dos alimentos ? 5. Há alguma diferença entre a quantidade de alimentos distribuída para meninos e raparigas? Quem recebe primeiro as refeição ( classe/idade, gênero, etc..) ? 6. Em relação às rações secas ("take-home") distribuídas pelo PMA para ser levadas à casa : quem recebe estas rações, como é feito o controle dos/das beneficiário/as ? 7. Qual a participação/responsabilidade do diretor e professores nas atividades relacionadas ao programa de cantinas escolares? (incluir uma análise de género: proporção e funções de homens e mulheres) 8. Os professores e o diretor participam nas compras locais para as cantinas escolares? (na utilização dos vouchers para a aquisição de alimentos produzidos localmente? ) (incluir uma análise de género: proporção e funções de homens e mulheres) Evaluation Report Version March 2021 109 | P a g e 9. Os professores/diretor receberam algum tipo de treinamento para desenvolver as atividades no projeto? Se sim, em quê? O treinamento foi suficiente? (incluir uma análise de género: proporção e funções de homens e mulheres) 10. Em sua opinião, quais os resultados do projeto para a qualidade da educação nessa escola? E para os alunos? E para a comunidade em geral? A literacía dos alunos melhorou? Porquê não ? 11. Caso fosse possível fazer ajustes nas atividade do projeto para melhorar o desempenho/resultados do mesmo, o que você recomendaria? 12. Como é o nível de formação dos professores na sua/vossa escola? como é a qualidade do ensino? Há materiais didáticos, livros escolares, cadernos, quadro preto, etc.? 13. As casas de banho oferecem condições mínimas de segurança às meninas? E. Comité de Gestão Escolar e Associação de Pais de Alunos 1. Qual a composição e participação do comité de gestão escolar e associação de pais de alunos nas atividades de alimentação escolar da escola? (incluir uma análise de género do comité e da associação de pais: proporção e funções de homens e mulheres) 2. Quais os cuidados aplicadas nas rotinas operacionais para garantir a boa qualidade e higiene da alimentação oferecida nas escolas? E uma gestão honesta, correta? 3. Ao decorrer do trabalho, têm havido casos de alguma iregularidade, desvio, ou distribuição indevida de alimentos? Ou de uso discutível do dinheiro para compras locais? 4. Os membros do vosso comitê receberam algum tipo de treinamento para desenvolver as atividades relacionadas ao programa de cantinas escolares? Se recebeu, sente necessidade de treinamento adicional? Se sim, em que aspectos? (incluir uma análise de género: proporção e funções de homens e mulheres) 5. Com que frequência esse comitê/membros do comitê desenvolvem atividades na escola, relacionadas ao programa de cantinas escolares? (incluir uma análise de género: proporção e funções de homens e mulheres) 6. Há algum tipo de comunicação formal entre esse comitê e os inspetores das escolas/ oficiais do governo local em relação às atividades desenvolvidas pelas cantinas escolares nas escolas (reunião, emissão de relatório ou outro tipo de documento)? 7. No caso das escolas piloto: como funciona na vossa escola o sistema financeiro para pagar os produtores locais de alimentos (vouchers ?) (planejamento, rotinas de controle, etc.). 8. Em sua opinião, quais são os resultados do programa cantinas escolares (projeto MGD) para a qualidade da educação e na vida dos alunos? Quais seus sucessos e dificuldades? Falhou na melhoria do nível de literacía, como se poderia fazer para que seja mais efetivo neste respeito? 9. Em sua opinião, de que forma esse projeto contribui/afeta (ou não) a vida da comunidade ao entorno dessa escola (famílias de alunos, moradores da localidade, agricultores e comerciantes locais)? 10. Caso houvesse possibilidades de ajustes nas atividades do vosso comitê em contribuição às atividades da cantina escolar, o que deveria ser feito? 11. Caso houvesse possibilidade de modificar/ajustar a forma como o programa cantinas escolares é posto na prática/conduzido nessa escola para melhorar seus resultados, o que deveria ser feito? Quais seriam os seus conselhos ou recomendações ? Evaluation Report Version March 2021 110 | P a g e F. COZINHEIRAS 1. Quantas cozinheiras trabalham nessa escola? são voluntárias ? Há algum tipo de incentivo/indemnização/pagamento para as cozinheiras? 2. Como é o sistema de trabalho, todas as cozinheiras trabalham da segunda à sexta ou há algum tipo de rotação? 3. Quais são as atividades das cozinheiras no programa cantinas escolares? Qual sua jornada de trabalho (quando começa e quando termina o trabalho na cozinha)? 4. Você recebeu algum tipo de treinamento para ser cozinheira nas cantinas escolares? O que você aprendeu nesse treinamento? Se recebeu, sente necessidade de treinamento adicional? Se sim, em que aspectos? 5. Qual é a sua opinião sobre a alimentação oferecida aos alunos: a quantidade é suficiente? e o que você pensa da qualidade da alimentação oferecida? Alguma recomendação em relação a quantidade ou qualidade da alimentação? 6. Qual é a rotina de distribuição de alimentos, passo por passo ? Rapazes e raparigas recebem a mesma quantidade de alimento? Quem é servido em primeiro lugar? 7. Que tipos de alimentos são usados regularmente? Caso haja a utilização de alimentos que não são distribuídos pelo PMA, de onde vêm esses alimentos (doações, horta escolar, comprado com dinheiro ou com vouchers ? 8. Qual é a sua opinião em relação às compras locais, se isto existir na vossa escola (adição de alimentos frescos, produzidos pelos agricultores locais, aos produtos secos (feijão, arroz, óleo e sal, distribuídos pelo PMA) nas refeições escolares? (benefícios e dificuldades, necessidades) 9. Você enfrenta alguma dificuldade para preparar os alimentos e distribuir as refeições aos alunos? Se sim, qual (quais)? 10. A cozinha tem os equipamentos e utensílios em quantidade e qualidade necessária para o preparo e a distribuição dos alimentos? 11. Quais os cuidados de higiene no preparo e distribuição dos alimentos (passo a passo)? E para a lavagem e armazenamento dos equipamentos e utensílios usados (passo a passo)? 12. De onde vem a água utilizada para cozinhar, lavar equipamentos/utensílios, e beber? O que é feito na escola/cozinha para garantir a sua qualidade? 13. Quem é responsável por fornecer a lenha/combustível para a cozinha? Há problemas de suprimento (fornecimento suficiente) ? Se sim, o que poderia ser feito? 14. Vocês as cozinheiras desta escola passaram por algum tipo de avaliação (controlo) de saúde (antes de iniciar atividades e de tempos em tempos)? Qual? 15. Desde que você começou a trabalhar como cozinheira na escola tem havido falhas na distribuição das refeições (dias sem alimentação ou dias com uma quantidade de alimentos insufi-ciente)? Se sim, qual foram os motivos, as causas ? 16. Caso seja possível fazer ajustes nas rotinas de trabalho na cozinha, e outras atividades ou aspetos do programa onde as cozinheiras participam, qual seria sua recomendação? G. Alunos (meninos e meninas) - grupos mistos de 10 alunos (5 meninos e 5 meninas) Evaluation Report Version March 2021 111 | P a g e 1. Quando é distribuída a alimentação na vossa escola (horário, em relação ao início da aula)? Isto está bem assim? Poderia ser melhorado? 2. A alimentação escolar está a ser oferecida todos os dias, cada dia ? Se não, você saberia responder quantas vezes houve falta de alimentos (e por quanto tempo) ? Neste caso porque é que isto aconteceu? 3. Quem recebe primeiro a alimentação em sua escola? (meninos ou raparigas, alunos mais jovens ou mais velhos, ou outro critério de organização da distribuição de refeições) 4. Qual sua opinião sobre a alimentação distribuída nas cantinas escolares: a quantidade é suficiente? Ou insuficiente, ou excessiva? A comida é gostosa, boa ? 5. (Pergunta delicada) Você comeu hoje antes de vir para a escola? Se sim, o quê? se não, quando foi a última vez que comeu antes de vir para a escola? 6. Qual a sua contribuição para as cantinas escolares (ajuda no preparo da comida ou na lavagem de pratos e panelas, contribuição com alimentos ou dinheiro, traz lenha ou água, outras ?) (atenção, perguntar separadamente a rapazes e meninas)? 7. Nas escolas piloto com compras locais : o quê você pensa sobre a adição de produtos frescos (verduras, legumes, frutas, ovos/peixe/carne) a alimentação escolar? 8. Em sua opinião, quais os benefícios de ter as cantinas escolares nessa escola (distribuição de refeições escolares pelo PMA/MGD)? Você percebe algum resultado positivo na sua vida como aluno/a? qual? 9. O que você aprendeu sobre saúde, nutrição e higiene na escola? Você pratica em sua casa algo que aprendeu na escola sobre higiene, saúde e nutrição? Ou é difícil demais? Se sim, quais as principais barreiras? 10. Os alunos costumam comprar comida (rebuçados, doçaria, bolachas, etc) nas vendedoras à porta da escola? Se sim, o que costumam comprar? 11. Caso fosse possível modificar ou ajustar o programa de cantinas escolares, para torná￾lo melhor, o que você recomendaria? 11. Como está a situação da higiene na vossa escola? As latrinas/casas de banho estão bem feitas? Dá para as utilizar de maneira segura, sem ser incomodado/a? H. Meninas (da 4ª à 6ª classe) – grupos de só raparigas (de 6 a 10) 1.Como funciona o sistema das rações secas para levar à casa nesta escola: quem recebe as rações (somente meninas ou meninos e meninas, alunos de todas as classes ou de 4-6, etc)? 2.Há algum tipo de controle pela escola sobre a distribuição das rações secas (arroz ou outra)? 3.Você recebe a ração seca todos os meses ou houve meses que não recebeu os alimentos? Caso não tenha recebido em algum mês, qual foi o motivo? Explicaram-lhes? 4.Como você percebe essa atividade, isto é, você acha que de alguma forma afeta sua escolaridade/participação nas atividades da escola? 5.E sua família, como eles percebem essa atividade? Há algum tipo de incentivo por parte de sua família para que você atenda regularmente as classes para poder receber a ajuda mensal de alimentos? 6.Caso sua escola deixasse de distribuir os alimentos da ração seca, isso afetaria de alguma forma sua participação nas aulas/estudo? Se sim, qual seria o impacto para você? Evaluation Report Version March 2021 112 | P a g e 7. O que poderia ser feito para melhorar o sistema das rações secas ? 8. Outros elementos que podem influenciar a decisão de deixar de ir à escola: as casas de banho/latrinas que não permitem segurança, privacidade, situações intra-familiares (delicado a perguntar), ...? I. AGRICULTORES LOCAIS (Mulheres e Homens) 1.Você participa de alguma associação/cooperativa ou produz e vende seus alimentos de forma independente? 2.O que você pensa do programa de cantinas escolares? 3.Você participa de alguma forma a este programa, isto é, vendendo produtos alimentícios para o programa/escola? 4.Se sim, que produtos e quantidades vende para a cantina escolar (por semana)? 5.Se sim, o que você pensa do sistema de pagamento (vouchers escolares)? Funciona bem ou traz alguma dificuldade para os agricultores? 6.Se não, quais são suas expectativas, como produtor rural, em relação a sua possível oportunidade futura de participação do programa de cantinas escolares? 7.Você (ou sua associação de agricultores) recebe ou recebeu algum tipo de apoio (assistência técnica e capacitação/treinamento relacionado ao programa cantinas escolares/ PMA? (pode ter sido feita por parceiros do programa, como FAO/NGOs, em colaboração como o PMA/MGD)? Se sim qual/quais? 8.Caso você tenha recebido capacitação/treinamento de que forma sua prática como agricultor/produtor rural foi melhorada (ou não)? 9.Há algum aspeto relacionado com a sua atividade em que gostaria de ser treinado? 10.Você participou ou conhece alguém que participou de atividades como aulas de alfabetização para agricultores (financiadas pelo projeto PMA/MGD)? Se sim, qual é a sua opinião em relação às mesmas? (incluir uma análise de género: proporção e funções de homens e mulheres) 11.Como produtor rural/agricultor, qual seria sua recomendação para melhorar sua participação no programa de cantinas escolares? Evaluation Report Version March 2021 113 | P a g e Annex 17 - Bibliography Presented throughout the text as footnotes Guinea -Bissau Country Office https://www.wfp.org/countries/guinea -bissau Bogotá-Colombia, March 2021, Final evaluation report Version 8