1 | Page ESS Phase II - Final Evaluation [Type a quote from the document or the summary of an interesting point. You can position the text box anywhere in the document. Use the Drawing Tools tab to change the formatting of the pk Effective Seed Storage Program Phase II: February 2013 – February 2015 Final Evaluation February 2015 Johan van Duijn – Independent Consultant 2 | Page ESS Phase II - Final Evaluation Survey analysis and report prepared by: Johan van Duijn Independent Consultant Survey conducted by: Mercy Corps, Catholic Relief Services (CRS), Together Improving Development (TID) and Organisasaun Haburas Moris (OHM) Front page images by Mercy Corps Timor-Leste 3 | Page ESS Phase II - Final Evaluation List of Acronyms AEZ Agro-Ecological Zone Aldeia Hamlet/Sub-village with a leader, but no council BCC Behavior Change Communication CDB Caritas Diosezana Baucau CRS Catholic Relief Services DAC Development Assistance Committee ESS Effective Seed Storage (program) FA Field Agent FAO Food and Agriculture Organization (of the United Nations) FGD Focus Group Discussion GPB GrainPro Bag HH Household IFAD International Fund for Agricultural Development IMM Ida Mos Manufahi (local partner) INGO International Non-Governmental Organization ISP Informal Seed Production KdP Kolegas da Paz kg Kilogram KII Key Informant Interview M&E Monitoring and Evaluation MAF Ministry of Agriculture and Fisheries MC Mercy Corps MFI Micro Finance Institution NGO Non-Governmental Organization OFDA Office of U.S. Foreign Disaster Assistance OHM Organizasaun Haburas Moris SILC Savings and Internal Lending Community SOP Standard Operating Procedure Suco Village - Biggest political unit with a development council. Composed of a number of Aldeias TID Together Improving Development USAID United States Agency for International Development 4 | Page ESS Phase II - Final Evaluation List of Contents List of Acronyms ...................................................................................................................................................... 3 List of Contents ....................................................................................................................................................... 4 List of Tables............................................................................................................................................................ 5 List of Figures and Boxes......................................................................................................................................... 6 Acknowledgements................................................................................................................................................. 7 Executive Summary................................................................................................................................................. 8 1. Introduction................................................................................................................................................... 12 1.1 Effective Seed Storage Program .......................................................................................................... 12 1.2 Evaluation Objectives.......................................................................................................................... 13 2. Methodology and limitations ........................................................................................................................ 14 2.1 Methodology ....................................................................................................................................... 14 2.2 Limitations........................................................................................................................................... 16 3. Program Achievements Against Indicators.................................................................................................... 17 3.1 Increase in # of months of food self-sufficiency.................................................................................. 17 3.2 Number of people benefiting from seed systems/agricultural input activities, by sex ...................... 19 3.3 Decrease in post-harvest seed losses.................................................................................................. 20 3.4 Availability of quality seed during planting season ............................................................................. 20 3.5 Access to improved seed storage systems.......................................................................................... 25 3.6 Access to BCC materials/training......................................................................................................... 26 3.7 Adoption of improved techniques....................................................................................................... 28 3.8 SILC indicators...................................................................................................................................... 29 4. Assessment of DAC criteria’s......................................................................................................................... 31 4.1 Relevance............................................................................................................................................. 31 4.2 Effectiveness........................................................................................................................................ 34 4.3 Efficiency.............................................................................................................................................. 39 4.4 Impact.................................................................................................................................................. 43 4.5 Sustainability........................................................................................................................................ 46 5. Lessons Learned............................................................................................................................................. 47 5.1 ESS Program......................................................................................................................................... 47 5.2 SILC....................................................................................................................................................... 49 6. Conclusions and Recommendations.............................................................................................................. 49 5 | Page ESS Phase II - Final Evaluation Annex A: Household Survey Template.................................................................................................................. 56 Annex B: Change in Used Storage Methods from Baseline to Endline................................................................. 69 List of Tables Table 1.1 DAC criteria and assessment questions 13 Table 2.1 Distribution of respondents per district (men and women) 15 Table 3.1 Increase in food self-sufficiency months compared to 12 months before joining program 17 Table 3.2 Food insecurity figures at Baseline and Endline 18 Table 3.3 Program components, total farmers and beneficiaries 19 Table 3.4 Changes in seed losses (% of households) 20 Table 3.5 Seed storage lossess and reduction in losses (%) 21 Table 3.6 Utilization (% of respondents) of selected improved storage methods per district 23 Table 3.7 Sources of seeds (% of households) 23 Table 3.8 Average amount of seed stored after harvesting (Kg - Improved and local) 24 Table 3.9 Average cultivated land area with maize; baseline, endline and change (% of respondents) 25 Table 3.10 Type of Silo purchased with vouchers (% per district) 25 Table 3.11 Households receiving information at time of handover unit (% per 27 Table 3.12 Application of improved technology (% of responding HHs) 29 Table 3.13 SILC indicators, target and achievements 30 Table 4.1 Coverage of maize farmers per district (%) 33 Table 4.2 Program achievements against indicators 34 Table 4.3 Selection criteria voucher recipients 36 Table 4.4 Calculation expenditures ESS Phase II 40 Table 4.5 Cost of interventions per beneficiary and justifications 40 Table 4.6 Cost per voucher per organization ($) 41 Table 4.7 Expenditures per cost category (in $ and %) 42 Table 4.8 Responsible person(s) in seed production, Baseline and Endline (% of households) 44 Table 4.9 Decision making power within households, Baseline and Endline (% of households) 45 Table 4.10 Program benefits expected to continue and their contributing factors 46 Table 6.1 Conclusions and recommendations – Achievements towards indicators 50 Table 6.2 Conclusions and recommendations – DAC criteria 52 6 | Page ESS Phase II - Final Evaluation List of Figures and Boxes Figure 3.1 Changes in number of months of food self-sufficiency compared to 12 months before joining program (N=401) 17 Figure 3.2 Reasons given for increased food availability from own production (percentage of HHs reporting increased food availability) 18 Figure 3.3 Storage methods of seeds (% of households growing improved or traditional varieties) 22 Figure 3.4 Method of information sharing during hand over of silo 27 Figure 3.5 Sources of additional information after purchase of silo (% of households – N=368) 27 Figure 3.6 Source of post-harvest training by district (% of respondents) 28 Figure 4.1 Variation in silo production by blacksmiths 38 Figure 4.2 Storage methods used for rice and beans (% of HH using for rice and beans) 44 Box 1 Improved storage methods 21 Box 2 Farmer mobilization process 36 7 | Page ESS Phase II - Final Evaluation Acknowledgements Wahyu Nugroho Mercy Corps Director of Agriculture and Food Security Joanna Walshe Mercy Corps Country Director Timor-Leste Albino Amaral Mercy Corps M&E Officer Goretti Oliveira Mercy Corps ESS Program Manager Torrey Peace Ian de la Rosa Catholic Relief Services Catholic Relief Services Head of Programs Country Representatives Alberto Freitas dos Reis Catholic Relief Services ESS Program Manager Marcelino Pinto Joana da C. S. Ximenes Catholic Relief Services Together Improving Development M&E Officer Executive director Appreciation also goes to the community members, blacksmiths and partner staff that participated in evaluation activities. 8 | Page ESS Phase II - Final Evaluation Executive Summary Since August 2011, Mercy Corps has been implementing the USAID/OFDA1 funded Effective Seed Storage (ESS) in Timor-Leste program, with the overall goal to design and develop sustainable and scalable farmer seed storage models in Timor-Leste. Based on the success of the Pilot Phase in two districts, Mercy Corps initiated in March 2013 a partnership with Catholic Relief Services (CRS) for a nation-wide expansion of the ESS program to reach 240,000 beneficiaries. The total committed support by OFDA, from August 2011 to February 2015 is $2,522,499. The objective of the evaluation is: to collect data and information and to conduct an analysis to evaluate the ESS program achievements. The evaluation also documents early learnings from the SILC component of the program and provides recommendations for the anticipated Phase III of the ESS program. This evaluation was conducted during the period of January 17 - February 24, 2015. Field data collection methods included a household survey with 409 respondents, Focus Group Discussions (FGD) in communities and with seed silo manufacturers, and interviews with staff of the program partners. The tables below provide key conclusions from the evaluation and recommendations for an expansion or follow-up program. Conclusions Recommendations Achievements towards indicators Indicator: Projected increase in # of months of food self-sufficiency due to seed systems/agricultural input for beneficiary households Baseline: 2.5 months Target: 30% increase = 0.75 months Target achieved: Average 2.1 month increase of food self-sufficiency = 84% • Include in future surveys productivity of maize and use of the production to analyze whether increase of maize production is actually used for food • Design more comprehensive food security interventions including nutrition for future programs Increasing the food availability by the program was not sufficient to address food insecurity. Indicator: # of people benefiting from seed systems/agricultural input activities, by sex Target: 240,000 The program reached 263.971 beneficiaries, surpassing the target by 10.0%. 47% of the beneficiaries of Phase II were women Indicator: Percentage of beneficiaries reporting decreased post-harvest losses for seeds Baseline: HHs reporting losses: 93% for improved variety; 95% for other varieties Target: 80% reporting decreased losses Decrease in post-harvest losses reported by households: 87.7% for maize, 80.3% for rice, and 73.7% for beans. Indicator: Increase in availability of quality seed during planting season Baseline: 39% of improved variety seeds are lost during storage and 45% of other varieties Target: 80% increase in availability: reduction of losses to 7.8% and 9.0% respectively Endline survey shows a reduction in seed storage losses, but the target is not achieved: • 23.2% of improved variety seeds are lost  • Incorporate follow-up strategies in succeeding programs to reinforce application of technologies (E.g. visits or SMS messages) 1 United States Agency for International Development/Office of U.S. Foreign Disaster Assistance 9 | Page ESS Phase II - Final Evaluation Reduction of 40.5% • 23.9% of local variety seeds are lost  Reduction of 46.9% FGD participants reported that if using improved storage 100% of the seeds are good for planting after storage. • Assess in future surveys seed survival for each of the different storage methods households are using instead of one combined survival rate to allow measurements for traditional and improved methods using silos. The evaluation survey assumed that beneficiaries would shift completely to the use of silos, but people continued to use also traditional storage methods. Indicator: Number of farmers with access to improved seed storage system, by sex Target: 40,000 Target achieved: 45,649, surpassing the target by 11.4%. Indicator: Number of farmers with access to BCC materials/training Target: 40,000 Target achieved: Total of 42,518, 6.3% above target The NGO worker was the most important source of follow￾up information (69.3%). The local leader also is important in providing information to 37.8% of the households. • Local leader standard as a participant in trainings and consider provision of a silo to motivate leaders as an incentive. Percentage of farmers adopting improved technique(s) Target: 80% Target was not achieved for all techniques: • 56.8% for better seed selection • 59.6% for better drying of seeds • 88.5% for storing local varieties separate from improved varieties • Storing food separately from seeds: 88.8% for local varieties and 96.9% for improved varieties. • Incorporate follow-up strategies in succeeding programs to reinforce application of technologies (e.g. visits or SMS messages). • Conduct demonstrations following the cropping calendar. • Conduct barrier analysis. DAC Criteria Relevance Relevance increased through inclusion of SILC. Savings groups seen as very important by community members to have a safe place to put money and reduce waste on unnecessary purchases. • Completion of group formation and training during ESS Phase III. • Incorporation of SILC indicators in Phase III evaluation survey. In several districts, vouchers already covered a significant portion of the maize-producing farmers. The use of the voucher coverage is likely to go beyond resource-poor households. • Reduce number of vouchers in Phase II. • Prioritize in Phase III districts and communities that have a current coverage by vouchers of less than 40%. • Ensure during selection processes that priority goes to resource-poor households. Effectiveness The following factors contributed to achievements: • Expansion to eastern district through CRS • Increased competition  cheaper silo materials and good profit margins of $5-7 per unit • Improved beneficiary selection process with feedback mechanisms for community members. • Consider the development of feedback mechanisms on the quality of products straight to the blacksmiths. 10 | Page ESS Phase II - Final Evaluation The following factors hampered achieving results: • Capacity and dedication of local partners to do sufficient field work in communities • Variation in capacity of blacksmiths to produce and deliver silos • Limited thinking of stakeholders towards market-led development • The program’s high number of beneficiaries has to some extent hampered the achievement of full adoption of the promoted techniques as program paid more attention in achieving number of beneficiaries to own improved seed storage systems but less in supporting adoption of promoted good practices. • Capacity assessments with blacksmiths prior to signing agreements with them. • Reduction of number of voucher beneficiaries to remove expectation to get subsidized silos and freeing time for follow-up activities and monitoring of silo and technique adoption. • More market support activities Efficiency Quantitative targets for silo distribution are met with only 76.7% budget expenditure. The cost per household for the seed storage intervention is $39.89. This amount could easily be justified by the prevention of loss of seeds, which can be equivalent to $20 per planting season. The target number of beneficiaries for seed systems were reached, but needed the deployment of additional staff. • Ensure allocation of sufficient field staff for beneficiary mobilization and training. A number of inefficiencies were observed in M&E. There were changes in formats, unclear use of information provided, and unused funds for a central database that could have facilitated some M&E processes. • Design a standard monitoring system at the beginning of the program, including database • Consider the use of e-vouchers and linked monitoring system/based • Identify focal persons for communication between organizations • Design feedback mechanisms/procedures on reports Impact Blacksmiths turned in to businessmen. • Prioritize business management training with new blacksmiths at start of Phase III. Reduction in seed losses and increased food self-sufficiency through high adoption of quality seed storage in the form of metal silos. • Final evaluation of ESS Phase II to sample respondents from beneficiaries since 2011 to assess continuity of improvements and durability of silos. Farmers with the capacity to pay to purchase silos at the full price, demonstrating the trust of farmers in the product. Sustainability Continuing benefits after the program: • Silo production by blacksmiths. • Increased food self-sufficiency. • Increased seed availability. Factors threatening sustainability: • Conduct barrier analysis and redesign BCC around 3 11 | Page ESS Phase II - Final Evaluation • Unavailability of silo materials at competitive prices. • Continuing wait and see attitude – non-beneficiaries waiting for next round of vouchers instead of purchasing with own resources. • Full price is still considered too high by vulnerable farmers, justifying the distribution of vouchers for this group. key messages:  Proper seed selection  Proper drying  Proper storage • Work towards free market principles allowing price competition between blacksmiths. The following lessons learned are identified for consideration during Phase III or future programs: • Overly high target number of beneficiaries has to some extent discouraged behavior change interventions, as program paid more attention in achieving number of beneficiaries to own improved seed storage systems, but less in supporting adoption of promoted good practices. • Production targets of blacksmiths need to be balanced with their capacity. • Number of field staff needs to be balanced with the working environment. • Feedback mechanisms during community mobilization and customer feedback survey to increase accountability towards beneficiaries. • Weak market systems challenge implementation of market driven development programs. • Inclusion of finance and operations departments in designing Standard Operating Procedures (SOP) to ensure accountability of activities. • Branding and marketing by professionals could be more effective and efficient than done by NGOs. • The travel needs for SILC activities limit women from becoming SILC Field Agents. 12 | Page ESS Phase II - Final Evaluation 1. Introduction Chapter 1 provides a summary of the ESS program and the objectives of the evaluation. The succeeding chapter provides details on the applied methodology and the experienced limitations. Chapter 3 presents the progress made towards the program’s indicators, followed by Chapter 4 which answers research questions to assess the Development Assistance Committee (DAC) criteria. Chapter 5 provides lessons learned and the report ends with conclusions and recommendations in Chapter 6. 1.1 Effective Seed Storage Program Since August 2011, Mercy Corps has been implementing the USAID/OFDA2 funded Effective Seed Storage (ESS) in Timor-Leste program, with the overall goal to design and develop sustainable and scalable farmer seed storage models in Timor-Leste. The program developed a market system for a metal-based seed storage system that is customized, locally manufactured and available in different sizes. Based on the success of the Pilot Phase that targeted over 27,000 beneficiaries across four sub-districts, Mercy Corps (MC) initiated in March 2013 a partnership with Catholic Relief Services (CRS) for a nation-wide expansion of the ESS program to reach 240,000 beneficiaries. The total committed support by OFDA, from August 2011 to February 2015 is $2,522,499. The expansion targets nation-wide, with priorities given to the neighboring districts/sub-districts of the pilot districts Ainaro and Manufahi as well as other maize producing districts – in particular Baucau, Viqueque and Lautem in the eastern part of the country. The program builds on the learnings from the Pilot Phase and the evaluation conducted in February 2013. The partnership with CRS allowed Mercy Corps to easily expand the program to the eastern districts of Baucau, Lautem and Viqueque. CRS worked with one local partner in each of the districts while Mercy Corps worked with two local partners in a total of eight districts. Only Dili and Oecusse districts are not part of the expanded program. The evaluation of the Pilot Phase concluded that the voucher system is an effective way to make quality storage units available to maize producers, and the expansion continued to use this strategy. To be able to reach a higher number of targets in Phase II, the program increased the number of blacksmiths involved in the production, and building their capacity to make the silos, and also encouraging diversification in creating other products. The promoted units are still the same size as in the Pilot Phase, 35kg with a price of $23, and one that is 75kg for $26. The design of the 75kg was modified to one with a big open lid to allow the use of GrainPro Bags (GPB) or plastic bottles to jointly store seeds and foods, or different kind of seeds within one storage unit. The subsidy of the units is still through a $20 voucher and the program distributed 25,000 vouchers, prioritizing resource poor farmers. 2 United States Agency for International Development/Office of U.S. Foreign Disaster Assistance 13 | Page ESS Phase II - Final Evaluation Starting from May 2014, the program incorporates Savings and Internal Lending Communities (SILC) in Ainaro, Manufahi and Baucau with existing ESS beneficiaries to expand access to credit and promote a culture of savings among poor farming households. The SILC provides training to group members in new skills such as bookkeeping, savings and lending policies, accountability, and good governance. 1.2 Evaluation Objectives The objective of the consultancy is: to collect data and information and to conduct analysis to evaluate the ESS program achievements. The evaluation is also expected to document early learnings from the SILC component of the program, and provide recommendations for the anticipated Phase III of the ESS program. This evaluation has two parts: 1) A household survey to assess program performance against indicators and 2) Key informant interviews (KII) and focus group discussions (FGDs) to assess the program against the Development Assistance Committee (DAC) evaluation criteria: • Relevance • Effectiveness • Efficiency • Impact, and • Sustainability. The Terms of Reference (TOR) of the consultancy provides guidance questions for the DAC criteria in Table 1.1 below. Table 1.1: DAC criteria and assessment questions DAC Criteria Assessment questions Relevance • To what extent are the objectives of the program still valid? • Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? • Are the activities and outputs of the program consistent with the intended impacts and effects? Effectiveness • To what extent were the objectives achieved/are likely to be achieved? • What were the major factors influencing the achievement or non-achievement of the objectives? Efficiency • Were activities cost-efficient? • Were objectives achieved on time? • Was the program or project implemented in the most efficient way compared to alternatives? Impact • What has happened as a result of the program or project? • What real difference has the activity made to the beneficiaries? • How many people have been affected? Sustainability • To what extent did the benefits of a program continue after donor funding ceased? • What were the major factors that influenced the achievement or non-achievement of sustainability of the program or project? 14 | Page ESS Phase II - Final Evaluation 2. Methodology and Limitations 2.1 Methodology This evaluation used a combination of tools to assess the program’s achievements and approaches: • Questionnaires at household level • Focus group discussions (FGD) with beneficiaries • FGD with SILC Field Agents • FGD with silo manufacturers • Interviews with staff of Mercy Corps and Catholic Relief Services • FGD with local partner representatives Below is a brief description of each of the activities: Household Questionnaires The evaluation intended to do a longitudinal survey by returning to respondents of the baseline survey in order to identify changes in the lives of beneficiary households. The consultant, however, noticed during the field visits that communities that were part of the baseline only recently purchased their storage units. After deliberation with program management and Monitoring and Evaluation (M&E) teams, it was decided to do a more purposeful sampling among the program beneficiaries, but still within the four districts of the baseline survey. These four were chosen due to the Agro-Ecological Zone (AEZ) they are part of. The required number of interviews for the evaluation is the same as the baseline and calculated by applying a 5% margin of error and a confidence level of 95% for a beneficiary number of 40,000. The minimum required number of interviews is 381. 3 The following steps were applied to select the respondents: • Generate list of beneficiaries that purchased a storage unit before April 2014. This would allow recipients to have used the unit for storage after harvesting maize. • Random numbering of the beneficiaries. • Interview the first 100 households. If a household was not available for interview, proceed to number 101 and onwards. Table 2.1 provides a summary of the respondents to the survey. 3 Calculated via the online sample size calculator at http://www.raosoft.com/samplesize.html 15 | Page ESS Phase II - Final Evaluation Table 2.1: Distribution of respondents per district (men and women) District AEZ Male respondents Female respondents Total respondents Percentage female Bobonaro Northern Slopes 51 51 102 50.0 Covalima Northern Highlands 52 48 100 52.0 Lautem Southern Slopes 51 55 106 48.1 Liquica Southern Highlands 66 35 101 65.3 Total 220 189 409 46.2 The endline questionnaire was based on the baseline template; however, the following modifications were made: • Additional questions on the food self-sufficiency of the household. • Questions helping attribution of improvements in food security to the program. • Adding questions on the use of the silos on other crops like rice and beans, and storage losses accordingly. • Questions on what training and Behavior Change Communication (BCC) respondents received. The complete survey form is presented in Annex A of this report. The consultant designed the survey form and database, and finalized in collaboration with Mercy Corps and CRS staff. The orientation and supervision of the survey was conducted by the international NGOs (INGO). After encoding the data, the database was sent to the consultant for analysis. Focus Group Discussions with Beneficiaries The consultant intended to visit communities in Liquica and Baucau districts. The latter had to be cancelled due to the security situation in the district. In Liquica, a FGD was conducted with a mixed group of men and women. Two FGDs were conducted in a second hamlet, one with male and one with female beneficiaries. FGD with Silo Manufacturers Mercy Corps invited all of the silo manufacturers to come to Dili for a FGD and 16 out of the 17 blacksmiths attended. The discussion started with a short exercise in one group on successes and challenges in the production, and a second group on the successes and challenges in the marketing of the silos. The output of the groups was used to discuss the DAC criteria, in particular Impact and Sustainability. FGD with SILC Field Agents 16 | Page ESS Phase II - Final Evaluation The FGD with SILC field agents was conducted in Maubesi town and brought together three of the five Field Agents (FA) active in Manufahi and Ainaro district. Among them they were supporting 20 groups at time of the meeting. All three field agents were men. FGD with Local Partner Staff The local partner meeting was held at the Mercy Corps office. Organizations present during the FGD were Together Improving Development (TID), Caritas Diosezana Baucau (CDB), and Kolegas da Paz (KdP), CRS’ three local partners, and Organisasaun Haburas Moris (OHM), partner of Mercy Corps. The second partner of Mercy Corps, Ida Moris Manufahi (IMM), did not attend, but the director was interviewed in Maubesi. To start the FGD, participants were divided into two groups to answer the following questions: • Group 1: ‘We think ESS is a good program because ......’ • Group 2: ‘If there would be another storage program I would do/include ........, because ......’ The output of the group work was presented and used as a starting point to discuss DAC criteria evaluation questions. Interviews with Mercy Corps and CRS staff Interviews were conducted with the Country Director of Mercy Corps and CRS staff involved in the ESS program. These staff included the head of office of Baucau, the project manager and M&E officer. Mercy Corps’ Director of Agriculture and Food Security Programs was consulted throughout the whole evaluation process. 2.2 Limitations The evaluation faced the following limitations: • Social and cultural events - The scheduled activities coincided with other activities. For example, the FGD with SILC group members fell at the time that an important religious event, ‘Cruz Joven,’ arrived in Maubesi obliging community members to attend related activities. As a result, about 40% of members were not able to attend the discussion. In Liquica, a scheduled FGD could not take place due to attendance of the community members a Parents Teachers Association meeting. As an alternative, a FGD was conducted with women of the other community visited. • Baseline respondents - The evaluation could not be longitudinal as initially planned and interview the respondents from the baseline survey. A part of the Baseline respondents were included too late as program beneficiaries. • Baseline question on food self-sufficiency – The baseline asked the respondents about food sufficiency instead of food self-sufficiency. As a result, the endline had to use questions relying on recall to assess progress towards the indicator, which can be less reliable. • Encoding inaccuracies - The consultant found a significant number of encoding errors that needed to be corrected. The corrections were both time consuming for the consultant and Mercy Corps’ M&E coordinator. 17 | Page ESS Phase II - Final Evaluation • Sequence of evaluation activities - The consultant was only available for a certain time period to conduct the fieldwork for the evaluation. This meant that the qualitative data gathering was done prior to the completion of the survey, leaving no opportunities to verify information coming from the data analysis. Recommendations are made in this report where verification could still be useful. • Security situation – The district of Baucau experienced some security incidents just before the consultant’s planned visit. As a result, the visit to Baucau was cancelled and interviews conducted by phone and Skype as much as possible. In person discussions with community members had to be cancelled altogether. 3. Program Achievements against Indicators 3.1 Increase in # of months of food self-sufficiency The complete indicator assessed is: Projected increase in number of months of food self-sufficiency due to seed systems activities/agricultural input for beneficiary households The target set for this indicator is an increase of 30% in number of months. Unfortunately, the baseline survey measured number of food security months, while the indicator actually measures the months of food self￾sufficiency. 4 For this reason, the endline survey questioned the respondents on ‘number of additional months of being food self-sufficient.’ In addition, they were asked about the ‘number of months of food self-sufficiency in previous 12 months,’ so that the project indicator (Projected increase in number of months of food self￾sufficiency due to seed systems activities/agricultural input for beneficiary households) could be measured. Table 3.1 provides analysis of the information coming from the interviews. The data in the first two columns comes directly from the survey. The third column is the difference between the two and represents the total number of food self￾sufficiency months households experienced compared to before joining the program. 4 ‘Food self-sufficiency’ indicates ability of a household to have food from own production, while in ‘food security’, food doesn’t necessary come from own production (i.e. buying from market). 5 Example for Bobonaro: 5.0 – 3.1 = 1.9 Table 3.1: Increase in food self-sufficiency months compared to 12 months before joining program District Reported increase food self-sufficiency months Reported food self-sufficiency months in previous 12 months Calculated months of food self-sufficiency months prior to joining program5 % Increase in food self￾sufficiency months Bobonaro 3.1 5.0 1.9 163.2% Covalima 2.2 6.1 3.9 56.4% Lautem 2.1 3.9 1.8 117.7% Liquica 1.1 3.5 2.4 45.8% All households 2.1 4.6 2.5 84.0% 18 | Page ESS Phase II - Final Evaluation The table demonstrates that the program achieved a 2.1 increase in months of food self-sufficiency. Compared to an average of 2.5 months before the project, this represents an increase of 84.0%, more than double the 30% target for the indicator. The data presented in Table 3.1 are averages and the reader needs to understand that not all households reported improvements in the availability of food from their own production. Figure 3.1 illustrates a breakdown of the responses. From the total of 401 households responding to this question, 14% reported that they experienced a decrease in number of months with food available, 48% an increase and 38% the same number of months. The net number of households with increased number of months of food self-sufficiency is 34.7%. The reasons given for an increased number of months with food availability from own production are presented in Figure 3.2. The respondents were allowed to provide more than one answer, and the most common was a higher production due to better seed selection (47.2%). Better production processes was reported by 42.5% of the households, making it the second most popular reason. The availability of better seeds also scores relatively high with 31.3%. Better storage is acknowledged as important (25.5%), but this does not lead necessary to a higher area planted for maize production. That answer scored lowest with only 17.7%. Overall attribution is much lower than during the evaluation of the Pilot Phase (in Ainaro and Manufahi Districts) during which each of the options scored at least 70%. Table 3.2: Food in security figures at Baseline and Endline Characteristic Baseline Endline % Food insecure households 46.6 54.4 # Food insecure months; food insecure HHs 0.9 1.8 # Food insecure months; whole sample 2.0 3.5 Figure 3.1: Changes in number of months of food self￾sufficiency compared to 12 months before joining program (N=401) Figure 3.2: Reasons given for increased food availability from own production (Percentage of HHs reporting increased food availability) 19 | Page ESS Phase II - Final Evaluation The baseline and the endline also assessed the overall situation within households, expressed in number of months that households were not able to satisfy the family needs. This includes from their own production as well as through purchase or other means. The overall characteristics are presented in Table 3.2 showing that 54.4% of the households experienced food shortage during at least one month out of the twelve prior to the endline. This is higher than the 46.6% reported during the baseline. The average months also increased in between the baseline and endline. The reasons behind a reduction in food security could not be explored during this evaluation, but reasons could include: • The households interviewed were different during the two surveys. • A difference in timing of the survey combined with recall period for respondents • The households experienced a change in access opportunities, for example a reduction in income to purchase food. The figures presented in this section demonstrate that increasing food availability alone is not sufficient to address food insecurity. To address food insecurity, and prevailing challenges in Timor-Leste around nutrition as a whole, will require more comprehensive programming. 3.2 Number of people benefiting from seed systems/agricultural input activities, by sex The complete indicator assessed is: # of people benefiting from seed systems/agricultural input activities, by sex. This is an OFDA indicator. The program has set a target of 240,000 beneficiaries to be reached by the program, and this includes beneficiaries reached by the Pilot Phase. Table 3.3 presents the beneficiaries for different program components. Table 3.3: Program components, total farmers and beneficiaries Component # of farmers % Women beneficiaries Total beneficiaries6 Pilot Phase Silos with voucher 2,337 34% 14,427 Silos without voucher 1,052 Not recorded 6,522 BCC – Post-harvest training 2,266 31% 14,049 Phase II Silos with voucher 24,718 26% 153,252 GrainPro Bag 9,936 61,604 Silos without a voucher 7,607 Not recorded 47,163 BCC – Post-harvest training 31,252 34% 193,762 BCC – Radio 3,000 18,600 BBC - Television 6,000 37,200 6 The average household size for rural areas is 6.2 (Census 2010) 20 | Page ESS Phase II - Final Evaluation Total 88,168 546,642 The components will have a significant overlap in terms of beneficiaries, in particular through the training to voucher recipients and the inclusion of GPBs in the silos. Given these, the total number of recipients for BCC activities is a reasonable total number of beneficiaries. This will bring the total of beneficiaries at 249,562 for Phase II, plus 14,409 during the Pilot Phase, making a total of 263,971. This is equivalent to surpassing the target by 10.0%. Since there will not be a complete overlap, the actual number of beneficiaries reached will be higher but impossible to assess. 3.3 Decrease in post-harvest seed losses The complete indicator assessed is: Percentage of beneficiaries reporting decreased post-harvest losses for seeds. The baseline reported that 93% of the growers of improved varieties reported losses during the storage of seeds. The percentage for traditional varieties is higher at 95%. Following the baseline, the program set a target for an 80% reduction. The program is mainly promoting improved seed selection, production and storage of maize seeds, but the Customer Feedback Survey indicates that many households are also using the storage for other crops. Table 3.4 also provides information on the other main crops: rice and beans. The table shows that farmers experienced a decrease in seed losses due to the use of improved storage facilities. The decrease in losses is highest for maize (87.7%), followed by rice (80.3%) and beans (73.7%). Table 3.4 shows the positive development towards producing more seeds with 20.8% of the households that store ‘improved’ seeds (and 11.7% for ‘traditional’ seeds) report a 100% survival of seeds. Moreover, during the FGDs – as part of this evaluation, majority of participants felt that there was a 100% survival rate when using improved methods. Section 3.4 discusses more on the availability of quality seed. 3.4 Availability of quality seed during planting season The complete indicator is: Percentage of beneficiaries reporting an increase in availability of quality seed during planting season Table 3.4: Changes in seed losses (% of households) Seed situation Maize Rice (in Husk) Beans N= 374 61 95 Decrease 87.7% 80.3% 73.7% The same 11.5% 18.0% 23.2% Increase 0.8% 1.6% 3.2% 21 | Page ESS Phase II - Final Evaluation This indicator is somewhat similar to that discussed in section 3.3. Table 3.4 in the previous section already presented that 87.7% of maize producers reported they have more seed available after the storage period. This section looks at seed availability by understanding the following: • Survival rate of seeds after storage • Percentage of households applying improved storage methods • Changes in sources of seeds • Changes in amount of seeds stored Table 3.5 presents the findings of seed losses expressed in percentages during the baseline survey, the target value and the information gathered during the endline survey. The table shows that the program was not able to meet the 80% reduction in seed losses as targeted. For improved seeds, a reduction in 40.5% was reached and local seeds reached 46.9%. The target was likely set too high. The evaluation assumed that the introduction of metal silos would automatically make people use only these for all saved seeds, and a lower percentage of losses were expected. However, the survey looked at all the storage methods that people are using and the data shows that many households are using more than one storage method and not just using improved storage methods – see Box 1 for the list of improved storage methods. For example, a household could use a silo, but still also hang seeds in the kitchen. - The average number of storage methods used for improved variety of seeds is 1.4. - The average number of improved storage methods for storing improved varieties is 1.2. - There are still around 21.8% of the 133 respondents growing improved varieties that do not use improved storage. - The average number of storage methods for traditional varieties is higher at 1.5. - The use of improved storage methods for storing traditional varieties are also higher, with 88.3% of households using improved storage. Overall, households still show reservations on the effectiveness of the improved methods and prefer to spread the risk. It will be interesting during the next evaluation to continue to see if there is a shift in terms of this reservation. Table 3.5: Reduction in seed losses (%) Variety Baseline losses (%) Target losses (%) (80% reduction) Endline losses (%) Actual reduction (%) Improved 39 7.8 23.2 40.5 Local 45 9.0 23.9 46.9 Box 1: Improved storage methods: • Storage in metal drum (at HH) • Storage in metal drum (shared) • Storage in metal silo (at HH) • Storage in metal silo (shared) • Airtight container just for seeds • Storage in GPB in metal container • Storage in airtight plastic container 22 | Page ESS Phase II - Final Evaluation The data further shows that about 10% of the households have not used the unit they purchased. This is likely due to the fact that there are no seeds to store during the period between buying the unit and the survey, but this possibly suggests that beneficiary selection and the BCC activities should be improved. Figure 3.3 presents the storage methods used as reported during the endline survey and while there are shortcomings, major improvements can be reported. The upper three methods in the figure are traditional methods, and usually having the highest losses during storage. During the baseline, 52% for improved varieties and 61% for local varieties used these methods, but at time of the endline these percentages were halved. The use of metal containers increased dramatically since the baseline. At time of the baseline, less than 1% of households used them, but at time of the endline 74.5% of the respondents used silos for local varieties and 58.6% use them for improved varieties. 7 The use of a GrainPro Bag alone as a storage method is relatively low since they were bundled in the purchase of a 75 kg storage unit. Annex B presents the data on storage methods for both the baseline and the endline. 7 Note that the total is more than 100%. This is because the number of households growing improved varieties (133) is lower than households planting local varieties (377). Applying the percentages it will not go beyond the number of vouchers distributed. Figure 3.3: Storage methods of seeds (% of households growing improved or traditional varieties) 23 | Page ESS Phase II - Final Evaluation Table 3.6 presents data for selected improved storage methods for each of the districts. The table shows figures that vary greatly. Variations are not just across the districts, but also within a district over the different kind of seeds. The cells are highlighted across district on a scale from white to red. White represents a higher value, while the darker the red, the lower the value; or in other words, the lowest application. Table 3.6: Utilization (% of respondents) of selected improved storage methods per district Location Metal Silo GPB in metal container Airtight plastic in metal container Improved Local Improved Local Improved Local Bobonaro 56.8 85.3 0.0 0.0 2.7 11.8 Covalima 81.8 82.5 9.1 11.3 0.0 9.3 Lautem 61.0 74.5 12.2 9.2 12.2 8.2 Liquica 52.3 55.3 13.6 4.3 11.4 4.3 All respondents 58.6 74.5 9.0 6.1 8.3 8.4 The table demonstrates the following: • Liquica district is pulling down the average for the use of the metal silo, but is the highest in using GPB or airtight plastic containers, like plastic bottles. The use of the GPB could be a result of availability of the GPB, but the use of plastic containers shows a diversification in the use of the silos. • Covalima shows high use of the metal silos for both improved and local seeds • Bobonaro uses the metal silos a lot more (85.3%) for local varieties, than for improved varieties (56.8%). The use of plastic inside of metal containers shows a same trend, 2.7% for improved versus 11.8% for local) • Although a lower difference, the use of metal silos in Lautem is also lower for improved seeds (74.5% versus 61.0%). Overall, the endline survey found that 78.2% of the respondents stated that they are using one or more improved storage method as given in Box 1. A slightly higher percentage of 88.3% of the households is using improved methods for local varieties. Only 33.7% of the households are planting the improved varieties Sele and/or Noi Mutin, but increased from the 23.9% during the baseline survey. It needs to be noted, however, that Noi Mutin was only released recently and some of the increase could be attributed to this. The sources of seeds are presented in Table 3.7, together with the figures found during the baseline. 24 | Page ESS Phase II - Final Evaluation The Table shows some interesting movements in the sources of seed that could be a result of the introduction of secure seed silos. The cells highlighted in green show: • An increase in seeds available within the household especially for improved seeds. From 21% during the baseline, 44% of the households said that they sourced the seeds from their own harvest. • Friends and/or relatives is noted as an important source, indicating that households could have spare seeds. This is especially true for improved sees, which are often shared. While only a source for 1% of households during the baseline, this increased to 11% at the end of the program. The latter could be related to a more general perception of people that everybody should benefit from interventions. This was given as a reason during FGDs for households wanting bigger silos to be able to store seed and give away to neighbors. The cells highlighted in red indicate: • A possible reduced need of households to buy seeds on the market. • A reduction in need to buy seeds from relatives or friends • A reduction in need of seeds from government. The percentage is still high though, and could partially be explained by the inclusion of a plastic bottle of seeds when one purchases a drum from International Fund for Agricultural Development (IFAD). The cell highlighted in blue shows an increase in sourcing seeds for free from NGOs, which increased since the baseline. The amounts of seeds saved are presented in Table 3.8 below. The table also gives the number of stored kilograms that was reported during the baseline. Both groups of varieties have a slight decrease in the volume stored. This could be due to a tendency to store less when the quality of seed is better. The recommended volume of seeds for a hectare of Sele is around 30 kg, making the Table 3.7: Sources of seeds (% of households) Seed source Improved varieties Local varieties Baseline Endline Baseline Endline Own harvest 21 44 75 82 Market 11 11 27 21 Bought seed bank/group 1 2 3 2 Bought relative/friend 4 2 10 5 Free relative/friend 1 11 6 12 Free government 69 62 8 7 Free NGO 14 23 3 3 Table 3.8: Average amount of seed stored after harvesting (Kg - Improved and local) Location Improved varieties Local varieties Baseline Endline Baseline Endline N Average Min Max N Average Min Max Bobonaro 8.0 35 18.6 1 50 24.6 102 19.4 2 150 Covalima 29.0 10 17.2 2 35 27.2 99 17.5 2 35 Lautem 39.0 41 16.0 3 75 14.3 97 25.2 5 180 Liquica 17.0 43 19.4 2 180 19.3 94 14.0 1 180 All respondents 23.3 129 17.9 1 180 21.3 392 19.1 1 180 25 | Page ESS Phase II - Final Evaluation volume of saved seeds enough for around 0.6 hectare. There is however a big variation in the volume of seeds stored, and the minimum values given in Table 3.8 suggest that a significant number might be saving less seed than the capacity of the unit. The data from the survey shows that, for example, out of the households that purchased a 35kg unit, 83.3% are storing less than 35 kg of seeds. The area cultivated by farmers is presented in Table 3.9. The majority of households are planting 0.5 - 1ha (42%) with maize, followed by 0.25 – 0.5 ha (34%). The numbers indicated that households are storing sufficient seeds for a planting season, but may not have enough when there is a crop failure. The endline shows a shift in area planted as compared to the baseline. The households are moving towards smaller areas planted, which could indicated an increase in productivity, as also shown in Table 3.1. This section started with stating that there is a percentage reduction in losses, but in fact it appears that the average quantity of seeds stored is going down. 3.5 Access to improved seed storage systems The complete indicator is: Number of farmers with access to improved seed storage systems The target for this indicator was 40,000 including farmers that received GrainPro Bags. Table 3.3 in section 3.2 already demonstrated the number of farmers that had access: • A total of 3,378 farmers had access to silos during the Pilot Phase, including 1,052 by paying the full amount. • A total of 32,325 farmers had access to silos during Phase II, including 7,607 without vouchers. • 9,936 farmers purchased a GPB separately or as part of the purchase of a 75 kg unit. The above numbers bring the total number of farmers to 45,649, 11.4% higher than the target. Of the farmers availing the storage units and GPBs, 26% were women. The blacksmiths did not record whether units sold without vouchers were purchased by men or by women. The distribution of the vouchers over the 35kg and 75kg units shifted between the Pilot Phase and the Phase II expansion. During the Pilot Phase, 56% of the units sold with vouchers were of the 35kg kind, while this was only 42% in Phase II. While there is a difference in design, the higher demand for bigger units was explained during the FGDs saying that they gave opportunities to store different kinds of seeds, and to use them to store food. The women were quite happy with the size of the 35kg, but only to store seeds. Table 3.9: Average cultivated land area with maize; Baseline, Endline and change (% of respondents) Land Size Baseline (N = 389) Endline (N = 400) Change < 0.25 ha 13 16 +3 0.25 – 0.5 ha 25 34 +9 0.5 – 1 ha 45 42 -3 > 1 ha 17 8 -9 26 | Page ESS Phase II - Final Evaluation Table 3.10 provides the distribution of the use of the vouchers for the different silos. The color scale is applied within the district to highlight the most popular model in the district in white, followed by different shades of red. The darkest red is the least popular. The table shows that in the districts of Bobonaro and Covalima, farmers preferred the bigger units (53.9% and 55.0%), while in Lautem and Liquica the smaller 35 kg unit (58.5% and 62.4%). The figures, especially for Liquica could be influenced by a lack of choice offered to some of the beneficiaries. The participants in the FGDs revealed that they only had the option of 35 kg units. With the exception of Lautem, a number of respondents in the districts also purchased 75 kg units with a small opening, which was a design introduced in the Pilot Phase. Around 20% of the respondents in the survey stated that they would like to have a bigger unit. Most other respondents did not comment on the design, with the exception of a few individuals that stated that: • The 35 kg unit should have a lid • The 35 kg is too big for seeds. (This is correct if we take in to account the seeds stored per household. See table 3.6) • Would like to have a 75 kg with a small opening. • Silos with a handles to make it easier to carry around The variety in responses indicates that blacksmiths could make customized versions for clients. It needs to be noted that there are few households that received more than one voucher. 3.6 Access to BCC materials/training The complete indicator is: Number of farmers with access to BCC materials/training The target for this indicator is the same as that of the previous indicator: 40,000 farmers. This section presents data on: • The distribution of farmers over different BCC materials. • The percentage of respondents receiving information on silo use at time of handing over of silo and the methods used. • Sources of information on silo use after the silo distribution. • Participation in post-harvest training. The numbers for this indicator can be found in Table 3.3: • 33,518 farmers received BCC training including 2,266 during the Pilot Phase Table 3.10: Type of Silo purchased with vouchers (% per district) Location 35 kg with small opening 75 kg with small opening 75 kg with big lid 75 kg with GPB Bobonaro 43.1% 2.9% 53.9% 1.0% Covalima 35.0% 2.0% 55.0% 12.0% Lautem 58.5% 0.0% 27.4% 24.5% Liquica 62.4% 6.9% 36.6% 3.0% All respondents 49.9% 2.9% 43.0% 10.3% 27 | Page ESS Phase II - Final Evaluation • The radio broadcaster gave a low estimate of 3,000 farmers as the audience of radio programs on improved seed storage • An estimated 6,000 farmers watched television programs on seed storage related programs sponsored by the ESS program. This makes a total of 42,518 farmers that had access to training and materials, 6.3% higher than the target. The number of beneficiaries is based on the number of registered participants during post-harvest training. Beneficiaries also had the opportunity to gain more information during the handing over of the storage unit, and information provided by program stakeholders. Table 3.11 presents the percentage of respondents that received information on how to use the storage when they received the unit. The lowest percentage of voucher recipients receiving information was in Liquica (81.2%). The highest was in Lautem with 100% demonstrating complete coverage. The average was 91.8%. Figure 3.4 illustrates what medium was used to deliver the information. Verbal was an important method with 39%, and ‘two or more’ was highest with around 57%. The low scoring of written materials (3%) and demonstration (less than 1%) are worrisome, but could be included in the ‘2 or more’ category. To get a better perspective on what methods are used, it could be better to remove the two or more category and require people to answer for each of the methods. The mobile phone videos aimed to take advantage of the increasing ownership of mobile phones that allow the sharing and showing of videos. Mercy Corps produced three videos on the topics of seeds selection, drying and storage and were distributed among: 1) Ministry of Agriculture and Fisheries (MAF) staff and national and district level; 2) Staff of international and local NGOs; 3) Shops selling agriculture inputs and 4) Shops selling mobile phones, and often music videos. The number of people, however, that have seen any of the videos is low (less than 1%), and succeeding programs will need to look at what is hampering their use and understand how this can be maximized. Table 3.11: Households receiving information at time of handover unit (% per district) District N % Bobonaro 101 90.1% Covalima 100 96.0% Lautem 98 100.0% Liquica 101 81.2% Total 400 91.8% Figure 3.5: Sources of additional information after purchase of silo (% of households – N=368) 28 | Page ESS Phase II - Final Evaluation Figure 3.5 presents the data on information sharing that respondents received after buying the storage unit. The NGO extension worker was most important for about 69.3% of the households, followed by local leaders with 37.8%. Radio and television were of minor importance, with a combined total of 3.1%. The MAF extension worker provides only follow-up to 27% of the respondents. Despite the program requiring training participation for voucher recipients, when asked whether they received training in post-harvest handling, 63.1% respondent answered positively. From these, 58.5% said they received this from NGO workers, with 9.1% responding from the MAF. A 32.4% said they received training provided by both the NGO and MAF. The low percentage of respondents having attended training is probably caused by other household members having participated in the training. The participation or complementation by MAF is collaborated by participants in FGDs. It was mentioned that training in production techniques was provided by MAF, but no further knowledge sharing on seed selection and storage. Figure 3.6 presents the complete figures per district and source of the training. The Y-axle gives both the number of households receiving training and the percentage of respondents per district. The Figure presents the following: • Training by the NGO alone is most common with 58.5% of the households. • Bobonaro and Liquica have the lowest number and percentage of households participating in post￾harvest training. • In Liquica MAF was the main source of training, which contradicts the earlier noted statement that FGD participants did not receive training. This could indicate that training by MAF is very suco specific. • While Bobonaro has a low number of recipients of training, the data shows that MAF and the NGO extension workers are working closely together. This could be due to NGO OHM having its main office in Bobonaro and therefore able to work closely with MAF. • Lautem has the highest coverage, 100% of households, in post-harvest training. In 29.3% of the cases this is done in the presence of the MAF extension workers. No training is provided by the MAF alone. • The overall average of 32.4% of the households receiving training from both NGO and MAF, and with a range of 25.0 – 41.9% across district is encouraging. 3.7 Adoption of improved techniques Figure 3.6: Source of post-harvest training by district (% of respondents) 29 | Page ESS Phase II - Final Evaluation The complete indicator is: Percentage of farmers adopting improved technique(s) The target for this indicator is 80% of the farmers to apply knowledge gained through trainings and other activities. Table 3.12 provides the percentage of respondents to the survey that state that they are applying the knowledge. The table provides information for the whole sample, and per district. Several scores did not reach the 80% target, but the higher scores are in line with the focus of ESS: 1) Seed selection (56.8%), 2) Better drying, (59.6%) and 3) Better storage, (>80%). The data for the districts show a high variation. The colors illustrate the following for each of the techniques: • Red, best performing district • Dark orange, second best performing district • Light orange, third best performing district • Yellow, the worst performing district. Considering the focus of the ESS program, Covalima households appear to perform best, followed by Bobonaro, then Lautem and Liquica. 3.8 SILC indicators The ESS program incorporated savings and lending activities as an additional component during the second year of Phase II. It did so by taking advantage of new OFDA guidelines that included a new sector Economic Recovery and Market systems, and the Sub-Sector of Microfinance. Table 3.12: Application of improved technology (% of responding HHs) Technique N % of N Bobonaro Covalima Lautem Liquica Better planting distances and seeding 250 35.2 23.8 26.6 37.4 60.6 Using improved seed varieties 250 23.2 9.5 10.1 37.4 27.3 Timing of harvesting 250 34.8 42.9 35.4 32.3 27.3 Better seed selection from field 250 56.8 57.1 59.5 63.6 24.2 Better drying of harvest 250 59.6 66.7 87.3 36.4 48.5 Storing improved seeds separate from local seeds 401 88.5 95.1 96.0 79.8 83.2 Storing food separate from seeds (local varieties) 393 88.8 82.4 100.0 82.5 91.5 Storing food separate from seeds (improved varieties) 130 96.9 94.4 100.0 97.6 97.7 30 | Page ESS Phase II - Final Evaluation The program adopted CRS’ SILC methodology, to increase the access of rural farmers to financial services. The methodology establishes groups of interested individuals with the primary reason to provide a venue for savings. Over time, groups may decide to provide loans among themselves and even outsiders. The experience and an increase in financial literacy might in the future make members eligible to access services from mainstream financial institutions. The inclusion in the program was approved in May 2014, hence leaving only eight months to build the internal capacity and formation and training to groups. This evaluation does not have the objective to evaluate the progress, but can merely present the achievements made so far. Chapter 5 includes some further observations to reflect on for future programming. Table 3.13 presents the indicators, target and achievements reached. All three indicators are standard OFDA indicators. Table 3.13: SILC indicators, target and achievements Indicator Target Achievement Number of people, by sex or MSEs newly receiving financial services due to USAID/OFDA support 1,000 750 (51% women) Percentage of financial service accounts/groups supported by USAID/OFDA that are functioning properly 100% (Target 50 groups to be formed) 100% of 41 groups formed and functioning Total USD amount channeled into the program are through sub-sector activities 0 0 The following need to be mentioned in regards to Table 3.13: • The SILC methodology is composed of nine successive steps. Not all groups have completed the nine steps, but are functioning as expected at the level they have acquired during the ESS Phase II. The remaining steps will be completed during Phase III. • The program absorbed a number of existing savings and lending groups. The members of these groups showed interest in the program because SILC provides a structure with higher transparency and accountability compared to the initiatives groups were formed under. • The program falls short on the targets primarily due to a later start than anticipated in the proposed program modification. • The OFDA indicators are focused on reaching a number of beneficiaries. Mercy Corps and CRS have developed internal indicators to improve the measurements of progress. Indicators include changes in saving habits and debts and changes in how this impacts access to productive assets and hunger scale. A baseline including these additional indicators was conducted in August 2014. 31 | Page ESS Phase II - Final Evaluation 4. Assessment of DAC criteria’s This chapter looks at each of the DAC criteria’s and the questions identified in the TOR. The questions are printed in each of the sections for reference. 4.1 Relevance The criteria for relevance are assessed through the questions in the box below, and this section is structured accordingly. Validity of Program The program objective reads as follows: Sustainable post-harvest protection of seed stocks to ensure seed security in rural Timor-Leste. The FGDs in communities confirm that the protection of seeds and grains remains a challenge within maize producing communities. The consultations surfaced that traditional methods of storage are still applied and lead to high losses. The endline survey conducted as part of this evaluation shows that even households that have purchased silos still are using traditional methods too. The Ministry of Agriculture and Forestry (MAF) does not have the means to promote secure seed storage methods, and in particular, support this by distributing or selling secure silos. Several programs, like that of FAO, have been supporting the production and distribution of silos, and more recently IFAD is providing $10 new oil drums to households. However, these programs are aiming for the storage of food, and not stressing the importance of having quality seeds for future production period. The ESS program has made important contributions to seed security in Timor-Leste, but this evaluation also concludes that the resulting increase in food availability does not suffice to reach food security. The increased food availability also does not address prevailing nutrition issues in Timor-Leste, and it is recommended that a more comprehensive approach be taken towards improving nutrition in terms of quantity and quality of food intake. An important component in this expansion Phase is the incorporation of saving and lending. The inclusion was recommended by the Pilot Phase evaluation and allowed through the latest OFDA proposal guidelines. Mercy Corps submitted a program modification to include the establishment of Savings and Internal Lending Evaluation questions: • To what extent are the objectives of the program still valid? • Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? • Are the activities and outputs of the program consistent with the intended impacts and effects? 32 | Page ESS Phase II - Final Evaluation Communities (SILC). The inclusion of SILC aims to bring financial services to communities that do not have access to such services due to their remoteness or other limitations. The access to financial services will contribute to the sustainability of the interventions, increasing the relevance for the program beneficiaries. Consultation with SILC group members, both men and women, highlighted that being a member of a savings group gives a purpose to extra money. In the past, extra cash was wasted on unnecessary expenditures. They also said that when they have the opportunity to take out loans this will be used for productive activities or the education of children; while spare money was previously often spent on cultural activities. Men and women groups did prioritize the use of loans differently. Women used them in the order of: 1) education, 2) business activities, and 3) agricultural production. Men mainly used them for business, for example trading of the production or animals from the community, followed by loans for the education of their children. Their reasoning is that if you have business, this will also generate money for education. Consistency of Program Logic The activities and outputs can be divided in four main groups: • Product development and promotion • Selection, orientation and training of voucher recipients • Selection of, and support to blacksmiths to deliver quality silos • Introduction of SILC Product development and promotion The design of products started during the Pilot Phase and based on the learnings the 35kg unit was retained while the 75kg unit to be redesigned as one with a big lid. The metal silos are still a very effective way of preventing seed losses, but application is not necessary as expected. Many people in FGDs say that the bigger units are more appropriate for food storage, and data presented in this report suggest that the 75kg is too big for seed storage. In this sense the program is not that relevant if strictly looking at ‘seed storage models’ as stated in the program goal. Possibly more promotion should take place on establishing emergency stocks, but even for these the 35 kg unit could be enough for most farmers. The promotion of silos is done by blacksmiths under the guidance of a business development advisor of Mercy Corps. The activities have included the presence of the product on public markets, using banners in shops and at the blacksmiths’ workshop and even big banners to promote the brand ‘Silo’ as a national brand. The inclusion of GPB as a product and promotion as such has been less successful than hoped. The product was imported (at a high cost using DHL) from the Philippines and was sold separately and included in the 75kg units, but not much is known how they are used. The product is currently also only available marketed by Loja Agrikultura in Dili, which at the time of the evaluation was out of stock and no new orders placed. Use of vouchers The use of the voucher system proved to be a good way of gaining the interest of farmers, while at the same exposing farmers to a market based approach towards input provision. As per the 2010 census, the total number of maize producing households is 82,607 in the 11 districts covered by the program. Table 4.1 shows the percentage of maize producers that are beneficiaries. The program has, including the Pilot Phase, 33 | Page ESS Phase II - Final Evaluation distributed 27,055 vouchers. This is 32.8% of the maize-growing households as presented in Table 4.1. The table shows that some of the districts have almost full coverage already. Manufahi has 90.7% coverage, followed by Manatuto with 89.7%. The latter was achieved in Phase II alone. With the exception of Bobonaro (8.3%) and Liquica (4.8%), all the other districts have coverage of above 20% of maize farming households. Assuming that the additional 8,659 are mainly new households brings the overall coverage up to around 42.0%. The use of vouchers is often applied as an incentive to have farmers try new technology. The high coverage of this program might already have surpassed that in some of the districts. This is further supported by almost 24.2% of the units being purchased through full payment, suggesting that the product is known and seen as effective. While there are arguments in communities that their own contribution is still too high, and the unit at full price too expensive, continuing the use of a high number of vouchers will work against market approach principles. Households will continue to expect a voucher and people are hesitant to purchase using their own resources. Blacksmiths Blacksmiths are playing a major role towards making the program more sustainable and scalable. The decision to continue to use blacksmiths near the locations of the farmers is still the most appropriate. Local partners made the argument that the program should work with even more blacksmiths who are closer to the beneficiaries. This, they argued, would increase access by farmers. The core activities, besides producing the silos, were business skills training and product diversification. The combination will assist the blacksmiths to continue making silos, and other products beyond the program duration. Introduction of SILC The introduction of SILC has also much to do with the sustainability and scalability of improved seed storage. The access to financial services should allow farmers to access loans. These could be used to buy replacement or additional silos, but also help farmers access emergency loans or invest in additional productive activities. Impact and effects 8 Source: Census 2010 9 Including 1120 of the Pilot Phase 10 Including 1217 of the Pilot Phase Table 4.1: Coverage of maize farmers per district (%) District # of Vouchers # of maize farmers8 Coverage (%) Aeliu 1,267 5,508 23.0% Ainaro9 3,483 7,166 48.6% Baucau 3,500 10,893 32.1% Bobonaro 931 11,176 8.3% Covalima 1,823 6,398 28.5% Ermera 2,881 13,962 20.6% Lautem 3,000 5,908 50.8% Liquica 349 7,244 4.8% Manufahi10 3,613 3,985 90.7% Manatuto 3,208 3,578 89.7% Viqueque 3,000 6,789 44.2% Total 27,055 82,607 32.8% 34 | Page ESS Phase II - Final Evaluation The main program activities are all contributing to an impact that should be long lasting. The counterpart contribution required by the program contributes to people making a shift from recipients of aid to active participants in developing communities and economies. For the long term, the input provision should be guided by market based approaches and possibly social services provided by the government. The ESS program fits with Timor-Leste’s goal of increasing food security and ESS will make a major contribution towards achieving this. The program is well received and its importance recognized by the MAF. The ESS program also fits seamlessly in the country strategies of both international NGOs. It contributes to Mercy Corps’ strategic objectives in the areas of sustainable nutrition, food and income security, access to financial services, and increase resilience of vulnerable rural and peri-urban Households to the impact of Disasters, Environmental Degradation and Climate Change. CRS is in the process of streamlining all livelihoods activities around food security and nutrition in the Least Developed sucos in the three districts covered also by ESS. 4.2 Effectiveness The Effectiveness criteria is assessed using the questions as per TOR and shown in the box below. Extent of achievements toward objectives Table 4.2 gives a summary of the achievements against the indicators. The figures are taken from Chapter 3 and deeper interpretation and analysis of the figures can be found there. The achievements of the program are highlighted using colors: green for fully achieved, yellow for partially achieved and orange for not achieved. Overall, it can be concluded that the program reached the majority of its targets, with a shortfall in the adoption of technologies. Table 4.2: Program achievements against indicators Indicator Baseline Target Achievements Projected increase in # of months of food self￾sufficiency due to seed systems/agricultural input for beneficiary households Recalculated11 months of food self-sufficiency through endline evaluation is 2.5 months. 30% Increase in number of months with food self￾sufficiency = 0.75 months • Average of 2.1 months increase in food self-sufficiency, equivalent to an 80.4% increase. 11 See Footnote 4 Evaluation questions: • To what extent were the objectives achieved? • What were the major factors influencing the achievement or non-achievement of the objectives? 35 | Page ESS Phase II - Final Evaluation # of people benefiting from seed systems/agricultural input activities, by sex 0 240,000 • 249,562 • 47% female Percentage of beneficiaries reporting decreased post-harvest losses for seeds 93% for improved variety12 growers report losses and 95% report losses of other varieties during storage 80% reduction from the Baseline • 87.7% of HHs report more maize seed available through using improved post-harvest handling • 80.3% for rice seeds • 73.7% for beans Increase in availability of quality seed during planting season 39% of improved variety seeds are lost during storage and 45% of other varieties 80% reduction from the Baseline. Equivalent to target losses of: 7.8% for improved varieties 9.0% for local varieties • 23.2% of Improved variety seeds are lost  Reduction of 40.5% • 23.9% of Local variety seeds are lost  Reduction of 46.9% Number of farmers with access to improved seed storage system, by sex 0 40,000 (silos and/or GrainPro bags) • Total of 45,649 • 11.4% above target • Including 9,936 by GrainPro Bags Number of farmers with access to BCC materials/training 0 40,000 • Total of 42,518 • 6.3% above target Percentage of farmers adopting improved technique(s) 0 80% • 88.5% storing improved varieties from local seeds • 88.8% storing food separate from seeds • Other techniques vary from 34.8% - 59.6% application Major contributing factors Geographical Focus NGOs Mercy Corps partnered after the Pilot Phase with CRS. CRS works in the eastern districts and the ESS program benefited from the established network of local partners. As a result, the program easily launched its activities in the districts of Baucau, Lautem and Viqueque. CRS’ local partners worked in each district, as opposed to the two Mercy Corps partners which covered eight (IMM: Ainaro, Covalima, Manufahi and Manututo. OHM: Aileu, Bobonaro, Ermera and Liquica). Informants reported that they felt that NGOs concentrating on a smaller area has led to better results; however, this feeling is not completely supported by other findings presented elsewhere in this report. In addition, CRS’ partners had challenges in meeting the targets. Having staff from the location was stated to increase effectiveness and collaboration with the MAF. Improved input supply The Pilot Phase evaluation analyzed the production costs of the silos. The evaluation demonstrated that the two blacksmiths were taking a loss on the big units, which was compensated by the high production of profit 12 This refers to Sele, the maize variety introduced by the government Seeds of Life program 36 | Page ESS Phase II - Final Evaluation making units of 35kg. Since the Pilot, more input suppliers have entered the market, providing cheaper alternatives to the metal sheeting materials previously used. The blacksmiths are now making a profit of $5-7 per unit, regardless big or small. The blacksmiths are now covering the transportation cost from their factory to the communities and there is, in certain areas, coordination in transport. This includes the bulk buying and transport of materials from Dili to the districts. Systemized beneficiary selection The program had a, by some informants called ‘ambitious’, high target number of farmers to be reached. The process of beneficiary selection to the delivery of the silos was streamlined in Phase II with increased local participation. This allowed local partners in many areas to move quickly from one community to another. In some areas there were still some challenges with blacksmiths not being able to deliver the orders on time. The program was responsive to this and through meetings with the blacksmiths increased collaboration and helped each other in reaching the targets. Table 4.3 presents the selection criteria for voucher recipients and, where relevant, the percentage of compliance as evidenced through the endline survey. Table 4.3: Selection criteria voucher recipients Selection criteria Endline survey 1 Maize producing household 98.0% 2 Willing to participate in the training 3 Willingness/interest to learn about post-harvest techniques 68.9% received training 4 Willing to participate in other project activities 5 Willing to make cash contribution in addition to voucher value 100% 6 Priority to resource-poor households, for example Female Headed Households (FHH), elderly farmers etc. FHH: 9.0% Elderly: 25.9% of HH heads are 60 or older 7 Household resides within target suco Table 4.3 shows the following: • Not all beneficiaries are maize farmers. • The farmers might have expressed willingness to be trained, but only around 70% stated to have received training. Reportedly this was identified during regular M&E activities and partners instructed to do additional trainings, but as per endline survey it appears that complete coverage was still not reached. • The 9% Female Headed Households (FHH) is lower than the national average of 16%13 and varied across districts. Lautem had the highest with 18.9%, followed by Liquica (6.9%), Bobonaro (5.9%) and 13 National Census 2010 37 | Page ESS Phase II - Final Evaluation Covalima (4%). It shows that in most of the districts the targeting of FHH was poor, or they were not able to meet some of the other criteria. Box 2 presents the planned steps in the mobilization process. The process includes a high level of transparency by publicly announcing the list of beneficiaries. Households that feel that they should be included can make their argument and if they are eligible can still be included in the list. The local partners reported that at times it was still challenging to get people to contribute to the cost of the silo, but the process of socialization, training and coordination with local authorities was seen as very productive. In fact, during the FGD with the local partner representatives, step two in Box 2 was, according to partners, in most of cases was done by the head of the village. Major factors hampering achievements Local partner capacity A number of local partners reportedly had many capacity gaps and this hampered the smooth operations of the project. These were found both in the lack in numbers or dedication of staff and the management capacity of the organizations. Mercy Corps also felt that that at times there was an unwillingness to be transparent and follow a high standard of compliance procedures, especially in handling voucher distribution. The output of the partners up to the second quarter of 2014 was critically low, making Mercy Corps decide to employ four field officers to augment the partner staff. CRS mentioned less challenges with the local NGO partners, thanks to their longer-term relationship and challenges faced were rather at the level of community mobilization than organizational. CRS also recognized that the already mentioned ‘ambitious’ targets fueled challenges in community mobilization. CRS and Mercy Corps utilized different approaches for partner capacity building. CRS’ partners stated that they received the following trainings to strengthen their systems and management: • Financial management • Bookkeeping for finance staff • Project management – activity management • Leadership training for the director • Procurement training • Monitoring and Evaluation Meanwhile, Mercy Corps employed a different approach. Instead of conducting ‘formal’ trainings on certain topics, Mercy Corps provided hands-on coaching on various issues including finance and operations, program Box 2: Farmer mobilization process: 1. Socialization (project staff, Chief Hamlet, MAF extension worker) 2. Selection of beneficiaries according to criteria (project staff, Chief Hamlet, MAF extension worker) 3. Dissemination of list in communities (verbal and written) 4. Selection final beneficiaries to training 5. Distribution of voucher at end of training 6. Informing blacksmith of number of voucher 7. Distribution of silos 38 | Page ESS Phase II - Final Evaluation planning, and M&E. Thus, when asked what kind of training they received from Mercy Corps, the partners stated that they received only Monitoring and Evaluation training and no other management trainings. This is most likely because as they did not view the coaching and one-on-one mentoring with finance or senior program staff as trainings. Capacity of blacksmiths The number of blacksmiths participating in the ESS program expanded from two in the Pilot Phase to 17 in Phase II. They were all trained by the Food and Agricultural Organization (FAO) to making silos for food storage. The blacksmith visited in Liquica produced silos for FAO from 2001 to 2004, after which he was engaged with other programs until 2006. Following this he did not produce silos until the ESS program in 2013. The blacksmiths were selected based on their willingness to participate in the program and revive their previous factories. Initially, 17 new blacksmiths were interested, but only 15 joined the training and program, in addition to the two active during the Pilot Phase. The blacksmith visited was, according to the informants, one of the least successful in terms of the number of units sold and in particular the timely production of the silos. He is, however, not a stand￾alone and other blacksmiths were unable to produce the number required. This resulted in others filling the gaps. This was initially uncoordinated and caused problems, but was resolved after a meeting. Figure 4.1 illustrates the variation in silo production across the blacksmiths. It shows that one produced about 20% of all silos, while others only produced a few hundreds. To increase the entrepreneurial capacity of the blacksmiths, all received training on business skills, like costing, pricing and marketing skills. The blacksmiths also participated in two training workshops on product diversification and visited a successful blacksmith in Baucau. The bigger producers have a number of employees and diversified to other products like water containers, cooling boxes and cooking utilities. The local partners stated that diversification is important for blacksmiths, but they expressed the importance that the product development should include marketing research. The most active blacksmiths promoted the silos in the markets, using pushcarts, and worked in partnership with kiosks. The kiosk received $2 per silo sold. Limited market based thinking The engagement with blacksmiths is an important component in the strategy to promote market-based interventions. This is in line with the program objective to establish sustainable mechanisms for farm input provisions. The program, and in particular expressed by Mercy Corps, is however challenged by market based Figure 4.1: Variation in silo production by blacksmiths 39 | Page ESS Phase II - Final Evaluation thinking being relatively new for most NGOs. This is partially also true for CRS, which is moving towards more market based approaches, but historically led its programs through vulnerability based approaches. The blacksmiths, even while they are an active player in the market place, still indicated that they are hesitant of market led input provisions. They expressed during the FGD that they prefer the government standardizing input (material) prices, rather than having fluctuations. The system now applied could actually be described as a hybrid. Inputs are provided following open markets, but the final product is provided at a fixed price. It can also be argued that the high number of subsidized silos is disturbing the development of an open market. Competition and jealousy Agricultural interventions in Timor-Leste are slowly moving to approaches that are requiring recipients to make a contribution. This includes ESS, and also the IFAD program asks for households to pay $10 per drum. There is, however, still the memory of free inputs and it is difficult to convince people to pay a contribution, according to local partners. At the same time, they say there are others who want to, but they are not part of the selected beneficiaries. Someone described it as ‘social jealousy,’ people that did not receive a voucher versus the ones that did. This was however minimized by refresher training for field staff on the mobilization process. High targets compromising quality The ambitiousness of the targets has already been mentioned several times in this report. There is broad agreement among participants in this evaluation that the program focus on meeting a high number of participant beneficiaries has to some extent hampered achieving the full adoption of the promoted techniques and storage systems. No follow-up visits or trainings were provided to reinforce learnings in earlier activities. The communities stated in the FGDs that demonstration was not done during the training. The training had a standard duration of two to three hours. This could be appropriate for the time available and attention span of most people, but without echoing the same message, regardless of the methodology, the activity only focused on distributing the units. 4.3 Efficiency The Efficiency criteria looked at cost-effectiveness and efficiency using the questions in the box below. This section is structured accordingly. Cost-efficiency Evaluation questions: • Were activities cost-efficient? • Were objectives achieved on time? • Was the program or project implemented in the most efficient way compared to alternatives? 40 | Page ESS Phase II - Final Evaluation This aspect of efficiency will be looked at from the angle of activity cost per beneficiary, and how the effects and impact can monetarily be justified. The costs per beneficiary need to be calculated for: • The seed storage silos • The introduction of SILC Table 4.4 below provides the total program costs as derived from financial information provided by Mercy Corps. The table summarizes the costs based on the actual expenditures as of January 2015. The total amount spent on the program as per January 31 is $1,992,527 (including Phase I and Phase II). To evaluate the expansion Phase the amount is reduced by cost of the Pilot Phase. In addition, to allow separate analysis for the silos and SILC the expenditure on SILC needs to be estimated.14 Table 4.4 presents the budget and costs of Phase II after deducting the cost of the Pilot Phase, $2,274,998 and $1,745,026 respectively. These figures resulted in a 76.7% total expenditure for Phase II and is extrapolated over the SILC budget, giving a total of expenditure for SILC of $139,463 and giving a remaining amount of $1,605,563 as expenditure on activities to improve seed storage. In the absence of exact figures spent on SILC this is the best way of segregating costs made for the seed storage and for SILC. Table 4.5 gives the cost per beneficiary and household for the seed storage and the SILC interventions. An interesting question to be answered is whether there are financial justifications to spend this amount on beneficiaries. Potential justifications for cost of the seed storage intervention are included in the table. Table 4.5: Cost of interventions per beneficiary and justifications Intervention Total cost ($) # of Beneficiaries Cost/beneficiary ($) Cost/HH ($) Seed Storage 1,605,563 249,562 6.43 39.89 Justifications • Community members stated that they would have to spend $20-30 to purchase seeds on the market when they loose their seeds during the storage period. This is about half of the cost of the intervention per household, or in other words, in two years the amount would be earned back. • The $20-30 is in the range of the purchasing cost of a silo, making it affordable for people to purchase them with the savings they make by not losing the 14 The expenditure summary shared with the consultant is a compilation of Pilot Phase, Phase II and SILC. Table 4.4: Calculation expenditures ESS Phase II Cost Category Total Phase II Budget Total Phase II Expenditure SILC expenditures ESS expenditures Salaries 206,067 138,634 22,902 115,732 Benefits 128,325 51,621 10,390 41,231 Travel 32,657 44,848 483 44,365 Equipment 70,000 65,500 0 65,500 Supplies 15,600 17,804 6,443 11,360 Program Activities 1,480,397 1,184,391 71,737 1,112,654 Other Direct Cost 111,290 79,021 7,663 71,358 Total Direct Cost 2,044,336 1,581,819 119,618 1,462,201 Indirect Cost Allocation 230,662 163,207 19,845 143,362 Total cost 2,274,998 1,745,026 139,463 1,605,563 % Spent by January 2015 76.7% 41 | Page ESS Phase II - Final Evaluation seeds. • The households that store food traditionally claim that they lose up to 90% of the food, leaving it only suitable for animal consumption. A silo of 75kg can save 75kg of food grain from destruction by pests. This amount will have a replacement value of about $37.50. It would only take a year to recover the investment cost per beneficiary household. SILC 139,463 750 185.95 1,152.89 It can be concluded that the benefits outweigh the costs for the seed storage intervention. The SILC has just been introduced and groups not fully operational yet. The only justification that can be given at this stage is that group members stated in the FGDs that they previously were spending money on unnecessary things due to the lack of a safe place to put the money. The members are currently saving $1-1.25 per week that will add up to $52-65 per year, far off from the cost per beneficiary. With the groups not fully operational yet, it is unknown how the savings will be used and what income this is generating. The complementing activities to increase target member’s financial literacy could lead to additional benefits that yet have to be seen. Assessments will have to be conducted during monitoring and evaluation activities. Timeliness of reaching objectives The positive costs and benefits assessment for the seed storage intervention is supported by an on time delivery of the outputs and achieving them within the budget. It must, however, be mentioned that some adjustments led to achieving the objectives or, important in this program, reaching the number of storage units distributed. 1) Mercy Corps recruited additional field officers in April/May 2014 to support local partners in attaining their targets. 2) Blacksmiths were encouraged to produce storage units faster and blacksmiths assisted colleagues in locations that initially were not in their coverage area. The question of local partners struggling to reach their targets raises an interesting question on the range of costs for the delivery of a silo. Table 4.6 provides the targets and the costs per unit per partner. Table 4.6: Cost per voucher per organization ($) INGO partners Budgeted cost per local partner Costs Mercy Corps CRS IMM OHM CDB KDP TID Total expenditure15 865,530 740,033 Contractual16 86,240 86,240 82,051 62,875 70,830 Vouchers 15,500 9,500 7,750 7,750 3,500 3,000 3,000 15 Calculated by total expenditure on ESS (Table 4.1 of $1,605,563) minus actual grant to CRS ($740,033). 16 These are not corrected since the savings are incorporated in the overall expenditure 42 | Page ESS Phase II - Final Evaluation $/Unit 55.84 95.49 11.13 11.13 23.44 20.96 23.61 The table shows that Mercy Corps is able to operate cheaper per voucher at $55.84 than CRS at almost double the cost with $95.49 per voucher. These figures already include the costs of local partners, but for illustration purposes: the cost per voucher as part of the total is for Mercy Corps’ partners with $11.13 much lower than for CRS’ partners. Their costs range from $20.96 - $23.61. The gap is actually bigger if considering that IMM and OHM did not spent their full budget as per January 2015. They spent only 63% and 69% respectively of the budget presented in Table 4.6. The under spending can be a proxy indicator for the quality of the management, which Mercy Corps struggled with. The differences in Table 4.6 show that the program, if only considering costs, has not been operating at the most efficient level possible. The inclusion of other aspects like effectiveness is hard because the survey was only implemented in one CRS district, Lautem. The data does show that Lautem has a full coverage of households receiving information when they receive the silo and 100% of the households participated in post￾harvest handling training. The other indicators do not indicate that their work has been more effective, thus justifying higher expenditure. The performance against indicators on seed survival and application of techniques is not notably better than other districts. For example, see Table 3.12 and its narrative. Efficiency compared to other programs The breakdown of the actual expenditures of the program is presented in Table 4.7. The table shows that 67.9% of expenditures went to program activities, which is a good percentage compared to other rural development programs. However, it needs to be noted that the category ‘Program Activities’ includes also the grant to CRS. CRS will also have its expenditures for salary, benefits and other operational costs comparable with Mercy Corps. As per budget expenditure in January, the cost categories recording the highest under spending were ‘Salaries’ with 73% of the budget spent, and of ‘Staff benefits and allowances’ at 46%. According to the program management, the following led to under spending in these areas: • A delay in staff recruitment led to expenditure for less than the budgeted 24 months. • The program was able to employ qualified staff at lower salaries than budgeted. • Mercy Corps was able to do cost sharing for the market development coordinator when it mobilized funds for other programs. • The program modification to include SILC was approved later than anticipated, leading to savings. • SILC field agents were reclassified as service providers with performance based remuneration, and not as staff members with a regular salary. Table 4.7: Expenditures per cost category (in $ and %) Cost category $ % of total Salaries 138,634 7.9% Benefits & Allowances 51,621 3.0% Travel 44,848 2.6% Equipment 65,500 3.8% Supplies 17,804 1.0% Program activities 1,184,391 67.9% Other Direct Cost 79,021 4.5% Total Direct Cost 1,581,819 90.6% Indirect Cost Allocation 163,207 9.4% Total expenditures 1,745,026 100.0% 43 | Page ESS Phase II - Final Evaluation Interestingly, Mercy Corps added four staff members as field officers for eight to nine months, yet stayed within the program budget. The need to add more staff to reach targets, to be able to do this within budget, and the already allocation for local partners, makes one wonder whether working with local partners is the most effective way of working. An argument exists for building capacity within the country, but one could argue at the same time whether, if no long-term engagement with the partners is expected, direct implementation is more efficient and effective. Other inefficiencies that were observed are: • Allocation of a promotion banner in non-strategic locations, for example in the center of Liquica. The blacksmith was outside the center and was already challenged by number of silos to be produced for the ESS program. If the banner created any demand, it would be unlikely filled by the blacksmith. The banner could have been better placed elsewhere where the silos were easily accessible. Another option would be to put the phone numbers of other producers on the banner. • The budget included money for M&E database development, but there are no expenditures against this. The partners also felt they were wasting time on adjusting to a number of times to changing reporting formats, calling for the need of a uniform system at the beginning and throughout the program. • The reporting channels were also felt to be unclear, or not consistently followed. It would regularly happen that monitoring information was requested from CRS ahead of schedules, or narrative information asked, but there is no feedback or evidence that the information is actually used. • Local partners stated that their target areas were more rice production areas, which made it difficult to convince farmers to use silos. Table 4.1 shows that there is a sufficient number of maize farmers in all districts, and it appears they were using their resources in the wrong locations. • The concern was expressed that the Business Development Coordinator who is supporting the blacksmiths in marketing events, sometimes does not inform local partners. The activities could be more effective if local partners are involved to mobilize people and prevent confusion among partners working in the area. • The baseline study covered communities in Liquica and respondents would have been the first to understand the program. Nevertheless, the local partner did not make use of this early exposure by working in these communities first. 4.4 Impact The Impact of the ESS program can be described as having positive and negative changes as a result of a development program. The Impact will be assessed using the questions in the box below, and come from the consultancy’s TOR. Evaluation questions: • What has happened as a result of the program or project? • What real difference has the activity made to the beneficiaries? • How many people have been affected? 44 | Page ESS Phase II - Final Evaluation Results of the program The following results can be attributed to the program: • The blacksmiths transformed from merely producing an order to businessmen that have business skills and can diversify their products. • The creation of a demand for inputs resulted in a more competitive market environment. More suppliers offering materials reducing the cost of silo production. • The buying of silos at full price proves there is acknowledgement of the importance of good seed storage and the quality of the provided product. • The endline survey demonstrates the following differences in the lives of beneficiaries: o Majority of households store seeds using improved storage methods like metal silos, at times combined with GPBs or airtight plastic containers o Increase in food self-sufficiency o Increase in seed availability, not only for maize, but also rice and beans o Majority of households are storing food separate from seeds o Majority of households are storing seeds of improved varieties separate from local varieties Figure 4.2 demonstrates that the silos are used beyond the original purpose of maize storage. Many farmers are also using the silos for rice and beans in seasons when maize is not available. Rice seed is stored in the 75kg unit with a GPB (38.1%), and 35 kg units (30.2%) and for 23.8% in 75kg with plastic bottles. Beans appears to be stored more in plastic bottles in the 75 kg units (38.9%), followed by the 35kg with 32.2%. This could be because beans are stored in smaller volumes. Nine households are storing, or have stored rice and beans together in a 75 kg unit using a GPB or plastic bottles. A number of households, 49 for rice and 32 for beans, reported that they also store rice and beans intended for seeds food in the silos. This follows a similar pattern as presented in Figure 4.2 for seeds. The baseline and the endline survey explored ownership and decision-making within the households. The program did not have any direct activities for men and women in the communities, hence the data was not reported earlier, but did aim for inclusion of women as beneficiaries. Table 4.8 presents the data from both the baseline and endline surveys. The cells in green highlight increases for the persons responsible, while pink highlight a decrease. Table 4.8: Responsible person(s) in seed production, Baseline and Endline (% of households) Responsible person(s) Preparing land Planting seeds Selecting seeds Drying seeds Storage of seeds Baseline Endline Baseline Endline Baseline Endline Baseline Endline Baseline Endline Figure 4.2: Storage methods used for rice and beans (% of HH using for rice and beans) 45 | Page ESS Phase II - Final Evaluation Main Male or husband 48% 65% 3% 6% 4% 6% 3% 5% 7% 11% Main female or wife 13% 9% 44% 57% 56% 65% 49% 51% 40% 52% Husband and wife jointly 27% 15% 23% 14% 28% 23% 31% 29% 44% 29% Someone else in the household 8% 2% 16% 11% 10% 3% 11% 5% 5% 4% Male, female and others 4% 9% 15% 11% 2% 2% 5% 9% 4% 4% Table 4.8 demonstrates the following: • Decreases can be noted in the joint responsibility of husband and wife and other members in the household for all activities. • Part of the decreases seems to move to a joint responsibility by all household members, the ‘Male, female and others’ group, but much of the increases going to ‘Main female, or wife.’ Women have increased responsibility especially in the activities for planting seeds, selecting seeds and storage of seeds women. • The highest increase in responsibility for ‘Main male or husband’ is recorded in ‘Preparing land’. • The increase in responsibility of women could be interpreted both ways, an increase in burden in terms of activities, or empowerment through the control and ownership over assets. Table 4.9 provides data from the surveys on the persons making decisions within the households. As with Table 4.8, green highlights increases and pink decreases. The table shows an overall shift towards more decision making by the ‘Main male or husband’. Women seem to have only gained more decision making power in the areas of ‘Investment in agriculture’ and the purchase of daily food. Table 4.9: Decision making power within households, Baseline and Endline (% of households) Responsible person(s) Selling large assets Buying large assets Investments in agriculture What food crops to plant Baseline Endline Baseline Endline Baseline Endline Baseline Endline Main Male or husband 21% 36% 17% 28% 47% 39% 13% 19% Main female or wife 16% 9% 18% 10% 18% 15% 28% 33% Husband and wife jointly 60% 54% 62% 59% 33% 43% 52% 44% Someone else in the household 3% 1% 3% 1% 1% 2% 7% 1% Male, female and others 0% 1% 0% 1% 0% 2% 0% 3% Responsible person(s) What cash crops to plant Attending meetings or activities Buying food What to do with farm income Baseline Endline Baseline Endline Baseline Endline Baseline Endline Main Male or husband 9% 16% 56% 60% 15% 13% 13% 26% Main female or wife 31% 29% 21% 11% 47% 54% 29% 15% Husband and wife jointly 55% 50% 21% 24% 32% 31% 55% 57% Someone else in the household 5% 2% 3% 2% 6% 1% 3% 1% 46 | Page ESS Phase II - Final Evaluation Affected number of people The program affected the following number of people: • 249,562 people through seed systems and agricultural input activities (47% female) • 4,650 through saving and lending activities • MAF suco extension workers through training • 34 partner staff through capacity building activities • 17 blacksmiths active in the program and an estimated 34 additional employees. • Owner and employees of Loja Agrikultura in Dili plus 17 kiosks selling silos and GP 4.5 Sustainability The Sustainability criteria aims to assess whether benefits are likely to continue. This section will use the questions in the box below to assess this, and what factors influenced sustainability. Continuation of program benefits The evaluation was conducted in January 2015, which is still part of the program period. The first question is therefore better formulated as: To what extent are benefits of the program likely to continue after donor funding? The original question is also less relevant since the donor will continue funding for another two years. Depending on the design of the Phase II, the commitment itself could be seen as an important contributing factor that benefits will be sustained. Table 4.10 summarizes the benefits of the program that are expected to continue and the major factors that influenced sustainability. Table 4.10: Program benefits expected to continue and their contributing factors Continuing benefits Contributing factors Silo production by a number blacksmiths • Business training • Marketing training • Product diversification to decrease dependence on silo production alone • Link to other CRS and MC programs Increased food self-sufficiency • Training of farmers in good seed production and storage methods Increased seed availability • Ownership and access to effective seed silos Male, female and others 0% 3% 0% 3% 0% 2% 0% 1% Evaluation questions: • To what extent did the benefits of a program continue after donor funding ceased? • What were the major factors which influenced the achievement or non-achievement of sustainability of the program or project? 47 | Page ESS Phase II - Final Evaluation • Coordination with the MAF extension workers • Knowledge imparted in the MAF extension workers Table 4.10 says on purpose ‘ a number of blacksmiths.’ The local partners expressed their concern that the smaller ones are still very dependent on the orders of silos through vouchers and expect them to stop business altogether after ESS. The blacksmiths also mentioned other programs like the IFAD oil drums as a threat. They consider silos for food storage as a potential product to sell, but are challenged by the IFAD program. Other factors that might threaten sustainability are: • Availability of materials in Dili. The blacksmiths experienced some gaps in the supply chain that forced them to temporarily stop production. • Continuing ‘wait and see’ attitude of people expecting more vouchers to be distributed in future programs. The blacksmiths stated that it is only the people who have not heard of the voucher system that are buying the silos with their own resources. • The farmers continue considering the full price for the silos too high. 5. Lessons learned The chapter is divided in two sections. Section 5.1 presents the lessons learned from the interventions as part of the original ESS program focusing on the silos. Section 5.2 gives initial lessons learned from the establishment of SILC groups. 5.1 ESS Program This section identifies and describes lessons learned as experienced or reported during the evaluation. Balance targets with sufficient follow-up activities The informants during this evaluation all shared the opinion that the program was very focused on reaching the targeted number of beneficiaries. It was felt that quality was compromised by limited interaction with beneficiaries in the form of training and a lack of follow-up activities. The program would have benefited from a better balance of mobilization and follow-up activities with beneficiaries. The under achievements towards indicators on technology adoption support this. During the program, it was experienced that mobilization processes were not followed by local partners and was attributed to the pressure to reach targets. Productive capacity of blacksmiths The program partners suffered delays due to the inability of a number of blacksmiths to deliver ordered silos in time. A number of reasons were given including not putting aside capital to purchase materials and not employing additional workers. Local partners also stated that for some, making silos is not their main livelihood. The factors above need to be balanced with program needs and/or possibilities. The price of the silos is not an issue, compared to the unprofitable operation of the blacksmiths during the Pilot Phase. The 48 | Page ESS Phase II - Final Evaluation blacksmiths actually mentioned that if there is no longer a project, they could consider lowering the price to attract more customers if material prices allow. Balancing field staff with reality of environment The lesson here is two-fold. First, the field staff, whether of the INGO or local partner, needs to be allocated according to the challenges that are present in the district. This means that one cannot base it on the size or area, but factors like terrain and accessibility need to be considered when deciding on the size of the area of responsibility of field staff. Different levels of accessibility may have to inform the area and number of beneficiaries per field officer. Secondly, the familiarity with the physical but also the institutional environment has been identified as an important factor to ensure success and an existing presence of the local partner in the district. It was found that partners give less attention to districts far from their base and communication with blacksmiths is at times poor. In the absence of the NGOs, the silos cannot be distributed causing delays. Direct implementation versus implementation through local partners The lesson here is related to the one above where it is identified that effectiveness relates to the presence of the local partner. A focus of many INGOs is to work with local organizations. An argument exists for building capacity within the country, but one could argue at the same time whether, if no long-term engagement with the partners is expected, direct implementation is more efficient and effective. This would be especially possible in areas where there are no existing local organizations. Feedback mechanisms increasing accountability The program applied two feedback mechanisms, one during the selection of beneficiaries and another one among beneficiaries in the form of a customer feedback survey. Both were experienced as methods that increase accountability and transparency towards beneficiaries, and increase the quality and satisfaction about the product that the program offers. Challenges of a weak market system The program experienced the advantages of an opening economy where more traders are stocking the supplies needed for the silos. However, there is still an attitude among people that much should be provided by the government and prices standardized. The program used vouchers to increase people’s to input markets, but the impact of the program in changing attitudes and marketing systems goes beyond the program indicators and thus the scope of this evaluation. Inclusion of finance department in designing Standard Operating Procedures (SOP) The SOP for the distribution of vouchers was questioned during an internal financial audit. The monitoring forms did not provide sufficient information to comply with accountability requirements. The inclusion of operations and finance departments in the development of SOPs for distribution and other systems will be ensured in the future to make procedures compliant. Branding and marketing by professionals 49 | Page ESS Phase II - Final Evaluation The program went through an elaborate process of branding the project and products, especially the silos. Consequently, the program partners also did the marketing. This participatory process might have been a good exercise, but at the same time, was considered time consuming. It was felt that using professionals to do the branding and assisting with the identification of appropriate marketing materials and channels could have been more efficient. 5.2 SILC The SILC activities have only recently started and thus it is too early to make recommendations or document lessons learned. The points below are more appropriately called attention points for the program management in the continuation of SILC activities. The points below are observations and interpretations of statements made by program participants: • The FGD participants referred to groups that were smaller and bigger than the SILC group size of 15- 25. • Women are occupying leadership positions in the groups, but it is difficult to find women that can take on the role of Field Agent (FA). Cultural and social rules prevented potential candidates to travel to trainings. As a result, only two out of nine FAs are women. Finding women that have the required educational background was also cited as a challenge to have women as FAs. • The concept of the Private Service Provider (PSP) as the future role of the FAs seems clear to most stakeholders, but the FAs themselves prefer that certain trainings like management will be provided directly to the group members. However, doing so will leave them with insufficient skills to operate as a full PSP. • The FAs participating in the FGD thanked the consultant for giving the opportunity to provide feedback. It was the first time, according to them, to have such an opportunity. They also appeared to have problems with payments, and had not received any compensation in six months, hampering the completion of their work. This could have been a result of the misunderstanding in the beginning of FAs being service providers or full staff. • The FAs cited that they were not always able to reach areas in time because of the lack of transport. The FAs ideally are as close as possible to the location of the groups. • The consultant found it challenging to get information on activities from the program staff. There was little understanding about SILC as well as little knowledge presented on what is happening in the field. • The existing group members are convinced that when others seeing SILC groups work, and the transparent manner it operates, interest to join groups will grow. They have the ambition to grow and form an association that can provide loans to outsiders as well. • The group members saw borrowing from family members as a common way of accessing loans if needed. The inability to pay could lead to conflicts and, as such, SILC could be seen as a conflict prevention measure. 6. Conclusions and Recommendations 50 | Page ESS Phase II - Final Evaluation This chapter is formatted as a table that provides conclusions and recommendations for future similar interventions where appropriate. Table 6.1 provides conclusions and recommendations on the achievements towards program indicators, while Table 6.2 looks at the DAC criteria. Table 6.1: Conclusions and recommendations – Achievements towards indicators Conclusions Recommendations Indicator: Projected increase in # of months of food self-sufficiency due to seed systems/agricultural input for beneficiary households Baseline: Recalculated months of food self-sufficiency through endline is 2.5 months Target: 30% increase = 0.75 months Target achieved: • Average 2.1 month increase of food self-sufficiency = 84% • Include in future surveys productivity of maize and use of the production to analyze whether increase of maize production is actually used for food The increase in maize availability was not sufficient to cover all food insecure months. • Design more comprehensive food security interventions including nutrition for future programs • Monitoring systems that allow assessing use of increased agricultural production • FGDs at start of Phase III to seek explanation for increase in food insecurity Food insecurity reportedly higher in endline than baseline. Increasing the food availability by the program was not sufficient to address food insecurity. The use of food self-sufficiency as an indicator appears to be insufficient to monitor changes in food security. Indicator: # of people benefiting from seed systems/agricultural input activities, by sex Target: 240,000 The program surpassed, at 263.971, the number of beneficiaries by 10.0%. 47% of the beneficiaries of Phase II were women. Indicator: Percentage of beneficiaries reporting decreased post-harvest losses for seeds Baseline: 93% for improved variety growers report losses and 95% report losses of other varieties during storage Target: 80% reporting decreased losses The program achieved a decrease in post-harvest losses within majority of the households: 87.7% for maize, 80.3% for rice, and 73.7% for beans. Indicator: Increase in availability of quality seed during planting season Baseline: 39% of improved variety seeds are lost during storage and 45% of other varieties Target: 80% increase in availability: reduction of losses to 7.8% and 9.0% respectively Target not achieved: endline data • 23.2% of Improved variety seeds are lost  Reduction of 40.5% • 23.9% of Local variety seeds are lost  Reduction of 46.9% FGDs reported that if using improved storage 100% of the seeds are good for planting after storage. • Incorporate follow-up strategies in succeeding programs to reinforce application of technologies (E.g. visits or SMS messages) • Assess in future surveys seed survival for different storage methods • Follow-up discussions with farmers why they are applying different storage methods. The evaluation survey assumed that beneficiaries would shift completely to the use of silos, but people continued to 51 | Page ESS Phase II - Final Evaluation use also traditional storage methods. Major achievements: • 74.5% of households use metal silos for the storage of local seeds, and 58.6% for improved seeds. This was less than 1% during the baseline. • However, 21.8% of households growing improved varieties report that they are not using any improved storage despite having purchased units. • The use of traditional methods of hanging maize seeds in the kitchen or a tree and store in sacks has been halved. Still, around 30% of households are still using these. The average of storage methods per household is 1.4. • Stress the susceptibility of improved varieties to weevils and increased importance for improved storage • Incorporate follow-up strategies in succeeding programs to reinforce application of technologies (E.g. visits or SMS messages) • Provide feedback to communities on the evaluation results with discussions why farmers are applying a combination of storage methods. The preference for improved methods varies across districts, for example: • Covalima scores high on overall use of improved methods • Bobonaro uses 85% metal silos for local seeds but only 56.8 for improved seeds. • Liquica scores worse on application of plain metal silos, but high on storing improved sheets in metal containers combined with a GPB bag or airtight plastic containers. • Discussion prior with beneficiaries to assess why silos are more commonly used for local varieties. • Conduct barrier analysis The cultivation of improved varieties increased from 23.9 to 33.7%. (The endline data also included the newly released NoiMutin, while baseline was only Sele.) Seed is more available within the household than at time of the baseline and less is purchased from the market. The average amount of kg of improved seeds saved is 17.9, which is much lower than the 35kg size of the smallest unit. The amount of saved seeds has reduced from 23.3 kg in the baseline indicating a reduced in amount to save. Farmers are also planting a smaller area of land, which could be due to the higher productivity. • Discussions with beneficiaries to verify whether there is a relation between productivity and amount of seeds saved. • Included the importance of emergency seed stocks in trainings. Indicator: Number of farmers with access to improved seed storage system, by sex Target: 40,000 Target achieved: 45,649, surpassing the target by 11.4%. The preference for silos changed from 56% of silos during the Pilot were small, to 58% of the silos in Phase II were big, 75kg units. Among the survey respondents, the distribution was equal with 49.9% being small units, but with a variation among districts. Small units were more popular in Lautem and Liquica, while the majority of beneficiaries in Bobonaro and Covalima preferred big units. 52 | Page ESS Phase II - Final Evaluation The variety in suggestions for models indicated that there is space for blacksmiths to do customized production. Indicator: Number of farmers with access to BCC materials/training Target: 40,000 • Total of 42,518 beneficiaries, 6.3% above target. During FGDs it was reported that no demonstration of the use of the unit was done during training. The data of the survey cannot confirm this since for part of the answers (57%) demonstration could be lumped in the option of ‘2 or more’ methods of receiving information. • Future surveys to ask separately whether information is received for each of the information sources. • Institutionalize the use of units for demonstration during trainings. Units should be available during training. The NGO worker was the most important source of follow￾up information (69.3%). The local leader also is important providing information to 37.8% of the households. • Local leader standard as participant in trainings and consider provision of a silo for leaders to motivated leaders as an incentive. The mobile phone videos that were produced to easily share information among individuals were seen by less than 1% of the households. • Check with recipient of videos whether they still have and use them. (E.g. MAF and NGO staff) • Assess barriers for not using the videos. The attendance in training of post-harvest handling is a requirement to receive a voucher, but only 63.1% responded that they received training. • Review monitoring processes and documentation of registering training participants Percentage of farmers adopting improved technique(s) Target: 80% Target was not achieved for all techniques: • 56.8% for better seed selection • 59.6% for better drying of seeds • 88.5% for storing local varieties separate from improved varieties • Storing food separately from seeds: 88.8% for local varieties and 96.9% for improved varieties. • Incorporate follow-up strategies in succeeding programs to reinforce application of technologies (E.g. visits or SMS messages) • Conduct demonstrations following the cropping calendar. • Conduct barrier analysis The SILC interventions started later than anticipated and only 75% of the targeted number of beneficiaries was achieved, and 41 out of the target 50 groups organized. • Completion of group formation and training during ESS Phase III. Table 6.2: Conclusions and recommendations – DAC criteria Conclusions Recommendations Relevance Continuing use of traditional methods confirms the relevance of the program ESS is the only program focusing on seed storage Beneficiaries consider the 75 kg unit too big for seed storage and more appropriate to store food. • Promote further the independence of blacksmiths in making customized products In several districts vouchers already covered a significant portion of the maize production. The use of the voucher as an incentive to try a new product is losing footing, and • Low number of vouchers in Phase II • Prioritize in Phase III districts and communities that have a current coverage by vouchers of less 53 | Page ESS Phase II - Final Evaluation coverage is likely to go beyond resource poor households. than 40%. The activities supporting blacksmiths will help them to become sustainable through good business management and product diversification. • Provide training to blacksmiths in new product identification and market research The ESS program fits with Timor-Leste’s priorities in food security and in the INGOs’ country programs. Expanded relevance through inclusion of SILC. Savings groups seen as very important by community members to have a safe place to put money and reduce spending cash on unnecessary things. • Completion of group formation and training during ESS Phase III. • Documentation of process and lessons Incorporation of SILC indicators in Phase II evaluation survey. The SILC group members prioritized business activities (men) and support the education of children (women) for use of future loans. Business would be primarily trading in agricultural products. • Provision of simple business planning to SILC group members to ensure viability of loans Effectiveness The following factors contributed to achievements: • Expansion to eastern district through CRS with established partners. • Increased competition in suppliers making silo materials cheaper and giving blacksmiths good profit margins of $5-7 per unit. • Improved beneficiary selection process with feedback mechanisms for community members. • Consider the development of feedback mechanisms on the quality of products straight to the blacksmiths. The percentage of Female Headed Households (FHH) was with an average of 9% lower than the national average of 16%. Only Lautem targeted with 18.9% a high percentage of HH. • Conduct barrier analysis with FHH to participate in the program. There are indications that the selection of beneficiaries is still done primarily by the head of the village. • Reinforcement of the mobilization process • Reduction of targets to allow more time for selection of target households like FHHs and vulnerable households. The following factors hampered achieving results: • Capacity and dedication of local partners to do sufficient field work in communities • Variation in capacity of blacksmiths to produce and deliver silos • Limited thinking of stakeholders towards market led development • The program’s high number of participant beneficiaries has to some extent hampered the achievement of full adoption of the promoted techniques as program paid more attention in achieving number of beneficiaries to own improved seed storage systems but less in supporting adoption of promoted good practices. • Capacity assessments with blacksmiths prior to signing agreements with them. • Reduction of number of voucher beneficiaries to remove expectation to get subsidized silos and freeing time for follow-up activities and monitoring of silo and technique adoption. • More market support activities 54 | Page ESS Phase II - Final Evaluation Efficiency Quantitative targets for silo distribution are met only 76.7% budget expenditure The cost per household for the seed storage intervention is $39.89. This amount could easily be justified by the prevention of loss of seeds in two planting seasons. The targets for the SILC component were not met due to the late approval of the proposal modification. The target number of beneficiaries for seed systems were reached, but needed the deployment of additional staff by MC and blacksmiths helping out colleagues that had difficulty meeting their orders. The cost per voucher/silo is almost double for CRS ($95.49) compared to Mercy Corps ($55.84) • Consider a detailed study assessing the cost of working with local and international NGO as partners versus direct implementations. Assessment should consider cost (extra personnel, offices etc.), effectiveness and future potential of local NGOs, among others. • Review of selection criteria and methodology for local partners. The portion of the expenditures going to program activities was at a good, high percentage of 67.9% but this includes also the contract with CRS that also includes personnel and other costs. The biggest under spending was in the cost categories ‘Salaries’ (73%) and ‘Staff benefits and allowances’ (46%) but mainly were because of savings through lower salaries, the late start of SILC and cost sharing with other programs. The overall budget was spent at 76.7%, but CRS spent 89.1% of its subgrant for the silo production and distribution. The capacity to employ more people and challenges with local partners calls for the need for a more detailed analysis of whether for certain programs direct implementation is more effective and efficient. A number of inefficiencies were observed in M&E. There were changes in formats, unclear use of information provided, and unused funds for a central database that could have facilitated some M&E processes. • Design a standard monitoring system at the beginning of the program, including database • Use of e-vouchers that can be directly linked to databases • Identify focal persons for communication between organizations • Design feedback mechanisms/procedures on reports Impact A more competitive market in silo materials leading to lower prices. Blacksmiths turned in to business men. Reduction in seed losses and increased food self-sufficiency through high adoption of quality seed storage in the form of metal silos. • Final evaluation of ESS Phase II to sample respondents from beneficiaries since 2011 to 55 | Page ESS Phase II - Final Evaluation Farmers with the capacity to pay purchase silos at the full price, demonstrating the trust of farmers in the product. assess continuity of improvements and durability of silos. Women have a higher responsibility over seed production than at time of the baseline. It is hard to conclude whether this should be seen as empowerment or an additional burden created by the project. • Conduct FGDs to verify findings on decision making and responsibilities Inquiries on decision making show that women get more decision making power in the areas of investments in agriculture and buying food. The former could indicate that women are experiencing empowerment as a result of the program. Sustainability Continuing benefits after the program: • Silo production by blacksmiths • Increased food self-sufficiency • Increased seed availability Factors threatening sustainability: • Availability of silo materials at competitive prices • Continuing to wait and see attitude – non￾beneficiaries waiting for next round of vouchers instead of purchasing with own resources • Full price considered too high by farmers • Low adoption of introduced seed production techniques • Redesign BCC around 3 key messages:  Proper seed selection  Proper drying  Proper storage • Lower number beneficiary number and increase resource allocation for refresher training and follow-up visits to communities to reinforce adoption of techniques • Work towards free market principles allowing price competition between blacksmiths. 56 | Page ESS Phase II - Final Evaluation Annex A: Household survey template Effective Seed Storage – Evaluation Survey Instructions for surveyor “Hello, my name is ______________. I am conducting a household survey for the NGOs Mercy Corps and CRS for a project to improve the storage of seeds and helping farmers to increase food production. We are collecting information from households in order to assess what the current situation is. Your household has been randomly selected for our study. Would it be possible for me to interview the head of the household or a representative? This interview will take about 30-45 minutes.” Identification info Response Household ID |__|__| Name of Respondent Sex of Respondent (please circle one) Male ………1 Female ……2 District Sub-district Suko Aldeia Interviewer Supervisor Name Signature Date Day Month Year 57 | Page ESS Phase II - Final Evaluation SECTION 1: HOUSEHOLD INFO No Question Response and coding Skip to 101 What is the sex of the head of household? (please circle one) Male ……….1 Female ……2 102 What is the age of the head of household? ……. 103 Marital status of the head of household (please circle one) Married ………………….1 Divorced/separated ……2 Widowed ………………..3 Was never married …….4 104 Education of the head of household? (please circle one) Primary School ……..…1 Junior High school …….2 Senior High school …….3 Undergraduate degree....4 None ……………………..5 Other; please specify ____________________ 105 Total number of people in the HH 106 How many members are 14 years old or less? 107 How many members are from 15 up to 24 and years old? 108 How many members are between 24 and 60 years old? 109 How many members are 60 years old or more? 110 How many of the adults in the HH between ages of 14 and 60 are unable to work? (disability, long term illness, chronically ill, etc. 111 Do you support a family member/household member with a disability? 0=no 1=yes 58 | Page ESS Phase II - Final Evaluation SECTION 2: REASON TO PURCHASE SEED STORAGE UNIT No QUESTION ANSWER SKIP 201 Did you purchase a storage unit with a voucher Yes ………1 No ............2 Yes  202 No  301 202 When month did you buy the unit ? 203 What type unit did you purchase with a voucher?: (Write the total number for each of the units. Write 0 of the kind of unit was not purchased 35kg with small opening 75kg with small opening 75kg with big lid 75 kg with GPB >75kg............ specify....... Grain Pro Bag 204 Did you buy additional units without a voucher? Yes ………1 No ............2 Yes  205 No  301 205 What month did you buy the additional unit(s)? 206 What type of addition units did you purchase without a voucher? (Write the total number for each of the units. Write 0 of the kind of unit was not purchased) 35kg with small opening 75kg with small opening 75kg with big lid 75 kg with GPB >75kg............ specify....... Grain Pro Bag 207 Where did you buy the units without a voucher? 1 = directly from blacksmith 2 = From Market 3 = From Kiosk 4 = From other beneficiary 5 = Other: please specify: ___________________________ 208 What is the most important reason you bought the units? (only tick 1) A I need it to store seed B I need it to store something else: Specify C Because other farmers are buying it. So I also want one, but I do not know what proper use is. D I can use it to store water E Other reasons, please specify: __________________________ 59 | Page ESS Phase II - Final Evaluation SECTION 3: SEED VARIETY AND STORAGE No QUESTION ANSWER SKIP 301 Do you produce maize? Yes/Los ………1 No/lae ............2 If no finish interview 302 Combining all the parcels what is the estimated area you usually plant with maize you planted last season? 1 0 – 0.25 ha 2 0.25 – 0.5 ha/ 3 0.5 – 1 ha 4 Bigger than 1 ha 303 Do you plant the Sele and/or Noi Mutin variety promoted by MAF and Seeds of Life? Yes..……1 No...........2 No 310 304 What is the portion of land you plant with Sele and/or Noi Mutin variety and what part with other/traditional varieties? Give the farmer 20 beans and let them divide the 20 over Traditional and Sele and/or Noi Mutin variety . Write down both of them to the right. Total of the two should be 20! Sele/Noi Mutin Other/ Traditional 20 305 The last time you planted Sele and/or Noi Mutin seeds where did you get your seeds from? Multiple answers possible 1 Own seed from previous harvest 2 Bought on market 3 Bought from community seed bank/ group 4 Bought from relative/neighbour/friend 5 Given for free by relative/neighbour/friend 6 Given for free by the 7 Given for free by an NGO 8 Given for free by the Church 9 Other/specify: ……………………………………….. 306 Do you store the seed separate from food grain? Yes in separate containers………1 Yes, in same 75kg container …2 No……………………..............3 307 After harvesting, how many kg of seeds of Sele and/or Noi Mutin variety do you keep for the next season? ……………kg 60 | Page ESS Phase II - Final Evaluation 308 How do you store the Sele and/or Noi Mutin seeds? Multiple answers possible 1 Storage in sacks 2 Storage above the fire place/ 3 Hanging in trees 4 Storage in metal drum (200L) – at Household 5 Storage in metal drum (200L)– shared with several households 6 Storage in plastic container – at Household 7 Storage in plastic container – shared with several households 8 Storage in metal silo – at Household 9 Storage in metal silo – shared with several households 10 Storage in plastic bags 11 Air-tight metal container just for seeds 12 Storage in Grain-Pro bag 13 Storage in Grain-Pro bag in metal container 14 Storage in airtight plastic container (for example bottle) inside metal container 15 Other: Please specify: …………………………………………………. 309 Of the amount of seed you stored last time, what portion could you still use to plant and what part has gone bad by rotting, weevils, or any other pests and diseases? Give the farmer 20 beans and let them divide the 20 over good and bad. Write down both of them to the right. Total of the two should be 20! 1 Good seeds 2 Bad seeds 20 310 Do you store seed of Sele/Noi Mutin together with seed of local varieties? Ye ………1 No.........2 311 Which of the local varieties are you growing? Tick the appropriate boxes. 1 Batar lais 2 Batar bo’ot 3 Suwan 5 4 Arjuna 5 Kalinga 6 Other, specify 61 | Page ESS Phase II - Final Evaluation 312 The last time you planted maize, where did you get the seeds for the varieties in Q311 from? Multiple answers possible 1 Own seed from previous harvest 2 Bought on marke 3 Bought from community seed bank/ group 4 Bought from relative/neighbour/friend 5 Given for free by relative/neighbour/friend 6 Given for free by the Government 7 Given for free by an NGO 8 Given for free by the Church 9 Other ……………………………………….. ( 313 Do you store the seed separate from food grain? Yes ………1 No............2 314 After harvesting, how many kg of seeds of these varieties do you keep for the next season? ……………kg 316 How do you store these seeds? Multiple answers possible 1 Storage in sacks 2 Storage above the fire place 3 Hanging in trees 4 Storage in metal drum (200L)– at Household 5 Storage in metal drum (200L)– shared with several households 6 Storage in plastic container – at Household 7 Storage in plastic container – shared with several households 8 Storage in metal silo – at Household 9 Storage in metal silo – shared with several households 10 Storage in plastic bags 11 Air-tight metal container just for seeds 12 Storage in Grain-Pro bag 13 Storage in Grain-Pro bag in metal container 14 Storage in airtight plastic container (for example bottle) inside metal container 15 Other: Please specify: …………………………………………………. 62 | Page ESS Phase II - Final Evaluation 317 Of the amount of seed you stored last time, what portion could you still use to plant and what part has gone bad by rotting, weevils, or any other pests and diseases Give the farmer 20 beans and let them divide the 20 over good and bad. Write down both of them to the right. Total of the two should be 20! 1 Good seeds 2 Bad seeds 20 318 Have you used improved storage for any other crop seeds? (Note: NOT food. This will be asked later) (Rice) Hare Beans Other: Other: 0 = no 1 = yes 319 Of the amount of seed you stored last time, what portion could you still use to plant and what part has gone bad by rotting, weevils, or any other pests and diseases Give the farmer 20 beans and let them divide the 20 over good and bad. Write down both of them to the right. Total of the two should be 20! 1 Good seeds 2 Bad seeds 20 20 20 20 Maize Rice Beans Other: _____ Other: ____ 320 Is the amount of good seeds now different than before using the unit? YES it is more…………....1 NO, it is the same …….2 NO it is lower ……………3 I have not used the unit yet for this crop 4 If all No, Skip to 322 321 What kind of container do you use to store these seeds? 1 = 35 kg unit 2 = 75 kg unit with small opening 3 = 75 kg with GPB 4 = 75 kg with plastic bottles 5 = Other metal airtight container 6 = Grain Pro Bag only 7 = GBP in a metal container I already had 63 | Page ESS Phase II - Final Evaluation 322 Have you used your improved storage for any other grains or crops for food? (Rice) Hare Beans Other: Other: Yes = 1 No = 2 323 Only for the crops they are putting in the seed storage: Of the amount of seed you stored last time, what portion could you still use to plant and what part has gone bad by rotting, weevils, or any other pests and diseases Give the farmer 20 beans and let them divide the 20 over good and bad. Write down both of them to the right. Total of the two should be 20! 1 Good seeds 2 Bad seeds 20 20 20 20 Maize Hare Beans Other: _____ Other: ____ 324 Is the amount of good food now different than before using the unit? YES it is more…………....1 NO, it is the same …….2 NO it is lower ……………3 I have not used the unit yet for this crop 4 If all No, Skip to 402 325 What kind of container do you use to store this crop? 1 = 35 kg unit 2 = 75 kg unit with small opening 3 = 75 kg with GPB 4 = 75 kg with plastic bottles 5 = Other metal airtight container 6 = Grain Pro Bag only 7 – GBP in a metal container I already had 64 | Page ESS Phase II - Final Evaluation SECTION 4: FOOD SELF-SUFFICIENCY 401 During the last 12 months, which months did the food for your family come from your own production? WORKING BACKWARD FROM THE CURRENT MONTH, TICK THE BOX IF THE RESPONDENT IDENTIFIES THAT MONTH AS ONE IN WHICH THE HOUSEHOLD HAD ENOUGH FOOD TO MEET THEIR NEEDS. . Dec Nov Oct Sept Aug July June May April March Feb Jan 402 Enumerator: Please count the number of months indicated in 401 Favor sura fulan ne’ebe hatudu iha numeru 401 Number/Numeru: …………… 403 Is this the same or different than the 12 months before you joined the project? No, now we have a shortage in more months…. …………………1  Go to 404 No, we have sufficient food in more months now … ……………2  Go to 405 The same ……………..…. 3  Go to 407 404 If only looking at own food production, how many additional months is there now a food shortage compared to the 12 months before the project? Number/Numeru:  Skip to 407 405 If only looking at own food production, how many additional months is there now sufficient food compared to the 12 months before the project? Number/Numeru: 406 Why do you have more food from own production now?? (Tick responses; more than one answer possible) A Better production practices leading to higher production 65 | Page ESS Phase II - Final Evaluation B Higher food production due to better seed selection practices C Higher food production due to better seeds D Higher production due to better weather condition E Bigger area planted due to more seeds available with storage unit F Better drying of harvest G Better storage of harvest H Other (please describe response): 407 In the past 12 months, were there any months during which your household did not have food to meet your family needs? (This includes from buying!) YES…… 1 NO…… 2 No/  Skip to 501 408 If the answer to 407 is YES, which months did your household not have food to meet your family needs? WORKING BACKWARD FROM THE CURRENT MONTH, TICK THE BOX IF THE RESPONDENT IDENTIFIES THAT MONTH AS ONE IN WHICH THE HOUSEHOLD DID NOT HAVE ENOUGH FOOD TO MEET THEIR NEEDS. Dec Nov Oct Sept Aug July June May April March Feb Jan 409 Enumerator: Please count the number of months indicated in 408 Number: …………… 66 | Page ESS Phase II - Final Evaluation SECTION 5: FEEDBACK ON STORAGE UNIT 501 Are you considering buying more storage units? YES …………....1 NO ……………..2 Yes  503 No  502 502 Why not buying more units? (tick answers; more than one response possible) A One is enough for my seeds B I don’t produce enough to need more C I like for storing food but these are too small D They are too expensive E Other (please describe response): __________________________________ 503 Do you have any feedback on the storage unit you purchased? If yes, please specify No……………………..1 yes …………………..2; Please describe: __________________ 504 When you bought the storage unit, did you receive any information on how to use this unit?? Yes ………………………..1 No …………………………2 Yes  505 No  601 505 How was the information provided?? Brochure/leaflet/sticker………1 Demonstration …………..……..2 Verbal …………………………… 3 Mobile phone video …………..4 2 or more of the above ……..5 Other (please specify) ……...6 _________________ 506 Did you learn more about the unit and seed storage through any of the following: Yes = 1 No = 2 Radio Television NGO extension worker PPL/MAF xtension worker Neighours Local leader 67 | Page ESS Phase II - Final Evaluation SECTION 6: TRAINING 601 Did you receive any training on post harvest handling techniques? Yes ……..1 No ……2 If No, skip to 701 602 Who provided the training? NGO extension worker ……..1 PPL extension worker ……….2 Both…… ……………………………3 603 Which techniques are you applying? (Tick where responding positively) A Better planting distances and seeding B Improved maize seeds (Sele variety) C Timing of harvesting D Better seed selection (from center of land and cobs) E Better drying of harvest F Other (please describe response): 604 Are there any other trainings on production or post harvest handling you would like to receive? Lae ……………………..1 Los …………………..2; Please describe: SECTION 7: SEED AND RESPONSIBILITIES Ask Question: Who in the household responsible for…… [PHRASE BELOW] 1 = main male or husband 2= main female or wife 3=husband and wife jointly 4= someone else in the household 5 = Male, female and others 701 Preparing the land for planting 702 Planting seeds 703 Selecting the seeds for next planting season 704 Drying seeds 705 Storing seeds – Looking after stored seeds 68 | Page ESS Phase II - Final Evaluation SECTON 8: DECISION MAKING WITHIN THE HOUSEHOLD Question Who in your household has the final say on the following: ……. [GIVER PHRASE BELOW] 1 = main male or husband 2= main female or wife 3=husband and wife jointly 4= someone else in the household 5 = Male, female and others 801 Selling of large assets (eg livestock, land, coffee) 802 Buying of large assets (eg livestock, land) 803 Investments in agriculture (eg seed, tools) 804 What to plant, when and where (food crops) 805 What to plant, when and where (cash crops) 806 Attending meetings or activities at the community level 807 Buying food for daily consumption 808 Deciding what to do with family income End of Survey Thank the respondent for their time. 69 | Page ESS Phase II - Final Evaluation Annex B: Change in used storage methods from Baseline to Endline Percentage of Households using storage methods Storage method Improved varieties (N=133) Local varieties (N=401) Baseline Endline Baseline Endline Storage in sacks 7.0 11.3 7.8 7.9 Storage above the fire place 33.3 13.5 45.0 17.9 Hanging in trees 11.8 3.0 8.0 3.8 Storage in metal drum (200L) at household 18.0 12.8 21.5 8.2 Storage in metal drum (200L)– shared with several households 0.8 0.8 1.3 0.8 Storage in plastic container – at Household 31.3 15.8 30.8 0.8 Storage in plastic container – shared with several households 2.5 0.8 0.0 1.8 Storage in metal silo – at Household 5.5 58.6 1.8 74.5 Storage in metal silo – shared with several households 0.0 0.0 0.0 0.5 Storage in plastic bags 11.8 3.8 1.3 1.5 Air-tight metal container just for seeds 0.0 1.5 0.5 1.3 Storage in Grain-Pro Bag 1.0 0.0% 0.3 0.5 Storage in Grain-Pro bag in metal container 1.0 9.0% 0.8 6.1 Airtight plastic container in metal container NA 8.3 NA 8.4