MIDTERM EVALUATION OF THE MALAWI IMPROVED SEED SYSTEMS AND TECHNOLOGIES (MISST) PROJECT July 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared by Dr. David Coombs, Dr. John Haydu, Dr. Geoffrey Kananji, Mr. Mayamiko Kakwera, Mr. Jim Phillips, and Miss Merrill Anne Jordan. MIDTERM EVALUATION: OF THE MALAWI IMPROVED SEED SYSTEMS AND TECHNOLOGIES (MISST) PROJECT Originally Submitted July 2, 2018 Revised Version Submitted August 9, 2018 AID-OAA-I-15-00022/AID-612-TO-17-00001 International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive, Suite 400 Vienna, VA 22182 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover Photo – Women explaining OFSP recipes for children in Mangochi Photo Credit: Dr. John Haydu TABLE OF CONTENTS TABLE OF CONTENTS ACRONYMS LIST Executive Summary 1 Project Background 6 Evaluation Purpose & Questions 8 Evaluation Design, Methods, & Limitations 9 Findings 13 Evaluation Findings for All Crops 13 Summary of Findings for All Components 36 Conclusions & Recommendations 37 ANNEXES Annex 1. Evaluation Statement of Work 45 Annex II. Data Collection Tools 56 Annex III. Theory of Change (TOC) Analysis 69 Annex IV. MISST Gender Context 72 Annex V. Conflict of Interest Forms 73 Annex VI. Statement of Differences 77 ACRONYMS LIST ADMARC The Agricultural Development and Marketing Corporation AEDO Agricultural Extension Development Officer Ag Div Agricultural Diversification Activity AISL Agriculture Input Supplies Limited ASSMAG Association of Smallholder Seed Multiplication Action Group CAC Consortium Advisory Committee CB Commercialization Branch CGIAR Consultative Group for International Agricultural Research CIP International Potato Center CIMMYT International Maize and Wheat Improvement Center CMO Consortium Management Office COMESA Common Market for Eastern and Southern Africa CVM Commercial Vine Multipliers DADO District Agriculture Development Officer DAOs District Agricultural Offices DAES Department of Agricultural Extension Services DARS Department of Agricultural Research Services DCA Development Credit Authority DT Drought Tolerant DTM Drought Tolerant Maize DVM Decentralized Vine Multipliers EPA Extension Planning Area EQs Evaluation Questions FAW Fall Army Worm FFP Food For Peace FGDs Focus Group Discussion FISP Farm Input Subsidy Program FTF Feed the Future FUM Farmers Union of Malawi FY Fiscal Year GAP Good Agricultural Practice (In this report it specifically includes all new technologies introduced by MISST) GDP Gross Domestic Product GOM Government of Malawi Ha Hectare HH Household IBTCI International Business & Technical Consultants, Inc. ICRISAT International Crops Research Institute for the Semi-Arid Tropics IITA International Institute of Tropical Agriculture Kgs Kilograms KIIs Key Informant Interviews KULIMA Kutukula Ulimi m'Malawi Project M&E Monitoring and Evaluation MAPAC Malawi Project for Aflatoxin Control MISST Malawi Improved Seed Systems and Technologies (also FTF-MISST in some documents) MK Malawian Kwacha MOH Ministry of Health MOT Ministry of Trade MOU Memorandum of Understanding MT Metric Tons MUSECO Multi-Seed Company NASFAM National Smallholder Farmers’ Association of Malawi NGOs Non-Governmental Organizations NJIRA Pathways to Sustainable Food Security NSC National Seed Commission OFSP Orange Fleshed Sweet Potatoes OPV Open Pollinated Varieties PACA Program for Aflatoxin Control in Africa PCI Project Concern International PERFORM Protecting Ecosystems and Restoring Forests in Malawi PITT Performance Indicator Tracking Table PSI Para-Seed Inspector PSSM Private Sector Seed Multipliers QPM Quality Protein Maize SADC Southern African Development Community SANE Strengthening Agricultural and Nutrition Extension Services SOW Statement of Work SQs Sub-Question SSTP Scaling Seeds and other Technologies Partnership STAM Seed Traders Association of Malawi SSU Seed Services Unit TOC Theory of Change UBALE United in Building and Advancing Life Expectations USAID United States Agency for International Development USD United States Dollar ZOI Zone of Influence 1 EXECUTIVE SUMMARY The Feed the Future-Malawi Improved Seed Systems and Technologies (MISST) project is a 4.5-year activity implemented by a Consortium of four International agricultural research institutes. MISST has four strategic objectives: 1. Improved functionality of input and output markets of selected value chains: groundnuts, pigeon pea, soybean, drought tolerant maize (DTM) and orange-fleshed sweet potato (OFSP); 2. Capacity of stakeholders in agricultural sector strengthened to deploy and promote new crop varieties and their allied technologies; 3. Adoption and uptake of improved technologies enhanced in the FTF (Feed the Future) of influence (ZOI); and 4. Increased resilience of vulnerable households and rural communities in the FTF ZOI. The MISST project is being implemented in Malawi’s FTF Primary Zone of Influence, which covers seven districts in Malawi. By the end of the 4.5 years, it is expected that 245,245 households will have directly benefited from the MISST project. The MISST mid-term evaluation took place from May-June 2018 and was guided by the below evaluation questions: 1. To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? 2. To what extent has MISST improved functionality of input and output markets of its focus value chains? 3. What challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. 4. To what extent will the outcome of MISST interventions likely result in improved seed production, quality and productivity? 5. How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? FINDINGS Based on the evaluation findings, the MISST project has made the following progress against key indicators for each of its strategic objectives:  In order to improve functionality of input chains of selected value chains for seed (objective 1) MISST has facilitated the production of 17,500 MT of cereal and legume seed and over 200,000 Orange Fleshed Sweet Potato bundles in total, and has during the last two years provided development assistance to 1,512 and 1,990 organizations, respectively (averaging 111% of target).  In relation to capacity building (objective 2), the number of stakeholders directly benefitting from MISST interventions has exceeded the target in each of the last three years (120% of target). The number of beneficiaries trained was 23% above target.  Regarding adoption and uptake (objective 3), new crops, varieties and technologies have been tried by 140,000 farmers and others (on target) and the number of organizations applying improved organization-level technologies or management practices is 1,741, or 133% of target over the last 2 years. 2  Improved resilience (objective 4) is measured by MISST as the use of drought tolerant varieties, early maturing varieties of soybean, OFSP, dryland cereals and conservation agriculture. These have been central to the selection of crops and varieties, and the work with farmers. During key informant interviews (KIIs) with stakeholders and beneficiaries the evaluation team made the following key observations:  Improved new crop varieties and technologies have been made available to seed companies, agro dealers and farmers especially in the primary and secondary ZOI with potential spill over to non￾USAID ZOI districts in Malawi.  Capacity of seed companies, agro dealers and farmers for seed production and marketing has been improved laying a foundation for growth of the seed sector.  Level of awareness of new crops, their varieties and allied technologies by farmers, agro dealers, seed companies and government has been significantly improved. This will underpin efforts to increase productivity in the country.  New varieties and technologies have been used by farmers and are likely to be continued provided that affordable, good quality seed is available.  The resilience of vulnerable communities has increased through the availability of drought tolerant crops, improved production technologies and ability to earn cash through the sale of surplus agricultural crops.  The development of output-markets has not yet been adequately addressed and commodity prices are negatively impacted by export restrictions on maize, soybean and pigeon pea.  A Malawi Aflasafe product based on local strains has been successfully developed and tested and has shown to be effective in the field. Registration dossiers are under preparation for submission to the government for product registration and commercialization.  The refurbishment of aflatoxin laboratory and supply of required equipment for aflatoxin research has been completed. This will strengthen local capacity to develop, test and deploy mitigation technologies  Project effectiveness has been limited by irregular funding, complex administration within the consortium as impacted by internal processes and other challenges, incidence of resource constraints of government partners, unusually dry growing seasons and fall armyworm infestations which impacted cereals (maize and sorghum), with maize being most affected. CONCLUSIONS & RECOMMENDATIONS Evaluation Question 1. To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? Conclusions  MISST was well designed to address the key issues of seed shortages of improved varieties for all key crops and also addressed the lack of technical knowledge and awareness of new varieties and technologies for seed companies, agro-dealers, community groups and farmers. Maintenance breeding was not included in the program and that remains a weakness in the input value chain.  The capacity of seed companies, agro dealers, farmers and government staff has increased following the training on production of new varieties (early seed generations and certified seed) and their allied technologies, as well as business development, seed production and inspection.  The project has addressed resilience through the introduction of drought tolerant maize, sorghum and millet varieties, drought tolerant and disease-resistant legume varieties, OFSP varieties, and 3 through the introduction of new technologies including conservation agriculture to enable crops to perform well in low rainfall conditions. Introduction of a range of new improved varieties has enhanced diversity and given farmers a wider choice of crops to grow other than maize.  Regarding adoption; demonstration plots and field days increased farmer awareness regarding the value of new varieties, their allied technologies and increased demand for the best performing varieties. Farmers are using new varieties and their allied technologies, and they will continue to do so, subject to the availability of seed and sufficient funds to purchase seeds when needed. Recommendations o Follow-up Project. A follow-on project would be beneficial to consolidate the gains made in strengthening seed companies and community producer groups, and to provide further outreach to reach more beneficiaries in the ZOI and beyond. A follow-up project should incorporate successful features of MISST, such as the sub-grant program, mega-demonstrations with their associated field days, including the investment and implementation of demonstration activities by local seed companies and the establishment of strategic partnerships for scaling. Evaluation Question 2. To what extent has MISST improved the functionality of input and output markets of its focus value chains? Conclusions  MISST activities have led to an increased production of quality seed/planting material of all the targeted value chains, enhanced the capacity of partner seed companies and strengthened market linkages.  The functionality of legume seed supply has increased substantially.  Output markets have not yet been adequately addressed. Recommendations o Agrodealer Capacity. Future projects should continue to increase capacity of agrodealers by fully equipping them with necessary knowledge and skills to transform their outlets into a one-stop shop. o Study on future of Revolving Seed Fund. The current revolving fund system should be reviewed to determine how best to strategically support growth of the legume and other under-invested crops. o Study on Output Markets. A thorough market assessment is needed to identify potential opportunities (and constraints) for specific agriculture commodities, especially those being promoted by MISST. o Seed Supply Catalogue. To further strengthen the input markets, any follow-on project should develop and publish a catalogue of seed companies and agrodealers. It could be developed through the Seed Traders Association of Malawi (STAM). Evaluation Question 3. What challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. Conclusions  The consortium arrangement was introduced by the United States Agency for International Development (USAID) after the start of the activity. This resulted in difficulties for the consortium members in rearranging their contractual and reporting requirements.  Changes in project scope were introduced by USAID and these were not closely planned with MISST.  There was weak communication between USAID and MISST and irregular formal consortium coordination meetings. 4  Irregular funding distributions and a reduction in the project budget have resulted in planning difficulties and reduced targets. Activities in the 2018 crop season were substantially reduced. Recommendations  Use of Lessons Learned. Lessons learned by USAID and the Consultative Group for International Agricultural Research (CGIAR) centers must be factored into the current and future programming to avoid the difficulties encountered during MISST. Some activities should be initiated before the end of MISST, and others activities will need to be included in a follow-on project. In particular the following issues should be considered: o The Structure and Management of a consortium o An active Consortium Advisory Committee o The quality of Project management by USAID. Evaluation Question 4. To what extent will the outcome of MISST interventions likely result in improved seed production, quality and productivity? Conclusions  While it is still too early to assess the level of adoption by the beneficiary group, it was clear that farmers wanted to adopt improved varieties and practices provided that inputs would be available, affordable and that the new practices would not have other serious negative consequences.  Adoption of OFSP has been rapid and strong and is expected to continue after MISST ends.  4,817 farm families bought seed of DTM varieties either using vouchers or cash for planting in 2017/18.  MISST farmers increased the area planted with improved soybean seeds by 60% in two years.  New sorghum and millet varieties were liked by farmers in the southern districts and due to the poor performance of maize in the recent dry seasons, it is expected that the new crops will be adopted; households are accepting the dietary change as a trade-off for increased food security.  Acceptance and planting of improved groundnut and pigeon pea were good but less than for other crops. Recommendations o Maintenance Breeding Program. Establish a maintenance-breeding program for production of early generation seed. o Strengthen Para-Seed Inspector (PSI) System. Continue strengthening of the PSI system following the recommendations made in the recent PSI impact assessment. o New Seed Varieties. Recommend the multiplication and commercialization of varieties that are not yet commercially available to farmers, especially several new groundnut and three OFSP varieties. o Support for Department of Agricultural Research Services (DARS). Provide support on policy and implementation related to seed certification, variety registration, and trade harmonization under Southern African Development Community (SADC) and Common Market for Eastern and Southern Africa (COMESA) initiatives. These initiatives should complement and not duplicate the interventions of other donors. o Support for Department of Agricultural Extension Services (DAES). At project outset, recommend that all program activities within target districts are clearly understood, agreed upon and coordinated with DAES activities. Future projects must ensure that government staff have the resources to visit, supervise and engaged in project activities. 5 Evaluation Question 5. How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? Conclusions  Cooperation with other partners appeared to be healthy and to benefit both parties. MISST partnership mechanisms include sub-grant agreements and Memorandums of Understanding (MOUs).  All partners reported that MISST worked well with them and that technical support from MISST was considered very valuable.  The level and quality of coordination with district government agencies varied between consortium partners, while some District Agriculture Development Officers (DADOs) are more proactive in partnerships than others.  MISST coordinated with five other USAID projects.1  The level of cooperation with each has varied substantially according to the level of mutual benefits received. Recommendations o Study to Identify Funding Sources for SMEs. A study would help to identify potential funding sources and key constraints that currently exist. The study should also propose realistic solutions within the existing policy framework and the current economic environment. Aflatoxin/Aflasafe Conclusions  IITA supported the development of Aflasafe using local strains. Field-trials have proved successful and government registration is expected. Commercial development of Aflasafe will need support financially, technically and at the policy level. MISST has created a functioning aflatoxin laboratory and has raised awareness of the problem of aflatoxin and the appropriate control methods. The laboratory needs to be fully incorporated and staffed by government with guarantee of continued funding.  IITA worked closely with all line Ministries to implement the Aflasafe project – including DAES to select participating communities and farmers for setting up trials; the Ministries of Health and Trade, plus DARS, will assist in the testing and identification of regulatory issues related to the testing and development of Aflasafe.  IITA organized a national technical working group to discuss product testing, development and government registration. The consortia also established a multi-stakeholder platform. Recommendations  Aflatoxin Lab Support. Should provide continued financial, human resource, and management support to the aflatoxin laboratory and work to achieve adequate and long-term funding through private or public sources.  Aflasafe Commercial Development. The commercial development of Aflasafe, possibly in conjunction with a business development project, must continue to be developed and pursued. 1 The United in Building and Advancing Life Expectations (UBALE) Project, Protecting Ecosystems and Restoring Forests in Malawi (PERFORM) Project, Pathways to Sustainable Food Security (Njira) Project, Agricultural Diversification (Ag Div) Activity, and the Strengthening Agricultural and Nutrition Extension Services (SANE) project. 6 PROJECT BACKGROUND Landlocked Malawi ranks among the world's least developed countries. The country’s economic performance has historically been constrained by policy inconsistency, macroeconomic instability, poor infrastructure, and high population growth among others that limit labor productivity. The economy is predominantly agricultural with about 83% of the population living in rural areas2. Thus, agriculture remains the backbone of the economy and vital for the livelihoods of most Malawians’ national and household food and nutrition security. Agriculture accounts for about one-third of Gross Domestic Product (GDP) and 80% of export revenues. Heavily dependent on rain-fed agriculture, climate change is increasingly affecting agricultural production and rural livelihoods by increased frequency of extreme weather events such as droughts and floods. The country experienced consecutive severe droughts in 2014/15 and 2015/16, and floods in 2016/17 (UNICEF, 2015)3. Increasing agricultural productivity and off-farm employment, including in agricultural value chains, are key for stabilizing production systems and livelihoods. High rates of rural poverty mean that climatic shocks often have devastating and long-lasting impacts on the livelihoods of smallholders. Crop production is concentrated on one main food crop (maize) and one main cash crop (tobacco). Maize is by far the dominant crop grown by almost every farmer in Malawi and accounts for almost 50% of the entire planted area (World Bank 2016). At the same time, the maize-centered approach to food security has contributed to a lack of dietary diversity at both household and national levels. Lack of crop diversity has made the agricultural sector in Malawi, particularly smallholder farmers, vulnerable to market and climate-induced shocks and inadvertently drives nutrition-related challenges such as child stunting and wasting. Hence, agricultural diversification away from reliance on maize and tobacco was a key objective of the Malawi Growth and Development Strategy II. There has been some progress in diversification into other crops, especially grain legumes and root crops. Production of grain legumes almost doubled, from 0.53 million MT in 2010/11 to 1.02 million MT in 2014/15. The commercialization of grain legume production was supported through strengthening of farmer organizations, increased market linkages with processors and exporters, private investments in these downstream segments, and increased trading levels on commodity exchanges and some use of warehouse receipts. Plant breeder’s rights are not yet available so breeding is limited. Consequently, it is carried out by government breeders and the few private breeders who exist have limited professionalism to manage trials and ensure the genetic purity of new varieties. Until recently, there have been few seed companies with most producing only maize seed. With the introduction of the Farm Input Subsidy Program, farmers receive subsidies for hybrid maize and legume seed and this has encouraged several new seed companies to enter the market. With the exception of international seed companies, most of the new seed companies have very basic processing equipment and their out-growers have limited basic knowledge and skills to manage a seed multiplication plot. Before MISST there were very limited quantities of breeder and basic seed available for multiplication, particularly of legume varieties. The Seed Services Unit has low capacity to carry out the inspection and certification of an increasing number of seed producers. Grain is commonly sold to farmers as seed by unscrupulous agro dealers.4 Gender disparities exist; women headed households, on average, own 1.2 hectares (ha) while their male headed counterparts, on average, own 1.4 ha. Yet women constitute 70% of the agricultural labor force and produce 80% of households’ food requirements (Integrated Household Survey (IHS), 2012). Food insecurity is rampant and nutritional indicators are suboptimal. The IHS (2012) estimates that 47% of the 2 World Bank (2016) https://data.worldbank.org/indicator/SP.RUR.TOTL.ZS 3 UNICEF (2015). Malawi Humanitarian Situation Report 4 Source for the paragraph: A scoping study on the status of the seed industry in Malawi- gaps, challenges and opportunities by M A R Phiri 7 children under five years of age are stunted, and more than one-third of the population consumes less than the required calories. Not surprisingly, Malawi was ranked as very food insecure (94 out of the 109 countries) on the Global Food Security Index (GFSI 2013). USAID’s Feed the Future MISST project was developed to advance US government plans to sustain economic growth by addressing shortages of improved seed of major staple and income crop and malnutrition challenges in Malawi. The MISST project is contributing to USAID’s FTF objectives of inclusive agricultural sector growth and improved nutritional status especially women and children. Specifically, the activity was designed to contribute to improved agricultural productivity, increased resilience of vulnerable communities and households and improved access to diverse and quality foods as well as strengthening the capacity of organizations, enterprises and farmers in Malawi through application of improved technologies and management practices. MISST is a 4.5-year (2014 – June 2019) activity, under the Feed the Future Initiative, implemented by a Consortium of four member institutions; CGIAR, the International Potato Center (CIP) promoting the OFSP value chain; the International Maize and Wheat Improvement Center (CIMMYT) promoting Drought tolerant and nutritious maize (DTM) value chain; the International Crops Research Institute for the Semi-Arid Tropics (ICRISAT) which is promoting groundnuts and pigeon pea value chains; and the International Institute for Tropical Agriculture (IITA) which is promoting improved soybean varieties and aflatoxin mitigation technologies. Additionally, ICRISAT hosts the Consortium secretariat on behalf of the four institutions. By project’s end in June 2019, MISST is expected to reach 242,245 households will have directly benefited from project activities. The activity is being implemented in Malawi’s FTF Primary ZOI, which covers seven districts that comprise Balaka, Dedza, Lilongwe rural, Machinga, Mchinji, Mangochi and Ntcheu. However, following a severe drought in the 2015/16 growing season, a program change was necessitated in which the activity extended its activities (except for OFSP) to a secondary zone of influence that included Blantyre rural, Chikwawa and Nsanje districts to cover sorghum, drought tolerant and nutritious maize, millet and soybean (only Blantyre rural). This expansion, which also increased the length of the activity to 4.5 years was to reach 48,000 additional households. MISST pools resources and expertise to strategically assist Malawi to unlock the potential of the targeted crops, to transform smallholder agriculture, and contribute to the attainment of the Country and USAID/Malawi’s goal of improving the quality of life of Malawians. The activity’s theory of change (TOC) was based on the premise that removing obstacles in the seed system value chain would lead to improved production, supply and distribution of good quality seed in the market of pigeon pea, maize, soybean, groundnut and OFSP. In addition, supporting seed producers with the right equipment and technical knowhow would increase the supply of quality seed available in the market, while subsequently expanding production capacity. The activity design anticipated that the additional seed supply would lead to a drop in the price of seed, making it more affordable to smallholder farmers. Additionally, enhancing extension and building the capacity of private- and non-state actors would increase information availability to the end users, create demand and consequently enhance uptake of yield-increasing technologies. In summary, the TOC proposed that if vulnerable rural farming households are targeted through the integrated delivery of technology packages across crop commodity types that improve functionality and competitiveness of value chains, then the rate of agricultural innovation (testing, adoption, scaled-use and market use) and the impact on the transformation of these households would be improved. The TOC is further discussed in Annex 4. The MISST project has four approved Strategic Objectives: 1. Improved functionality of input and output markets of selected value chains, including groundnuts, pigeon pea, soybean, drought tolerant (DT) maize and OFSP; 8 2. Capacity of stakeholders in the agricultural sector strengthened to deploy and promote new crop varieties and their allied technologies; 3. Adoption and uptake of improved technologies enhanced; and 4. Increased resilience of vulnerable households and rural communities within the FTF ZOI. EVALUATION PURPOSE & QUESTIONS Evaluation Purpose. Consistent with USAID’s 2011 Evaluation Policy and its 2016 revision, the primary goal of this mid-term evaluation is to provide evidence which will enable USAID/Malawi to determine the extent to which MISST is on track to achieve its desired results. This evaluation assessed the effectiveness of the MISST design, its implementation process, and will inform decision makers as to any types of changes or course corrections that may be required to help MISST reach its objective. Through this evaluation, USAID also seeks to inform strategic and programmatic options for future engagement to support seed production, dissemination, and agriculture growth in Malawi. The three major objectives of this evaluation are: 1. To review, analyze, and evaluate the effectiveness of the MISST project in achieving program objectives and contributing to USAID/Malawi’s efforts to improved agricultural productivity in Malawi; 2. To evaluate major constraints in achieving expected project results; and 3. To provide specific recommendations and lessons learned on strategies and approaches USAID/Malawi should pursue in the remaining period of MISST’s implementation and for future seed production and distribution activities. Evaluation Questions. The Statement of Work (SOW) developed by the USAID/Malawi Office of Sustainable Economic Growth (SEG), in partnership with the IBTCI Monitoring, Evaluation and Learning Support (MELS) team, defined five evaluation questions (EQs) that guided the design, fieldwork and analysis of the mid-term evaluation. Additionally, the IBTCI evaluation team developed additional sub-questions (SQs) to better address specific topics to ensure useful evidence-based findings and conclusions would be achieved. The following are the principal five evaluation questions that were addressed by IBTCI together with the sub-questions. Evaluation Question 1. To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? The response to this question should address the four anticipated results and identify MISST’s areas of weakness and strengths in addressing these results. ● Sub-Question 1: Was the activity design appropriate to achieve the overall objectives – and has this changed? ● Sub-Question 2: To what extent has MISST met its targets for achieving Objectives 1-4? ● Sub-Question 3: How have project activities contributed to meeting the overall objective and what have been the key obstacles and challenges? ● Sub-Question 4: Has the design and implementation been appropriate to improve resilience of vulnerable households and rural communities? Evaluation Question 2. To what extent has MISST improved the functionality of input and output markets of its focus value chains? ● Sub-Question 5: How and where has the activity improved the performance and functionality of input markets for seed and new technology? What specific interventions were used and how has this benefited smallholder farmers? 9 ● Sub-Question 6: Output markets for seed and new technology? What specific interventions were used and how has this benefited smallholder farmers? Evaluation Question 3. What challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. ● Sub-Question 7: What challenges has the activity faced with regard to the technical and financial situation of beneficiaries and partners and how has it responded to them? ● Sub-Question 8: What challenges has the activity faced with regard to the policy environment and the ability of government to participate in the activity and how has it responded to them? ● Sub-Question 9: What challenges has the activity faced with regard to the structure, management and resources of the consortium, and how has it responded to them? ● Sub-Question 10: How well have consortium members coordinated their activities and how effective has the consortium approach been in achieving project goals and objectives? Evaluation Question 4. To what extent will the outcomes of MISST interventions likely result in improved seed production, quality and productivity? ● Sub-Question 11: Have the direct beneficiaries accepted and adopted the improved seed products and technologies? If there has been resistance, why, and how has the activity addressed this? ● Sub-Question 12: How sustainable will the improved seed provision be in terms of new varieties and good quality, affordable seed and improved technology? ● Sub-Question 13: What impact is the activity likely to have on grain production in the ZOI through introduction of new varieties and technology? ● Sub-Question 14: What is likely sustainability of the aflatoxin laboratory and the longer-term uptake of Aflasafe? ● Sub-Question 15: What long term effect will the activity have on the resilience of vulnerable households and rural communities in the ZOI? Evaluation Question 5. How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? ● Sub-Question 16: How successful were the activity partnerships with implementing organizations including private sector seed companies, input supply firms, government extension offices and other NGOs? ● Sub-Question 17: How successfully did the activity coordinate and cooperate with other non-partner organizations, such as projects, government programs and seed companies? EVALUATION DESIGN, METHODS, & LIMITATIONS Evaluation Design. The evaluation team utilized several complementary and inter-related forms of gathering information and data. The team concentrated on collecting data and user feedback through individual or group interviews utilizing semi-structured interview protocols. This approach allowed greater flexibility from variations in the questions and the topic being emphasized. This more flexible approach was helpful in addressing time and logistical constraints that were likely to surface and could require adjustments to samples and locations as the field activities progress. A basic work plan was developed that included a detailed evaluation methodology, which comprised an evaluation design matrix that explicitly linked evaluation questions to particular data collection approaches and data sources (Annex 2-B). This helped the evaluation team to carefully consider and systematically 10 organize all of the linkages between questions and methods. The main types of inquiries to acquire information came from review of project documents, key informant interviews, expert opinions, and focus group discussions. As mentioned above, USAID provided five principal evaluation questions to guide the investigative process, and the evaluation team developed a series of sub-questions under each principal question to glean out important specific information within those broader inquiries. How the team approached the questions depended on the interview approach being used and the particular respondent being interviewed -- a government individual, a “partner organization”, a consortium implementer, a seed￾company partner and beneficiary, or a smallholder farmer beneficiary. The combination of each (respondent type vs. interview method) required a different approach. For instance, key informant interviews (KIIs) were direct one-on-one sessions with fairly predefined questions, whereas focus group discussions (FGDs) with multiple people demanded a more flexible and probing approach recognizing differences in individual personalities. The MISST evaluation incorporated a four-tier evaluation design (see Table 1 below) to ensure that sufficient qualitative and quantitative data was collected and analyzed to give a full understanding of the activity. The evaluation team initiated the work by reviewing the Malawian agricultural sector and seed sector reports and documents to grasp the context on which the MISST activity was implemented. The review was extended to MISST project documents such as annual reports, work plans, Monitoring and Evaluation (M&E) data - notably the Performance Indicator Tracking Table (PITT) and baseline studies. The full list of documents reviewed, interviews held, and focus group discussions is provided in Annex 2. The information from the document review provided the basis for the in brief meeting and refinement of the evaluation data collection tools. Table 1: MISST evaluation approach with respect to types and sources of information. Tier A Tier B Tier C Tier D Desk Review & Data Collection Preparation Primary Data Collection & Preliminary Analysis Prepare Draft Report Prepare Final Report  Document identification  Review of sector and project documents  In-brief with USAID and Kickoff with MISST  Identification of Key Informants (KIs)  Development and finalization of field schedules  Development and finalization of data collection tools  Data collection-Field/Site observation, KIIs and FGDs  Ongoing data integration and data analysis  Debrief to discuss preliminary findings and enable feedback with USAID  Debrief with project staff and partners  Final data analysis, drafting of conclusions and recommendations  Draft report production  Incorporation of feedback on draft report  Production of final report 11 Challenges & Opportunities in Implementation Tier A Tier B Tier C Tier D Desk Review & Data Collection Preparation Primary Data Collection & Preliminary Analysis Prepare Draft Report Prepare Final Report  The documents initially shared with the evaluation team by both USAID and the implementers contained inaccuracies and most of them were in draft form  Field visits were successfully conducted.  In selected cases the attendance of farmer beneficiaries was low  All in all, beneficiaries and project stakeholders made time for the evaluation team  The field schedule was very tight.  The Debrief with USAID and project team was a useful opportunity for validating findings of the evaluation. The in-brief meeting was held with the evaluation team, the activity team and the consultant in the second week of the work. In addition to the in-brief meeting the evaluation team benefitted from the kickoff meeting with project team (MISST Consortium members) where information was provided which informed the scheduling of the consultative meetings with KIIs and farmer beneficiaries through the FGDs. The fifteen days of data collection for the MISST evaluation kicked-off with interviews with project leaders before proceeding to project partners and collaborators in Lilongwe. The evaluation made use of expert opinion interviews (See Annex 2c and 2d) to evaluate the relevancy of MISST design and implementation. In that regard, the evaluation team interviewed experts in aflatoxin and seed-related work. KIIs were also conducted with seed companies who included both MISST sub grantees and non-grantees. Senior government officers whose departments worked with MISST were also interviewed. The officers falling under district agricultural extension services and researchers from department of agricultural research services made up another set of the KIIs. Interviews were also conducted with agro-dealers, commercial vine multipliers and commercial farm owners who worked with MISST activity. Importantly the evaluation team conducted focus group discussions and site visits with farmers, who were the ultimate beneficiaries of the activity. In total, 23 focus group discussions were conducted across all the districts in the MISST ZOI. The farmers were usually gathered along the value chain on which they participated in the MISST project. MISST evaluation activity worked on Soybean, DTM, OFSP, aflatoxin, pigeon peas and dry land cereals value chains. Figure 1 presents the spread of interviews conducted by respective groups in MISST. In summary the MISST evaluation team used mixed method approach for data collection. The approach was used to address the preset evaluation questions of MISST provided by USAID/Malawi. 12 Figure 1. Spread of interviews by category of groups in MISST evaluation (n=72), June 2018 Sampling. Representation of the stakeholders in MISST and the seed sector was the prime goal in drawing the sample. The sampling frame was informed by the project value chains and the MISST ZOI. More importantly, both of the host-country national evaluators had extensive knowledge of most stakeholders within Malawi’s seed sector, including private sector seed companies, government agencies including DARS and DAES, many of the agro-dealers, other non-governmental organizations (NGOs), and locations of all target villages within the ZOI. So respondent selection within the ZOI was purposive with respect to Regions, Districts, Extension Planning Areas (EPAs) and Sections to ensure comprehensive coverage, but individual respondents often depended on who was available within the time frame and structure of the team’s interview schedule. Furthermore, when the team recognized a particular gap in the coverage, direct efforts were made to fill those gaps with additional interviews, so the approach was also quite fluid depending on the dynamics of the situation. The FGDs and site visits were spread across the respective districts and urban bias was avoided. With farmer beneficiaries the evaluation team purposively worked on ensuring representation of both men and women. Limitations. Methodological limitations and challenges for this evaluation included ensuring that samples of interview sources were sufficient to support evaluation findings. First, this suggests that the interview sources selected are appropriate and knowledgeable regarding the issues and activities to be addressed. Second, that a sufficient breadth of sources is interviewed to ensure the evaluation is complete and accurate. Third, the evaluation team needed to take systematic actions to counter any biases that might occur in (a) reporting by data collection sources; (b) interpretations of collected data by the evaluation team; and (c) ensuring “actual” results could be measured, which is only possible if data can be gathered and analyzed beyond respondent perceptions. Key biases the team tried to avoid included:  Recall Bias. The evaluation team anticipated that it would encounter some recall bias, given the two￾year plus period elapse and some instances of staff turnover in public sector counterparts.  Response Bias. Some Key Informants may try to deliberately avoid direct criticism or controversy regarding project implementation. 13  Selection Bias. Selection bias is an inherent risk when implementers help to facilitate contact with project beneficiaries, as they may select the most active, responsive, or engaged beneficiaries— meaning that the evaluation team may only hear from key informants who report positive experiences. The evaluation team aimed to observe and speak with a sufficiently wide range of key informants to mitigate this risk  Baseline & PITT. The evaluation team had challenges with comparability of data reported in the baseline study, data in the PITT and data reported by individual consortium members. This was partly because targets have changed several times during the activity lifetime with changes in the implementation area and funding availability. Quantitative data on targets and results are not available for all years in the PITT. The tables presented in this report include all data available to the team. In addition to avoiding the various potential biases mentioned above, the team used triangulation methods by re-checking document sources, and/or re-interviewing respondents when it was necessary to obtain clarity on key issues. At times when re-interviews were required, these were conducted in person, or if not feasible because of time and distance, additional information was acquired through phone call interviews. FINDINGS The findings section is divided into two parts, the first part covers all crops, as most of the program objectives, approaches and issues are similar across crop-types. The exception is for the discussion on Aflatoxin contamination and its biological control agent “Aflasafe”, which are both unique in their objectives, approaches and implementation processes. Within the “all crops findings” section, all five questions are addressed in their entirety. The exact same approach is then applied to Aflatoxin and Aflasafe for groundnuts and maize, the two crops affected by the aflatoxin fungus. Following the findings discussions, separate sections on conclusions and recommendations are addressed, as each of these topics follow logically from findings, thus ensuring an integrated approach to the evaluation analysis. Again, similar to the findings section, conclusions and recommendations use the five-question framework to ensure comprehensive and integrated coverage of all major issues identified and addressed throughout the evaluation process. As mentioned in the Evaluation methods and limitations sections above, there are many cases where target and/or actual data is not available for all crops and all years. This may be due a lack of activity or possibly because the monitoring system did not capture the data. In any case, all the data that was made available to the team through the PITT and other documents is included in the tables below. Evaluation Findings for All Crops5 Evaluation Question 1. To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? Suitability of Design. The program design addressed most needs of the seed sector and included all the major food security crops in the ZOI. Although common beans would have been a useful addition to the program, they are not a designated FTF crop so were not included. 5 Findings for the four (4) crop-related components are considered jointly since the target crops cover many common issues however, the fifth component (Aflasafe), are discussed separately. 14 The program addressed the provision of seed and vegetative planting material for OFSP according to the current market models and opportunities. Seed production of maize is primarily carried out by the private sector, for both hybrid and open pollinated varieties. Legumes and dry land cereal seed production is both private sector and community-based due to the low perceived value of the seed business. In the case of OFSP, a new model (commercial and decentralized vine multiplication) was established as little existing vine production existed prior to MISST. Although MISST addressed the need for increased volumes of basic seed and trained technicians from seed companies in maintenance breeding, it did not fully address the shortage of capacity in Malawi for maintenance breeding and the production of initial stocks of high purity seed from the breeder. A four-year program is insufficient to develop and introduce new varieties but, as is normal with comparable programs, MISST took the existing seed material and used it for implementing activities with farmers, while multiplying early generation seed of the newest varieties for development after the end of the MISST program. This difficulty is severe for crops with a low seed multiplication rate, notably the legume crops, but is not significant for the dry land cereals. The CGIAR centers have been the main mechanisms for incubating small seed companies. ICRISAT, in particular has incubated over 11 seed companies through its Malawi Seed Alliance. Under MISST, buy￾back contracts were included to guarantee production and provide financial security to pre-basic and basic seed producers. However, the model’s disadvantage is that it does not foster the further growth in relationships between seed producers and commercial markets. The evaluation team observed that the important objective of developing output markets is not a key strength of the CGIAR staff working on MISST. If stakeholders had been engaged at the design stage, it may have been possible to include commercial partners and make a stronger effort to develop output markets for the produce of farmer’s increased production. An alternative would have been to leave output markets to another USAID intervention with a stronger focus on private-sector business development activities. Progress Against Targets. MISST has measured its performance against FTF and custom indicators. Progress against selected key indicators are presented in this section. The evaluation team used data from the MISST PITT report, annual beneficiary surveys and individual component PowerPoint presentations. The results include data to the end of 2017, which includes three out of a potential five main cropping seasons. Gender disaggregation is included in the tables where it is available. Objective 1: Functionality of Input and Output markets. Although this first MISST objective identified program activities for both input and output markets, the clear emphasis of the activity was on input markets only, focusing on seed and planting materials for the selected value chains, with little resources directed at output markets. First of all, it was quite clear from the interviews with consortia members that there was clearly no intention to work on markets for grain and tubers for consumption. Some sales-oriented activities within “input markets” were erroneously construed as an integral part of “output markets” by MISST implementers. Second, although clear targets were defined in terms of metric tons of basic and certified seed production and sale for input markets, there were no such targets established for the sale of grain in the output markets. CIMMYT has recently started engaging the school feeding programs stakeholders to consider orange maize which is rich in Vitamin A to be used for school meals and other nutritional specific intervention other consortia members did not actively pursue output market programs. In Malawi both formal and informal output markets exist, the latter being relatively bigger. The activity initially focused in increasing production via better access to seed. The Government of Malawi was able to sign a forward contract for the supply of 100,000 MT of legumes chiefly pigeon pea and this target was easily met with the help of improved seed but the contract was cancelled when India cut down their 15 import contracts in 2017. Output markets are especially prone to externalities but still require other actors to work with productivity enhancers such as MISST in the future. Project records show that MISST has facilitated the production of over 17,500 MT seed and 200,000 bundles of OFSP vines through its work with partners. Despite many difficulties related to weather, pests and diseases, MISST has reached 74% of its target for basic seed production (Table 2). The initial production targets were set to allow for 30% adoption of new crops and varieties in the areas where MISST carried out its interventions, however targets have been reduced as a consequence of reduced funding. The number of organizations receiving organizational development assistance is presented Table 3, below. From the data, it is clear that MISST has reached its targets for this indicator, with the exception of women’s group organizations, which fell substantially short in 2016. This may be due to a lack of organized women’s groups within the FTF zone of influence. In many cases the women are part of farmer groups who then form a second tier of grouping usually an association as in the case of National Association of Smallholder Farmers of Malawi (NASFAM). This may cause them to be under-represented in the statistics. Table 2. Seed production of target MISST crops – Cumulative Totals (MT), 2018 data MISST Crop All Classes Breeder Basic Certified Target Actual Achieved Target Actual Target Actual Target Actual Soybean 2,893 2,408 83% 12 11 380 300 2,500 2,095 DTM 10,600 7,409 70% 100 51 500 315 10,000 7,042 Groundnut 2,800 2,086 75% 60 43 2,000 1,716 740 327 Pigeon pea 800 708 89% 12 8 350 209 438 491 Sorghum 400 356 89% 5 4 85 83 310 270 Millets 23 20 87% 8 7 6 5 9 8 Total 17,516 12,987 74% 198 125 3,321 2,629 13,997 10,233 Table 3. Number of associations receiving organizational development assistance, 2o15-17 data MISST Organization Type 2015 2016 2017 Actual (no target) Target Actual Target Actual Total Number Organizations 71 818 995 743 756 For-profit Enterprises 38 100 119 94 251 Producer Organizations 18 400 823 644 502 Women’s Groups 15 318 53 3 3 Objective 2: Capacity of stakeholders in agricultural sector strengthened. The number of householders benefiting directly from MISST interventions is recorded in Table 4, below. As can be seen, MISST has exceeded the target numbers in all three years for which data is available. The number of individuals receiving short￾term agricultural productivity training is presented in Table 5. MISST has exceeded its targets for this training for all categories of people and include similar numbers of male and female participants. Table 4. Number of householders benefiting directly from project interventions, 2015-17 data MISST Households (HHs) 2015 2016 2017 Target Actual Target Actual Target Actual Households benefiting 800 16,144 79,895 80,544 67,350 82,059 16 Table 5. Private or government individuals receiving MISST short-term agricultural productivity training, 2015-17, MISST Interview Group 2015 2016 2017 Target Actual Target Actual Target Actual Individuals 800 17,084 40,634 35,864 28,700 40,276 By Category Producers 800 16,623 39,538 34,401 27,591 40,276 Government 0 117 278 1,017 771 2,261 Private sector firms 0 338 804 154 117 239 Civil Society 0 6 14 292 221 238 By Gender Male 300 13,177 16,254 n/a n/a n/a Female 500 3,907 24,380 n/a n/a n/a Objective 3: Adoption and uptake of improved technologies enhanced. Adoption and uptake from MISST activities have been measured through application of improved management methods by organizations, application of new technologies by farmers, and the effect of the MISST program on sales and gross margins of farm households. The “for-profit” and “producer associations” targets were exceeded. However, in the case of women’s groups, targets in 2016 appear to have been unreasonable compared to the actual numbers of women receiving assistance (Table 6). As suggested in the case of Table 3, this may be due to a lack of women’s groups or under-reporting. The continued sustainability of new varieties and technologies after project completion is a function of several inter-related factors which are discussed more fully in a later section. MISST has introduced six improved OFSP varieties which were not in the ZOI before MISST intervention and has also worked on supporting the multiplication of parental materials for DTM which were not available to seed companies. Farmers reported through FGDs that these and other new technologies have been fully demonstrated throughout the ZOI and have been demanded by smallholder farmers. In one FGD, farmers said that they now know and appreciate the different maize varieties which are drought tolerant. This has been made possible because of the demonstration plots mounted in this area by a seed company which was supported by CGIAR centers. The activity registered significant progress on the input market (seed and vines) through the agro dealer support, the establishment of the vine multipliers and support to seed companies. The improved varieties are now available in the marketplace and, provided that the Farm Input Subsidy Program (FISP)6 does not end resulting in a sudden collapse of the seed industry, the seed companies are likely to continue producing these new varieties demanded by farmers. Indeed, it was observed that some seed companies are operating outside the FISP program, so profitability is indeed possible for some companies even without FISP. Continued improved seed availability to farmers, from a seed producer standpoint, is likely to continue. This is discussed further in a later section. The number of organizations that applied improved organization-level technologies or management practices has exceeded the target in 2016 and 2017. The activity worked through membership-based organizations such as NASFAM, STAM and FUM to attain this indicator. Seed companies' accessing inbred lines is also included in this indicator. 6 Our engagement with stakeholders was suggestive of the uncertainty of the continuation of FISP. At the time of the evaluation, Government of Malawi (GOM) was owing seed companies money in excess of MK2 billion from 3 previous seasons. This proved to have caused serious cash flows in most seed companies. 17 Table 6. Number of organizations that applied improved organizational technologies or management practices, 2015-17, MISST data Interview Group 2015 2016 2017 Target Actual Target Actual Target Actual Community Organization 66 812 995 443 680 For-profit Enterprise 52 62 107 48 226 Producers Association 14 400 876 390 451 Women’s Groups 350 7 3 3 Trade & Business Assoc. 5 2 0 The number of farmers and others who have applied improved technologies or management practices shows the scale of MISST’s demonstration programs7. The four major crops promoted were clearly maize, groundnut, soya and OFSP. Each crop category had comparable targets and actual figures. Gender parity was achieved in 2016 and 2017 although in 2015 when no targets were set, the number of male participants was much higher than females. Table 7. Number of farmers and others who have applied improved MISST technologies or management practices, 2015-17, MISST data 2015 2016 2017 Actual (no target) Target Actual Target Actual Individuals 18,029 58,495 40,668 56,850 80,905 By Category: Groundnuts 9,775 13,634 13,352 Maize 10,026 13,984 20,071 Millet 400 231 Pigeon peas 1,042 1,453 6,947 Sorghum 626 600 2,910 Soy bean 387 12,000 16,019 13,565 15,000 OFSP 9,473 13,213 21,394 By Gender: Male 13,207 23,398 19,635 27,081 39,621 Female 4,361 35,085 21,033 28,769 41,284 The area of land under improved technologies or management practices shows that, although the numbers of participants met the targets (Table 7), the land area planted was significantly less when compared to the targets, particularly in the case of groundnuts (Table 8). This could be either that target beneficiaries either chose to plant less of their land area under improved crop varieties (due to lower sales of cash crops in the previous year) or the estimated land area per beneficiary was less than the actual area. Fall army worm may also have reduced production in 2017. Both the target and the actual areas represent less than 5% of the total crop land in the ZOI (including plantations and fallow land - Teluguntla et al., 2015). Detailed figures for land areas under annual food crops are not readily available but the original target – before budget cuts was to reach 30% of farmers in the ZOI. 7 The total population of the 7 ZOI districts was recorded as 4 million in the 2008census. The FAO estimates that there are 2 million smallholder farmers (10% in 2016 of the population – World Bank 2018). The number of farmers in ZOI might be 400,000. 18 Table 8. Area of land under improved crops, technologies or management practices, 2016-17, MISST data Crop Category 2016 2017 Target Actual Target Actual Area (Hectare (ha)) 46,334 20,137 32,745 26,563 By Category: Groundnuts 6,092 9,926 3,541 Maize 5,626 9,167 5,995 Millet 15 27 Pigeon peas 353 575 1,506 Sorghum 130 130 585 Soy bean 4,000 3,680 4,000 3987 OFSP 1,643 2,677 1,801 By Gender: Male 18,534 5,167 8,319 9,819 Female 27,800 7,870 12,752 6,154 Joint 7,100 11,528 10,590 The value of smallholder incremental sales and gross margins are presented for the two major legume crops in Table 9. Sales of groundnuts in 2016 were only 50% of the baseline estimate, so the incremental value is negative. Overall sales are substantially below targets for both crops. The sales of groundnuts in 2016 were only 50% of the baseline estimate the incremental value is negative. Overall sales are substantially below targets for both crops. Late seed delivery, poor rainfall and high incidence of rosette disease affected the crop performance and reduced yields. By implementing through membership-based organizations such as NASFAM, STAM and the Farmers Union of Malawi (FUM), the number of organizations that applied improved organization-level technologies or management practices has exceeded the target in 2016 and 2017. Table 9. Value of smallholder incremental sales and gross margins for groundnuts and soybeans, 2016-2017, MISST data Incremental Sales and Gross Margin Value (US Dollars) 2016 2017 Target Actual Target Actual Value (USD) 891 3,738 1,409 Groundnuts -460 1,900 357 Soy bean 1,3521 1,839 1,051 Gross margin (USD/ha) Groundnuts 215 176 215 351 Soy bean 250 357 275 315 Objective 4: Increased resilience of vulnerable households and rural communities. Resilience was achieved in the activity through the introduction of new crops, more climate-safe crops, more nutritious crops and by improving growing crop technologies. The introduction of drought tolerant and nutrient dense maize varieties (DTM), early maturing and disease resistant soybean varieties, OFSP as a means of diversifying crop production and providing a source of vitamin A. Maize and millet were also introduced in southern districts where maize production has become very unreliable due to lack of regular rains. New technology to increase crop productivity including conservation agriculture has also brought more sustainable crop production and therefore, resilience. 19 Resilience is also captured in the indicators by the home consumption of targeted crops by direct beneficiaries and by the adoption of resilience practices. The quantity of food set aside in 2016 and 2017 is shown in Table 10. The average quantity across all commodities is 58 Kilograms (kgs) per beneficiary producer. This represents 10% of the annual food consumption per individual.8 Note that in the case of groundnuts and soybeans, quantities set aside for home consumption are substantially less than for pigeon pea and OFSP. Groundnuts and soybeans are typically sold as cash crops where pigeon peas and OFSP are used more for human consumption. Table 11 shows that nearly 95,000 individuals in the MISST project adopted risk reduction practices during the two years that data has been collected and presented. These practices included crop diversification, cultivation of drought tolerant maize varieties and dry land cereals and cultivation of OFSP by farmers subject to periodic income and food insecurity. Gender parity was nearly achieved in 2017. Table 10. Quantity of food set aside for consumption and numbers of direct beneficiaries, 2016-17, MISST data Crop Category Quantity Food Set Aside for Home Consumption (MT) Number of Direct Beneficiary Producers 2016 2017 2016 2017 Groundnuts 613 541 19,346 13,352 Pigeon peas 36 204 2,074 6,947 Soybean 1,502 299 19,446 15,000 OFSP 1,097 2,314 18,845 21,394 Total 3,248 3,358 59,711 57,693 Table 11. Number of implementing risk-reducing practices and/ or actions to improve resilience to climate change, MISST data Category 2016 2017 Target Actual Actual (no target) Individuals 52,875 54,853 39,818 By Gender: Male 37,545 38,949 22,594 Female 15,330 15,904 17,224 Contribution of Activities to Overall Objective. The activities of MISST have been highly valued by all stakeholders in key informant interviews and FGDs. As shown in Table 2, MISST has enabled a substantial increase in the production of early generation seed (both Breeder and Basic) and has therefore reduced this bottleneck for smaller seed companies. The upgrading of the seed testing laboratory at DARS/SSU facilities should be a valuable contribution to the seed sector but is not yet functional due to a lack of government funds to complete the works. Sub-grants have also been used to help seed companies to address gaps and bottlenecks in their production and storage, including rehabilitation of 3 seed warehouses and provision of 3 sets of irrigation equipment. This financial support has led to an increase is seed production by MISST partners. During all key informant interviews and FGDs with government, commercial and farmer stakeholders it was noted that training, demonstration activities and field days have strongly enhanced the awareness of improved seed varieties, technologies & new technologies among farmers, agrodealers and DAES staff. Seed companies and government reported a strong increase in demand for seed of improved varieties 8 Based on 128kg annual food consumption (MALAWI STRATEGY SUPPORT PROGRAM, Policy Note 11 / March 2012) and a family size of 5 20 following involvement by farmers in the MISST crop demonstrations. While the inclusion of embedded MISST staff in District Agricultural Offices (DAOs) was seen by the DAES staff as very effective for coordination of activities and communication with the offices in Lilongwe. Key informant interviews and FGDs revealed that some aspects of the implementation caused problems for partners and reduced the success of field demonstrations. Late delivery of seed and late disbursement of funding to partners caused difficulties to the timeliness of activities. Planting time cannot be delayed without causing reduced yields, increased risk of drought damage and additional risk of pest and livestock damage to late maturing crops. This problem was noted during many partner focus group discussions as the major weakness of the program. In Fiscal Year (FY) 2017, the groundnut seed which ICRISAT provided to farmers and out growers particularly in Lilongwe and Mchinji had poor germination rates. This was noted in almost all the fields and interviews conducted in Lilongwe and Mchinji. A follow up with ICRISAT provided an explanation that the problem was due to defective seed treatment. Where possible the seed was replaced but this could not be done for all small-scale demonstrations. The source of the faulty dressing was traced back to a company in Kenya. Tighter quality controls for imported chemicals have been set up to ensure that the problem does not reoccur, Although some of the activities were designed to run for only a single year, others were intended to run for 2 or more years. Changes to the program as a result of changing USAID priorities and financial difficulties led to many activities that were planned to be multi-year being implemented for only a single year. Local government interviewees and farmers reported that this limited the effectiveness of the activity to ensure that farmers fully understood the value of the new crops and technologies and is likely to have an impact on their longer-term adoption. Contribution to Resilience. Resilience to beneficiaries has been improved through several MISST interventions: ● Farmers are more willing to diversify away from monocrop maize production. Farmers reported, through FGD results, a willingness to diversify their crops and diets particularly in disaster prone districts where MISST was being implemented. Seed availability of many other crops was the most limiting factor for their crops. Focus groups and government interviewees reported that crop diversification provides a richer diet to smallholder farm households and also reduces the risk of severe hunger as a result of failure of a single crop. ● Sorghum and millet have been established in the most drought-prone areas of Nsanje and Chikwawa. The farmers at FGDs reported acceptance of millet and sorghum as an option to the staple (maize). Traditionally these areas grew the two crops but the local varieties were prone to damage by livestock. The improved varieties, due to their early maturity characteristic, partly address the livestock-crop conflict. Communities have accepted that these crops give a much more reliable harvest of nutritious grain and have recognized that the crops are more appropriate to their agro-ecologies even though maize remains the staple food of choice. ● FGDs and key informant interviews suggest that the introduction of new DTM varieties and other crops with shorter growing seasons has enabled farmers to produce crops in drier seasons when traditional varieties are more likely to fail. ● Through direct observations, key informant interviews and FGDs held with participants in Chiwamba EPA (Lilongwe); Mwaiwa and Katuli EPAs in Mangochi and Ntaja EPAs in Machinga, it was revealed that participants had embraced with considerable enthusiasm the production of the OFSP. The evaluation team were shown different food products which the participants produce and consume from OFSP. Case studies were provided of how farmers have, within a short period of the activity implementation, earned significant income from OFSP. The introduction of OFSP has increased the 21 production and recognition of this important crop for food security with a high nutrition value, including additional income generating opportunities to farm households. ● Biofortified crop varieties (OFSP, Provitamin-A maize and Quality Protein Maize) provide an improved intake of protein & vitamins. FGDs with farmers in areas where demonstrations had been carried out revealed a clear awareness of this and a willingness to grow these crops even if there is a yield penalty when compared with the highest yielding non-fortified varieties. Seed companies have started commercial production of these varieties on the strength of the demands of producers. CIMMYT and another project reported that agencies that are implementing school feeding programs for the vulnerable are planning to use biofortified maize in order to maximize the nutritional benefit of the programs. The evaluation team was not able to verify this with the programs concerned. ● Project records show that the introduction of OFSP has increased the production and recognition of this important crop for food security with a high nutrition value, including additional income generating opportunities to farm households. ● Interviews suggest that biofortified improve intake of protein & vitamins, which will be valuable for communities and also for the vulnerable beneficiaries of school feeding programs. ● A key aim of the work with smallholder farmers is to enable the transition from subsistence farming to market led farming. Key informant interviews and FGDs revealed confirmed that cash brings with it the potential to store wealth and provide a safety net for future crises. According to government interviewees and IITA, soybean is increasingly becoming one of the most important cash crops in Malawi – driven by good market opportunities and the expansion of poultry and fish industries. FGDs with farmers involved in the program in the seven districts reported that the soybean program has been successful at enabling this transition as a second valuable source of cash, in addition to groundnuts. They also reported that they are using soybeans as a protein supplement to existing foods. Evaluation Question 2. To what extent has MISST improved the functionality of input and output markets of its focus value chains? Findings in this section were obtained from FGDs and key informant interviews. Functionality of Input Markets. MISST has improved the functionality of the input chain through the production of early generation seed and planting material, primarily through bulking up seed to ensure that seed companies, agrodealers, OFSP vine multipliers and community seed producers have good quality product to sell to farmers. MISST strengthened their partners through technical backstopping of seed producers and business development of seed producers, agro-dealers and vine producers, in particular the mobility support to SSU, and the PSI training improved the frequency of field inspections at seed farms. In addition to the training and technical assistance, some partners were assisted in the physical development of their facilities through sub-grants for renovation of warehousing and irrigation for maize inbred line production. CIP facilitated the creation, and training of commercial and decentralized vine multipliers so that the markets in the ZOI were fully supplied with vines for root production. The para-seed inspector training was widely praised by participants and by government. Para-seed inspectors were previously trained by the Seed Services Unit (SSU) but the system had lapsed due to lack of financial resources. With the increase in seed production stimulated by the FISP program there was a much greater need for seed inspection services than could be met by the SSU alone. The PSI program has trained 180 new PSIs and also retrained 30 SSU seed inspectors. The program was not limited to MISST partners and several other seed companies – including multi-nationals – were able to take advantage of the program. PSIs are an important resource for the seed industry, being able to inspect growing crops, provide advice to seed producers and to maintain a higher-level quality control within seed companies 22 than was available previously. Senior and junior government officers and seed companies all reported that this led to more timely inspection and improved out-grower production. A spin-off benefit of the program is an improvement in the linkage between SSU staff and the DAES. This results from the need to communicate inspection assessments from the PSIs to the SSU, and also from the new found certification skills that exist within the DAES, as some staff did receive PSI training. There still remains an issue of clarifying the role of PSIs on quality assurance of seed delivery, then end chains of seed value chains. This is critical as quality assurance for production has made progress through the piloted PSI initiative but more gains would be realized if seed delivery was similarly supported. Market development was stimulated through the demonstration program, particularly the mega￾demonstrations, which gave farmers the opportunity to assess varieties and to meet with the seed companies and agrodealers who are the potential suppliers of seed. It was reported by government, farmers and seed companies that the demand for seed of the improved varieties was greatly increased though the strong participation of farmers as well as input suppliers. Government and farmers both reported the ability of farmers to demand the “best varieties” from suppliers. Despite this rather sudden demand, seed supply did not always match demand. For instance, in one case drought tolerant, pro vitamin A (orange) maize seed was introduced for demonstrations in 2016/17 but, despite creating a strong demand, the seed is still not available through the local seed companies. It usually takes several years for performance trials, registration and market testing before a seed companies is ready to invest in a new product such as pro vitamin A maize. One seed company is now multiplying and marketing pro-vitamin A maize in Malawi. CIMMYT reported that this particular case was unusual, in that access to additional imported pre-basic seed was interrupted, thereby delaying further seed multiplication. It was noted in interviews with seed companies, DAOs and implementing partners that while demand for a new variety can develop rather suddenly, following a favorable field demonstration, an inevitable supply lag in satisfying demand will occur due to the time necessary for seed multiplication, certification, and eventual distribution to wholesale and retail outlets. For the legumes, MISST worked through national farmer organizations (Association of Smallholder Seed Multiplication Action Group (ASSMAG), NASFAM and FUM) in order to encourage the development of local seed production through clubs and associations. Consortium members were involved in the testing of seed quality to ensure that the seed satisfied certification standards. Despite some quality issues, supply in legume seed banks has increased substantially. Beneficiary tracking by CIMMYT also enabled the mapping of the agrodealer network and outlet shops to enable seed companies to identify where new agrodealerships are needed and the effectiveness of the existing dealers. Functionality of Output Markets. For most crops, output markets have not yet been adequately addressed by MISST, in part because output markets are difficult to develop until there is sufficient produce to sell. Seed production by companies and at the farmer multiplier level takes 2-3 years to enable the production of excess grain, beans and roots. Difficult growing conditions have also reduced production. Despite this, sorghum, soy and OFSP have all been available in sufficient quantity to necessitate market development. OFSP roots are mainly traded locally in informal fresh root markets. CIP organized promotional activities to create more awareness and demand for the roots including live theatre in 21 markets, videos, jingles and songs by a popular artist, trainings of partners and farmers in marketing and a radio promotion campaign with FTF Ag Div campaign. Farmers reported low prices for dryland cereals, pigeon pea and soybean; at present crops that do not have well established markets in the ZOI, particularly in remote areas. In fact, the main challenge for all FTF MISST value chains is the is the rather weak output market systems that mostly target external markets and are very prone to their associated externalities. DAO staff reported that markets are essential to improving smallholder farm income and low prices are a clear disincentive to increase productivity and adopt new crops and technology. 23 CIMMYT has begun work to develop markets for drought tolerant nutrient-dense maize, with the main emphasis being the large school feeding programs that currently exist within the country. It was also apparent from the stakeholder workshop in Lilongwe that IITA interprets the phrase “market outputs” in the activity proposal as the seed produced by partner organizations (e.g. seed companies and community producers) rather than the harvested crop from farmers once it leaves the farm gate. This confusion also stems from the amalgamation of separate proposals from the centers into a single consortium proposal with objectives that were lightly different from the originals. Evaluation Question 3: Challenges in Implementation & MISST Responses Findings in this section were obtained from FGDs and key informant interviews. Financial Issues. Key stakeholders in the seed industry all face financial challenges with the exception of the international seed companies who have greater resources and access to financing. The smaller seed companies have been key players in MISST but have found it difficult to obtain bank loans as they lack acceptable collateral and, if successful, must still face very high interest rates. Seed companies pay all the production costs during the growing season (November to March) but do not receive payment for the seed until the agro dealers sell the seed nearly 12 months later. In cases where the sales are dependent on the FISP program, then that repayment period may be extended by an additional six months. Lack of availability of credit to small seed companies means that it is difficult for them to find the means to invest in infrastructure, high quality inputs and well-trained technicians. MISST sub-grants have assisted the smaller companies to develop some infrastructure – warehouses and irrigation - and in some cases have assisted with the purchase from two Districts of breeder seed from abroad and marketing activities. This is in addition to MISST sub-grants that are issued to increase the volume of seed being produced and to support demonstrations and field days. Sub-grants transfer the responsibility of managing funds and procurement from MISST to the sub-grantees who reported that this was in some cases a new skill and handled successfully. The team observed that the quality of the work on warehouses was variable, in one case a roof was poorly constructed and destroyed in a storm. CIMMYT acknowledged that they need to monitor more closely the technical specifications and workmanship of physical projects. ICRISAT manages a revolving fund for early generation legume seed, which, although not large enough to meet all a firms’ financial needs, does reduce the need for long-term credit for some legume seed producers. Government departments, particularly at the district level, lack sufficient operational funds which, affects their ability to participate in MISST project activities. This particularly applies to staff mobility, as local officers frequently lack funds for local taxis or for fuel or maintenance of government-owned motorcycles. To some extent MISST has been sensitive to this problem and, when possible, has included funding to pay for local transport. However, USAID regulations do not allow for direct financial transfers to government agencies, so the process can result in time-consuming delays. Policy. MISST was not designed as a policy-focused project and, except for consortium members participating as CGIAR experts in policy forums, it has not attempted to influence policy at the activity level. That being said, there are many aspects of government policy that have impacted MISST success, which include the following: ● A shortage of government spending on DARS and DAES reducing their capacity to produce breeder’s seed and outreach to farmers and to control the quality of seed in the market. ● The lack of a system of plant breeders’ rights reduces the incentive for international seed companies to invest in crop variety development in Malawi. 24 ● The Government of Malawi does not permit export of maize or soy grain limiting opportunities for sales. The Government of India introduced a ban on import of pigeon peas to stimulate domestic production thereby seriously affecting grain prices at a time when production had increased to meet demand from India. ● The FISP program has significantly stimulated the demand for hybrid maize seed, and to some extent for legume seed. The expansion in the number of seed companies from four in the late 1990’s to 28 today has been largely attributed to the stimulus of FISP. Project Development & Supervision. The development of MISST and its evolution since the start of its implementation has been a complex and inefficient process. The initial “Call for Proposals” was issued by USAID to individual CGIAR centers following a Country Strategy Review. Following selection of the concept notes, a consortium of four CGIAR centers was requested to submit one proposal by its lead ICRISAT. However, USAID advanced funds to each center in year 1 (2014), but required that a consolidated finance and technical report be submitted to the Mission. Each center submitted its financial report separately but one consolidated technical report was submitted by ICRISAT to USAID on behalf of the consortium. USAID decided that it would be more convenient for its financing mechanism, and improve the effectiveness of the program, to replace the four contracts with a single contract with a consortium leader. Meanwhile, implementation began in December 2014 and the ZOI began mobilizing staff. In the case of CIMMYT and the CMO, it took nearly 10 months to recruit dedicated project staff. An inception phase was also not included in the original design and as a result, nearly one year of the three-year program had already passed by the time staff were in post and activities became fully operational. MISST was initially designed to operate in the seven FTF districts. Following a severe drought in the southern region in early 2015, USAID decided to extend MISST to three districts in the secondary ZOI for maize, millet and sorghum, and then maize was removed from the secondary zone for the 2016/17 growing season. MISST was given a no-cost extension of one year to December 2018 in order to complete its activities. Technically closing MISST in December 2018 made no sense since December to April is the main growing season. Consequently, to make the most of the 2017/18 growing season, MISST was given an extra 6 months up to June 2019 with an additional $3 million in order to carry out activities through the 2017/18 season. As is evident from the above description, major changes have occurred to MISST management and also to programming content, both of which have affected performance, as has irregular funding. The latest budget for MISST was $24.7 million but this was reduced in 2017 following budget cuts across USAID. USAID expects to make up part of the cut but this is not certain. Table 12, below, shows the budget and the funding by year; the pipeline figure represents the remaining resources received by MISST for future activities, i.e. the difference between the amount received from USAID and the amount expended by MISST. 25 Table 12. MISST project funding budget from 2014-2019 FISCAL YEAR BUDGET CUMMULATIVE BUDGET DISBURSMENT PIPELINE YR1, FY14/15 2,702,857 2,702,857 6,900,000 4,197,143 YR2, FY15/16 7,017,940 9,720,797 6,000,000 3,179,203 YR3, FY16/17 6,491,596 16,212,393 - (3,312,393) YR4, FY17/18 4,933,887 21,146,280 - (8,246,280) YR5, FY18/19 3,279,589 24,425,869 - (11,525,869) Totals 24,425,869 12,900,000 Source: MISST CMO From the table, it is clear that no funds have been received from USAID since mid-2016, and that as a whole MISST has run a deficit in the last 2 years. USAID made it clear in November 2017 that pre-financing by consortium members was not acceptable to USAID. The budget shortfall has not affected all consortium members equally as CIP, being the most affected, decided in December 2017 to close down its activities and terminate staff contracts for those financed by MISST. However, CIP plans to restart activities if additional funds become available. The other consortium members have reduced activities including cancelling the sub-grant program and restricting the field program to 5 districts -- Lilongwe rural, Mangochi, Balaka, Machinga and Mchinji. Both USAID and MISST commented that the consortium did not initially understand the USAID financing mechanism, which led to disbursement delays to individual consortium members. Staff changes within USAID, lack of regular communication between USAID and MISST, absence of clear guidelines and documentation to agreed changes in project scope, and slow approval of work plans also affected project scheduling and performance. Consortium Structure and Management. The four MISST consortium partners are independent members of CGIAR and each report to a board of trustees, not to a central CGIAR leadership. The headquarters of these four Centers reside in Mexico, Nigeria, India and Peru. Management activities of each center are the responsibility of the country manager and HQ staff, which has slowed the decision￾making process within each consortium member. The Consortium Management Office (CMO) was established to manage consortium issues, particularly for preparing consolidated financial and technical reports for submission to USAID. This process has been challenging due to the time needed for each consortium member, to submit requests for funding to their individual HQs for approval, prior to consolidation by the CMO. The Consortium Advisory Committee (CAC), with ICRISAT as chair /convener, was set up to coordinate the activities of MISST and USAID and was supposed to meet every three months. In practice it has only met on 4 occasions out of a possible 13, although there have been undocumented Skype conferences in between. Given these circumstances, it is clear that more frequent meetings could have improved coordination among MISST members. Coordination of Activities. Despite the challenges noted above there were significant benefits in operating as a consortium rather than as separate entities: ● Working groups were set up and managed by centers and the CMO to address consortium-wide issues. These groups utilized the specific expertise of the leading partner and ensured that a common approach to each issue was maintained across the consortium. The working groups and their lead partners were: 26 o Seed systems - ICRISAT o Value Chains - IITA o Gender – CIMMYT o Nutrition - CIP o Consortium issues and M&E - CMO ● One center was nominated to coordinate MISST activities in each district ● Mega demos, reviews and reporting were all shared ● The single MISST reporting structure avoided double counting of beneficiaries ● Consortium members were able to work together in more ad hoc ways to support each other’s activities. Environmental Issues. Uneven rains during project implementation and late seed delivery to the EPA staff and farmers affected field-level performance activities. This was partly caused by ad hoc changes to the activity, delayed funds disbursement and late approvals of work plans which affected procurement and distribution plans. During field visits MISST team members estimated that at least 50% of the cereal and legume demonstrations over the life of the activity suffered from abnormal drought, which led to less effective demonstrations, field day cancellations and reduced seed and grain yields. Despite the drought conditions the DTM performed better than traditional maize varieties and the sub-grant approach used by CIMMYT led to fewer late seed deliveries than for the legumes. January and February 2018 were particularly dry with no rain in the secondary zone districts. The emergence of Fall Army Worm (FAW) as a major pest of agricultural crops across southern Africa has equally harmed demonstrations and seed multiplication in the last 2 seasons despite control measures. Evaluation Question 4: Will MISST Interventions Result in Improved Seed Production, Quality & Productivity? Findings in this section were obtained from FGDs and key informant interviews. Acceptance & Adoption of Improved Seeds & Technologies. Many beneficiary groups, local government staff and partner organizations were interviewed during the course of the evaluation. Nearly all respondents noted that farmers were keen to participate in the program, especially when the new varieties and technologies gave better results than traditional varieties and practices. In these cases, farmers wanted to adopt varieties and practices into their farms provided that inputs were available, affordable and that the new practices would not have other serious negative consequences. Even in drought-affected demonstrations, farmers were able to compare their traditional seeds, technologies to those under MISST. Orange Fleshed Sweet Potatoes. Adoption has been rapid and strong. For OFSP participants it was observed that they have been growing crop for third and fourth time and only received support during the initial growing season. The support was mainly the vines and trainings. CIP was successful in reaching over 56,000 beneficiaries through training on OFSP Good Agricultural Practice (GAP), creating awareness and distributing vines. CIP achieved this primarily by working closely with DAES at the District and EPA levels, sub-grants to We Effect, Concern Worldwide, CADECOM Dedza, Farmers Union and CADECOM ZARDD, integration partnerships with other USAID-funded programs including Tetra-Tech PERFORM, INVC, and Project Concern International (PCI)-NJIRA, and collaborations with Welthungerhilfe, InterAID and Farmers World Ltd. This process of working with DAES included embedding staff at the District offices and coordinating all aspects of demonstration development, extension staff training and farmer training. CIP and DAES also invited SSU staff from two districts for training. In addition, CIP conducted 27 business training for commercial and decentralized vine multipliers farmer awareness through demonstration trials and field days. CIP also conducted awareness campaigns via radio through the Ag Div activity in coordination with the Farm Radio Trust and conducted live theatre to 21 markets in the USAID ZOI. Input markets for OFSP vines are currently informal and there is no official system analogous to seed certification. MISST supported the establishment of the association of vine multipliers to facilitate the development of the market and is also providing technical input in the draft seed act which will see the root and tubers seed (vines) regulated. Both the association and the legislation are efforts to formalize the market. CIP noted that a lot of work is needed to sensitize the growers on the benefits of using quality assured planting materials if the input market for root and tubers is to be formalized. Most sales by Decentralized Vine Multipliers (DVM’s) are done locally, usually through awareness programs stimulated by demonstration field days. DVM’s noted that they sell both the vines and roots, and also use roots for home consumption. Farmers reported that OFSP is more drought tolerant than maize, provides, nutrition especially Vitamin A, food security through crop diversification and provides cash income through the sale of roots in local markets. One DVM farmer used to plant maize in his wetland which earned him roughly MK70,000 annually. This past year he planted OFSP and earned MK420, 000 from the sale of vines and an additional MK70, 000 from the sale of roots in just one growing season. He claimed sales of OFSP received higher prices than local varieties due to strong demand from consumers. The orange-fleshed sweet potato is most liked by consumers both in the local and urban markets. Additionally, because his farm is on a wetland, he can produce two crops in a single year, doubling his income. MISST worked to strengthen seed systems through the development of Commercial Vine Multipliers (CVMs) and DVMs, by organizing trainings on seed production and inspection and by bringing stakeholder in the OFSP seed value chain together. Farmers were trained using the Mother Baby Trial approach (training at planting, after establishing and at harvest). Storage training included trainings on post-harvest handling and storage with demonstrations of household-level storage structures). = CIP’s work on processing into chips, biscuits, fortified powders and purees was carried out outside the MISST project but it is possible that there will be synergies between the production activities and the processing. Drought Tolerant and Nutritious Maize. Varieties were widely accepted at sites even that were not seriously affected by severe drought as farmers could see that DTM varieties performed better than older varieties, especially with respect to higher yields. Drought tolerant and nutritious maize varieties were developed to be competitive under both optimum and stress conditions as not disadvantage farmers during the good years. The drought tolerant orange or pro-vitamin A maize was welcomed by farmers in spite the color as farmers are mostly used to white maize. Stress tolerant maize varieties in combination with conservation agriculture showed better yield gains when compared conventional tillage practices. CIMMYT beneficiary monitoring and evaluation tracking through agro-dealers (reported in March 2018) showed that 4,817 farming families accessed and bought seed of different DTM varieties either using vouchers from FISP or cash for planting in 2017/18, with purchases being highest in the driest districts of Mangochi and Balaka. The price of Hybrid DTM seed is quite high and has to be paid each year, nevertheless the improved yields justified most farmers paying the higher price. FISP pays part of the seed cost for many farmers but even without FISP farmers claim that they will continue to purchase the seed. Seed companies associated with the activity are investing in the production of hybrid DTM varieties to replace the previous open pollinated varieties (OPV). Nevertheless, OPV varieties represented 52% of sales by agrodealers in 2017/18. This is not surprising as they have been on the market much longer than most DTM varieties. OPVs are still preferred by many farmers because of their lower prices and because they can be recycled for repeated 28 use, reducing the financial burden on poor farmers. However, with repeated use, OPV yields will decline markedly within 2-3 years. Soybean targets were met in 2017/8 by reaching 43,000 smallholders farming households with improved varieties from 2,500MT of seeds. This tonnage covered 15,000ha with improved varieties and management practices. Seed-producing farmers reported that yield increases of 50% had been achieved. The high yields provided the starting material for local seed banks to sustain the production and distribution of early generation seeds. MISST developed 8,500 demonstration plots and trained 25,000 farmers in the use of GAPs in harvest years 2016 and 2017. Adoption and use of these improved seed are demonstrated by the fact that the total area planted with new varieties (including locally multiplied seed) increased by 60% from 114,000 ha to 188,000 ha during the activity life. The process of achieving breeder seed production involved developing basic seed service contracts with private sector seed multipliers (PSSM) who were then responsible for multiplying basic seed after receiving IITA breeder seed. IITA took delivery of the basic seed, tested it for germination and purity through SSU with a fee of $1.5 per kilogram. The basic seed was then sold to seed companies and seed multipliers who purchased it from IITA at $2.0/kg for certified seed production. Additionally, three improved soybean varieties have been promoted along with local multipliers being trained in the use of quality seed, planting seeds on ridges with double rows, use of inoculants, fertilizers and pesticides. In terms of availability of pre-basic seed after MISST project ends, two of the larger seed companies and two new local seed companies are currently making arrangements to develop their own, and in another case, acquire pre￾basic seed from an international supplier (QualiBasic Seed Company Ltd - AFSTA) headquartered in Nairobi, Kenya, but with subsidiaries in Zambia and South Africa. Dry land cereals and legumes have been produced in the southern region before but using land races that have longer maturity periods and are very tall and increasing drudgery during harvesting that is mostly done by women the new sorghum and millet varieties being promoted by MISST and tested under two unusually dry growing seasons have considerably shorter growing periods and their yields are more reliable than maize. Although changing food consumption patterns is always difficult, farmer groups in Nsanje and Chickwawa districts reported that there really was no choice but to adopt the new crops and those households have clearly accepted this dietary change as a tradeoff for food security. During the FGDs in Chikwawa and Nsanje the participants reported that they had seed multiplication plots and demos in the first year. This was followed by production of crops without support in the second year. MISST was unable to work in the secondary ZOI in the second year and so technical support was not available to the growers. Acceptance and plantings of improved groundnut and pigeon pea has been good but less spectacular than soybean, sorghum and millet, partly due to problems with seed quality (in FY17 as discussed earlier) and delivery timing, but also because the popular varieties have inherent challenges such as susceptibility to disease and pests e.g. CG7 that has is highly susceptible to rosette disease especially when intermittent drought occurs during early stages of crop growth as occurred during the two years of under investigation. More recently released improved groundnut varieties such as CG8-14 are currently being bulked and tested by MISST project farmers but will not be widely available as certified seed for distribution until 2019 especially for drought tolerant and rosette resistant material. In the case of pigeon pea, the crop is primarily a cash crop and depends heavily on external markets although MISST is promoting the crop for local use in the central region. Introducing new crops and technologies bring with them new challenges for growers – farmers reported problems with livestock damage to crops that were harvested after maize (particularly pigeon peas), bird damage on dry land cereals and pest and diseases on pigeon pea crops after flowering. These difficulties 29 may reduce adoption rates if local solutions are not found. Lack of markets for the produce may also reduce the rate of adoption. Once household demand is met farmers need to sell the excess production for cash but prices for pigeon peas and the dry land cereals tend to be very low. Conservation agriculture practices in maize, intercropping maize with pigeon peas, OFSP tuber storage and other technologies have been demonstrated and evaluated by many farmers using participatory approaches, farmers have been willing to try these different technologies but it is still early to say if the take-up will be permanent. Sustainability can be considered in three categories, firstly the new varieties and technologies; secondly the functionality or performance of the input markets, and thirdly the continuing support to farmers at the local level to maintain the advances made and improves on them where feasible. The new varieties and technologies that have been released are now available in the market and, provided that the FISP subsidization program does not end resulting in a sudden collapse of the seed industry, the seed companies reported that they are likely to continue producing the new varieties that are demanded by farmers. Already some seed companies were observed to operate outside the FISP subsidy program, so profitability absent the subsidy program is indeed possible for some companies. Capacity building and training has increased the efficiency of seed companies and agrodealers and those input market improvements are likely to be continued. In several interviews with the seed companies which had sent their staff to the para-seed inspector training, they stated that the PSI had improved the efficiency of the companies. One multinational seed company claimed that through the PSI training they were able to reduce the losses they incurred by improved monitoring their out-growers and also improved their internal quality control. A local seed companies claimed that the PSI had reduced transactional costs by reducing the losses due to certification failure. The sustainability of local legume seed production is less certain as farmer producer groups and seed banks established during the activity are still relatively new. Farmer organizations which participated in the activity (i.e. NASFAM, STAM, FUM, and ASSMAG), and local government staff reported that they will continue to support these groups and farmers although this support is likely to be at a lower level with the end of MISST financial support. Farmer groups and DAES representatives both believe that the adoption of the new varieties will be sustained into the near future. Interviewees noted that sustainability of the sector would be improved if there was greater access to seed companies through catalogues with variety descriptions and contact details for seed suppliers. Although individual companies produce commercial literature neither government nor the seed trade organizations produce literature listing a wide range of potential products and suppliers. It was observed that the agro dealer network has been strengthened by several projects including MISST. It was observed that mobility of farmers is low and that their access to retailers who can supply goods and services and advice at the local level is limited. Farmers noted that it would be of considerable value if they had better access, at the sub-district level, to agro dealers with products and the ability to give product-related and more general agricultural advice. The prospect of SSU sustainability is also critical to the availability of breeder seed to the seed companies and to the retraining and re-licensing of para-seed inspectors. DARS believes that government financial sustainability will improve following the recent adoption of a new seed policy and the creation of a National Seed Commission (NSC) but, as yet, there are no guarantees of government sourced funding. MISST provided some support to DARS on policy and implementation related to seed certification (digitization of records) but was not involved with reform of variety registration or national trade policy development. It was widely reported in interviews that DARS needs further technical assistance to be able to properly support the seed sector. OFSP root and vine production is likely to be sustained given its rapid uptake, although in some areas the program has been a victim of its own success. For instance, some vine producers have increased supply above market needs and have been unable to sell all their vines. This development has impacted CVMs 30 more than DVM’s because the former produce only vines for sale, while the latter produce both vines and tubers, thereby diversifying their access to markets. It is likely that there will be spillover effects from the spread of vines through social networks to indirect beneficiaries. These effects will not be seen in the activity surveys as these only include direct beneficiaries. Impact is difficult to assess at this stage of the activity and will be determined in end-line surveys. Based on discussions with beneficiary groups, it seems likely that MISST benefits will result in increased production and improved resilience for vulnerable households and rural communities. All stakeholders claimed that easier access to markets and higher prices for grain would improve the overall impact. It is likely that MISST will have a beneficial impact on nutrition due to the strong adoption of OFSP, acceptance of sorghum as staple food in dry areas and an increased contribution of biofortified crops to major school feeding programs. Although nutrition indicators were dropped at the beginning of the activity MISST (led by CIP) is working with government staff and project partners at district level to provide training on nutrition that integrates all MISST-supported value chains Evaluation Question 5: How did MISST work with other partners and did integrated activities promote or hinder attaining results? Findings in this section were obtained from FGDs and key informant interviews. MISST works with partners through sub-grant agreements and, where there is no financial commitment, through MoU’s. All partners (including government, seed companies, agrodealers, vine multipliers and farmer organizations) reported that that MISST worked well with them and that technical support from MISST was considered very valuable. Several partnerships were broken off at short notice during 2017 due to funding shortages and this caused difficulties for partners who had already made commitments. During discussions with DADOs it was noted that some MISST components did a better job coordinating with district government agencies than others, with CIP rated highly in this regard. On the other hand, it was also noted during interviews that some DADO’s are very proactive with MISST seeking close collaboration, whereas others are more passive with implementation activities. The transfers of government staff also compounded partnerships in some districts MISST has operated alongside six other USAID projects funded through FTF or Food for Peace (FFP). The level of cooperation with each has varied substantially according to the level of mutual benefits received. UBALE is an agricultural market and livelihoods project being implemented in the southern districts. Until MISST was extended to include three southern districts in late 2016, there was no clear opportunity for cooperation with MISST. However, once operational within the southern districts, MISST supplied improved seed and technology while UBALE provided the outreach and links to commercial partners. Before the start of the 2016/17 seasons the activity agreed that MISST varieties and technologies would be included in the UBALE demonstrations and seed fairs. Due to late funding and non-clarity on implementation modalities, between both projects, seed was not provided in time for planting with the main demonstrations, although DTM and sorghum were planted later on government sites. Following this failure, the cooperation was much closer in the 2017/18 cropping season with sorghum and millet varieties included in the demonstrations. In Chikwawa District, ICRISAT reports that MISST and UBALE jointly mounted 1,052 demonstration plots using seed provided by the MISST. UBALE and DAES gave some excess seed to 103 farmers in Doro EPA where they grew the crop following a cluster system who were then expected to harvest 8MT of seed for sale to surrounding areas. UBALE reported that the new varieties were favored by growers over traditional varieties as yields were higher, seed quality better and 31 growing seasons shorter. They also reported that unimproved varieties typically yield 200 kg per ha while new varieties yield more than 1,000 kg per hectare. PERFORM operates in the primary ZOI and introduces conservation agriculture and OFSP through its agriculture component. An MOU was established between the activities and a strong joint program operated during 2016/17 and 2017/18 with mutual benefits. Delayed inputs hampered the program to some extent in the first season but despite this, OFSP and DTM together with a small number of Groundnut and Pigeon pea demonstrations were organized and supported through PERFORMs extension network. Njira works to promote food security in two ZOI districts including through the introduction of improved varieties, technologies, and capacity building through training. MISST has worked closely with all 4 crop components and report that the cooperation has been very successful. Njira is not involved with crop marketing, which has not been addressed in the joint project activities. Ag Div (started in 2017) reports that it has close cooperation with MISST on OFSP but much less on groundnut and soybean, although these are also part of its remit. Both Ag Div and MISST agree that this should be strengthened. Ag Div is also working at the strategic/policy level on seed issues with the development of seed roadmap, which appears to be an area where the two organizations should be working together but are not. SANE supports linkages between research and extension. Of the 54 mega-demonstrations in 2016/7, SANE participated in all of them to ensure maximum value. SANE assisted with transport of participants and other facilitation activities. Due to funding restrictions for both projects the activities were scaled down in 2017/8. Both projects felt that the collaboration was worthwhile and improved the linkage between extension workers and farmers to DARS and the District Centers. SSTP (Scaling Seeds and other Technologies Partnership - closed 2017) was closely related to the aims of MISST and in the original project document it was envisaged that there would be close cooperation. While that did not occur, there has been some positive synergies in that agro-dealers developed under SSTP are now better able to participate in the seed value chain and improve the sale and delivery of seed produced under MISST to farmers. Both SSTP and ICRISAT stated that the cooperation did not occur due to the difficulty of joint programming when the programs started at different times and also because the two programs used different methods of implementation. Kulima EU (implementation expected May 2018) - Nutrient dense DTM varieties like MH43A and Quality Protein Maize (QPM) will be promoted on a cost-sharing basis at 18 KULIMA sites in Mulanje, Kasungu and Mzuzu, which are outside the FTF-ZOI. The collaboration with KULIMA is expected to be a strong opportunity to create demand for these varieties more widely than the MISST core area. Introduction of Nutrient dense maize varieties in the KULIMA (Kutukula Ulimi m'Malawi) project aims at further linkage with the school feeding interventions being implemented under Afikepo, a nutrition specific sister project of KULIMA. AFLASAFE PROJECT FINDINGS – MAIZE & GROUNDNUT Findings in this section were obtained from FGDs and key informant interviews. Evaluation Question 1: To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? 32 The primary goal of the Aflasafe project is the development, promotion and commercialization of biological control of aflatoxin in the groundnut and maize value chains to improve public health, increase trade, augment smallholder income, and enhance food security in Malawi. The goal is comprised of four major objectives: 1. Quantify the scale of the aflatoxin problem in groundnut and maize at harvest, in farmers’ stores, warehouses and markets; 2. Develop a commercially-ready biological control product for aflatoxin to reduce groundnut and maize crop contamination; 3. Demonstrate product utility in crop value chains, develop a business plan, guide commercialization opportunities (including product manufacturing) and steer guidelines for intellectual property management; 4. Enhance technical capacity of national partners, extension services, farmers, and other actors in the groundnut and maize value chains to incorporate aflatoxin bio-control in integrated management to minimize aflatoxin contamination. Accomplishments. In spite of the financial and subsequent implementation challenges that affected this component, significant progress has been made to date on all four objectives. Regarding objective 1, through its soil sample testing in all three regions of the country, and specifically within the 10 districts of MISST operations, the severity of aflatoxin contamination in groundnuts and maize was confirmed (Table 13). The Aflasafe product has been tested and is currently being used in several African countries and has been proven to effectively control aflatoxin. IITA tested and modified the Aflasafe product to control specific disease strains in Malawi that were identified through extensive soil sampling. This new product designed for Malawi was tested and proven effective and documentation and is now being submitted to DARS for registration purposes. In terms of objective 3, discussions have been underway between DAR, IITA, and several private sector businesses (e.g. Agriculture Input Supplies Limited (AISL)) to discuss plans to commercialize the production of Aflasafe. In an interview with the Program Manager of the Crop Control Board, Ministry of Trade, he stated that the Aflasafe project was indeed on track to achieve all of its objectives, as he (Mr. Mphatso Dakamau) has been intimately involved since the activity’s inception. Finally, objective four has been met with demonstrations, field days and extensive trainings of DAES extension workers, DARS research staff, and farmers. Specific details for each of the four objectives are presented immediately below. To accomplish the first objective, the activity collected a total of 2,181 soil samples (Maize: 1510; Groundnut: 671) in 24 Districts in the Northern, Central and Southern regions of Malawi. From these samples, strains were selected for product formulation at two levels: 1) strains for Malawi-specific product Aflasafe MW02, and; 2) strains for a regional product Aflasafe MWMZ01. In the 2016/2017 growing season, 143 ha (76 ha of maize and 67 ha of groundnut) from 559 farmers’ fields were treated with Aflasafe for testing. A total of 1,154 households benefited from Aflasafe training and awareness raising activities on aflatoxin management. At harvest, 458 maize samples (229 treated and 229 untreated control) and 468 groundnut samples (234 treated and 234 control) were collected from 463 farmers’ fields. Altogether (2015-16; 2016-17; 2017-18) more than 3,000 farmers and agricultural extension staff have been trained on Aflasafe and aflatoxin management of which 44% were women. The Department of Agricultural Research Services provided building space for an aflatoxin research and training laboratory. The space was upgraded by IITA to improve the infrastructural and technical capacity of national partners, extension services, farmers, and other actors in the groundnut and maize value chains. The installation of equipment is complete and the laboratory is fully functional. Laboratory technicians have been recruited and trained on surveillance, aflatoxin detection, fungal identification, and related research on aflatoxin issues. 33 Results from two years (2016 and 2017) of testing Aflasafe show its effectiveness in controlling the aflatoxin strains, which are native to Malawi (Table 13). In the case of groundnut, with the MW02 treatment, the control shows 614 parts per billion (ppb) in 2016 and 66ppb in 2017. Under Aflasafe MW02 treatment, aflatoxin residue was reduced to 15ppb and 6ppb, respectively. Similar results were shown for the Aflasafe product MWMZ01.The situation for maize showed similar efficacy for the MW02 treatment; the control had 115ppb in 2016 and 20ppb in 2017 whereas the treated plots were reduced to 1ppb in 2016 and 2017. Table 13. Efficacy data results for Aflasafe conducted over two seasons in MISST - ZOI Treatment Aflasafe MW02 Aflasafe MWMZ01 Groundnut Maize Groundnut Maize 2016 (n=62) 2017 (n=113) 2016 (n=69 2017 (n=115) 2016 (n=70) 2017 (n=102) 2016 (n=69) 2017 (n=114) Control (ppb) 614 66 115 20 444 49 52 30 Treated (ppb) 15 6 1 1 16 3 1 3 Aflatoxin Reduction (%) 98 90 95 95 96 94 98 91 Source: IITA PowerPoint, 2018 Household Resilience. Aflasafe is anticipated to have both direct and indirect benefits on household resilience – Aflatoxin free groundnuts and maize will eliminate health issues related to family consumption of contaminated products, which is currently ongoing. In focus group discussions, farm families noted that prior to their interaction with MISST, family members were consuming contaminated maize and groundnuts, and were also feeding it to their livestock. Once they learned, through MISST’s awareness raising activities, of the significant health dangers associated with aflatoxin, they have stopped consuming infected crops, which they were trained to identify by MISST staff. Summary ● Prevalence of aflatoxin in groundnuts and maize has been established through extensive soil and crop sampling in MISST target districts. ● Two Aflasafe products (MWMZ01 and MW02) have been developed for Malawi. ● Field evaluation of Aflasafe products has been successful in 11 districts and 22 EPAs across three regions within three agro-ecological zones. ● Aflasafe products have shown more than 80% reduction in aflatoxin. ● More than 3,000 farmers and staff have been trained on aflatoxin management. ● Refurbishment of the aflatoxin laboratory and supply of required equipment for aflatoxin research has been completed and will be incorporated into DARS. ● Several technicians have been recruited and are being trained on surveillance, aflatoxin detection, fungal identification, and related research on aflatoxin. ● A PhD Pathology student funded by MISST will graduate from the University of Arizona in 2019. DARS should make arrangements to hire this individual to operate and manage the Aflatoxin laboratory upon his return. ● Registration dossiers for Aflasafe release are under preparation for submission to the government (DAR). ● Partnerships are being discussed to enable future commercialization of Aflasafe. Aflatoxin Testing Laboratory. At project start-up there was a lack of commitment from Ministry of Agriculture, particularly with respect to assigning DARS staff to run the Aflatoxin lab. Introducing new technologies requires government to follow certain legal and regulatory procedures that must be followed, often requiring 2-3 seasons of testing before release. DARS was to hire staff but failed to do so, so now 34 it has no trained staff to run the lab. A recent meeting between DARS and IITA was held to rectify the situation. DARS has agreed to provide staff for the Aflatoxin lab, in addition to hiring some IITA staff. DARS is considering creating a Commercialization Branch (CB) to operate the lab as a “for-profit” entity in coordination with IITA. The primary function of the DARS-CB would be to monitor the quality of the Aflasafe product to ensure standards are being maintained, and to monitor the quality of grain produced by farmers to ensure it meets safety standards for consumers. MISST funded a PhD student in the USA who is expected to graduate in early 2019. In an interview with a large private sector agribusiness firm, Agriculture Input Supplies Limited (AISL), they also expressed interest in commercializing Aflasafe as it is in the process of constructing a lab for the development of inoculants. The facility will also process, treat and package seeds ranging from under 5kgs to 50kgs. IITA mentioned that several private sector processors could be stationed in different regions of the country and work in tandem with DARS-CB. The idea would be to establish a network of Aflasafe processors who would work in coordination with distributors to sell to farmers. In turn, this network would be responsible for monitoring, via SSU, the proper application and use of Aflasafe by farmers. After harvest, samples would be tested to ensure the grains were at acceptable aflatoxin levels in parts per billion. This “closed system” would be necessary to ensure health and safety standards for consumers were being adhered to. Again, other agencies critical to this process will be the Ministry of Health, Ministry of Trade, Malawi Project for Aflatoxin Control (MAPAC) and Program for Aflatoxin Control in Africa (PACA). Evaluation Question 2: To what extent has MISST improved the functionality of input and output markets of its focus value chains? Not relevant to Aflasafe. Evaluation Question 3: What challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. Challenges. First, it should be noted that the IITA Aflatoxin component has different goals and objectives than the other four MISST components, which are pursuing seed system development. As described in an earlier section the funding to IITA was irregular and the budget was cut by 50% in 2017. These financial challenges had significant repercussions on the scale and scope of future activities, Specifically, IITA adjusted its Work Plan and scaled back individual farmer training, demonstrations and field days on Aflasafe; it also reduced collection of laboratory samples by roughly 30%. In spite of this scaling back of technical and program activities, IITA continued its contribution to the district-level Mega Demonstration plots with its Aflasafe trials, comparing treated versus untreated groundnut and maize plots. Finally, CMO management meetings were held monthly to review progress and achievements and discuss future planned activities. Developing Aflasafe Product. IITA is leading the effort but works closely with Ministry of Health (MOH) and Ministry of Trade (MOT), coordinates carefully with its partners and gives regular progress updates. During interviews, the MOT mentioned MISST is on the right track for achieving its objectives in developing a viable and affordable Aflasafe product. Three main steps are required to get Aflasafe up and running, and all three involve financial issues with respect to both potential private sector partners and government agencies (DAR, Ministry of Health, and Ministry of Trade). Step 1 is finding the appropriate Aflasafe bio-control for Malawi through trials and testing; Step 2 involves the registration process and approval from DARS; and Step 3 is commercialization of the Aflasafe product, through both government 35 and private sector involvement. To date, Step 1 has been completed, and step 2 is still in process – a dossier is being compiled and will be submitted to DARS in the next couple of months. Following the technical evaluation, there are many economic issues that need to be addressed; the Ministry of Trade has raised the following questions, which are being discussed and addressed by the IITA team: ● What is the market potential for Aflasafe? ● What will be the cost of product development and who will pay for it? ● What will be the break-even point in production of Aflasafe (necessary to set a price affordable to farmers)? ● Should product development and commercialization be publicly or privately led? ● Malawi Bureau of Standards will need to become involved in regulation, in terms of ensuring Aflasafe quality standards are maintained, and food safety issues adhered to for consumers. ● What will be the most efficient process to produce and distribute Aflasafe? ● Price incentives will be necessary for aflatoxin-free maize and groundnuts. To acquire a premium, jargon-free consumer awareness programs of aflatoxin health risks, and benefits of aflatoxin free produce must continue to be addressed through multiple additional media outlets. ● What oversight controls are needed to maintain bio-control effectiveness and sustainability? Sustainability. According to key informant interviews with IITA, government and other projects and agencies, the sustainability of Aflasafe as a “product” will depend on the process that is used to launch commercialization. The process is as follows: 1) Registration of Aflasafe products (target date for dossier submission is the 3rd Quarter 2018); 2) Commercialization for scaling up (3rd Quarter 2018 to 2021); 3) agronomic trials to improve product performance based on feedback from commercial partners; 4) increasing scope of product use; 5) monitoring of product performance; 6) awareness /training/ advocacy; 7) Impact assessment on health and income (as animal feed and human consumption; 8) Investments needed: Approx. $2 million for 3 years for commercialization; $500,000 per year for 3 years for technical support. 9 To increase the likelihood that Aflasafe will become accepted and utilized as an aflatoxin control agent, IITA has anticipated the following commercialization roadmap to full-scale commercialization and release to the agriculture sector. Some of these activities will be initiated within the current MISST program but most will need to be addressed in the future. 1. Product registration. 2. Identify potential commercial manufacturer for Aflasafe. 3. Run a pilot program: to market test the product. 4. Market development including branding. 5. Testing for quality compliance for processors and exporters of grain to meet aflatoxin safe limits. 6. Using the improved grain quality to open up new food processing opportunities. 7. Partner with leading supermarkets/food processors to develop publicity campaigns Aflatoxin free products. 8. Partner with the relevant organization and government ministries to raise awareness on aflatoxin health effects and enforce regulation. 9 These figures are tentative estimates that were calculated by IITA in discussions with government agencies and private sector firms. 36 Interviewees were optimistic about the technical value of the product and noted that Aflasafe is in commercial production in Nigeria and other countries. Government interviewees were supportive of the process and committed to its success. Interviewees also recognized, however, that there is a lot of work to be done to achieve the steps outlined and, moreover, they are unclear how the product will be adopted by farmers unless there is a premium for the healthy grain. They stated that this needs to be achieved by a combination of product quality control regulation and recognition by the market of the added value of improved grain quality. Evaluation Question 4: To what extent will the outcome of MISST interventions likely result in improved seed production, quality and productivity? The IITA Aflatoxin project involves the development and use of the Aflatoxin Laboratory for testing purposes, as well as the development and use of the Aflasafe product. Together these two project outcomes will have the potential to improve the production, quality, and productivity of maize and groundnut seed, as well as aflatoxin free grain for livestock feed and human consumption. Evaluation Question 5: How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? The Aflasafe project has required a highly integrated and collaborative approach to achieve its objectives. In fact, without this integrated approach through government and private sector partners, achieving the final objectives would be impossible. Consequently, IITA worked closely with all line Ministries to implement the Aflasafe project – including DAES (to select participating communities and farmers for setting up trials; MOH and MOT and DARS to assist in the testing and identification of regulatory issues related to the testing and development of Aflasafe. In addition, IITA organized a national technical working group that meets three times yearly to discuss ongoing issues related to product testing, development and government registration. The consortia also established a multi-stakeholder platform (steering committee), including private sector partners through the assistance of the Association of Commerce & Industry. At the testing level, farmers have been intimately involved in various research trials, and are now aware of aflatoxin issues in maize and groundnuts and the important role of Aflasafe in mitigating the effects of aflatoxin. SUMMARY OF FINDINGS FOR ALL COMPONENTS ● Improved new varieties have been made available to seed companies and farmers. ● Capacity of seed companies, agrodealers and farmers for seed production and marketing has been improved. The training of para-seed inspectors was particularly highly valued as it alleviated bottlenecks in crop certification and improved the capacity of seed companies to monitor the quality of their production. ● DARS/SSU has been supported through the renovation of the seed lab and support to the mobility of inspection staff. ● The awareness of new crops, varieties and technology by farmers, agrodealers, seed companies and government has been significantly improved. ● New varieties and technologies have been used by farmers and are likely to be adopted provided that affordable, good quality seed is available. ● The resilience of vulnerable communities increased through improved availability of drought tolerant crops and varieties and the ability to earn cash through the sale of agricultural crops. ● The development of output markets has not yet been adequately addressed by MISST. 37 ● Aflasafe has been successfully tested and has proven to be effective in the field. Registration dossiers are under preparation for submission to the government ● The refurbishment of aflatoxin laboratory and supply of required equipment for aflatoxin research has been completed. ● Project effectiveness continues to be limited by irregular funding, complex administration within the consortium organization, resource constraints of government partners, unusually dry growing seasons, and fall armyworm infestations on maize. CONCLUSIONS & RECOMMENDATIONS Question 1. To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? Conclusions MISST has made significant progress against key indicators for each of the four strategic objectives it is expected to achieve by end of project. MISST was well designed to address the key issues of seed shortages of improved varieties for all key crops and also addressed the lack of technical knowledge and awareness of new varieties and technologies for seed companies, agro-dealers, community groups and farmers. Maintenance breeding was not included in the program and that remains a weakness in the input value chain. Objective 1. In order to improve functionality of input markets of selected value chains for seed, MISST has facilitated the production of 17,500MT of cereal and legume seed and over 200,000 Orange Fleshed Sweet Potato bundles. Improved new crop varieties and technologies have been made available to seed producing companies, agro dealers and farmers and awareness of new crops, their varieties and allied technologies by farmers, agro dealers, seed companies and government has been significantly improved. IITA initiated and supported the development of Aflasafe using local strains. Field trials were conducted and have proven successful and government registration is expected when the 2017/8 data analysis is completed and dossier submitted. MISST has created a functioning aflatoxin laboratory and has raised awareness of the problem of aflatoxin as well as developing the appropriate control methods. Objective 2: In relation to capacity building, the number of stakeholders directly benefitting from MISST and the number of beneficiaries trained have both exceeded the target in each of the last three. The capacity of seed companies, agro dealers, farmers and government staff has increased following the training on production of new varieties and their technologies, as well as business development, seed production and inspection. Objective 3. New crops, varieties and technologies have been tried by 140,000 farmers and others and the number of organizations applying improved organization-level technologies or management practices is over target in each of the last 2 years. Farmers using new varieties and their allied technologies are likely to continue to do so, subject to the availability of seed and sufficient funds to purchase seeds when needed. Objective 4. The activity has addressed resilience through drought tolerant maize sorghum and millet varieties, drought tolerant and disease resistant legume varieties, OFSP varieties, and through the introduction of new technologies including conservation agriculture to enable crops to perform well in low rainfall conditions. Introduction of a range of new improved varieties has enhanced diversity and given farmers a wider choice of crops to grow other than maize. Recommendations Follow-up Project. MISST has largely succeeded in achieving its objectives and reaching its targets but national uptake of improved seed by smallholder farmers is still low. The causes of this are well covered by the 38 report and roadmap developed by the Ag Div project in 2017. Part of the issue is the under-investment in non-maize seed value chains and the continued lack of awareness of smallholder farmers of the value of improved seed, varieties and production technologies. MISST succeeded in its target districts and its ZOI, but many adjacent districts also need support. A follow-on project would be beneficial to consolidate the gains made in strengthening seed companies and community producer groups, and with further outreach could reach more beneficiaries in the ZOI and beyond. If a follow-on project is considered, it should incorporate successful features of MISST, such as the sub-grant program, mega-demonstrations with their associated field days, including the investment and implementation of demonstration activities by local seed companies and establishing strategic partnerships for scaling. Use of “Lessons Learned”. The administrative, financial, and reporting problems encountered in setting up MISST and operating the consortium was quite severe. Use of “lessons learned” by USAID and the CGIAR centers must be factored into future programming to avoid the difficulties encountered during MISST to ensure more efficient implementation. Some activities should be initiated before the end of MISST, and others activities will need to be included in a follow-on project (or both), implemented by the CG centers or by other USAID projects. Aflatoxin & Aflasafe Conclusions The primary goal of the Aflasafe project is the development, promotion and commercialization of biological control of aflatoxin in the groundnut and maize value chains. Briefly the four major objectives are: 1. Quantify the scale of the aflatoxin problem in groundnut and maize; 2. Develop a commercially-ready biological control product for aflatoxin; 3. Demonstrate product utility, develop a business plan, guide commercialization opportunities and steer guidelines for intellectual property management; 4. Enhance technical capacity of national partners, extension services, farmers, and other actors in the groundnut and maize value. Regarding the first two objectives, IITA tested and modified the Aflasafe product to control specific disease strains in Malawi that were identified through extensive soil sampling. To support the testing of the samples, DARS provided building space for an aflatoxin research and training laboratory. The space was upgraded by IITA to improve the infrastructural and technical capacity of the laboratory. The installation of equipment is complete and the laboratory is fully functional. IITA initiated and supported the field-trial phase of Aflasafe; these trials have proved successful and government registration is expected when the 2017/8 data analysis is completed. Regarding the third objective, discussions have been underway between DAR, IITA, and several private sector businesses (e.g. AISL) to discuss plans to commercialize the production of Aflasafe. Finally, objective four has been met with demonstrations, field days and extensive trainings of DAES extension workers, DAR research staffs, and farmers. In an interview with the Program Manager of the Crop Control Board, Ministry of Trade, Mr. Mphatso Dakamau stated that the Aflasafe project was indeed on track to achieve all of its objectives. Recommendations Continue Discussion Process. To ensure sustainability of the aflatoxin laboratory, discussions between IITA and DARS have occurred and government commitments have been made to properly staff, manage and 39 fund the laboratory. These discussions must continue with benchmarks and milestones until the lab is fully operational and has secure and sustainable funding. Acquire a Laboratory Manager. A PhD Pathology student funded by MISST will graduate from the University of Arizona in 2019. Arrangements must be made by DARS and IITA to hire this individual to operate and manage the Aflatoxin laboratory upon his return. Continue Towards Aflasafe Commercialization. Aflasafe has proved that it is technically effective and can contribute both to human health and as a means of adding value to Malawian agricultural produce. In order for Aflasafe to become launched as a “product” it will require significant resources. USAID should assist the process of commercialization by continuing support to IITA as the technical partner but also through a more commercial project dedicated to product development. Partners in this effort will include government agencies, industry trade associations, seed companies, and input supply firms (e.g. AISL). The key next steps are outlined in the findings section. Question 2. To what extent has MISST improved the functionality of input and output markets of its focus value chains? Conclusions Input Markets. Under this component MISST has enabled the production of good quality seed planting material of all targeted value chains. At the same time, it has enhanced the capacity of partner seed companies to produce higher volumes of good quality seed than before the activity started. Through its work with seed companies, agrodealers and local authorities, MISST has strengthened market linkages particularly through field days where all parts of the seed chain were invited and participated. The functionality of legume seed supply has been increased substantially with the support of ICRISAT and IITA and the inclusion of legumes in the FISP program. MISST has also benefited from the presence of the legumes development trust that advocates for inclusion of legumes in government’s development activities. Moreover, several local and international seed companies are now producing groundnut, soy and other legumes. The need for ICRISAT and IITA to act as a seed producer and seller of basic seed in this emerging market will be declining as the ability of seed companies increases. Community-based production has been supported by MISST to meet the gap in certified seed production. The sustainability of these groups will vary according to their leadership and support from government and other organizations. Output Markets. Output markets have not been adequately addressed by this project. Some work has been done on encouraging home consumption of OFSP, and ensuring that protein and vitamin A rich maize varieties are utilized in school feeding programs. However, it is not clear that consortium members understood that output markets were part of the activity objectives or the concept was misinterpreted. In some cases, it was apparent that the term “output market” was taken as referring to the sale of seeds and planting material by companies or local farmer organizations rather than markets for final output as grain or roots for consumption. The lack of work on output markets reflects the incomplete harmonization of objectives when the stand-alone center programs were merged under a single program. Recommendations Seed Supply Catalogue. To further strengthen the input markets, any follow-on project should develop, publish a catalogue of seed companies and agrodealers including the crops, varietal descriptions, and contact numbers. Commercial organizations and STAM should participate in production, funding and dissemination to all stakeholders including every EPA office in the country. Updates could include additional publicity and practical information of value to farmers and extension workers. The catalogue 40 could be developed through STAM and, while electronic dissemination may be easiest, it may not be appropriate for EPA offices. Agrodealer Capacity. Increase capacity of agrodealers by fully equipping them with necessary knowledge and skills to transform their outlets into a one-stop shop able to provide full information on GAP, including insect pest management, disease control, proper use, storage and handling of farm chemicals, and generally to support and complement DAES extension workers. Study on future of Revolving Seed Fund. The revolving seed fund was set up with USAID and Irish Aid support to promote the production of legume seed and is currently managed by ICRISAT and IITA. The system nurtures the growth of local legume seed companies given their interest by large multi-national companies despite a major shortage of commercial legume seed nationally. With the development of a more commercially focused legume seed production system encouraged by MISST, the current revolving fund system may need a review to determine how best to strategically support growth of the legume and other under-invested crops to meet the needs of Malawi’s seed market. Options that have been proposed include privatization of the fund to an independent agency, combining it with a new maintenance-breeding unit, or maintaining the status quo. At the same time the future role of ICRISAT as a seed producer should be examined to assess if and how this function might be transferred to a viable private sector entity or other means to encourage growth in these under-invested value chains. Study on Output Markets. Opportunities to develop output markets were not adequately addressed under MISST. It would be advantageous to conduct a thorough market assessment to identify potential opportunities (and constraints) for specific agriculture commodities, especially those being promoted by MISST. The study should include the potential to develop a warehouse receipts system, to re-activate the commodity exchanges previously run by The Agricultural Development and Marketing Corporation (ADMARC), and to examine processing and export opportunities. Previous projects have attempted to introduce a warehouse receipts system, so lessons should be drawn from those experiences. Specific opportunities should be led by commercially-focused programs. Question 3. What challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. Conclusions Project Development & Supervision. The development of MISST and its evolution since its inception has been a complex and inefficient process. USAID contracted and advanced funds to the four individual centers in in 2014 but later decided to replace the four contracts with a single contract with a consortium leader. For consortium members, the process of adapting their contracts and reporting systems from USAID to ICRISAT, as consortium leader, was difficult. Implementation and staff mobilization began in December 2014 but in the case of CIMMYT and the CMO, it took nearly 10 months to recruit dedicated project staff. An inception phase was also not included in the original design and as a result, nearly one year of the three-year program had already passed by the time staff were in post and activities became fully operational. MISST was initially operated in the seven FTF districts but following a severe drought in the southern region in early 2015, USAID extended MISST to three districts in the secondary ZOI for maize, millet and sorghum although maize was removed again after the first year. MISST was given a no-cost extension of one year to December 2018 in order to complete its activities and this was extended to June 2019 with USD 3 million in additional funds to enable the activity to operate during the 18/19 growing season. As is evident from the above description, major changes have occurred to MISST management and also to programming content, both of which have affected performance, as has irregular funding. 41 Consortium Structure and Management. The four MISST consortium partners are independent members of CGIAR and each report to a board of trustees, not to a central CGIAR leadership. Management activities of each center are the responsibility of the country manager and headquarters staff, making for a lengthy decision-making process within each consortium member. Nevertheless, once the set-up issues were resolved the consortium has worked well and the reporting and finance issues within the consortium have been largely resolved. The CMO was initially established to manage the consortium but its main role has been to prepare and submit consolidated financial and technical reports. The Consortium Advisory Committee was set up to coordinate the activities of MISST and USAID but has met 4 times in 3.5 years. Given these circumstances, it is clear that more frequent meetings could have improved coordination among MISST members. Funding. Funding has been irregular and distributed funds were less than planned. Only USD 12.9 million has been received by the activity to date, the last payment being made in mid-2016. The activity was notified in 2017 of a reduction in funding but it is still not clear how much the cut will be and when the next payment will be received. This uncertainty, together with an initial misunderstanding about the USAID procedures for disbursement, make planning very difficult for MISST and its constituent centers. MISST reduced its activities to 5 districts for the 2017/2018 season and CIP has stopped work altogether until further funding is available. Coordination of Program Activities. Despite the numerous challenges, there were significant benefits in operating as a consortium rather than as separate entities. Working groups were established and managed by centers and the CMO to address consortium-wide issues. These groups utilized the specific expertise of the leading partner and ensured that a common approach to each issue was maintained across the consortium. The working groups covered; Seed Systems, Value Chains, Gender Inclusion, Nutrition and Consortium issues and M&E. One center coordinated MISST activities in each district and Mega demonstrations, program reviews and project reporting responsibilities were all shared among consortia members. The development of a single MISST reporting system avoided double counting of beneficiaries across members. Finally, due to the consortium coordinating activities, consortium members were able to work together in more ad hoc ways to support each other’s activities. Ultimately the benefits of working as a consortium have outweighed the complications with setting it up. Recommendations Use of “Lessons Learned”. The administrative, financial, and reporting problems encountered in setting up MISST and operating the consortium was quite severe. Use of “lessons learned” by USAID and the CGIAR centers must be factored into future programming to avoid the difficulties encountered during MISST to ensure more efficient implementation. Some activities should be initiated before the end of MISST, and others activities will need to be included in a follow-on project (or both), implemented by the CG centers or by other USAID projects. In particular the following issues should be considered:  Consortium Structure Assessment. If USAID believes the Consortium approach and the CMO model should be considered for a future project, a review panel or working group should be formed to assess the best way forward to make it truly effective. Lessons learned from MISST should be highlighted and recommendations made to create an effective and efficient operational and management model.  Consortium Advisory Committee (CAC). Regular meetings with required attendance (including by video-link) is recommended if the Consortium model is being considered in the future. Improved communication, in terms of frequency, transparency and content should be agreed upon and established, including proper protocol to minimize disruptions. 42  Project management by USAID. USAID should improve its project management. In particular it should ensure that all changes to the program are fully discussed with the activity, clearly documented and communicated to the activity in writing, work plans and other administrative issues should be processed more quickly to allow the program to run smoothly. Financial issues relating to budget changes and timing of disbursements need to be more clearly communicated with the activity. Question 4. To what extent will the outcome of MISST interventions likely result in improved seed production, quality and productivity? Conclusions A wide assortment of beneficiaries were interviewed during the evaluation process, and nearly all respondents noted that farmers were keen to participate in the program, especially when the new varieties and technologies gave better results compared to those traditionally planted. Farmers wanted to adopt improved varieties and practices provided that inputs were available, affordable and that the new practices would not have other serious negative consequences. Even in drought-affected demonstrations, farmers were able to compare their traditional seeds, technologies to those under MISST and were eager to acquire the improved varieties and technologies. Adoption of OFSP has been most rapid and strong and is expected to continue after MISST ends. CIMMYT beneficiary monitoring and evaluation tracking through agro-dealers showed that 4,817 farm families accessed and bought seed of DTM varieties either using vouchers from FISP or cash for planting in 2017/18. Adoption and use of improved soybean seed were demonstrated by the fact that MISST farmers increased the area planted with improved seeds from 114,000ha to 188,000ha in two years. New sorghum and millet varieties have considerably shorter growing periods and are not as tall as old varieties, making them easier to grow and harvest. Due to the poor performance of maize in the recent dry seasons it is expected that the new crops will be adopted; households accepting the dietary change as a trade-off for food security. Acceptance and plantings of improved groundnut and pigeon pea were good but less spectacular than for other crops. Recommendations Maintenance Breeding Program. Establish a maintenance-breeding program for production of early generation seed. At present there is no facility in Malawi capable of doing this task for publicly bred varieties. Options that might be considered are to contract the work of basic seed production to an outside agency, a seed company such as QualiBasic Seed, Ltd., or to create a small technical agency within DARS with the necessary financial support and linkages (or franchise) to one of those agencies. Strengthen Para Seed Inspector System. Continue strengthening of the PSI system with support for refresher training and, more generally, by following the recommendations made in the recent PSI impact assessment. New Seed Varieties. Recommend the multiplication and commercialization of varieties that are not yet commercially available to farmers, especially several new groundnut and three OFSP varieties. Support for DARS. Provide support on policy and implementation related to seed certification (digitization of records), variety registration, and trade harmonization under SADC and COMESA initiatives. These initiatives should complement and not duplicate the interventions of other donors. 43 Support for DAES. At project outset, recommend that all program activities within target districts are clearly understood, agreed upon and coordinated with DAES activities. Future projects must ensure that government staffs have the resources to visit, supervise and engaged in project activities. Question 5. How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? Conclusions In general, the cooperation with other partners appeared to be healthy and to benefit both parties. MISST partnership mechanisms include sub-grant agreements and, where there is no financial commitment, through an MOU. All partners interviewed including government, seed companies, agrodealers, vine multipliers and farmer organizations reported that MISST worked well with them and that technical support from MISST was considered very valuable. During discussions with DADOs it was noted that some MISST components did a better job coordinating with district government agencies than others, with CIP rated highly in this regard. On the other hand, it was also noted during interviews that some DADO’s are very proactive with MISST seeking close collaboration, whereas others are more passive with implementation activities. The transfers of government staff also compounded partnerships in some districts MISST has also coordinated with five other USAID projects funded through FTF or Food for Peace (FFP). The level of cooperation with each has varied substantially according to the level of mutual benefits received. Cooperation with UBALE was slow to get going after the extension of MISST to the southern districts. In the first year the cooperation was unsuccessful due to a shortage of time to plan the actions and a lack of communication, collaboration was much more successful in the second year. PERFORM operates in the primary ZOI and introduces conservation agriculture and OFSP through its agriculture component. All 4 crop components of MISST have worked closely and successfully with the Njira project which works to promote food security in two ZOI. Ag Div and MISST collaborate closely on OFSP but much less on groundnut and soybean despite their common interest. SANE supports linkages between research and extension and cooperated with MISST to maximize the value of the mega-demonstrations in 2016/7. Recommendations Study to Identify Funding Sources for SMEs. The lack of credit facilities is a major constraint to small business development. USAID has an active credit facility Development Credit Authority (DCA), which uses risk￾sharing agreements to mobilize local private capital. Currently it is not widely used and there are probably other sources of soft loans available from donors and development banks. A study would help to identify potential funding sources and key constraints that currently exist. The study should also propose realistic solutions within the existing policy framework and the current economic environment. Aflatoxin & Aflasafe Conclusions The Aflasafe project has required a highly integrated and collaborative approach to achieve its objectives. In fact, without this integrated approach through government and private sector partners, achieving the final objectives would be impossible. Consequently, IITA worked closely with all line Ministries to implement the Aflasafe project – including DAES to select participating communities and farmers for setting up trials; Ministry of Health and Ministry of Trade and DARS to assist in the testing and identification of regulatory issues related to the testing and development of Aflasafe. In addition, IITA organized a national technical working group that meets three times yearly to discuss on-going issues related to product testing, development and government registration. The consortia also established a 44 multi-stakeholder platform (steering committee), including private sector partners through the assistance of the Association of Commerce & Industry. At the testing level, farmers have been intimately involved in various research trials, and are now aware of aflatoxin issues in maize and groundnuts and the important role of Aflasafe in mitigating the effects of aflatoxin. Recommendations Aflatoxin Lab Support. Continued financial, human resource, and management support to the aflatoxin laboratory so that it becomes economically sustainable. Although “fees for service” should be considered as a viable income-generating component, this will not be sufficient by itself; so additional “hard sources” of government funding from DARS will be necessary, must be actively pursued and acquired. Aflasafe Commercial Development. The commercial development of Aflasafe, possibly in conjunction with a business development project, must continue to be developed and pursued. Although IITA has been working with public and private sector entities to develop a systematic and comprehensive way forward, it is still in its early stages. A formal Steering Committee or Working Group consisting of key public and private sector players should be established. A detailed business plans, including timelines, key participants, financial and market-based objectives, critical project activities, and key milestones should be incorporated into a formalized document that can be agreed upon, funded, and implemented. 45 ANNEX 1. EVALUATION STATEMENT OF WORK Mid-term Performance Evaluation of USAID’s Malawi Improved Seed System and Technologies (MISST) Activity United States Agency for International Development (USAID)/Malawi, Office of Sustainable Economic Growth (SEG) December 2017 POINT OF CONTACT AOR, MISST Project Eric Witte Agricultural International Affairs Specialist Agriculture Research and Policy Office Bureau for Food Security US Agency for International Development RRB 3.7-054 USAID/BFS/ARP 1300 Pennsylvania Ave., NW Washington, DC 20523 Carter Hemphill COR, Monitoring, Evaluation, and Learning Support (MELS) Project Office of Sustainable Economic Growth USAID/Malawi Tel: 265 (0) 177-2455, Ext. XX E-mail: chemphill@usaid.gov Project Information Project Name Malawi Improved Seed System and Technologies (MISST) Award Number AID-BFS-G-11-00002 Original Project Dates December, 2014−June, 2019 Original Funding $24.7 m Implementing Partner ICRISAT, IITA, CIMMYT & CIP 46 Contents I. Background II. Program Components and Description III. Evaluation Purpose IV. Evaluation Questions V. Evaluation Methodology VI. Existing Sources of Information VII. Deliverables VIII. Evaluation Team Composition IX. Level of Effort X. Scheduling and Logistics XI. Budget 47 I. Background The Feed the Future Malawi Improved Seed Systems and Technologies (MISST) project is a 4.5-year Project implemented by a Consortium of four CGIAR institutions. The centers include: The International Potato Center (CIP) promoting the orange fleshed sweet potato (OFSP) value chain; The International Maize and Wheat Improvement Center (CIMMYT) promoting Drought tolerant maize (DTM) value chain; The International Crops Research Institute for the Semi-Arid Tropics (ICRISAT) which is promoting groundnuts and Pigeon Peas, The International Institute for Tropical Agriculture (IITA) promoting soybean and aflatoxin mitigation technologies. ICRISAT hosts the Consortium secretariat on behalf of the four CGIAR institutions. A brief on the proposed value chains and their potential impacts is presented below. 1. Maize represents 90% of the total cultivated land, is grown by 97% of farm households, and accounts for 60% of total calorie intake and 48% of protein consumption. It is also a major source of income for most households. Whereas adoption of improved maize varieties is improving, many of them are insensitive to drought. The CIMMYT and partners have developed Drought Tolerant Maize that are being deployed for climate change proofing of Malawi’s maize-based cropping systems. 2. Groundnut is an important legume in smallholder farming systems of Malawi and is the highest paying legume crop that supports diverse pathways out of poverty. Approximately 20% of smallholder farmers in Malawi grow groundnuts out of which 25% grow improved groundnut varieties. 76% of those farmers that grow improved varieties experience enhanced food security and high per capita consumption (7.3 kg/capita). The crop also generates about US $ 6 million annually. 3. Pigeon pea is a crop of increasing importance in Malawi with potential to climate change proof farming communities in dryer districts of the ZOI. Pigeon pea is drought tolerant, with a deep tap root system that can fix nitrogen and improve soil fertility, as well as provide affordable dietary protein and calories. It is a shrub and therefore can reduce carbon footprint (mitigate climate change) and provide fuel wood and fodder. ICRISAT will deploy varieties with a yield advantage of 50% and higher processing ability to unlock productivity, nutrition and income opportunities. 4. Soybean is a crop of increasing demand in Malawi, currently estimated at 120,000 tons, with a potential demand of 175,000 tons/annum. This demand could rise to 250,000 tons/annum by 2020. Currently demand outstrips supply. IITA in partnership with the Department of Agricultural Research Services (DARS) have highly productive and resilient varieties. The aim in this project is to deliver these varieties to farming communities. 5. Sweet potato is another focus project crop. The Demographic and Household Survey of 2010, shows that 47% and 63% of children under the age of five are stunted and anemic respectively. These dismal statistics point to the limited access to critical nutrients such as vitamin A. The orange-fleshed sweet potato (OFSP) is a rich source of pro-vitamin A. In this project, orange-fleshed sweet potato will be used to perform multiple functions of supplying calories, delivering vitamin A to vulnerable communities, and generating incomes. In addition, the activity seeks to mitigate Aflatoxins, which are naturally occurring highly carcinogenic toxins produced by fungi of the genus Aspergillus species. In Malawi, groundnut and maize are often contaminated with aflatoxins and when consumed, affect people’s health and growth. In this project, IITA will test Aflasafe, an aflatoxin mitigation technology to find solutions for addressing this deadly threat to the country and region. The activity is implemented in Malawi’s FTF primary Zone of Influence (ZOI) which covers seven districts namely: Balaka, Dedza, Lilongwe rural, Machinga, Mchinji, Mangochi and Ntcheu. The activity pools 48 resources and expertise to strategically assist Malawi to effectively unlock the potential of the targeted crops, to transform smallholder agriculture, and contribute to the attainment of the Country and USAID/Malawi’s goal of improving the quality of life of Malawians. II. Program Objectives, Components and Description MISST has four approved Strategic Objectives given as: ● Improved functionality of input and output markets of selected value chains: groundnuts, pigeon pea, soybean, drought tolerant (DT) maize and orange-fleshed sweet potato (OFSP). ● Capacity of stakeholders in agricultural sector strengthened to deploy and promote new crop varieties and their allied technologies. ● Adoption and uptake of improved technologies enhanced in the ZOI. ● Increased resilience of vulnerable households and rural communities in the FTF zone of influence. By the end of the activity in June 2019, it is expected that 268,000 households (including multiple beneficiaries) will have directly benefited from the activities of the activity. III. Evaluation Purpose The primary purpose of this mid-term performance evaluation is to assess the effectiveness of the MISST design and implementation processes. This evaluation will also inform management what course corrections, if any, are needed. Additionally, through this evaluation, USAID seeks strategic and programmatic options for future engagement to support seed production and dissemination in Malawi. There are three major objectives of this evaluation. They are as follows: 1. To review, analyze, and evaluate the effectiveness of the MISST activity in achieving program objectives and contributing to USAID/Malawi’s efforts to improved agricultural productivity in Malawi; 2. Evaluate major constraints in achieving expected project results; and 3. Provide specific recommendations and lessons learned on strategies and approaches USAID/Malawi should pursue in the remaining period of MISST’s implementation and for future seed production and distribution activities. IV. Evaluation Questions The contractor’s evaluation of MISST must be comprehensive. Whenever possible, the evaluation team should ensure that data is gender disaggregated and subjected to gender analysis. The contractor shall assess MISST performance and achievements against the performance indicators, targets, reporting requirements, outputs and deliverables described in the approved M&E plan, Annual Work Plans and the agreement. In doing so, the team should pay particular attention to required FTF indicators. While this evaluation is not a data quality audit, the contractor shall assess and describe the quality and use of performance monitoring data and information generated during MISST implementation. Evidence-based conclusions and recommendations on how MISST used performance monitoring data and information in performance management are required when describing MISST effectiveness in component management. The following evaluation questions should be addressed: 1. To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? The response to this question should (at a minimum) address the four anticipated results and identify MISST’s areas of weakness and strengths in addressing these results. 2. To what extent has MISST improved the functionality of input and output markets of its focus value chains? 3. What challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? 49 Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. 4. To what extent will the outcome of MISST interventions likely result in improved seed production, quality and productivity? 5. How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? V. Evaluation Methodology The Contractor must propose a robust evaluation methodology to answer the key evaluation questions identified above. The evaluation methodology must follow a mixed–method approach using both quantitative and qualitative techniques in gathering reliable data and valid evidence of project outcomes or impacts including review of relevant MISST, USAID, GOM, FAO, and other secondary data sources. The contractor must develop a Mixed Methods Evaluation Design Matrix that will detail the data source, data collection methods, data collection instruments and analysis of data to answer each of the evaluation questions. By using a mixed-method approach, the evaluation team is expected to gain insight on the impact of MISST project activities (mostly from quantitative data collected by the activity and others) and the processes (mostly qualitative information provided by the activity staff and key informants) that lead to those impacts. It should generate gender-disaggregated data and reflect attention to gender relations such as the participation of women in training. Sequential and iterative approaches should be used to integrate the mixture of methods at various stages of the evaluation. The evaluators should utilize several different, yet complementary and inter-related forms of gathering information/data such as those described below. The contractor is expected to utilize its expert judgment and evaluation best practices in selecting which methodological components to include in the evaluation design. Document Review. Evaluation team members will review documents throughout the evaluation process including program reports, relevant studies to ensure that comprehensive and grounded best practices will be identified. Key Informant Interviews. The team will conduct one-on-one interviews with a variety of stakeholders including the leads for each of the participating research centers, most relevant GOM ministries and agencies, bi- and multilateral development partners supporting development and seed production and dissemination activities in Malawi. This will be done administering a semi-structured or structured questionnaire. Self-assessment. The IPs will respond to a self-assessment either through a questionnaire or standard interview checklist put together by the evaluation team and approved by USAID before use. Expert Opinion Survey. Utilizing expert opinion is a technique is used increasingly in evaluations. The evaluation team, with approval of USAID, can apply this method as well. Focus Group Discussions (FGDs). FGDs (small group of 6 to 10 people) will be used to lead open discussion through a skilled moderator to gather semi-structured qualitative data. The preselected participants will discuss issues and concerns based on a list of key themes drawn up by the moderator. No more than 10 questions will be addressed by a group. These sessions will encourage free flowing discussion about the activity. Mini Survey. This type of survey is small (30-40 participants) and can be performed rapidly in the field without analytical software such as SPSS or a large questionnaire. The sample size is not statistically 50 significant; however, this type of analysis can be used to triangulate with other methods. Because of the small sample size this type of survey can be implemented quickly when time and resources are constrained. All the methodological strengths and weaknesses should be explicitly described in the evaluation report. Prior to the start of the evaluation, the evaluation team shall meet with EG office to refine the evaluation methodology and address any other concerns the EG office may have. Limitations to Data Collection. Due to the compressed timeline of the evaluation, the evaluation team will not have adequate time to conduct large surveys that are statistically significant. Wherever possible the team will triangulate findings using multiple sources. VI. Deliverables All deliverables are internal to USAID, MELS and the Evaluation Team unless otherwise instructed by USAID. Evaluation deliverables include: In-brief Meeting with USAID/Malawi. Within two working days of international team members’ arrival in Malawi; Inception Report: At the In-brief meeting, the Evaluation team will provide an Inception report that will outline key aspects of the Evaluation, including the Draft Work Plan, Methodology, Evaluation Design Matrix, and proposed Data Collection Instruments (e.g., interview guides). See more detail below. ● Work Plan. The Contractor will prepare a detailed work plan that includes task timeline, methodology outlining approach to be used in answering each evaluation question, team responsibility, document review, key informant and stakeholder meetings, site visits, survey implementation, travel time, debriefings (for USAID, implementing partner and, if decided, the GOM), draft and final report writing. The work plan will include a data analysis plan. The work plan will be submitted to the MELS COR and MISST AOR at USAID/Malawi for approval no later than the fifth day after the Evaluation team arrives in Malawi. ● Evaluation Design Matrix. A table that lists each evaluation question and the corresponding information sought, information sources, data collection methods, data analysis methods, and limitations and assumptions. The matrix should be finalized and shared with USAID/Malawi before evaluation field work starts. It should also be included as an annex in the evaluation report. ● Data Collection Instruments. Development and submission of data collection instruments to USAID/Malawi during the design phase and after the evaluation is completed; Regular Updates with USAID. The Evaluation Team Leader will brief the MELS COR and MISST AOR on progress with the evaluation on at least a weekly basis, in person or by electronic communication. Any delays or complications must be quickly communicated to USAID/Malawi as early as possible to allow quick resolution and to minimize any disruptions to the evaluation. Emerging opportunities to strengthen the evaluation should also be discussed with USAID/Malawi as they arise. Out-briefing with USAID. The Evaluation Team will present the major preliminary evaluation findings to USAID/Malawi through a PowerPoint presentation before the team’s departure from country. The debriefing will include a discussion of project achievements and issues as well as any preliminary recommendations. The team will consider USAID and MELS comments and incorporate them in the Draft Evaluation Report. Stakeholders Workshop. The team will present the major findings from the evaluation to key stakeholders (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from the country. The debriefing will include a discussion of achievements and activities only, with no recommendations for possible modifications to project approaches, results, or activities. 51 The team will consider key stakeholder comments and incorporate them appropriately in drafting the evaluation report. Draft Evaluation Report. A draft report on the findings and recommendations should be submitted to USAID/Malawi and MELS 10 working days after departure of international team members from Malawi. The report must be no more than 35 pages in length (excluding annexes) and comply with the Checklist for Assessing USAID Evaluation Reports (see annexes). The written report should clearly describe findings, conclusions, and recommendations. The draft report must be of high quality with no grammatical errors or typos. A report is high quality when it represents a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the activity, what did not and why. The draft report must have well-constructed sentences that are presented in a way that clearly presents findings, conclusions and recommendations. The report should answer all the evaluation questions and the structure of the report should make it clear how the questions were answered. The draft report must meet the criteria set forth under the Final Report section below. USAID will provide comments on the draft report within 10 working days of submission. Final Evaluation Report. The Contractor will a Final Evaluation Report that incorporate Mission comments and suggestions no later than five working days after USAID/Malawi provides written comments on the Draft Evaluation Report. The format of the final report is provided below. The report will be submitted in English, electronically. The Contractor must ensure that Appendix I of the USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report is followed. The final report should at a minimum meet the following criteria to ensure the quality of the report: ● The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the activity, what did not and why. This includes a short paragraph of key takeaways from the evaluation. ● Evaluation report shall address all evaluation questions included in the scope of work. ● The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the MELS COR and MISST AOR. ● Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. ● Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). ● Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. ● Sources of information need to be properly identified and listed in an annex. ● Recommendations need to be supported by a specific set of findings. ● Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. The format of the final evaluation report should strike a balance between depth and length. The report will include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, findings, conclusions, lessons learned and recommendations. The report should include a brief paragraph in the executive summary of “key takeaways” from the evaluation. Where appropriate, the evaluation should utilize tables and graphs to link with data and other relevant information. The report should include, in the annex, any dissenting views by any team member or by USAID on any of the findings or recommendations. The report should 52 not exceed 35 pages, excluding annexes. A second version of this report excluding any potentially procurement-sensitive information will be submitted (also electronically, in English) to Development Experience Clearinghouse (DEC) and for dissemination among implementing partners and other stakeholders. All quantitative data, if gathered, should be (1) provided in an electronic file in easily readable format; (2) organized and fully documented for use by those not fully familiar with the activity or the evaluation; (3) owned by USAID and made available to the public barring rare exceptions. A thumb drive with all the data could be provided to the MISST AOR and MISST Chief of Party. The final report will be edited and formatted by the Evaluation Team and reviewed by MELS and provided to USAID/Malawi 10 working days after the Mission has reviewed the content and approved the final revised version of the report. VII. Evaluation Team Composition The team will include two international and two local consultants. The former should include specialists with the following areas of expertise: project evaluations and assessments (ideally agriculture and economic development evaluations), seed production, seed commercialization, economic growth, technology dissemination and co-governance principles. The local consultants should have a background in agriculture and the seed sector in Malawi. Team Leader/Seed Sector Expert (International). This expert will serve as Team Leader for the MISST evaluation. The Team Leader will provide overall leadership for the team, and s/he will finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations into a high￾quality document. S/he will lead the preparation and presentation of the key evaluation findings and recommendations to the USAID / Malawi team and other major partners. The TL should have the following attributes/qualifications and experience: ● S/he should have a postgraduate degree in an appropriate field, with area of expertise in seed systems strengthening, agricultural value chains, seed policy, agronomy, and agricultural economics, with at least 10 years of international experience in the seed sector with experience conducting USAID evaluations, ideally for seed sector, agricultural input projects. Relevant experience in Malawi or Eastern/Southern Africa preferred. ● S/he should have extensive experience in conducting quantitative and qualitative assessments. ● The Team Leader must be familiar with USAID regulations and systems including performance monitoring, gender policies and guidance, project management, budgeting and financial analysis, and reporting. ● Experience in international donor development program management and overseeing multiple program areas simultaneously is preferred. ● Should be experienced in preparing documents that are objective, evidence-based, and well organized. ● Excellent oral and written skills in English are required. Senior Agricultural Expert (International). The Senior Agricultural Specialist or Agricultural Economist will be responsible for assessing the effectiveness of project implementation according to the M&E plan, while addressing the evaluation questions above. S/he will provide technical assistance and analytical ideas with respect to the seed sector and challenges facing the agricultural input sector in Malawi, as well as the capacity constraints that impede seed development and commercialization at both the local and national level. 53 S/he will participate in team meetings, and in the absence of the team leader, lead key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The Senior Agricultural Expert will have: ● A postgraduate degree in agriculture, agriculture economics or a related field. ● A minimum of 8 years of international field experience in agricultural development with an understanding of seed sector including seed commercialization in southern Africa. ● M&E experience with at least one similar USAID project, or that of another international donor, and have familiarity with USAID regulations and systems. Agriculture Specialist (Local). The local Agriculture Specialist will have experience in the Malawi agricultural sector preferably in the seed sector and understand the needs of the Malawi smallholder farmer. S/he will participate in team meetings, key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he also will participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The local Agricultural Specialist should: ● Have an advanced degree in agriculture or related field. ● Be a Malawi national with at least 5 years of field experience in evaluation, preferably with some sectoral experience with agricultural and improved seed. Evaluation Specialist (Local). The local Evaluation Specialist will participate in team meetings, key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he also will participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The local Evaluation Specialist should: ● Have a degree in monitoring and evaluation, international development, economics, agriculture, or a related field. ● Be a Malawi national with at least 2 years of field experience in evaluation, preferably with some sectoral experience with agriculture and improved seed. Logistics Specialist (Local). The local Logistics Specialist will be responsible for providing program logistics support, arranging appointments and maintaining the schedule, providing interpretation and translation, and assisting with the preparation of project reports, as needed. Conflict of Interest. All evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the activity being evaluated. USAID will provide the conflict of interest forms. See Annex 1 for the Template. VIII. Existing Sources of Information The evaluation team should consult a broad range of background documents apart from project documents provided by USAID /Malawi These should include, but are not limited to, documents on the agricultural context in Malawi, the seed commercialization process, requirements for bringing seed to market and improved seed varieties both in and outside of Malawi. The evaluation team also should review relevant GOM national strategies and policies, as well as relevant projects and strategies of other aid agencies active in Malawi. USAID, MELS, and the MISST project will provide the assessment team with a package of briefing materials, including: ● The agreement for MISST activity; ● M&E plan for MISST; ● MISST Baseline Survey; 54 ● Annual Beneficiary Survey reports ● Project quarterly and annual reports, work plans and management reviews developed as part of routine monitoring; ● Training reports; ● DQA reports; ● USAID/Malawi Country Development Cooperation Strategy 2014-19 (Public version); ● USAID Malawi DO2 PMP; ● MISST Indicator Tracking Table (PITT); ● Training & Beneficiary database; ● USAID Evaluation Policy; ● Checklist for Assessing USAID Evaluation Policy IX. Level of Effort (LOE) of Study Team by Task Deliverables Below is an estimate of the evaluation level of effort (LOE). Level of Efforts of Team Members by Task Deliverables Task/Deliverable Duration / LOE in Days Team Leader/Se ed Specialist Sr. Agricultu ral Expert Local Agricultural Specialist Evaluation Specialist Review background documents and home-based preparation work 5 5 2 2 Travel to Malawi 2 2 0 0 Team planning meeting and meeting with USAID 2 2 2 2 Development of Evaluation Work Plan (concurrent with document review and initial meetings) 2 2 2 2 Development of data collection instruments 2 2 2 2 Information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits 20 20 20 20 Discussion, analysis, and preliminary draft evaluation report in country including discussion with USAID 5 5 5 5 Debrief meetings with USAID (preliminary draft report due to USAID) 1 1 1 1 Debrief meetings with key stakeholders 1 1 1 1 Team Leader meets with Technical Specialists and USAID to synthesize findings/discussion 1 1 1 1 Depart Malawi/Travel to U.S. 2 2 0 0 Finalization of draft and internal review (IBTCI) remote 4 4 2 2 USAID provides comments on draft report 0 0 0 0 Team revises draft report and submits final to USAID (out of country) 10 5 5 5 USAID completes final review 0 0 0 0 Editing and formatting of report 0 0 0 0 Total 57 52 43 43 55 X. Scheduling and Logistics Funding and Logistical Support. USAID/Malawi’s MELS project will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate local staff. It will take care of arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. The MELS Logistics Specialist will arrange field visits, local travel, hotel, and appointments with stakeholders. Scheduling (see above table). Include a tentative evaluation schedule showing all the timings for the key tasks to be undertaken as part of this evaluation. Have a sample that I can share with you. Work is to be carried out over a period of approximately 10 weeks. At this point in time, we anticipate that the evaluation would begin on/about February 5th, with work in Malawi commencing on/about February 15th, field work completed on/about March 10th and final report and close out before the end of March. A six-day workweek is authorized for the evaluation team while in Malawi, however, no overtime or premium pay is authorized. The evaluation team will submit a work plan with timeline as part of the evaluation methodology proposal and develop a Gantt chart displaying the time periods during which activities occur. Team mobilization will include: travel approval; airline tickets; visa; lodging; work facility and vehicle transport arrangements; dates for meetings with USAID/Malawi EG staff and key contacts; in-country travel agenda; and accommodations. 56 ANNEX 11. DATA COLLECTION TOOLS a) Evaluation questions and sub- questions The Statement of Work (SOW)) defines five evaluation research questions (EQs) to guide the design, fieldwork and analysis of the mid-term evaluation. In addition, the IBTCI team developed additional sub￾questions (SQs), to better address specific topics to ensure useful evidence-based findings and conclusions were obtained. The following are the principal five evaluation questions that were addressed together with IBTCI’s sub-questions. Evaluation Question 1: To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? The response to this question should address the four anticipated results and identify MISST’s areas of weakness and strengths in addressing these results. ● Sub-Question 1: Was the activity design appropriate to achieve the overall objectives – and has this changed? ● Sub-Question 2: To what extent has MISST met its targets for achieving Objectives 1-4? ● Sub-Question 3: How have project activities contributed to meeting the overall objective and what have been the key obstacles and challenges. ● Sub-Question 4: Has the design and implementation been appropriate to improve resilience of vulnerable households and rural communities? Evaluation Question 2: To what extent has MISST improved the functionality of input and output markets of its focus value chains? ● Sub-Question 5: How and where has the activity improved the performance and functionality of input markets for seed and new technology? What specific interventions were used and how has this benefited smallholder farmers? ● Sub-Question 6: Output markets for seed and new technology? What specific interventions were used and how has this benefited smallholder farmers? Evaluation Question 3: What challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. ● Sub-Question 7: What challenges has the activity faced with regard to the technical and financial situation of beneficiaries and partners and how has it responded to them? ● Sub-Question 8: What challenges has the activity faced with regard to the policy environment and the ability of government to participate in the activity and how has it responded to them? ● Sub-Question 9: What challenges has the activity faced with regard to the structure, management and resources of the consortium, and how has it responded to them? ● Sub-Question 10: How well have consortium members coordinated their activities and how effective has the consortium approach been in achieving project goals and objectives? Evaluation Question 4: To what extent will the outcome of MISST interventions likely result in improved seed production, quality and productivity? ● Sub-Question 11: Have the direct beneficiaries accepted and adopted the improved seed products and technologies? If there has been resistance, why, and how has the activity addressed this? 57 ● Sub-Question 12: How sustainable will the improved seed provision be in terms of new varieties and good quality, affordable seed and improved technology? ● Sub-Question 13: What impact is the activity likely to have on grain production in the ZOI through introduction of new varieties and technology? ● Sub-Question 14: What is likely sustainability of the aflatoxin laboratory and the longer-term uptake of Aflasafe? ● Sub-Question 15: What long term effect will the activity have on the resilience of vulnerable households and rural communities in the ZOI? Evaluation Question 5: How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? ● Sub-Question 16: How successful were the activity partnerships with implementing organizations including private sector seed companies, input supply firms, government extension offices and other NGOs? ● Sub-Question 17: How successfully did the activity coordinate and cooperate with other non-partner organizations, such as projects, government programs and seed companies? 58 b) Data Collection Matrix MISST Data Collection Matrix – Assessing appropriateness of project design based on achievement of goals and objectives Sub-questions Type of Answer or Evidence Required 1 Data Collection Methods 2 Data Sources Sampling or Selection Approach Data Analysis Methods Evaluation Question 1: To what extent is MISST on track, in terms of progress and outcomes, to meet its overall objective? In addition, address MISST’s areas of strengths and weaknesses throughout project implementation of the five value chains (GN, PP, SB, DTM and OFSP). SQ1. Was the activity design appropriate to achieve the overall objectives and has this changed? SQ2. To what extent has MISST met its targets for achieving Objectives 1-4? SQ3. How have project activities contributed to meeting the overall objective and what have been the key obstacles and challenges. SQ4. Has the design and implementation been appropriate to improve resilience of vulnerable households and rural communities? SQ1. Comparison of program documents and work plans against needs of Malawi as expressed by stakeholders. SQ2. Comparison of results against targets for quantitative indicators. SQ3. Quantitative and qualitative information and feedback from beneficiaries and partners (DAES, DARS, MOAIWD, RTCDT, and other NGOs SQ4. Analysis of design, work plans, results in quantitative and qualitative form for resilience activities. 1. Review of annual and quarterly reports. 2. Gather and analyze data from secondary sources including Y1, Y2, Y3 MEL reports. 3. Semi-structured Interviews with MISST senior project management (CIP, CIMMYT, IITA, ICRISAT) 4. Primary data collection through KII field interviews in 7 districts with partner organizations and target beneficiaries. 1. Quarterly and annual reports. 2. Project description reports (training assessments). 3. USAID CDCS 4. USAID PAD 5. Project inception report 6. Project M&E Plan 7. Primary data collection through key Informant interviews and or mini-surveys. 8. Possible Focus Group discussions. 1. Data from secondary sources must be applicable to current total period of performance; 2. KII selection and number should balance participants with opposing roles and responsibilities (“Mirror Image approach) to ensure triangulation and clearly identify risks of bias or undue influence. 3. Gender￾disaggregated data if appropriate. 4. Consider Group Interviews to enable a better stakeholder balance in the feedback collected. 1. Review secondary sources to measure progress toward meeting PMEP targets for FY15, 16, and 17 indicators; 2. Conduct content reviews of technical assistance programs to ensure, quality, appropriateness and sufficiency. 3. Compare primary data acquired during the evaluation with secondary data sources to ensure continuity. 4. When appropriate, analyze gender relations and their impacts participation in training, research, and/or other roles. 59 MISST Data Collection Matrix – Assessment of improvements in input and output markets in target districts Type of Answer or Evidence Required 1 Data Collection Methods 2 Data Sources Sampling or Selection Approach Data Analysis Methods Evaluation Question 2: To what extent has MISST improved the functionality of input and output markets of its focus value chains? SQ1. How and where has the activity improved the performance and functionality of input markets for seed and new technology? SQ2. What specific interventions were used and how has this benefited smallholder farmers? SQ3. How and where has the activity improved the performance and functionality of output markets for seed and new technology? SQ4. What specific interventions were used and how has this benefited smallholder farmers? SQ1. KII with input supply firms and target beneficiaries. Comparisons of inputs from Y1 to Y3. SQ2. Documentation of changes in functionality of input supply markets as perceived by input supply recipients. SQ3. Comparison of results against targets for quantitative indicators and feedback from stakeholders and observation of evaluators for qualitative targets. SQ4. KII with stakeholders and direct project beneficiaries to identify & document impacts. 1. Review of annual and quarterly reports. 2. Gather and analyze data from secondary sources including Y1, Y2, Y3 MEL reports. 3. Primary data collection through semi-structured Interviews with MISST senior project management (CIP, CIMMYT, IITA, ICRISAT) 4. Primary data collection through KII field interviews in seven (7) districts with partner organizations and target beneficiaries. 1. Data from secondary sources must be applicable to current total period of performance; 2. KII selection and number should balance participants with opposing roles and responsibilities (“Mirror Image approach) to ensure triangulation and clearly identify risks of bias or undue influence. 3. Gender-disaggregated data if appropriate. 4. Consider Group Interviews to enable a better stakeholder balance in the feedback collected. Review secondary sources to measure progress toward meeting PMEP targets for FY15 & 16 indicators; 2. Conduct content reviews of technical assistance programs to ensure, quality, appropriateness and sufficiency. 3. Compare primary data acquired during the evaluation with secondary data sources to ensure continuity. 4. If appropriate, analyze gender relations and their impacts participation in training, research, and/or other roles. 60 MISST Data Collection Matrix – Identifying project challenges and effectiveness of MISST Consortium structure Type of Answer or Evidence Required 1 Data Collection Methods 2 Data Sources Sampling or Selection Approach (if Needed) Data Analysis Methods Evaluation Question 3. What primary challenges has MISST faced in implementing its activities and how has MISST responded to those challenges? Are there any recommended alternatives to these approaches? Particular attention is necessary on the current consortium structure of MISST and the degree of its effectiveness. SQ1. What challenges has the activity faced with regard to the technical and financial situation of beneficiaries and partners and how has it responded to them? SQ2. What challenges has the activity faced with regard to the policy environment and the ability of government to participate in the activity and how has it responded to them? SQ3. What challenges has the activity faced with regard to the structure, management and resources of the consortium, and how has it responded to them? SQ4. How well have consortium members coordinated their activities and how effective has the consortium approach been in achieving project goals and objectives? SQ1. Key informant interviews with CIP, CIMMYT, ICRISAT and IITA senior management, key partners and target beneficiaries. SQ2. KIIs with CIP, CIMMYT, ICRISAT and IITA senior management, key partners and target beneficiaries. Interviews with key Ministry of Agriculture (MOA) officials. SQ3. Interviews with CIP, CIMMYT, ICRISAT and IITA senior management. KII with key partners. 1. Review of annual and quarterly reports. 2. Gather and analyze data from secondary sources including Y1, Y2, Y3 MEL reports. 3. Semi-structured Interviews with MISST senior project management (CIP, CIMMYT, IITA, ICRISAT) 4. Primary data collection through KII field interviews in seven (7) districts with partner organizations and target beneficiaries. 1. Data from secondary sources must be applicable to current total period of performance; 2. KII selection and number should balance participants with opposing roles and responsibilities (“Mirror Image approach) to ensure triangulation and clearly identify risks of bias or undue influence. 3. Gender-disaggregated data if appropriate. 4. Consider Group Interviews to enable a better stakeholder balance in the feedback collected. 1. Review secondary sources to measure progress toward meeting PMEP targets for FY15 & 16 indicators; 2. Conduct content reviews of technical assistance programs to ensure, quality, appropriateness and sufficiency. 3. Compare primary data acquired during the evaluation with secondary data sources to ensure continuity 4. If appropriate, analyze gender relations and their impacts participation in training, research, and/or other roles. 61 MISST Data Collection Matrix – Degree of MISST success in improving seed production, quality and productivity Type of Answer or Evidence Required 1 Data Collection Methods 2 Data Sources Sampling or Selection Approach (if Needed) Data Analysis Methods Evaluation Question 4. To what extent will the outcome of MISST interventions likely result in improved seed production, quality and productivity? SQ1. Have the direct beneficiaries accepted and adopted the improved seed products and technologies? If there has been resistance, why, and how has the activity addressed this? SQ2. How sustainable will the improved seed provision be in terms of new varieties and good quality, affordable seed and improved technology? SQ3. What direct impact is the activity likely to have on grain production in the ZOI through introduction of new varieties and technology? SQ4. What is likely sustainability of the aflatoxin laboratory and the longer￾term uptake of Aflasafe? SQ5. What long-term effect will the activity have on the resilience of vulnerable households and rural communities in the ZOI? SQ1. Use of KII’s and Focus Group Discussions (FGDs) of project beneficiaries, MOA district extension staffs and key partners. SQ2. Use of KII’s and Focus Group Discussions (FGDs) of project beneficiaries, input supply firms, MOA district extension staffs and key partners. SQ3. KIIs with CIP, CIMMYT, ICRISAT and IITA senior management, key partners and MOA. SQ4. Interview with IITA to determine progress, effectiveness, and future of Aflasafe technology. SQ5. Analysis of project results to date with respect to improvements in smallholder farmer productivity, increased output and use of other resilience activities. 1. Review of annual and quarterly reports. 2. Gather and analyze data from secondary sources including Y1, Y2, Y3 MEL reports. 3. Semi-structured Interviews with MISST senior project management (CIP, CIMMYT, IITA, ICRISAT) 4. Primary data collection through KII field interviews in seven (7) districts with partner organizations and target beneficiaries. 1. Data from secondary sources must be applicable to current total period of performance; 2. KII selection and number should balance participants with opposing roles and responsibilities (“Mirror Image approach) to ensure triangulation and clearly identify risks of bias or undue influence. 3. Gender-disaggregated data if appropriate. 4. Consider Group Interviews to enable a better stakeholder balance in the feedback collected. 1. Review secondary sources to measure progress toward meeting PMEP targets for FY15 & 16 indicators; 2. Conduct content reviews of technical assistance programs to ensure, quality, appropriateness and sufficiency. 3. Compare primary data acquired during the evaluation with secondary data sources to ensure continuity. 4. If appropriate, analyze gender relations and their impacts participation in training, research, and/or other roles. 62 MISST Data Collection Matrix – Extent to which MISST collaborated with partners and effectiveness of collaboration Type of Answer or Evidence Required 1 Data Collection Methods 2 Data Sources Sampling or Selection Approach (if Needed) Data Analysis Methods Evaluation Question 5. How did MISST integrate and work with other partners to achieve its results? To what extent did MISST integration activities promote or hinder attaining of its results? SQ1. How successful were the activity partnerships with implementing organizations including private sector seed companies, input supply firms, government extension offices and other NGOs? SQ2. How successfully did the activity coordinate and cooperate with other non￾partner organizations, such as projects, government programs and seed companies? SQ1. Key informant interviews with CIP, CIMMYT, ICRISAT and IITA senior management. KII’s with private sector seed companies, input supply firms, government extension offices. SQ1. Key informant interviews with CIP, CIMMYT, ICRISAT and IITA senior management. KII’s with private sector seed companies, input supply firms, government extension offices, other NGOs. 1. Quarterly and annual reports. 2. Project description reports (training assessments). 3. USAID CDCS 4. USAID PAD 5. Project inception report 6. Project M&E Plan 7. Key Informant interviews 8. Possible Focus Group discussions. 1. Data from secondary sources must be applicable to current total period of performance; 2. KII selection and number should balance participants with opposing roles and responsibilities (“Mirror Image approach) to ensure triangulation and clearly identify risks of bias or undue influence. 3. Gender-disaggregated data if appropriate. 4. Consider Group Interviews to enable a better stakeholder balance in the feedback collected. 1. Review secondary sources to measure progress toward meeting PMEP targets for FY15 & 16 indicators; 2. Conduct content reviews of technical assistance programs to ensure, quality, appropriateness and sufficiency. 3. If appropriate, analyze gender relations and their impacts participation in training, research, and/or other roles. 63 c) Key Informants Interviewed Date Name Title / Role Organization Place/Location 15th May Mr. Mphatso Dakamau Program Manager MAPAC City Centre, Lilongwe 15th May Mr. Supply Chisi Business Development Officer STAM Area 4, Lilongwe 15th May Lanneck Makawuse EXAGRIS Mchinji Road, Lilongwe 15th May Dr. Innocent Chimimba / Ibrahim Benesi Chief Executive Officer MUSECO Kanengo, Lilongwe 15th May Mr. Frank Masankha Farm Service Coordinator NASFAM City Centre, Lilongwe 15th May Mr. Blessings Mwale Deputy Chief of Party PERFOM City Centre, Lilongwe 16th May Dr Maiden Kumbukani Munthali Chief Executive Officer DEMETER SEED COMPANY Kanengo, Lilongwe 16th May Mr. Edward Kabaghe Executive Director PANNAR SEED COMPANY Kanengo, Lilongwe 16th May Julekha Issat Monitoring and Evaluation Officer PEACOCK SEED COMPANY Area 18A, Lilongwe 16th May Dr. Limbakani Matumba Director of Research and Outreach LUANAR Lilongwe 16th May Renata Krzywon’- Schramm Head of Projects Welt Hunger Area 9, Lilongwe 17th May Daniel van Vugt and Paul Demo MISST OFSP Project Manager and Country Director CIP Area11, Lilongwe 17th May Naomi Kamanga Chief Of Party Thomas Banda Finance Manager CMO Chitedze, Lilongwe Joy Ndalama M &E Specialist￾17th May Willie Kalumula Project Manager CIMMYT Lilongwe(Chitedze) 17th May Mr. Soka Chitaya Project Manager-Cereals and groundnut’s ICRISAT Lilongwe(Chitedze) 17th May Mr. Joseph Atehnkeng Program Manager(Aflasafe) IITA Lilongwe (Chitedze) 17th May Mr Soka and Joseph Atehnkeng Project Manager-cereals and groundnuts and Project Manager-Aflasafe-IITA ICRISAT and IITA Lilongwe (Chitedze) 18th May Mr. Yotamu Hastings District Agriculture Development Officer District Agriculture Office (Lilongwe West) Lilongwe (Old town) 18th May Carl Larkins COP Feed the Future Initiative Lilongwe (Area 3) Lonester Sibande Lead Monitoring and Evaluation 18th May Mr. Jacob Nyilongo Director of Program Development and Learning FUM Lilongwe (Area 4) 64 Date Name Title / Role Organization Place/Location 18th May Mr. Hector Malaidza DARS Lilongwe (Chitedze) 18th May Mr. Sanne Chipeta WE EFFECT City Centre, Lilongwe 18th May Mr. Manuel Milz Program Director (agricultural development Agronomist) Inter aide Area 47, Lilongwe 20th May and 8th June Dr Patrick Okori Country Representative ICRISAT Lilongwe 21st May Mr. Matembo Mtawali Crops Officer(Serials) DAES Mchinji District Mr. Malekano Chingeni Crops Officer (legumes) 21st May Mr. Kawale Agriculture Extension Coordinator CIP Chiwamba- Mchinji District 21st May Marriam Swale Field Officer NASFAM Kawelawela-Mchinji District 21st May Mr. Mwanza AEDO DAES Tembwe EPA- Mchinji 22nd May Mr. Dennis Emmanuel Zingeni DADO(District Agricultural Development Officer) District Agriculture Office Balaka District 22nd May Osmond Chapotoka DADO District Agriculture Office Dedza District 22nd May Yona Mhango Agriculture Extension Officers DAES Kabwazi EPA-Dedza Ernest Chatunda District 23rd May Owen Kumwenda DADO District Agriculture Office Mangochi 24th May Crops Officer District Agricultural Development Office Machinga District 25th May Alice Sakaliche AEDIC District Agriculture Office Machinga District 25th May Dane Fredenburg Chief of Party Catholic Relief Services (UBALE Project) Jane Deputy Chief of Party Blantyre Juma Agriculture Technician 25th May Michael PCI Malawi (NJIRA Project) Zomba 25th May Mr. Chidwala Crops Officer District Agriculture Office Chikwawa District 26th May Meshen Moodley Human Resources Business Partner ETG Blantyre 65 Date Name Title / Role Organization Place/Location 28th May Dr. Obed Mwenye Kennedy Commodity Team Leader - Potato Breeder Bvumbwe Research Station Thyolo District 28th May Dr. Meshack Soko Pest Control Board Chair Bvumbwe Research Station Thyolo District 28th May Jane Pankuku Root and Tubers Development trust Board Chair Universal Industries Blantyre 28th May Ceaser Nyasulu DADO District Agriculture Office Nsanje District 29th May Philip Beki Assistant DADO District Agriculture Office Chikwawa District Roja Kanyimbiri Crops Officer 29th May Ephraim Mkundula Managing Director Agro Dealer Chikwawa District 29th May Linda Maska Crops Officers District Agriculture Office Blantyre Martha Mbau 30th May Martin DADO District Agriculture Office Ntcheu District McBeth Dzuunde Crops Officer 30th May Mr. P J Bengo Agro-dealer Ntcheu District 30th May Smart Ntolera Marketing & Sales Officer Agro-dealer Mangochi District William Tsokonombwe Managing director 30th May Owner and senior staff Pantochi Seeds Mangochi district 30th May Mary Phiri Owner Global Seed Company Masoto Estate, Mangochi District 1st June Mr Patrick Chingati-Phiri Managing Director CPM Agri-Enterprises Lilongwe 1st June Mr. Fredrick Kawalewale Managing Director AISL Kanengo, Lilongwe. 1st June Dr Makhumba Acting Director of Research DARS DARS Chitedze, Lilongwe 4th June Chief of Party SANE Lilongwe 4th June Grace Mhango Regional President WASAA Lilongwe 4th June Shane Phiri Managing Director Global Seeds Lilongwe 4th June Mar Phiri Professor – Agricultural Economics Luanar University Lilongwe 66 d) Focus Group Discussions Date MISST Component or Organization Place/District # of Participants 21st May ICRISAT-EXAGRIS Farm Estate 57, Mchinji 3 21st May ICRISAT Malingunde EPA, Lilongwe 9 22nd May CIMMYT Bazare EPA, Balaka 20 22nd May ICRISAT Phalula EPA, Balaka 15 22nd May CIP-Umodzi Club 5 22nd May IITA Kabwazi EPA, Dedza District 2 23rd May ICRISAT & CIMMYT-DAES Nankumba EPA, Mangochi District 4 23rd May CIP & IITA Maiwa EPA, Mangochi District 43 24th May ICRISAT-DAES Nanyumbu EPA, Machinga District 5 24th May CIP CVM-Machinga District 1 24th May CIP Mwayiwathu Club-Machinga District 25th May ICRISAT & CIMMYT-DAES Kalambo EPA, Chikwawa District 6 28th May CYMMIT EPA, Nsanje 28th May IITA-DAES Mtonda EPA, Blantyre 13 28th May CYMMIT EPA, Nsanje 8 29th May ICRISAT N’dyachichi Club (PANTHOCHI), Blantyre 29th May ICRISAT Lilangwe EPA, Blantyre 3 30th May IITA &ICRISAT-DAES EPA, Ntcheu 5 31st May CYMMIT-Peacock Peacock Farm-Salima 3 1st June CIP CVM-Lilongwe 1 67 e) Documents Reviewed Name Organization Year Author Malawi Report-Markets Matters Inc. TASAI 2016 TASAI Malawi Road Maps AGRA 2014 AGRA Promoting the Growth and Development of Small holder Seed Enterprises for Food Security Crops FAO Paul J.H Neate￾Robert G. Guei Malawi Seed Industry Seed Trade Association of Malawi (STAM) 2014 Bunda College of Agriculture Malawi Seed Sector Baseline Study AFSTA 2010 H.R. Mloza Banda G. Kaudzu I. Benesi A scoping study on the Status of the Seed Industry in Malawi￾Gaps, Challenges and Opportunities FEED the FUTURE 2017 Mthakati Alexander R. Phiri Malawi Seed and Grain Roadmap FEED the FUTURE 2018 National Seed Policy DARS Annual Work Plan ICRISAT-MISST FY 2018 Management Team Annual Work Plan ICRISAT-MISST FY 2017 Management Team Annual Work Plan ICRISAT-MISST FY 2016 Management Team Annual direct Beneficiary Survey FEED the FUTURE-MISST 2017 Donald Makoka Annual Direct Beneficiary Survey FEED the FUTURE-MISST 2016 Donald Makoka Project Monitoring and Evaluation Plan ICRISAT-MISST FY 2016 Joy Sebastian Ndalama Project Monitoring and Evaluation Plan ICRISAT-MISST FY 2017 Joy Sebastian Ndalama Project Monitoring and Evaluation Plan ICRISAT_MISST FY 2018 Joy Sebastian Ndalama PITT Indicators ICRISAT-MISST 2014-2018 Joy Sebastian Ndalama Integrated Activities in the Agriculture Sector: Multilevel Case Study in Malawi USAID-Malawi 2018 Andrew I. Epstein Alison Smith, Leslie G. Hodel Geetha Nagarajan Consortium Proposal ICRISAT-MISST 2015 Consortium Management 68 Name Organization Year Author MISST Expansion Proposal All Consortiums 2016 Consortium Management Quarterly Reports ICRISAT-MISST FY 2016 To FY2018 Consortium Management Annual Reports ICRISAT-MISST FY 2015 To FY 2017 Consortium Management PERFORM Mid-team evaluation report PERFORM Project July 2017 IBTCI FISH Mid-team evaluation report FISH Project September 2017 IBTCI Impact Assessment on the PSI Training Programs F Msiska, Wotech Investment for ICRISAT May 2018 ICRISAT Aflasafe Commercialization Strategy in Malawi Joseph Atehnkeng, IITA. 2018 IITA 69 ANNEX III. THEORY OF CHANGE (TOC) ANALYSIS TOC Concept. From a project implementation standpoint, the underlying purpose of a Theory of Change (TOC) lies in its ability to demonstrate progress on the achievement of intended outcomes. Evidence of success confirms the theory and indicates that the initiative has been effective. Therefore, the outcomes in a TOC must be coupled with indicators that guide and facilitate measurement. Indicators are necessary to operationalize the outcomes, to make them understandable in concrete, observable and measurable terms. The relationship of indicator to outcomes must be clearly mapped out and both rest on the assumption of certain preconditions that must be in place for the outcome to occur. For instance, greater output of maize per hectare rests on the assumption that critical inputs necessary to increase productivity will (or are) available and accessible to the farmer. If this precondition is in place, introduction of a new technology or improved variety by a project could then result in increased maize productivity per hectare. Under the TOC umbrella, the Results Framework (RF) is the backbone of USAID’s strategic planning process and establishes Mission and partner accountability. The RF sets the parameters for the activity design process and provides a framework for monitoring and evaluation. A Results Framework is a diagram of the cause and effect logic for achieving a development objective over a defined time period. The RF uses causal logic, i.e. if lower-level results are achieved, and then the next higher-level results can be achieved, as long as the critical assumptions mentioned earlier hold. The development hypothesis identifies the specific problem that the development objective addresses, and explains the causal logic between results, generally through a series of if-then statements that describe the theory of change underlying those relationships. MISST Objectives & TOC Construct. The MISST consortium seeks to strengthen agricultural technology delivery in Malawi’s Feed the Future Zones of Influence (ZOI) which cover 10 districts namely: Balaka, Dedza, Lilongwe rural, Machinga, Mchinji, Mangochi, Ntcheu, Blantyre rural, Chikwawa and Nsanje. Project success will in part depend on its integration partners like PROMOTE, UBALE and SANE to continue with promotional and up scaling activities in districts within the Zone of Influence (ZOI). The activity pools resources and expertise to strategically assist Malawi to effectively unlock the potential of the targeted crops, to transform smallholder agriculture, and contribute to the attainment of the Country and USAID/Malawi’s goal of improving the quality of life of Malawians. MISST project effectiveness will be assessed against the extent to which it is achieving its four strategic objectives, which are: 1. Improved functionality of input and output markets of selected value chains: groundnuts, pigeon pea, soybean, drought tolerant (DT) maize and orange-fleshed sweet potato (OFSP). 2. Capacity of stakeholders in agricultural sector strengthened to deploy and promote new crop varieties and their allied technologies. 3. Adoption and uptake of improved technologies enhanced in ZOI. 4. Increased resilience of vulnerable households and rural communities in the FTF zone of influence. The MISST goal, objectives and program components are linked through a theory of change that posits that if: a) resilience of vulnerable households is increased; b) target communities adopt and utilize improved crops and agricultural technologies; c) capacity of stakeholders to produce and deliver new technologies is increased; and d) the functionality of input and output markets are improved, then the quality of life of rural Malawians, including smallholder farm families, will be improved. Contextual Analysis. A logical next step is to evaluate the extent to which the objectives within the TOC framework are actually being achieved, and where within the activity have difficulties arisen? The following discussion will briefly address some of the more salient components of the MISST project and 70 their facility, in hindsight, to actually contribute to the four primary objectives listed above, which ultimately would then translate into the attainment of the Country and USAID/Malawi’s goal of improving the quality of life of Malawians. 1. Improved Functionality of Input and Output Markets. Proper functioning of input and output markets is essential to improving farm family income and quality of life. As a “seed systems” project, MISST has focused quite heavily on the input markets in particular, much less so with the output markets. Input markets consist of all the products and services (e.g. seeds, fertilizers, agro-chemicals, farmer training) that contribute to the productivity of farm households. Some examples of input market activities include capacity building of agro-dealers & seed companies, para-seed inspectors (PSI) training to MISST partners and non-partners; improved linkages between DAO/ Agricultural Extension Development Officer (AEDOs) and SSU staff and enhanced technical support to farm associations such as ASSMAG, NASFAM and FUM to build and improve seed producer and seed bank capacity. Conversely, output markets refer to all the activities that enhance farmer income once the produce leaves the farm gate. Examples include aggregation of product by village traders, sorting and grading to differentiate quality, processing to add value, and distribution of farm or processor output to consumer wholesale and retail markets. Some examples of output market activities by the MISST consortium include the farm demonstrations and field days that stimulated demand for new varieties and improved linkages that were established among farmers, agro-dealers and seed companies. The MISST consortium however, is not known for output market expertise and the evaluation team observed that, in general, output markets were not adequately addressed by the activity. As a result, many project beneficiaries received low prices for their products (for a variety of reasons), but these low prices were a disincentive to increase productivity and adopt new crops and technologies. 2. Stakeholders Capacity in Agricultural Sector Strengthened. In general, it was observed that nearly all stakeholders benefitted from MISST project activities, including farmers, agro-dealers, smallholder farm associations like ASSMAG and NASFAM, and various government agencies, as well as PSI’s mentioned earlier. For instance, MISST staff that were embedded in DAO offices as key contact people were especially appreciated, as it allowed extension officers to more fully understand and engage in upcoming project activities. In particular, training activities and MEGA demonstrations were especially mentioned as a superior learning venue for stakeholders as the breadth and depth of new seed varieties and improved technologies being demonstrated were significant. These demos included aflatoxin trials where farmers were trained to identify aflatoxin infected plant material, the effectiveness of Aflasafe as a control agent, as well as the health risks associated with consumption of tainted maize or groundnuts. Challenges to implementation, however, also affected the potential benefits accruing to stakeholders. Some partners mentioned late delivery of inputs as well as late reimbursements from the buy-back programs; others mentioned low germination rates of groundnut seed, and financial constraints, particularly with respect to high interest rates, significant collateral required for the loan, and prohibitive repayment periods for crop & equipment loans. 3. Adoption and Uptake of Improved Technologies Enhanced in MISST ZOI. In general, there was favorable farmer acceptance of new technologies. Farmers and communities were willing to adopt dry-land cereals where there was a history of repeated maize failure. DTM was highly preferred by beneficiaries despite financial exposure due to higher purchase costs. OFPVs were especially well embraced both from and income generation standpoint from the selling of vines, to the selling and consumption of the tubers, which were particularly appreciated for off-season production in low-lying areas. Farmers claimed the new sorghum varieties introduced by ICRISAT matured in roughly half the time of traditional sorghum varieties and had notably higher yields. Many of the DADOs reported that MISST activities were very helpful for farmer adoption of improved varieties, which often translated into better food security of smallholder farm families. Finally, seed companies noted that adopting hybrid technologies resulted in increased market opportunities and subsequent higher sales. 71 Although many of these benefits accrued to beneficiaries, project activities were not implemented without difficulties. For instance, MISST had a complicated history starting at inception -- call for proposals were sent to each separate CG center and contracts were then issued individually. Each CG center was also responsible to its own HQ, which slowed down the approvals process within each center. The following year central reporting and financing was instituted through ICRISAT via the CMO mechanism, but the CMO was not clear about the USAID fund disbursement process resulting in more confusion. Mobilization took 10 months for CIMMYT because they were not already operating in Malawi, then USAID increased the activity’s scope by adding three districts in the south. Irregular funding and subsequent funding cuts also adversely affected project operations and planning. 4. Improvement in Farm Household Resilience. USAID views resilience as the “ability of people, households, communities and countries to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth.” Its approach to resilience requires identifying ways to “layer, integrate, and sequence humanitarian relief and development assistance in chronically vulnerable communities.”10 Improvement in resilience through the MISST project activities was realized primarily through crop diversification, by introducing improved varieties and technologies, by strengthening and improving linkages between input and output markets, and by enhancing the capacity through training of key stakeholder groups. For instance, sorghum and millet, which require less water, fewer inputs, and have good nutrient capabilities, were established in drought prone areas in the ZOI. CIMMYT and partners developed Drought Tolerant Maize (DTM) that was deployed for climate change proofing in Malawi’s maize-based cropping systems. Soybean is a legume crop of increasing demand in Malawi and has recently become the second largest cash crop after groundnuts. The orange-fleshed sweet potato (OFSP) is a rich source of pro-vitamin A and has been successfully used to perform multiple functions of supplying calories and generating incomes. Finally, aflatoxins are naturally occurring highly carcinogenic toxins produced by fungi. IITA has tested aflatoxin mitigation technologies and found solutions via Aflasafe to address this deadly threat to the country and region. All together to date, these strategies have had a definite positive impact on the resilience of vulnerable smallholder farmers in MISST zones of influence. In spite of these successes, challenges have also emerged. The first is from the policy environment where impacts from Malawi government maize and soy export controls have reduced the market demand for these commodities thereby reducing the prices farmers received for their products. Similarly, a second policy action by the Indian government-imposed import restrictions to pigeon pea crops, thereby adversely affecting market value and the production planning of farmers. A third indirect policy issue is the lack of “Plant Breeders Rights”, which limits incentives for international seed companies to invest in crop variety development in Malawi. Finally, environmental issues have affected the activity as well as the resilience of vulnerable households. Uneven rains during project implementation affected field demonstration performance; half of the cereal and legume demonstrations had losses from drought leading to reduced effectiveness, field day cancellations and reduced seed and grain production. Fall Army Worm (FAW) also affected project work in the previous two years as well as maize yields from smallholder households. 10 USAID, 2012. “Building Resilience to Recurring Crisis: USAID Policy and Program Guidance.” https://www.usaid.gov/sites/default/files/documents/1870/USAIDResiliencePolicyGuidanceDocument.pdf 72 ANNEX IV. MISST GENDER CONTEXT In developing the MISST project, the gender aspect was carefully considered because it is well known fact that gender disparities exist in Malawi. The existing social and cultural constraints mean that women usually farm with fewer resources and/or with less decision-making powers. Men and women also have varying capacities or resources to make use of new technologies. The activity implemented technology promotion activities in a manner that harnesses the gender opportunities of women and men in the farming communities and along the associated value chains. The target value chain crops include orange-fleshed sweet potato (OFSP), groundnuts, pigeon pea, soybean, maize and sorghum, all of which are vital for cash income and food security. The activity had a good representation of both women and men to ensure that adoption rates of improved crops and technologies were optimal. Special efforts were made to build capacity of women as carriers of extension messages to fellow women. The purpose was to assure that positive impacts resulted from women investments, increasing the likelihood that benefits from increased crop productivity and incomes were achieved. Technologies introduced were to mitigate the impacts of drought as well as to address the dietary needs of women and children, the former of whom perform both reproductive and economic roles in their communities. Given that legumes and sweet potatoes are generally regarded as women’s crops, delivery of these technologies focused more on women, children and other vulnerable groups. Consideration of gender issues is an important part of technology dissemination and product development process and were addressed in MISST program activities. For instance, in order to encourage technology use and uptake, overall targets were set to assure that roughly 50% participation of women was achieved at project various levels. However, upon review of quarterly reports, this target may or may not be reached by the end of the activity. For example, on OFSP capacity training of seed sector stakeholders involved 2929 soybean farmers that were trained in 30 field training sessions, 1540 were female and 1329 were males. In another case, the number of farmers and others applying improved technologies showed that 16,217 were males and 8,405 were females. The same applied to the number of households benefiting directly from MISST interventions and the number of people implementing risk-reducing practices, where 11,221 were males and 4,605 were females. Processing and food utilization activities at the household level would entice more women to participate in crops like soybean, sweet-potato, pigeon pea and sorghum. For example, a group of farmers visited in Balaka had 100 women and less than 15 men with DTM, soybean and pigeon pea as target value chain crops. In other words, although the extent of women varied depending on the particular activity, women appeared to be included in fairly strong numbers across most value chains. This is indicative of the significant role women play in all aspects of agriculture production in Malawi, from the “field to the plate”. Gender is a crucial aspect of agriculture in Eastern and Southern Africa including Malawi. It is not limited only to the participation of women but it also refers to other disadvantaged groups and ethnicities, including the “most vulnerable” in segment of society. However, the main focus is on women, because in the case of Malawi they contribute up to 70% of the required on-farm labor. Mainstreaming gender as an issue is essential for two reasons: 1) the number of women working in agriculture is significant and often they do not receive equal economic and or social opportunities; and 2) women who manage the agricultural approach to household food security and income often utilize different production strategies compared to male managers. Hence, training women farmers and bringing them to the mainstream where decision-making takes place can increase technology adoption thereby increasing overall productivity and in turn reduce poverty and malnutrition. In short, closing the gender gap is not only necessary to increase productivity and provide socio-economic benefits to local communities, just as importantly it has substantial implications for public health and nutrition. 73 ANNEX V. CONFLICT OF INTEREST FORMS 74 75 76 77 ANNEX VI. STATEMENT OF DIFFERENCES There are no statements of differences from any of the evaluation team members. 78 ANNEX VII. MISST Stakeholder Dissemination Workshop Notes Mid-term Evaluation of the FTF Malawi Improved Seed Systems and Technologies (MISST) Project Stakeholder Dissemination Workshop Date Held: June 11, 2018 Venue: American Embassy Public Affairs Section (PAS) Auditorium in Lilongwe, Malawi Purpose of the Stakeholder Dissemination Workshop: The United States Agency for International Development (USAID) contracted International Business & Technical Consultants Inc. (IBTCI) to provide monitoring, evaluation and learning support to USAID/Malawi’s Sustainable Economic Growth (SEG) Office under the Monitoring, Evaluation and Learning Support (MELS) contract. Under this contract, the IBTCI team conducted a mid-term evaluation for the Feed the Future (FtF) Malawi Improved Seed Systems and Technologies (MISST) project. The primary goal of this mid-term evaluation was to provide evidence supporting USAID/Malawi in determining whether MISST is on track to achieve its desired results or not, as well as identify any changes or course corrections that may be required. Through this evaluation, USAID also seeks to identify strategic and programmatic options for future engagement to support the effective functioning of the seed sector in Malawi. In view of the above, USAID organized a stakeholders’ workshop to discuss preliminary findings and provide the evaluation team with useful feedback. An invitation was sent to over 50 relevant stakeholders on May 24, 2018 by Mr. Kumbukani Chirwa of USAID/Malawi. Less than half of those stakeholders invited attended the meeting. Agenda: Time Activity Responsible Person 8:00 – 8.30am Arrival and Registration Ellen Chipumphu / Margaret “Maggie” Chibwana 8:30 – 8:45am Introductions Gilbert Kamiyala 8:45 – 9:00am Opening Remarks Cullen Hughes 9:00 – 9:45 am Presentation of Preliminary Findings IBTCI MISST Evaluation Team 9:45 – 10:30 am Plenary Discussion All 10.30 – 10.45 am Way forward Martin Banda 10:45 – 11:00 am Closing remarks Martin Banda Arrival and Registration The stakeholder workshop included participants from USAID, USAID-supported projects, Pannar Seed Company, a Farmer Organization representative, a representative of the Seed Services Unit in the Department of Agricultural Research Services under the Government of Malawi, MISST key project staff and the IBTCI MISST evaluation team. It was observed that a large number of MISST private sector and government partners did not attend as anticipated, despite receiving the event invitation two weeks prior to the session. Introductions Introductions of USAID participants were made at the start of the workshop by Mr. Gilbert Kamiyala of USAID/Malawi, followed by the introductions of the other participants. Opening Remarks Mr. Gilbert Kamiyala of USAID/Malawi was the convener of the meeting, which was opened with remarks by Cullen Hughes. In his remarks, Cullen emphasized that the dissemination workshop was a 79 learning event, not just for MISST, but for all USAID-funded projects. The purpose was to present the findings of the MISST Evaluation and to obtain feedback and recommendations from other key stakeholders. In addition, the evaluation results were expected to inform USAID and the Consortium Partners on how the administrative and programmatic structure of the MISST project succeeded in achieving its objectives, and the types and sources of challenges it faced during implementation. Not only would this inform the project of next critical steps it should take before closeout, but it would also provide USAID with valuable strategic and programmatic insight related to options for future activities in support of Malawi’s seed sector. Presentation of Preliminary Findings Evaluation Team Leader and Seed Sector Expert, Mr. David Coombs and Senior Agricultural Expert, Mr. John Haydu, presented the team’s preliminary findings via PowerPoint to the various stakeholder representatives in attendance. The workshop PowerPoint presentation is attached as Appendix 1. Plenary Discussion During the Plenary Discussion, participants were encouraged to ask questions about the preliminary findings presented by the MISST team. Below are points related to discussion that took place:  One participant commented that in addition to findings she would have liked to have seen some comparative analysis of MISST’s impact. The participant was also looking forward to recommendations in addition to findings, and to more quantitative analysis of MISST progress, although it was unclear what was meant by more “quantitative analysis. The presenters clarified that the purpose of the Stakeholder meeting was solely to present findings, as the team had not yet progressed far enough in the evaluation process to make conclusions and recommendations.  Another participant wanted to express appreciation for the improved seed varieties that MISST has supported and released. o The evaluation team shared with stakeholder participants that MISST introduced 6 OFSP improved varieties developed by DAR into the project ZOI, which were not present prior to MISST. In addition MISST has supported and coordinated activities with the Department of Agriculture Research, Bvumbwe Research Station in the development of three (3) new OFSP varieties, which are in the final stages of testing for release by DAR. o The IBTCI Team also clarified the following:  15 varieties of DTM had been commercialized with support from MISST  MISST supported the release of 4 pro-vitamin A maize varieties in 2016 MISST was designed to scale up existing technologies rather than to carry outbreeding or adaptive research  During discussion, clarification was also sought on the evaluation finding that MISST provided limited financial support for government extension programs due to USAID regulations, which in some cases do not allow direct cash infusion to support government extension workers. This issue repeatedly arose from field-level DAES during teams interviews that many extension staff could not participate in MISST project activities due to lack of mobility, and resource restrictions  Input into the evaluation came through from a collaborating partner under the USAID/SEG office. During the discussion, the partner hinted that they worked with MISST on mega-demos and their role was to run a facilitated mega-demo which enhances learning.  During the discussion, it also became apparent that participants were of the view that MISST should have indeed built in an inception phase. During this inception phase, stakeholders should have been expected to give input into the design and implementation plan of the project.  Participants expected recommendations from the presentation. The team noted that USAID did not want the team to discuss recommendations at this stage, but would, of course, be included in the final report. 80  Participants also discussed the project design’s focus on breeder’s seed. The focus on early generation seed was because the national breeders were not able to produce sufficient seed, and this resulted in a shortage of certified seed for farmers.  Another comment was raised regarding continued strengthening of stakeholder capacity involved in the Seed sector.  Stakeholders also wanted to follow the progress of the seed system since it appears that the seed sector is still struggling with certain maize varieties. o The team noted that substantial progress has been made with increasing the availability of DT maize varieties although not all varieties are yet commercially available. Additional comments and explanations were added by CIMMYT and CIP on OFSP o Both IBTCI and implementing partners reported that MISST introduced 6 OFSP improved varieties in the ZOI for the project which were not present prior to MISST. It was reported that MISST has supported 3 OFSP varieties which are yet to be cleared by the relevant authorities. o 15 varieties of DTM had been commercialized with support from MISST o MISST supported the release of 4 pro-vitamin A maize varieties in 2016  Comments were made on the implementation and coordination challenges for the MISST project. The project team commented regarding the lack of clarity from USAID during the initial implementation period. A case in point was on contracts which initially given directly to each CGIAR implementing institution, but later was issued through ICRISAT who then distributed the funds to each CIGIAR institution. This new consortium arrangement came with some challenges of coordination and reporting across the CGIAR centers. Workshop participants also discussed changes of staff at USAID as a challenge on the leadership for the MISST during the period of assessment.  Comments were also made on the need to have an analysis on how gender and nutrition influenced the effectiveness of mega demonstrations. However, Daniel Van Vugt, CIP and OFSP Project Manager noted that although nutrition was in the original project design, the nutrition component was removed by USAID, because MISST was a seed production project, not a nutrition project, and so they were no longer to report on nutrition.  On the Sustainability of Improved Seeds (FISP and Demand) participants noted that they would like to see recommendations on what should be done after FISP has phased out.  Stakeholders commented also on the coordination of ICRISAT and Seed Companies, noting that seed companies look at ICRISAT as a competitor in the seed sector. o ICRISAT explained that, it is not in income generating activity but just to cover the cost and also the farmers to benefit.  One participant asked a question about how MISST activities promote or hinder other USAID projects from attaining results. The following points were made: o UBALE Project -They were collaborating well in Mega Demonstration mostly on seed Multiplication. o Ag Diversification Project - They are working with different farmers mostly on the seeds that the industries are demanding. o SANE Project - collaborated in demonstrations  On Markets, stakeholders notes that the consortiums hard different understanding on output and input markets (because of the proposals submitted).  On output markets, stakeholders brought up the following points: o Farmers should look at farming as a business 81 o Legumes – The consortium has linked with seed companies to purchase the seeds and sell them to the community. o Early intervention with farmers is needed to better understand what seeds are needed in the field and how much the seed companies are producing.  Related to input markets, it was noted during discussion that if the evaluation was done during the planting season it should have been able to identify the collaboration of MISST and the Agro￾dealers  On acceptance and adoption of improved seed and technologies e.g. Sorghum, stakeholders wanted to know the kind of varieties that farmers preferred.  Discussion points were also raised on the following question: “Will MISST Interventions Result in Improved Seed Production, quality and Productivity?” For example: o How far did MISST go to obtain recognition by SSU of the Para Seed Inspectors? o SSU noted that para seed inspectors do the technical part and SSU certifies the seeds. However, there is still a need for clarification related to the duties of both the para seed inspectors and SSU on who should certify the seeds.  Stakeholders also wanted to know if there will be more quantitative data in the report since it looks more qualitative and not quantitative. o The team responded that the presentation was mostly qualitative as they had not yet had a chance to review the quantitative data received from the project. The team said that it would include quantitative data in the report (this was done)  Discussion participants also had questions related to the following points: o They wanted to know if the MISST Evaluation Team had visited any groundnut processing companies. The Team responded that they had not. o What are the next steps for the evaluation? The team explained that a draft report would be submitted by the team within10 days and, following a commenting and revision process the final report would be shared with stakeholders. This was further explained by Mr. Martin Banda (see below).  Finally, one participant noted that they would like to see in the Evaluation Recommendations a section on how MISST benefited other partners. Way Forward Mr. Martin Banda of USAID/Malawi provided comments on a way forward to the evaluation as discussed below:  MISST was not designed as a standalone program. The intervention was meant to push out germplasm which was not getting out to farmers via certified seed. At that point in time the flagship program was INVC.  He outlined the timelines for the report: o Conclusion and recommendations of the report (next day) o Draft report (10 days from the day of the presentation) o Feedback from USAID and partners (16th Day after submission of the draft report) Conclusion Mr. Martin Banda, of USAID/Malawi, concluded by indicating that USAID is looking forward to the report and it will circulate the draft report to the project partners for their input. He thanked the participants for their contributions and engagement in the workshop. 82 APPENDIX 1. Workshop PowerPoint Presentation 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107