INOVA, DAI FEED THE FUTURE MOZAMBIQUE AGRICULTURAL INNOVATIONS ACTIVITY (FTF INOVA) Midterm Performance Evaluation Report OCTOBER 28, 2020 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), A Tetra Tech Company. FEED THE FUTURE MOZAMBIQUE AGRICULTURAL INNOVATIONS ACTIVITY Midterm Performance Evaluation Report Contracted under AID-656-C-17-00002 Mozambique Monitoring and Evaluation Mechanism and Services (MMEMS) The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 3 CONTENTS ACRONYMS ............................................................................................................. 5 EXECUTIVE SUMMARY ........................................................................................ 6 BACKGROUND ...................................................................................................................................................... 6 RECOMMENDATIONS .......................................................................................................................................... 9 INTRODUCTION .................................................................................................. 11 BACKGROUND .................................................................................................................................................... 11 THE DEVELOPMENT PROBLEM .............................................................................................................. 11 USAID’S RESPONSE ..................................................................................................................................... 13 EVALUATION PURPOSE AND USE ................................................................................................................ 14 METHODOLOGY .................................................................................................. 15 RESULTS ................................................................................................................ 15 APPROACH ............................................................................................................................................................. 15 THE FTF INOVA APPROACH AND CAUSAL LOGIC ..................................................................... 15 FTF INOVAS ADAPTION TO MARKET SYSTEMS DEVELOPMENT ............................................ 19 IDENTIFICATION OF MARKET CONSTRAINTS AND OPPORTUNITIES................................ 22 EVIDENCE ON MARKET CAUSALITY TO INFORM FUTURE INVESTMENTS ........................ 23 COMPARISON OF MARKET SYSTEMS DEVELOPMENT APPROACHES USED BY FTF INOVA AND SIMILAR INTERVENTIONS ............................................................................................................ 23 COMPARISON OF USAID AGRICULTURAL INVESTMENT APPROACHES............................. 26 USE OF CONSORTIUM PARTNER STRENGTHS .............................................................................. 28 PROGRESS ............................................................................................................................................................... 28 PROGRESS ALONG THE CAUSAL CHAIN ......................................................................................... 28 FACTORS PROMOTING AND HINDERING PROGRESS ............................................................... 33 FTF INOVA FACILITATION OF CHANGE .......................................................................................... 34 FTF INOVA EXPANSION .......................................................................................................................... 35 FTF INOVA MEL AND CAPTURING CHANGE ................................................................................. 36 FTF INOVA MEL AND EFFECTIVE LEARNING .................................................................................. 36 LESSONS AND SUCCESS STORIES ........................................................................................................ 38 CHANGE .................................................................................................................................................................. 39 THE SCALE OF IMPACT ............................................................................................................................ 39 RELEVANCE OF PARTNERS/ACTORS .................................................................................................. 40 POSSIBLE SCALING AGENT FUNCTIONS .......................................................................................... 42 SUSTAINABILITY .......................................................................................................................................... 43 UNEXPECTED RESULTS ............................................................................................................................ 43 FTF INOVA AMBITION AND RISK TAKING ...................................................................................... 44 GENDER ................................................................................................................................................................... 45 GENDER STRATEGY ................................................................................................................................... 45 DIFFERENT IMPACTS ON MEN AND WOMEN ............................................................................... 47 COVID-19 ................................................................................................................................................................ 48 ADAPTATIONS IN RESPONSE TO COVID-19 .................................................................................. 48 CONCLUSIONS .................................................................................................... 48 APPROACH ............................................................................................................................................................. 48 COMPARISONS ..................................................................................................................................................... 49 USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 4 CONNECTION TO GOVERNMENT .............................................................................................................. 49 PROGRESS ............................................................................................................................................................... 50 SCALE ........................................................................................................................................................................ 50 SUSTAINABILITY ................................................................................................................................................... 51 UNEXPECTED RESULTS ..................................................................................................................................... 51 RECOMMENDATIONS ........................................................................................ 52 ANNEX A: SCOPE OF WORK............................................................................ 54 SUBJECT ACTIVITY DESCRIPTION ................................................................................................................ 55 1.2.1 ACTIVITY AT A GLANCE ............................................................................................................... 55 1.2.2 FTF INOVA ACTIVITY PURPOSE .................................................................................................. 55 1.2.3 SUBJECT ACTIVITY THEORY OF CHANGE ............................................................................. 56 EVALUATION CONTEXT ......................................................................................................................... 57 STATEMENT OF WORK..................................................................................................................................... 57 EVALUATION PURPOSES ......................................................................................................................... 57 EVALUATION OBJECTIVES AND LEARNING AREAS ..................................................................... 57 METHODOLOGICAL APPROACH ........................................................................................................ 59 DATA SOURCES .......................................................................................................................................... 60 ANALYSIS METHODS AND DATA DISAGGREGATION ............................................................... 61 GENDER APPROACH ................................................................................................................................. 62 EVALUATION PROCESS ............................................................................................................................ 63 REPORTING AND DISSEMINATION .................................................................................................... 65 RECOMMENDED COMPOSITION OF EXTERNAL COMPONENT OF TEAM ........................ 69 ANNEX B: PEOPLE INTERVIEWED ................................................................. 70 ANNEX C: THE FTF INOVA RESULTS CHAIN .............................................. 75 ANNEX D: EVALUATION GRID ........................................................................ 76 ANNEX E: COMPARISON OF MSD APPROACHES ...................................... 77 ANNEX F: COMPARISON OF OTHER USAID AGRICULTURAL INVESTMENTS ...................................................................................................... 80 ANNEX G: MINI-SURVEY RESULTS: AGRODEALER PURCHASES AND SALES ...................................................................................................................... 83 ANNEX H: FTF INOVA THEORY OF CHANGE SEPTEMBER 2020 ........... 85 USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 5 ACRONYMS CLA collaborating, learning, and adapting FAO Food and Agriculture Organization of the United Nations EVI Eco Ventures International FTF Inova Feed the Future Mozambique Agricultural Innovations Activity GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit HEI Higher Education Institutions ICT information and communication technology MEL monitoring, evaluation, and learning MSD market systems development MSI Management Systems International MZN meticais OCHA United Nations Organization for the Coordination of Humanitarian Assistance TA technical assistance UCM Universidade Católica de Moçambique USAID United States Agency for International Development USG U.S. Government VBA Village Based Agent USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 6 EXECUTIVE SUMMARY Management Systems International (MSI) is pleased to present this midterm performance evaluation report to the United States Agency for International Development (USAID). This evaluation assesses Feed the Future (FTF) Inova progress and effectiveness and recommends course corrections to maximize the likelihood of success in the remaining implementation period. The evaluation addresses the following questions (see Annex A for a complete list of sub-questions): 1. Understand the FTF Inova approach to market systems development (MSD), its effectiveness, and how it compares with similar efforts elsewhere. 2. Understand FTF Inova’s progress in improving agricultural outcomes 60 percent into implementation. 3. Gauge whether FTF Inova has fostered significant transformational change in the agriculture sector. 4. Understand FTF Inova’s gender approach and implications. 5. Consider the above questions in the context of the COVID-19 pandemic. The evaluation took place as Mozambique fell under a Level 3 state of emergency in response to the COVID-19 pandemic. This response prevented travel to and within Mozambique, face-to-face interviews, in-person group interviews or workshops, and in-person debriefings. The evaluation addressed these challenges by using remote, online communication to gather data, hold briefings, and share results. BACKGROUND FTF Inova, a five-year initiative with a budget of nearly $21 million, aims to strengthen sustainable business relationships using an MSD approach in agriculture. The activity’s geographical focus is central and northern Mozambique above the Rio Save. FTF Inova’s overall goal is to support sustainable, agriculture-led economic growth by catalyzing more inclusive and competitive agricultural market systems. FTF Inova reports in its revised monitoring, evaluation, and learning (MEL) plan (2019) that the precise pathways toward its goal are uncertain because of complexity, low trade levels, and the limited number of firms in target markets. Rather than chart potential causal pathways in a theory of change, FTF Inova has based its approach on testing ideas formed with private sector partners and tracking how these ideas progress toward its goal. FTF Inova refers to this as a probe–sense–respond approach which helps them manage complex contexts. Through its partnerships, FTF Inova ‘probes’ with a variety of innovations, integrates the results and feedback, and responds. This has been the driving principle behind the design of the approved FTF Inova MEL plan. The FTF Inova MEL plan does not include a results framework. MSI used a mixed-methods approach for this evaluation. The approach included review of relevant documents and analysis of the activity’s quantitative monitoring and evaluation data as developed and collected by FTF Inova. The evaluation team held 52 key informant interviews (KIIs) and four focus group discussions (FGDs) with FTF Inova partners, market observers and experts, representatives from other similar activities, consortium members, and USAID. The team used a story-based “most significant change” approach in interviews with partners. Changes identified in interviews were used to develop an illustrative theory of change to help assess progress and effectiveness. The team captured and analyzed data from KIIs and FGDs using an iterative approach to identify common themes and patterns and then regularly triangulated and cross-checked them with secondary data. This process informed the report’s findings, conclusions, and recommendations. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 7 Evaluation Question 1. Understanding FTF Inova’s Approach to MSD FTF Inova has implemented a flexible, systems-oriented, learning-based approach to support market system change in the agriculture sector. The approach emphasizes sharing and co-creating ideas with the private sector to form co-investment partnerships. Partnerships start small and can deepen with learning and success. Support focuses more on technical advice and less on grants or subsidies. FTF Inova’s partner selection has led to engagement with firms that have the capacity and resources to work with the activity. At the time of fieldwork for this evaluation, FTF Inova intended to prioritize larger partnerships with tested partners, focusing on input (e.g., seed and agrochemicals) and market support sectors (e.g., logistics, transport and information and communication technology).1 Using lessons from implementation, the activity can now better present its approach, clarify market causality, and communicate a theory of change. This work has begun (Annex H). FTF Inova shows mixed results in identifying market constraints and opportunities. The activity’s approach has achieved some positive results, indicating correct identification. Other experiences have been disappointing and show that not all constraints were identified or that, assuming profitable opportunities exist, the best ones were not addressed. FTF Inova has experienced two significant disappointments arising from insufficient understanding of constraints. The approach has led to comparatively less investment addressing constraints in agricultural finance and output market–oriented2 opportunities. FTF Inova’s active interventions were showing signs of progress, suggesting improving constraint and opportunity identification. Comparison with other MSD activities shows that FTF Inova has had a relatively broad focus with a longer period of implementation flexibility. The activity has generally given less emphasis to grants and subsidies than have similar MSD initiatives.3 Comparison with other USAID agricultural investment shows that commodity/value chain initiatives present valuable lessons in achieving impact at scale, using grants, and leveraging connections to government. FTF Inova does not have the connection to government found in earlier USAID agricultural investment programs. Opinions and perspectives differed on achievements of various approaches, mostly on the use of grants/subsidies, the scale and timing of benefits, and sustainability. There is a lack of rigorous evidence to inform discussion about the most appropriate approach in each situation. The consortium has not built on the strengths of its NGO partner organization. The evaluation reflects differing viewpoints from Technoserve and DAI regarding their working relationship. FTF Inova has employed Technoserve staff as the majority of its technical team but is reported to have limited the NGO’s access to monitoring and evaluation data, participation in decision making, and influence on strategic vision. Evaluation Question 2. FTF Inova’s Progress FTF Inova has made the most progress at activity and output levels; it has yet to effectively deliver significant outcomes. At the time of fieldwork, achievements were being challenged by the reaction to COVID-19. FTF Inova has made the most progress in input markets and support services. This progress 1 DAI comments on the midterm evaluation draft report suggest the intention to also support output market actors (international exporters, domestic agroprocessors, and supermarkets). 2 FTF Inova calls these “supply chain” market systems. 3 USAID representatives confirm that “this has been one of the objectives of the activity from the beginning, not to bring subsidies to distort the market but rather to facilitate market changes through TA and in-kind grants. The grant was to be used only if the market players were not able to solve a specific constraint by themselves.” USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 8 has occurred in a difficult environment, with multiple factors—long-term structural macro- and microeconomic, governance, climate, and disease—hindering business. FTF Inova is performing well in facilitating change in partner and stakeholder practices and improving relationships between firms and with farmers. At the time of fieldwork, FTF Inova, though remaining open to ideas, was not aiming to expand beyond its supported innovations and intended to amplify work with potential for the largest impact. It will replicate proven models and concentrate any new innovations around a smaller set of partners. Since the approach FTF Inova has with partners can take time to develop a relationship and test ideas first before scaling up, it is reasonable for FTF Inova to focus at the end of the project on current partnerships and getting proven concepts to scale rather than developing new partnerships. FTF Inova’s MEL system is strong; it is capturing early signs of change and impact and using these signs internally. The implementation team has used information effectively to limit failure and adapt activities. The activity is reported as making valuable contributions to global MSD forums but with limited participation in local forums or with local stakeholders. Evaluation Question 3. The Significance of Change FTF Inova–facilitated changes were not yet at sufficient scale to significantly impact the Mozambican agriculture sector or smallholder farmers’ welfare. Change was at local and district levels. It is still too early for FTF Inova to be transformational or to determine whether results are sustainable. Potential for sustainable impact at scale does exist in input markets. FTF Inova is working with appropriate firms within the agricultural market system, including seed companies and input distributors that can be scaled to provide inputs to many farmers (e.g., one input firm can reach up to 40,000 farmers). FTF Inova has experienced disappointments in cashew and spray service provision support and has learned from the experience while maintaining relationships with the main parties involved. FTF Inova is ambitious in achieving scale. The activity is relatively cautious when partnering with firms. It is willing to invest time piloting, learning, and adapting carefully with the aim of developing sustainable, business-based solutions. It avoids taking significant risks early with its partners to ensure sustainable scale over the long term. Evaluation Question 4. Understand Gender FTF Inova has shifted from its initially proposed more targeted gender approach and has adopted gender mainstreaming. The shift to mainstreaming arose from a focus on MSD-oriented business innovation. Some innovative gender activities have been implemented.4 FTF Inova aims at a program level to mainstream gender within all research, outreach, and implementation activities; however, a mainstreamed gender lens has not been uniformly applied at a field level. Evaluation Question 5. COVID-19 Pandemic COVID-19 is having a deep negative impact on FTF Inova partners. FTF Inova is proposing valuable adjustments to assist companies, such as introducing information technology solutions; using media and 4 See FTF Inova reports of work in actively educating agribusinesses on the untapped business opportunities that female producers represent and the FTF Inova study on gender norms in farming, input use, and distribution in Manica Province (August 2018). USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 9 virtual meetings; embedding health support services to prevent infection; and supporting identification and pursuit of business opportunities that may arise amid the crisis, such as supporting local food production and distribution. RECOMMENDATIONS FTF Inova should continue with plans to focus on input and support market systems and develop a theory of change.5 FTF Inova should continue with plans to focus on farm inputs and support service sectors. It should develop an overall theory of change and individual results chains for focal areas to clarify intended activities, outputs, and outcomes. The theory of change should be linked to indicators that show progress, especially at outcome level. FTF Inova should further develop its interest in output markets. FTF Inova should provide more information on its work in output markets in its regular reports showing how they will contribute to scale. This should include information on the commodities or products involved, who the buyers are in which output markets, estimates of sales and market size in volume and value terms, trends in growth, forecasts of growth and an analysis of margins and investment returns for the partners it supports. This should be monitored to show investment returns. FTF Inova should review its communication strategy. FTF Inova should review its external communication strategy and produce materials that communicate more effectively with external non￾specialists—for example, with an infographic approach that avoids use of technical language and communicates clearly with non-MSD practitioners (e.g., government, donors, NGOs, and private sector partners). To support future intervention design, FTF Inova should reflect and report on the implications of its partner selection process, its support to finance, and its levels of investment in output market opportunities. The FTF Inova end-of-program report should reflect on the positive and negative implications of its partner selection process. This reflection should consider whether small and medium-size enterprises with good ideas but insufficient resources to contribute to partnerships were excluded and whether large-scale opportunities were missed due to the approach. The FTF Inova report should also consider why the activity has evolved to focus on input and support markets and explore the value of an initial broad-based market assessment to identify output market opportunities. The report should also explore why support to address finance constraints and output market opportunities did not evolve to the same extent as farm input–related support, consider the implications of this, and provide recommendations for future MSD activity design. USAID should conduct an independent ex-post evaluation. USAID should hold an independent ex-post evaluation approximately two years after FTF Inova ends. This evaluation should assess activity impact, scale, and sustainability. It should also aim to further inform the debate on systems versus sectoral/sub sectoral approaches and the use of grants when trying to achieve sustainable impact at scale. FTF Inova should strengthen its linkage to government. To help ensure consistency with local government policy, FTF Inova staff should visit provincial-level government to inform them of their 5 Since fieldwork for this evaluation, FTF Inova has begun developing a theory of change (see Annex H). USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 10 activities. At a national level, FTF Inova leadership should review and potentially develop a closer working relationship with GoM/MADER, USAID and SPEED+ (or similar activities) to improve activity performance. FTF Inova support to local food production needs to be consistent with cross-border trade policy, particularly in the COVID-19 context. FTF Inova should improve the use of consortium strengths. FTF Inova should review the use of consortium partner strengths and needs to reset the relationship between DAI and Technoserve. Monitoring and evaluation data and progress reports should be shared with Technoserve. Technoserve country-level senior staff members should be meaningfully involved in FTF Inova strategy review processes. All partners should be involved in developing learning to inform the MSD debate on best practice. FTF Inova should review and potentially enhance its contribution to national-level learning about economic development. National-level economic development practitioners report interest in FTF Inova and call for an increased contribution to national-level learning. The activity should explore ways, with partners, to enhance its contribution to the Mozambique MSD practitioners’ group and other economic development forums to share learning and inform others working in a similar field. FTF Inova should review and strengthen its gender strategy. FTF Inova should uniformly apply its mainstreamed gender strategy across its portfolio. The MEL team should monitor this and ensure that managers are appropriately discussing gender with partners. The activity should also review its 2017 gender action plan and revisit its gender norms mapping exercise to recommend pilot entry points for women in specific markets and develop clear, targeted gender interventions. FTF Inova should inform others on additional COVID-19 adaptations in national-level communication and finance. FTF Inova should be “on the agricultural business pulse” and inform others on what they should do and where they should go to support the response to COVID-19. This effort needs to go beyond USAID resources. The FTF Inova leadership should inform and engage broad agricultural stakeholders and practitioners, including key financial actors such as the World Bank and African Development Bank. Communication should also inform government on shifting markets and changes to cross-border trade to support scenario planning and adjustments to transport restrictions. Agricultural businesses will require rapid access to finance to meet working capital requirements. FTF Inova can help by using DAI and Technoserve connections with businesses and financial institutions. FTF Inova and partners should analyze constraints, needs, and potential finance sources and methods and then rapidly design and provide appropriate support. There are no recommendations on partner selection in the executive summary… INTRODUCTION Management Systems International (MSI) is pleased to present this midterm performance evaluation to the United States Agency for International Development (USAID) as part of the Mozambique Monitoring and Evaluation Mechanism and Services project (MMEMS), through which MSI supports USAID/Mozambique monitoring and evaluation efforts and verifies, monitors, and supports USAID program performance in Mozambique. This evaluation assesses Feed the Future (FTF) Inova progress and effectiveness and provides recommendations to maximize the likelihood of success in the remaining implementation period. The evaluation took place as Mozambique fell under a Level 3 state of emergency in response to the COVID-19 pandemic. This response prevented travel to and within Mozambique, face-to-face key informant interviews (KIIs), in-person group interviews or workshops, and in-person debriefings. The evaluation team addressed these challenges by conducting a remote evaluation. BACKGROUND THE DEVELOPMENT PROBLEM Agriculture is the backbone of Mozambique’s economy, employing about 80 percent of the active workforce (90 percent of the female workforce and 70 percent of the male workforce).6 More than 80 percent of the population depends on subsistence-based, low-productivity agriculture. In 2017, agriculture generated 21 percent of gross national product (GNP),7 suggesting low returns to agricultural labor. The devastating impact of tropical cyclones Idai and Kenneth destroyed much agricultural production and muted growth prospects for 2019.8 However, accelerating agricultural growth remains a key means of eradicating poverty and increasing food security in rural areas, where poverty is widespread. The World Bank predicts that real gross domestic product (GDP) growth in Mozambique will be below 2 percent in 2020, below the average of 3.7 percent experienced between 2016 and 2018.9 The country ranked 180th out of 189 countries in the 2019 Human Development Index, as reported by the United Nations Development Program, and nearly half of its population (46 percent) lived in poverty in 2014.3 Economic growth has been driven primarily by a few large-scale capital investment projects, with very limited benefits to the poor living in rural areas.10 Poverty is mainly a rural phenomenon and particularly persistent north of the Rio Save in Niassa, Nampula, and Zambézia Provinces, where rates are above 60 6 Republic of Mozambique, Ministry of Agriculture. 2011. Strategic Plan for Agricultural Development (PEDSA) 2011–2020. 7 World Bank database. 8 https://www.worldbank.org/en/country/mozambique/overview. 9 http://pubdocs.worldbank.org/en/383301492188164250/mpo-moz.pdf (Figure 1). 10 World Bank, “Smallholder Irrigated Agriculture and Market Access Project. Project Appraisal Document” Report No. PAD2792 (World Bank, 2018). USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 12 percent. Household consumption is nearly three times lower in rural than in urban areas. Eight out of 10 poor people live in rural areas.11 The FAO12 reports that the agriculture sector is characterized by many small-scale farmers, rainfed production, low mechanization, use of labor-based production techniques, a weak policy environment, little use of external inputs, and low productivity. “Smallholder farmers account for most of this sector's production. Approximately 3.2 million smallholder farmers account for 95 percent of the country's agricultural production. Roughly 400 commercial farmers produce the remaining 5 percent. Agriculture is practiced on less than 10 percent of the arable land, largely in flood- and drought-prone areas. Difficult access to credit and markets, low use of improved inputs, and dominance of rainfed agriculture make the sector vulnerable to shocks.” The agricultural markets where FTF Inova operates have few buyers and sellers and low volumes of overall transactions.13 The Government of Mozambique and donors have distributed free inputs and have acted as buyers of locally produced agriculture products. Low transaction volumes and distorted buyer/seller incentives resulting from direct government intervention can “crowd out” other sources such as banks, media, consultants, and information and communication technology (ICT) firms. Availability of cheaper agricultural goods imported from the region and prevalence of weak road networks in rural locations present further challenges to agricultural development. According to a UN OCHA report in June 2020, “the destruction caused by back-to-back disasters and now COVID-19 is escalating an already alarming food security situation and exhausting families’ coping capacities. Prior to COVID-19, an estimated 2 million people were projected to be severely food insecure across Mozambique. Households have not yet recovered from the devastation caused by cyclones that affected more than 770,000 hectares of staple crops. At the same time, global disruptions in trade will affect food supply, resulting in lower production, higher import costs, and higher food prices. Households are likely to exhaust savings and resort to negative coping mechanisms.”14 The COVID-19 pandemic has significantly affected markets and the private sector. The International Growth Centre15 reports that grocery stores and markets have experienced a reduction in mobility of 30 and 21 percent, respectively, following the implementation of social distancing measures. “The slowdown in economic activity severely impacted the private sector. For example, in April 2020 more than 217 companies opted to suspend the work contracts of around 6,400 workers. The tourism and transport sectors have also been heavily impacted by the suspension of visas to the country and by social distancing measures. Border closures led to a rapid increase in prices of essential products.” As many of FTF Inova’s insights depend on its staff being physically in the field with partners, it is likely that much of its work will suffer from the restrictions on movement and in-person interaction.16 11 Data from Inquerito Sobre o Orçamento Familiar (2014/2015). 12 http://www.fao.org/mozambique/fao-in-mozambique/mozambique-at-a-glance/en/. 13 M.A. Bear, “Stocktaking Report on FTF Inova Mozambique Progress in Achieving Impact at Scale” (2019). 14 https://reliefweb.int/report/mozambique/flash-appeal-covid-19-mozambique-may-december-2020. 15 https://www.theigc.org/blog/mozambiques-response-to-covid-19-challenges-and-questions/. 16 FTF Inova Project Continuity Plan. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 13 USAID’S RESPONSE USAID funds the FTF Inova activity to stimulate more inclusive agricultural markets for farmers.17 FTF Inova, a five-year initiative with a budget of nearly $21 million, aims to strengthen sustainable private commercial relationships18 using a market systems development (MSD) approach. The geographical focus of the activity was extended from 26 districts in Cabo Delgado, Manica, Nampula, Tete, and Zambézia Provinces to anywhere above the Rio Save where market opportunities exist. FTF Inova aims to further sustainable, agriculture-led economic growth—critical to reducing poverty and hunger—by catalyzing more inclusive and competitive agricultural market systems. Market systems are more inclusive when market actors have strengthened and expanded access to markets and trade opportunities, which they pursue and benefit from, thereby increasing the value and volume of sales of both the buyers and the producers. Inclusivity also refers to the extent to which traditionally marginalized actors such as smallholder farmers, especially those who are young and/or female, are engaged in and benefiting from the expansion of markets and trade. Market systems are more competitive when market actors can adapt effectively to sell products that meet demand requirements (price, quality, quantity) and, at the same time, ensure profits that enable market actors in the system to thrive. Competitiveness is not determined by the number of market actors but by how well the market actors can face competition, incrementally innovate their business models, and succeed. FTF Inova reports in its revised monitoring, evaluation, and learning (MEL) plan (June 4, 2019) that the precise pathways toward its goal of market systems change are uncertain because of the complexity and “thinness” of the agricultural markets in Mozambique.19 Rather than chart the multiple potential causal pathways that may lead from interventions to outcomes (as there is low certainty and low agreement on what will work), FTF Inova’s approach is based on real-time hypothesis testing, which involves setting out an initial “big picture” vision and ideas for change, supporting innovations, and tracking how these innovations are moving toward the vision. The overarching vision is as follows: “Innovations that build alliances between farmers and other actors in the core of the agricultural market system will contribute to equitable economic growth, provided that they are also underpinned by services and functions that are rooted in other parts of the market system.” The Inova MEL plan does not include a results framework, a graphic useful to evaluators shaping an inquiry of USAID activities. In fact, as stated in the 2018–2019 MEL plan, “FTF Inova does not follow a traditional theory of change approach.” The FTF Inova MEL plan recognizes an area of uncertainty between its work with partners and its vision of improved market systems where causality is not clear or predictable. FTF Inova refers to this as the 17 USAID awarded FTF Inova (contract number AID-656-C-17-00001) to DAI Global, LLC on February 22, 2017. FTF Inova has a budget of $20,971,049 and is to be completed by February 22, 2022. DAI leads the implementation, supported by TechnoServe, Ecoventures International, and MarketShare Associates. 18 FTF Inova comments on the SOW emphasized relationships. 19 In thin markets, limited numbers of entrepreneurial growth firms—companies growing faster than their peers or the broader economy—have difficulty finding and transacting with each other at reasonable costs. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 14 black box of causality.20 FTF Inova therefore aims to gradually “unpack the black box” of complex market behaviors by testing and collecting evidence on business innovations during each intervention.21 This midterm evaluation presents an opportunity to learn by unpacking FTF Inova’s black box. The evaluation will contribute to FTF Inova and USAID learning about the activity’s causality and support development of a theory of change while enough implementation time remains for learning to be beneficially applied. EVALUATION PURPOSE AND USE This evaluation assesses FTF Inova progress and effectiveness and provides recommendations to maximize the likelihood of success in the remaining implementation period, in keeping with best practice for monitoring and evaluation and USAID’s collaborating, learning, and adapting (CLA) approach. Given the FTF Inova activity’s experimental nature, this type of iterative learning approach is particularly useful. The evaluation addresses the following questions (see Annex A for a complete list of sub-questions): 1. Understand the FTF Inova approach to MSD, its effectiveness, and how it compares with similar efforts elsewhere. 2. Understand FTF Inova’s progress in improving agricultural outcomes 60 percent into implementation. 3. Gauge whether FTF Inova has fostered significant transformational change in the agriculture sector. 4. Understand FTF Inova’s gender implications. 5. Consider the above questions in the context of the COVID-19 pandemic. To assess effectiveness, the evaluation clarifies the activity’s causal logic and compares outputs with outcomes.22 The evaluation also compares FTF Inova with other similar projects and approaches. It assesses FTF Inova processes for capturing lessons learned, including partner and stakeholder feedback, and the extent to which systems are used to refine FTF Inova activities and adjust implementation to enhance the activity and ensure relevance to agricultural systems transformation. To assess progress, the evaluation compares FTF Inova’s achievements with the links in its causal logic. It reviews the factors promoting or hindering progress; the FTF Inova method of facilitation; and the potential for expansion, scaling up, and sustainability. The evaluation will provide the Mission, the U.S. government (USG), and the wider donor community with usable lessons for adaptation under USAID’s CLA framework. The evaluation will also provide the 20 The current Inova black box–based explanation lacks the clarity provided by a traditional logical hierarchy based on activities, outputs, and outcomes. Inova reports that the uncertainly inherent in their approach means that causality is unclear and that using a logical flow of activities, outputs, and outcomes is inappropriate or impractical. The midterm evaluation provides an opportunity to explore this view and generate some outputs and outcomes that reduce ambiguity. 21 FTF Inova MEL plan submitted October 2018 (version 2), revised May 2019. 22 Collaborative work with FTF Inova staff helped develop a theory of change and clarify the activity’s causal logic during the evaluation period (see Annex H). USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 15 Mission with background, conclusions, and recommendations to inform planning for follow-on activities and the new country development cooperation strategy. The main users of the evaluation findings will be the Office of Agriculture Environment and Business at USAID/Mozambique and others at the Mission’s discretion. The FTF Inova lead implementer (DAI) and the other members of the implementing consortium (Technoserve, Eco Ventures, and Marketshare) will also likely use the evaluation findings to inform activity implementation. The Government of Mozambique and key donors in the sector might also use recommendations from the evaluation to design new interventions in the sector. METHODOLOGY The evaluation applied a consultative, mixed-methods approach. The team collected primary qualitative data through KIIs and focus group discussions (FGDs). A list of people interviewed is provided in Annex B. The team collected quantitative data from secondary sources such as relevant documents (Annex C) and the activity’s monitoring and evaluation data. The team also conducted a mini-survey of agro-dealers to confirm sales and sources of inputs. Given COVID-19-related travel and meeting restrictions in Mozambique, the team used online approaches, including virtual group/team meetings, phone-based interviews, and video. A separate report was generated to describe learning from this approach. The MMEMS approach drew on utilization-focused methodologies. The team conducted initial briefings to review and confirm evaluation questions and discuss with USAID how to best share the evaluation results and recommendations. The team actively engaged staff from the four organizations implementing the activity. In addition to formal interviews, the team held feedback briefings for follow-up questions with Inova and USAID. These events were managed in a way to protect confidentiality and the evaluation’s independence. After finalizing the evaluation, MMEMS will conduct a three-month follow-up to determine how the evaluation results will be used and how they have informed decision making. RESULTS APPROACH THE FTF INOVA APPROACH AND CAUSAL LOGIC The FTF Inova Approach The FTF Inova approach at the time of this evaluation’s data collection aimed to explore and work in the broad agricultural market system. The approach is relatively flexible and not limited to a commodity, sector, or selection of subsectors, as is usually the case in a value chain activity. FTF Inova aims to move from this broad coverage to more focused investments by a “test, learn, and focus” approach that works closely with private sector partners and emphasizes the joint creation of investment ideas. Specific USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 16 investments are based on building strategic relationships and networks, leveraging business investment, encouraging innovation, and achieving sustainability with businesses. The activity’s focus is on engagement and co-creation—looking for partners with ideas and working on these ideas together to develop interventions that support competitiveness and pathways for growth. Technical assistance and advice are emphasized over subsidies, grants, or more formal research.23 FTF Inova uses two methods to find and start engagement with potential partners: (1) issuing calls for proposals through which a private sector actor can submit ideas and (2) proactively engaging businesses through formal and informal meetings. In formal meetings, FTF Inova presents itself and what it does to private sector representatives; follow-up meetings are then held with interested parties. For example, FTF Inova conducted a stakeholder workshop in Nampula in connection with the Associação Comercial, Industriale Agrícola de Nampula (ACIANA). Interested business representatives then went to meet FTF Inova as a follow-up to the workshop. In informal, ad hoc meetings, a business representative calls into an FTF Inova office or meets an employee to start discussing ideas. Evaluation fieldwork revealed proactive engagement (the “come to us” approach) as the method most used by FTF Inova to start working relationships with the private sector. Business representatives interviewed as part of the evaluation consistently describe having a series of meetings with FTF Inova staff to present, discuss, negotiate, and agree on ideas for the activity to support. If the parties agree, the relationship deepens from an initial application to a “deal note,” a written agreement to do something together. One partner can have a series of deal notes that grow in scale as the relationship with FTF Inova builds. Key informants describe a process that can start with brainstorming, study, and analysis and lead to pilots and learning as interventions are jointly developed. Interventions are usually implemented by the partner and supported by FTF Inova technical assistance. In-kind grants and equipment distributions are possible but tend not to form a large proportion of the FTF Inova support package. As an example, one business representative described how FTF Inova, through a market study and consultant support, “helped us to connect ourselves to the farmers and build healthy relationships with them by changing small things in the way we approached our field workers and farmers.” Another partner reported approaching FTF Inova looking for a grant. They talked and shared ideas. FTF Inova presented the idea of supplier clubs. This idea was agreed on and is reported by the partner representative as “radically changing” their approach from aiming to remove communities from their land to working with them. FTF Inova intends to focus its approach. The activity reports24 that its approach is evolving in three ways: prioritizing and, where necessary, reducing the number of intervention areas and interventions; pursuing larger and deeper partnerships and activities with tested industry partners; and focusing more strongly on supply chain management (farm inputs and support sectors such as transport and ICT). 23 FTF Inova Annual Work Plan 2020: “On the one hand, grant mechanisms with a significant contribution from the grantee may allow FTF Inova to lend resources to an activity that can be fully led by a market actor; in other cases, the best route to scale may be for FTF Inova to use its credibility, networks, and knowledge to serve as a convener and a connector, or serve as a negotiator to help locate resources elsewhere in the system” (11). 24 FTF Inova Annual Work Plan 2020, 6. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 17 The FTF Inova Theory of Change FTF Inova does not follow a traditional theory of change approach.25 The activity did develop results chains in 2017 (see Annex D) that key informants reported as unused. Review by the evaluation team suggests that the 2017 results chain does not clearly communicate FTF Inova’s approach outside of the activity. To help clarify the activity’s causal logic, the evaluation team developed an illustrative, generalized theory of change based on key informant responses and FTF Inova documents (Figure 1).26 It shows FTF Inova’s co-creation of interventions and technical support at an activity level leading to the desired strategic relationships, leverage, and innovation at output level. This is then seen as contributing to increases in income, trade, and profits that in turn support the intended goal of agriculture-led growth. This general theory of change is developed in more detail in the section of this report discussing progress along the causal chain (Figure 2). FIGURE 1. ILLUSTRATIVE REPRESENTATION OF THE FTF INOVA CAUSAL LOGIC Source: Authors’ interpretation of FTF Inova theory of change based on document review and key informant responses. FTF Inova Partner Selection Inova selects partners that have the capacity and the willingness, motivation, or commitment to do something (the “skill and the will”). FTF Inova determines partner capacity by ability to finance or co￾finance an initiative. Observers suggest that this is easier for larger partners such as international or regional businesses or relatively strong Mozambican companies. FTF Inova measures partner commitment through observation of behavior such as willingness to commit staff time and effort to test and work on ideas. 25 FTF Inova MEL plan submitted October 2018 (version 2) and revised May 2019. 26 A more detailed theory of change is presented later in this report to illustrate FTF Inova progress. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 18 Other important factors emphasized by key informants include the following: ● A partner’s strategic potential, meaning the business’s proximity to the market system or positioning where FTF Inova perceives a need for more actors ● A partner’s innovativeness, sustainability, and connection to/inclusion of smallholder farmers ● The idea’s alignment with what FTF Inova is trying to facilitate ● The idea’s potential to bring change (or be a “disruptor”) ● The profit potential or potential for commercial sustainability ● Avoidance of firms expecting support (i.e., a free grant) ● Number of donors already working with the potential partner FTF Inova evaluates partner proposals based on a criteria grid (Annex E). Staff members involved in this process emphasize that it is less of a selection methodology and more of a way to think continuously about whom to work with.27 Rejection or acceptance and movement to the next stage rarely occur. Follow-up and further engagement are the usual outcome.28 FTF Inova staff report that they now know most of the actors in Mozambican agricultural markets.29 Some large firms are reported as not interested in working with FTF Inova. Activity representatives perceive these firms as mainly profit driven and happy with the status quo. They are seen by FTF Inova as more exploitative and not having the shared value of wanting to balance profit with improving the situation of farmers or other actors in the chain. Evaluation fieldwork found evidence of frustration with the FTF Inova partner selection process, particularly from smaller potential partners with fewer resources. FTF Inova looks for actors that want to innovate and have resources to invest. Some observers report that this results in selection of those who can invest and against those who cannot. These observers suggest that actors who can invest and make changes are often the same, so the same people benefit from development support. Partners with good ideas but fewer resources to implement change (such as small and medium-size enterprises) can be excluded and frustrated by the approach. 27 FTF Inova staff also highlight the contrast in their approach to that commonly used in a procurement or grant award process. The FTF Inova partner evaluation is a first step in the process rather than the culmination of a process. It is a way to discuss ideas and decide how to proceed. FTF Is continuously asking, “Is this partner still committed to pro-poor transformation of the market system? Does the idea have potential for impact at scale?” 28 Outright rejection or acceptance of the initial application rarely occurs; the more usual outcome is to follow up with further engagement to understand the idea and the partner better before approving the application to enter the co-creation process. Assigning an application "follow-up" status can culminate in an approval/disapproval decision within a few days or weeks and is usually the result of a key question from the evaluation committee not addressed in the initial application. Applications do not remain indefinitely in follow-up status. 29 FTF Inova has received about 200 applications and reports tending to know applicants already. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 19 FTF INOVA’S ADAPTATION TO MARKET SYSTEMS DEVELOPMENT As part of FTF Inova’s inception phase, eight FTF value chains (pigeon pea, common bean, soybean, bananas, sesame, cashew, groundnut, and cowpea) were assessed, focusing on the activity’s zone of influence. The value chain analysis built on initial findings and recommendations from a previous USAID global Leveraging Economic Opportunities agricultural value chain analysis in Mozambique. FTF Inova’s value chain selection was based on three criteria: competitiveness, development impact (including scalability), and feasibility to generate impact within three to five years.30 The value chain analysis’s primary objective was to generate a strategy for interventions that would yield short-term results, test critical assumptions about evolving market system dynamics through direct engagement with market actors, and allow longer-term strategy development.31 The August 2017 analysis supported the development of FTF Inova’s Year 1 work plan. Cashew, sesame, groundnut, and pigeon pea were selected as priority value chains for Year 1. FTF Inova also produced five strategy summaries for cashew, groundnuts, pigeon pea, and sesame in September to October 2017. The first section of the 2017 FTF Inova value chain analysis gives an overview of the Mozambican agricultural market system focusing on key crosscutting market systems, input supply, the seed sector, financial services, business support services, and logistics. It then presents more detailed analyses for each of the eight value chains. The overview section identifies Export Trading Group and Olam as “strong potential partners.” It also highlights viable growth opportunities and reports “many seed and fertilizer companies and major suppliers of crop protection products.” It identifies finance as a “principal constraint at multiple functional levels in all of the value chains” and reports that improving finance for agriculture could have the greatest impact on productivity and should be an area or priority for FTF Inova.32 Each of the individual value chain analyses provides a four- to five-page sector overview and presents information on sector dynamics, end markets, vision for equitable growth, identified interventions (in the cashew value chain), leverage points, a map of value chain functions, exporters/processors, traders, and production. Discussion under these section headings includes policy considerations. Analysis does not present detailed margin or quantitative investment analysis. It sometimes mentions previous support (e.g., TechoServe/CLUSA support to soya); however, it does not emphasize lessons from previous development partner (including USAID) experience. The FTF Inova value chain analysis identified areas for future support in some commodities and not others. The cashew analysis identified possible FTF Inova support to (1) help cashew processors adopt and adapt supply chain management practices that deliver access to higher-value export markets and that ensure farmers’ access to quality inputs and technical support and (2) strengthen service provider systems that support buyers to achieve market certification and traceability. The cashew analysis also 30 See Table 1 in FTF Inova Value Chain Analysis August 2017 for further details. 31 FTF Inova Value Chain Analysis August 2017. 32 An FTF Inova proposed agrifinance strategy was developed in March 2020. This strategy was shared with the evaluation team in September 2020. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 20 reported Condor Nuts expressing interest in working with FTF Inova to secure certification standards. FTF Inova later worked with Condor Nuts to address high-value export markets. The sesame analysis reported Olam showing a strong interest in expanding its investment in white sesame and K2 requesting support to establish a customer relationship management system that could embed several marketing innovations. A market actor partnership agreement was reported as being signed with K2 in the 2017 value chain report. Other analysis was less successful in identifying a way forward, sometimes despite reporting growing markets. The soya analysis reported steadily rising demand for soybean, which was claimed to consistently exceed domestic production by approximately 50 percent. This analysis recommended maintaining dialogue with the existing donor projects, NGOs, and private sector actors as opposed to identifying clearer opportunities. Intervention areas and entry points were not identified for beans, cowpeas, or banana. FTF Inova also performed a Manica-Sofala rapid market assessment in September 2017. This assessment developed a list of possible value chains with growth potential in the Beira Corridor apart from the eight FTF value chains researched in the earlier assessment. Value chains had a fruit and vegetable focus and included avocado, litchi, macadamia, mango, export vegetables (baby corn, chilies, broccoli), and local vegetables (cucumber, cabbage, lettuce, tomato). Research for each value chain was again based on competitiveness, development impact, and feasibility to generate impact within three to five years. Recommendations from this study included exploring ways new marketing strategies could be expanded into potential new districts of Machaze, Buzi, Chibabava, and Machanga and exploring interventions to reduce food insecurity and support climate change adaptation. For Litchi the study recommended engaging Westfalia33 to address export markets and collaborating with Sumol+Compal34 to develop their strategy for local fruit procurement. The study recommended local vegetable support to promote initiatives to spur the local procurement of vegetables by working with a Campesinas greenhouse project. It also recommended further engagement with the Support the Friends of Agribusiness Forum and supporting the expansion of high-value vegetable production. For mango, it recommended expanding the use of and training in crop protection measures.35 Crosscutting support activities were also recommended in food safety, trade show engagement, and investment facilitation such as facilitating business-to-business linkage through trade shows and collaborating with the USAID–Southern Africa Trade Investment Hub to address market information and transaction costs. FTF Inova then developed five strategy summaries, one each for cashew, groundnuts, pigeon pea, and sesame. Each strategy summary provided a one-page discussion of the vision for market systems change, current market conditions, targeted constraints, and entry points, followed by a value chain map and a results chain. Entry points include the identification of partners (including CDA and K2 for inputs and Condor Nuts for cashew, which the activity worked with subsequently) and strategies that FTF Inova has since used—for example, (1) helping input suppliers strategically coordinate to provide market signals that encourage smallholder farmers to adopt inputs and prioritize expanding their market footprints more strategically by geographic area, (2) working in the cashew agro-inputs industry to increase marketing of products and services to smallholder farmers, and (3) seeking other ideas through 33 Westfalia could not be found on the list of FTF Inova actors presented to the evaluation team at the time of fieldwork. 34 Sumol+Compal could not be found on the list of FTF Inova actors presented to the evaluation team at the time of fieldwork. 35 DAI comments that FTF Inova and USAID did not believe mango presented good opportunities. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 21 a process of requests for proposals for innovative business models that shift cashew buying relationships from competition to cooperation between processors and smallholder farmers. After the 2017 analysis, FTF Inova decided to move away from using results and value chains.36 By 2018, the activity was emphasizing a broader market system approach. In its 2018 baseline report, FTF Inova reported that it “takes a market systems approach to increase equitable growth of the Mozambican agriculture sector by catalyzing continual strengthening of relationships and linkages between market actors throughout the agricultural market system” and that it “takes a portfolio approach to market systems development.” In its 2019 annual work plan, FTF Inova detailed its use of the more holistic Market System Change Wheel37 as the starting point for developing support to agricultural market systems. The Market Systems Change Wheel allowed the activity to take a much broader and flexible approach than when focusing on eight value chains. USAID representatives reported that this adaptation was made because markets were limited and the added flexibility would likely increase the range of FTF Inova’s investment opportunities. The FTF Inova zone of influence was also expanded to allow it to work anywhere north of the Rio Save. These explanations are consistent with other key informants’ accounts of FTF Inova’s significant effort looking for potential partners. Allowing the activity to work in a wide range of markets over a larger geographical area allowed potential engagement with more businesses. Another reason for the activity’s adaptation reported by some USAID staff members was the perceived need to try new approaches after investing for 15 to 20 years in Mozambican agriculture in the same way and still facing the same challenges. USAID representatives also reported that they wanted FTF Inova to be innovative and groundbreaking. With the increased use of market system development, the activity also began to deemphasize the results chains used in its initial value chain summaries. The activity reported in its 2018 baseline study that the reason for this was that “the [MSD] approach means that predicting change along a linear path is not possible.” FTF Inova began to emphasize an area of uncertainty between its test innovations with partners and its vision of improved market systems where causality is not clear or predictable. To explain the reasons for this change, the activity reported that “the use of methods that rely on some form of linear causality, such as result chains, [is] therefore being rejected by the activity as it seeks a flexible monitoring system that allows it to accommodate and detect unpredicted outcomes that signal market system change.”38 FTF INOVA began using probe indicators to explore markets and test innovations through the perceived area of unclear causality (the black box) and toward its vision. 36 FTF Inova continued to conduct further studies, including on coverage of farmers' preferences for seed products, dispute resolutions in outgrower schemes, transport needs and costs for the agricultural sector, and the feasibility of weather index insurance. 37 https://www.msdhub.org/agricultural-market-system-change-wheel.html. 38 FTF INOVA Baseline Report. May 2018, 1. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 22 IDENTIFICATION OF MARKET CONSTRAINTS AND OPPORTUNITIES FTF Inova shows mixed results in identifying market constraints and opportunities. The activity’s approach has achieved some positive results (see “Lessons and Success Stories”), pointing to correct identification. Other experiences have been disappointing (see “Unexpected Results”), suggesting that not all constraints were identified or that, assuming profitable opportunities exist, the best ones were not addressed. Evaluation fieldwork results suggest that FTF Inova has experienced two significant disappointments—in cashew and Isolmoc spray services—through insufficient understanding or definition of opportunities and constraints. At the time of the evaluation, FTF Inova’s active interventions were showing signs of progress, suggesting improvement in constraint and opportunity identification. It is not clear whether a sufficiently thorough overall analysis of the market situation was carried out to help understand constraints and opportunities before the activity was put in place. Parts of the 2017 value chain analysis were useful in identifying potential partners (i.e., CDA and K2) and opportunities but not sufficiently rigorous to sufficiently understand opportunities and constraints (i.e., Condor Nuts). Some observers point to current market opportunities (in bananas, mangoes, and forestry) in Mozambique that FTF Inova is not addressing.39 Within the timeframe of the evaluation, it was not possible to determine whether these would have presented better opportunities. FTF Inova has identified constraints and opportunities in two main, overlapping ways. The most used method is joint identification with partners as part of the investment co-creation process. An example found during fieldwork involved a major input supplier and output buyer. FTF Inova discussed ideas based on this partner’s views of business constraints or opportunities. However, the partner had not adequately assessed the constraints and so the assessment was incorrect. Also, estimates of potential gains were inflated and FTF Inova did not have enough information as implementation was rushed. The second FTF Inova method is to perform studies or action research—for example, the 2017 value chain study exploring fruits and vegetables. Also, a financial study has looked at remittances and prepayment of inputs with agro-dealers. This research is reported to have led to FTF Inova supporting actions offering smallholder farmers the opportunity to pay for inputs in advance when they have money to guarantee inputs. FTF Inova also conducted research with a supermarket to learn about domestic supply chains where agricultural products are sourced in northern Mozambique and transported for sale in Maputo stores. A behavioral study interviewed farmers to find ways of addressing farmer supply-side constraints to improving access to inputs. This study has informed FTF Inova’s work with seed companies and input distributors to take inputs closer to farmers by combining supplies and increasing coordination. This work was evolving in early to mid-2020 but was hindered by COVID-19-related restrictions, making it unclear whether the constraints and opportunities addressed were correct. FTF Inova’s intervention co-creation process has relied largely on ideas coming from partners that activity staff perceive as committed and able to contribute. This process is bound by the information available to the persons involved and their levels of creativity, knowledge, and experience. FTF Inova 39 DAI representatives report FTF Inova and USAID concluding that these were not viable opportunities based on initial value chain studies. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 23 cannot invest in constraint- or opportunity-related ideas that it does not know about or its staff do not think can be successfully addressed. At the time of fieldwork, the FTF Inova approach has led to the activity emphasizing and primarily addressing constraints in input and support service markets. At the time of fieldwork, opportunities in output markets were not emphasized and had not clearly emerged as focal areas. This may have been due to several factors, such as lack of opportunities, partners not presenting fundable ideas for output markets, and FTF Inova staff not identifying or perceiving many fundable opportunities in output markets. EVIDENCE ON MARKET CAUSALITY TO INFORM FUTURE INVESTMENTS Over its three and a half years of implementation, FTF Inova has built significant experience working with businesses in Mozambique’s central and northern agricultural markets. Staff turnover has been low, although the chief of party left at the end of August 2020, and learning has been emphasized, increasing knowledge and experience of market causality. During the activity’s initial stages in consultation with USAID staff members, FTF Inova emphasized that its work in “complex” markets suggested that cause￾and-effect relationships were identifiable in retrospect. Document review showed that the activity has not emphasized causality in its reporting. At the time of fieldwork, senior FTF Inova managers reported that “as time passes, the project and our partners learn. This allows us to identify what works and what doesn't. It allows for solutions to emerge. As they emerge, viable change pathways become clearer (theories of change).” Now that implementation is further advanced, the implementer should be able to use evidence to infer a cause-and-effect relationship to plan interventions40 and inform future investments.41 The evaluation found initial evidence of market causality, including FTF Inova facilitating business partnerships and supporting movement from a franchise model to development of agro-dealer networks to reduce costs for businesses and prices for farmers. Ngala spray service providers report improved services leading to expansion, and radio stations point to improvements in their services as well. Support to farmer groups to aggregate produce is also reported to have increased trade. These examples are discussed further below. COMPARISON OF MARKET SYSTEMS DEVELOPMENT APPROACHES USED BY FTF INOVA AND SIMILAR INTERVENTIONS The evaluation compared FTF Inova’s approach with those of four similar MSD interventions (Honduras Transforming Market Systems, Innovation for Agribusiness [InovAgro I II III], Hortisempre, and Women in Business) (see Annex F for additional details). Comparison shows that all MSD initiatives emphasize research and learning (particularly in early phases), flexibility, adaptation, and work with the private sector on market-oriented, business-based solutions. 40 M.A. Bear, “Stocktaking Report on FTF Inova Mozambique Progress in Achieving Impact at Scale 2019,” 6. 41 FTF Inova reports (COP email July 7, 2020) that “the viable change pathways we identify and pursue through our partnership process are crucial for informing future investments (based on the theory of change we continuously develop and update). Indeed, it is mainstreamed so thoroughly into every stage of our internal learning and reporting that it is not easy to disentangle or point to a single document.” “We could select one or two ‘viable change pathways’ and give you an example of how our understanding has developed and evolved, with reference to internal and external project documents.” This offer was accepted on July 7, 2020, by the midterm evaluation team leader. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 24 Support comes mostly in the form of technical assistance, with grants and distributions usually at levels lower that 25 percent of overall budgets. TABLE 1. SUMMARY OF MSD INITIATIVE APPROACHES Initiatives Central Theme Approach Ideas Focus of Support FTF Inova Agriculture Initial studies and analysis to explore markets, in which interventions are co￾created with substantial flexibility, using an MSD approach Joint co-creation with partners with some research, calls for proposals with due diligence and analysis TA based Grants/In-kind support: approx. 10% of original budget Honduras Transforming Market Systems42 Job creation Initial studies and analysis lead to selected sectors and then design emphasizes an MSD￾based approach Emphasize continuous co￾creation; discuss and flesh out ideas TA based Grants/In-kind support: approx. 20% of budget Innovation for Agribusiness (InovAgro I II III)43 Seeds and six value chains Use MSD to develop and test models, supporting ones that work Engage with partners that have ideas and then go through a process of due diligence and analysis TA based Grants/In-kind support: < 20% of budget Hortisempre44 Irrigation Use MSD in a facilitatory role with entrepreneurs During implementation, emphasis is on learning, making changes, and adaptation TA based Grants/In-kind support: < 10% of budget 42 Start 2018 ongoing, led by ACDI VOCA and funded by USAID. https://www.acdivoca.org/projects/transforming-market￾systems/ 43 Duration: 2010–2020. Implemented by DAI and funded by Swiss Agency for Development and Cooperation. https://www.dai.com/our-work/projects/mozambique-innovation-agribusiness-inovagro 44 Duration: 2013–2020. Implemented by SwissContact and funded by Swiss Agency for Development and Cooperation. https://www.swisscontact.org/nc/en/country/mozambique/projects/projects-mozambique/project/-/show/horti-sempre-en.html USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 25 Women in Business (WIN) 45 Women micro￾entrepreneurs Use MSD to focus support to viable business cases Mix of internal idea development, calls for proposals, and specific consultancies TA based Grants/In-kind support: approx. 25% of budget Fieldwork results suggest that the MSD approach in the selected comparative initiatives informs thinking around a central theme. For example, Hortisempre focuses on irrigation and uses market-based system thinking as a framework to inform, plan, and monitor its more detailed activities. TMS Honduras focuses on jobs and InovAgro on seeds in six value chains that could be considered sectors/subsectors. Although approaches are similar, FTF Inova has had a comparatively wide remit and has been able to probe, learn, and adapt support in agriculture broadly and over a relatively long time. TMS Honduras respondents report that they use an inception phase to study, analyze, and select sectors based on job￾creating potential and then move to detailed design emphasizing a systems-based approach within a year. Observers with knowledge of FTF Inova or with significant experience in MSD in Mozambique suggest the activity uses a cautious approach when applying grants and subsidies.46 Data presented in Table 1 show that FTF Inova allocates approximately 10% of its overall budget to grants and in-kind support. This compares to Honduras (TMS 20%, WIN 25%), InovAgro (less than 20%) and Hortisempre (less than 10%). Access to finance is a significant and recognized constraint in Mozambique. Grant financing is viewed as both potentially distorting markets and as an “enabler.” MSD implementers in Mozambique claim recent success in initiatives that have supported small-scale farmers to access machinery with a 50 percent grant. These farmers are described as becoming “agricultural hubs,” and the approach was adopted and adapted by a recent World Bank project. Support to poultry meat processing that used grant finance tools is also claimed to be attaching several businesses to quickly “crowd in.” Grant proponents also point to the grant financing some FTF Inova partners received to fuel their initial expansion. Advocates for grants recognize their disadvantages in terms of expectations, dependence, and distortion, but they suggest that grants are an important tool to support small and medium-size enterprises for whom only expensive commercial loans are available. 45 Duration: 2018–2023. Implemented by TechnoServe and funded by Swedish International Development Cooperation Agency. https://www.technoserve.org/our-work/projects/women-in-business-making-markets-work-for-women-entrepreneurs/ 46 USAID representatives report that this has been one of the objectives of the activity from the beginning—not to bring subsidies to distort the market but rather to facilitate market changes through TA and in-kind grants. The grant was to be used only if the market players could not resolve a specific constraint by themselves. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 26 COMPARISON OF USAID AGRICULTURAL INVESTMENT APPROACHES The evaluation compared the FTF Inova approach with those of four USAID agricultural investment activities in Mozambique (EMPRENDA, AGRIFUTURO, P4I, and RAMA) (Table 2; see Annex G for additional details).47 Comparison shows significant differences in the approaches used by the selected initiatives. Although all activities use adaptive management, the FTF Inova approach emphasizes systems thinking, learning, and flexibility to a greater extent than the other initiatives. P4I has a targeted, grant-focused approach in which targeting has been adapted over its four funding rounds. Although RAMA implementation has also adapted, this activity uses a largely predetermined approach based on four pillars to support climate￾smart agriculture. Emprenda and Agri Futuro both used a sectoral value chain–based approach. Fieldwork results show that Emprenda has achieved success at scale, particularly in the cashew and poultry sectors.48 TABLE 2. SUMMARY OF USAID AGRICULTURAL INVESTMENT INITIATIVE APPROACHES Initiatives Approach Ideas Research and Learning Focus of Support EMPRENDA ($25 million activity implemented from 2005 to 2009) Sectoral approach based on comparative advantage Sectoral studies that gave detailed industry/sector plans Research emphasized at start; learn with implementation Technical/managerial support focus AGRIFUTUR O ($25 million activity implemented from 2009 to 2015) Agribusiness development focused on value chain approach; included an enabling environment component Business development ideas came from working with businesses and depended on their needs Most research was on the enabling environment component Technical support focus Grants – about 6% of total budget used mainly to bring in experts and machinery and fill gaps in the value chain Partnering for Innovation ($15.5 million Grant-based, fixed-price investments Private sector presents ideas in answer to Research, diagnosis, and Grants make up at least 50% of the total budget 47 The comparison is illustrative; no counterfactual was developed. Further study is required to add rigor. 48 Emprenda claims having increased the number of cashew processors from 1 to 12 within two to three years as investors copied the promoted model. Advocates for MSD question the sustainability of this success and point out that processors are now subsidy dependent. Emprenda also claims having increased the size of the poultry sector from less than 1 percent of GDP to approx. 4 percent of GDP. Advocates for MSD point out that this took seven years to achieve. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 27 spent in Moz.; implemented from 2014 to 2021) calls for proposals learning are not emphasized No direct technical assistance RAMA (approx. $10 million 2017 to 2022) Climate-smart agriculture focus based on four predetermined pillars Ideas come from implementation experience Low levels of formal research; learning happens with implementation Technical support focus No grants The Emprenda approach identified sectors (value chains) based on analysis of comparative advantage. (The MSD activity TMS Honduras also had an initial inception phase to determine focal areas.) Emprenda then performed more detailed studies to verify comparative advantage, identify constraints, and develop an industry strategy, including supportive policy changes. Implementation emphasized use of international business expertise to conduct the detailed studies, managers with industry experience, and supportive government actions. Business-based logic underlay the Emprenda approach as it does that of FTF Inova. FTF Inova does not, however, emphasize the use of managers with specific industry experience (as Emprenda claims to do in the cashew sector) as private sector partners take a leading implementation role. FTF Inova does not have the direct political connection reported by Emprenda.49 FTF Inova’s approach is long term and aims to effect behavioral change. Activity staff members report that it is more qualitative and that its outcomes can be relatively intangible.50 Although COVID-19 presents significant challenges, this situation may change as FTF Inova implementation continues and the activity produces clearer examples of sustainable impact at scale. Both value chain and MSD approaches have merits and are valuable. The review of previous USAID agricultural investments suggests that Emprenda presents valuable lessons: a sectoral approach allowing targeted investment based on analysis of comparative advantage to achieve scale at least in the short term, use of international investment analysis expertise, and government connection and policy support. 49 Key informants report that this is because of contractual differences that allowed Emprenda senior staff to directly engage government. In the FTF Inova contract, any government contact must be made through USAID. FTF Inova reports (chief of party during a briefing on June 26) that they go to SPEED+ on governance issues and must follow strong USAID protocols. 50 DAI reports that FTF Inova's outcomes are not meant to be intangible, just not always conducive to quantification at the activity and market system level. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 28 USE OF CONSORTIUM PARTNER STRENGTHS Each of FTF Inova’s implementing partners—DAI,51 Techno Serve,52 Eco Ventures,53 and Market Systems Associates54—has unique strengths that FTF Inova is using in different ways. DAI, as the lead organization, has the most engagement in FTF Inova, bringing strengths in leadership and MSD technical expertise. DAI has several projects and programs in Mozambique, including SPEED+,55 Inovagro, SeedTrade,56 and Financial Sector Deepening.57 Eco Ventures contributes on-demand coaching and facilitation support in certain areas, such as supplier clubs. DAI reports they are valuable advisors on strategy to FTF Inova senior managers. Market Systems Associates provides regular planned engagement in MEL, gender, and learning. It is in regular contact and “always on SLACK.”58 Technoserve, an established NGO in Mozambique,59 has a long history supporting agriculture and business, with significant experience in cashew, soya, and poultry as well as in training farmers and female entrepreneurs. It was the lead partner in Emprenda and a consortium member in Agri Futuro. Five FTF Inova staff members are from Technoserve. FTF Inova staff members report that FTF Inova has drawn on Technoserve experience. For example, the Technoserve WIN project helped introduce businesses such as supermarkets and cashew processors to FTF Inova. Cashew processors had received Technoserve support to get Hazard Analysis Critical Control Point status. Key informants also report that FTF Inova Technoserve staff have a countrywide biannual staff meeting where learning is shared from each other’s programs. These key informants point to Technoserve learning on the cashew sector being shared and helping with the FTF Inova spray service training program. The relationship between DAI and Technoserve, particularly at a senior country level, has not made optimal use of consortium partner strengths. Perceptions of the reasons for the relationship differ between the two partners. Fieldwork shows different expectations exist regarding levels of collaboration, information sharing, influence on decision making and how the organizations would draw on each other’s strengths. Key informants report that Technoserve’s status as an NGO—the only NGO among the partner organizations—has presented challenges in managing partners’ level of effort. PROGRESS PROGRESS ALONG THE CAUSAL CHAIN Overall, FTF Inova has made moderate progress working in a difficult environment and in the face of significant challenges (see “Approach”). Most progress has been made at activity and output levels of the 51 https://www.dai.com/our-work/projects/mozambique-feed-the-future-agricultural-innovations-inova 52 https://www.technoserve.org/our-work/where-we-work/mozambique/ 53 https://www.eco-ventures.org 54 https://marketshareassociates.com 55 https://www.dai.com/our-work/projects/mozambique-support-program-economic-and-enterprise-development-speed 56 https://www.dai.com/our-work/projects/southern-africa-feed-future-southern-africa-seed-trade-project 57 https://www.dai.com/our-work/projects/mozambique-financial-sector-deepening-fsdmoc 58 SLACK is a communication platform. https://slack.com/intl/en-et/ 59 https://www.technoserve.org/our-work/where-we-work/mozambique/ USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 29 causal chain. Some key informants report modest changes in outcomes. However, at the time of fieldwork, these achievements were being challenged by the COVID-19 response. FTF Inova has made the most progress in input markets and support services (i.e., spray services and radio). FTF Inova has performed a range of activities that have led to many of its intended outputs being produced. Figure 2 shows the FTF Inova activities that partner organization representatives consistently identified during fieldwork, such as providing technical advice and training; facilitating workshops, meetings, and platforms; holding tenders; negotiating initiatives; piloting; facilitating market access; and monitoring progress. It then shows the outputs key informants report as produced from these activities, such as building capacity of about 200 agro-dealers, helping distributors expand networks and routes, creating partnerships, building relationships with farmers, creating positive incentives for change, and facilitating group-based platforms and groups/clubs. FTF Inova progress at an outcome level was emerging and relatively small in scale at the time of fieldwork. A small number of key informants report rapid progress. For example, one company60 representative reported that the implementation of FTF Inova recommendations allowed for a tenfold improvement in the company’s profitability compared with the loss in the previous year: “It was like instantly climbing from the ground floor to the 10th floor.” Other key informants report increases in trade at relatively low volumes, such as FTF Inova support enabling them to purchase 20 metric tons of rice or enabling farmers to sell an additional two truckloads of produce. One key informant explained that following FTF Inova advice to form farm-level producer groups reduced conflict between communities and landowners. FTF Inova’s facilitation approach has developed capacity and improved relationships. For example, capacity development is reported in agro-dealers (e.g., numbers of dealers, proximity to the farmer), information and knowledge, financial and account planning, radio stations, and machinery centers. Input distributors report changing their business model as a result of FTF Inova technical advice. This model included working in partnership with seed producers, using catalogues to allow customers to preorder, and moving from a franchise model to development of agro-dealer networks and new supply routes. Another agricultural business reported an FTF Inova–supported market study as crucial in its decision to deepen its involvement in the seeds business. Input distributors and seed suppliers also report improved collaborative relationships due to FTF Inova support. Relationships between businesses and farmers are also reported as having improved with FTF Inova support. Input providers, buyers, radio stations, and supermarkets report improved linkage in this area. FTF Inova has also facilitated formation of a wide range of groups and platforms. These range from online logistics hubs to more traditional business common interest platforms and farmer groups to aggregate produce. One key informant also reported that implementation of FTF Inova technical advice had led to the successful introduction of positive incentives rewarding best-performing farmers. FTF Inova performance indicators are not clearly linked to a theory of change, making it difficult to assess progress by causal link from the review of these indicators. However, a general review (presented in Table 3) points to progress. Encouraging results are found for investment leverage 60 FTF Inova reports this company’s scale as small. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 30 (EG.3.1-14), individuals participating in activities (EG. 3-2 and EG.3.2-24), number of hectares using improved practices (EG.3.2-25), number of farmers with improved commercial relationships (Custom Indicator #3), and number of firms with improved relationships (Custom Indicator #4). This suggests investment growth, rising numbers of participants using improved practices on their land, and more commercial interaction between farmers and businesses. These results are consistent with fieldwork findings. Indicator EG.3.2-27 suggests growth significantly above target for agricultural finance. Over the first six months of 2020, however, some of this growth may have been reversed. The increase reported by the indicator relates to a cashew processor purchasing cashew; the processor had been in financial difficulty and was “inactive” in terms of FTF Inova support at the time of fieldwork. FTF Inova representatives report that this partner firm was active in Quarter 2 FY 2020 and its figures were included in the quarterly report. The FTF Inova performance indicator of sales (EG.3.2-26) is significantly below target at Quarter 2 FY 2020, suggesting that the enhanced capacities, relationships, practices, and interactions (output-level factors) are not leading to more business or trade (outcome level). The overall target for this indicator was $6,353,526 for 2019, rising significantly to $29,267,590 in 2020. FTF Inova reported sales of $4,175,431 at Quarter 2 FY 2020 before the full implications of COVID-19 started to emerge, expecting this sales figure to rise in the second half of the year. Quarter 2 FY 2020 results for this indicator also include a significant contribution from the cashew processor in financial difficulty. FTF Inova reports that the total cumulative value of sales reported in Quarter 2 is distributed among the key partners as follows: cashew processor (35 percent), seed distribution company (29 percent), and seed supplier (36 percent). FTF Inova reports that other partners “on hold” or “inactive” are not included in indicator results.61 61 FTF Inova MEL manager email July 7, 2020. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 31 FIGURE 2. ILLUSTRATIVE THEORY OF CHANGE Source: Authors’ estimates based on fieldwork results. TABLE 3. SUMMARY OF FTF INOVA PERFORMANCE INDICATORS62 Ref Indicator Baseline (2018) FY 2020 Cumulative Achie ved FY 2020 Overall Target YOUT H-3 Percentage of participants in USG￾assisted programs designed to increase access to productive economic resources who are youth (15–29) [IM￾level] Youth unemployment is 42.7%, and underemployment is up to 80% 770 youth/4,462 participants (18%) 22% EG.3.1- 14 Value of new USG commitments and private sector investment $15,000 1,140,061 (76%) 1,492,612 62 The table does not include FTF Inova’s gender indicator as it is reported later in this document. The table also does not include the FTF Inova youth indicator as it is not seen as relevant to any of the evaluation questions. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 32 leveraged by the USG to support food security and nutrition [IM- level] EG. 3-2 Number of individuals participating in USG food security programs FTF Inova results start from 0; 22,000 farmers participated in FTF Mozambique programming in 2016 15,664 (71%) 21,026 EG.3.2- 24 Number of individuals in the agriculture system who have applied improved management practices or technologies with USG assistance 12% of commercial farmers use fertilizer, 10.3% use pesticides, 16.9% use improved seed63 13,268 (81%) 16,136 EG.3.2- 25 Number of hectares of land under improved technologies or management practices with USG assistance Producers applied improved technologies and management practices on more than 138,000 hectares of land as a result of FTF Mozambique investments 14,522 (64%) 22,823 EG.3.2- 26 Value of annual sales of farms and firms receiving USG assistance [IM-level] $79,200 annual sales of FTF Mozambique￾supported farmers 4,175,431 (13%)64 29,267,590 63 FTF Inova Baseline Report 2018, EG.3.2-17. Number of farmers and others who have applied improved management practices or technologies with USG assistance. 64 FTF Inova MEL representative report sales data can be delayed as they are sensitive; partners may believe (incorrectly) that they may be used for tax purposes. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 33 $2,272,371 annual sales of FTF Inova partner firms EG.3.2- 27 Value of agriculture￾related financing accessed as a result of USG assistance [IM-level] 2.3% of rural households receive credit for agriculture 3,500,000 (1300%) 265,573 Custom Indicato r #3 Number of farmers who have a new/improved commercial relationship with partner firms 0 7,401(70%) 10,488 Custom Indicato r #4 Number of FTF Inova partner firms who have new/ improved commercial relationships with other firms 0 29 (107%) 27 Source: FTF Inova. FACTORS PROMOTING AND HINDERING PROGRESS FTF Inova works in a difficult environment with many long-term structural factors that limit progress. At a macroeconomic level, these factors include import subsidies favoring urban consumption over local production; inflation (i.e., rising input costs); business taxation; poor infrastructure, roads, and ports; corruption; land laws reducing collateral; and the shortage and high cost of finance. Significant parts of the geographical areas where FTF Inova works have been hit by two cyclones, and the COVID-19 pandemic is now hindering business interaction and trade, especially at regional and international levels. For example, one FTF Inova partner reports that the COVID-19 context has forced the company to delay planned expansion, stopped close work with farmers, and hindered collaboration with FTF Inova experts. At an operational level, FTF Inova works in markets that have a small number of private sector actors that tend to be small in scale, and trade is at relatively low levels. FTF Inova often refers to this situation as “thin” markets. Commercial agriculture levels are very low. Seed companies report farmers preferring open pollinated varieties of maize seed and rarely using inorganic fertilizer. Although these practices increase resilience and have environmental advantages, they limit annual seed sales and productivity compared with use of hybrids with supporting agrochemicals. In addition to being small and seasonal, agricultural markets are often distorted by government and NGO distributions that can crowd out the private sector or shift incentives in a way that influences their behavior. This can be beneficial, USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 34 such as when government reduced its role in seed distribution and some businesses saw an opportunity to step into this space. Business partner numbers, capacity, and attitudes have led to challenges such as a low number of potential partners with ideas or difficulty agreeing on deals. Because of the history of donor support, many firms perceive activities such as FTF Inova as a source of low-cost or free finance. Unrealistic expectations can challenge deal formation and long-term relationships between FTF Inova and its potential clients. Observers also report that development projects and programs can compete with one another to support businesses. FTF Inova’s focus on developing ideas while providing relatively small financial contributions is reported by some as hindering progress as businesses are drawn to projects giving larger amounts of cash. Low numbers of partners with viable ideas have forced FTF Inova to look more, introduce,65 and then test ideas lightly.66 Experience also shows that private sector priorities can change with personnel changes or market fluctuations. Private sector partners report that the time taken (often three to four months) to contract partners in the deal note process also hinders progress. USAID policies and procedures are also reported as reducing flexibility to take opportunities and as hindering progress. The evaluation did not find any clear factors that promoted FTF Inova progress.67 Key informant responses show that it was easier to identify factors hindering progress. As the evaluation proceeded and respondents were probed more deeply about positive factors, responses were less clear as people struggled to identify positives that triangulated with other data. FTF INOVA FACILITATION OF CHANGE FTF Inova is performing well in facilitating change in partner and stakeholder practices. At the time of the evaluation, there were early signs of firms benefiting from market opportunities as a result of this facilitation. Overall, respondents value and appreciate FTF Inova facilitation for its continuous support in developing ideas and fostering team spirit. This facilitation, often seen as time consuming, can lead to significant changes from the original ideas presented by the business partner. One supermarket was reported as moving from an original proposal for $1 million (50% of the entire FTF Inova budget) to going through a yearlong, often “bumpy” process of discussing ideas, piloting, and getting to know each other. As a result of this process, at the time of fieldwork, FTF Inova and the supermarket were moving forward with a new grant for significantly less than $1 million to build a supplier network using women’s groups. At the time of the evaluation, it was not clear whether the supermarket’s initial business expectations were being met. FTF Inova full-time staff and short-term technical assistance conduct facilitation by coaching companies and stimulating strategic thinking to target smallholder farmers. Facilitation support is reported to have 65 DAI refers to this as “seeding” ideas. 66 That is, initial low levels of support that grow with success. 67 FTF Inova reports “positive probes” in its FY 2020 Q3 report, which was presented after fieldwork. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 35 helped develop new brands, formed partnerships, supported new businesses, increased product ranges, and built understanding of smallholder farmer needs. For example, facilitation has helped an agricultural input distributor reduce input costs by improving supply routes and develop marketing materials such as a catalogue. A seed supplier reports that FTF Inova facilitation saved them from the need to build a warehouse, thereby reducing their costs. A commodity dealer reports that FTF Inova facilitation helped them change the system by which inputs are provided to farmers by working more closely with input suppliers. Another large-scale agrochemical business valued FTF Inova’s assistance in addressing “last￾mile” delivery to farmers. Both examples are yet to provide significant impact, with COVID-19-related restrictions having delayed progress. Facilitation has also increased awareness in service companies of potential in agricultural markets. For example, a logistics company is now better informed to address transport issues for agriculture products. The company reports that FTF Inova’s work has improved their understanding, enabling them to develop a better marketing strategy. FTF Inova has also facilitated change in market research companies and their perception of the agriculture sector as a potential opportunity for their services. The activity has also supported commercial radio stations to work more in the agriculture sector. FTF radio activity support found during fieldwork includes providing radio messaging services and advertising agricultural inputs, which have increased sales for these companies. FTF Inova facilitation has not always been sufficiently informed to effect change. Communication has not always been clear, and negotiations have not always succeeded. One partner felt that promised monitoring and evaluation support had not been delivered. Another key informant felt that staff did not fully understand smallholder farmers’ incomes in Mozambique. In this case, activity staff had proposed giving best-performing farmers incentives that far exceeded normal smallholder farmer income levels by the potential partner. Also, in this case, the potential partner deemed FTF Inova suggestions for staff salary scales inappropriate compared with market rates and potentially damaging to the business’s sustainability. In another case, a partner reports that the FTF Inova deal was based on exaggerated farmer numbers and unrealistic expectations of what smallholder farmers could afford.68 Both examples suggest that FTF Inova knowledge has not always been sufficient or that intervention implementation has been rushed. FTF INOVA EXPANSION With approximately 18 months of the activity remaining, FTF Inova is not aiming to expand beyond its supported innovations. In its 2020 annual work plan, FTF Inova reports its intention to “focus on amplifying innovations with the largest impact, replicating proven models, and concentrating new innovations around a smaller set of partners.”69 68 DAI representatives question this observation, suggesting that precise farmer participation could not be determined in advance. 69 During iterative finalization of the evaluation report, DAI commented that FTF Inova would support several new interventions in the output markets and always remain open to innovations. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 36 FTF Inova remains open to new ideas and innovations, but it is also pragmatic about how far it can go with testing and scaling up brand new ideas with only 18 months left. At the time of the evaluation, the activity was proposing to focus on innovations with the greatest potential for sustainable impact at scale as determined by its experience and learning. FTF INOVA MEL AND CAPTURING CHANGE The FTF Inova MEL system is good at capturing early signs of change in interventions with partners. Activity claims for impact are relatively consistent with evaluation findings. During fieldwork, FTF Inova staff were largely aware of the situations reported by their implementing partners. For example, MEL staff confirmed that input distributor sales were growing in veterinary medicines for broiler poultry and fertilizer as reported by the companies’ representatives independently. The evaluation mini-survey of agro-dealers (Annex G) was also generally consistent with MEL information. FTF Inova MEL is conducted at two levels:  Agent level. Staff use a dashboard that registers monthly change in sustainability, value to smallholder farmers or a subgroup of smallholder farmers, and scale (potential to change a market system).  System level. Staff use a set of tools and metrics (indicators) that are both inductive (based on observation) and deductive (staff investigate changes detected). Activities are implemented in close collaboration with FTF Inova staff so that managers and technical advisors can detect change. Staff observe progress and inform MEL staff, who can then follow up. For example, using this approach, MEL staff reported seeing raw material purchases and sales falling at a supported cashew processor. As a result, FTF Inova senior managers reviewed the activity and put support on hold. FTF Inova maintains an indicator dashboard for each intervention. Examples seen by the evaluation team suggest that dashboard data are consistent with the situation on the ground and are used by managers to capture early signs of change. FTF Inova staff report using partner data to perform continuous triangulation. For example, staff review agro-dealer records and conduct short surveys to add data. They cross-check data sets for consistency and double counting. They compare data and visit data providers such as agro-dealers. FTF Inova MEL staff report knowing that farmers spend on average 2,681 Meticais (approx. $38)/year) on inputs. Therefore, they can check farmer numbers and compare them with total input sales. FTF INOVA MEL AND EFFECTIVE LEARNING Learning through observation and study is a major FTF Inova principle. Activity staff members emphasize learning as quickly as possible during an intervention using a “probe, learn, adapt” approach. Learning enhances partner interventions, making them more effective. Activity internal learning is emphasized, but USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 37 key informants report that external learning is emphasized at a global rather than a national level.70 Some consortium members report that FTF Inova does not share monitoring and evaluation data. Fieldwork responses show that communication material is often complex and not easily digestible by non-MSD practitioners.71 The co-creation approach used by FTF Inova promotes continual learning and adaptation. Examples highlighted by activity representatives include those discussed in earlier sections, such as linking an input distributor with a seed supplier, developing a catalogue, and supporting supply routes for agricultural inputs. Quarterly portfolio reviews are also used as a team-based approach to learning. FTF Inova studies are based on the activity’s CLA plan. Examples provided during fieldwork include a dispute resolution study that explored contracts and low literacy among smallholder farmers. This study reviewed how contracts could be made clearer so that fewer contracts fail. A gender norms study was conducted whose findings and recommendations were used to inform partners about the market opportunities that female farmers present.72 FTF Inova builds on and develops solutions informed by learning from partners. For example, in response to a partner study that observed that individuals needing transport could be assisted by transport trucks that return to their bases empty after delivering goods, FTF Inova developed a concept note and business plan and is now supporting an online platform that registers transporters and connects them to individuals looking for transport. Review of FTF Inova learning-related communication shows that it is detailed, complex, and difficult to quickly digest by information-rich, time-scarce managers. Reports are most easily digested by MSD practitioners and often use a specialist vocabulary (e.g., “probes,” “pivots,” “nudges,” “thin markets”) that is less clear to non-specialists. Traditional tools for communication and learning, such as theories of change, are not emphasized. Review of FTF Inova on-the-ground work suggests that activities could be described in clearer, simpler language. International observers report that FTF Inova communicates instructively on global platforms such as the Beam Exchange and73 MarketLinks74 and with USAID/Washington. However, some national observers report that FTF Inova is less willing to share learning. These key informants report that FTF Inova rarely contributes to a Mozambican-level MSD practitioners’ group.75 Given the wealth of market￾based, private sector interventions in Mozambique and the interest in MSD, FTF Inova could make a valuable contribution at this level. 70 DAI representatives perceive FTF Inova as the most active member of the Mozambique MSD network and that FTF Inova gives equal emphasis to national-level and global learning. 71 DAI reports planning to devote more resources to this area in its final year. 72 “Qualitative Study on Gender Norms in Farming, Input Use, and Distribution in Manica Province” (August, 2018). 73 https://beamexchange.org 74 https://www.marketlinks.org 75 DAI representatives feel that FTF Inova is a major technical contributor to this group. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 38 LESSONS AND SUCCESS STORIES Success Story: Community Radio Supporting Agriculture Lesson: Successful innovations do not always require cash grants; however, they may be smaller in scale and take time to grow. “We never asked or received cash funds from Inova. We received technical training and business ideas and that is what unlocked our success and sustainability.” – Radio station manager Most radio listeners are farmers and, with support from FTF Inova, the station became an important partner for agriculture support. In 2018, Inova invited the company to a community radio forum to explore opportunities for working together. At the end of the forum, FTF Inova challenged the company to produce a radio program for a seed company. The challenge was accepted, and a deal note was signed. FTF Inova provided technical assistance and training to the radio staff on marketing techniques, how to profitably run the community radio, and how to design and frame radio programs for farmers with specific messages for each phase of the agriculture production cycle. The station started producing programs with specific agriculture technical information to educate farmers, including best agriculture practices in each phase of the agriculture production cycle. Radio programs also contained promotional messages for a seed company that paid for this service. Programs recommended that farmers use the company’s seeds and the techniques taught in the broadcasts. The program ran for three to four months in 2019. After that, the station continued broadcasting agriculture-related programs. Success areas were two new agricultural input–related clients working with FTF Inova that also started using the radio station’s marketing service for paid advertising. Some donor projects supporting agriculture in the province also started paying to air their programs and activities. “We can confidently say that Inova was instrumental in linking the radio to the agriculture sector and helped us to get closer to clients from the agriculture inputs market. Inova showed us how and where to start.” – Radio station representative Success Story: Building Stronger Partnerships with Farmers Lessons: ● Relationships matter and small changes can lead to win-wins. ● Co-creation builds ownership and pride in what is being developed. “Inova helped us to connect ourselves to the cashew farmers and build healthy relationships with them by changing small things in the way we approached our field workers and cashew farmers.” – Business representative USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 39 This partner provides spraying services and cashew tree treatment. The relationship with FTF Inova started in 2018 at an outreach meeting in Nampula launching the activity. Following first contact, FTF Inova sent its experts to visit the partner. After discussions about business opportunities and an analysis of the company’s approach, FTF Inova invited representatives to work together to improve the way the partner was conducting its business. A deal note was signed and FTF Inova commenced market studies. Activity consultants were sent to the field and interviewed farmers and field workers. From this, FTF Inova came up with recommendations for contracts with farmers to provide farmers with tree treatment services and ensure purchase of production at an agreed price. The partner also started giving incentives to its spraying workers and cashew farmers. For example, the best cashew farmer was given spraying equipment. Partnerships with farmers became stronger and tree treatment improved, increasing productivity. University students were engaged to monitor fieldwork. The company also built on Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) support, which provided staff training on value links, an important complement to the FTF Inova support. Services expanded to two other districts. Collaboration with FTF Inova also improved the company’s approach to gender. The company became sensitive to employing female workers in its field-spraying activities. FTF Inova was instrumental in changing company employment policy. “Due to health concerns that we are now aware of, we do not employ female workers in spraying servicers. However, we favor the employment of female workforce for our retailer network and warehouses.” Company management feels very confident about the future, even with COVID-19, in consolidating and expanding its market standing in the cashew value chain beyond Inova support. CHANGE THE SCALE OF IMPACT By July 2020, FTF Inova–facilitated changes were not at sufficient scale to impact the Mozambican agriculture sector or significantly impact smallholder farmers’ welfare. Examples of local and district-level change were found, however, during fieldwork, a finding supported by FTF Inova reports. The FTF Inova Quarter 2 FY 2020 report states: "Until now, probes by individual partners can be brought to scale.” This FTF Inova report goes on to identify partners showing signs of encouraging transformational change. Though evaluation results show that it is still too early for FTF Inova to be transformational, potential for impact at scale does exist, particularly in input markets. This scale is possible if inputs can reach more customers in a manner that allows a lower cost while delivering sufficient profitability for the actors involved. Examples of ongoing change reported during fieldwork include the following: USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 40 ● An input provider claimed greater efficiency in input use and a slight increase in yields when correct dosages were used. These changes were not yet at scale. ● Spray service provision expanded from 8 to 20 providers, with potential to cover 1,000–1,500 cashew farmers, compared with the current coverage of 540 farmers. ● Three hundred ninety-one families were organized into 12 supplier clubs, and this organization is expanding on its own. ● Seed businesses are expanding within a region to another one or two districts based on FTF Inova– supported market research. These changes were limited due to COVID-19-related restrictions. Examples of FTF Inova partner plans for expansion reported by key informants during fieldwork include the following: ● An input distributor is aiming to add six new shops per year at district level. ● Agrochemical producer and supplier are aiming to supply 40,000 farmers, and FTF Inova support is expected to accelerate the expansion process by two to four years. ● A commodity trader is expanding its farmer performance classification idea to 40,000–50,000 farmers; FTF Inova support is again expected to accelerate the process by two to four years. ● Some districts are now creating supplier clubs on their own based on the positive examples from FTF Inova–supported clubs. ● A radio station plans to expand its activities to include input trade fair organization and conference equipment rental services with FTF Inova support. Potential impact from FTF Inova support exists mostly in input distribution and in partners performing market support roles, such as spray service provision and radio. Some significant commodity traders and input providers reported the intention to expand where FTF Inova support would speed up the expansion process. This confidence in proceeding with an initiative is to be expected in large-scale partners of an activity that aims to support a business’s own ideas. These businesses emphasize that FTF Inova support will allow them to expand more quickly and effectively. RELEVANCE OF PARTNERS/ACTORS FTF Inova is working with firms within agricultural market systems, including seed companies and input distributors, that are relevant to achieve scale and benefit many farmers. FTF Inova has not developed partnerships in finance and output markets to the expected degree given the significance of finance as a constraint and the importance of output markets as contributors to economic growth. Input distribution companies offer a network of stores, with extension services that are close to farmers. FTF Inova needs to be clear and consistent in its approach to supporting input distribution. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 41 Different perspectives were found on the vision for the sector. Some respondents suggest that a monopoly market structure may evolve and that this structure is relevant to best supply large numbers of farmers, although a monopoly could raise prices. Others suggest that FTF Inova should support a more competitive market structure to avoid potential monopoly pricing power. Fieldwork found evidence of competition between input supply firms supported by the activity. This area needs review to avoid duplication and overlap in support and to ensure competitive input pricing. In addition to input distribution, FTF Inova is working with commodity buyers and input producers that are relevant to achieving scale and that benefit many farmers. Partners report plans to expand, and seed companies report further interest in the market. In finance, FTF Inova has worked with village saving and loan associations (VSLAs), one bank and supported agro dealers stocking inventory. The activity has not worked significantly in ICT fintech solutions, which some observers consider important.76 Given that access to finance is a significant constraint to small and medium-size enterprises in Mozambique, it is surprising that FTF Inova has not developed relationships with more firms in this area. Some USAID representatives report that addressing finance would not be an effective use of FTF Inova staff time and effort as their abilities are stronger in other areas. The activity has explored VSLAs, limiting work to connecting VSLAs to agricultural input distribution. It is also working with one commercial bank, has maintained a light-touch relationship with another bank, explored mobile payments, and linked partners to finance providers.77 In addition, the activity is working to centralize inventory with wholesalers to enhance efficiency in inventory management and deliver cost savings. FTF Inova states that by stocking inventory on behalf of agro-dealers78 and delivering them just in time, 79 wholesalers are de facto financing the supply chain.80 DAI reports that FTF Inova has supported funding through suppliers' clubs giving 4,445 farmers access to inputs. The activity is also developing relationships with a Weather Index Insurance provider and reports working with a commercial bank to find agriculture-specific financial service packages that reduce the normal requirements for accessing finance. DAI reports FTF Inova has facilitated connections between 10 companies and Absa Bank for asset and working capital finance. During fieldwork Absa representatives reported further exploring five of these connections although final agreements had not been made at the beginning of July 2020. In October 2020, DAI reported that Absa was processing a US$2 million working capital loan for Agrico Marketing that was likely disbursed before the end of 2020 and that Miruku expected to receive US$200,000 working capital and asset finance to upgrade their grain processing plant. Financial institutions interviewed as part of fieldwork suggest that FTF Inova could enhance the 76 DAI notes that FTF Inova has worked with Greenfingers Mobile to design a fintech solution to manage payments in outgrower schemes. 77 DAI report having offered to support Condor Nuts access a financial package and identify an investor. 78 DAI report that FTF Inova has supported financing the stocking system of 200 agro-dealers through just-in-time input delivery 79 FTF Inova Annual Report 2019, 29. 80 DAI comments that the de facto financing amount from a just-in-time delivery system can be approximated by applying the prevailing interest rate to the value of the inventory at any given time. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 42 relevance of the partners it works with by building the capacity of companies using outgrower schemes81 and suppliers with potential to improve client management, stock, and operational plans82 by adopting ICT fintech-based solutions. These observers suggest increasing fintech support to improve credit tracking and repayment. FTF Inova has fewer working relationships in output markets than in input markets and support systems, raising questions about the activity’s targeting and selection of partners. One example is the perceived market opportunity provided by employees working in the oil and gas sector in northern Mozambique. FTF Inova’s partner selection process has not led the activity to work with firms that could supply these markets. In addition, technical assistance supporting FTF Inova suggested that these markets were volatile as worker numbers rise and fall depending on the global economy, weather events, and COVID￾19-related restrictions. They also pointed out that quality requirements were high and difficult for smallholder farmers to consistently achieve. The activity has also not developed partnerships that address regional or international export markets. Fieldwork shows that FTF Inova has addressed opportunities in import substitution only in the last year. Some observers suggest that export opportunities exist in banana, mango, and timber. DAI reports that FTF Inova did explore opportunities in these areas and decided not to proceed with investment.83 The approach has also not forced the activity to work with “difficult” companies that might be vital to the overall market system. Working with such companies may have led to frustration and failure. POSSIBLE SCALING AGENT FUNCTIONS Agent (FTF Inova partner) functioning can be scaled to exceed intended results in farmer numbers in inputs. Less evidence was found of innovations being unexpectedly copied (beyond a relatively small increase in supplier clubs), benefits to others, or recommendations to others being repeated. Other businesses reported developing networks of shops and agro-dealer networks like those facilitated by FTF Inova’s support. However, this development was ongoing when FTF Inova started to support this type of expansion and cannot, at the time of evaluation reporting, be interpreted as copying the supported model. A radio station reported another FTF Inova partner becoming their client unexpectedly and without any intervention by the activity. One unexpected spillover effect was a reduction in conflict between landowners and communities. 81 Outgrower schemes, also known as contract farming, are broadly defined as binding arrangements through which a firm ensures its supply of agricultural products through individuals or groups of farmers. 82 A financial institution reports that “two companies need to improve the presentation of their business plans but have a great potential to be approved. Here is where Inova could play a role.” 83 DAI comments that the activity’s initial value chain analysis considered bananas but agreed with USAID that the plantation scale needed to achieve international competitiveness would not benefit smallholders. The activity ruled out forestry because of the level of illegal logging. A mango study recommended collaboration with Sumo and Compal to develop their strategies for local fruit procurement and exploring market-based approaches to expand crop protection measures. However, Sumo and Compal were not interested in a partnership in this area, and the crop protection companies Inova engaged did not see mangoes as a viable market size. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 43 SUSTAINABILITY It is unclear and too early to say whether the results achieved by FTF Inova are sustainable. FTF Inova emphasizes sustainability as a key advantage and central theme of their approach. Many partners are optimistic that results are sustainable, suggesting that the relationships and linkages that the activity has facilitated will last. One partner reported, “We would miss Inova, but we have learned enough so that we can carry on by ourselves when Inova is gone.” Another partner planning to expand into seed processing using the lessons learned from an FTF Inova–funded market assessment study claimed it does not depend on Inova support. A partner working in radio felt the station “has been around for 12 years and will continue for many years to come. Inova helped us to expand our clientele and we will build on that.” A representative of an FTF Inova–supported supermarket stated that “the time taken to do the market assessment and to build relationships makes the project a market-based solution that will last beyond Inova’s existence.” Another radio station employee felt that supported initiatives would continue beyond the activity’s lifespan. This key informant reported that the linkages facilitated by Inova opened a window that is now being used by others (e.g., seed producers) to market their products. Another partner felt that the connections FTF Inova had facilitated “would continue after Inova is gone, because it did not provide any financial support beside the linkage to customers.” Representatives of larger-scale partners involved in commodity trading and input distribution raised questions about sustainability. One large-scale partner pointed out the low-margin, low-volume markets in northern Mozambique and the challenges to sustainability posed by weather events and health pandemics. Input distributors questioned agro-dealers’ long-term viability and the high cost of setting them up. At the time of fieldwork, COVID-19-related restrictions were significantly disrupting business, delaying payments, and straining relationships. Buyers (including government) were not paying for products delivered, and cash flow/working capital requirements were becoming serious issues and a challenge for sustainability. UNEXPECTED RESULTS FTF Inova has experienced two significant disappointments. The activity has learned from the experiences and maintained relationships with the main parties involved. The first disappointment was with a cashew processor aiming to achieve a quality-related price premium in export markets. FTF Inova aimed to help create a competitive advantage84 by supporting cashew certification and achieve premium prices. Key informants report that “discussion was progressing well but soon the company started experiencing financial difficulties and it had to adjust its priorities. The company relies on investors and financiers for its financial needs, but investors were also facing the same hardships.” As a result, the intervention has been put on hold in the third quarter of 2020. It appears that FTF Inova could not address the financial issues. Capital requirements were highlighted in an FTF 84 DAI report this included investing an initial $17,000 from the FTF Inova activity fund. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 44 Inova stocktaking report (December 2019).85 FTF Inova did link the cashew processor with commercial finance, but the amounts required were significant (approx. $9 million) and not all had been met at the time of fieldwork.86 A second disappointment was with an agrochemical supplier. This initiative aimed to support professional spray services87 bundled with high-quality chemicals and information on their use to increase yields and returns. Partner representatives report that the company invested 11,004,000 MZN (approx. $150,000) in chemicals and then found that farmer numbers were exaggerated; there were not the expected 3,900 smallholder farmers (only 2,324), nor 3,200 hectares (only 2,076 hectares). The price of the promoted package88 to farmers was twice as high as they were accustomed to paying for generic inputs. The spraying service providers started demanding salary and bicycles from the partner, and farmers believed that the drought’s effect on cotton-flowering buds had been caused by the new spraying product and new seed variety. At the end of the season, the region was hit by Cyclone Idai, which caused the germination of cotton inside the bark. The agrochemical supplier had to find an alternative market for the chemicals imported in excess. Some failure is expected in MSD initiatives, and FTF Inova staff report learning the importance of contractual arrangements89 and avoiding rushing implementation.90 The agrochemical example can be seen as a “perfect storm” of weather and poor planning. Both above examples suggest that even with FTF Inova’s claimed careful approach to partner selection, their knowledge of their client was insufficient as investigation did not accurately determine farmer numbers or financing requirements. Key informants for the cashew initiative also felt that FTF Inova changed the initial idea too much. In the cotton example, the partner representative suggested that the package was not appropriate for fields below 2 hectares and that application of the chemical package required great discipline. FTF INOVA AMBITION AND RISK TAKING FTF Inova has a cautious approach when partnering with the private sector. The activity does not partner with government. FTF Inova is ambitious in achieving scale, but it emphasizes sustainability and is willing to invest time to achieve this. It does not aim to achieve quick wins or harvest “low-hanging fruit” as sometimes seen in development programs. The “probe, learn, adapt” approach invests a lot of time and effort in developing mutually respectful relationships that do not always work. The FTF Inova 85 M.A. Bear, “Stocktaking Report on FTF Inova Mozambique Progress in Achieving Impact at Scale 2019,” 28. The processor may need to find financing partners to cover the upfront costs of new supply chain management systems and the working capital required for procuring cashews at much higher volumes. 86 DAI comments that support was ongoing in September 2020 and may see some success. 87 DAI report this included an in-kind commitment of $110,000 from FTF Inova of which $74,000 was expended. 88 DAI report the package increased productivity three- or four-fold in tests. 89 As a result, disagreements arose over smallholder farmers’ debt due to the weather. 90 FTF Inova staff recognize that they rushed the partnership to “catch the season.” DAI report that this may have been one reason that the cotton concessionary and the agro-chemical distributor did not reach a commercial agreement over how to resolve smallholder farmers’ debt in the case of weather issues. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 45 approach of partnering small and building up means that risks do not tend to be taken early but can increase with time. Investing in opportunities is often difficult. FTF Inova looks at market investment opportunities by sharing ideas with a partner business. Potential partners negotiate for support, and private sector expectations can be unrealistic and maintaining relationships challenging. Some partners reported that FTF Inova was not willing to take risks to implement new and innovative ideas (e.g., in insurance). According to one partner, Inova is open to piloting ideas but is not willing to engage in long-term development activities. However, overall evaluation results show that FTF Inova will engage long term if pilots are successful. A clearer limitation of the cautious FTF Inova approach is that if a large-scale high-budget opportunity arises early before the activity perceives a strong relationship as having developed, FTF Inova may not take the associated risk and decline to invest because a pilot is seen as too costly. In this case, the initiative and opportunity will be missed. Interviews with FTF Inova staff show that they maintain their ambition for scale. For example, they recognize the failure with the cotton spray service provider but feel they have learned from it and have maintained good relationships with two out of the three partners involved in the initiative.91 These partners have scale potential, and fieldwork findings confirm that a working relationship with these partners has remained intact FTF Inova continue to recognize spray service provision opportunities in the cashew sector. Donor representatives also report that spray service provision opportunities exist in the cashew sector. If FTF Inova partners remain engaged and planned support is successful, there is potential for good value for money. Planned support involves an FTF Inova commitment of approximately $90,000, with the activity claiming potential to reach up to 50,000 farmers. GENDER GENDER STRATEGY FTF Inova takes a gender mainstreaming approach to its portfolio implementation. In its results framework, FTF Inova has explicitly stated that it will engage with women, ensuring that 35 percent of those benefiting from the activity are women.92 The 2020 gender strategy report also states that “FTF Inova actively identifies and pursues opportunities to work with female-owned firms or firms receptive to adopting a gender perspective in their business. FTF Inova also claims to actively educate agribusinesses on the untapped business opportunities that female producers represent and helps them adopt gender-inclusive practices.” FTF Inova conducted a qualitative study on gender norms in farming, input use, and distribution in Manica Province (August 2018) and fieldwork shows that FTF Inova has engaged some partners to make them more aware of this potential. FTF Inova was working 91 As evidence of learning, DAI reports their belief that these relationships have been essential to two new strategic partnerships. 92 FTF Inova Gender Strategy and Results June 2020, 1. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 46 with nine (approx. 14 percent)93 female-owned or -managed firms, six (approx. 10 percent) of which were active at the time of fieldwork. FTF Inova’s gender strategy has shifted to mainstreaming from an initially proposed more targeted approach described in the May 2017 FTF Inova gender action plan. This action plan outlined a strategy based on a women’s economic empowerment framework for market systems developed for USAID.94 The approach stated that the activity would “identify opportunities for structural transformation and bottom-up change” where change included addressing women’s opportunities and choices.95 The gender action plan listed three activities to identify opportunities and interventions for change: ● Gender norm mapping exercise to recommend pilot entry points for women in specific markets ● Development of women’s empowerment interventions to improve activities ● Gender audit to identify challenges integrating gender into an organization’s or project’s systems and operations At the time of fieldwork, the more targeted approach and activities outlined in the gender action plan had been given a lower priority. As opposed to taking the targeted approach given in the plan, FTF Inova’s strategy has been based on mainstreaming, as presented in the June 2020 gender strategy. Key informants report that this shift arose from a focus on MSD and innovation, with debate on the extent to which an activity can develop gender initiatives when focusing on business and operating with a limited scope (e.g., in thin markets). Evaluation results suggest that “FTF Inova will continue to mainstream gender within all research, outreach, and implementation activities, actively identifying and pursuing opportunities to work with female-owned firms or firms receptive to adopting a gender perspective in their business.”96 The FTF Inova annual work plan 2020 does clearly identify supported innovations that aim to have a gender impact. FTF Inova also includes gender in its partner capacity assessment and has an activity gender screening form to facilitate positive gender outcomes in the initiatives it supports. FTF Inova’s gender strategy is reported as guiding partnership negotiations. Gender-related responses collected during fieldwork were mixed, showing that mainstreaming has not been applied uniformly. One respondent claimed “never [to have] discussed gender with Inova.” However, other respondents contacted during fieldwork reported that “paperwork submitted to Inova always had to have gender at the forefront.” Awareness has also been raised. One company revealed that Inova was instrumental in changing its employment policy, reporting it now favors having a female workforce in its retailer network and warehouses.97 93 FTF Inova partner records provided to the evaluation team suggested that the activity had 63 active or inactive partners. 94 Erin Markel and Lindsey Jones, “Women’s Economic Empowerment: Pushing the Frontiers of Inclusive Market Systems” (USAID, 2014) (https://microlinks.org/sites/default/files/resource/files/WEE_in_Market_Systems_Framework2__508_Compliant.pdf) 95 “FTF Inova (May 2017) Year 1 Gender Action Plan,” 3. 96 FTF Inova emails seen by the evaluation team between DAI and Market Share, May 2020. 97 An MSI gender expert points out the need to clarify when we are talking about women’s economic empowerment (WEE) and when we are talking about gender. WEE is a tool for achieving gender equality but it does not encompass the full meaning of “gender integration.” We can empower the women; however, if no work is done with men and the sector as whole, the USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 47 DIFFERENT IMPACTS ON MEN AND WOMEN Fieldwork found little evidence of different impacts on men and women. FTF Inova reports that the activity is approximately achieving its gender targets (Table 4). TABLE 4. FTF INOVA GENDER INDICATOR Number Indicator Achieved Q2 of FY 2020 FY 2020 Cumulative Achieved FY 2020 Overall Target GNDR-2 Percentage of female participants in USG-assisted programs designed to increase access to productive economic resources 31% 1,224 females/3,943 participants 31% 1,345 females/4,462 participants 35% Source: FTF Inova. Private sector partners reported being asked by FTF Inova staff to disaggregate farmer numbers by gender and age. Some large, active FTF Inova partners reported feeling that numbers of men and women are naturally equated and that gender issues would sort themselves out naturally. Other partners reported FTF Inova demanding and sponsoring participation of female representatives in workshops. FTF Inova is achieving its targets (31 percent achieved in Quarter 2 FY 2020 from a target of 35 percent)98 for its indicator (GNDR-2) of percentage of female participants in USG-assisted programs designed to increase access to productive economic resources. However, review of FTF Inova documents shows that the activity does not generally claim different impacts on women. FTF Inova documents tend to report gender-disaggregated information or describe gender-related activities, such as the partner “organiz[ing] many more gender-sensitive, customer-oriented promotional events across Manica Province”99 as opposed to describing impacts. The December 2019 stocktaking report does not describe different gender impacts, mentioning gender once (p. 9) and women twice (pp. 2 and 46). Interest in gender is growing. USAID representatives report that they would like to see FTF Inova addressing gender, as in the 2017 FTF Inova gender action plan, to give more economic opportunities to women. This goal is similar to that of FTF Inova implementing partner Technoserve’s WIN project, which aims to more directly support business cases where women can co-invest. efforts will not be sustainable. This expert suggests that gender is about relationships and dynamics between men and women; it would be important to understand these dynamics and how perceptions have changed. 98 FTF Inova Quarterly Report Oct. 1, 2019–Sept. 30, 2020, 34. 99 FTF Inova Quarterly Report April 1–June 30, 2019, 11. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 48 COVID-19 ADAPTATIONS IN RESPONSE TO COVID-19 FTF Inova is adapting its strategy and responding positively by looking for opportunities. A stronger link to government would help with the COVID-19 response, as discussed in the recommendations section of this report. The activity has sought approval to adapt its work plan for the remainder of FY 2020. Adaptation has been developed based on three principles: ● Revising activities to minimize disruption from COVID-19-related restrictions by introducing ICT solutions to replace public gatherings or training. Where physical presence in the field is restricted, the project will use media and virtual meetings. ● Revising and adding activities that strengthen market relationships, particularly with smallholder farmers, by embedding health support services, such as providing information on COVID-19, best practices to prevent infection, and access to remote medical consultations. ● Revising and adding activities that improve agribusinesses’ economic resilience by supporting them to identify and take advantage of the business opportunities that moments of change and crisis offer (e.g., supporting local vegetable production to replace normally imported produce). Additional adaptations in response to the COVID-19 pandemic are described in the recommendations section of this report. FTF Inova’s strong connection to its partners allows detailed knowledge about how the pandemic is affecting their businesses. It will therefore be in a strong position to help these partners when businesses reopen. It is recommended that the response to COVID-19 be seen as a process. FTF Inova’s proposed support is consistent with this view. CONCLUSIONS APPROACH FTF Inova is focusing its approach. FTF Inova has implemented a flexible, systems-based, learning￾based approach to support agricultural businesses. The approach emphasizes the co-creation of ideas with the private sector and partnerships that start small, evolve, and deepen. Support focuses on technical advice as opposed to grant subsidies. At the time of fieldwork, FTF Inova intended to focus its approach in three ways:100 ● Prioritizing the number of intervention areas and interventions ● Pursuing larger and deeper partnerships and activities with tested industry partners 100 DAI commented during report finalization that FTF Inova also intends to focus on output markets. DAI further comment that this includes work with Olam, Miruku, Novo Madal, AICAJU, the Mozambique Cotton Association, Ka Da Terra, and Adicional, and launching new partnerships with ETG, VIP-Spar and Five Star. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 49 ● Placing a stronger focus on farm inputs and support sectors (e.g., finance, transport, ICT). FTF Inova has not used a theory of change or results chains to plan, monitor, or describe its approach. Using lessons from implementation, the activity can now better clarify market causality and present a theory of change. FTF Inova’s partner selection has led to partnerships with firms that have the capacity to work with the activity and enough resources to invest in co-created ideas. Evidence was found of frustration with the FTF Inova partner selection process, particularly from smaller potential partners with fewer resources. Mixed results in identification of market constraints and opportunities. FTF Inova shows mixed results in identifying market constraints and opportunities. The activity’s approach has achieved some positive results, indicating correct identification. Other experiences have been disappointing and show that not all constraints were identified or that, assuming profitable opportunities exist as claimed by some observers, the best ones were not addressed. FTF Inova has experienced two significant disappointments as a result of insufficient understanding of constraints. FTF Inova’s active interventions were showing signs of progress, suggesting improving constraint and opportunity identification. Evaluation results show that FTF Inova’s emphasis on the co-creation of ideas led to a focus on working in input and support markets. Investment in output market–oriented opportunities and finance were not emphasized by FTF Inova at the time of evaluation data collection.101 COMPARISONS Comparison with other MSD activities shows that FTF Inova has had a relatively broad focus, with greater implementation flexibility. The activity has given less emphasis to grants and subsidies than have similar initiatives. Comparison with other USAID agricultural investment shows that commodity/value chain initiatives present valuable lessons in achieving impact at scale, using grants, and leveraging connections to government. Opinions and perspectives differed on the value of different approaches, and there is a lack of clear evidence to prove which approaches are most appropriate in a given situation, although field observations indicate long-term lasting impacts and sustainability with the MSD approach. CONNECTION TO GOVERNMENT FTF Inova does not have a clear connection to government and does not emphasize engagement in national policy forums or debates. Comparisons show that working with government in a coordinated, mutually supportive manner can contribute to achieving impact. Such coordination would be especially helpful now amid the pandemic, which is hampering business and cross-border trade. 101 Since performance of evaluation fieldwork DAI have emphasised FTF Inova’s work in output markets. This emphasis is encouraged if market opportunities exist. Proving or disproving these opportunities is beyond the scope of the evaluation. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 50 PROGRESS Progress at activity and output level in input and service markets. FTF Inova has made the most progress at activity and output levels and is not yet effectively delivering significant outcomes. Some key informants report relatively small changes in outcomes, and, at the time of fieldwork, these achievements were being challenged by the COVID-19 response. FTF Inova has made the most progress in input markets and support services (e.g., ICT, radio, market research).102 This progress has occurred in a difficult environment, with multiple factors—macro- and microeconomic, climatic, governance, and disease—hindering business. FTF Inova is performing well in facilitating change in partner and stakeholder practices and has improved relationships. Overall, respondents’ value and appreciate FTF Inova facilitation and its continuous support in developing ideas and fostering team spirit. This facilitation, often seen as time consuming, can lead to significant changes from the ideas presented by the business partner. FTF Inova is not aiming to expand beyond its supported innovations. At the time of fieldwork, FTF Inova intended to amplify innovations with the largest impact, replicate proven models, and concentrate new innovations around a smaller set of partners.103 FTF Inova’s MEL system is strong. It is capturing early signs of change and impact and using them internally and globally in MSD discussion forums. The FTF Inova facilitation approach emphasizes learning. Its MEL system provides the activity with rapid information on early signs of change. The implementation team has used information effectively to limit failure and adapt activities. Internal learning is well used, and the activity makes valuable contributions to global MSD forums. Evidence suggests that the activity is less open to sharing learning in national forums. SCALE It is too early for FTF Inova to be transformational. FTF Inova–facilitated changes were not at sufficient scale to impact the agriculture sector or significantly impact smallholder farmers’ welfare. Fieldwork revealed examples of local and district-level change, but evaluation results show that it is still too early for FTF Inova to be transformational. Impact may be achieved at scale—particularly in input markets—if inputs can reach more customers and allow a lower cost while delivering sufficient profitability. FTF Inova is working with relevant partners in input (e.g., seed companies) and service markets (e.g., transport, radio, and marketing companies). FTF Inova is working with 102 DAI report increased traction in output markets as arising since evaluation fieldwork. 103 DAI comments on the midterm evaluation draft report are that “while it is true that in its final 18 months, the project does not plan to launch many brand-new activities that will not bear fruit before the end of the project, it remains open to new innovations.” USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 51 appropriate firms in the agriculture market system, including seed companies and input distributors that can be scaled to provide inputs to many farmers. SUSTAINABILITY It is too early to determine the sustainability of FTF Inova’s results. FTF Inova staff emphasize sustainability as a key advantage and central theme of their approach. Many partners are optimistic that results are sustainable, suggesting that the relationships and linkages the activity has facilitated will last. UNEXPECTED RESULTS FTF Inova has experienced two significant disappointments using its MSD approach. The activity has learned from the experiences and maintained relationships with the main parties involved. FTF Inova is relatively cautious when partnering with the private sector and avoids taking significant risks early in its relationships with partners. FTF Inova has a cautious approach when partnering with the private sector. The activity does not partner with the government. FTF Inova is ambitious in achieving scale, but it emphasizes sustainability and is willing to invest time in probing, learning, and adapting to achieve this. Partnerships do not always succeed, and the activity aims to learn from its mistakes. FTF Inova’s contribution to partnerships is prudent, starting small and growing with experience. Risks are not taken early but can increase with time. If large-scale, high-budget opportunities arise early, the FTF Inova approach may decline to invest because a pilot may be too costly; opportunity can thereby be missed. Targeting gender. FTF Inova has shifted from an initially proposed targeted approach in its 2017 gender action plan to gender mainstreaming, now presented in its June 2020 gender strategy.104 Some innovative gender activities have been performed, although the degree of focus on gender varied from partnership to partnership. No differences were found in impact from the activity on men and women. COVID-19. FTF Inova is well connected to the partners it works with and in a strong position to help them through the crisis. FTF Inova is proposing valuable adjustments in its support, such as the following: ● Introducing ICT solutions such as media and virtual meetings ● Embedding health support services—for example, providing information on COVID-19, best practices to prevent infection, and access to remote medical consultations ● Supporting identification and pursuit of business opportunities that may arise amid the crisis, such as supporting local food production 104 Feed the Future Mozambique Agricultural Innovations Activity (FTF Inova) Gender Strategy and Results June 2020. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 52 RECOMMENDATIONS FTF Inova should continue with plans to focus on input and support markets and develop a theory of change to explain causality. FTF Inova should continue with its plans to focus on farm inputs and support sectors (e.g., input distribution, spray services, and radio). FTF Inova should develop an overall theory of change and present individual results chains for intended focal areas. The theory of change/results chains (see Annex H) should be linked to indicators that show progress, especially at outcome level (e.g., reduced cost of inputs to farmers, increased productivity, increased income). FTF Inova should further develop its interest in output markets. FTF Inova should provide more information on its work in output markets in its regular reports showing how they will contribute to scale. This should include information on the commodities or products involved, who the buyers are in which output markets, estimates of sales and market size in volume and value terms, trends in growth, growth forecasts and an analysis of margins and investment returns for the partners it supports.105 This should be monitored to show investment returns. FTF Inova should review its communication strategy. As part of developing its theory of change, FTF Inova should review its communication strategy. FTF Inova should produce materials that communicate more effectively to non-specialist/non-MSD practitioners. This could include having complementary versions of the theory of change for internal/specialist use and external/non-specialist use. The external version—e.g., with an infographic style approach—should avoid technical language and communicate clearly with non-MSD practitioners. USAID should conduct an independent ex-post evaluation. USAID should hold an independent ex-post evaluation approximately two years after FTF Inova ends. This evaluation should assess activity scale and sustainability. It should also aim to inform the debate around systems versus sectoral/sub￾sectoral approaches and the use of grants when trying to achieve sustainable impact at scale. FTF Inova should strengthen its linkage to government. To ensure consistency with local government policy, FTF Inova staff should visit appropriate provincial government representatives to inform them of their activities. At a national level, FTF Inova needs to work in close collaboration with USAID and SPEED+ to ensure complementary effort. FTF Inova support to local food production and/or trade needs to be consistent with cross-border trade policy, particularly given the COVID-19 context. FTF Inova should enhance the use of consortium strengths. The FTF Inova incoming chief of party should reach out to FTF Inova consortium members and look for ways to enhance partner strengths, particularly to make better use of partners’ experience and knowledge. All partners should be involved in designing the work plan, reviewing strategy, sharing, debating, and developing learning to inform MSD best practice. 105 i.e. Olam, Miruku, Novo Madal, AICAJU, the Mozambique Cotton Association, Ka Da Terra, and Adicional, and launching new partnerships with ETG, VIP-Spar and Five Star. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 53 FTF Inova should continue to strengthen its valuable contribution to national-level learning about economic development. Participants in national-level forums feel that FTF Inova should continue to increase its contribution to national-level learning—for example, to the MSD practitioners group, where members feel they could learn more from FTF Inova and other economic development forums to share learning and inform others working in a similar field. To support future intervention design, FTF Inova should report on the implications of its partner selection process, its support to finance, and its levels of investment in output market opportunities. The FTF Inova end-of-program report should reflect on the positive and negative implications of its partner selection process. This reflection should consider whether small and medium-size enterprises with good ideas but insufficient resources to contribute to partnerships were excluded and whether large-scale opportunities were missed due to the low-risk approach. The report should reflect on how and why the activity has evolved most strongly on input and support markets and consider the value of an initial broad-based market assessment. For example, how were the value chain studies used? Did they identify investable business opportunities? Did they help to find private sector partners and, if so, how and which ones? The report should also explore why the activity’s support to finance and output markets did not evolve significantly, consider the implications of this, and provide recommendations for future MSD activity design. FTF Inova should apply its gender strategy uniformly. FTF Inova should apply its mainstreaming gender strategy uniformly across its portfolio. It should also assess and specifically invest in targeted opportunities to support women. The MEL team should monitor this and ensure managers are discussing it with partners and including gender considerations in all interventions. The activity should also revisit its 2017 gender action plan to pilot entry points for women in specific markets to develop clear, targeted gender interventions. FTF Inova should inform others on additional COVID-19 adaptations in national-level communication and finance. FTF Inova should be “on the agricultural business pulse” and inform others on what they should do and where they should go to support the response to COVID-19. This needs to go beyond USAID resources and inform other key actors such as the World Bank and African Development Bank. Communication should also inform government on shifting markets and changes to cross-border trade and support public-private dialogue, such as advocating for government change (e.g., adjusting transport restrictions and supporting scenario planning). Agricultural businesses will require rapid access to finance to meet working capital requirements. One FTF Inova commercial banking partner reported that they had lost physical contact with clients and it was hard to understand clients’ situations. Managers of some companies are stranded abroad and are not aware of the real situation. Again, FTF Inova can help through its relationships with DAI and Technoserve and connections with businesses and financial institutions. FTF Inova and its partners should perform more work to understand needs and potential sources and methods of finance. Following this analysis, appropriate support should be provided, including grants. ANNEX A: SCOPE OF WORK USAID has worked closely for years with partners such as private sector companies and associations, civil society organizations, and the Government of the Republic of Mozambique (GRM) to increase investment and incomes in the agriculture sector. The GRM has set forth an ambitious agenda to increase equitable, pro-poor agricultural growth and sustainably reduce poverty and hunger. Aligned with GRM’s vision, USAID/Mozambique’s Feed the Future Mozambique Inova (FTF Inova) activity seeks to provide the framework for ensuring that successes in Mozambique’s agricultural sector are enduring, that market actors are resilient, and that women benefit from growth in the agricultural sector. The USAID Mozambique Feed the Future Agricultural Innovations Activity (FTF Inova) is a five-year (2017- 2022) project that started in February 2017. FTF Inova takes a market systems development (MSD) approach to agriculture-led economic growth in Mozambique that contributes to USAID’s Global Food Security Strategy. In its first two years of implementation, FTF Inova developed strategic relationships with market actors (i.e. input distributors, seed suppliers, agro dealers, agro processors) to test innovative business models and new technologies. The innovations promoted by FTF Inova aim to improve the incomes of Mozambique’s smallholder farmers (SHFs) and drive long-term, sustainable, and inclusive growth in the agriculture sector. FTF INOVA has adapted its approach significantly as it has rolled out, from initially being focused on input distribution system only to taking the more holistic MSD approach. Segmenting interventions by one market system only was felt to “lead to missed opportunities to leverage interesting ideas with high potential to catalyze systemic change.”106 FTF INOVA’s implementation approach now tries to facilitate change in the wider market system to achieve sustainable outcomes. FTF Inova is implemented by the DAI Consortium, including DAI Global, LLC, with sub-partners Technoserve and MarketShare Associates. Hereafter this group will be referred to as “the implementing partner,” or “IP.” DAI previously led a similar agricultural project, INOVAGRO, funded by a different donor. This evaluation occurs approximately 65% through the implementation period. 106 Feed the Future Mozambique Agricultural Innovations Activity Quarterly Report (January 1 through March 31, 2018) Maputo, April 2018, p. 6. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 55 SUBJECT ACTIVITY DESCRIPTION 1.2.1 ACTIVITY AT A GLANCE FIGURE 1. ACTIVITY FACTS (SOURCE: FTF INOVA FACT SHEET AND WORK PLAN) Name Inova: Feed the Future Mozambique Agricultural Innovations (FTF Inova) USAID Contract Number AID-656-C-17-00001 Funding USD $20,971,049 Activity Period February 21, 2017 to February 20, 2022 Implementing Partner Team (IP Team) DAI Global, LLC, with sub-partners Technoserve and MarketShare Associates (the “DAI Consortium”) Feed the Future Goal Reduce hunger and poverty in target areas of Mozambique Feed the Future Purpose Increase equitable growth of the agricultural sector FTF Inova Strengthening sustainable private commercial relationships using a Market Systems Development Approach Geographic Focus Feed the Future Zone of Influence: 26 Districts in Cabo Delgado, Nampula, Zambezia, Tete, and Manica Provinces expanded to where market opportunities exist. USAID/Mozambique Office Agriculture, Environment and Business (AEB) USAID Contracting Officer’s Representative (COR) Elsa Mapilele Chief of Party (COP) Luca Crudeli 1.2.2 FTF INOVA ACTIVITY PURPOSE FTF Inova’s overall goal is for sustainable, agriculture-led economic growth, which is critical to reducing poverty and hunger. The aim of FTF Inova’s market systems approach is to catalyze systemic changes, which are changes in the way core functions of supply and demand, supporting functions, and rules perform to ultimately improve the participation of target groups (such as people living in poverty, youth, marginalized groups, etc.) within the market system.107 FTF INOVA aims to increase equitable growth and incomes in the agriculture sector in Mozambique by stimulating purposeful business innovation and investment, expanding the number of enterprises that can 107 FTF Inova. Monitoring, Evaluation, and Learning (MEL) Plan. May 2019. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 56 compete and upgrade their products and services in selected markets, and improving relationships and linkages between those firms and other market participants throughout the agricultural market system. FTF INOVA works with selected private sector partners to develop and implement interventions in “specific market functions that exhibit a credible pathway towards inclusive change in the agricultural market system.”108 Partners are selected through an Annual Partnership Program Announcements (APPA) that invite applicants to present their market development ideas. Successful applicants are invited to enter a strategic partnership arrangement called Market Actor Partnership Agreements (MAPA), similar to a memorandum of understanding. FTF INOVA managers then work with the private sector partner to develop ideas. FTF INOVA provides support to successful applicants in: ● Technical assistance and training. ● Co-financing with farmers and agribusiness to promote investment. ● Leveraging activity funds to strategically buy down risk. FTF INOVA manages each successful applicant within portfolios that include: input distribution network systems (i.e., input suppliers), supply chain management system (i.e., cashew and maize output buyers), interconnected systems (i.e., agricultural radio programs), and business services system (i.e., finance, Information and Communication Technology [ICT], and logistics). 1.2.3 SUBJECT ACTIVITY THEORY OF CHANGE The Inova MEL (monitoring, evaluation and learning) Plan does not include a results framework, a graphic useful to evaluators shaping an inquiry of USAID activities. In fact, as stated in the 2018 to 2019 MEL Plan “FTF Inova does not follow a traditional Theory of Change approach.” The FTF INOVA MEL Plan recognizes an area of uncertainty between its work with partners and its vision of improved market systems where causality is not clear or predictable. FTF INOVA refers to this as the ‘black box’ of causality.109 FTF Inova therefore aims to gradually “unpack the black box” of complex market behaviors by testing and collecting evidence on business innovations during each intervention.110 108 Feed the Future Mozambique Agricultural Innovations Activity Quarterly Report (January 1 through March 31, 2018) Maputo, April 2018, p. 6. 109 The current Inova probe–sense–respond approach lacks the clarity provided by a traditional logical hierarchy based on activities, outputs and outcomes. Inova report that the uncertainly inherent in their approach means causality is unclear and using a logical flow of activities, outputs and outcomes is / was inappropriate or impractical. The mid-term provides an opportunity to explore this and potentially generate some outputs and outcomes that reduce any vagueness and / or ambiguity. 110 FTF INOVA Monitoring, Evaluation, And Learning (MEL) Plan Submitted October 2018 (version 2) Revised May 2019 USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 57 This mid-term evaluation presents an opportunity to learn through “unpacking the black box of causality” in Mozambique’s agricultural market system. The evaluation will contribute to FTF Inova and USAID learning about the activity’s causality and support development of a Theory of Change while enough implementation time remains for learning to be beneficially applied. EVALUATION CONTEXT This SOW is written as Mozambique is under a Level Three State of Emergency response to the COVID-19 Pandemic, which effectively eliminates the option of international staff and consultants travelling to Mozambique, internal travel in the country, face-to-face key informant interviews (KIIs), in￾person group interviews or workshops, and in-person debriefings. The methodology included in this SOW attempts to address these challenges by undertaking a remote evaluation responding to the evaluation questions. STATEMENT OF WORK EVALUATION PURPOSES The purpose of this external mid-term evaluation is to assess progress to date under this activity in order to inform the IP and USAID of any possible course corrections. This is in keeping with best practice for Monitoring and Evaluations and particularly the approach is in keeping with the Agency’s Collaborating, Learning, and Achieving (CLA) approach. Given the experimental nature of the FTF Inova activity, it is particularly useful to apply this type of iterative learning approach to this activity. To assess the effectiveness of FTF Inova’s MSD approach the mid-term evaluation will, contrast it with other similar approaches, gauge progress at slightly past the midpoint, and suggest improvements to maximize likelihood of success in the remaining period of the activity. The review will assess FTF Inova processes for capturing and documenting lessons learned, including feedback from the perspective of partners and stakeholders. Assess the extent to which these systems are used to refine FTF Inova activities, adjust implementation, and enhance activity efficiency or effectiveness and ensure that the activities are relevant to agricultural systems transformation. EVALUATION OBJECTIVES AND LEARNING AREAS FIGURE 2. EVALUATION OBJECTIVES AND LEARNING AREAS Evaluation Objectives Learning Areas Understand FTF Inova’s approach to MSD, understand its effectiveness, and how it compares to similar efforts elsewhere. 1. Clarify the causal logic (unpack the black box)– and related interventions – of the activity so that the reader can understand the FTF Inova approach to market system development (MSD). USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 58 2. Assess the extent to which the MSD approach used by the activity have: a) correctly identified market constraints and opportunities, and b) gathered evidence on market causality that can inform future investments. 3. Compare the FTF Inova MSD approach with other interventions using MSD approaches in Mozambique and elsewhere (i.e., Honduras, etc.). 4. Compare and contrast Inova’s approach with other USAID agriculture investments in Mozambique, such as EMPRENDA, AGRIFUTURO, P4I, RAMA NC, etc. What are opportunities for the future? 5. How is Inova building on the strengths of its partner organizations (i.e., TechnoServe), besides receiving staff from these organizations? Understand FTF Inova’s progress towards improving agricultural outcomes in Mozambique 60% into implementation. Describe the progress in each of the links of the causal and/or market chains specified in A.1. What factors appear to be promoting and hindering progress? How well is FTF Inova facilitating change in partners’ and stakeholders’ practices leading to market opportunities? Is FTF Inova aimed at expansion beyond the supported innovations (systemic market change)? How well is the MEL system capturing whether there are any early signs of impact from interventions? How effective is the team at learning and using learnings to improve activities and how they are being implemented? Identify key lessons learned and 1-2 success stories. Gauge whether the changes fostered by FTF Inova appear to be significant in the development context of Mozambique? Are changes currently at sufficient scale to impact Mozambican agriculture sector and smallholder farmers welfare? Is Inova working with and/or relevant to appropriate firms within the agricultural market system that would position interventions to be relevant to large numbers of farmers? Does it appear that improvements in Agent functioning as a result of FTF Inova can be scaled to go beyond its intended results?111 Do the results achieved by FTF Inova appear to be sustainable and are they likely to continue beyond the project life span? Are there any areas in which the MSD approach used by FTF Inova appears to have been less impactful/failed? 111 These can include spill over effects, being unexpectedly replicable, benefits others, recommendations to others or being repeated. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 59 Understand whether FTF Inova is ambitious when partnering with the private sector and government to take risks and develop transformative agricultural opportunities?112 Understand the gender implications of FTF Inova. What is FTF Inova’s gender strategy? What are the differential impacts on men and women? Consider the above questions in the context of the COVID-19 Pandemic What adaptations in strategy and implementation tactics have already been made? What additional adaptations are needed for future strategies in response to the pandemic Does FTF Inova offer any unique traits that place it particularly well for helping Mozambican farmers and businesses restart after the economic effects of the COVID 19 virus subside? METHODOLOGICAL APPROACH Collaborative Process The evaluation is expected to apply a consultative approach, using both quantitative and qualitative methods for data collection and analysis. Given COVID-19-related travel and meeting restrictions in Mozambique, innovative approaches will be needed to achieve the evaluation purpose. These include employing a consultative process (discussed in greater detail, below) and using on-line virtual group/team meetings, phone-based interviews and video. Combined with the use of secondary data, this innovative approach can provide a rich, informative high￾quality product to support the choices made by management decision makers. It is also expected to contribute to learning about ‘remote-based’ virtual forms of evaluation and how they can enhance value for money and minimize climate-related impacts while retaining sufficient quality to support mid-term decision making. The team will use MS Teams software to manage all collaboration in an efficient and transparent manner. This includes documents storage, meetings, chats, online simultaneous document editing, video conferencing (with breakout rooms). Other food security USAID-funded Activities have found that evaluation teams which draw on staff from IPs increase the probability of feasible recommendations being produced. The evaluation will, therefore, actively engage staff from the three organizations of the IP. This will be achieved by using Most Significant Change (https://www.betterevaluation.org/en/plan/approach/most_significant_change) and Appreciative Enquiry (https://www.betterevaluation.org/plan/approach/appreciative_inquiry) techniques, coupled with traditional analytic approaches, that: 112 This includes examining how FTF Inova looked at the range of market investment opportunities, the ease and /or difficulties involved in investing in these opportunities and exploring who, how and why partners were chosen. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 60 Engages IP staff during the inception phase to fine tune design of the evaluation. IP staff will advise on what data and information should be collected. For example, FTF Inova staff will advise and provide recommendations on which investment activities and supported actors should be consulted in the mid￾term performance evaluation based on how long they have been underway and their potential to address evaluation questions. A cross section of supported activities that show what is working well, what is average and what is underperforming will be selected. Inviting one member from each of the IP member organizations to periodically review progress. This will serve to maximize the quality of data available to the team and build the organizational commitment to implementing the results of the evaluation. This SOW is drafted with that approach in mind.113 The approach is meant to maximize learning and use while ensuring that the process is led by an external perspective, as desired by the Mission, while being supportive of the FTF Inova MEL Plan. Holding regular feedback/verification events on-line that share and discuss information with stakeholders and contributors to explore causality within the FTF Inova ‘black box’ so that learning happens about what is valued. These events will explore potential results, and how things might work well in the future. This will feed into planning and prioritizing processes by exploring what worked well, learning and how to adjust/adapt further to sustain, build and enhance change. Knowledge generated from the mid-term evaluation will help to align FTF Inova’s interventions to capitalize on opportunities for greater impact. Findings from the mid-term will also inform FTF Inova support interventions that are not directly assessed. The evaluation will also draw on FTF Inova’s dedicated team of MEL personnel and its system for ongoing data collection and analysis. Program MEL output and outcome data will be drawn on where possible. DATA SOURCES The evaluation will consider both qualitative and quantitative data. Qualitative data will include secondary data and primary information from interviews and group discussions (GDs). Primary qualitative data sources will include USAID, the IP, MSD Agents, farmers, GRM officials, and other stakeholders and partners at central, provincial and local levels. Quantitative data will largely be existing secondary data from Agents (i.e. those receiving In FTF Inova support) production and sales data, activity data, reporting documents, useful technical references and farmer data, to the extent available. 113 But, lesser engagement by IP members is also possible, if the IP is no longer committed to what was stated in the MEL Plan, such as (i) joining in some interviews and GDs that collect information; (ii) engaging in analysis; and (ii) determining which information is most significant through collaborative feedback briefings. The evaluation is likely to be slightly less useful with such compromises, however. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 61 Data Collection and Tools The focus of all data collection activities will be on the learning areas in Figure 2. Qualitative data collection will include virtual group interviews and virtual key informant interviews, based on interview guides approved by USAID. Working in consultation with the IP, the evaluation team will develop instruments to guide group and individual interviews, guidelines to register the information collected in the interviews and a preliminary Findings, Conclusions and Recommendations (F/C/R) matrix that will assist the compilation of data collected from primary and secondary sources. The data collection will be supported by notes from individual interviews and GDs, copies of documents, and other record forms. Interviews will be recorded digitally, as possible, and written records of all interviews will be prepared within a day of the interview and stored for sharing among team members for eventual uploading to USAID. The effort will rely heavily on information and communication technology (ICT.) MS Teams will be used innovatively to capture questions and answers through text, voice recording and messaging functions. Opportunities will also be maximized to collect and use video evidence. For example, where appropriate agents with camera phones will be invited to make short videos or provide pictures that show FTF Inova’s work. The evaluation team will collect relevant activity data and documents, such as MEL plan data, quarterly reports, technical products and other secondary data to judge performance and access effectiveness of the FTF Inova M&E system. Sampling It is anticipated that data collection will focus on the geographic areas where FTF Inova operates. The initial selection of interviewees and discussion group participants will be included in the work plan for approval by USAID. ANALYSIS METHODS AND DATA DISAGGREGATION Quantitative Data Analysis of quantitative data made available by FTF Inova and Agents will depend on how the data have been captured. Ideally, the data will be in a coded file (Excel format or similar), which could be imported into a data analysis software tool and analyzed for standard quantitative measurements such as: frequencies, mean, median, mode, and other inferential statistics as appropriate and relevant to the data analysis. Quantitative outcomes will be reported using tables, charts, and graphs in the evaluation report. Data will be disaggregated by sex and social demographics for each indicator examined, as feasible and useful, and be reported as such in the final evaluation report in cases where the findings vary significantly by USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 62 demographic cohort. This will depend on the nature of data for disaggregation and respondent size under each of the disaggregated groups. Qualitative Data The qualitative data gathered through the KIIs and GDs will be analyzed using systematic qualitative methods to determine common themes and patterns in the data. These themes and patterns will be cross-compared and triangulated to develop a picture of what has occurred, and is occurring, under each evaluation question. Where possible, demographic information, such as gender, will be considered during the analysis process and included in the evaluation report. Qualitative data used in the evaluation report will generally be anonymized, in order to ensure respondent confidentiality. Respondents’ sex and data collection location will be identified with any specific quotations, in order to provide context for the quotation. General aggregation of qualitative data will be enumerated but not quantified. Overall Analysis Overall data analysis will be organized in a preliminary F/C/R Matrix, based on evaluation questions. As data are analyzed, findings will be aggregated in this (a ‘rolling’ F/C/R Matrix). This will allow comparison, triangulation, cross checking and the identification of any gaps. This synthesis can be regularly shared and jointly reviewed to verify findings as they are aggregated. Primary data will be combined with available secondary data in the preliminary F/C/R Matrix through the crosschecking process to compare initial results and determine patterns and themes. Where patterns and themes differ, additional data will be sourced. Consistent patterns and themes will be developed into tentative findings. As additional data is produced and analyzed these tentative findings will become stronger and initial conclusions developed. As the matrix is developed, its contents can be easily drawn upon and used in report writing and feedback events. This iterative feedback process involving IPs and USAID should help ensure information exists to answer all evaluation questions and fully address cross-cutting issues. GENDER APPROACH Consistent with USAID’s evaluation policy and recognizing that effects of integration and the success of the FTF Inova activity might vary across gender, a gender perspective will be applied to the entire evaluation process. Starting with the background document review, evidence related to gender-based differences in outcomes will be examined, and project documents will be reviewed with a gender￾sensitive lens, in order to inform a better understanding of gender dynamics in the implementation environment. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 63 A gender balance will be maintained among respondents during the recruitment phase, especially among beneficiaries. Since most interview teams will include at least one-man, female respondents will be given the option to withhold responses if they feel uncomfortable at any time. Gender balanced GDs will be organized in order to encourage active participation by all respondents, and to limit a potential for one gender to dominate the conversation at the expense of another. In some cases, the team may decide to separate men and women in GDs to maximize engagement. During the data analysis phase, all participant-level data (both qualitative and quantitative) will be disaggregated and reported by sex. Analysis and discussion of findings, and where appropriate, conclusions and recommendations, will be included in the final evaluation reporting (both written and oral). EVALUATION PROCESS The evaluation is designed for a four-month period, beginning April 2020. Remote preparation will require one week; data collection and analysis will last six weeks, with a draft report delivered within two weeks of data collection ending. External Team Component Preparation The External Component of the team will conduct a desk review of available literature, including activity documents, survey reports, research and other relevant documents identified by key stakeholders as well as broader MSA literature on best practices and lessons learned worldwide. The evaluation team is expected to engage beneficiaries and reliable observers that have valuable insight into FTF Inova’s work. These may include government officials, local NGOs (including NGO’s working on gender issues and agriculture), implementing partners, subcontractors, and other beneficiaries, partners and stakeholders in order to acquire the data needed to respond to the evaluation questions. Interviewees, including beneficiaries, will be invited to take phone videos, where appropriate, that show FTF Inova’s work. Will also talk to people who have led and worked on other USAID agricultural projects in Mozambique. Inception Report: Preliminary Evaluation Work Plan and Design Within two weeks of beginning the desk review, an Inception Report, including preliminary design for the evaluation, will be submitted to USAID/Mozambique. The report will include suggested task timeline, logistical arrangements, methodology outlining the approach to be used in answering each evaluation question, team responsibilities, data analysis plan, and report writing tasks and timeline. But, in preparing this material, most evaluations have the benefit of a clearly articulated theory of change (TOC.) As there does not yet appear to be one drafted (if there is, the TL will use that TOC) the TL will draft a TOC for purposes of the evaluation, based on information available. This will be fine-tuned in the Team Planning Meeting, described below. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 64 The evaluation design will be finalized in discussion with the full evaluation team (including the IP and USAID). The evaluation design will include: (1) a detailed evaluation design matrix that links the evaluation questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their features; (3) the list of potential interviewees and proposed selection criteria; (4) known limitations to the evaluation design; and (5) a dissemination plan. The External Component of the evaluation team will sign Disclosure of Conflict of Interest forms prior to engagement. The evaluation team will need to have online meetings with the Mission to discuss the SOW in its entirety, broadly describe team members’ role and responsibilities, discuss the process for devising a work plan, develop data collection methods, review and clarify any logistical and administrative procedures for the assignment and instruments, and prepare for the in-brief with USAID/Mozambique. Team Planning Meeting (TPM) The Inception Report represents the External Component’s best effort to understand FTF Inova, design the instrument, and consider data sources. Within a few days of sending that report to USAID and the IP, the TL will convene a TPM of the entire evaluation team so that all can understand how this innovative evaluation will be conducted and to consider all the material in the Inception Report and improve it, in all areas, based on the larger team’s understanding. The product will be the operational work plan, including all elements of the Inception report – now finalized. Although it is considered “final” at this point, the team is likely to adapt some of its elements during implementation. This is true of any work plan but is especially true in the case of evaluations with an innovative approach, such as this one. Critical points in this process will be for the team to agree upon a TOC to use for the evaluation and to clarify roles, responsibilities, communication etiquette, and time commitments of each team member. In-brief Presentation Within two days of the TPM and based on the finalized work plan and methodology developed in the TPM, the evaluation team will then hold an official virtual in-briefing with USAID/Mozambique and IP. Data Collection The team will embark on virtual KIIs and virtual GDs, as outlined in the work plan. There is likely to be additional secondary data incorporated into the study at this time as well. Interim Briefings The evaluation TL (or his/her delegate) will brief USAID on progress with the evaluation on at least a weekly basis, via email, transmitted by MMEMS. Any delays or complications will be communicated to USAID/Mozambique as quickly as possible to allow quick resolution and to minimize any disruptions to the evaluation process. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 65 The TL may also hold feedback meetings with USAID during data collection and initial analysis, as needed. These feedback meetings will be based on emerging learning from the F/C/R Matrix. This matrix can be shared for joint review using MS Teams. These feedback meetings will provide opportunities to define results, learning and jointly refine findings, conclusions and recommendations. Debrief Presentation The evaluation team will present the major findings from the evaluation to USAID/Mozambique and partners through a PowerPoint (or similar) presentation again using MS Teams. The debriefing will cover preliminary findings, conclusions and recommendations. Feedback will be incorporated into the draft report. Draft Evaluation Report The draft evaluation report will be consistent with the guidance provided in the following section and submitted to USAID/Mozambique within two weeks after the debriefing. The written report should clearly describe findings, conclusions, and recommendations. USAID will provide a single set of consolidated comments on the draft report within five working days of submission. The draft report will also be shared with the IP for a single set of integrated comments and an opportunity to disclose a statement of difference, if applicable. REPORTING AND DISSEMINATION Evaluation Report MMEMS will submit a final report that incorporates the Mission’s and IP’s comments and suggestions no later than ten days after USAID/Mozambique provides a single set of written comments that integrate all Mission feedback on the team’s draft report. All evaluation data and records, including copies of secondary quantitative data sets, transcripts of interviews and focus groups, and notes from direct observations will be submitted to USAID in electronic form in easily readable format, organized, and documented for use by those not fully familiar with the intervention or evaluation and owned by USAID. The draft evaluation report should meet the following criteria: ● The report should be in line with USAID Evaluation Policy (see Appendix I – Criteria to Ensure the Quality of the Evaluation Report) and USAID Secretariat Style guide. ● The report will be in English and will be no longer than 30 pages, excluding executive summary, table of contents, and annexes. ● The report should include a 3-5-page Executive Summary highlighting findings and recommendations. ● The report should represent a thoughtful and well-organized effort to objectively respond to the evaluation questions. ● The report shall address all evaluation questions included in the SOW (see Figure 2). USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 66 ● Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides shall be included in an Annex in the final report. ● Limitations to the evaluation shall be disclosed in the report, with attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). ● Evaluation findings should be specific, concise and supported by strong quantitative or qualitative evidence. ● Recommendations should be action-oriented, practical, specific, and evidence-based. ● The final report should be edited and formatted. ● MMEMS will submit the final evaluation report to the Development Experience Clearinghouse, once clearance is provided by the Mission. ● The final evaluation report must be 508 compliant and comply with the USAID Evaluation Policy: http://www.usaid.gov/ sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Findings and Recommendations Workshop MMEMS will facilitate a Findings and Recommendations Workshop one-three months after submission of the final evaluation report, which will include USAID, the IP, and relevant stakeholders. This will provide an opportunity to engage USAID and stakeholders in verifying that recommendations were implemented and to identify further opportunities for application of what was learned in the evaluation. This engagement will lead to a more tailored plan on how to collaboratively apply evaluation lessons directly to relevant decision making, including policy development, and larger sectoral assessment needs. The workshop will be a one-day session, facilitated by MMEMS, with the aim of maximize use of performance evaluation results. Learning Document MMEMS will produce a concise learning document detailing the innovative ‘virtual’ collaborative methodology used, challenges faced, and results achieved. The document will also perform a comparative analysis with conventional physical field-based evaluations to facilitate learning and inform future evaluations. Deliverables The process and deliverables are presented in Figure 3, together with anticipated LOE for each team member throughout the process, to aid in planning. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 67 FIGURE 3. EVALUATION PROCESS, DELIVERABLES, AND ESTIMATED LOE Note: TS = Technoserve; ML = MarketLeader; SEA = Senior Evaluation Advisor (MMEMS) Stage/Process Deliverable Specifications Responsible; with estimated LOE in () Timing Preparation / Inception Collaborative planning Virtual Team Planning Meetings External Component meetings Facilitated by IM First week External Component and IP meetings Led by TL (4), with engagement by MASA (3) and SEA. USAID (2), DAI (1) TS (1) MS (1) Preliminary list of potential KIIs and GDs Table of persons to be interviewed TL (1), in consultation with MASA (1), with engagement of USAID (1), DAI (1) TS (1) MS (1) First week List of Organizations to be contacted Review activity documents Summary of findings from review of secondary data Word document, in English TL (5), MASA (3), DAI (1) TS (1) MS (1) Second week Inception Report with Detailed MTPE Work Plan Building on this SOW, it should include the following sections: TL, in consultation with MASA End of week two ∙ Overview ∙ Purpose ∙ Methodology ∙ Chronogram ∙ Detailed evaluation matrix ∙ Risks and limitations Data Collection and Analysis Data collection Perform virtual KIIs and GDs TL (15), MASA (15), DAI (1) TS (1) ML (1); USAID (3) Weeks three, four, five and six. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 68 Analysis Preliminary findings matrix Develop rolling preliminary findings matrix TL (5), MASA (5), DAI (1) TS (1) ML (1) Weeks four, five, six and seven. Debriefing and feedback events Preparation and performance of de￾briefings with USAID / other stakeholders TL (3), MASA (2), USAID (1) DAI (1) TS (1) ML (1) Week seven / eight Report Writing Draft report Format as described above TL (10), MASA (6), USAID (3), DAI (1) TS (1) ML (1) Week seven and eight Final report Iterative finalization of report following feedback TL (2) Week nine and ten Post Evaluation Report Develop Learning Document Learning document Document detailing learning from applying an innovative ‘virtual’ collaborative methodology to evaluation of an activity using the MSD approach MMEMS staff Weeks nine and ten Findings and Recommendations Workshop Tailored plan Tailored plan on how to collaboratively apply findings and lessons learned MMEMS staff; USAID (1), DAI (3) TS (3) ML (3) One to three months after agreement of final report USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 69 RECOMMENDED COMPOSITION OF EXTERNAL COMPONENT OF TEAM The External Component of the evaluation team will be comprised of short-term evaluators and MMEMS full-time staff. MMEMS will provide the required logistical support for these individuals. The evaluation design and LOE assume that all tasks listed in the previous section will be conducted. MMEMS Technical Oversight (MMEMS Senior Evaluation Advisor). An international professional experienced in managing USAID evaluation projects, including knowledge and expertise in M&E systems, collaborative approaches in evaluations. She will be responsible for the overall client management, provide quality assurance and technical oversight. MMEMS Management Support (Intervention Manager). A MEL Specialist will provide management, logistic, and technical support (including participating in some data collection and analysis, as needed.) Team Leader. An international expert in evaluation and Market Systems Development. He/she will be an independent thinker and the technical lead responsible for the evaluation design, developing the evaluation work plan, implementation of the work plan, ensuring data quality and leading analysis and report writing. (Estimated LOE = 45 days) Mozambique Agricultural Sector Specialist. A Mozambican agricultural expert familiar with market approaches to agricultural interventions. He/she will participate in the same areas as the TL, under his/her guidance. (Estimated LOE = 35 days) USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 70 ANNEX B: PEOPLE INTERVIEWED Name Role Organization Contact FTF Inova Partners (Actors) Redacted CEO Founder and CEO, AppLoad Lda. Redacted Redacted CEO CdA Redacted Redacted Miruku Agro￾Indústria, SARL Redacted Redacted K2 Redacted Redacted Director Ecotri, S.A. Redacted Redacted Empresa de Comercialização Agricola Redacted Redacted Manager Ka Da Terra Supermercados, Lda Redacted Redacted UCM Universidade Catolica de Mocambique Redacted Redacted Manager Olam Moçambique, Lda. Redacted Redacted Marketing Director Bayer Moçambique Ltd Redacted Redacted Agro Imbeu & Parque de Maquinas, E.I. Redacted Redacted ASTRON Associação dos Transportadores Rodoviários de Nampula Redacted Redacted Condor Nuts – Industria de Processamento de Caju, Lda Redacted USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 71 Redacted Prodeca Sarl Redacted Redacted Forcier Mozambique, Lda Redacted Redacted Associação Comercial, industrial e Agrícola de Nampula (ACIANA) Redacted Redacted Hollard Moçambique Companhia de Seguros, S.A. Redacted Redacted IAM and Instituto de Fomento do Caju, IP Redacted Redacted Groupo Madal S.A.R.L Redacted Redacted Isolmoc Montagem e Comercio de Isolamentos e Revestimentos, Lda. Redacted Redacted Manager Winnua Lda. Redacted Redacted General Director Sithole Comercial Redacted Redacted Ngala Investimentos Redacted Redacted ABSA Redacted Redacted Coordinator Radio SIRT Chimoio Redacted Redacted Coordinator Community Radio of Monapo Redacted Other non-supported actors knowledgeable / involved in the systems where FTF Inova invests Redacted Chairmen of Board JFS, SA Redacted USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 72 Redacted IITA Redacted Redacted Director General Instituto de Amendoas de Mocambique (IAM) Redacted Other USAID Agricultural Investment Activities in Mozambique - interview requested 15th June Redacted Former Manager EMPRENDA TechnoServe Redacted Redacted Currently Project Manager Seedtrade AgriFuturo Redacted Redacted Partnering for Innovation Fintrac Redacted Redacted Chief of Party RAMA Land O Lakes and Winrock Redacted Other MSD Activities (interview requested 15th June) Redacted Program Director Women IN Business (WIN) TechnoServe Redacted Redacted Chief of Party Honduras Transforming Market Systems ACDI VOCA Redacted Redacted Senior Director, Agriculture and Community Development ACDI VOCA Redacted Redacted Project Manager Hortisempre, SwissContact / Swiss Agency for Development and Cooperation (SDC) Redacted Redacted Team Leader Innovation for Agribusiness (InovAgro I, II, III)[1] DAI / Swiss Agency for Development and Cooperation Redacted Implementing Partners (interview requested 15th June) USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 73 Redacted Researcher Market Share Assoc Redacted Redacted Senior MEL specialist Market Share Assoc Redacted Redacted Market Systems Specialist EcoVentures International Redacted Redacted International Program Director EcoVentures International Redacted Redacted EcoVentures International Redacted Redacted Country Director TechnoServe Mozambique Redacted FTF Inova Staff (interview requested 15th June to take place week 29th June to 3rd July Redacted Chief of Party FTF Inova DAI Redacted Redacted Value Chain Manager FTF Inova Redacted Redacted MEL Manager FTF Inova DAI Redacted Redacted Communication Specialist FTF Inova DAI Redacted Redacted Portfolio Manager - Chimoio FTF Inova DAI Redacted Redacted Portfolio Manager￾Nampula/Maputo FTF Inova DAI Redacted Redacted Partnership Manager FTF Inova DAI Redacted Redacted FTF Inova COR USAID Redacted Redacted Agriculture Team Leader, Agriculture, Environment and USAID Redacted USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 74 Business (AEB) Office Redacted FTF Inova A-COR USAID Redacted Redacted Team Leader for the BEE (policy) office USAID Redacted Redacted Director, Regional Economic Growth Office USAID Redacted Redacted Market Systems Specialist USAID Bureau for Resilience and Food Security (RFS) Redacted Experts / Observers Redacted SUGGESTED BY FTF Inova as he has used the work we are doing with Casa do Agricultor as example of Bottom of the Pyramid distribution and market creation in Africa. Harvard scholar. Redacted Redacted SUGGESTED BY FTF Inova as an MSD practitioner, able to help with visioning and understanding of the broad strategic approach. Consultant / MSD Practitioner Redacted Redacted Private Sector Advisor Mozambique DFID Redacted Redacted Researcher SPEED+ (for. Inova) Redacted USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 75 ANNEX C: THE FTF INOVA RESULTS CHAIN USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 76 ANNEX D: EVALUATION GRID USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 77 ANNEX E: COMPARISON OF MSD APPROACHES Activity Honduras Transforming Market Systems Innovation for Agribusiness Hortisempre Women in Business Approach Focus: supporting jobs. Initial phase: studies and analysis. Selected sectors based on job-creating potential; then design emphasizes a systems￾based approach (like Inova). Flexibility: no pressure to take decisions and deliver results. Focus: seed. Always work though a company/private sector (9 to 10 seed companies)—key part of their MSD work. Test models and support ones that work. Have value chains (6, Maize, G Nuts, Soya, Pigeon Pea, and Sesame). Focus is on irrigation; central to project design and decided by the donor. Emphasized technical support in a facilitator role and linkage between actors. Work with entrepreneurs. Focus: women micro￾entrepreneurs. A lot of adaptation (like Inova); tailored design over first one to two years from sectors to current focus with an evaluation to do this. How ideas are generated and who has the initial idea Emphasize co-creation; all staff involved, continuous, not phased, flesh out and discuss ideas to make a fundable project. Done through direct contact / interaction; discuss / mine ideas; constant process with associations government, forums, workshops, and peers. If ideas are lacking, issue call for proposals. In a sector, a call can focus on an area (e.g., gender). Unsolicited proposals three to four times a year. In early days, focus was on identifying issues—talked a lot with Pannar and Seed Co—to define need; there was a lot of negotiation, analysis, and surveying. PS drove this process. Now engages with private sector companies that have ideas and goes through a process of due diligence and Central idea of irrigation determined by the donor; implementation involves a lot of learning, making changes, deciding to target other areas, reviewing, and reengineering. Mix of internal idea development (studies, internal learning, testing small ideas), calls for proposals, and specific consultancies. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 78 analyzes the idea. If idea is accepted, they develop a deal note (work plan + budget) that can include co-financing. Levels of Research / Diagnosis Research-/learning-based approach; learning is emphasized throughout the process. Research/analysis is emphasized in first eight months to select sectors. A lot of research (selecting among and within VCs). Perform analysis (relationships in markets). There was a lot of analysis and learning in 2017 in preparation for the last phase, and there were a lot of evaluations and modifications to scale successful models. A lot of interaction and innovation—“innovation within innovation.” Test models—use pilots (e.g., buyer points). Expect models to be copied. Focus on TA and internal learning. “Every day you learn” is a permanent process; adaptive and very dynamic – always looking for options. Conducts internal reports with donor to look at progress and results, their own business model, the results chain, and the M&E plan; use lessons to improve. Internal M&E involves studies and surveys. Learning, study, and research are emphasized. WIN emphasizes internal knowledge and learning; research is mostly internal and talked about a lot among the WIN team—share learning. Has commissioned studies. Cost structures KI knowledgeable on budget for activities felt that co￾financing (grants/in-kind) was less than 20% of the total budget. He did not know further budget details. % of budget on grants and in-kind support (e.g., hire consultants to help, buy vehicles, or give cash) needs 50:50 contribution from business partner—done for testing something new, rigorous on what to accept and negotiate; USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 79 strict not free money. Admin is about 15% of budget. Administrativ e and technical spend Admin 15%, tech 60%, grants and in-kind 25% Proportion of spending on grants/loans 20% of budget is on grants / in-kind support. Less than 20%. Less than 10% grants. 25% Total budget Not available $6.7 million for four years. $7 million five-year project. Length of implementati on 2018 ongoing up to seven years. 2010–2020 2013–2020 2018–2023 USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 80 ANNEX F: COMPARISON OF OTHER USAID AGRICULTURAL INVESTMENTS Activity EMPRENDA (2005 to 2009) AGRIFUTURO (2009 to 2015) Partnering for Innovation (2014 to 2021) Multi country Initiative RAMA (2017 ongoing) Broad Approach Identify sectors/value chains/industries based on comparative advantage. Then 1) verify comparative advantage, 2) identify constraints, 3) develop an industry strategy (including policy changes). Then seek donor funding to implement that strategy and engage an industry program manager (preferably someone with industry experience at the general manager level) to manage the strategy implementation. Agrifuturo had an agribusiness development focus and used a value chain approach. Within each value chain, it looked for businesses that it could work with and motivate. Two main components: 1. Business development— worked with private sector to improve performance. Starting point were firms that TS and Clusa were already working with. 2. Enabling environment— studies on gaps and policy. Global, managed from Washington, DC. P4I call their approach an investor approach, using public funds on grant￾based, fixed-price investments based on milestones. Partners must contribute 50% of budget. Support relatively secure businesses. Cover all of Mozambique—most support is north of Rio Save. Have had four “rounds” in Mozambique: release a request for proposals (advertised on website by the USAID mission and on social media). Work in broad agriculture sector. First round: MFIs, Off takers, and two NGOs supporting agriculture extension and seeds. Second round: Targeted logistics and distribution—had identified a need to connect agriculture inputs to out growers/millers in soya. Third round: Targeted “nontraditional” businesses, meaning the private sector. Fourth round: Cyclone response— shorter, one-year partnerships (usually Activity has a climate-smart agriculture focus. Design based on four predetermined pillars: behavior change, model farmers, sustainable extension, and markets. Design is flexible/adaptable and has become more focused. Have adapted based on hands-on implementation experience (promotion and experimentation). Now focus is on intercropping with smallholder farmers in line with activity objectives. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 81 two years); supported four partnerships for a total of $2 million. Ideas Activity was based on sectoral studies, gave detailed industry/sector plan. This was then implemented by managers with detailed knowledge and experience of the subsector. Most work built on/developed Emprenda ideas. Focused on value chain from Emprenda. Business development ideas came from working with businesses to hear of their struggles and help them usually with TA. Actual support would depend on the individual business needs. Linkage often emphasized. Looked more at ways of running the business, whereas Inova is looking for systems-based ideas to help. Private sector comes with ideas in answer to P4I calls for proposals. Then ideas are selected based on criteria: - Recovery and resilience impact - Sustainability - Implementation plan - Organizational capacity - Budget - Leverage ratio (private sector contribution) Negotiate with those chosen—do not change the business idea, although in some cases they focus on a certain part of the proposal or change the geographical area. Ideas and adaptions to the activity are based on implementation experience and come from staff working with partners (e.g., seed companies) and farmers. Levels of Research / Diagnosis / Learning Study sectors first in detail. Learn with Implementation. Learning happened with implementation. Each industry had policy issues and needed connections Most research was on the enabling environment component—e.g., on ports (expensive, reducing Mozambique’s competitiveness). Outputs were used by SPEED in lobbying Ministries of Transport and Agriculture. Levels of research are much lower than those of Inova. When proposal is accepted, important points are the budget and the 50% contribution and payments being clearly linked to milestone delivery by the private sector. There is no M&E budget (has a theory of Low levels of formal research. Supports a national university funding of four interns. May use results from these studies. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 82 and power. Government was supportive (e.g., in cashew and poultry). change). Reports on six standard indicators. Relies on private sector to provide indicator data (e.g., sales, farmer numbers). Used a local M&E firm to collect data as private sector data were considered unreliable (in 2011/12). Not continued as funding was cut. Administrativ e and technical spend TA was the main expense. TA was the main expense. 10–20% of total budget spent on implementation/supporting grants. Admin is approx. 25%. Tech/field staff is 75%. Proportion of spend on grants / loans Not available Grants: about 6% of total budget, used mainly to bring in experts and machinery and fill gaps in a business. Followed traditional/usual call for proposals processes used by USAID. Grants make up at least 50% of the total budget; can provide up to $2 million to partners (much bigger than Inova). P4I do not provide direct technical assistance funding. No money for subgrants. $1.6 in original budget, used approx. $200,000, then stopped all grants. Total budget $25 million $25 million Spent about $15.5 million in Mozambique. Total budget is approx. $72 million. Budget was cut from $16 million to $6 million. Then $4.6 million was added back to address cyclone. Length of implementati on Five-year project ended. Five-year project ended. 10-year initiative ending in September 2021. Five years, three years into implementation. USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 83 ANNEX G: MINISURVEY RESULTS: AGRODEALER PURCHASES AND SALES Agro dealer Main things bought and volumes What are the main items you sell? Type of buyers (e.g., men/women/commercial farmers) 1 Seeds/flower seeds/oil pumps/shovels/hoes Seasonal Retailers (mostly men) 2 Fertilizer 50–100Kg; 20L insecticides; seeds 300–500g Insecticides; fertilizer 3 Fertilizer (60–80 bags of 50kg) Fertilizer Smallholder farmers (mostly men) 4 Horticultural seeds (100–150 packages); corn seeds (50kg), fertilizer (10–20 bags of 50kg); insecticides; pesticides Fertilizer; lettuce seeds; tomato seeds; cabbage seeds Smallholder farmers (mostly women) 5 Cabbage seeds (10 packages); lettuce seeds (10 packages); onion seeds (10 packages); fertilizer (2 bags of 50kg); insecticides Cabbage seeds; tomato seeds; lettuce seeds; fertilizer Smallholder farmers (mostly men) 6 Corn seeds; tomato seeds; onion seeds; cabbage seeds (500 packages in total); insecticides Fertilizer; tomato seeds; onion seeds; lettuce seeds Smallholder farmers (mostly men) 7 Seeds (100–150 packages); insecticides (4 boxes) Cabbage seeds Smallholder farmers (mostly men) 8 Tomato seeds (10 packages); cabbage seeds (10 packages); onion seeds (10 packages); lettuce seeds (10 packages); pesticides; fertilizer Tomato seeds; cabbage seeds; onion seeds Smallholder farmers (mostly men) 9 Veterinary medicines (10L); corn seeds (50kg); fertilizer (10 bags of 50kg); pesticides; cabbage seeds; carrot seeds; cabbage seeds; pepper seeds Cabbage seeds; carrot seeds; cabbage seeds; pepper seeds Smallholder farmers (mostly men) 10 Vegetable seeds (10–20 packages); pesticides (10–20 bottles of 250ml) Vegetable seeds Smallholder farmers (mostly men) 11 Fertilizer; chainsaw (1) Insecticides; fertilizer Retailers/Smallholder farmers (mostly women) USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 84 12 Insecticides; pesticides; herbicides; personal protection material; corn seeds; sesame seeds; veterinary medicines; pesticides Corn seeds (40-50 Tons); sesame seeds (80 tons), veterinary medicines; pesticides State-owned companies, NGOs, Resellers, Smallholder farmers (mostly men) 13 Fertilizer; veterinary drugs; seeds; insecticides; pesticides Seasonal Smallholder farmers (mostly men) 14 Tomato seeds (1 box); cabbage seeds (5 packages); onion seeds (1 box of cans); cabbage seeds (10 packets); okra seeds (10 packages); corn seeds (50kg); fertilizer (4 bags of 50kg); pesticides (1–2); insecticides (25–50L) Tomato seeds Smallholder farmers (mostly men) 15 Tomato seeds (50–100kg); pepper seeds (50–100kg); cucumber seeds (50–100kg); cabbage seeds (50–100kg); onion seeds (50–100kg); bean seeds (50–100kg); carrot seeds (50– 100kg); beet seeds (50–100kg); cabbage seeds (50–100kg); okra seeds (50–100kg); lettuce seeds (50–100kg); fertilizers (2–3 tons); pesticides (5000L); herbicides Fertilizers Institutions/smallholder farmers (mostly men) 16 Tomato seeds (6–7 packages); lettuce (12 packages); fertilizer (50kg) Mustard kale seeds Smallholder farmers (mostly men) USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 85 ANNEX H: FTF INOVA THEORY OF CHANGE SEPTEMBER 2020 Option 1 USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 86 Option 11 USAID.GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 87 FtF INOVA’s Emergent Theory of Change Narrative Boundary: Agricultural Market System in Mozambique USAID .GOV FTF INOVA MIDTERM PERFORMANCE EVALUATION | 88