1 Final Performance Evaluation of Securing Water for Food Grand Challenge for Development [August 2020] This evaluation was made possible by the support of the United States Agency for International Development (USAID), Sweden through the Swedish International Development Cooperation Agency (Sida), the Foreign Ministry of the Kingdom of the Netherlands (FMA￾NL), and the South African Department of Science and Technology (DST). The contents of this evaluation are the sole responsibility of David Hemson, Ken Caplan, Stephanie Monschein and Nohemi Voglozin and Dexis Consulting Group and do not necessarily reflect the views of the SWFF partners. Photo credit: Stephanie Monschein FINAL PERFORMANCE EVALUATION OF SECURING WATER FOR FOOD GRAND CHALLENGE FOR DEVELOPMENT David Hemson, Ken Caplan, Stephanie Monschein and Nohemi Voglozin Dexis Consulting Group CONTENTS Executive Summary........................................................................................................................................................2 Evaluation Purpose and Evaluation Questions ..............................................................................................................7 Project Background .....................................................................................................................................................11 Evaluation questions, design, methods and limitations................................................................................................12 Risks and Limitations of the Evaluation Design ...........................................................................................................21 Findings........................................................................................................................................................................24 Section 1: Relevance ....................................................................................................................................................24 Section 2: Effectiveness................................................................................................................................................30 Section 3: Efficiency .....................................................................................................................................................45 Section 4 Impact ..........................................................................................................................................................54 Section 5: Sustainability................................................................................................................................................89 Conclusions, and Recommendations.........................................................................................................................105 Section 1: Relevance ..................................................................................................................................................105 Section 2: Effectiveness..............................................................................................................................................106 Section 3: Efficiency ...................................................................................................................................................109 Section 4 Impact ........................................................................................................................................................110 Section 5: Sustainability..............................................................................................................................................115 Annex 1: Evaluation Statement of Work ..................................................................................................................117 Annex 2 SWFF Evaluation Questions by OECD DAC Criteria ...............................................................................130 Annex 3 Key Informant Interviews by Country........................................................................................................132 Annex 4 Surveys of Innovations................................................................................................................................134 Annex 5 Metadata......................................................................................................................................................135 Annex 6 Data Collection Instruments ......................................................................................................................136 Annex 7 Tables of Field Visit KIIs..............................................................................................................................183 Annex 8 Sustainability of innovations: evidence from performance evaluations ......................................................190 Annex 9 Statement of Difference..............................................................................................................................195 Annex 10 Evaluation Team........................................................................................................................................210 Annex 11 Conflict of Interest Forms ........................................................................................................................216 Acronyms ...................................................................................................................................................................... 1 Report Tables Table 1. The Sample: Initial status (as at November 2018) 14 Table 2. The Sample: Rounds (as at November 2018) 14 Table 3. The Selected Sample: Country and Innovations, Visits and Revisits 15 Table 4. Demographics of Surveys (First Visit, Revisit and Field Evaluator) by Innovation and Country 18 Table 5. Surveys of Customers/End-Users, Revisit Surveys (Gender, poor and household characteristics) 19 Table 6. Surveys of Customers/End-Users, Field Evaluator Surveys (Gender and household characteristics) 20 Table 7. Surveys of Customers/End-Users, Field Evaluator surveys (levels of poverty and income) 21 Table 8. Self-description of innovations, sets of priorities 24 Table 9. Locality of innovations supported by Round, 2020 25 Table 10. Profit or Non-profit Innovations, All Rounds 26 Table 11. Chart Innovation Success Rate 31 Table 12. Comparative Stage of Growth over Innovation Runway period (Years 1-5): Initial Development Period, SWFF vs. Non-SWFF 32 Table 13. Comparative Stage of Growth over Innovation Runway period (Years 1-5): Mature Development Period, SWFF vs. Non-SWFF 32 Table 14. Side-by-Side Comparative Stages of Development Timelines for SWFF vs. Non-SWFF from development stage to commercialization over five years 33 Table 15. Graphic: SWFF innovator assessment of SWFF/NON SWFF technical support, Not at all/Somewhat Satisfied 34 Table 16. Graphic, “More than Expected” Satisfaction with Technical Support, SWFF vs. Non￾SWFF Innovators 35 Table 17. Comparative effectiveness of organizational support by area: Effective vs. Not at all 36 Table 18. Funding Sources by Stage 37 Table 19. SWFF Stages of Growth (Growth, Midpoint, Maturity) of Five Years 2014-2018 42 Table 20. Comparative Information Across Global Challenge Funds 48 Table 21. SWFF Comparative Administrative Burden 52 Table 22. Stage of Development: Plans for Scaling Up, 2019 54 Table 23. Stage of Development: Status, Meeting Targets, 2018 55 Table 24. Status of Innovator Partnerships 55 Table 25. Innovation Programmatic Emphasis 56 Table 26. Resilience to climate change 57 Table 27. Customer/End-User Impact Responses, Benefit from innovation 58 Table 28. Customer/End-User Impact Responses (Women and Poor) 59 Table 29. Impact on Customers/End-Users, Income 61 Table 30. Impact on Customers/End-Users, Income (Women and Poor) 62 Table 31. SWFF contribution, extent, and type 64 Table 32. Impact on Customers/End-Users, Improved access to water 66 Table 33. Impact on Customers/End-Users, Improved access to water, women and poor 67 Table 34. SWFF Innovation effect on “crop per drop”: less water and better productivity (water efficiency) 68 Table 35. SWFF Innovation effect on “crop per drop”: less water and better productivity (water efficiency, Women, Men and Poor) 69 Table 36. Reported Improvement in Agricultural Productivity 72 Table 37. Reported Improvement in Agricultural Productivity (Women and Poor) 73 Table 38. Agricultural productivity Outcomes and Targets 75 Table 39. Agricultural Productivity Targets 75 Table 40. SWFF Innovation Effect on Inputs 76 Table 41. Customers/End-Users responses, Degree of Change in farming practices, 2 or more changes in farming practices due to innovations 77 Table 42. Private Sector Investment (N=23) 83 Table 43. SWFF-Funded Innovations sources of funding, Year 1 to Year 5 84 Table 44. Perceived Fairness and Value of Targets 86 Table 45. Milestones, targets, and reporting system 87 Table 46. Innovator Perception of SWFF Targets, Milestones, Support, and Rounds 87 Table 47. Sustainability indicator: use of innovation on the long term (5 to 10 years) 90 Table 48. Sustainability indicator: use of innovation on the long term (5 to 10 years) (Women, Men and Poor) 91 Table 49. Sustainability indicator: Recommendation of innovations to other farmers by Customers/End-Users 92 Table 50. Willingness to pay for innovation 93 Table 51. Status of innovation by Round Status of innovation, 2020 96 Table 52. Analysis of Affordability, Use and Financial Sustainability 97 Table 53. Women’s participation as customers, managers, and owners 101 Table 54. Demographics of innovations, gender, SWFF/non-SWFF 101 Annex Tables Table 1. Gender Repartition by Innovations (in-countries surveys), 2019 183 Table 2. Stage of Development of Innovations (Scale, in-countries surveys) 184 Table 3. Revisit surveys undertaken by Dexis (in-countries surveys), 2019 185 Table 4. Gender Repartition by Innovations (in-countries surveys), 2019 186 Table 5. Stage of Development of Innovations (Targets Met, in-countries surveys) 187 Table 6. Stage of Development of all Innovations (Scale, data from surveys & KII interviews) 188 Table 7. Stage of Development of all Innovations (Targets Met, data from surveys & KII interviews) 189 Table 8. Analysis of Affordability, Use and Financial Sustainability with supporting evidence 192 1 ACRONYMS AST Adaptive Symbiotic Technologies CEO Company Executive Officer DST South African Department of Science and Technology EQ Evaluation Question FAO Food and Agricultural Organization FE SWFF Field Evaluator Survey FGD Focus Group Discussion FP Founding Partners GCD Grand Challenge Development IIAC Innovation Investment Advisory Committee KII Key Informant Interview LOE Level of Effort MFA-NL Ministry of Foreign Affairs of the Kingdom of the Netherlands MTR Mid Term Report M&E Monitoring and Evaluation OECD Organization of Economic Co-operation and Development OECD/DAC OECD Development Assistance Committee QCA Qualitative Comparative Analysis Rd. Round: Rd. 1, Rd. 2, etc. SIDA Swedish International Development Cooperation Agency SOW Statement of Work TAF Technical Assistance Facility USAID United States Agency for International Development USG United States Government WE4F Water and Energy For Food Grand Challenge WWW World Water Week 2 EXECUTIVE SUMMARY As a Global Challenge Fund supporting the uptake of innovations, the goal of Securing Water for Food (SWFF) is to source, incubate and accelerate high-potential technical solutions and/or business models that lead to higher agricultural productivity with lower overall water demands in the food value chain. SWFF innovators scale innovations designed to support the program’s overall goal to reduce water scarcity and poverty. The focus areas of SWFF include: • water efficiency and reuse, especially targeted at the food value chain; • water capture and storage, in particular in regions where rain occurs at limited times; and • saltwater intrusion, especially in coastal aquifers or deltas and estuaries. The SWFF Founding Partners – United States Agency for International Development (USAID), Swedish International Development Cooperation Agency (Sida), Ministry of Foreign Affairs of the Kingdom of the Netherlands (MFA-NL), and the South African Department of Science and Technology (DST) -- share the common goal of advancing international development through improved access to sustainable water sources for agricultural applications. Values held in common by the Founding Partners include sustainability, efficiency, sourcing of market-based solutions, gender inclusion, climate adaptation and mitigation, ensuring benefit to poor people, and a commitment to avoid negative effects (particularly with regard to water-related services). The purpose of this evaluation has been to determine the extent to which SWFF contributed to the expected program outcomes and results achieved by innovators across the portfolio during project implementation. Have SWFF innovations led to the production of more food using less water, thus ensuring greater food security and raising incomes particularly of poor farmers? The evaluation team has sought to: 1) validate the results already reported by SWFF and answer the questions set by funders on why and how the outcomes were achieved by innovators, 2) identify the contributing factors to success particularly as regards to SWFF programming, and 3) uncover the unintended positive and negative consequences that arose as innovators implemented their models with SWFF support. The performance evaluation concurrently examined SWFF/TAF (Technical Assistance Facility) program efficiency and impact through the lens of the program delivery, the risks taken, and the technical assistance provided. The sources of data for the evaluation have included SWFF annual monitoring data; program documentation; a number of sets of surveys of end-users and Focus Group Discussions (FGDs) with a selected 21 SWFF innovators, 17 of which were visited in country (including site visits); and a series of Key Informant Interviews across non-visited innovators, SWFF’s programmatic implementation team, the Founding Partners, and members of the Innovation Investment Advisory Committee (IIAC). The evaluation makes assessments at three levels: 1) the meta-level, 2) the SWFF program-level, and 3) the awardee-level. These levels roughly correlate to the OECD evaluation categories with impact and sustainability linked to the meta-level, and relevance, effectiveness and efficiency generally linked to the program and awardee levels. 3 The questions set by the funders in the original Statement of Work (Section C) are presented with evidence and findings on SWFF’s Relevance (V.a, A1a), Effectiveness (A1b, B3, B5, B7), Efficiency (B1, B2, B4, B6), Impact (A1c, A1d, A1e, A1h, A2, A3) and Sustainability (A1f, A1g). A representative sample of 21 of the 40 SWFF innovations in 7 countries was determined on the basis of geographic region, round, status, focus type and number of innovations in a country. This included Bangladesh (Lal Teer, Practical Action); Egypt (ICBA); Ghana, (Ignitia, Skyfox); India (EDR, Adaptive Symbiotic Tech, MyRain, SWAR, Naireeta, WASTE Stichting, NewSil); Kenya (IRK Sunculture, CSDES M-FODDER, ITIKI CUT, Hydroponics Africa); South Africa (Ecorangers, Reel Gardening, CUT); and Uganda (Aquaponics Farming, Green Heat). Of the 21 innovations, two were unresponsive and there was insufficient time or other issues with others. Finally, 17 awardees were interviewed in field visits. Surveys were made of customers/end-users in site visits; in 14 surveys, 221 questionnaires with impact modules relating to the benefit planned from innovations in infrastructure, service/information technology or products were concluded. Most of the innovations in the first round of surveys were subsequently revisited in a second round during September-November 2019. In addition, the data from 12 SWFF Field Evaluator surveys which adopted an instrument comparable to that of the evaluation team’s Revisit surveys has also been analyzed and incorporated into the analysis. Key Informant Interviews were undertaken with Founding Partners, TAF staff, innovators and importantly the IIAC. Through participation at World Water Week in August 2018, 39 KII were concluded adding other current innovations, alumni and graduates not in the original sample. The research team gained a comprehensive overview of the entire SWFF Program as well as detailed observation at the site level. This qualitative data was captured and coded in an electronic template and auto-coded in Nvivo; the survey data was analyzed in SPSS. In addition to interviews and surveys, analysis has been made from the many SWFF performance evaluations to assess the financial sustainability of innovations in the post-SWFF period. It was found that the Program has great relevance in contributing to improving agricultural productivity in developing and emerging economies. There is widespread local need in developing and emerging economies for the innovations and potential for local ownership. Innovators have the challenge of helping generate effective demand to meet the emphasis around poorer and more vulnerable customers as well as meeting the demand from better off farmers for their infrastructure, products and services. SWFF innovations have been found to be demonstrably more focused on women, the poor and vulnerable, with a sharper focus than surveyed Non￾SWFF innovations on gender integration with women as customers and emphasizing its work with vulnerable populations. There are challenges particularly in the ability of the intended customers/end-users (poor, very poor and women) to pay. Thus, effective demand for the products and services of innovations often depends on locally available small-holder finance at low interest rates for poor or very poor farmers. The funders should consider the most effective ways of managing access to low interest locally available -finance that extends credit to farmers in such a way as not to over-burden either the farmer with debt or the innovator with bureaucracy. This will help build a stronger base of innovations and impact in areas of greatest 4 need. To ensure greatest relevance for target farmers, it is recommended that there should be greater developing and emerging economies participation in WE4F. The Program was found to be effectively run, ensuring close attention to detail, and with a hands-on approach with the awardees. The flow of reliable and authoritative information enables innovators and SWFF to “pivot” as new information becomes available. It also helps make difficult decisions where necessary. Year-on-year SWFF innovations have progressed upwards from inception to midpoint and finally maturity. By Year 5, only 20% of innovations are still at their midpoint while 80% (compared to 60% Non-SWFF innovations) have entered their mature (commercialization) phase. As testimony to effectiveness, there are high levels of satisfaction with SWFF technical support and low levels of satisfaction with Non-SWFF technical support. SWFF support to Communications and/or branding, Partnerships and networking, Business Practices / Organizational capacity, and Access to finance is regarded as Effective/Very Effective. The only support in which Non-SWFF technical support scores higher (I.e. for which SWFF has not been as engaged) is in Product design and production. Overall, 100% of SWFF innovators report that the added value of SWFF’s technical support is “Significantly Higher” than other support. The Founding Partner interaction provides effective oversight and direction. They bring complementary agendas to the SWFF partnership, as per the various needs and capacities of each agency. USAID’s effective management of the challenge fund is highly regarded. Greater emphasis should be given to broadening developing and emerging economies membership base of the Founding Partners in the new initiative and linking these partners effectively to the Program. Such partners could help locate and mentor suitable innovators particularly from developing and emerging economies. It is recommended that SWFF should foster more exchange of information on the barriers which innovators are encountering, particularly in the critical first year, and efforts to overcome them. Initiating wider links with related embassy and other programs would further enhance SWFF’s effectiveness. Innovators strongly appreciate the dedicated support of the TA Facility and feel it is efficiently set up, organized and managed; it provides the “right kind of support at the right time from the right people”. Some 74% of innovators cite the TAF as being decisive in its contributions towards their success and outcomes. Compared to other programs, SWFF is regarded as adding “tremendous” value, specifically in terms of monitoring and evaluation and other reporting capabilities. To reinforce efficiency and effective targeting of support, it is recommended that WE4F should further reinforce innovator sustainability/viability, and programmatic (or innovator) learning and adaptive management as well as continuing the reporting on the achievement of primary outcomes. Innovators regard SWFF TAF as “Highly efficient” and providing “High value for money. While the burden SWFF places on awardees appears to be higher than most other grand challenge programs, these same processes allow for SWFF to tailor its funding and technical support more directly to achieving the aims and objectives of the awardees as well as those of the founding partners. 5 Relative to other GCFs, SWFF brings highly intensive management, proactively providing a wide range of technical assistance and interaction. While incomplete and inconclusive, an exercise in comparing the overheads of nine GCFs in the Swedish International Development Cooperation Agency (Sida) and USAID portfolios found a range in administrative costs/total funds of 4.9% as a low (Sustainability and Resilience) to 57.1% (MAVC) as a maximum. Of those listed, SWFF sits in the middle with fund management at 26.2% of the total budget and 40.9% of fund grant value. SWFF’s proactive approach to providing a wide range of technical assistance and interaction opportunities is well appreciated by awardees. The critical aspect of SWFF is its impact on the water efficiency and agricultural productivity of customer/end-user farmers. SWFF strongly contributed to outcomes: analysis of surveys find (90%) of beneficiaries have improved access to water and (95%) water efficiency in terms of crop per drop directly due to SWFF’s innovations, changes in farming practices or crop yields due to climate change (56%) increased resilience of crops to climate change (68%), undertaking two or more changes in farming practices (84%). Overall an average of 84% of participants report benefit and 84% report increased income. Such impact can be uneven: women and poor have lower levels of beneficial impact, even with the relatively higher emphasis on their participation. While the inclusion of the vulnerable depends on the business model, very poor groups and women are positively impacted by SWFF￾supported innovations but always not to the same degree as the overall population. This is what the users report and survey data captures; what is not always clear why this should be the case. Farmers reported an increase in agricultural productivity through increased yields, a greater number of growing seasons or a diversification of the crops grown, and more efficient use of inputs (labor, time, pesticides, fertilizers, etc.). women and the poor reported lower levels of water efficiency and improved income. Resilience to extreme weather conditions is less evident but can be inferred: better adaptation to extremes by increased water access and a reduction in reliance on rainfed farming, targeted and improved irrigation, and better control over adverse climate conditions and pests. Measured impact is crucial to each innovator’s credibility. As such, surveys should be undertaken at modest expense in the new initiative to ensure baseline and impact in particular in relation to inclusion (women, poor and very poor), changes in income, and climate resilience. Resilience to weather extremes in agriculture should be included in the design of future innovations as this will certainly increase their impact. Since climate conditions impacting on agriculture are becoming more pronounced, express criteria on climate resilience should be included in applications to WE4F. Sustainability was assessed at a meta, program and innovation level. A critical aspect of innovation is use of the data to learn from failure. Round by round, innovations are succeeding to meet the needs of the poor or very poor over the years and are gaining in viability, but more could be learned from success and failure. Gender integration is identified as a key social component of sustainability, without growing participation by women as customers of the innovation in the management and ownership, innovations will be constrained in impact and hampered in sustainability. SWFF should continue the drive for gender integration accentuating economic and organizational advantages. Quotes from innovators provide testimony to how the innovations fit into a socio-economic context: 6 We have reached the Break Even point from 2015 onwards. Though we are not making huge profits, we have enough work to do for the next 10 years. Socially it is challenging, however, we are giving priority to organizations that are keen to work with women. We are working to survive in an economic climate in which competitors are keenly aware of the potential profit of our technological innovation. Some people want to grab our idea by slightly changing our name and approaching rich farmers. Making a social program sustainable is very difficult in this country at this point of time. In terms of financing, data from the Online Survey does not show a striking difference in sources of funding between SWFF and Non-SWFF innovations in the growth period and midpoint. There is however divergence in the “mature” phase (Years 4 and 5) SWFF innovation showed greater use of public funds and lower use of private and own (savings, family or other local) sources of funding. Since SWFF funding is public this indicates the logical progression in their development. In relation to gender, in the average SWFF innovation, women constitute 35% of all staff compared to 11% of other innovations from the online impact survey. Women executives and managers constitute 7% of all staff compared to 3% in other innovations. Sustainability in terms of customers’ commitment to continued use and recommendation to others is recorded at a high level. Overall, 94% of all Customers/End-Users report they will use the innovation into the future with the lowest level recorded for Practical Action (53%). Overall, 96% of all Customers/End-Users would recommend the adoption of the innovation they use by other farmers. Practical Action records the lowest level at 53%. (The reasons for this could be explored further – perhaps users fear that a more widespread use would flood the market.) Previously, it had been recommended that, as well as celebrating and learning from success, SWFF should improve its systematic and intentional learning from failure and make reports of progress and challenges more widely available. This has now been taken up in the Failures, Pivots, and Lessons Learned Report and analyzed with some rigor. This Report recommends pivots in relation to identified failures in program coordination, acceleration facilitation, grants and finance, and monitoring and evaluation. A number of the identified issues reflect the findings from data analysis and many of the recommendations are aligned to the pivots proposed. Overall, taking all the innovations funded by SWFF from its inception, 23 of the 40 innovations (58%) successfully graduate from SWFF. The key question in discussion is the financial sustainability of innovations in the post-SWFF period when no further funding is available. Analysis of 8 graduate innovations conducted from performance evaluations and other sources indicates that all are operational 2-3 years after funding, most are assessed as durable and profitable or covering costs and the remainder, while operational, have unsure durability. Subsequent Grand Challenges such as WE4F should continue to systematically record and analyze the reasons for failure, regarding gender, institutional stability, finance, technology, and other dimensions. In relation to environmental sustainability, many innovators expressed strong commitment to green technologies, the use of organic pesticides, and emphasis on improved soil 7 quality. In addition, several products (such as NewSil) contribute to reducing or eliminating fertilizers and pesticides altogether. Since more governments now are increasingly interested in declaring agricultural zones or entire states as organic, future Grand Challenges such as WE4F should consider such a target in selection criteria. EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The primary purpose of this evaluation is to determine the extent to which the Securing Water for Food (SWFF) Grand Challenge for Development (GCD) program contributed to the expected SWFF program outcomes and results achieved by innovators during project implementation and post project implementation. The primary focus areas have been whether SWFF innovations have led to the production of more food using less water, thus ensuring greater food security and raising incomes particularly of poor farmers. The evaluation team has sought to validate the results already reported by SWFF and answer the questions on why and how the outcomes were achieved by innovators, identify the contributing factors particularly as regards to SWFF programming, and uncover the unintended positive and negative consequences that arose as innovators implemented their models. The performance evaluation concurrently examined SWFF/Technical Assistance Facility (TAF) program efficiency and impact through the lens of the program delivery, the risks taken, and the technical assistance provided. Data collected regarding the operations as well as the financial and human resources spent on the SWFF/TAF has been used to assess the results and, within time constraints and as feasible, to compare SWFF’s operational efficiency and effectiveness with other grand challenge funds. The sources of data from which the evaluation report have been compiled include: • SWFF TAF annual monitoring data via DevResults and the SWFF awardee database; • SWFF TAF program documentation; • Site visits with 17 SWFF innovators; • For innovators visited, surveys and Focus Group Discussions (FGD) with users/customers of the innovation; • For innovators visited, FGDs with non-users where appropriate and feasible; • Applicant survey already conducted by SWFF of three rounds of awardees, finalists, and non-finalists. • All rounds of innovators that have received two or more years of funding; innovators with one year of funding; and SWFF non-innovators; • Additional survey data from rejected SWFF proposal applications and finalists to increase sample size of counterfactual; • Qualitative information gathered from interviews with current SWFF portfolio innovators that the team were not able to visit, non-recipients of SWFF funds, founding partners, TAF staff, members of the Innovation Investment Advisory Committee (IIAC). • In the period September – December 2019 Dexis local consultants undertook 12 Revisit surveys out of the 17 Innovations originally visited. These surveys used a set of standard 8 questions which have been tested in the field and achieved a larger sample than the original or first visits. • In addition, SWFF deployed Field Evaluators to conduct surveys at 11 Innovations namely: Naireeta Bhungroo, AST, Skyfox, WASTE, WGI, ITIKI, MNP, Hydroponics Africa, Greenheat, Aybar and Aqysta. The resulting data has been included in data analysis and evidence considered for the Evaluation Questions. • Final follow up surveys made available from SWFF TAF. The Evaluation Design includes the initial criteria and indicators for assessing the efficiency and impact of SWFF, which were refined in later discussions with the SWFF Founding Partners, and a proposal for the methodology and sample selection. KEY ISSUES TO BE EVALUATED The summary of the SWFF program below sets out the key outcomes anticipated from SWFF and Awardee activities to be evaluated; it is anticipated that these activities will lead to at least 40 projects in 30 eligible countries. The expected outcomes of SWFF were that: 1. At least eight proposals/innovations that improve water availability and efficiency in the food chain have been adopted, brought to scale and/or commercialized by businesses in at least eight developing and low-to-middle income countries (see p 58 of partners PAD); 2. Demand for and availability of these innovations have increased. 3. More food has been produced with less water or more water has been made available for food production in the eligible countries (NL aim in line with NL policy is 25% resource efficiency improvement by the program innovations as compared to standard practice in the implementation countries; the Swedish International Development Cooperation Agency (Sida) aims for 20% resource efficiency improvement by the program innovations as compared to standard practice in the implementation countries). 4. This program will also contribute to increased water-related resilience to climate change (climate change adaptation). 5. Impact indicators include (see SWFF PMEP for specific program indicators and indicator targets): a. Percentage of food productivity / volume water increase b. Volume of water usage reductions through efficiency-increasing innovations in the food value chain c. Volume of water captured and stored for food production d. Percentage increase in agricultural yields/farmed area due to SWFF innovations e. Number of direct customers/end-users of the program (disaggregated by gender) f. Number of innovations adopted, brought to scale, and/or commercialized g. Number of poor people reached (in some cases, individual SWFF awardees had expected outcomes regarding poverty). In line with the GCD Monitoring and Evaluation (M&E) operational plan, the present evaluation fully incorporates the GCD Results frameworks and will evaluate on indicators at three levels: 1) 9 the meta-level (enabling analysis across GCDs); 2) the program-level (SWFF level); and 3) the awardee-level. These levels roughly correlate to the OECD evaluation categories with Impact and Sustainability linked to the meta-level, and relevance, effectiveness and efficiency generally linked to the program and awardee levels. EVALUATION QUESTIONS This final program evaluation should determine whether or not the SWFF program as a whole led to the production of more food using less water based on the performance of the innovations that SWFF funded. During this evaluation, the evaluator will examine evidence provided to SWFF by SWFF innovators in order to determine the impact of the program around the following questions: A. SWFF Innovator’s Impact 1. To what extent has SWFF contributed to the outcomes and results outlined by the indicators in the SWFF PMEP during project implementation and post project implementation? In answering this question, the evaluator should take into consideration the following: a. Is there demand and local ownership for the SWFF innovations (individually and across all innovations)? b. Does the SWFF TA overcome organizational capacity barriers for innovators? Were there additional barriers that were not addressed? c. Have vulnerable groups (the poor, women, ethnic minorities) been positively and/or negatively impacted (through income, employment, water/environmental benefit or harm) from SWFF supported innovations? Specifically, has SWFF reduced the number of people in poverty as a result of supporting SWFF￾supported innovations and has SWFF increased the number of women benefitting from SWFF-supported innovations. d. Did SWFF-supported projects increase water efficiency/make water more accessible? Did SWFF projects meet their water efficiency/availability targets? Overall, across all innovators, did the program meet the water efficiency/availability targets? e. Did SWFF-supported projects lead to more agricultural productivity and resilience to climate change? Did SWFF projects meet their agricultural productivity targets? Overall, across all innovators, did the program meet the agricultural productivity targets? f. Were SWFF-supported projects environmentally sustainable (i.e., did they provide positive environmental benefit, or did they do more environmental harm than good)? g. Were SWFF-supported projects likely to be financially and socially sustainable by the organizations supporting the innovation? h. What is the balance between public/social engagement and private/public engagement? To what extent have private funds been generated that contribute 10 to the developmental objectives of the program both during and following SWFF awards. 2. How much of the measured change (outcome or result indicator) in the SWFF program can in fact be attributed to the SWFF-supported projects? That is, what portion of the result is not explained by the projects examined by the evaluation? (SWFF recognizes that this is a difficult question to answer, but wants the evaluator to make the best effort to answer this question). 3. To what extent are there differences between the planned SWFF-supported projects and what was actually delivered in Year 1 and then Years 2-3 of the projects? B. SWFF Program’s Impact and Efficiency 1. To what extent were the SWFF results to date in balance with the level of effort and resources (funds, human resources including by the FPs, TAF, interns, consultants)? 2. To what extent is the level of effort and resources spent by applicants/innovators in balance with the added value SWFF brings? 3. How effectively have investment risks been managed by the program? (number of failed projects, timeliness of reaction on problems observed etc.) 4. To what extent was the TA Facility efficiently set up, organized and managed? 5. To what extent did the SWFF TA Facility provide SWFF innovators with technical assistance that led to an immediate success (a support engagement is defined as an immediate success if deliverables formally agreed to by the awardee in the work plan were delivered as the awardee expected) and long term success (a support engagement is defined as a long-term success if the product or advice delivered is actually adopted by the innovator and results in recognized value, such as a shift in strategy, an effective partnership, additional funding, new financial forecasting capabilities, or an improved manufacturing approach or product design)? To what extent did the SWFF TA Facility provide SWFF innovators the technical assistance that innovators deemed useful? Did SWFF enable certain (new) (combinations of) expertise to be deployed that would otherwise likely not have been deployed/used (by the individual partners)? 6. To what extent are the administrative costs for managing SWFF above, below, or on par with the cost of similar Challenge funds? (Special Consideration should be made for funds that provide technical assistance to their innovators.) 7. How well did founding partners interact within SWFF and what lessons should be taken from their interactions? Following the Evaluation Design, in preparing the data collection instruments, the evaluation questions were gathered under the following section headings for ease of use. These section headings have been applied in structuring the report as follows. SOW EQ Report EQ question organization Section C-V.a, A1a Section 1 - SWFF Program Relevance A1b, B3, B5, B7 Section 2 - SWFF Program Effectiveness B1, B2, B4, B6 Section 3 - SWFF Program Efficiency A1c, A1d, A1e, A1h, A2, A3 Section 4 - SWFF Impact A1f, A1g Section 5 - SWFF and Sustainability 11 PROJECT BACKGROUND As an innovation and acceleration initiative, the goal of Securing Water for Food (SWFF) is to source, incubate and accelerate high-potential technical solutions and/or business models that find new and sustain existing water supplies as well as lower overall water demands in the food value chain to reduce water scarcity and poverty. SWFF innovators scale innovations designed to support the program’s overall goal. The focus areas of SWFF include: • water efficiency and reuse, especially targeted at the food value chain; • water capture and storage, in particular in regions where rain occurs at limited times; and • saltwater intrusion, especially in coastal aquifers or deltas and estuaries. The SWFF Founding Partners (United States Agency for International Development (USAID), the Swedish International Development Cooperation Agency (Sida), Ministry of Foreign Affairs of the Kingdom of the Netherlands (MFA-NL), and the South African Department of Science and Technology (DST)) share the common goal of advancing international development through improved access to sustainable water sources for agricultural applications. Values that are common to the Founding Partners include sustainability, efficiency, sourcing of market-based solutions, gender inclusion, climate adaptation and mitigation, benefit to poor people, and a commitment to avoid negative effects (particularly with regard to water-related services). The SWFF Founding Partners recognize that often 100 investments are made in the private sector to get one successful innovation to wide-scale adoption. SWFF aims to have at least 10-20% of the innovations it supports reach some level of wider-scale adoption (greater than 10,000 customers/end-users). In support of these aims, USAID (the lead program implementer) has contracted the SWFF Technical Assistance (TA) Facility to provide and/or facilitate TA services and assist USAID in the implementation of the program to help innovators accelerate their progress to reach wider-scale adoption. The Innovation Investment Advisory Committee (IIAC) provides advice on selection of innovators, assessment of progress and determination of benchmarks to make sure that the Founding Partners support those innovations that most likely will achieve success. To date, four calls for proposals have been announced and selection of innovators completed. Through SWFF, the Founding Partners hope to source and accelerate high potential solutions that will have multiplier effects at various levels of a country’s economy. The following three hypotheses are both meaningful and practical measurements of SWFF’s development impact potential: 1. By investing in science and technology innovations at the water and agricultural nexus, the pace of development in both sectors will be substantially faster than if we relied on “traditional” development programming alone. 2. By sourcing technologies and business model innovations that have already demonstrated potential at the pilot stage, Tier 2 SWFF-supported innovators have greater likelihood of being brought to scale (reaching at least 1 million people) and Tier 1 SWFF-supported innovators have a greater likelihood of reaching wider scale adoption (reaching more than 10,000 customers/end-users). 3. By investing in acceleration-oriented technical assistance and facilitating partnerships, we will substantially increase the likelihood that innovators will have the knowledge, tools, and resources to bring their innovations to scale. Source, incubate and accelerate high-potential technical solutions and/or business models that find new and sustain existing water supplies as well as lower overall water demands in the food value chain to reduce water scarcity and poverty SO 1 DEFINE Data & Evidence Used to Define Specific Barriers to Development SO 2 MOBILIZE Partners and Solvers Mobilized SO 3 SCALE Viable Science and Technology Innovations Scaled SO 4 IMPLEMENT Solvers Help to Overcome Critical Barriers to Development IR 2.1 Partnerships Built to Advance Program IR 2.2 Innovations Sourced from Diverse Solvers IR2.1 Partnerships Built to Advance Program IR 3.2 High Potential Solutions Are Adopted S01-S04: 31 indicators are maintained by TAF/SWFF IR1 IR2 IR3 IR4 IR5 TECHNICAL ASSISTANCE& SCALING, 10 Indicators GRANTS ft FINANCIAL MANAGEMENT, 6 Indicators MONITORING AND EVALUATION, 6 Indicators COMMUNICATION AND PARTNERSHIPS, 5 Indicators TA FACILITY ADMINISTRATION, 4 Indicators IR4.1, 4.2 & 4.3: 28 indicators are maintained by Awardees with assistance from TAF/SWFF IR 4.1 Increased access/availabiilty to and more efficient, safe use of water for sustainable food producers of all scales throughout the supply chain in low and middle income countries IR 4.2 Improved Resilience and water availability due to climatic and seasonal variability IR 4.3 Reduced risks from salinity due to technological advances G o a l Sub -Objectives Intermediate Results 12 The SWFF program has aimed at supporting at least 40 projects in 30 eligible OECD/DAC 1-4 countries. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS The original Results Framework sets out a single Objective and four Sub-Objectives as shown in the graphic below and sets of related Intermediate Results. The strategic objectives Define, Mobilize, Scale and Implement describe the key activities within SWFF and the related indicators and milestones of progress. Figure: SWFF Results Framework This is primarily a performance evaluation which focuses on and measures these results or outcomes and impacts at two levels: firstly, that of the strengthening of innovations through support from SWFF/TAF and secondly, by these scaled up innovations achieving measurable water efficiency and increasing food production. Dexis has been encouraged to work collaboratively with the Triple Line evaluation which ran concurrently with earlier stages of this evaluation and notes the Innovations interviewed, provisional findings and recommendation. This strengthens the counterfactual to SWFF in other GCFs and provides access to comparative data. There are a number of levels at which attribution for outcomes and impact is possible: firstly, that of SWFF/TAF in giving resources and providing support for innovations/awardees, secondly by the innovations themselves, and thirdly by support from partnerships and government and other agencies. 13 These distinctions demonstrate the complexities of attribution which is considered not separately but in combination as contributions in achieving competence in the management of innovations and impact from the application of the technology embodied in the innovation. The Outcomes Harvesting approach which does not start from work plans or from a theory of change is open to retrospectively identifying the full range of drivers after identifying and validating outcomes. The familiarization with these procedures, full access to this data and the evaluation methodology itself has been discussed with SWFF. The focus of Outcomes Harvesting is on the substantiation of Outcomes and this is found, in part, through the validation of data. Outcome Harvesting Approach The methodology suggested in the SOW is that of Outcome Harvesting, which is an approach in which evaluators identify, formulate, verify, analyze, and interpret 'outcomes' (in summary, substantiate outcomes) in programming contexts where relations of cause and effect are not fully understood. This is particularly suitable for this evaluation where activities, responsibilities, and outcomes tend to overlap between SWFF and Awardees in the specific details of each innovation. The primary means of sourcing information was through KIIs with TAF, FP, IIAC and Innovators followed by divergent sources which provide other perspectives on Outcomes which help provide triangulation such as from a) the disparate views as captured in the KII and b) the customers/end-users (in FGD and surveys) who provide a perspective from below. The method involved in the analysis of this data is set out in the Data Analysis Plan below. Outcome Harvesting is an evaluation approach in which evaluators, grant makers, and/or program managers and staff identify, formulate, verify, analyze and interpret 'outcomes' in programming contexts where relations of cause and effect are not fully understood. The KII particularly provided the setting in which the outcomes anticipated and unanticipated from different program contexts could be better understood. Selection of Innovations and Sites The selection of innovations was undertaken with key criteria (including sites not previously visited) agreed with the client. The sampling frame for most of the data collection and analysis, in particular the meticulous review of performance monitoring data, existing and proposed survey data, interviews, and desk review will consist of the entire universe of innovators. The team developed a sampling universe in Excel in which all innovations from the SWFF program, including alumni and graduates, are listed. Across the columns the site selection criteria were found and entered. The team then ranked the innovators using a sort with multiple levels of criteria in order of importance, e.g. if geographic region was the most critical data cut, followed by country, focus area, round, visit status and then other available data on criteria. The tables below indicate the “demographics” of the sample; although the SOW required eight site visits the multiple sorting identified the following 20 innovations as the sample, to which Egypt was added as it had not previously been visited. The tables below provide the initial status of the sample as it was derived to be representative of the full 40 innovation. In terms of the current status of projects, 13 of the sample were current, five were Alumni and another three were Graduates. The Alumni were 14 somewhat over-represented in selection which partly compensates for under-representation in responses to the Post Awardee Survey. Table 1. The Sample: Initial status (as at November 2018) Project Status Selected Others Grand Total Alumni 5 9 14 Current 13 2 15 Graduate 3 8 11 Grand Total 21 19 40 Note: This is the basis on which selection of the sample was made at the beginning of the evaluation; the status of the original sample has changed over time. The final status is reported in tables in the section on Sustainability. The sample was selected methodologically to represent innovations on the basis of the then current status, the current place in rounds and country location. At the time of sampling, the current innovations constituted the greatest proportion of the sample, followed by first and second rounds. During the three year period up to the completion of the evaluation there has been considerable change in the status of the projects which is reported in the section on Sustainability. The sample also is effectively representative of the four Rounds undertaken in SWFF (the two innovations in Round are consolidated in one row). Table 2. The Sample: Rounds (as at November 2018) Rounds Selected Others 16 Rd. 1&2 10 8 18 Rd. 3 9 3 12 Rd. 4 3 7 10 Grand Total 21 19 40 Note: This is the basis on which selection of the sample was made; the status of the original sample has changed over time. The sample amounts to 21 out of the 40 Innovations1 (53%) on the Awardee Database, a reasonably representative selection from the continents and focus areas.Although these 21 Innovations were initially selected, the budget provided for only 8 sites to be visited; but rigorous field visit planning resulted in 17 innovations being visited and surveys concluded on 14 sites. 1 One of the innovations, CUT is based in South Africa but has a subsidiary in Kenya. 15 Table 3. The Selected Sample: Country and Innovations, Visits and Revisits Country and Innovation Country total and status Bangladesh 2 Lal Teer Visited and Revisit Surveys Practical Action Visited (FGD) and Revisit Survey Egypt 1 ICBA Visited and Revisit Surveys Ghana 2 Ignitia Visited (survey conducted) no Revisit Skyfox Visited and Revisit Surveys India 7 EDR Not visited Adaptive Symbiotic Tech Not sufficient time to visit MyRain Not responsive SWAR Visited, survey conducted, no Revisit Naireeta Visited and Revisit Surveys WASTE Stichting Visited and Revisit Surveys NewSil Visited and Revisit Surveys Kenya 4 IRK Sunculture Visited CSDES M-FODDER Not responsive ITIKI CUT Visited and Revisit Surveys Hydroponics Africa Visited, survey conducted South Africa 3 Ecorangers Visited survey conducted, no Revisit Reel Gardening KII, recent evaluation, no Revisit CUT Visited and Revisit Surveys Uganda 2 Aquaponics Farming (WGI) Visited and Revisit Surveys Green Heat Visited and Revisit Surveys Grand Total 17 out of 21 sites visited, 12 revisits For a number of reasons (such as the recent other evaluations, unavailability of innovators and insufficient time) not all of the 21 innovations in the sample were visited during the First Visit. These obstacles led to 17 innovations being visited and surveys undertaken. In the Revisit interventions, 16 there were other reasons for not visiting, including overlapping evaluations, inconvenient times and other good reasons not to undertake further surveys; these reasons were provided to the SWFF Team Lead and agreed upon. Late in 2019, there were surveys undertaken by SWFF Field Evaluators using a similar instrument as in the case of the Dexis surveys. The Revisit and Field Evaluator surveys have achieved a considerably larger sample of beneficiaries from the First Visit. The use of common indicators and questions from instruments previously tested in the field has also achieved greater comparability and reliability. Data Collection: Qualitative, KII There were initially 24 proposed Key Informant Interviews with persons selected on the following criteria: leading FPs, DST, IIAC from emerging and developing countries, most experienced IIAC (generally participating in three rounds) and gender dimensions. Fortunately, participation in World Water Week (August 23-31, 2018) supported by SWFF offered an opportunity to undertake considerably more KII than initially planned and a total of 39 were completed. A well developed and appropriately structured instrument aligned to Nvivo was prepared. This provided the instrument for data capture during the KII and FGDs and, in Nvivo, led on to auto￾coding from a set of pre-codes prepared for all major questions. FGD and KII Preparation and Data Capture A facilitator fully familiar with the language of beneficiaries moderated the FGD sessions together with the team member. Each session was recorded, and the Consultant indexed discussions to identify key sections of an interview for full transcription for potential reporting to give concrete expression to new insights, key trends and common assumptions. Data Analysis Plan In undertaking Data Analysis, the team has given priority to the computation of respondent￾assessed outcomes, thematic outputs derived from KII, the ranking of issues and innovation priorities, and findings made from the analysis and structuring of the qualitative data. The two processes of qualitative and quantitative analysis have been drawn on simultaneously and interactively; leading on to tables, rankings, and sections of narratives. Qualitative Analysis Qualitative data sources include the KII which provide a range of expert opinion from the TAF itself, in the FPs, the IIAC and the Innovators. The KII instrument was designed to incorporate the Outcome Harvesting approach to elicit responses on the identification and self-definition of outcomes, of retrospective causation for these outcomes and substantiation of both these Outcomes and Drivers within and beyond the SWFF framework. The KII instrument was designed and formatted in Word to provide for the selection of pre￾coded attributes and auto-coding in Nvivo to lead to rapid data capture and analysis. The question guide was developed into a Nvivo aligned instrument, tested, reviewed and improved. While the use of Nvivo was initially more time consuming it did enable rapid data capture and auto-coding is leading to and more rigorous analysis and quicker reporting. 17 Quantitative Analysis With data provided from innovators the team planned the field visits to identify the universe of beneficiaries/customers/end users from which to draw a sample for surveys, focus group discussions, observations/site visits, and interviews. This provided a purposive sample stratified by region, site, and gender and further stratified by availability. Analysis was initially made in Excel to establish the frequencies, etc., and the metadata compiled (such as number of responses by project status, open-ended responses requiring coding, etc.). Data quality in the wider sense is central to the evaluation and metadata files have been assembled. The Dexis Agricultural Expert reviewed the drafts of KII and questionnaires were customized to each innovation to ensure that no gaps remained on agricultural aspects. She has given particular attention to measures of agricultural production and to the types and forms of interventions. She examined the types of crops and their significance to households; for instance: whether crops are subsistence or cash crops and whether the intended intervention is for a partial or greater aspect of agricultural production. The details of the First Visit, Revisit and Field Evaluator surveys of beneficiaries are presented below: 18 Table 4. Demographics of Surveys (First Visit, Revisit and Field Evaluator) by Innovation and Country # Innovation Country Visit (12 innovations) Revisit (12 innovations) FE (11 innovations) Respondents Respondents Respondents N Male% (N) Female (N) N Male% (N) Female% (N) N Male% (N) Female% (N) 1 Aqysta Nepal N/A N/A N/A N/A N/A N/A 52 85% (44) 15% (8) 2 AST India N/A N/A N/A N/A N/A N/A 50 88% (44) 12% (6) 3 Aybar Ethiopia N/A N/A N/A N/A N/A N/A 50 90% (45) 10% (5) 4 CUT ITIKI Kenya 4 50% (2) 50% (2) 31 35% (11) 65% (20) 62 27% (17) 72% (45) 5 CUT Swayimane South Africa 23 22% (5) 78% (18) 31 3% (1) 97% (30) N/A N/A N/A 6 Greenheat Uganda 8 50% (4) 50% (4) 35 40% (14) 60% (21) 58 52% (30) 49% (28) 7 Hydroponics Africa Kenya 13 15% (2) 85% (11) 34 20% (7) 80% (27) 48 22% (11) 77% (37) 8 ICBA Egypt 24 100% (24) 0% (0) 19 100% (19) 0% (0) N/A N/A N/A 9 Lal Teer Bangladesh 25 80% (20) 20% (5) 31 94% (29) 6% (2) N/A N/A N/A 10 MNP (Ecorangers) South Africa 27 78% (21) 22% (6) N/A N/A N/A 65 75% (49) 24% (16) 11 Naireeta Bhungroo India 25 4% (1) 96% (24) 25 32% (8) 68% (17) 51 2% (1) 98% (50) 12 Newsil India 10 80% (8) 20% (2) 21 57% (12) 43% (9) N/A N/A N/A 13 Practical Action Bangladesh N/A N/A N/A 30 0% (0) 100% (30) N/A N/A N/A 14 Skyfox Ghana 10 80% (8) 20% (2) 34 76% (26) 24% (8) 59 59% (35) 40% (24) 15 Waste India 22 23% (5) 77% (17) 26 58% (15) 42% (11) 56 58% (33) 41% (23) 16 WGI (Aquaponics) Uganda 10 30% (3) 70% (7) 30 47% (14) 53% (16) 48 72% (35) 27% (13) Overall 201 103 98 347 156 191 599 344 255 Average 17 51% (9) 49% (8) 29 47% (13) 53% (16) 55 58% (31) 42% (24) Source: First visit, revisit and SWFF Field Evaluator surveys Note: Not all First Visit surveys undertaken are entered, those innovators which did not have a Revisit or Field Evaluator survey are excluded. Inclusion would have led to too many gaps in the data in subsequent tables. To reduce the complexity and potential missing data, 2 innovations have been taken off the list of initial visits (Ignitia, Ghana and SWAR, India) as they were not selected to be revisited neither by the Dexis team, nor by the field evaluators. This drops the total number of innovations 19 providing full data during the First Visit from 14 to 12 and the total number of respondents from originally 221 to 201). A comprehensive set of data on the demographics for the First Visit, Revisit and Field Evaluator surveys are presented in the table. The relatively small samples from the First Visit included data from: 12 innovations, with a total of 201 respondents (male: 51% (103); female: 49% (98). There are considerably larger samples in the Revisit and Field Evaluators Surveys. The 12 Revisit surveys have a total sample of 347 with 156 men and 191 women; the comparable figures for the Field Evaluator surveys are a total sample of 610 including 344 men and 266 women respondents. To what extent have the surveys achieved gender balance among respondents? Overall, the survey interview design of Revisit and FE surveys ensured sensitivity to gender by ensuring a proportion of the sample to women in each case: 53% of the respondents’ sample being female. A high level of women respondents can be seen in the revisit surveys for Practical Action, Bangladesh (100%), CUT Swayimane, SA (97%), and Hydroponics Africa, Kenya (80%). For the FE surveys, a high level of women respondents can be seen for Naireeta Bhungroo (98%), Hydroponics Africa, Kenya (77%), and CUT ITIKI, Kenya (72%). Table 5. Surveys of Customers/End-Users, Revisit Surveys (Gender, poor and household characteristics) # Innovation Name N Total % Male (n) % Female (n) % Poor (n) average household size 1 Practical Action 30 0% (0) 100% (30) 60% (18) 5 2 CUT Swayimane 31 3% (1) 97% (30) 100% (31) 7 3 Hydroponics Africa 34 20% (7) 80% (27) 80% (27) 6 4 Naireeta Bhungroo 25 32% (8) 68% (17) 84% (21) 6 5 Newsil 21 57% (12) 43% (9) 81% (17) 5 6 Waste 26 58% (15) 42% (11) 16% (4) 4 7 Skyfox 34 76% (26) 24% (8) 16% (5) 8 8 Lal Teer 31 94% (29) 6% (2) 6% (2) 4 9 ICBA 19 100% (19) 0% (0) 37% (7) 6 Total (Average) 28 49% (13) 51% (15) 53% (15) 6 Source: SWFF Field Revisit Surveys Note: Data in the tables is ranked according to percentage of women respondents, from highest to lowest. The table above indicates that in the Revisit surveys there was an average sample of 28 and that of that 49% (13) were men and 51% (15) were women; those who were identified as poor were 53% (15) of the sample and that the average household size was 6. 20 The sample indicates a high proportion of women and poor being represented in the sample. Generally, those innovations with the highest number of women respondents were found also to have high levels of poor. Table 6. Surveys of Customers/End-Users, Field Evaluator Surveys (Gender and household characteristics) # Innovation Name N Total % Male (n) % Female (n) Male average age Female average age Average household size 1 Naireeta Bhungroo 51 2% (1) 98% (50) 45 45 6 2 Hydroponics Africa 48 22% (11) 77% (37) 41 46 5 3 ITIKI 62 27% (17) 72% (45) 55 47 6 4 Greenheat 58 52% (30) 49% (28) 56 55 9 5 WASTE 56 58% (33) 41% (23) 51 47 4 6 Skyfox 59 59% (35) 40% (24) 42 40 9 7 WGI 48 72% (35) 27% (13) 49 46 6 8 MNP 65 75% (49) 24% (16) 62 62 6 9 Aqysta 52 85% (44) 15% (8) 44 40 6 10 AST 50 88% (44) 12% (6) 55 51 8 11 Aybar 50 90% (45) 10% (5) 42 55 6 Total 599 58% (344) 42% (255) 49 49 6 Source: SWFF Field Evaluator Surveys Note: Data in the tables are presented according to percentage of women respondents, from highest to lowest. The table above indicates that in the Field Evaluator surveys, there was a total sample of 610 and of that 58% (344) were men and 42% (266) were women; the average household size was 6. Men and women interviewed were of the same average age, 49 years old. In tables produced from Revisit and Field Evaluator Surveys there has been no disaggregation between poor and very poor as this distinction was not calibrated in interviews. However, both surveys paid a special attention to poor in the sample, especially CUT Swayimane, SA (100%), WGI, Uganda (87%) and Naireeta Bhungroo, India (84%) for the Revisit surveys, and Naireeta Bhungroo, India (100%), Skyfox, Ghana (71%) and Waste, India (67%) for the Field Evaluator surveys. Household size across innovations varies between 4 to 8 and, particularly in the case of large households, can be taken as indicating the spread of benefits within the household. 21 Table 7. Surveys of Customers/End-Users, Field Evaluator surveys (levels of poverty and income) # Innovation Name % extreme poor (n) % low income (n) Extreme Poor average age Low income average age % women extreme poor (n) % women low income (n) 1 Naireeta Bhungroo 100% (51) 0% (0) 45 0 98% (50) 0% (0) 2 AST 90% (45) 4% (2) 54 50 83% (5) 0% (0) 3 Skyfox 71% (42) 20% (12) 41 39 87% (21) 4% (1) 4 WASTE 67% (38) 14% (8) 50 44 86% (20) 8% (2) 5 WGI 62% (30) 20% (10) 44 61 53% (7) 38% (5) 6 ITIKI 61% (38) 1% (1) 49 24 48% (22) 2% (1) 7 MNP 60% (43) 31% (22) 62 60 68% (11) 45% (5) 8 Hydroponics Africa 58% (28) 12% (6) 43 53 59% (22) 16% (6) 9 Greenheat 51% (30) 15% (9) 56 57 71% (20) 14% (4) 10 Aybar 0% (0) 14% (7) N/A 46 0% (0) 40% (2) 11 Aqysta N/A N/A N/A N/A N/A N/A Total 62% (345) 13% (77) 49 43 65% (178) 17% (26) Source: SWFF Field Evaluator Surveys Note: Data in the table above is presented according to percentage of extreme poor respondents, from highest to lowest. The Field Evaluator Surveys have a sufficient sample size to disaggregate categories such as “women who are extremely poor”; in analysis we have tried to get the “vulnerable” and “extremely vulnerable” perspectives. The definition of poor relates to the levels of median income in each country issued by the World Bank and discussed for the relevance with the innovators. There has been a challenge in distinguishing between low income and low/middle income as most Customers/End-Users are in the low-income category. RISKS AND LIMITATIONS OF THE EVALUATION DESIGN The evaluation team has been conscious of the risks and limitations associated with delivering an evaluation of the quality expected by the Founding Partners and the wider SWFF community. To overcome these risks and limitations, the team has drawn on the considerable strengths of the team members with two having significant experience in the water sector and one as a well￾qualified agricultural expert. Team members have brought significant experience in a number of SWFF portfolio countries including Kenya, South Africa, Ghana, Bangladesh and India. One 22 member of the team has had extensive experience with SWFF itself in the MTR. All have had solid experience with multi-country and multi-level evaluations. The team has brought considerable experience in the compilation of datasets and data analysis using Excel extensions and experience in qualitative data capturing and analysis using Nvivo software. The team recognizes the following risks and limitations that may have affected our ability to draw conclusions with high levels of confidence on some aspects of the program: Risks and Mitigation Measures Risks Mitigation Measures Taken Insufficient face-to-face time with innovators in country due to scheduling challenges. While the time frames were relatively tight in terms of innovator visits, efforts were made to connect with innovators by Skype/phone and email in instances where follow-on questions could not be addressed due to time limitations for face-to-face meetings in country. Stockholm World Water Week (August 23- 31, 2018) also allowed a significant further opportunity to meet with most of the innovators that had been visited (with the exception of Egypt). Stockholm also allowed for an opportunity to attend SWFF sessions and to organize key informant interviews with all remaining innovators as well as with the Founding Partners and members of the IIAC and the TAF. There were follow￾up emails on occasion. Insufficient access to customers and non-customers to understand the success factors for the innovations. Efforts were made to plan as far in advance with the innovators to impress upon them the need to meet with as many customers and non-customers as feasible within the allotted time. Time and logistical limitations did prove challenging, although the team did meet the target set of speaking directly with 200+ customers of SWFF innovations and well over 50 non-customers. In addition to the first visit, the revisit of 12/17 of the innovations in the sample with a tested instrument enabled a greater number of farmers to be included in the survey and greater depth of analysis to be undertaken. In addition, the data from 11 Field Evaluation surveys provided additional data on customers/end-users and further analysis to be made. In the first phase there were three different team members travelling to undertake the first visit of innovators. There were potentially resulting in slightly different approaches and different interpretations of the data presented. In the second phase there were Revisit surveys of the innovations in the sample. In addition SWFF Field Evaluators conducted a In the first phase the team was careful to schedule the survey protocols in advance in such a way as to ensure that it visited as many innovators as possible. Contact among the team and SWFF TAF was maintained between rounds of visits made to cross-reference and ensure that the team’s approach was as consistent as possible. The Key Informant Interview protocol was based on the questions noted in Section C with clear indications of those questions that were for innovators and customers as well as for TAF, IIAC and FP interviews. Well-rehearsed methodologies were shared for Focus Group Discussion and surveys. A matrix was designed for uniformly capturing information from the desk review. Conducting the visits in two rounds (3 countries each) allowed for an interim period to compare notes and further refine the approach. This helped harmonize the approach in advance, identified differences in interpretation were resolved before the team undertook interviews in Stockholm. In the second phase, prior to local consultants being mobilized to undertake the Revisit surveys, a standardized set of questions on key indicators were assembled. These were exchanged with SWFF TAF to ensure that the Field Evaluator instruments carried identical questions. 23 Risks Mitigation Measures Taken number of surveys of innovations with some overlap. There was the risk that even these standardized questions may not have carried the same meaning between interviewers and across innovations. Careful preparation had to be made to ensure the standardization of questions. In addition the visit protocols ensured that there was no overlapping during FE and Dexis visits. The research team has worked to mitigate variance in results and findings from data analysis by triangulating with access reports and interviews. Despite these efforts, there may be results which can not be fully explained without further research. Data collection methods needed to be consistent across visits. The team had not fully anticipated the challenge of working through Fulcrum to collect data in the field and needed to shift to ODK and then to Kobo (which is fully compatible with ODK). Although challenging while in the field, the team was able to consolidate the approach with the support from local consultants. The Revisit instrument was developed and shared with Field Evaluators conducting surveys of farmers. using experience of the first visit, this instrument was developed with the benefit of testing in the field. Agriculture expertise may not have been consistently applied across all visits. The team has made a concerted effort to engage the agriculture specialist both at the macro level to review the surveys and KII protocols but then also to review the findings through a metadata analysis of the ODK/Kobo reports. Efforts were also made by each travelling team member to enlist the guidance of the agriculture specialist to prepare for his or her visit. This involved sharing marked-up SWFF documents in advance to clarify aspects around agriculture to be reviewed while the team member was in country. Local consultants were also brought in with specific expertise in and familiarity with the agriculture sector in country. The instrument used in Revisit and Field Evaluation surveys was designed to carry appropriate agricultural indicators. Limitations and Mitigation Measures Limitations Mitigation Measures The challenge of ensuring that sampling is sufficiently representative of the portfolio. Efforts had been made to ensure that the information being collected during visits has been sufficiently diverse to cover as many different kinds of interventions across as many geographies within the portfolio as possible given the time and funding constraints. While the team met the target of 200 customers of SWFF innovations, indeed in some instances, the sample size was more limited. The team member then needed to approach the task in the sense of confirming that the SWFF data makes logical sense based on the information collected in country rather than validating through a significant sample size. The comparability of data across a diverse group of innovators This remains a generic challenge for SWFF. The team has been mindful of this at all stages of the analysis. The careful coding of the KII protocol and the design of the surveys has been critical to ensuring as much comparability as possible. Clarity on contribution / attribution with regard to SWFF’s acceleration support. The team will work to triangulate from multiple sources in assessing contributions and proportional contributions from many points of support. The comparability across different Challenge Funds in terms of costings will prove challenging as they each operate in different ways and offer different support. Finding the right comparators may not be straightforward. SWFF recognizes that this will have been a challenge and the team has liaised with TripleLine to help clarify approaches to this issue and sought advice from policy makers who have experience in a number of Challenge Funds. The evaluation team has conducted its own initiative to gather information on funding, costs of management and other indicators of the efficiency in the allocation of funds. 24 FINDINGS SECTION 1: RELEVANCE Question 1: Technically, what types of innovations have been supported? How many innovations have been awarded funding? What are the basic demographics/ descriptors of innovators? (H2) SWFF has the objective of helping farmers around the world to grow more food with less water through enhanced water storage and more effective irrigation practices, improved soil quality, and the use of saline water. SWFF does this by ensuring that entrepreneurs and scientists behind groundbreaking new approaches are getting the support they need to apply and expand their solutions. The relevance of this overarching strategy in an era when water use is increasingly contested, and agriculture requirements are growing exponentially is unequivocal. To understand SWFF’s relevance, the team explored the focus of the innovation’s supported, the locality of the innovation, business goals (including for-profit and non-profit) and social objectives of the innovations, their types in terms of outputs (products and services), and the demographics of management. This is all with a view to analyzing how SWFF supports innovation, spreads risk and ensures potential social and agricultural returns (as detailed more in the sections below on effectiveness, impact and sustainability). SWFF’s internal M&E data and key informant interviews guided the findings from a more operational standpoint. To understand the focus of applicants to the SWFF program, a set of questions was sent out to SWFF finalists (not awarded entry into the program) and grantees. 25 SWFF awardees and 76 non-awardees provided details on the objectives of their enterprise and the focus of their innovation, as shown in the table below. Table 8. Self-description of innovations, sets of priorities Answer Options [Percentage (Respondents)] % of SWFF N=25 % of Non SWFF N=76 Water efficiency - 77% (78) 84% (21) 75% (57) Green technology - 56% (57) 48% (12) 59% (45) Irrigation - 55% (56) 72% (18) 50% (38) Extremely vulnerable/extreme poor - 45% (45) 48% (12) 43% (33) Improved water access - 39% (39) 24% (6) 43% (33) Gender integration - 37% (37) 48% (12) 33% (25) Green energy - 29% (29) 4% (1) 37% (28) Infrastructure - 22% (22) 8% (2) 26% (20) Women customers - 18% (18) 24% (6) 16% (12) Access to credit - 16% (16) 20% (5) 14% (11) ICT/App (mobile or other) - 9% (9) 12% (3) 8% (6) Source: Online Impact Survey 2019. Note: Please select up to five keywords that best describe your objectives and innovations. Since innovators could report more than one priority, percentages reflect the total mentions of the top 11 responses. 25 Based on how they self-categorize, as expected, most SWFF innovations conform closely to the programmatic goals of greater agricultural productivity with more efficient use of water (84% of SWFF and 75% of non-SWFF respondents). Irrigation makes up almost three quarters of the SWFF respondents but only half of non-SWFF. Green technology is used by half of SWFF respondents to describe their activities but almost 60 percent of the larger pool of non-SWFF respondents define their offering in this way. SWFF has a larger focus on gender integration (48% vs 33%), extremely vulnerable/poor populations (48% vs 43%), and women as customers (24% vs 16%), reflecting SWFF’s focus in this area. Non-SWFF innovations self￾describe more as providing green energy (37% to 4%) and, somewhat unexpectedly, improved access to water (43% to 24%). What is perhaps surprising is that, given the focus on water efficiency, not more SWFF enterprises are focused on some form of innovation in infrastructure to enhance efficiency and higher crop per drop yields. The number of innovations accepted into the program in each Round has gradually declined from 16 in Round 1, 12 in Round 3 and 10 in Round 4. Although there has been a downward trend, the locality and orientation of innovations has changed in favor of developing and emerging economies and with more of an emphasis on meeting the needs of poor or the poorest farmers and particularly those of women farmers. There are significant differences technically between innovations; some are capital intensive with outputs such as a standalone water filtration infrastructure, others provide a readily sellable product such as treated seeds, and others provide a service, usually an information service through a mobile application. Reviewing the sampled offering of 23 Current, Alumni and Graduate innovations, 7 are infrastructure based, 8 are product based and a further 8 provide a service. The nature of the product or service logically affects the ability of farmers to invest and thereby innovators to scale up in certain markets, i.e., those with expensive infrastructure being slower to take off than those with relatively inexpensive additional units with information technology customers. Depending on the cost, when the product or service is used in the agriculture cycle (pre-planting to post-harvest) also has a bearing on farmers’ ability to invest or purchase what is on offer. The response of the customer base, noted below, highlights the relevance of the SWFF-supported innovations to farmers. SWFF prioritizes interventions in developing and emerging economies where there is the greatest need and potentially the greatest demand for the innovations it supports. Northern innovations have been encouraged to explore markets and establish subsidiaries in developing and emerging economies and local innovations (those based in the country or continent) are prioritized. This strategy has been reinforced Round by Round as the number of local innovations indicates in Table 8. Table 9. Locality of innovations supported by Round, 2020 Rounds Local Non￾local Grand Total 1 5 11 16 2 2 2 3 6 6 12 4 9 1 10 Grand Total 20 20 40 Source: Awardee Results database, SWFF TAF 26 In Round 1, most of the innovations supported were non-local to developing and emerging economies with 11 compared to 5 local innovations. By Round 3, this had been adjusted to 6 local and 6 non-local innovations, and by Round 4, the 9 local innovations were in a majority compared to the 1 non-local innovation. This strategy is strongly supported by the Founding Partners particularly in support of local innovations. “There is a clear shift to companies registered in developing and emerging economies. Any American or Swedish company can be registered in a developing and emerging economies, but the Swedish International Development Cooperation Agency (Sida) is keen to see more indigenous companies particularly with women in leadership positions.” (Sida Interview Respondent) A number of innovations that came from developed countries have risen to the challenge with the encouragement of SWFF to develop local partnerships to explore and expand these markets. One of the Round 1 graduates mentioned they have a local partner with extensive contacts to help market their product. While there has been extensive emphasis on gender integration, most of the existing innovations are male led. Among 21 key informant interviews providing clear responses to gender balances, the SWFF portfolio is represented by 15 male led innovations with 4 women led innovations and 2 institutions which are led jointly by women and men. The SWFF leadership is well aware of this and from initially nudging innovations in this direction is now intervening actively to encourage the necessary shifts in gender balance at all levels in the enterprise. Gender training and assessments are also helping innovators to recognize the business benefits of considering gender aspects more systematically in their business model and approach to their market. The social character of the innovation could be seen as a determinant of its commitment to poverty alleviation and the inclusion of women, vulnerable ethnic groups and the very poor as key target markets. From the 21 SWFF interviewees, 13 innovations were business ventures from inception and the remaining 8 described their innovations as predominantly focused at least initially around a social or societal commitment with a non-profit orientation. As the table shows, this proportion is roughly the same across all awardees with 22 describing their status as for-profit and 18 as non-profit organizations. Table 10.Profit or Non-profit Innovations, All Rounds Rounds Non-profit Profit Grand Total 1 7 9 16 2 1 1 2 3 7 5 12 4 3 7 10 Grand Total 18 22 40 The business orientation of innovations has shifted back and forth over the Rounds. In Round 1 just under 56% were for-profit innovations at application stage; in Round 3 just over 42%, and in Round 4 70% are for-profit innovations. A number of SWFF innovations have 27 reoriented their business model from non-profit to for-profit enterprises (CUT, CSA-MNP, World Hope, etc.) while engaged in the SWFF program. This could be to broaden their customer base and prepare themselves for other (commercial) funding opportunities. Registration status notwithstanding, the emphasis in the program is entrepreneurial in nature, with some innovators identifying as social entrepreneurs, or entrepreneurs with a social mission. The enterprises in the SWFF portfolio actually include a wide range of characteristics particularly of the founders. One innovation reports it is “quite heavily supported by a national government” as a state-run institute. An analytical chemist with prior SWFF funding pioneered one innovation. An economist who created four innovations in a variety of fields founded another SWFF-supported innovation. A further innovation was founded by an educationalist, who has lectured in Britain and America and has eight innovations to his name. Further demonstrating the wide variety of backgrounds, one innovation founder is a former trade unionist who is dedicated to eliminating poverty throughout his country and the world. SWFF’s tier system enables more funding to Tier 2 innovations based on their track record and potential for scale. Unlike other Grand Challenge Funds, SWFF has a relatively modest quota of Tier 2s including Reel Gardening (R1), Skyfox (R4) and Ignitia (R4). While SWFF has limited funding to support innovations per Round, the Tier system does not appear to be “crowding out” other innovations given the fewer numbers of high-quality applications from Tier 2 prospects. FINDING: In terms of SWFF’s relevance, much of the current development paradigm revolves around tailored or adapted market responses to lifting the poor out of poverty. Emphasizing access for marginalized and vulnerable populations has proven to be a constant thread of the SWFF program, with innovations largely selected as a function of their appropriateness for poorer farmers in increasingly water-stressed environments. SWFF has successfully transitioned to having a majority of local innovations. The balance between male and female-led innovations though remains heavily in favor of male-led enterprises. While the trends in the portfolio at application stage are not clear, several enterprises are making the transition from non-profit to for-profit. RECOMMENDATIONS: Going forward, WE4F should continue to encourage women led innovations to apply and make the proportion of women in management a criterion for selection to provide additional points in the selection process. WE4F should, from inception, follow the portfolio of awardees to understand how shifts in the business model (and registration status) are changing the willingness to or emphasis on reaching the poor and marginalized. Are these businesses using cross-subsidies or other mechanisms to continue to reach these target groups? Question 2. Is there demand and local ownership for the SWFF innovations (individually and across all innovations)? (H2) With regard to local ownership and local demand, of the 40 awardees, 20 are local innovations established by country nationals, i.e. locally registered, locally owned, and locally staffed. The narrative below seeks to unpack the different aspects of being “local” for SWFF. 28 SWFF has moved from predominantly more applicants from developed economies and awardees in the initial Rounds to a more balanced Northern-Southern applicant and awardee pool (as noted above). That said, there was a sense from some interviewees that the program is not yet grounded in the south – i.e. that there is insufficient participation from developing and emerging economies across the board. Efforts to redress this have been made in terms of the composition of the IIAC, seeking more active engagement from South Africa’s Department of Science and Technology as a Founding Partner, and efforts by the TAF and others to find channels of advertising SWFF to more developing and emerging economies audiences. It is recognized that identifying developing and emerging economy innovators who can apply for this kind of funding is not always straightforward due to language constraints, a lack of familiarity with or ability to navigate the application process, an inability to muster the arguments in sufficient detail to pass the initial screening stages, and other related challenges. These are all acknowledged by SWFF and efforts have been made to try to overcome them while recognizing the limitations of a small SWFF/TAF team with not a lot of spare resources. That said, this challenge further suggests the need to engage governments from developing and emerging economies, reach out to Founding Partner and other interested embassies, identify NGOs operating in this space, and identify other channels to expand the pool of applicants from the South even farther. In terms of “TA support providers”, there has been a move towards more local provision, often on the basis of cost considerations. Innovators also seem to appreciate technical assistance from TA support providers who, where possible, are sufficiently familiar with the local context and thereby can make more tailored recommendations. In most countries, there needs to be local registration and even some designated share of local ownership for foreign enterprises. Whether this translates into genuine local ownership and whether local shareholders have any influence on the direction of the company has been difficult to gauge for “non-local” awardees. For some innovators, finding skilled local staff has been challenging (either because the skillsets are not readily available or because people with the right skillsets are already working for large companies). SWFF TAF has provided support in some instances (Ignitia, for example) to sharpen their approach to hiring local staff. That said, it was noted in numerous conversations and in the sessions at a SWFF un-conference that the awardees remain very small firms for the moment and there is not a lot of spare cash and scope for expanding the employee base. Across the private sector, the competition is fierce in many SWFF local economies for talented staff who can manage accounts, effectively reach out to farmers and engage with other types of stakeholders, and help to advance the business in other ways. Indeed, for most of these organizations, individual staff members need to fulfill a wide variety of roles. Interestingly, innovators noted in the un-conference in Stockholm, 2019, that in many instances applicants for positions are looking for clear and unequivocal job descriptions with defined targets and roles. The mindset of many applicants is generally not that of wanting to make a contribution to the enterprise across a wide range of functions in a small organization. (Several also noted the challenge of the transition period as the company grows and requires more specialized skills and staffing.) All that said, a number of innovators have a stated goal to ensure a greater percentage of local staff in senior positions, like that of Alba to “reach 75 per cent Cambodians in the senior management team”. 29 In terms of local production, for some innovators like WASTE, construction inputs using locally sourced raw materials requires further innovations to ensure sufficient quality and viability of the product. A number of innovations that came from developed countries (such as AST and NewSil) are locating production facilities in the countries in which they are expanding markets. With regard to local demand, for the innovation by farmers, this is reliant on a range of factors that revolve primarily around level of risk and price. Poor farmers are less able to take significant risks particularly for innovations that involve major investment in land or infrastructure. There was some suggestion that local demand could have been fostered further through connections to other related programs from embassies or by working with governments. For most innovators, there appears to have been limited or no contact between them and Founding Partner embassies / ministries. Connections to NGOs who also work with farming communities (or even tapping into synergies between innovators in the same country like SkyFox’s work with fishponds and farmers and Ignitia’s weather messaging service) could also make for meaningful partnerships and help each expand their customer base. There is thus a perception that SWFF efforts to engage more at this level would help innovators to overcome certain barriers or to gain greater uptake more quickly. Partnerships will be very innovator specific in terms of whether, given the nature of investment required, innovators will want to link up with providers of finance, municipalities for access to municipal lands, agricultural extension agencies for promoting uptake of the product or service, major corporations who could provide markets for offtake of farmer production, universities for internship programs to bring a new generation into the awardee’s workforce, or other opportunities. A partnership mapping exercise, based on the goals of individual innovators should help to flesh out these opportunities.2 Ultimately as a multi-donor fund, the connection to SWFF has helped to extend innovators’ credibility at the local level. In some contexts, this has also proved challenging as the expectation is that the innovator was there to provide more traditional development support at no or low cost (i.e. “people didn’t understand why we would sell something while having support from USAID”). FINDING: SWFF has made progress in further localizing its support to awardees. There is widespread local need in developing and emerging economies for the innovations and potential for local ownership. The challenge is the ability to pay. Innovators are generating effective demand as well as meeting existing demand from better off farmers for their infrastructure, products and services. The ability to pay in the local context often includes the availability of inexpensive locally available financing options for poor or very poor farmers. RECOMMENDATION: Further efforts to localize WE4F support would involve greater engagement with funder embassies and other donor programs with NGOs operating in related activities. Rather than treating awardees as operating in isolation, this could create synergies among funded activities allowing for greater visibility, reach and impact. Embassies and other donors are likely to also be able to make the links to policy and national government programs that would support the awardees in positioning and marketing their innovation in 2 WASTE, for example, has created a link with commercial banks in India and Kenya to provide credit to households for sanitation investments. (See FINISH Mondial project at https://www.waste.nl/causes/finish-mondial/) 30 the local context. More effort could be placed on mapping the kinds of partnerships that would help innovators strengthen their offering and expand their customer base. For example, WE4F should consider the most effective ways of partnering with or incorporating programs that offer locally available low-interest finance without over-burdening the innovations. SECTION 2: EFFECTIVENESS Question 7. To what extent did the SWFF TA Facility provide SWFF innovators with timely and appropriate technical assistance that led to the creation or enhancing of a viable business centered around an innovation that saved water / generated more agricultural product? In addition to being a funding source, SWFF negotiates targets and milestones with carefully selected innovations and provides technical support to fill mutually identified gaps to augment their capacity and encourage sustainable growth. The data from interviewees, surveys, field visits, and SWFF secondary data was analyzed to determine whether the target and reporting system supported the creation of a viable business, and whether the direct TAF support and the contracted technical support was effective. In answering this question, the evaluation team reviewed two main types of assistance under the SWFF Program and an alternative including SWFF TAF support provided directly to awardees in M&E, financial reporting, and management; SWFF-funded technical assistance (TA) from contracted providers, and technical assistance paid for by SWFF innovations directly. The data analysis supports the finding that the SWFF support to innovations (both directly provided and contracted out to service providers) is strongly appreciated and that it has led to the development of mature innovations. The overwhelming majority of innovators during interviews reported that the technical assistance was timely and appropriate, helping them to create viable businesses. A minority of respondents was concerned about TA not always being available when needed or TA providers not being the best suited for their needs, an issue which is further explored below. With this exception, the interviewees give high approval of TAF support and technical assistance from service providers, stating that it was comprehensive but also helped innovations overcome particular obstacles. The Innovation Success Rate table below shows data analysis of responses from SWFF and non-SWFF innovations to the Online Impact Survey regarding the development stage they reached over how many years. 31 Table 11. Chart Innovation Success Rate Source: Data analysis from Online Impact Survey Development Stage questions Analysis of the chart above finds that SWFF innovations were more likely (64% vs 44%) than non-SWFF innovations to reach Maturity (i.e., early commercialization or commercialization stage) through all the Growth Stages (development and initial pilot). Out of the 19 SWFF innovations and 64 Non-SWFF innovations with reported data, 18% (n=4) of SWFF innovations did not reach maturity with SWFF support compared with 46% (n=33) of Non￾SWFF with non-SWFF support. Not counted in this analysis are innovations that were in growth stages in 2018 (the last reported year) or those with no data for this question in the online impact survey. Tables 12 and 13 below, Comparative Stage of Growth over Innovation Runway period, compares the growth trajectory or innovation runway for SWFF and Non-SWFF innovations over a five-year period. Not all innovations went through five years according to the data. The bolded cells show the year of growth (Y1-Y5) that the highest percent of innovations were in at each Stage of Development (Development through to Commercialization). The first table compares SWFF to non-SWFF growth during early stages (development, initial pilots, multiple pilots). The second table compares SWFF to non-SWFF innovations in the mature phases of growth (Early Commercialization and Commercialization). More in-depth data would need to be collected to ascertain sustainability of such a trajectory. The majority of SWFF innovators that responded to the Online Impact Survey were either graduates or alumni at the time of or soon after taking the survey. This needs to be explored more, but could suggest sustainability beyond the SWFF period of performance. This is likely a period of less than one year though and a sustainability study can be undertaken now since more time has lapsed since Round 1-3 innovators left SWFF. One limitation of the survey is that because it captured non-SWFF respondents as well, we did not explicitly ask about SWFF status at time of response. As shown in the tables, SWFF innovations have what the team concludes to be (based on secondary research and understanding of innovation runways) a more sustainable “runway” to sustainable growth and outcomes, with incremental changes over years that we know to 32 be paired with technical assistance to fill gaps, while non-SWFF innovations at times attempt full Commercialization by Year 1, potentially without such assistance. Table 12. Comparative Stage of Growth over Innovation Runway period (Years 1-5): Initial Development Period, SWFF vs. Non-SWFF Development Stage Initial pilot stage Multiple pilots Year SWFF Non￾SWFF SWFF Non￾SWFF SWFF Non￾SWFF Y1 100% 100% 32% 51% 0% 26% Y2 0% 0% 59% 5% 33% 31% Y3 0% 0% 9% 11% 33% 19% Y4 0% 0% 0% 5% 29% 14% Y5 0% 0% 0% 0% 5% 10% Source: Data Analysis from Online Impact Survey Question: How has your innovation developed over the past 5 calendar years? Table 13. Comparative Stage of Growth over Innovation Runway period (Years 1-5): Mature Development Period, SWFF vs. Non-SWFF Early commercialization Commercialization Year SWFF Non-SWFF SWFF Non-SWFF Y1 7% 25% 10% 32% Y2 7% 13% 0% 4% Y3 40% 44% 10% 21% Y4 40% 13% 50% 29% Y5 7% 6% 30% 14% Source: Data Analysis from Online Impact Survey Question: How has your innovation developed over the past 5 calendar years? According to data collected by innovators (SWFF and Non-SWFF) through the Online Impact Survey, 53% (n=10) of SWFF innovations compared with 43% (n=28) of non-SWFF innovations make it from Development Stage (earliest stage) to Commercialization (most mature); and 79% (n=15) of SWFF compared to 50% (n=32) of non-SWFF make it from Development Stage (earliest stage) to Early Commercialization (second most mature). This data supports the finding that more SWFF innovations reach a sustainable commercialization phase from either development stage or the initial pilot stage. Overall, the time it takes for SWFF innovations to reach maturity (defined as early commercialization or commercialization is 4.0 years while it takes non-SWFF innovations 3.1 years, giving SWFF innovations a longer innovation “runway”. However, only 44% of innovations reach early or full commercialization from pilot with 46% (n=33) of innovations not making it past multiple pilots. SWFF however sees a 64% “success” rate in reaching early or full commercialization with only 18% (n=4) of innovations not making it past multiple pilots. The chart in table 14 below compare the development of SWFF and non-SWFF innovations. The first chart shows the SWFF innovations’ development stage timeline while the second chart shows the Non-SWFF innovations’ development stage timeline. This definition of maturity for the effectiveness section only considers stage of development towards commercialization and not the broader SWFF objective of social criteria and benefits to women, poor, youth, or other vulnerable populations. These broader issues are explored more in depth in Sustainability. 33 Table 14. Side-by-Side Comparative Stages of Development Timelines for SWFF vs. Non-SWFF from development stage to commercialization over five years Source: Online Impact Survey: How has your innovation developed over the past 5 calendar years? In the transition from Year 1 of their life span to Year 5, there is a higher level of SWFF innovations starting at the pilot stage, with a handful of non-SWFF innovations beginning in midpoint and maturity stages. In Year One 93% of the SWFF innovations at in the growth phase and 7% are at midpoint. Year five 20% of innovations are at their midpoint and 80% in their mature phase. In each phase there are higher percentages of SWFF innovations than Non-SWFF innovations in the more mature phase. One would expect, based on knowledge of an innovation runway, that the growth trajectory (blue line) would decrease at a steady rate over five years (showing that innovations are reaching higher stages of development) and a grey line (maturity) that steadily increases over time. In each successive year, there is a shift from pilot stage to midpoint, and from midpoint to maturity. As expected, the proportion of pilot innovations declines over the five-year period. Non-SWFF innovations show a similar transition but at a faster, yet lower overall, rate. 80% of SWFF innovations vs 60% of non-SWFF innovations are in maturity at Year 5. The data collected from the Online Impact Survey, looking at experiences of both SWFF and non-SWFF innovators in terms of the effectiveness of SWFF vs non-SWFF TA, buttresses the data collected through interviews and in both iterations of field-based data collection. SWFF 34 innovators cited SWFF TA as being either “more than expected” or “significant” in terms of satisfaction and effectiveness, while non-SWFF technical support as “not at all” or “somewhat”. Instead of disaggregating the data, Table 15, presents the data as “Not at all/Somewhat” vs. “Significantly more than expected/more than expected” to show the hard line drawn between SWFF and Non-SWFF TA. Table 15 below with red (Non-SWFF Support) and blue (SWFF Support) is the SWFF respondents to the Online Impact Survey that cited SWFF vs Non-SWFF support as being “not at all” or “somewhat satisfied”. The Non-SWFF Technical Support (red line in chart below) is assessed by SWFF innovations as more “Not at all/somewhat” effective than SWFF Technical Support (blue line). In relation to every goal, non-SWFF technical support is assessed as more likely to be ineffective. In improving primary customer/end-user outcomes of innovations (agricultural productivity, water efficiency, access to water) the non-SWFF Technical Support was 80-90% more ineffective than SWFF support. Table 15. Graphic: SWFF innovator assessment of SWFF/NON SWFF technical support, Not at all/Somewhat Satisfied Source: Online Impact Survey 2019. Chart Satisfaction with SWFF vs Non-SWFF Technical Support “. As a result of SWFF or other innovator/accelerator technical support (i.e. not funding), have you been able to:” Responses to SWFF vs Non-SWFF technical support of “not at all” or “somewhat” Table 16 below with only one color is the SWFF respondents to the Online Impact Survey that cited SWFF vs Non-SWFF support as being “more than expected” or “significantly more than expected”. As demonstrated by the use of only one color in the chart (grey for SWFF “more than expected”) SWFF innovations cited only SWFF support (and not non-SWFF technical support) as being “more than expected” in its ability to improve outcomes, leverage funding, enhance business practices, increase production, and better target customers. Using satisfaction with Technical Support as a measure, the highest level of satisfaction is with SWFF technical support in improvement in access to water for beneficiaries and least satisfaction with increased production capacity. Non-SWFF Technical Support do not receive any level of satisfaction, more than “somewhat” in areas of “increase your female customer base” and “increase production capacity.” 35 Table 16. Graphic, “More than Expected” Satisfaction with Technical Support, SWFF vs. Non-SWFF Innovators Source: Online Impact Survey 2019. “As a result of SWFF or other innovator/accelerator technical support (i.e. not funding), have you been able to:” Innovator responses to “more than expected” and “significantly more than expected” SWFF support was, across all types of support provided, cited as “more than expected”. The table and analysis below discuss the sub-categories of support and the findings. Overall, there are higher levels of satisfaction with SWFF than Non-SWFF technical assistance. The data, under the Annex Effective 2.2 Contrasting Comparative outcomes of Technical Support, shows poor levels of satisfactory outcome “Not at all” in relation to improving access to credit from the use of technical support from SWFF (38%). By way of contrast Non-SWFF technical support, high levels of “Not at all” are recorded for access to credit, business support (67%), water efficiency and leveraging funding (50%). These show high levels of dissatisfaction. Greatest levels of satisfaction (“significantly more than expected”) are recorded for Non-SWFF technical assistance at very low levels (0%). SWFF supported technical assistance records “Significantly more than expected” in relation to agricultural productivity and expanding customer base (20%) and with improved business practices, expanded female customer base and leveraging funding (16%). The data shows poor levels of satisfactory outcome “Not at all” in relation to improving access to credit for from the use of technical support from SWFF (38%). By way of contrast Non-SWFF technical support, high levels of “Not at all” are recorded for access to credit, business support (67%) water efficiency and leveraging funding (50%). These show high levels of dissatisfaction. Greatest levels of satisfaction (“significantly more than expected”) are recorded for Non-SWFF technical assistance at very low levels (0%). SWFF supported technical assistance records “Significantly more than expected” in relation to agricultural productivity and expanding customer base (20%) and with improved business practices, expanded female customer base and leveraging funding (16%). Overall, there are higher levels of satisfaction with SWFF technical assistance than Non-SWFF technical assistance. FINDING: The SWFF support structure is understood by innovators as helping them advance towards viability, and the support is widely acknowledged as critical to an innovation’s uptake by users. The constraints of working in a tight system of planning, targeting and reporting are less than the expansion of horizons and capacity development. The country visits are highly appreciated and can lead to the clarification of key aspects of targeting and reporting. RECOMMENDATION: None 36 Question 7a. Does the SWFF TA support help the awardee to overcome organizational capacity barriers? (These could include strategy development, financial forecasting capabilities, improved manufacturing approaches or product design, barriers to accessing funding, etc.). Table 17 below presents columns recording the most (Very effective, effective) and least effective (“Not at all”) organizational support. It suggests evidence of SWFF support being more effective (very effective/effective) in all areas of organizational support compared to Non-SWFF support. SWFF support is recorded as more “Very effective” or “effective” for all activities except for product design and production was cited as the only activity in which SWFF support was less effective (64%); than non-SWFF as 75% of SWFF innovators recorded this as “effective/very effective”. Business practices/organizational capacity was least likely to be labeled “not at all effective”. SWFF support for access to finance was also “more effective” than other types of technical support. Table 17. Comparative effectiveness of organizational support by area: Effective vs. Not at all Effective (very or effective) Not at all effective Technical Support Area (SWFF N) (Non-SWFF N) SWFF % (N) Non-SWFF % (N) SWFF % (N) Non-SWFF % (N) Product design and production (11) (12) 64% (7) 75% (9) 18% (2) 0% (0) Communications and/or branding (19) (11) 89% (17) 64% (7) 5% (1) 0% (0) Partnerships and networking (23) (14) 74% (17) 64% (9) 9% (2) 7% (1) Business Practices / Organizational capacity (24) (13) 71% (17) 54% (7) 4% (1) 8% (1) Access to finance (17) (13) 71% (12) 54% (7) 6% (1) 23% (3) Source: Online Impact Survey. “For each area of technical support received, from various entities, how effective was each support? Please respond for all types of support received from SWFF, non-SWFF, and internal entities.” As shown in the chart below, there are not many differences in funding allocation between SWFF and non-SWFF innovations. 37 Table 18. Funding Sources by Stage Online Impact Survey 2019: Funding Sources and Level SWFF vs. Non-SWFF The data carried in the graphic indicates there is a heavier reliance on public funds by SWFF innovations, likely due to SWFF itself being public funding, but there is diversification among the three categories: public, private, self-funded. The private sector funding levels are higher for non-SWFF than for SWFF, likely because of the need to find alternative funding streams when not in the SWFF program. Declining private funding into mature growth is a point to note. But it is important to note that the data shows a potential heavier reliance on SWFF funding over the years in the SWFF program (public funding). FINDING: SWFF support is focused on helping innovations to scale up and meet targets; the vending system has been considerably improved to meet innovators’ needs but some innovators would like to explore further the engagement of TA support providers to be more flexible and useful. By years 4 and 5 in the SWFF program, there is a reduction in private funding and an increase in public funding, compared to non-SWFF innovators, potentially highlighting the need for a review of barriers to diversifying funding sources prior to SWFF ending. RECOMMENDATION: More TA support providers from developing and emerging economies should be listed and other aspects of choice explored such as contributions from innovators themselves to match funds to engage TA support providers. There should be a strengthened focus in the WE4F program on identifying barriers to diversifying funding sources and supporting innovations in access to private capital or income-generation (through directly related revenue or other fundraising) prior to SWFF ending. Given the existence of many synergistic or related USAID and other Founding Partner programs, projects, and activities in the countries in which these innovations operate more can be done to identify and coordinate with existing programs that work in areas of low-interest finance, private sector engagement, women’s saving groups, agricultural and market systems development, and gender. Given the current operating environment, there may be changes post-COVID that need to be explored. 38 Question 7b. Did SWFF TAF enable certain (new) (combinations of) expertise to be deployed that would otherwise likely not have been deployed/used (by the individual partners)? SWFF aims to bring together the necessary expertise through a variety of means; through TAF itself, the TA support providers, SWFF network of innovators and by new partnerships. There is a wide range in responses to the question with most mentioning SWFF activities resulting in a combination of such expertise through new partnerships although a sizable minority does not. A number also felt uncertain whether SWFF itself rather than their own initiative was instrumental to these new partnerships. The direct support from SWFF is highly appreciated with a number of innovators reporting that TA support providers had helped innovations to break out of the pilot stage: We were just starting up as a pilot when SWFF extended support. Without SWFF, we wouldn’t have been a viable innovation…the idea never would have arrived. Many innovators show as great an appreciation for SWFF enabling contact between the innovators both within and outside of the SWFF network. The open SWFF forums, even more than direct interventions some suggested, led to the creation of new strategic partnerships. SWFF Conferences led to us being able to call on different people in that network and we do that frequently. We’ve partnered with SWFF awardees to improve our product and with international companies and agencies; all through SWFF. For this innovator, SWFF conferences opened opportunities to increasing the agricultural productivity from their product as well as getting advice about their institutional status as a social enterprise with an international organization. Another innovation wanting to attract investment reported that SWFF facilitated contacts which led to funding as well as for attracting additional Dutch social impact funding for the provision of low interest rate loans to customers. SWFF “helped a lot” in giving credibility to this innovation, in identifying partners, and in using its international links to secure investment. While innovators acknowledged SWFF enabled new combinations of expertise, they were also keen to show their own initiative particularly in relation to partnership with government, possibly a sensitive area for direct SWFF intervention. One innovation stated, “We have made our own links such as with the South African Weather Services because we can’t give forecasts without their knowledge.” On balance SWFF sought to provide support in expanding combinations of expertise but also had the expectation that innovators needed to take the initiative such as in SWFF conferences and beyond to explore expanding these relationships. FINDING: Finding the appropriate combination of expertise in support of an innovation is intrinsically difficult; while most feel SWFF has helped decisively, some innovators want to further explore ways of prioritizing their needs prior to engaging TA. RECOMMENDATION: Further experimentation with the vendor system in matching funds or another funding mechanism would be rewarded. 39 Question 7c. Were there additional barriers that were not addressed by the SWFF TAF? The term “additional barriers” is understood as those barriers outside of the SWFF framework which the innovators encounter and work to surmount. While most interviewees felt there were such unresolved barriers, a large number were uncertain, and a sizeable group felt additional barriers were not addressed. Indeed, not all innovators felt that the additional barriers could be successfully addressed by SWFF as these sometimes were part of a national or even international context far greater than the Program. For example, with regard to human resources, one innovator noted that, It is challenging to find the right person with the right commitment and inclination to work in the field. Getting someone to work at the office is very easy. This kind of work (as a rural development professional) is not known to a lot of people in India. People in India are looking for a white-collar job and this work does not fit in that criteria. Other interviewees gave somewhat contradictory responses, either reporting barriers but not receiving support but also appreciating advice on their business model. No such support was received. There were areas in which we were struggling. Though there are several problems that we are facing, we were asked to give the data. We did appreciate the support however, on our business model. Some barriers were identified by an innovator as very challenging but were not directly identified as being within the SWFF field of work. An unusual information technology challenge was encountered as very challenging to overcome; it took some time to identify just how this would be solved. No response from SWFF was reported but through iterative work with a TA support provider to reach the solution. A few innovators felt they had barriers which were particularly technical and possibly unique. This point is returned to below. FINDING: Additional barriers are reported by innovations but not all expected a resolution from SWFF itself. Some barriers (such as the lack of human resources) need longer time and iterative work to overcome. RECOMMENDATION: WE4F should further engage with innovators to list the barriers experienced and undertake online exchanges on the most common. Question 7d. Were there positive or negative unintended consequences of the support? Every intervention has unintended consequences of one kind or another. Predominantly interviewees reported positive consequences, others positive and negative, and a single respondent reported only negative unintended consequences. In the section on Sustainability the unintended negative consequences from innovations (not from SWFF support) are explored. 40 Among those that had positive unintended consequences, one innovator reported that their award led to unexpected access to “different grants with other funders like with FAO”. Improving internal systems could also lead to having the capacity to write better proposals. Upgrading software and reporting requirements helped us to understand the business better including targeting of funding…ensuring that requests for funds were mapped clearly against activities on the ground, etc. The institutional and personal contacts between TAF and innovators have had some positive consequences which were not entirely planned. SWFF staff told of one innovator as follows: Little by little the innovator listened more and more while previously he just was not open to the support. With space and time…he realized the business was going to collapse and he slowly changed. This led to starting a company and taking charge as CEO. Some of these positive consequences have gone beyond the innovation itself into a much wider social field with unintended results, in this case greater consciousness on the extent of climate change and the conservation of water. We ensured that the farmers are sensitized on the utilization of water, which is a scarce resource. We could also ensure that farmers realize the impact of climate change on availability of water. They have learnt the hard way and are now using micro irrigation systems. Amongst those who had experienced contradictory impulses in relation both to unintended consequences, some were aware of changes within themselves. We now think in a business way which may not be a negative per se but the sales training we had in the end took us away from the focus on our intended beneficiaries. The question of purely negative unintended consequences arising from innovations is explored more fully in Sustainability. FINDING: From the interviews SWFF support has mostly positive and not negative unintended consequences directly for the innovations or in a wider social context. An unintended negative consequence is that support has not been extended; another consequence of good performance may be continued dependence on available assistance. RECOMMENDATIONS: None. Question 7e. To what extent did the SWFF TA Facility provide SWFF innovators with technical assistance that innovators deemed useful? As discussed in detail above, most interviewees were very satisfied or satisfied with direct TAF support and the technical assistance from SWFF contracted service providers; while a small group was either unsure, “somewhat” or “not satisfied”. The overwhelming impression in interviews is appreciation of the advice communicated either on regular calls or online and particularly during field visits. The views of the minorities were not uniform, expressing 41 disappointment in some dimension with a suggested partner or a lack of specific technological advice or with delayed response. Most interviewees felt that the assistance they had received from TAF led to immediate success, or success over the longer term; these constituted the overwhelming majority the remainder were unsure or felt assistance was not successful. 23 SWFF innovators (100% for this question) in the online impact survey responded that SWFF value added is Significantly Higher compared to other technical support they’ve received. It could be extrapolated from SWFF’s stronger emphasis on technical support than other programs, that the technical support was considerably more effective than funding alone. SWFF innovators consistently valued the technical support higher than simply funding in growing their business and organizational capacity for more sustained outcomes. FINDING: Technical Assistance from service providers as well as direct support from the TAF are both key features of the SWFF design and are highly appreciated; many innovators consider it more important than the actual funding. Possibly because of its importance there is some concern to expand its parameters where possible. RECOMMENDATION: Demand-driven Technical Assistance as well as TAF Support and training based on identified business capacity gaps to address them should be made available in WE4F. Question 8. How effectively have investment risks been managed by the program? (number of failed projects, timeliness of reaction on problems observed etc.) (H3) Has the milestones-based tiered grant structure led SWFF to continue funding only the most promising innovations over time? (H3) SWFF innovations have a more sustainable period of commercialization, even if it takes longer to achieve, suggests that SWFF has effectively managed risk through its selection of innovations and the support provided. As shown in the comparative charts below, the SWFF innovations have higher percentages of “maturity” in years four and five than non-SWFF innovations 42 Table 19. SWFF Stages of Growth (Growth, Midpoint, Maturity) of Five Years 2014-2018 Source: Online Impact Survey, question: “How has your innovation developed over the past 5 calendar years?” The chart above indicates that on a year-by-year basis, the SWFF innovations advance from early consolidation of an innovative idea to mature market presence. It appears that the investment risk has been well managed by use of the IIAC and other procedures of review to redress weaknesses and build on strengths. From a comparative perspective, the Non-SWFF innovations follow the same line of advance but not at the same speed and do not achieve the same final proportion of achieved maturity. Some innovators also feel that necessary micromanagement by TAF has radically reduced risk taking. Responsiveness through regular review, modern communications, availability of key personnel, willingness to be in the field, capacity to provide assistance, reallocation of targets or of funds as needed have all contributed not just to the innovators’ success but also to SWFF’s management of risk. The IIAC experts provide an invaluable and much appreciated component of SWFF through their multi-disciplinary assessment of risks and rewards across innovations. It is generally acknowledged that a key challenge in assessing the innovators is that the IIAC will likely not really know the markets in which the innovators are operating. The program helps to unlock the reward through the seed money and reduce the risk through the requirements and reporting responsibilities placed on the innovators. An IIAC member acknowledges that the latter requirement may be burdensome and that their expert opinion may, in some instances, be overridden by the Founding Partners. Some analysis of the success rate of these Founding Partner “overrides" or the further work required of SWFF/TAF to ensure the success of these innovators not recommended by the IIAC but still brought into the program may be instructive. SWFF prioritizes the funding of selected innovators as Tier 2 awardees and their additional funding does not appear to have crowded out the Tier 1 awardees, and it may lead to the further strengthening of the smore successful innovations. 43 FINDING: As explained above, The Program is effectively run, ensures close attention to detail, and has a hands-on approach with the awardees. SWFF innovations have a more sustainable period of commercialization, even if it takes them longer on the innovation runway. The evaluation team finds that the SWFF risk has been effectively managed by the program in its selection of innovations, their tiering of funding, and their ultimate sustainability in success. RECOMMENDATION: SWFF should foster more exchange of information on the barriers that innovators are facing and efforts to overcome them. Initiating wider links with related embassy and other programs would further enhance SWFF’s effectiveness. Question 9. How well did founding partners interact within SWFF and what lessons should be taken from their interactions? By all accounts, SWFF has been a labor-intensive effort for all involved. Founding partners have had to stretch to meet the requirements (not in some ways unlike the innovators themselves) particularly in terms of time to dedicate to the program. SWFF has been run with a higher degree of intensity than most other challenge funds (as also noted by the TripleLine review for the Swedish International Development Cooperation Agency (Sida)) with more regular meetings and high expectations of partners to review lengthy applications and provide input to decision-making particularly as to whether innovators should be advanced to subsequent stages in the program. SWFF has a productive division of labor. The appearance is that with shrinking in￾house capacity, other founding partners have had to run to keep up with USAID, who installed a dedicated staff member to oversee the day-to-day functioning of the fund. With this, a high level of trust has grown out of an appreciation for the dedication and commitment as well as the attention to detail shown by the USAID manager. Regular meetings and consistent reporting have served to keep partners up to date (even as partner representatives changed) both on the progress of the innovators but also of SWFF as a whole. As a result, there is a high level of familiarity with the inner workings of SWFF by founding partner representatives, including the processes used to select, support, and graduate the innovators, as well as many of the innovations and their contexts. Founding partners make important policy as well as funding contributions. There has also been an appreciation for the diverse but generally complementary agendas of the different partners. With time, partners have come to anticipate the particular themes and kinds of questions that each has brought to the table about the program overall and with regard to innovator selection and support. A clear example is the Swedish International Development Cooperation Agency (Sida) emphasis on gender for which the agency put forward guidance, offered some analysis and leveled clear expectations as to how it expected SWFF to incorporate gender into the program. Gender did not become a measured indicator but there was a clear expectation that innovators would recognize and seek to optimize their impact in this area. This became more embedded in the selection criteria in later rounds. Thus, there was a sense from interviewees that founding partners could influence the design and shape of the program, although less on the day-to-day which was run by USAID rules. IIAC members also noted that founding partners had veto power that in some instances reinstated the application of an innovator that they had rejected. Although recognizing the “political” nature of these kinds of programs, some IIAC members questioned why they 44 as a diverse group could take considerable time to review and reject an application based on their professional judgment that was then still brought into the program. USAID is highly visible while most founding partners have low visibility in SWFF. The general sense from innovators is that the founding partners were not very visible throughout the process. Obviously working with USAID on a weekly and monthly basis in what was often described as an intense but supportive relationship left the other partners a bit in the background. That said, numerous interviewees noted that there were missed opportunities to tap into complementary support that could perhaps have been provided by embassies and related programs. Although nothing was to stop innovators from making these connections, it seems that SWFF itself was not forging these links. This ties in to other aspects of the review, which notes that, due largely to resource constraints, efforts to address bottlenecks in the wider enabling environment were not really embedded in SWFF. SWFF has an uncertain influence on other donor funding programs. It is difficult to note with any certainty how influential SWFF has been on other donor programming. Informal conversations with colleagues from related offices seemed aware of but not very familiar with SWFF. It was suggested that efforts by USAID to reach out to DGIS and to Swedish International Development Cooperation Agency (Sida) in a wider vein would have helped to spread the word more effectively and keep SWFF “on the front burner”. While interviewees suggested that SWFF had a positive influence on other programming, little actual evidence was put forward for the team to review of how other programs had been shaped in SWFF’s image or adapted elements of SWFF’s approach. That said, interviewees were doubtful that with limited human resources that SWFF could be replicated in any meaningful way in-house as implemented by USAID. Although there was an appreciation for the results and the contribution of the lean TAF and its efforts to ensure value for money, funders were less familiar with the actual running costs for SWFF and the staffing and other costs being absorbed by USAID beyond the $34m budget. So there remains a question of how much it actually costs to run a program of this nature and design. The founding partners include key developed country funding agencies and, despite the participation of South Africa, there is a deficit of participation from developing and emerging economies governments and institutions. Interviewees also mentioned that the frequent reference back to the developed country founding partners and their way of working left the program a bit skewed in terms of not necessarily being grounded in how businesses take off in different contexts. Ideally South Africa as a late-comer fourth partner would have played a more prominent role to redress the balance somewhat. It was suggested that future such arrangements should ensure more developing and emerging economies representation at an earlier stage. This has been expressed by a founding partner: It was obvious that SWFF needed strong perspective from developing and emerging economies and we helped push for more developing and emerging economies partners to join. There are few developing and emerging economies with the necessary depth of knowledge and of scientific community to make a significant contribution to the IIAC and in other dimensions. Those which had the agency and the capacity to make an impact were not as responsive as other partners, and therefore eventually were engaged less often. This is a challenge to be addressed in the next iteration because of the importance of the strong emerging economy’s perspective in a locally-driven program like this. 45 The challenge is to ensure that the quality of representation is appropriate in the quality of representation – South Africa has academic strength, knowledge of how things work, network, and connections. As noted above, broadening the base of Founding Partners needs to be considered in the context of strengthening links between SWFF innovators and to learn from other donor programs in developing and emerging economies and regions. Possibly connections with other Challenge Funds will provide avenues to add countries or regional organizations to provide additional strength and representativity. FINDING: The founding partners bring complementary agendas to the SWFF partnership, which meets the various needs and capacities of each agency. USAID’s effective management of the GC is highly regarded. RECOMMENDATION: Greater emphasis should be given to broadening a developing and emerging economies membership base of the Founding Partners in a new initiative and linking these partners more effectively to the Program. Such partners could help locate and support suitable innovators particularly from developing and emerging economies. Prospects with regional organizations in Africa and Asia should be explored. SECTION 3: EFFICIENCY As each challenge fund is different in its objectives, reach, approach and administration, a series of proxy indicators have been used to reflect on SWFF/TAF’s relative efficiency. As per the questions in the original SOW for the evaluation, these proxies revolve around the perception of awardees through surveys and interviews of the application process, support received, reporting requirements and other aspects of the program. Information from other global challenge funds is used to inform the analysis by comparing costs of the overall program and specific aspects of management. Question 10. To what extent were the SWFF results to date in balance with the level of effort and resources (funds, human resources including by the FPs, TAF, interns, consultants)? To inform the response to this question, the evaluation team relied on Key Informant Interviews with innovators in particular. In terms of comparative efficiency of the SWFF/TAF, 22 out of 27 KII respondents (81.5%) have a positive view of the administration of SWFF with the vast majority of those respondents (18) perceiving the SWFF/TAF as highly efficient. (See Table 5 titled TAF Comparative Efficiency in Annex.) When asked specifically about value for money, although most had limited understanding of SWFF budget allocations at the macro level, the vast majority (14 out of 21 or 67%) suggested that SWFF’s overarching systems and administration reflected high value for money. One third (7 respondents) were uncertain regarding SWFF’s value for money. Across interviewee categories (innovators, IIAC members and founding partners), while SWFF costs are perceived to be higher than other GC funds as a function of the extensive technical assistance and support provided, the outputs and outcomes across the program as 46 a whole are in balance with the level of effort and resources spent or indeed represent high value for money. When leveraged funding figures are incorporated into the equation, recorded satisfaction from the founding partners in particular is that much stronger. Indeed, a study finds the social rate of return metrics exceed the conservative expectations of impact investment funds as well as comparable foreign aid investments.3 The issue of SWFF costs in relation to customer/end-user numbers is further explored below in Section 5 on Sustainability. Founding partners and IIAC members noted the significant amount of time required of them to stay engaged at a level that allowed them to be confident in the decisions being taken (particularly around whether to allow an innovator to continue in the program). Innovators and founding partners all acknowledge the high level of effort needed by innovators to meet negotiated milestones and maximize their gains as a result of SWFF support over the years of funding. Reporting is demanding, check-ins take time and resources, and field visits require planning and significant staffing capacity. However, those same innovators attribute their success on some level to the support SWFF has provided – the resources, technical assistance, and network available; and the targets and milestones, which have all contributed to their growth as an organization. FINDING: Respondents across the board recognize the higher costs and levels of effort required to administer SWFF, but overwhelmingly suggest that the resultant impacts from the attention to detail and tailored support are in line with or well exceed the costs. Question 10a. To what extent was the TA Facility efficiently set up, organized and managed? Based on KIIs with innovators, the majority of innovators overwhelming appreciate the dedicated support of the TAF. While not overly familiar with the arrangements by which the TAF was originally established and how it operates, respondents noted that the TAF provides the right kind of support, at the right time, from the right people. Frequent contact from a small number of TAF staff (with minimal turnover during the course of the program) has led to an intimate knowledge of the innovators and their businesses. This was reflected numerous times in the personal testimonials during interviews but also during more public discussions at SWFF un-conferences, for example. FINDING: At no point during the evaluation did questions emerge around the efficiency or effective management of the TAF to deliver support to SWFF awardees and respond to requests from the founding partners or IIAC. Indeed, the TAF is widely appreciated as providing exemplary service from a small, dedicated team of professionals. Question 10b. To what extent are the administrative costs for managing SWFF above, below, or on par with the cost of similar challenge funds? (Special consideration should be made for funds that provide technical assistance to their innovators.) 3 Saad Usmani, April 2020. The Economic Benefits of Supporting to Private Social Enterprise at the Nexus of Water and Agriculture: A Social Rate of Return Analysis of the Securing Water for Food Grand Challenge for Development. 47 Global challenge funds have emerged as a mechanism to support a wide range of initiatives from supporting local innovation to promoting civil society voice to developing health care solutions. Even for those focused primarily on growing businesses through incubation and acceleration programs, data on global challenge funds is generally difficult to compare given the disparities regarding objectives, approaches to disbursing funds and providing assistance, target participants for the program in terms of stage of development and location, and the fund’s administrative arrangements. As noted in the IPE Triple Line study entitled Evaluation of Sida’s Global Challenge Funds: Lessons from a Decade Long Journey published in 2018, “This is clearly a complex area of analysis and the Swedish International Development Cooperation Agency (Sida) should develop a challenge fund (CF) activity costing framework to understand the effective costs and benefits of CF engagement and enable comparison across different models of CF.” (p150) It is unclear whether Swedish International Development Cooperation Agency (Sida) or USAID have already created this framework for cross comparison and whether that includes costings accompanied by some comparative review of the impact that the support has on the enterprise and also on the customers / users. To respond to the evaluation question, efforts were made to contact a range of GCF fund managers with the team receiving a handful of responses. Financial information broken down by fund management costs is not readily available on GCF websites or in GCF reports as information of this nature is generally seen as proprietary. Quick interviews with individuals familiar with GCFs suggested that administration and fund management usually ranges from roughly seven to thirty percent of the total cost of the fund. The corresponding (albeit incomplete) results from the data collection exercise are presented in the table below. 48 Table 20. Comparative Information Across Global Challenge Funds Challenge Fund Beneficiaries Awardees FM as % of Total Fund Budget FM as % of Fund Grant Value Leveraged Funds Source(s) AECF REACT SSA 16m (by 2017) 268 companies 30% US$658m IPE Triple Line Sida Report + Website: //aecfafrica.org/about-us/who-we-are (accessed 22 Jan 2020) (Awaiting further info) Amplify Change (2014) N/A 757 N/A N/A N/A Amplify Change website (Accessed 14 April 2020) Demo Environment 22.3 28.7 IPE Triple Line Sida Report Global Innovation Fund (GIF) 87m 38 24.2 31.9 US$69m concurrent US$165m follow-on GIF Annual Review (Dated May 2019) Innovations Against Poverty (IAP1) 19k new jobs + 567k access to goods & services 25 N/A N/A US$12m+ IAP Website: Accessed 14 April 2020 Making All Voices Count (MAVC) (2012-17) 3.15m 178 57.1 133.0 US$2.5m+ MAVC Final Evaluation and correspondence with IMC Worldwide Powering Ag (PAEGC) 234k 24 15.1 17.8 US$68.5 PAEGC Financial Information (FY19) and correspondence with USAID Securing Water for Food (SWFF) 7.25m 24 (active & graduate) 26.2 40.9 US$25m+ SWFF Correspondence and SWFF Results Database Sustainability & Resilience (2016) N/A N/A 4.9 5.2 N/A IPE Triple Line Evaluation of Sida's GCF. As from S&R Activity Report 2017 - No updated figures could be determined. 49 The table above provides information on nine challenge funds that were recently cross referenced in a Swedish International Development Cooperation Agency (Sida) IPE Triple Line study. This has now been updated by the evaluation team with new data which includes the number of beneficiaries, number of awardees, fund management as a function of the total fund amount, fund management as a function of fund grant value, and the amount of funding leveraged for and by awardees. (The total value of donor commitments, the total value of support to awardees, and the allocation for monitoring and evaluation and quality assurance is incorporated in tables within the Impact tables Annex.) As noted above, while every effort has been made to ensure that the data is accurate, further work is needed in several instances to validate the data taken from websites and informal email exchanges with the grant manager or the funders. The table shows a significant spread in administration / fund management as a percentage of total budget for nine GCFs that are in the Swedish International Development Cooperation Agency (Sida) and USAID portfolios. These range from 4.9% (Sustainability and Resilience) as a low to 57.1% (MAVC) as a maximum. Of those listed and third from the highest, SWFF sits in the middle with 26.2% of the total funding going to fund management. it is not always clear whether donor costs are factored in (in terms of time and expenses). As a percentage of total grant values, SWFF fund management cost is second from the highest at 40.9%. The figures show a more intensive work program and involvement with innovators for fund managers like SWFF/TAF. This aligns with the findings on question 10c below on the relative burden (and level of close interaction) that awardees find when working with SWFF. Some sense of scale is also introduced based on the number of beneficiaries, the number of awardees, and the amount of funding leveraged for and by the grant recipients. Each of these indicators comes with nuances and differences in how the figures are tabulated. Apart from MAVC, at US$34.3m, SWFF’s fund management costs are higher than all the others in the list. With over 7.25 million farmers positively impacted over the life of the program, the customer/end-user numbers are significantly higher than most as well though. Comparing such figures could be misleading, however, for some challenge funds may work through economies of scale – providing similar support to a wide range of groups, while SWFF’s approach is to offer quite tailored assistance. Thus, higher costings suggest a more hands-on approach to working with (potentially fewer) recipients and lower costs denote a less intensive modality of grant giving. Of note, the higher fund management costs as a percentage of the overall budget do not necessarily correlate to high or low overall fund sizes. For instance, Making All Voices Count is an outlier as it works with “innovative approaches to fostering accountable, responsive governance.” (MAVC Website, accessed 20 April 2020) Significant investment was made in forging trusting relationships through a stream of facilitated activities. Thus, like with SWFF, the funding could be considered to be more of a fund manager program of work with awardees as opposed to primarily administering grants to awardees. For SWFF, working intensively with a smaller number of awardees to enhance their viability alongside a focus on socially minded objectives has yielded benefits. Thus, the mixed approach of offering grant money and technical assistance has made sense. 50 In highly intensive challenge funds like SWFF, management may proactively do the following: • Provide (ongoing) hands-on technical assistance (business modeling, tailored networking opportunities, technical support to advance the product / service, etc.). • Provide a matching grant for enterprise development or a fully-funded enterprise development program. • Take an equity share, incorporate a revolving fund, or deploy another mechanism that would likely require more involvement of the fund manager. • Provide technical assistance from in the region or internationally, and whether some of the assistance from externals is provided pro bono. • Actively promote collaboration with local organizations. • Actively promote learning from both success and failure (and degree of patience with funding). • Use a milestone- or outputs-based funding arrangement (with significant time and effort required to manage the arrangement from both the funder and recipient sides). • Require a holistic approach to and potential study of other development objectives (like gender, poverty, etc.), which would require further inputs to frame, analyze and monitor. Additional factors that could influence the level of spend or the fund management figures include: • The nature of the sector and how investible it is, the kinds of returns that can be expected over what period, and ultimately whether the instrument gets involved in capital expenditures for expansion and/or operating expenses, or sticks to providing organizational support (development of business and operational plans, staffing structures, funding models for external finance, etc.). • The stage of the enterprise recipients (“distance from commercialization”) in terms of business development in relation to their absorption capacity for larger grants but also in terms of the higher administrative support that is required at earlier stages. • Whether cross-program activities are included in the figures like conferences and convenings, cross-innovator visits, etc. Advantages of the SWFF arrangement that justify (higher levels of) fund management costs (in the 40-50 percent range of total budget as per the table above) largely revolve around the ability to shape responses to the fund’s objectives as well as manage risk or respond to the risk appetite of the funders. Such advantages include: • Heavy emphasis on high quality launch activities to attract good quality applications. • Robust vetting and selection processes (with resources allocated to also provide feedback to bidders who are unsuccessful in final stages of the application process). • Clarity on the recipient enterprises’ current position and likelihood of success with sufficient time and capacity to negotiate targets and milestones. (This generally allows for building solid relationships between the management and awardees with a high level of trust and transparency.) • Ability to report to funders with a high degree of accuracy in terms of how funds are being spent, their impact on the enterprises, and their impact on the market and development objective more broadly. 51 • Familiarity with the enterprises allows for an ability to pinpoint with some degree of accuracy and a high degree of concurrence with awardees the needs of the enterprise to strengthen organizational structures and sales. • Sufficient familiarity to make informed decisions as to when to advance an enterprise to next stages and when to remove an enterprise from the process. • Sufficient familiarity to introduce enterprises to appropriate external funders, media outlets, and other interested parties and potential partners. This also includes tailoring participation in communities of peers (other grantees) to share ideas, advice and learning (as noted in the IPE Triple Line study, p55). • [As noted in the Triple Line study, p50] Sufficiently hands-on processes to “review and refine the design criteria from one round to the next, supporting a process of continuous adaptation and improvement” to the fund, but also potentially to informing wider development practices of the funder. Disadvantages of the SWFF approach include: • The SWFF model is very labor intensive for the donor(s). • The SWFF model may not match the capacity of smaller organizations that may have helpful technologies but are not at a stage to absorb SWFF’s intensive approach. • Application, M&E, visit and general reporting burden may be high for some recipients (as noted in the subsection below). • High expectations on the part of recipients for high-quality tailored support. • High expectations on the part of funders for high quality data. • High costs (including staff and management time) associated with building up a M&E capacity and data collection that are largely geared towards the needs of a particular funder. Of note, the SWFF final evaluation TOR did not include a specific component of examining the value for money or efficiency elements of SWFF/TAF from a financial point of view. The TAF was referred to by one innovator as “lean and mean” and the evaluation team notes the vigilance of the COR, who is highly intentional about the money spent under the SWFF program. That said, overall, the evaluation team has not had reason to question the administration or fund management costs given the advantages of the approach listed above and the numerous positive impacts noted throughout this report. In many open-ended KII no interviewee mentioned any waste for superfluous activity in SWFF/TAF. Indeed, IPE Triple Line concludes that “in general the more intensively managed funds, with a more hands-on approach, had a greater degree of success in ensuring sustainable development outcomes.” (p49) It is understood that IPE Triple Line has been commissioned by USAID to conduct a similar comparative study to that completed for Swedish International Development Cooperation Agency (Sida). Further analysis on cost and impact comparisons would likely result from this upcoming work. FINDING: SWFF is found to have a highly intensive mode of management. Its administration and fund management costs are generally higher as a function of the total value of support to 52 awardees than other challenge funds, but such comparisons need to be considered with care. The significant level of support provided to awardees and the impacts need to be considered. Support activities are well regarded by innovators as providing valuable assistance. RECOMMENDATION: Swedish International Development Cooperation Agency (Sida), USAID and other funders should agree on a framework for cross comparison to enable a more rigorous assessment of the efficiency and effectiveness of different models of intervention. A comparative review should include the impact that the support has on the enterprise and also on the customers / users. Question 10c. To what extent is the level of effort and resources spent by applicants/innovators in balance with the added value SWFF brings? Anecdotal responses during KIIs with innovators suggested that SWFF/TAF has been more involved in their operations and their organizational development than other funders. For the most part, SWFF awardees are overwhelmingly positive about receiving the financial award and the tailored support from the challenge fund. Thus, the general consensus during the interviews was that, although anecdotally higher than with other funding sources, the support from SWFF has been very much worth the effort. To unpack this further, the evaluation team sought to distil the perceptions around different aspects of participating in the SWFF program on a comparative basis with other funding sources that the innovators have accessed. The table below captures the responses from the online survey. Table 21. SWFF Comparative Administrative Burden Aspect (Respondents) Significantly more Somewha t more About the same Somewhat less Significantly less Application Process (21) 38% (8) 29% (6) 29% (6) 5% (1) 0% (0) Financial Reporting (21) 29% (6) 29% (6) 43% (9) 0% (0) 0% (0) Reporting / Monitoring (20) 19% (4) 52% (11) 19% (4) 5% (1) 0% (0) Agreeing TA / Working with TA providers (21) 14% (3) 28% (8) 38% (8) 10% (2) 0% (0) Hosting visits (21) 52% (11) 25% (5) 19% (4) 5% (1) 0% (0) Source: Online Impact Survey 2019, Level of effort to achieve support The table above highlights that in each category awardees perceive SWFF to be significantly or somewhat more taxing than other funds with which they have had experience. (No respondents saw SWFF as significantly less burdensome in any category.) Again, juxtaposing this against satisfaction, impact and value for money suggests that the added “burden” of working with SWFF is not necessarily problematic. In fact, in many instances, it likely makes the majority of innovators’ engagement with SWFF more productive and potentially enjoyable. 53 From the perspective of the founding partners, these aspects of rigor are what make SWFF a solid investment. If the application process is generally more challenging (as noted by 67 percent of respondents), it could mean that accepted applicants are of a higher quality, more suited to deliver on SWFF’s aims, more aware of what will ultimately be expected of them, and thereby more likely to succeed. Similarly, for financial reporting, 58 percent of respondents noted that SWFF processes are significantly or somewhat more demanding than other funds. This along with the reporting and monitoring demands (at 71 percent suggesting these were significantly or somewhat more than other funds) are seen by SWFF not just as providing information for the funders but also in getting awardees to understand different aspects of their business. Alongside providing critical information for the founding partners, such reporting practices will ultimately help the enterprises when seeking both public and private investors. Altogether 42 percent of respondents suggested that agreeing on technical assistance (TA) and working with TA providers was significantly or somewhat more burdensome than the processes with other funds. This, alongside the significant percentages of innovators who note satisfaction in the TA support received (based on surveys conducted by TAF and anecdotally confirmed by the evaluation team) suggests that technical assistance has been helpfully tailored to meet the needs of the vast majority of awardees. Similarly, with the higher percentages of respondents (77%) noting a significant or somewhat higher burden of SWFF in terms of hosting visits compared to other grant makers, this needs to be juxtaposed against the significant number of innovators who are grateful for the interest that SWFF has taken in and deep familiarity with their business as well as the helpful feedback received during and after the visits. From the initial round of interviews conducted for the evaluation in 2018, a handful of innovators - some stronger organizations and those that are from the “North” - did not see the value of SWFF’s hands-on approach as strongly as those in the South and those still solidifying their innovation and approach. During interviews and off the record, numerous innovators complained about but attributed much of the success in scaling and expansion to the negotiated milestones. As one innovator notes (but many agree), SWFF has a “much higher level of reporting and admin compared to other programs, but a very high rate of return on the investment based on the number of farmers reached and seed produced.” FINDING: While the burden SWFF places on awardees appears to be higher than most other grand challenge programs, these same processes allow for SWFF to tailor its funding and technical support more directly to achieving the aims and objectives of the awardees as well as those of the founding partners. RECOMMENDATIONS: Comparing incubation and accelerator support for emerging businesses with challenge funds that support civil society or public sector efforts is misleading and unhelpful. If not already in process at the inception of WE4F, efforts should be made to undertake further comparative cost and impact analysis of challenge funds. The Founding Partners should, however, establish a methodology focused on consistent definition of terms, leading to comparisons of like-with-like. Further analysis and a webinar with Tripleline, the Evaluation Team and the Founding Partners would be a useful step towards identifying a unified approach and methodology. 54 SECTION 4 IMPACT The following section provides an overview and the details of SWFF’s impact at both the innovator and program level. The findings are drawn from evidence largely on the first site Visits (2018), site Revisits and Field Evaluators (FE) surveys (2019), and key informant interviews (KIIs) with innovators and local partners in nine countries. Additional information comes in the form of KIIs with innovators, IIAC, and Founding Partners at Stockholm World Water Week (WWW); focus group discussions with beneficiaries; and secondary M&E data. Question 3. Were solutions sourced through SWFF adopted and utilized at scale? The table (Stage of Development: Plans for Scaling Up) below, presents data on innovations plans for scaling up. Table 22. Stage of Development: Plans for Scaling Up, 2019 Innovations: plans for scaling up Total, N=21 Percentage No plan to scale up in place 0 0% Initial plan in place but evidence largely anecdotal 5 24% Early plan in place with quantitative evidence collected 9 43% Matured plan in place for scale 4 19% Replication and expansion in place with clear evidence 3 14% Source: Data from surveys & Key Informant Interviews Of those interviewed at the WWW or in country during field visits, most (86%; 18 out of 21) of the SWFF innovators in 2018 had at least a plan to scale in place, at an early or matured phase, and are meeting or exceeding some targets (except for one innovation). These categories in plans for scaling up are comparable but differ slightly to those of the TAF Innovation Stages in Portfolio Review and add value from the perspective of the innovator. Altogether 77% (16 out of 21) of innovator interviewees are meeting or exceeding at least some of their targets in scaling up. For some, the evidence to scale is largely anecdotal (N= 5, 24%), while for others there is a replication and expansion plan in place with clear evidence to scale (N= 3, 14%). Innovations entered into the SWFF program at various levels of maturity: for some, SWFF was the way to take an idea or a pilot to fruition and have used SWFF for everything from technical assistance to organizational capacity building, while for others SWFF presented stop-gap funding for an innovation both tested and already effective, inside of an organization and well-established in some spheres. Table 23 (Stage of Development: Status, Meeting Targets) below, presents data on innovations status in meeting scaling up targets. 55 Table 23. Stage of Development: Status, Meeting Targets, 2018 Innovations, status on targets Total N=21 Percentage Not meeting targets 1 5% Early stage, i.e. pre-scaling up stage 4 19% Meeting some targets in scaling up 8 38% Meeting all targets in scaling up 6 29% Exceeding scaling targets 2 10% Source: Targets Met, (KII data from surveys & KII interviews) As shown in the table above in addition to planning there are responses to the question of scaling up, 39% (8 out of 21) of innovations are meeting (29%) or exceeding their targets (10%) in scaling, while 57% (12 out of 21) that are at an early stage are meeting some targets. Only 5% (N=1) of innovations are not meeting any targets in scale. Data on the strengthening or development of partnerships by SWFF-funded innovations is presented in Table 24 (Status of Innovator Partnerships). Table 24. Status of Innovator Partnerships N=18 Innovator was able to establish new partnerships as a result of SWFF support 14 (78%) Innovator was able to strengthen existing partnerships as a result of SWFF support 2 (11%) SWFF did not impact on innovator partnerships 2 (11%) Source: KIIs: site visits and WWW Scaling up is closely linked to innovations developing partnerships to add capacity to make qualitative change. In working to meet demanding targets and milestones, the innovators are finding it critically important to establish new partnerships or to strengthen existing partnerships. SWFF support has been critical to scaling up; it has helped to established new and strengthen existing partnerships for 89% (16 out of 18) of innovations, which has spurred the way for expansion and growth in the districts in which the innovation currently works as well as into new regions and even countries. FINDING: In interviews conducted during 2018-19 most innovations reported a plan to scale up (growth and expansion in new markets) and were able to strengthen existing partnerships and establish new partnerships with SWFF support. Subsequent developments are to be found in the Section on Effectiveness and Sustainability. 56 RECOMMENDATION: WE4F should define realistic target goals for scaling up and develop a clear plan for strengthening existing partnerships or establishing new partnerships. Regular assessment of targets and plan should be adapted to the socio-economic environment in which innovations operate. Question 4. To what extent has SWFF contributed to the outcomes and results outlined by the indicators in the SWFF PMEP during project implementation and post project implementation? Attribution in the clearest sense of the word is not possible in a performance evaluation of this nature and to meet this challenge the methodology relies on Outcomes Harvesting methods including triangulation between different types and sources of information. Without clear baselines on a well-defined set of indicators that carefully track SWFF’s support and hold constant support (financial and technical) from all other donors, as well as the absence of a robust counterfactual, the evaluation team has focused its attention to SWFF’s contribution – both from the perception of the evaluation team and directly the innovators – to the results and outcomes achieved by each innovation. To mitigate issues with attribution, in addition to interviews with innovators (as well as other key stakeholders) to help ascertain the level of SWFF contribution to outcomes there have been a series of surveys of End-Users. These surveys have used a set of clearly-defined impact indicators including: improvements in income, level of perceived benefit, level of satisfaction, improvements in access to water and water efficiency, and contribution towards improvements in agricultural yields. Subjectivity in perceived benefit has been triangulated with data from, for instance, the SWFF Final follow up survey. Table 25 (Innovation Programmatic Emphasis) on SWFF’s Innovations’ Programmatic Emphasis breaks down the emphasis of innovations on the key SWFF objectives of productivity increases, pro-poor focus, and water efficiency (innovations can have more than one focus across the three metrics, so the Ns will not add to 25). As shown, nearly all innovations that formed part of the KIIs had a clear primary focus on agricultural productivity increase (86%) and clear water efficiency gains (76%). Considered from another perspective, 64% have a clear primary pro-Poor focus. Table 25. Innovation Programmatic Emphasis SWFF Programmatic Emphasis, N=25 # of Innovations Clear agricultural productivity increases 80% (20) Clear pro-Poor focus 64% (16) Clear water efficiency (and access) gains 76% (19) Source: KIIs: site visits and WWW. Note: More than one focus is possible per innovation There is more than one emphasis possible per innovation and percentages indicate the proportion of the total 55 emphases mentioned. 57 In Table 26 (Resilience to climate change) below, 82% of the innovators indicate that their contributions in these areas of emphasis were increasing the resilience to climate change. Table 26. Resilience to climate change Assessment N=22 Innovation clearly helps increase resilience to climate change 18 82% Innovation makes negligible contribution to resilience to climate change 1 5% Innovation somewhat or peripherally increases resilience to climate change 3 14% Source: KIIs: site visits and WWW. Tables below outline responses to perceived benefits, improvements in income, and level of overall satisfaction with the innovation across 16 innovations visited in person either by the evaluation team or by the SWFF Field Evaluators. This data comes primarily from one-on-one survey data collected during site visits or Revisits or after focus group discussions. Table 27 (Customer/End-User Impact Responses; Benefit from innovation) presents the interviewees’ perceived benefit from the use of the innovations. Data in the table is ranked (high to low) by levels of benefit and by innovation as found in the Revisit/FE surveys. 58 Table 27. Customer/End-User Impact Responses, Benefit from innovation % Benefit from innovation YES (N=721) # Innovation Visit (N) Revisit / FE (N) 1 Aybar N/A 100% (50) 2 CUT Swayimane 100% (23) 100% (31) 3 MNP (Ecorangers) 74% (20) 100% (65) 4 Naireeta Bhungroo Too early 100% (25) 5 WASTE 86% (19) 96% (24) 6 AST N/A 94% (47) 7 aQysta N/A 88% (46) 8 WGI (Aquaponics) 80% (17) 87% (26) 9 Hydroponics Africa 100% (13) 85% (29) 10 CUT ITIKI 100% (37) 84% (26) 11 ICBA 100% (25) 84% (16) 12 Newsil 100% (10) 81% (17) 13 Green Heat 100% (8) 80% (28) 14 Skyfox 100% (10) 66% (21) 15 Practical Action 76% (32) 63% (19) 16 Lal Teer 100% (20) 55% (17) OVERALL 93% (234) 85% (487) N/A= Not Available Source: Visit, Revisit and SWFF Field Evaluator Surveys Note: The evaluation team has canvassed local consultants for the differences, over time, for the proportion of end-users reporting benefit from Skyfox and Lal Teer. In both cases high proportions of end-users report benefit; intentionally different end-users were sampled in each survey. In the table above, there are high levels of benefit reported. Taking the column of Revisit and FE surveys, 85% (N=487) of all beneficiaries surveyed in country perceived a strong tangible benefit or improvement to their quality of life since using the innovation. Although that percentage is slightly lower than the one gathered during the First Visits (93%; N=234), there is a strong indication that Customers and End-Users of the different innovations highly benefited from their use and that their overall quality of life was substantially improved. 4 out of 16 Revisit/FE innovations (25%, N= 171) have a 100% customer/end-user benefit level, while 13 out of 16 innovations (81%, N=430) of Revisit/FE innovations have at least an 80% customer/end-user benefit level. The management of cash benefits from an innovation is illustrated in the case of Practical Action; cash from sales has been used to invest in a variety of assets. The end-users in Cox’s Bazar were able to sell their produce to the 1.2m Rohingya refugees and local communities. The project had managed to establish a “symbiosis process of benefits and managing demand and supply” to 59 achieve good prices at the farm-gate. This then led to a broadened asset base for End-Users as some 60% immediately invested in livestock to achieve higher income just after the harvest. Their End-Users had a ready market in Eid-Ul Adha for bulls. Other households (some 15-20%) were able to buy land, start enterprises or host such as celebrations.4 In the case of Naireeta Services’ Bhungroos, benefits can be seen at several levels. Customers/End-Users have more water available, especially in the dry season (as reported by 23 out of 51 farmers (45%)5 ) for their agricultural activities, for cattle farming, for domestic purposes such as cleaning, bathing, and drinking.This availability of water allows for an increase of agriculture productivity, an improvement of food security and an enhancement of the economic status of farming households. Indeed, 11 of the 51 farmers (24%) stated that with access to this water storage system, they can now rely on the availability of water and in turn, rely on agricultural activities as a source of livelihood; thus improving their income and the agricultural sustainability of the region. Farmers are able to increase/diversify the number of crops they grow (94%), to get an extra season of cultivation (8 out of 51 farmers, 16%), to improve their crop yields (5 out of 51 farmers, 9%), and specifically with less usage of water (14 out of 51 farmers, 27%), to observe an increased resilience of crops (17 out of 51 farmers, 33%), and to generate income from non-farming activities6 . While other farmers have to pay hefty prices for access to water or to rent a pump, members of the Naireeta innovation get water free of charge and do not need to pay to rent a pump. They are also able to have one or more growing seasons and do not have any longer to rely exclusively on rainfall for their activities; they are able to grow an increased number of crops as well as grow larger amount of the most economically valuable crops. Table 28 (Customer/End-User Impact Responses (women and poor)) below presents the interviewees’ perceived benefit from the use of the innovations, especially for women and the poor. Data in the table is ranked from highest to lowest based on the percentage of women who benefited from the innovation. Table 28. Customer/End-User Impact Responses (Women and Poor) % Benefit from innovation YES (N= 401) # Innovation Name % Women benefited (n) % Poor benefited (n) 1 AST 100% (6) 89% (42) 2 Aybar 100% (5) N/A 4 Information available from the innovation and made available from SWFF TAF, Final Follow-up Survey. 5 SWFF Field Evaluator Report, November 2019 6 SWFF Field Evaluator Report, November 2019 60 % Benefit from innovation YES (N= 401) # Innovation Name % Women benefited (n) % Poor benefited (n) 3 WGI 100% (13) 96% (29) 4 MNP 100% (16) 100% (43) 5 Greenheat 96% (27) 93% (28) 6 Aqysta 88% (7) N/A 7 WASTE 73% (17) 81% (31) 8 Hydroponics Africa 62% (23) 67% (19) 9 ITIKI 51% (23) 50% (19) 10 Skyfox 45% (11) 45% (19) 11 Naireeta Bhungroo 22% (11) 24% (12) OVERALL 76% (159) 72% (242) Source: SWFF Field Evaluator Surveys When looking at disaggregated data relative to the benefit from the innovations through the lens of vulnerability, it can be noticed that women and vulnerable populations (poor and very poor) benefited. Overall, 76% (N=159) of all women and 72% (N=242) of all Poor and very poor benefited; this is roughly equivalent to the average of 85%. Independently of the survey, Aybar reports that 95% of its End-Users were poor or very poor.7 The highest percentages of women benefitting from the innovations are found in AST, Aybar, WGI and MNP (100%) while the lowest numbers appear for Skyfox (45%) and Naireeta Bhungroo (21%). Whether it is for all the respondents or specifically for Women and the Poor, the principal benefit cited was “the benefit from the innovations for the most important crop”. Table 29 (Impact on Customers/End-Users, Income) presents the findings related to the impact of the innovations on respondents’ income improvement. Data in the table is ranked from highest to lowest based on Revisit/FE surveys data. 7 Data from reports provided by SWFF TAF, Final Follow-up Survey. 61 Table 29. Impact on Customers/End-Users, Income % Positive change in income YES (N= 840) # Innovation Visit (N) Revisit / FE (N) 1 Green Heat 100% (8) 100% (35) 2 CUT ITIKI 100% (4) 100% (31) 3 Newsil 90% (9) 100% (21) 4 ICBA 100% (25) 100% (19) 5 Lal Teer 100% (20) 97% (30) 6 Hydroponics Africa 100% (6) 94% (32) 7 Skyfox 100% (10) 94% (32) 8 Naireeta Bhungroo Too early 92% (23) 9 CUT Swayimane 65% (15) 88% (27) 10 WASTE 46% (10) 88% (23) 11 Aybar N/A 86% (86) 12 MNP (Ecorangers) 93% (25) 81% (106) 13 Practical Action 76% (32) 70% (21) 14 AST N/A 68% (68) 15 AQysta N/A 56% (58) 16 WGI (Aquaponics) 83% (15) 42% (49) OVERALL 88% (179) 85% (661) Source: Visit, Revisit, and SWFF Field Revisit Surveys Even though data from the different surveys might yield different results for each innovation, there seem to be a consistency in the overall positive impact of the innovations on the respondents’ income. For more than 80% of innovations (13 out of 16, N= 486), at least 70% of respondents seem to have seen an increase in their income since using the innovation. There are two particular variances which can be addressed. Firstly, WASTE end-users report a considerable improvement in income (46-88%) which could be explained by the improved functioning of water storage and availability of co-compost increasing incomes. Secondly, there is the apparent decline of income over time in WGI (83-42%); while reasons have been pursued these have not been conclusive. These variances need to be more fully explored; the data arises directly from the responses of users captured in surveys and are only fully apparent after analysis. Small variances may relate to changes in sampling procedures, sample size and changes in the phrasing of questions, more substantial variances invite aggregation, clarification of the questions and further research. 62 Green Heat and CUT ITIKI consistently report a positive impact on respondents’ income, yielding the same 100% result for each survey. Ignitia, not included in these surveys, has independently reported that incomes of End-Users are very low “due to shaky sales”, but an average of $480 had been earned over a season and farmer incomes had risen from $1.9 to $3.1 per day.8 In terms of overall satisfaction with the innovations, Revisit and FE surveys (which were undertaken at the same time) did not provide data to access change over time. Table 30 (Impact on Customers/End-Users, Income (Women and Poor)) presents the findings related to the impact of the innovations on women and poor respondents’ income improvement. Since this level of disaggregation was only undertaken with Field Evaluator surveys there is a smaller number of innovations represented. The data is ranked from highest to lowest based on percentages of improvement in income data for women. Table 30. Impact on Customers/End-Users, Income (Women and Poor) % Positive change in income YES (N= 250) # Innovation Name %Women Significantly Improved Income (n) %Poor Significantly Improved Income (n) 1 Greenheat 89% (25) 80% (24) 2 Skyfox 83% (20) 88% (37) 3 MNP 56% (9) 79% (34) 4 AST 50% (3) 53% (24) 5 Aybar 40% (2) N/A 6 Aqysta 38% (3) N/A 7 WASTE 34% (8) 34% (13) 8 WGI 30% (4) 10% (3) 9 Naireeta Bhungroo 24% (12) 24% (12) 10 ITIKI 17% (8) 13% (5) 11 Hydroponics Africa 2% (1) 10% (3) OVERALL 42% (95) 43% (155) Source: SWFF Field Revisit Surveys 8 Data made available to SWFF TAF, Final Follow-up Survey. 63 While the overall improvement in income for women and poor are similar (42% for women and 43% for the poor), they are considerably lower than the average perceived improvement in income for all respondents (85%, N= 840). The highest improvements in income for women are seen for Green Heat (89%) and Skyfox (83%) and the lowest appear for ITIKI (17%) and Hydroponics Africa (2%). In addition, improved productivity has been reported to also increase the households’ food security. The highest percentages of income change for women are also found in the reported income change for poor. Greatest income improvement could be seen for the poor with Skyfox (88%), Green Heat (80%), and MNP (79%). In other records, MNP further reported that although incomes of End-Users had increased, the primary advance was not in income but rather to livestock which are managed as a form of insurance against economic stress. Livestock was healthier and households more resilient to climate change.9 The considerable variance between the average perceived improvement in income for all respondents (85%) and women and poor needs further examination. The following innovations carry responses well below this average: AST, Aybar, aQysta, WGI, CUT ITIKI, WASTE, Naireeta Bhungroo, Hydroponics Africa. There appear to be two major factors in explanation: firstly a number of the innovations (including WASTE and Bhungroo) were not fully operational at the time of surveys and secondly there appear to be aspects of participation in the innovation (such as land ownership, lower both for women and poor) which could be hampering full participation. The evidence of inequality in such a primary benefit needs further investigation. The presentation and analysis of the effectiveness in relation to the activities of SWFF innovations is now completed and we now turn to the reported effectiveness of the SWFF TAF and its leadership. The extent and types of SWFF’s contribution to outcomes and results outlined by the indicators in the SWFF PMEP is presented in Table 31 (SWFF contribution, extent, and type) below. 9 Report from MNP to SWFF TAF made available to the evaluation team, Final Follow-up Survey. 64 Table 31. SWFF contribution, extent, and type Extent of SWFF Contribution N=28 Decisive contribution at certain points 29% (8) Decisive contribution throughout 50% (14) Tangential contribution from SWFF to outcomes and results 21% (6) Type of SWFF contribution N=33 (21 innovators made multiple responses) SWFF contribution to organizational strengthening is clearly linked to outcomes and results 24% (8) SWFF financial contribution is clearly linked to outcomes and results 64% (21) SWFF contribution to technical aspects is clearly linked to outcomes and results 12% (4) Source: KIIs – site visits and WWW In assessing the extent of SWFF’s contribution from key informant interviews in the table above, 100% of innovator respondents (N=28) stated that SWFF contributed to the outcomes and results achieved writ large as they reported to the SWFF PMEP. 50% (14) identified the support by SWFF as being decisive contribution throughout, while 29% (N=8) identified the contribution as decisive at certain points. 21% (N=6) felt the SWFF contribution to outcomes was tangential. In relation to the type of SWFF contribution, the greatest contribution was seen in financial contribution (64%, N=21) while 24% (N=8) highlighted organizational strengthening and 12% (N=4) highlighted technical aspects. FINDING a: In relation to SWFF’s contribution to outcomes; there is a strong indication that SWFF made a decisive contribution throughout the innovation’s operational life to the outcomes and results in terms of (in this order); financial, organizational strengthening, and technical contributions. The financial contribution of SWFF was reported as playing the greatest role (64%, n= 21) in reaching outcomes and results, followed by the contribution to organizational strengthening and to technical aspects. FINDING: Customers and End-Users highly benefited from the use of the different innovations and their overall quality of life was substantially improved. There is an overall positive impact of the innovations on the respondents’ income. However, benefit from the innovations and improvement of income from the use of innovations is lower for women and poor than the average for all respondents. This is particularly true in the case of improved income which is substantially lower for women and poor. RECOMMENDATION a: Establish clear baselines on a well-defined set of indicators that will carefully track SWFF’s or other programs’ support and hold constant support (financial and technical) from all other donors. Set up a robust counterfactual along a set of clearly defined 65 impact indicators to validate the contribution of WE4F and other programs to outcomes and results of funded innovations. RECOMMENDATION b: Further investigation is needed into uneven benefits from innovations for women and poor. Gender Integration should be at the forefront of innovations and programs to help reduce the systemic inequality that exists in communities. The greater inclusion of women will assist in overcoming the limitation in the opportunities of one-half of its population. In addition, attention needs to be given to poor End-Users to ensure greater benefit from innovations and improved income. Question 4a. Did SWFF-supported projects increase water efficiency/make water more accessible? Did SWFF projects meet their water efficiency/availability targets? Overall, across all innovators, did the program meet the water efficiency/availability targets? Based on key informant interviews with innovators, surveys and focus group discussions with beneficiaries, as well as data validation of existing performance indicator data, there is strong evidence that SWFF-supported projects contribute to increased water efficiency and make water more accessible to Customers and beneficiaries. The level of contribution and attribution of SWFF varies by innovator based on the extent of other donor funding and technical support as well as the innovator’s perception of the value. The Annex table on Outcomes and Targets in Water Efficiency and Access, is based on KII respondents during first country site visits and WWW; not all innovators are working in water accessibility and efficiency and within that subset not all innovators answered all related questions, thereby bringing down the Ns. Innovators reported significant changes to water accessibility, whether it is verifiable (45%, N= 5) or not (7%, N= 1). In addition, 87% (N=13) of innovators interviewed agreed that targets regarding water savings were realistic against 13% (N= 2) who thought that they were unrealistic. In relation to the key question of efficiency and savings, 60% (N=12) of innovations showed very significant water savings and water efficiency due to SWFF supported innovations; 4 innovations (36%) did not have verifiable data. The tables below (improved access to water, and water efficiency) have been constructed to illustrate key common indicators across the 16 surveys’ respondents10; considerably more data is available on the degree of benefit, of improvement of income, improved access to water, improved water efficiency, and improved agricultural productivity as a result of SWFF innovations. Table 32 (Impact on Customers/End-Users, Improved access to water) presents findings relative to the impact of innovations on Customers or End-Users regarding the improvement in access to water. Data in the table is ranked from high to low by improved access to water based on Revisit and FE surveys. 10 Not all beneficiaries answered each question and surveys varied slightly by innovation to capture nuances in technical focus and innovation type 66 Table 32. Impact on Customers/End-Users, Improved access to water % Improved access to water YES (N= 279) # Innovation Revisit / FE 1 Greenheat 91% (32) 2 Practical Action 90% (27) 3 WGI (Aquaponics) 90% (27) 4 Naireeta Bhungroo 84% (21) 5 CUT Swayimane 77% (24) 6 Skyfox 74% (25) 7 Aybar 60% (51) 8 Hydroponics Africa 53% (18) 9 ICBA 53% (10) 10 Waste 50% (13) 11 AQysta 38% (39) Overall 70% (279) Source: Revisit and SWFF Field Evaluator Surveys (Note: Newsil, MNP, AST, CUT ITIKI and Lal Teer have been removed from the original table because they represent innovations that have low numbers because improved access to water is not the innovations’ main mandate. However, some Customers/End-users still think that some innovations such as Lal Teer and CUT ITIKI did help in their access to water. Although it might not be some innovations' first mandate, they do benefit end-users in more ways than anticipated.) Overall, 70% (N=279) of respondents reported an improvement in their access to water due to SWFF’s innovations. In addition, in 10 out of 11 of the innovations (N= 240) respondents reported an improvement of 50% or more in their access to water. Respondents cited the greatest impact on access to water due to the use of the innovation for Green Heat (91%), Practical Action (90%) and WGI (90%) and the least for WASTE (50%) and aQysta (38%). These lower values in innovations can be explained by the fact that for these innovations, there was not an explicit commitment to increase water access for respondents. Indeed, for Lal Teer for example, there is no commitment to improved access to water, rather to the improved production in high-saline areas of southern Bangladesh from the combined use of locally developed salt-tolerant vegetable seeds, easily adoptable cultivation methods, microfinance assistance, information and communication technology (ICT) support, and extension advisory services. The same goes for CUT ITIKI which is an early warning system that integrates indigenous 67 and scientific drought forecasting using a mobile application, web portal, and SMS service to pool weather information through a network of sensors that monitor weather conditions for small￾scale farmers. However, even though the improvement in the access to water might not be some innovations’ first mandate, they did benefit Customers/End-Users in more ways than anticipated, namely here by increasing their access to water. Data on the improvement of access to water, relative to women and poor, is presented in Table 33 (Impact on Customers/End-Users, Improved access to water, women and poor). Data in the table is presented according to percentage of improvement in access to water for the poor, from highest to lowest. Table 33. Impact on Customers/End-Users, Improved access to water, women and poor % Improved access to water YES (N= 139) # Innovation % Women (n) % Poor (n) 1 Aybar 100% (5) N/A 2 WASTE 69% (16) 73% (28) 3 Aqysta 63% (5) N/A 4 WGI 53% (7) 36% (11) 5 Greenheat 32% (9) 23% (7) 6 Hydroponics Africa 24% (9) 32% (9) 7 Naireeta Bhungroo 24% (12) 24% (12) 8 Skyfox 8% (2) 16% (7) OVERALL 47% (65) 34% (74) Source: SWFF Field Evaluator Surveys (Note: MNP, AST, and CUT ITIKI have been removed from the original table as they do not have improved access to water as an objective.) Improved access to water numbers for women and poor, follow the same trends as the results for all respondents, although these values are lower than the average improved access to water reported above (62%) for all respondents. Overall, only 47% (N=65) of women and 34% (N=74) of poor Customers/End-Users reported an improvement in their access to water from the SWFF innovations. Women had the highest improvement in access to water for Aybar (100%), WASTE (69%), and aQysta (63%) and the lowest for Hydroponics Africa (24%), Naireeta Bhungroo (24%) and Skyfox (8%). These results show that more work needs to be done, especially in relation to 68 women and the poor, to achieve equal improvement in water access for all members of communities. Although it may be disconcerting to find some innovations funded by SWFF did not meet their goal of improving the availability or access to more water for food production, it doesn’t necessarily mean that they didn’t help produce more food using less water, which is another goal of SWFF. In some innovations e.g. MNP for example, there is no commitment to improved access to water, rather to better herding and sale of cattle; in CUT ITIKI the goal of the innovation is to improve information which is, again, not an explicit commitment to improved water access. The same goes for Newsil which is geared towards reducing crop loss in times of water stress and drought through the application of Silicic acid to food crops in an affordable and environmentally friendly way; or for AST whose BioEnsure® is a fungal seed and plant treatment that, when sprayed onto seeds, helps plants to adapt to water-related stress, helps crops grow in sub-optimal conditions and use 50 percent less water. Water efficiency, measured here in the ability of SWFF innovations to help produce more food using less water, has been accessed with interviewees for 12 innovations. The data presented in the table on SWFF Innovation effect on “crop per drop”: less water and better productivity (water efficiency), is only for respondents who answered YES to an increase in the crop per drop, meaning to using less water and achieving an increased crop yield. Data in the table is ranked from highest to lowest based on Revisit data. Revisit surveys were able to provide data for innovations such as Naireeta and WASTE for which data is lacking from the First Visits because either they weren’t available, or it was too early in the innovation’s cycle to effectively access this variable. However, data on water efficiency is still not available for MNP. Table 34. SWFF Innovation effect on “crop per drop”: less water and better productivity (water efficiency) % Crop per drop increased YES (N= 398) # Innovation Visit Revisit 1 WGI (Aquaponics) 100% (10) 90% (27) 2 Hydroponics Africa 100% (13) 85% (29) 3 Waste N/A 85% (22) 4 Skyfox 88% (7) 59% (20) 5 Greenheat 100% (6) 54% (19) 6 CUT Swayimane 61% (14) 45% (14) 7 ICBA 96% (23) 37% (7) 8 Practical Action 100% (91) 37% (11) 9 Lal Teer 100% (20) 26% (8) 69 % Crop per drop increased YES (N= 398) # Innovation Visit Revisit 10 Newsil 100% (10) 19% (4) 11 CUT ITIKI 100% (37) 19% (6) Overall 95% (231) 50% (167) Source: Visit and Revisit Surveys Overall, there is a significant difference between data from First Visits, for which 95% (N=231) responded positively in the effectiveness of the innovations to increase water efficiency, and Revisit surveys (47%, N=168). This is a significant difference and has to be accounted for by the interviewers possibly phrasing their translation of the question into “improved water efficiency” in the case of the First Visit and “significant water efficiency” in the Revisit. The highest crop per drop are seen in both Revisit and FE surveys for WGI (Aquaponics) (100% and 90%) and for Hydroponics Africa (100% and 85%); this could be a reflection on the provision or highly productive use of water. WASTE has high impact as the stored water resources can be drawn on in the dry season and have a marked effect on the harvest; Hydroponics is a form of intensive agriculture. The innovations with lowest levels of crop per drop are Newsil and CUT ITIKI, The explanations offered are that Newsil treatment impacts on the resilience and final product irrespective of the changed usage of water, CUT ITIKI provides information on key points in the agricultural cycle and not on the most efficient usage off water. Data on water efficiency, relative to Women, Men and Poor, is presented in Table 35 (SWFF Innovation effect on “crop per drop”: less water and better productivity (water efficiency, Women, Men and Poor)). Data in the table is presented according to percentage of improvement in water efficiency for women, from highest to lowest. Table 35. SWFF Innovation effect on “crop per drop”: less water and better productivity (water efficiency, Women, Men and Poor) % Crop per drop increased YES (N= 186) # Innovation Name % Women Increased water efficiency (n) % Men Increased water efficiency (n) % Poor Increased water efficiency (n) 1 MNP N/A N/A N/A 2 WASTE 74% (17) 76% (25) 82% (31) 70 % Crop per drop increased YES (N= 186) # Innovation Name % Women Increased water efficiency (n) % Men Increased water efficiency (n) % Poor Increased water efficiency (n) 3 Hydroponics Africa 62% (23) 73% (8) 61% (17) 4 Greenheat 25% (7) 23% (7) 23% (7) 5 WGI 15% (2) 6% (2) 0% (0) 6 Naireeta Bhungroo 14% (7) 0% (0) 14% (7) 7 Skyfox 8% (2) 34% (12) 17% (7) 8 ITIKI 4% (2) 6% (1) 5% (2) OVERALL 29% (60) 31% (55) 29% (71) Source: SWFF Field Evaluator Surveys Responses disaggregated to women, men and poor are as follow: women (29%, n=61), men (31%, n=55) and poor (29%, n=72). These results show even lower percentages than those for all respondents which register an average of 47%. Since there is not a great variance, these differences could be explained by the process of averaging; the evidence is that women and poor share similar benefits to water efficiency as all respondents. WASTE and Hydroponics are also the most water efficient innovations for women (74% and 62%), men (76% and 73%) and poor (82% and 61%). The lowest levels for water efficiency are recorded by respondents for Naireeta Bhungroo, Skyfox and CUT ITIKI. Specifically, this relates to water efficiency and does not imply that Bhungroo does not improve access to water which is recorded. The high levels of water efficiency for WASTE could possibly be explained by the fact that the innovation involves the recycling of household wastewater (kitchen and bathroom) for irrigation use during dry periods, therefore not only considerably cutting down the total amount of water used by Customers/End-Users overall, but also making water available during the yearly dry season. The success of Hydroponics Africa could be the adaptation, based on crop types, water availability, user type, land area, climate, and culture, of the hydroponic method employed by End￾Users. These systems have been tailored to the specificities of End-Users helping farmers produce maximum yields in small areas, without soil, while using 80 percent less water. In the case of Naireeta Services’ Bhungroos, despite all the benefits that have been stated by customers/end-users, some of the impact of Bhungroos (which have suffered from irregular monsoons), especially in regards to water efficiency, are still not entirely registered by farmers. For example, farmers expressed concerns regarding the level of salinity in the water made available by the innovation that does not always allow them to use most of the water delivered by the innovation for their crop farming. However, they understand that it is a water harvesting 71 tool that reduces salinity and recharges groundwater after 2-3 cycles of continuous water storage. Therefore, they are ready to wait a few years to fully utilize the benefits of the innovation. However, at the same time, since customers/end-users could not see instant benefit for agricultural purposes, they are skeptical and apprehensive about incurring the one-time cost needed to use the innovation. FINDING: Most of the SWFF-supported projects increased water access for their Customers / End-Users, some quite decisively. In some innovations (e.g. MNP) this is not an explicit goal. Unlike other outcomes there is a wide range of recorded impact. More consideration needs to be given on how to ensure innovations achieve better results in terms of improvement in water access attainment and/or improvement for communities. Possibly those innovations such as WGI (Aquaponics), Hydroponics Africa, WASTE, Skyfox and Bhungroo (when it is fully functional) which are or could be the most successful in this aspect need greater support. The more efficient use of water is a critical aspect of SWFF’s mission. In terms of water efficiency, across all innovators, decided evidence of success can be seen only for a few. Although detailed analysis has not been undertaken on the planned targets and results for each innovation, it may be concluded that the results in water efficiency are more modest than the expected targets for a number of innovations. RECOMMENDATION: Focused monitoring and evaluation of the water efficiency indicator is needed to deepen understanding of the variance in responses. Special attention also needs to be given to understanding the variance between the average and levels of water accessibility to vulnerable groups such as women and the poor. Given the lower levels recorded overall for water efficiency, a strong recommendation would be to develop novel methods to make quantitative estimates for this important variable for WE4F. These methods would include rigorous measures of water use as farmers generally have a better idea of yields to establish crop yields per unit of water utilized. Innovations funded through WE4F could work in conjunction with customers/end-users to determine how to accurately ascertain measures of crop yields and water used to achieve such yields, being sensitive to measures being expressed in farmers’ traditional units of measures. Question 4b. Did SWFF-supported projects lead to more agricultural productivity and resilience to climate change? Did SWFF projects meet their agricultural productivity targets? Overall, across all innovators, did the program meet the agricultural productivity targets? Assessment of the impact of innovations on agricultural productivity was undertaken by asking respondents questions about the types of crops grown using the innovation, changes in the number and kind of crops grown before and after the start of the use of innovations, the changes in yields from one growing season to another, the number of growing seasons with the use of innovations compared to without, the changes in inputs and resources allocated to agricultural activities with and without the use of innovations, the level of food security for each household and the community since the use of innovations, the changes in the family income, and the 72 different ways in which the innovations have helped in the practice of the farmers’ agricultural activities. The products of innovations have been used on a wide range of crops, whether mainly cash crops (reportedly mostly grown by men) or subsistence crops (reportedly mostly grown by women), or both in some cases. They are also used to grow crops that are deemed staple11 for the various countries in which the innovations are present. Innovations are also used by farmers at different stages of farming activities, such as soil preparation, sowing, weeding, irrigation, fertilizer application, and harvesting. Overall, the innovations have shown to have an impact on the increase of the agricultural productivity by influencing different factors, depending on the innovation. Table 36 (Reported Improvement in Agricultural Productivity) presents the data relative to the increase in agricultural productivity (in terms of crop yield). The data is ranked from highest to lowest based on Revisit/FE surveys data. In the case of MNP for example, that will include increased number of cattle. Table 36. Reported Improvement in Agricultural Productivity % Increased crop yield YES (N= 697) # Innovation Visit Revisit / FE 1 Aybar N/A 100% (50) 2 Lal Teer 100% (20) 100% (31) 3 Newsil 100% (10) 100% (21) 4 ICBA 67% (16) 100% (19) 5 CUT ITIKI 75% (28) 97% (30) 6 WGI (Aquaponics) 100% (10) 97% (29) 7 WASTE 91% (10) too early: 7 96% (25) 8 Naireeta Bhungroo Too early 92% (23) 9 Green Heat 100% (6) 89% (31) 10 AST N/A 82% (41) 11 Hydroponics Africa 100% (6); too early: 7 79% (27) 12 aQysta N/A 76% (41) 13 Skyfox 90% (9) 71% (24) 14 Practical Action 82% (75) 66% (20) 15 CUT Swayimane 17% (4) 58% (18) 16 MNP (Ecorangers) 78% (21) 52% (34) OVERALL 81% (215) 85% (482) Source: Visit, Revisit and SWFF Field Evaluator Surveys 11 Subsistence crops are crops grown for consumption; staple crops are those deemed the most dominant part of a population’s diet 73 Overall, respondents reported an average increase in their crop yield due to the use of SWFF innovations. In the Revisit column, an average of 85% (N=482) of End-Users observed an improvement of their agricultural production, specifically related to the quantity and quality of their harvest, due to the use of the diverse innovations supported by SWFF. Given the comparable result obtained from the First Visits surveys (81%, N= 215), it can be concluded that an increase in respondents’ agricultural productivity is a definite byproduct of the use of SWFF innovations. Aybar reports that for the main crop yields per hectare gained on average some 20% (3200 kg/ha to 4600kg/ha) with lower water consumption.12 The results do not relate to crops alone, in the case of MNP for example, that will include increased number and value of cattle. Table 37 (Reported Improvement in Agricultural Productivity (women and poor)) below presents the data relative to the increase in agricultural productivity (in terms of crop yield). Data in the table is presented from highest to lowest values for women. Table 37. Reported Improvement in Agricultural Productivity (Women and Poor) % Increased crop yield YES (N= 312) # Innovation Name % Women Increase crop yield % Poor Increase crop yield 1 AST 100% (6) 82% (37) 2 Aybar 100% (5) N/A 3 WASTE 95% (22) 94% (36) 4 Greenheat 92% (26) 86% (26) 5 ITIKI 66% (30) 63% (24) 6 Aqysta 63% (5) N/A 7 Hydroponics Africa 54% (20) 57% (16) 8 MNP 31% (5) 51% (22) 9 WGI 23% (3) 20% (6) 10 Naireeta Bhungroo 14% (7) 14% (7) 11 Skyfox 8% (2) 16% (7) OVERALL 59% (131) 54% (181) Source: SWFF Field Evaluator Surveys On average, 85% of respondents reported higher crop yields using the innovation. The numbers are considerably lower for women (59%, N= 131) and poor (54%, N=181) than for all respondents; this does not negate this significant achievement of SWFF innovations but is an indicator of some unevenness in impact. In 7 out of 11 innovations (N= 114), there is a 50% or higher response on improving agricultural productivity. AST, Aybar, WASTE, Green Heat, and 12 Data from report from innovation submitted to SWFF TAF. 74 CUT ITIKI are innovations which have a marked impact through design: for example, AST identifies and reproduces seeds which are drought resistant, WASTE helps bridge farmers over the dry season, and CUT ITIKI provides valued advice on the time to plant and reap. The impact of the increase of agricultural productivity can be assessed in terms of increase of the harvest for the same amount of inputs, or as a significant saving of resources (human, time, water, money, etc.) or as a more effective use of inputs (water, fertilizers, labor, pesticides, etc.) for a similar yield as in previous growing seasons. Innovations that are focused specifically on water capture and storage or water re-use allow farmers to increase the number of growing seasons per year, thus increasing the agricultural productivity and strengthening the resilience of agricultural systems to the effects of climate variability and change. The data collected shows, in some cases, agricultural productivity was reported to have been increased by providing improved irrigation systems (examples include aQysta, WASTE), by providing high production seeds and resistance to soil salinity (examples include Lal Teer, ICBA) in addition to a low consumption or re-use of water (examples include WGI), by helping plants to adapt to water-related stress environment and thrive in less than optimal growing conditions (examples include AST, Newsil, CUT ITIKI), or by making previously unused land available for farming (examples include Aybar). This agricultural productivity is measured by comparing the previous productivity of the crop with its current productivity in the same space and same land. However, this estimation of the agricultural productivity is not uniform across innovations. Overall, water efficiency (as reported by users in surveys) has been found to be low (Revisit: 50%) compared to agricultural productivity (Revisit: 85%). This could be explained by the fact that all the SWFF- funded innovations have a clear mandate of improving agricultural productivity whether it is through for example making agriculture possible in challenging environments, increasing the number of growing seasons or increasing yields by using performance-enhanced seeds. In addition, agricultural productivity could be deemed to be more or less easily and approximately quantifiable in the field by users. Indeed, farmers, in most cases, can recall or roughly estimate their agricultural production, sometimes for several past seasons, as they use traditional measuring system that can be easily converted in conventional units of measure. Further explanation comes from the objectives of innovations themselves. Some SWFF-funded innovations do not have a mandate to increase water efficiency even if, in some cases, they end up doing so. Some innovations’ objectives are, for example to increase crops resilience (making them grow in extreme environments) or to serve as prediction tool for timely and targeted use of limited resources. In addition, from the interviews carried out during first and revisit surveys, as well as by SWFF field evaluators, it seems that, conversely to agricultural productivity, water efficiency estimation or accurate measure by farmers of the quantity of water used over the span of a growing season, proved more challenging than anticipated. Possibly farmers are more aware of improved yields rather than the more complex idea of improved yields being achieved with less water. These points can help explain the imbalance between results for the agricultural productivity and water efficiency. 75 Data presented in the table below (Agricultural productivity Outcomes and Targets) provides an overview of various assessments of estimation of agricultural productivity outcomes and targets from the First Visits (2019). This data is derived from KII which could not readily be repeated to show change over time; the record of the results from surveys provide updates. Table 38. Agricultural productivity Outcomes and Targets Productivity outcomes, N=23 Productivity data is based on logical estimates 8 35% Productivity estimates are based on very rough estimates 3 13% Productivity increases are based on actuals 12 52% Productivity increases, N=23 Productivity increases not verifiable 5 22% Productivity increases significant 6 26% Productivity increases very significant 12 52% Source: KIIs – First site visits and WWW. Note: The source for the various categories employed above comes from the questions, codes and analysis of the KII. The responses of key informants to questions have a) been appraised by the interviewer against the evidence of site observation b) checked, where possible, against the innovation reports and c) again checked, where possible, against survey data. Significant, here means evidently and incontestably materially changed, very significant here means evidently and incontestably changed to a considerable degree, not verifiable here means that data was not available to authenticate a judgment. Values in the table Agricultural productivity Outcomes and Targets demonstrates that the measure of agricultural productivity increase was mainly based on actual measured quantitative values (52%, N= 12) based on actual measurements and in others it is based on extrapolations from the limited available data and/or rough estimates and approximations. Productivity increases were judged by 52% (N= 12) of respondents to be very significant and another 26% (N=6) are significant. In the table below on outcomes for targets in productivity is assessed. Table 39. Agricultural Productivity Targets Productivity Targets, N=23 Targets realistic 20 87% Targets realistic but not achieved 1 4% Targets unrealistic 2 9% Source: KIIs – First site visits and WWW. Note: Since all targets are the product of joint consultation, “unrealistic” may indicate a change in conditions or over-confidence in making the target. The overwhelming verdict is that the targets were attainable: 87% (N= 20) of innovators interviewed felt their targets were realistic; 9% (N= 2) felt they were unrealistic. The estimation of agricultural productivity has many challenges. For example, in the case of ICBA in Egypt, farmers and the researchers all suggested that agricultural production has increased with the use 76 of the salt-tolerant seeds. The productivity increases are seen as very significant as they are largely turning non-arable land into productive land. The agricultural productivity is based on actuals as the numbers are derived from the volume of crops that are brought to the station for processing. However, with this innovation, a key problem arises. As for most genetically modified or enhanced seeds, farmers must go back to the initiator to purchase the seeds for each growing season. Thus, they cannot practice the ancient tradition that is to save the seeds from the previous season for planting during the next season. The agricultural productivity can also be evaluated in the more precise use of inputs: resources saved whether it is on time spent irrigating, value of pesticides effectively used, cost of labor spared, etc. For example, with Ignitia in Ghana, it is difficult to measure the increased productivity with any degree of certainty, but the forecasts provided by the innovation are used by farmers to know when to plant, to make a more effective use of pesticides to ensure rainfall is not diluting or washing the spray off the plant, or to make an informed decision on when to harvest so high levels of rainfall will not spoil the crop. Climate change is making weather patterns less predictable and so the forecast helps farmers to know when to plant to save as many resources as possible and be as efficient as possible. Table 40 (SWFF Innovation Effect on Inputs) below presents the outcomes of the effectiveness of inputs when used in conjunction with the SWFF innovations. Table 40. SWFF Innovation Effect on Inputs Effect on use of inputs, N=22 Innovation results in more effective use of inputs~ fertilizer and water 1 4.5% Innovation results in more effective use of inputs~ labor ~ fertilizer ~ water 1 4.5% Innovation results in more effective use of inputs~ labor ~ fertilizer ~ water ~ pesticides 12 55% Innovation results in more effective use of inputs~ labor ~ water 2 9% Innovation results in more effective use of inputs~ water 6 27% Source: KIIs – First site visits and WWW. Note: None of the respondents mentioned less effective use of inputs but was not asked explicitly. These results show that there is a positive influence of SWFF innovations on the effectiveness of inputs regardless of the types of inputs used. However, effectiveness of inputs with the use of SWFF innovations is the highest for a combination of the maximum number of inputs: labor, fertilizer, pesticides and water (55%, N= 12). The only other indicator of high value is that of more effective use of water, 27% (N=6). More work should be done to increase the effectiveness of SWFF innovations while using a minimal set of inputs as most farmers cannot afford to buy all the necessary inputs. For innovations such as Practical Action, the agricultural productivity and the achievement of targets are more evident: “Where there used to be barren land, farmers can now be productive”. The productivity increases for these farmers are 100% in most cases because the baseline was zero. Although the pumpkin growing activities are more labor intensive and most of the farmers didn’t use to grow pumpkins, the innovation allows for crops to be grown in a space where during half of the year farmers couldn’t plant anything and therefore make no money. Most beneficiaries 77 see increases in productivity and income if they follow recommended agricultural procedures and take care of their crops. Another example of evident value added of the innovation is the case of WGI Aquaponics in Uganda. There is evidence of yield increases because farmers are not only growing more vegetables than they were prior to having the unit (because additional seeds come with the unit), but also, they are involved in pisciculture or fish farming to eat (or sell). The recycled water with the nutrients is a great fertilizer for the plants and increases yields. The innovation also helps with food security and resilience to climate change because it reduces the reliance on fish in lakes, which is depleting, and it also provides the farmers with a more condensed and efficient way to grow vegetables and fish, providing them with a complete meal. However, overall, little evidence has been demonstrated to adaptation to climate change or consideration of climate change when designing or implementing the innovations. This could be explained by the fact that “addressing climate change was not a direct objective but rather a contextual factor for SWFF”, as an innovator stated. Despite this, in the section below, a number of changed farming practices are recorded and analyzed. The evidence on the effect of climate change is presented in the Annex Table Changes in crop yields or farming practices due to climate change. Respondents interviewed overall agree (62%, N=367) that they have experienced climate change and that they need to adapt their farming practices to it. Respondents believe that extreme weather conditions especially the changes in rainfall and temperature, have influenced their farming practices. For example, the greatest experience of changes in farming practices or crop yields due to climate change is cited by Customers/End-Users of Aybar (100%) and Naireeta Bhungroo (96%) and the least by Lal Teer (16%). The variance could be explained by the direct experience of farmers of extreme weather and the utility of these innovations in helping their response to such extremes: Aybar by draining excess water away from crops, and Naireeta Bhungroo by filtering, injecting, and storing rainwater underground for use in lean periods to provide food security. Lal Teer, by comparison, is already intrinsically an adaptation to such changes by providing salt-tolerant seeds and cultivation methods designed specifically for high-saline areas. In that effect, there is a worldwide tendency from farmers who are using the innovations to make changes, to some degree or other, in their farming practices (Table 41: Customers/End-Users responses, Degree of Change in farming practices, 2 or more changes in farming practices due to innovations). Table 41. Customers/End-Users responses, Degree of Change in farming practices, 2 or more changes in farming practices due to innovations % 2 or more changes in farming practices YES (N= 362) # Innovation Revisit / FE 1 Aybar 100% (50) 2 Green Heat 100% (35) 78 % 2 or more changes in farming practices YES (N= 362) # Innovation Revisit / FE 3 Lal Teer 100% (31) 4 WGI (Aquaponics) 100% (30) 5 CUT ITIKI 94% (29) 6 CUT Swayimane 94% (29) 7 ICBA 89% (14) 8 Hydroponics Africa 88% (30) 9 Practical Action 83% (25) 10 WASTE 73% (19) 11 Skyfox 65% (22) 12 Naireeta Bhungroo 60% (15) 13 Newsil 57% (12) 14 aQysta 40% (21) 15 AST N/A 16 MNP (Ecorangers) N/A OVERALL 81% (362) Source: Revisit and SWFF Field Revisit Surveys Generally, a high level of farmers (81%) made 2 or more changes in their farming practices in an attempt to mitigate the effect of climate change. These farming changes overall transcend geographical boundaries as they appear to be in one way or another a combination of the same practices. The most substantial farming practices changes are cited by Aybar, Green Heat, Lal Teer and WGI Customers/End-Users (100%) and the least by Newsil (57%) and aQysta (40%). The range of reported changes is narrower than with other indicators; this seems to show that making two changes is accepted as necessary to respond to climate effects. Interviewees have also acknowledged that they were able to make adequate changes in their farming practices solely due to the use of the innovations. These innovations have been accepted as being a welcomed ally in their preparedness to climate change. The most frequent changes include “introduction of new crops”, “deciding when to plant”, “reduction of water usage”, and “changes in irrigation practices”. Although these combinations or changes in farming practices cannot be necessarily deemed as “positive” or “negative”, they can definitely be labelled as being “adequate” given the current climatic conditions faced by farmers. In addition, a higher resiliency of crops due to the use of SWFF’s innovations has also been stated by farmers. See the Annex table for Impact on resilience of crops to climate change. Across all innovations surveyed, 71% of respondents recognized the pertinence of innovations as helping in increasing the resilience of their crop to climate change. The highest resiliencies of crops due to the use of the innovations have been reported for Green Heat, WGI and Newsil by 100% of respondents. Responses from women and poor do not vary from those of all customers/end-users. 79 Innovations that inherently tackle more or less the challenges of climate change are mostly the ones that allow a forecast of the weather (i.e. Ignitia in Ghana) or of the potential stress areas on farmers’ land (Future Water in Mozambique), so they can address preemptively possible problems. Other innovations whose results should lead to an increased resilience over time to climate change are Practical Action in Bangladesh because it improves the agricultural productivity on otherwise barren land, and Skyfox in Ghana because it facilitates water storage, which is a key response to enhancing resilience to climate change and allowing farmers to continue farming regardless of changing weather patterns. Farmers face risks related to price, customer demand, yield, resources, and climate that make income unstable from year to year. A critical question is access to credit, insurance, and inputs; the issue of credit in African countries being associated not only to unequal opportunity and access to information but also to high interest rates. If farmers are impeded by credit constraints, they will not be able to take advantage of the different technologies that are presented to them even when these technologies are available to produce more or better. Indeed, a lot of work is still left to be done when it comes to the access to credit. See the Annex table 62 for Customers/End-Users responses, Access to credit). Data in the table is presented from highest to lowest improved access to credit. Overall, only 33% (N=183) of respondents had a positive answer regarding the ease of access to credit or improved access to credit due to the use of innovations. This is undoubtedly a challenge to all innovations. Some such as Naireeta Bhungroo and Green Heat appear to have institutionalized a response to this demand, others such as ICBA and MNP (Ecorangers) do not appear to have such institutionalized access to credit. In Annex table 63 (Customers/End-Users responses, Access to credit (women and poor); the responses of women and poor are presented. With a few exceptions, the proportion of farmers accessing credit are generally lower for women (27%, N=55) and poor (23%, N=70). It is worth noting that some of the low numbers are because most of the time, the answer is "no need for access to credit". Also, although the overall numbers are low, a high percentage of women and poor have seen an improvement in their access to credit, especially for Green Heat, Skyfox, and WASTE. FINDING: Results established evidence of noticeable reported increases in farmers’ agricultural productivity and resilience to climate change as being a definite byproduct of the use of SWFF￾supported projects. A high level of farmers (81%) are making 2 or more changes in their farming practices. Innovations offer farmers an array of solutions that overall transcend geographical boundaries as they appear to be in one way or another a combination of the same practices. Hence, a high proportion of farmers report they are taking advantage of innovations to make two or more changes to their farming practices repertory. RECOMMENDATION: More research is needed to increase the effectiveness of SWFF innovations while using a minimal set of inputs as most farmers cannot always afford to buy the necessary inputs. WE4F needs to focus on mitigation of climate change (and thus increased resilience) as a key direct objective by integrating Climate Smart Agriculture in the design and implementation of the supported innovations. Climate-smart agriculture (CSA) is an approach 80 that helps to guide actions needed to transform and reorient agricultural systems to effectively support development and ensure food security in a changing climate. Question 4c. Have vulnerable groups (the poor, women, ethnic minorities) been positively and/or negatively impacted (through income, employment, water/environmental) from SWFF supported innovations? Specifically, has SWFF reduced the number of people in poverty as a result of supporting SWFF-supported innovations and has SWFF increased the number of women benefitting from SWFF￾supported innovations?13 Across the SWFF-supported innovations, the impacts on vulnerable groups (poor, women, ethnic minorities) has been overwhelmingly positive in terms of benefits to them, but their targeting depends both on the business model and the initial focus of the innovator (for profit vs. social engagement). Most innovations target women as a primary or secondary customer/end-user, a byproduct of the smallholder farming culture in Africa, Asia, and Latin America where women comprise a majority of this group. However, gender relations in Africa and Asia are complex; for many of the innovations there is a male designated as the “owner” of the innovation, but in a majority of those cases it is the wife who is responsible for the day to day management and upkeep (e.g. WGI, Hydroponics Africa). In many countries, women participate heavily in farming activities, but not as the owner of land, and therefore is the immediate customer/end-user of non-income￾based impacts, such as time saved. For example, every Hydroponics Africa customer/end-user (majority women), without prompting, cited time savings as the primary benefit, even those without a harvest season completed (and therefore no income increase). Since women are the primary small-scale farmers (for both cash and subsistence) in Africa and Asia, it is the women and their children that bear the burden of watering, fertilizing, harvesting, and planting. Therefore, it is they that are most impacted by the vast time savings that comes with hydroponic farming, leaving time for more income-generating activities, education, or family. Complementarily, for innovations like Green Heat, Uganda’s biogas water recycler, even when owned or managed by men (though not all are), affects women cooking time, also freeing them for other activities in addition to improving health by reducing smoke in the house. Innovations such as Naireeta Bhungroo and WASTE attenuate gender differences by bringing in women into the management of saving groups or of a marketing company to rebalance unequal gender relations. For example, Naireeta Services’ Bhungroos trained and emboldened women to operate and monitor the Bhungroos14 . Most end-users are women and poor farmers. Indeed, this demographic represented most interviewees during the first (96%), the revisit (68% women, 84% poor) and the Field Evaluators (98% women, 98% poor). In the last year of their SWFF award, 13 This section relates gender in relation to access to the innovations and female-specific impacts; in Question 11 gender issues are explored in relation to women’s access as Customers and to participation in management and ownership of innovations. 14 SWFF Final Annual Report, March 2020 81 Naireeta Services continued to expand their customer base, which consisted of extremely poor farmers in 10 regions throughout India. By curtailing desertification, the initiative helped women be empowered and build resilience to environmental changes. In addition to providing food security during rainy periods and ensuring crop survival in lean periods, Bhungroo systems supplemented household water needs. End-users, most of whom were women, reported various benefits from the use of the innovation: increase of agricultural productivity, improvement of households’ food security and improvement of the economic status of their households. Innovations like Water Governance’s Aquaponics Farming that are also owned by women directly benefit women income in poor and underserved areas in the cities and border areas. The tables which include Revisit and Field Evaluator surveys have a sample capable of measuring pro-poor intentions. The question is whether the SWFF’s emphasis on the poor, women, ethnic minorities has been strong enough to secure a pro-poor result in the context of both innovation and development. The question is whether this is possible in a sustainable/financially viable business model. At times, given SWFF’s emphasis on innovation and private sector enterprise but USAID’s writ large goal of development and an emphasis on poor, there is a disconnect between having a viable, profitable, and sustainable entity, and one which serves the populations USAID cares more about. For those that focus on the very poor/poor, how successful have they been at also integrating a sustainable private sector enterprise model into their business plan? This is discussed from another perspective in Question 11 in the context of the tension between commercial viability and the target market of traditional development assistance. SWFF can be seen sometimes to have two potentially competing objectives that need to be carefully considered, balanced, and intentionally targeted. As we discuss below in Question 11, the business model needs to be intentionally designed to incorporate subsidies from the wealthier Customers or the government to include poorer farmers, while not losing profitability and sustained market presence. During discussions with innovators and other key stakeholders, it’s clear that both profitability and sustainability as well as a target market of vulnerable populations are equally important, but it’s not clear that consistent technical support and guidance is given to 1) for-profit firms that don’t have experience in social entrepreneurship and 2) non-profits that have never had a profitability mandate or clause directly in a donor-funded contract. These are not findings nor recommendations at the moment, but considerations for exploration in future programs to determine how to approach this delicate balance and provide assistance to each variation of innovator that allows them to achieve both goals simultaneously and without sacrificing quality or performance. Complementarily to the dilemma we outline above to balance the business model side of the equation, there’s the issue with targeting poorer farmers in a for-profit model who are more risk averse, with less access to cash, and generally apprehensive about adopting new technology. Perhaps a staggered payment plan or loan model will alleviate concerns, but innovators’ beneficiaries cited high cost, risk aversion, apprehension to adopt untested technology, and 82 cultural preference for traditional methods as reasons for their initially not adopting, not wanting to pay, or as reasons their neighbors haven’t yet joined the model. FINDING: Overall, women and vulnerable groups have been positively impacted but not to the level of the average. Women and the poor do not, for example, access credit at the same level as most Customers. Though these impacts are not as striking as expected, they cannot negate the stride that has been made in Gender-integration by SWFF-supported innovations. More emphasis was placed round after round on integrating women at all levels of innovations, whether as Customers/End-Users or at different levels of the innovations themselves. RECOMMENDATION: Provide consistent technical support and guidance to each variation of innovator to balance both the business (profitability) and social (vulnerable groups) model side of the equation to allow them to achieve both goals simultaneously and without sacrificing quality or performance. Cross subsidies need to be considered and included in the design of innovations. Gender Integration (or mainstreaming) should be a key consideration in innovation design. This strategy for integrating gender concerns in the analysis, formulation and monitoring of policies, programs and innovations to promote gender equality and the empowerment of women and the vulnerable in population and development activities should be considered for all innovations because different communities’ groups demand different approaches. Question 4d. What is the balance between public/social engagement and private/public engagement? To what extent have private funds been generated that contribute to the developmental objectives of the program both during and following SWFF awards? A little less than half (N=11) of innovations in the SWFF portfolio self-identify as social engagement dominant; 13 of 24 (54%) interviewed identify as a business venture from the start. However, the issue of private sector funding/investment or lack thereof has not appeared to have been adequately addressed. Those that are predominantly social engagement focused need support becoming more investment friendly and profit driven, while those on the business venture side need a mix of technical assistance to push them towards financial sustainability and investor interest and investor matching. Data on private sector investment is presented in the table below. 83 Table 42. Private Sector Investment (N=23) Level of readiness, N=22 Not ready for private investment 59% (13) Investor-ready 41% (9) Level of investor interest, N=19 Significant interest from private investors 26% (5) Limited interest from private investors 32% (6) No interest from private investors 42% (8) Level of investor access, N=10 No access to private investment 42% (11) Some access to private investment 15% (4) Extensive access to private investment 15% (4) SWFF contribution to interest, N=22 Private investors interested based on opportunities through SWFF 32% (7) Source: KIIs – First site visits and WWW. Note: Private investment here refers to all funds not ascribed to SWFF itself or from government and includes an innovators own capital, grants from foundations or scientific bodies, loans, and financial costing of human resource or other support from other agencies. Table 42 shows the level of investment readiness, investor interest, and access, as perceived by the innovator during First Visits and WWW interviews, but also as assessed by the evaluation team based on existing data and first site visit observations. Subsequent to these interviews there is evidence of changed attitudes and new interest in private sector funding. 15 . The existence of private sector funding for SWFF-supported innovations has been reported as low and, in many cases, non-existent. At the time of interviews in 2018, the majority of innovations (N=13) were not yet ready for private investment, which could explain why the majority of innovators also had no access to private investment (N=11) or no interest from private investors (N=8). Most of the innovators who were leveraging, or sourcing additional funds were doing so through other public, government, or social engagement entities (host country or international donor), while actual private investment levels had been low. According to one innovator: Many angel investors have approached us. But most of them are focused on profit only. But we would like to work with farmers. We are looking for impact investors instead of venture capital. We are waiting for people with right values and ethics to work with us on the venture capital side. 15 At the time of the present final year of evaluation, several gatherings in which these findings on private funding interests to SWFF-funded innovations from 2018 could have been updated by in-person interviews with innovators, haven’t been possible due to several factors including the cancellation of meetings because of the worldwide COVID19 pandemic. 84 This quote exemplifies the disconnect at times between a for-profit and sustainable enterprise and one which is pro-poor and social engagement driven. Investors are looking to profit, and the majority of innovators (even those with a matured plan to scale and sustainable future) are looking to break even and have enough funds to sustain the enterprise, but not to ‘get rich.’ Because the typical investor may not be well suited for the type of innovators SWFF has supported (however successful), this is an area that key respondents agree needs a closer look and a more well-defined plan. There are challenges in matching innovations to investors. According to one respondent, “the program forces innovators to seek additional funding which is a must as it allows a level of validation. SWFF has a thorough evaluation and determination of the innovators but there is not a funding strategy discussion or how to understand the market provided to the innovators. Not sure we have done enough analysis to understand what is the matching funding that innovators are bringing.” This also helps demonstrate that while securing private sector funding is a core SWFF objective for innovators, little is currently done in an intentional and systematic way to help innovators identify the right funding stream or understand the financial markets in which they are operating, nor to understand what types of funds the innovators are currently bringing in to match SWFF funding and to help them determine whether those are the best fit. However, key stakeholders both in the IIAC and FPs as well as innovators identify private investment (be it providing technical assistance to build the capacity or actively working with innovators to find investment opportunities) as a major weakness of the SWFF program. It is an area that has lacked in performance against expectations. Those citing the weakness do so with the understanding that this is a complex and multifaceted topic: • some innovators are not yet ready for private investment • there is a finite amount of resources and many other SWFF priorities – like just working with innovators to get their innovation ready for market and sold on some level of scale. Table 43 (SWFF-Funded Innovations sources of funding, Year 1 to Year 5) presents an overview of the different sources of funding for SWFF-funded innovations throughout the length of the program. Table 43. SWFF-Funded Innovations sources of funding, Year 1 to Year 5 Funding Source % Public 37% Private 29% Self 34% Source: SWFF Online Impact Survey 2019, Sources of funding The most important source of funding for SWFF-Funded innovations are public funds (37%), followed by innovators’ own funds (34%) and private funds (29%). It is commendable on their 85 part to see the trust and faith innovators have in their products in order to invest so much of their own funds in their innovations. It is also admirable to notice the huge proportion of public funds that innovators have been able to procure over their time in the program. In addition, innovators were able to secure 29% of their funding from private investors, which leads to believe that with the right amount of maturity of innovations and an adequate funding strategy plan, innovators can discover “Impact investors” who will be driven by both profit and social engagement. SWFF does encourage its innovators to seek additional/outside funding, but the source of that funding seems to largely be public sector or cost-sharing with the innovator’s other projects. While cost efficient for SWFF in terms of accountability, it does not necessarily adhere to the principles of innovation in a private sector, a for-profit context. If SWFF intends for its innovations to be sustainable, financially viable, for-profit entities, then the capacity of organizations to reach the level of maturity at which they’re able to secure private sector funding at fair rates is a critical component of M&E and performance targets, technical assistance, and one’s general trajectory down the innovation runway. FINDING: SWFF-funded innovations have been able to secure different types of funding, therefore finding a balance between public/social engagement and private/public engagement. The most important sources are public funding and innovators’ own funds. Although private funds are the least prevalent source of funding, it is worth noting that the novelty of SWFF-funded innovations as well as their worthy cause appeal more and more to “Impact Investors” who are driven not only by profit but also by a social engagement towards vulnerable groups. RECOMMENDATION: Develop with innovators a customized funding strategy that will offer a balance between profit and social engagement. It will allow them to appeal to a larger and more diverse pool of funding sources. Innovations should not only be financially viable, for-profit entities but also socially engaged and sustainable. Some effort should also be devoted in monitoring SWFF graduates to measure continued public and private engagement in innovations and inform the design of novel and sustainable potential donors engagement strategies. Question 5. How much of the measured change (outcome or result indicator) in the SWFF program can in fact be attributed to the SWFF-supported projects? That is, what portion of the result is not explained by the projects examined by the evaluation? As quoted in a previous table, 85% of innovations cited SWFF as the major contributor to their success and outcomes. Either all outcomes are dependent on SWFF support or critical transitions are made possible by SWFF support. A few respondents stated that “major outcomes [were] not linked to SWFF support” and cited a relatively minor contribution. Others felt that SWFF accelerated their progress towards their objectives, from whatever stage they had started. 18 innovations reported that SWFF raised either or both their international and national profile, 86 leading to strengthened or new partnerships, investment, additional donor funds, or an expansion of market access. The extent to which outcomes and results are not attributed to SWFF is more difficult to assess. The data from which the evaluation team presents its findings on this question is based on the perceptions of innovators and partly also on the judgment of the consultant. In the section on Effectiveness and Efficiency the relative benefit to innovations from technical assistance and other support is appraised to determine a truer level of SWFF attribution. Based on the data the team has, majority of End-Users/Customers report that noteworthy changes contributed to the SWFF supported projects. FINDING: Although it is difficult to assess the extent to which the measured outcomes and results can be in fact attributed to SWFF-supported projects alone, end-users/customers reported that SWFF-funded innovations made a high-level contribution in the form of overall benefit from innovations, mostly in terms of improvement in agricultural output, income, water access and availability, and resilience of crops to climate change. RECOMMENDATION: None. Question 6. To what extent are there differences between the planned SWFF￾supported projects and what was actually delivered in Year 1 and then Years 2-3 of the projects? As shown in Table 44 (Perceived Fairness and Value of Targets) below, the majority of innovators felt that targets set in year 1 (59%) and years 2 and 3 (66%) were fair, even those that weren’t met. 61% of innovators highly valued the targets and milestones in general; while 22% somewhat valued them; 17% did not value them and over the three years an average of 20% felt that the targets were unrealistic. Table 44. Perceived Fairness and Value of Targets Perceived fairness of Year 1 targets, N=17 Year 1 targets fair 10 59% Year 1 targets modest 3 18% Year 1 targets unrealistic 4 23% Perceived fairness of targets for Years 2 and 3, N=18 Targets for years 2 and 3 fair 12 66% Targets for years 2 and 3 modest but challenging 3 17% Targets for years 2 and 3 demanding and unrealistic 3 17% Value of targets and milestones, N=18 Innovator did not value targets and milestones 3 17% Innovator somewhat valued targets and milestones 4 22% Innovator highly valued the targets and milestones 11 61% Source: KIIs – site visits and WWW. 87 Although in the table below innovators suggested modifications to the M&E system and complained about reporting requirements, 83% (15) of innovators said that the targets and reporting system were helpful in creating a viable business and therefore that SWFF support helped them in meeting those targets. One innovator very clearly stated that they hated the milestones, but in the end, it was the milestones that pushed them to reach their potential, expand market access, and improve their business model. Another innovator’s targets and SWFF TA contributed to the redesign of the innovation, reducing input costs and time to construct, allowing production and sales to match the speed with which marketers were able to find new clients. Table 45. Milestones, targets, and reporting system (N=18) % Targets and reporting system helpful in creating a viable business 15 83% Some targets and aspects of reporting seen as somewhat helpful 1 6% Targets and reporting system seen as unhelpful 2 11% Source: KIIs – site visits and WWW. Without the targets pushing the innovators towards a specific and time-bound goal, there is the potential that the innovations would have stalled. Overall, based on KIIs, innovators tend to find the targets to be a heavy yoke but understand the advantages of a) improved credibility and b) better internal practices leading to viability. As shown in Table 46(Innovator Perception of SWFF Targets, Milestones, Support, and Rounds) below, 58% of innovators felt that the SWFF rounds were treated differently, and 42% felt that SWFF requirements were perceived to change over time. Table 46. Innovator Perception of SWFF Targets, Milestones, Support, and Rounds Awareness of Support, N=18 Innovators greater awareness of available support 12 67% Innovators unclear about available support 6 33% SWFF Rounds, N=12 Rounds treated consistently 5 42% Rounds treated differently 7 58% SWFF requirements clear? N=16 Same TAF pressure on targets year to year but better understood and responded to 1 6% SWFF requirements (gender and poverty, for example) clear from the start 8 50% SWFF requirements perceived to change over time 7 44% FINDING: Targets, milestones and the reporting system were all valued by most innovations. These helped facilitate a greater reach of their potential, the expansion of their market access, the improvement of their business model or the redesign of the whole innovation when needed. They also improved the innovations’ national and international credibility and better their internal practices leading to viability. 88 RECOMMENDATION: Develop a standard set of targets, milestones and reporting system that could however be adapted to specific innovations as needed. Overall, results show that results for women and poor are lower than the average for all respondents. The reasons for this variance need to be explored. Apart from the key aspect of access to land and other resources there are those of education and social emancipation. Women may also prioritize production for domestic consumption rather for markets. A gendered division of labor and allocation of assets may affect the accurate estimation of variables such as income or agricultural production. Overall, results for women and poor are lower than those of all respondents. Reasons for these lower numbers for women and poor could be that they are in general less educated than men and wealthier members of communities, therefore different estimations when answering surveys questions might be off or not accurate. In addition, they are less likely to record inputs and outputs relative to their activities, less likely to sell their production in markets and more likely to consume at home, making accurate agricultural estimations also difficult. Moreover, there is overall a gender division of labor and allocation of assets that can affect the accurate estimation of variables such as income or agricultural production. It is also possible that there is an unequal power relations that limit women and poor in their choices and autonomy, in their access and control over resources (economic, productive: land, equipment, tools, credit), in their decision￾making and participation (access to information and capability of organization, training). Therefore, it is not just about agricultural productivity that women and poor would be at a disadvantage but in their overall lives in general. The results gathered on all variables from 2018 to 2020 inform the following overall findings and recommendations. Overall findings and recommendations are presented below to guide new programs. OVERALL FINDINGS: SWFF strongly contributed to outcomes and results. Impacts in terms of improvement in agricultural output, income, water efficiency and in the use of inputs (labor, time, pesticides, fertilizers, etc.) are reported. Impact of innovations on adaptation and resilience in the face climate change is more evident through greater adaptation and preparedness. A high level of farmers (81%) made 2 or more changes in their farming practices. These practices are embedded in better adaptation to climate change due to a greater availability of water, better information in adaptation to rainfall agriculture, targeted irrigations, and appropriate farming practices. Targets, milestones, and reporting system were valued by most innovations. They facilitated the greater reach of their potential, the expansion of their market access, the improvement of their business model or the redesign of the whole innovation when needed. They also improved the innovations’ national and international credibility and better their internal practices leading to viability. OVERALL RECOMMENDATIONS: Gender integration should be at the forefront of innovations and programs to help reduce the systemic inequality that exists in communities. The 89 greater inclusion of women will assist in overcoming the limitation in the opportunities of one￾half of its population. At the inception of a new program, baselines and robust counterfactual along a set of clearly defined impact indicators need to be established. These will constitute the definitions and data to validate the contribution of SWFF and other programs to outcomes and results of funded innovations. A focus on the adaptation and mitigation of climate change as a key direct objective of future programs, and therefore the resilience to climate change, is proposed by integrating Climate Smart Agriculture in the design and implementation of the supported innovations. Consistent technical support and guidance is needed for each innovator to balance both the business (profitability) and social (vulnerable groups) model to achieve impact which would be evenly shared between men and women, and better off and poor. Section 5: Sustainability Question 11. What is the likelihood that SWFF-supported innovations will have a sustained market presence (i.e. are financially and socially sustainable)? The prospects for an innovation’s sustained growth are assessed in terms of a growing base of customers willing and able to afford the innovation, access to investment, supportive regulation, and expanded focus on women as customers and users of the innovation. Interviews with innovators highlighted a wide range of internal and external variables that ultimately determine if SWFF-supported innovations will have a sustained market presence. Interviewees mentioned that regulation could work in favor or against certain innovations. The uptake of the innovation may be reliant on a pass-through organization (like the Mobile Telecom Provider for Ignitia’s messaging service) or government certification for the land (like SkyFox’s fishponds). Government policy on competing imports could have an impact, as would the creation of more regional markets and the infrastructure to support this. Among innovations in India there was an inclination to seek a market presence through incorporation or support from state agricultural agencies. Finance could be more or less easy to procure. Growth may be more or less well managed. Competitors may appear on the market or the market may change with more mechanized or larger scale operations buying out smallholders who then become laborers. Ultimately, the evaluation team identified a few key indications of potential success: namely, whether there was increased willingness to invest in and fund the innovation from public and private sources, whether current customers see themselves continuing to use the innovation, whether they would recommend the innovation to others, and whether they would be willing to pay for the innovation going forward and could afford it. Many innovation products are free e.g. WASTE, NewSil, CUT Swayimane and others. Rather than charging, these innovations are hoping to get state funding or incorporation in agricultural extension and continue with free provision. 90 An interviewee, for instance, saw expanding markets for their product: The prospects are good. The market for perishables is getting stronger with, at times, gluts in production seeing the markets respond across the region. Foreign owned grocery stores and major restaurants are also starting to buy more locally rather than importing from South Africa or elsewhere. In terms of sustained operational viability, interviews with innovators suggested that several are already or soon would be making a profit from the innovation, i.e., “operating in the black without counting on grants and other financial assistance”. Several interviewees highlighted that, in practical terms, it takes ten years for such innovations to overcome initial obstacles, start to take off and eventually become deeply embedded in the local economy. This is a time beyond the length of SWFF support and has to be taken into consideration; even for a succeeding innovation, sustainability may not be attained within the project life of SWFF. A measure of sustainability is the adoption of the innovations by customers and end-users, measured for our purpose by three indicators in the willingness to 1) continue to use the innovation in the next 5 to 10 years (Table 47: Sustainability indicator: use of innovation on the long term (5 to 10 years) and Table 48: Sustainability indicator: use of innovation on the long term (5 to 10 years) (women, men and poor)), 2) recommend the innovation to others and therefore expanding the reach of the innovation to new customers and new markets (Table 49: Sustainability indicator: Recommendation of innovations to other farmers by Customers/End-Users and Table 48: Sustainability indicator: Recommendation of innovations to other farmers by Customers/End-Users (women, men and poor)), and 3) pay to use the innovation (Table 50: Willingness to pay for innovation) Table 47. Sustainability indicator: use of innovation on the long term (5 to 10 years) % Use of innovation in the future YES (N= 528) # Innovation Revisit / FE 1 Aybar 100% (50) 2 Hydroponics Africa 100% (34) 3 Skyfox 100% (34) 4 CUT Swayimane 100% (31) 5 Lal Teer 100% (31) 6 Waste 100% (26) 7 Newsil 100% (21) 8 ICBA 100% (19) 9 CUT ITIKI 97% (30) 10 Naireeta Bhungroo 96% (24) 11 MNP (Ecorangers) 95% (62) 12 Green Heat 94% (33) 13 aQysta 92% (48) 14 WGI (Aquaponics) 90% (27) 15 AST 84% (42) 91 % Use of innovation in the future YES (N= 528) # Innovation Revisit / FE 16 Practical Action 53% (16) OVERALL 94% (528) Source: Revisit and SWFF Field Evaluator Surveys Overall, from the table above, a high proportion of respondents (94%) report they will continue to use the innovation in the next 5 to 10 years, suggesting a high probability of adoption of the innovations. Some of the reasons cited for a continued use of the innovations in the next 5 to 10 years from surveys and site visits include: saving labor time and costs; increasing food security, income and crop yields; that the innovation is environmentally friendly; source of employment; diversification of diet; gaining of knowledge through trainings, inclusion of women farmers, opportunities to share ideas with fellow farmers. All innovations report high levels of support except for Practical Action. Table 48 (Sustainability indicator: use of innovation on the long term (5 to 10 years) (Women, Men and Poor)) shows disaggregated values for women, men and the poor. Table 48. Sustainability indicator: use of innovation on the long term (5 to 10 years) (Women, Men and Poor) % Use of innovation in the future YES (N= 895) # Innovation Name % Women Use innovation in the future (n) % Men Use innovation in the future (n) % Poor Use innovation in the future (n) 1 AST 100% (6) 82% (36) 82% (37) 2 Aybar 100% (5) 100% (45) N/A 3 Naireeta Bhungroo 100% (50) 100% (1) 100% (51) 4 Skyfox 100% (24) 100% (35) 100% (42) 5 WASTE 100% (23) 96% (32) 100% (38) 6 WGI 100% (13) 97% (34) 100% (30) 7 ITIKI 97% (44) 100% (17) 100% (38) 8 Greenheat 96% (27) 76% (23) 96% (29) 9 MNP 93% (15) 95% (47) 95% (41) 10 Hydroponics Africa 78% (29) 100% (11) 85% (24) 11 Aqysta 75% (6) 95% (42) N/A OVERALL 94% (242) 95% (323) 95% (330) Source: SWFF Field Evaluator Surveys In terms of commitment, the values for women (94%), men (95%) and the poor (95%) follow the same trend as for the overall (94%) use of innovations in the next 5 to 10 years. This could 92 indicate that the communities as a whole recognized the value of the innovations in their lives and are committed to their own betterment by capitalizing in the innovations’ benefits. Table 49 (Sustainability indicator: Recommendation of innovations to other farmers by Customers/End-Users) shows data on the willingness of respondents to recommend the innovation to their network. Table 49. Sustainability indicator: Recommendation of innovations to other farmers by Customers/End-Users % Recommendation of innovation to other farmers YES (N= 535) # Innovation Revisit / FE 1 Aybar 100% (50) 2 Green Heat 100% (35) 3 Hydroponics Africa 100% (34) 4 Skyfox 100% (34) 5 CUT Swayimane 100% (31) 6 Lal Teer 100% (31) 7 CUT ITIKI 100% (31) 8 WGI (Aquaponics) 100% (30) 9 Waste 100% (26) 10 Newsil 100% (21) 11 ICBA 100% (19) 12 MNP (Ecorangers) 96% (63) 13 Naireeta Bhungroo 96% (24) 14 aQysta 92% (48) 15 AST 82% (41) 16 Practical Action 57% (17) OVERALL 95% (535) Source: Revisit and SWFF Field Revisit Surveys Most respondents would recommend the innovation to other farmers, suggesting a high degree of satisfaction with the innovations. For almost 70% (11 out of 16) of innovations, the percentage of likelihood of recommendation of the innovation by the respondents to others is 100%, and almost 90% (15 out of 16) of innovations have a percentage of recommendation higher than 80%. The highest values (100%) could be seen for innovations such as Aybar, Green Heat or Hydroponics Africa, while the lowest values are for AST (82%) and Practical Action (57%). In the case of Naireeta Services’ Bhungroos, farmers had an overall positive attitude towards the Bhungroo systems and 100% of farmers interviewed reported a continued use of the innovation in the next five to ten years, while approximately 96% of farmers stated that they’d recommend the innovation to others. As of March 2020, Naireeta Services installed 184 Bhungroo rainwater 93 harvesting systems, reached nearly 7,000 end-users/customers, helped produce nearly 50,000 metric tons of produce, and helped store more than 63,500 liters of water for agriculture16 . SWFF innovations get a high score for the measure of sustainability in the question “attitude to recommending innovation to other Customers/End-Users”. In the Field Evaluator surveys, there is a high overall figure of 95% of respondents who “would recommend”, which is higher among women Customers/End-Users (94%) and slightly lower for the poor (90%). The highest values of recommendation of innovation by women are found for innovations such as aQysta, AST, and Aybar. The lowest values are seen for CUT ITIKI (86%) and Hydroponics Africa (70%). These values show overall adoption of the innovations by women, men and poor. For many of the innovations, there is a tension between commercial viability and the target market they are trying to reach under the SWFF program. See Annex Table 5.2 on Sustainability. Table 50. Willingness to pay for innovation % Willingness to pay for innovation YES (N= 622) # Innovation Name % would pay for innovation (n) % Women would pay for innovation (n) % Men would pay for innovation % Poor would pay for innovation 1 AST 98% (44) 100% (6) 86% (38) 87% (39) 2 Aybar 76% (38) 80% (4) 75% (34) N/A 3 WASTE 58% (33) 43% (10) 69% (23) 57% (22) 4 MNP 55% (36) 56% (9) 55% (27) 53% (23) 5 Green Heat 50% (29) 57% (16) 43% (13) 50% (15) 6 WGI 43% (21) 53% (7) 40% (14) 56% (17) 7 Hydroponics Africa 37% (18) 32% (12) 54% (6) 17% (5) 8 Aqysta 35% (18) 50% (4) 31% (14) N/A 9 Skyfox 15% (9) 25% (6) 8% (3) 21% (9) 10 CUT ITIKI17 0% (0) 0% (0) 0% (0) 0% (0) 11 Naireeta Bhungroo 0% (0) 0% (0) 0% (0) 0% (0) OVERALL 42% (246) 45% (74) 42% (172) 38% (130) Source: SWFF Field Evaluator Surveys 16 SWFF Final Annual Report, March 2020 17 In the case of CUT ITIKI and Naireeta Bhungroo, poor are not being asked to pay for the services and either government sources or cross-subsidies are covering the costs. 94 The willingness to pay for an innovation, whether it is a higher, equal or lower price for the innovations is reported by 42% of all respondents, and specifically by 45% of women, 42% of men and 38% of poor interviewed. Only five out of eleven innovations (AST, Aybar, WASTE, MNP and Green Heat) had 50 percent or more of those surveyed suggest that they would be willing to pay a market price for the innovation. Two (CUT ITIKI and Naireeta Bhungroo) had no farmers prepared to pay for the innovation. Disaggregated data among women, men and the poor show a similar trend. However, overall, when end-users recognize a clear benefit from the innovations, they are willing to either spend the exact amount of money they are spending now to have access to the innovation or to even pay more. Indeed, farmers’ valuation of an innovation increase with prior awareness, prior use or prior witness of benefits garnered from the use of innovations by others. On the other hand, some end-users are still willing to pay for the use of the innovation only if they would be spending less than what they are paying now18 . These issues are further explored in the section on Sustainability. At present, some innovations are making their products either freely available to customers (which is a challenge which must be surmounted to achieve financial sustainability), have users making subsidized payments, or are anticipating government support (e.g. WASTE) for aspects of their operation in the future while selling some of their produce to cover costs. Innovations for which customers or end-users have to spend any amount of money to use it would have to come up with innovative strategies to be equally financially accessible to every member of the community, offer their products at a reasonable price while also being profitable and grow as a business. When there is uncertainty surrounding a new technology (as it is the case for these novel SWFF￾funded innovations), a possible solution could be to offer a one-time price subsidy for the new technology, which could spur demand and increase future adoption. These initial (and temporary) subsidies for a new technology can be an effective way of scaling up adoption and expansion, increasing profitability and balancing the innovators’ need to not only cater to all groups of end￾users, including vulnerable groups, but to also be profitable. Achieving this balance between profit and social engagement would be a great way to attract investors, whether public or private. Therefore, a recommendation to innovators would be to invest in extension services and advertising by all actors in their business supply chain to raise awareness about the technology. This would also motivate investors, especially impact investors19 , who could see a guaranteed return on their investment, both financially and socially. In instances where willingness to pay and affordability would prove to be a problem, the business model would need to be deliberately designed in such a way whereby wealthier consumers or government agencies cross-subsidize poorer farmers or for which local low-interest loan mechanisms can be offered to poorer customers. These would need to be determined on a case-by-case basis as for some innovations like Naireeta Bhungroo, for which end-users were meant to contribute labor rather than money 18 It is however worth noting that within the scope of the present evaluation, only the willingness to pay for an innovation has been measured, not the ability to pay nor the affordability of innovations have been assessed. 19 Impact investors are investors who are driven by both profit and social engagement. 95 and landowners were meant to cover some of the construction costs but then benefited from having a well available on their land. This is particularly true of the more investment-heavy infrastructure-based innovations (like aQysta’s water pumps or SkyFox’s fishponds), where an investor would be welcomed but which may see government as the primary buyer or interlocutor in some countries. Investors would otherwise need to wait several years before recovering their investment. Product-based innovations may struggle with how best to ramp up production, which may also require significant investment while service-based information technology innovations may quickly attract customers and become viable. There is a keen awareness among even socially oriented innovations of the need to become economically viable even as they target the very poor. As an innovator stated: “We have reached the Break Even point from 2015 onwards. Though we are not making huge profits, we have enough work to do for the next 10 years. Socially it is challenging, however, we are giving priority to organizations that are keen to work with women. We are working to survive in an economic climate in which competitors are keenly aware of the potential profit of our technological innovation. Some people want to grab our idea by slightly changing our name and approaching rich farmers. Making a social program sustainable is very difficult in this country at this point of time.” For some innovations, farmers will logically gauge their risk only after two or three growing seasons (as was mentioned in the ICBA/DRC program in Egypt, for example). For farmers who are used to receiving free inputs and services, there may be a lag until they are willing to cover the full market price of the innovation or the government is willing to allow innovators to charge. Without this track record with paying farmers, investors may also not be willing to take the risk. As noted by one IIAC member, “for the kinds of nascent ideas that are involved in SWFF, it may be hard to find investment capital.” Results show that even though in some instances, the innovation itself may not be a big moneymaker but rather the complementary products and services are what generate the income for the enterprise. This is the case for example with the salt-tolerant seed varieties in Egypt, whereby the crop processing stations might in fact be what brings in the money. The sustainability of innovations is tied into the stages of development of the innovation itself, the finance it attracts, and its market presence. The viability of the innovations within their three years of support (and beyond) is set out in Table 51 which indicates the status of innovations by the Rounds in which they entered as awardees of SWFF. The progress of each innovation beyond its Round through subsequent years towards successful graduation does not necessarily indicate sustainability into the future. While still within the SWFF fold, innovations continue to receive funding and this may mask a lack of market presence. Despite this, SWFF TAF has a rigorous review procedure which will identify such deficits and, if these are not addressed, will terminate support. While a few alumni are still in the field, they have been assessed (using the rigorous criteria IIAC deploys) as not progressing towards a sustained future. 96 Table 51. Status of innovation by Round Status of innovation, 2020 Round Alumni Graduated Grand Total 1 7 9 16 2 1 1 2 3 6 6 12 4 3 7 10 Grand Total 17 23 40 Source: Awardee Results database Note: There is no “Active” column as each Round is now complete and all active innovations are now Alumni or Graduates. The status of innovation in 2020 above can be compared to the sample of all innovations undertaken late in 2018. The earlier table sets out the initial status of the sample as a group of innovations representative of all 40 innovations in 2018; this table demonstrates the final status. At the time of the first visit surveys, there were 14 active innovations in the sample in their initial status, now all have moved on to become alumni (those who entered the SWFF program but did not complete the three years) or graduates (those who completed the three years of funding). There were initially 15 alumni and 11 graduates in the sample. Of these, 14 innovations then active early in 2019, 2 are now alumni and 12 are graduates. This transition indicates that of those active in early 2019, 83% have successfully achieved the status of graduates and, year by year, have met the rigorous criteria for support. Reviewing the final status in the table above, among Round 1, there are 9 graduates and 7 alumni. All but one of the alumni were terminated after the first year. Round 2 awardees had a single grant and are not like other rounds. In Round 3, there are an equal number of alumni and graduates while in Round 4 there are only 3 alumni but 7 graduates. These results show that the proportion of succeeding innovations has tended to rise over the rounds; in Round 1 56% were graduate innovations, in Round 3 this dipped to 50% and in Round 4 the succeeding innovations has risen to 70%. Overall, taking all the innovations funded by SWFF from its inception, 23 of the 40 innovations (58%) successfully graduate from SWFF, a lower figure than those who were active in early 2019 as mentioned above. These numbers clearly demonstrate the likelihood that SWFF-supported innovations will have a sustained market presence, i.e. being both financially and socially sustainable. The subsequent development of these graduates is positive; indicating that, 2-3 post￾SWFF they are not only are all still operational but most are durable and profitable into the future. The continuous effort made by SWFF throughout the length of the program to actively learn and develop the apparatus for monitoring, supporting and intervening to ensure progress by each innovation or to terminate support definitely paid off. The Innovation Investment Advisory Committee (IIAC) brought multi-disciplinary expert opinion into decisions on successful applicants and, where necessary, advised the termination of support for an innovation. 97 The evidence indicates that lack of progress towards targets is evident early in an award: of the 15 alumni, 13 did not receive support after their first year after the IIAC determined they were not progressing in milestones and targets. This early termination is possibly more decisive than in the experience of venture capital start-ups and reflects probably concern over the responsible use of public funds. Some interviewees, however, felt that a 10-year period might be more appropriate for the measure of a successful innovation. However, SWFF did not have such resources. Since decisions were based on rigorous examination by the IIAC of each innovation on the evidence of performance, there was some detachment of TAF itself and the Founding Partners from these decisions. While SWFF support being terminated did not mean that innovations were failures and died, many continued with other support or, on a smaller scale, kept the idea of the innovation alive. The alumni continued in the SWFF fold and were invited to conferences and often retained and engaged the country and international networks they had built as awardees. SWFF did not rush to judgment in cases where targets were not met; TAF and the IIAC gave close attention to the obstacles which could cause a lack of performance which have been found to include a compressed period for implementation before an agricultural season, complications in receipt of funds, adverse weather conditions, misunderstandings about timelines and the definition of customer/end-user, delays occasioned by an environmental assessment, and legal disputes over ownership of a patent. From the evidence of visits, it is difficult to develop a typology of these obstacles although some interviewees are vocal in mentioning the disparity between the USAID financial year (September-October) and the agricultural seasons in developing and emerging economies (in India roughly July-June) which, it was implied, operated to the disadvantage of developing and emerging economies innovations. Analysis of post-SWFF innovations To get a longer view on the progress and sustainability of SWFF projects a study has been made of continuing demand, affordability, user commitment and financial sustainability. Use has been made of the number of performance evaluations funded by SWFF to gather the data on these indicators and to classify graduate innovations in terms of their financial sustainability. A full report on this analysis is included in the ANNEX X below. A summary is presented here. In the section below the table with supporting evidence is reduced to a summary characteristic in each column: High, Moderate, or Low; followed by an assessment of profitability and durability. Table 52. Analysis of Affordability, Use and Financial Sustainability Affordability/Will ingness to pay Commitment to use/promotion Profitability/durability AQYSTA HIGH HIGH DURABLE/MEETING COSTS AST HIGH/MODERATE HIGH BEYOND DURABLE/PROFITABLE AYBAR HIGH/MODERATE HIGH DURABLE/CONTINUING 98 Affordability/Will ingness to pay Commitment to use/promotion Profitability/durability FUTURE WATER LOW LOW/NOT WILLING TO PAY NOT DURABLE/UNSURE GREEN HEAT HIGH/MODERATE HIGH/PROMOTE BEYOND DURABLE/PROFITABLE MNP HIGH HIGH/MODERATE BEYOND DURABLE/PROFITABLE WGI MODERATE/AVAILA BLE FREE HIGH/IMPROVED CREDIT DURABLE/NOT PROFITABLE WORLD HOPE LOW HIGH VERY UNSURE Source: Performance evaluations, the Social Rate of Return report and SWFF Annual Report, 2019. The analysis of affordability, commitment and financial sustainability has been undertaken. The higher level of affordability (Aqysta and MNP) and high/moderate levels (WGI; AST, Aybar and Green Heat) are scored from high commitment to use and willingness to pay. In some innovations like Green Heat there is relatively limited access to the biogas tank infrastructure but the use is high. Unfortunately, there is also evidence of the destruction of a proportion of the installed infrastructure. Low levels of affordability (World Hope and Future Water) are linked to uncertain or no revenue stream and are linked to unsure or very unsure financial sustainability. Virtually all innovations score continued commitment to use and promote the innovation among other farmers (Aqysta, Aybar, Green Heat, MNP, World Hope, and WGI). The only exception is Future Water which has a low commitment to use. Free access to the innovation is not necessarily linked to high levels of use or willingness to pay in the future, as indicated by Future Water. In relation to financial sustainability, the following innovations were found to be both durable and profitable: AST, Green Heat and MNP. Another group were durable but not necessarily profitable: Aybar and Aqysta. Finally, there were three with unsure financial sustainability (World Hope, WGI and Future Water) were also found to unsure durability and profitability (despite still being operational 2 or 3 years after receiving SWFF funding). The general conclusion that can be drawn is that with some exceptions, those innovations which have graduated from SWFF, that is completed 3 years receiving funding and technical assistance, have continued operating 2-3 years after this support. Virtually all have attracted and held the commitment of users to the innovation and most have further expanded their user base. A number have continued as durable and profitable innovations, a further section as durable and meeting their costs but not necessarily profitable. The concluding 3 are assessed to be of unsure financial sustainability; largely, it appears, because of a low willingness to pay. Further evidence on sustainability is available from analysis of the online impact survey. Recognition of these obstacles has led to changing practices such as making no-cost extensions and other measures to keep innovations from joining the alumni group. At times, the advice of the expert IIAC was heard but heeded as in the case of no cost extensions. 99 The financial sustainability of innovations in their ability at securing funds from various sources and at different stages of their development is presented below. Given the complexity of the data, years (recorded over 5 years) have been aggregated into 3 different development stages: Growth, Midpoint and Maturity. Graphic 1. Funding sources over Innovations’ Development Source: Online Impact Survey 2019 Note: Percentages represent number of innovations reporting access by type by year of intervention, not by total funding accessed. At all stages of the development of innovations from Year 1 to Year 5, for both SWFF and non￾SWFF innovations, there is a combination of three main sources of funding (Public, Private and Self). While there is no striking difference between SWFF and non-SWFF innovations in the growth period and midpoint; there is however divergence in the “mature” phase (Years 4 and 5). SWFF innovations showed greater use of public funds and lower use of Private and own sources of funding. This is not unexpected as SWFF support itself is a form of public funding. In addition, although the difference in the extent of these various types of funding at each development phase is evident, non-SWFF innovations tend to invest a higher proportion of personal funds in their innovations than SWFF’s. Moreover, SWFF innovations tend to obtain, over their expansion, higher proportion of public funds. It can be concluded that participation in SWFF and support from TAF made SWFF innovations more skillful at securing public funding in their maturity phase. This could be an indication of the social and environmental sustainability of SWFF funded innovations in the sense that allocation of public funding could equal collective agreement on relevance for communities, adoption of innovations by an audience larger than the intended/targeted communities, and confidence in the overall impact of innovations in communities’ lives. 0% 10% 20% 30% 40% 50% 60% Public Private Self Public Private Self Public Private Self Growth (Y1+Y2) Midpoint (Y3) Maturity (Y4+5) Funding Sources over Innovation's Development SWFF Non-SWFF 100 The social sustainability of an innovation in its gender dimensions was also evaluated: in markets, in participation, and in the management and ownership of innovations. Using KII data source and the online surveys, every innovation records SWFF’s clear contribution to their incorporating gender relations into their planning, reporting and activities. The supporting rationale for undertaking gender is presented both as a conjunctural imperative and as a strategic advantage. SWFF believed that “reaching the untapped markets of women smallholder farmers in emerging economies with yield increasing or labor-saving technology is an idea whose time has come. By tackling low productivity barriers that prevent women from being empowered on the farm, you can help unleash the potential of farming to be a leading driver of economic growth and food security, with women at the center of it all… Treating women and the vulnerable as equal to men is good for societies and good for business” 20. SWFF also noted the advantages that gender integration in innovations provides in mixed teams being more innovative, in opening new segments through marketing and making for better performance and productivity gains.21 There is opportunity as well as the challenge of broadening the base of the market for products and services. The KII with innovations all accord high recognition of SWFF’s prioritization of gender integration and a growing awareness of the need to achieve more equitable gender relations. At the same time, while there is growing awareness there is evidence of a considerable lag in the innovation’s responses to what has been termed the “hidden market” of latent or pent up demand from women farmers. This lag results in a loss in net social welfare through the ineffective market allocation of goods and services. The responses in interviews reflect considerable unevenness. Most integration (achieving the highest levels of women participation) occurs at the customer level, there is considerable improvement in participation in management but a very uneven participation in ownership. This polarization is evident in the uneven spread at the high level and low level of participation, particularly in management and ownership. In Table 53 (women’s participation as customers, managers, and owners), 38% of innovators record women participating as customers at a high level and equally 38% report a low level of customers as women. There appears to be either market segmentation or undue attention being paid to the male customer. 20 SWFF. August 2016. Gender report. 21 Sida. 2015. Gender workshop presentation. 101 Table 53. Women’s participation as customers, managers, and owners Women’s participation, customers, N=24 Women participating as customers, high level 9 (38%) Women participating as customers, improving 6 (25%) Women participating as customers, low level 9 (38%) Women’s participation, management, N=24 Women participating at management level; achieved/high level 7 (29%) Women participating at management level; improving 11 (46%) Women participating in business; none, low level, unsure 6 (25%) Women’s participation, business ownership, N=24 Women participating in business, highly integrated 9 (38%) Women participating in business, improving 7 (29%) Women participating in business, low level 8 (33%) Source: Analysis of coding from KII Note: Due to rounding off, the addition of all percentages may not equal 100%. Analysis of responses on management and ownership show a somewhat similar dichotomy between high and low levels of participation but with two important differences. The first is that the highest level of in women's participation appears in management (46%) and a significant proportion of innovations (38%) have achieved a high level of integration in ownership. This may indicate greater women’s participation than in comparable GCF but the 33% of innovations with “low level ownership” and the 29% of “improving” indicate the remaining challenge. The analysis indicates that most of the innovations still have considerable challenges in achieving gender integration particularly in management and ownership. The social sustainability of SWFF and Non-SWFF innovations in their gender integration in the composition (staffing) of their organization is presented in Table 54 (Demographics of innovations, gender, SWFF/NON-SWFF). Table 54. Demographics of innovations, gender, SWFF/non-SWFF SWFF non-SWFF Average per innovation % of total Average per innovation % of total Total Total 46 287 Men Average # of staff/average % composition 30 65% 256 89% N=96 Women Average 16 35% 31 11% N=93 102 SWFF non-SWFF Average per innovation % of total Average per innovation % of total Total Total 46 287 Female Executive Average (% of total staff) 3 7% 6 3% (% of total females) 19% 19% N=92 Source: SWFF Impact Survey 2019, How many people are in your organization (innovation/country 1) Note: Women in management is calculated as a proportion of all women. In relation to the composition of those engaged in innovations (“people in your organization”), it is notable that SWFF innovations report a higher proportion of men in the organizations than non-SWFF innovations. These gender distinctions about the composition of the engaged may be relatively neutral, possibly reflecting the gender composition of farmers in these areas. There is, however, an indication of SWFF innovations achievement in having a somewhat higher percentage of women in senior positions of all engaged women, compared to those in non-SWFF innovations. Further study is needed to continuously review the data to unpack the gender dynamic within innovations and, particularly, the progress of women into management and executive positions. Clearly SWFF has deep commitment to identifying and resolving challenges and identifying failures substantial material on the obstacles arising in growth and other issues which cause an innovation not to perform according to expectations. The svery careful vetting around the likelihood of success for applicants and the milestones and targets that kick enterprises out when they start to falter provide a number of points in time for pivots to be made. The record of failure provides an invaluable stock of well documented knowledge. Despite difficulties of disclosure and of complexity, this can be mined as a record or manual of successful and unsuccessful engagement with obstacles. This is being undertaken. Encountering failure has been a sign of the higher level of risk that SWFF has been prepared to undertake; the lack of failure would have indicated both low expectations and a conservative approach. While venture capitalists shun discussion of failure22 , in a public body, diagnosis and strategic thinking about failure is invaluable to necessary learning. Boldly in this spirit, SWFF’s Failures, Pivots, and Lessons Learned Report23 , provides a close analysis and examination of identified failures in program coordination, acceleration facilitation, grants, and finances, and monitoring and evaluation. The pivots propose corrective actions taken by SWFF and can be taken as lessons learned for future Grand Challenge Funds, including WE4F. 22 Venture capitalists "bury their dead very quietly," Mr. Ghosh says. "They emphasize the successes, but they don't talk about the failures at all." The Venture Capital Secret: 3 Out of 4 Start-Ups Fail – WSJ https://www.wsj.com/articles/SB10000872396390443720204578004980476429190 23 SWFF Failures, Pivots, and Lessons Learned Report, December 2014 – December 2019 103 FINDING: There is clear evidence of a high likelihood that SWFF-supported innovations will have a sustained market presence (i.e. financially and socially sustainable). Although the SWFF’s evaluation process which measures the overall success of an innovation was deemed at times rigorous, exacting and even overwhelming by innovations, it revealed necessary changes. Some of these are incremental, others necessarily more decisive. It is regarded as helping to approach sustainably with more rounded indicators such as gender integration and rigorous financial indicators such as marketability. The external perspective from IIAC as well as openness to flexibility from SWFF proved beneficial in the objectively examining promising innovations. The Failures/Pivot Report indicates the willingness of SWFF to learn both from success and failure in a methodical and well-structured way. The identified failures range across the program itself to the details of innovations and provide a feedback loop which provides an active rectification and fresh turns through the pivots and results. RECOMMENDATION: In future endeavors such as WE4F the spirit of inquiry, openness, identifying failure should lead on to continuous improvement. The adoption of the Failures/Pivot Report shows there is the possibility of drawing on the accumulated documentation and review to long term advantage. This approach has which has shorter or quicker feedback loops than that of a midterm or final monitoring and evaluation report, provides a solid foundation for similar programs. Future programs should continue to capitalize on the experience gained from SWFF to build and keep momentum to achieve greater success in the sustainability (in the dimensions of gender, institutional stability, finance, technology and other indicators) of funded innovations by: • Developing, in collaboration with innovations, novel funding strategies adaptable in the functioning environment of emerging economies. These could include a combination of private and public funds to offset the potential loss that will arise in the attempt to make innovations equally affordable to all members of the communities. Examples of various sources of funding to consider include: subsidization, crowdfunding, grants, donations, private and public investments, corporate matching-gifts, non-cash donations and peer￾to-peer loans. In addition, the incorporation of innovations into agricultural extension services (as a number of innovations in India are considering) needs to be reviewed. • Identifying the most promising innovations in terms of 1) gender inclusion, growth, and marketability and 2) key components that are at the nexus of water, food and energy to continue to support in WE4F. Such innovations should already demonstrate their potential. • Working with program funded innovations at developing strategies and skills to secure a higher proportion of a range of private funds to match the start-up public funding to secure their expansion. 104 Question 12. Are SWFF-supported projects environmentally sustainable (i.e., did they provide positive environmental benefit, or did they do more environmental harm than good)? The SWFF portfolio encompasses a wide variety of innovations and contexts. For the vast majority of interviewees, no significant environmental considerations were noted, and several are actively improving the environment. In support of the agricultural gains for farmers, many innovations introduce more environmentally friendly farming practices. One interviewee stated that, “Many of the innovators already have an ecosystem-environmental consciousness and [SWFF] further emphasizes this.” As a result, in some instances, the innovators appear to have become conduits for environmental advocacy and wider environmental awareness. Amongst others, environmental benefits include: • More efficient use of water, • Reduced reliance on groundwater, • Reuse of greywater and other soil conditioners, • Changes in crops to less water-reliant varieties, • Reduced contamination of drinking water, • Reduced desertification, and • Reduced or at least more targeted use of pesticides and chemical fertilizers alongside training on pest management and improving biodiversity. Negative impacts (see Annex sections on Sustainability), when identified by respondents, were all considered “minor” negative impacts. No major negative impacts have been reported. Negative impacts include increased burden on women, increased debt levels, and creation of conflict within family about gender role or creation of conflict within the community on the order of use of innovation or jealousy from other members of the community who don’t have access to the innovations or who cannot afford them. Examination of negative impacts on women and poor show no significant divergence from the impact on all participants. Although regarded as minor, the negative impacts are worth reviewing. For WGI these are minor and include increased burden on women, increase debt levels and potential conflict within family about gender roles. For Lal Teer, they are also declared as minor: increased burden on women’s workload and created conflict with buyers. Practical Action records the same issues. For Green Heat the impacts are all minor: increased burden on women and create conflict within family about gender role. There is evidence of the environmental sustainability of SWFF-supported projects, albeit more explicit for some innovations than for others. Agricultural policy in many countries, regions and localities is tending towards establishing organic agricultural practices. In India, nine states— Karnataka, Mizoram, Kerala, Andhra Pradesh, Himachal Pradesh, Madhya Pradesh, Tamil Nadu, Maharashtra and Gujarat—have adopted organic farming policy or law. These policies reduce or terminate the subsidy or supply of fertilizers and pesticides to farmers. Several innovations are engaged in the establishment of organic farming by promoting their innovation to reduce fertilizers and pesticides. These innovators are committed particularly to maintaining biodiversity, improving soil fertility, and encouraging ecological pest management. 105 Some innovations make a direct contribution to an improved environment. Take the case of WASTE which processes both fecal sludge and organic solid waste with co-composting and sells this fertilizer to local farmers. This facility has provided the “honeysuckers” which had previously spewed the sludge into the forests at night with a ready alternative considerably improving the natural environment and water quality. For many innovators, there may also be particular deforestation considerations due to more land being under cultivation. This brings resource use and resource management aspects more into view. Thus, overall water abstraction for farming may in fact have increased even while significant water efficiencies result directly from the introduction of the innovation or related support to farmers around more efficient irrigation technologies and practices. There is significant confidence in USAID’s scrutiny around environmental conditions and several innovators noted that USAID had supported them in conducting an environmental impact assessment. Ultimately, the environmental focus of the IIAC and USAID’s own restrictions and on-the-ground checks (also with the help of a USAID/LAB Environment Officer) have helped to flag potential environmental issues. The evaluation team suspects that these could be captured more clearly in an overview analysis of the SWFF portfolio’s wider environmental impacts. For some innovators, such as those employing sensors, there may be a need for Life Cycle Analysis assessments for the products to also understand end of life / disposal issues. FINDING: Negative impacts are not linked to environmental issues and regarded as minor; they include issues such as changed gender roles and increased burden on women. Among many innovators there is a commitment to green technologies, organic pesticide-free agriculture, and improved soil quality. As well as this commitment, several products (such as NewSil) contribute to reducing or eliminating fertilizers and pesticides. There is a keen f among innovators to achieve organic farming through the technologies they promote. Unintended negative impacts arise from the use of the innovations, but they are mainly reported as minor. RECOMMENDATION: Governments are increasingly interested in declaring agricultural zones or entire states as organic. Future programs should consider such a target in selection criteria. WE4F should monitor closely as well unintended negative impacts and work at mitigating them as soon as they arise. CONCLUSIONS, AND RECOMMENDATIONS SECTION 1: RELEVANCE Question 1: Technically, what types of innovations have been supported? How many innovations have been awarded funding? What are the basic demographics/ descriptors of innovators? (H2) FINDING: In terms of SWFF’s relevance, much of the current development paradigm revolves around tailored or adapted market responses to lifting the poor out of poverty. Emphasizing 106 access for marginalized and vulnerable populations has proven to be a constant thread of the SWFF program, with innovations largely selected as a function of their appropriateness for poorer farmers in increasingly water-stressed environments. The portfolio of SWFF innovations score comparatively poorly in Green Energy and Improved access to Water. SWFF has successfully transitioned to having a majority of local innovations. The balance between male and female led though remains heavily in favor of male-led enterprises. While the trends in the portfolio at application stage are not clear, several enterprises are making the transition from non-profit to for-profit. RECOMMENDATIONS: A greater emphasis is needed on Green Energy in WE4F. Going forward, WE4F should continue to encourage women led innovations to apply and make the proportion of women in management a criterion for selection to provide additional points in the selection process. WE4F should, from inception, follow the portfolio of awardees to understand how shifts in the business model (and registration status) are changing the willingness to or emphasis on reaching the poor and marginalized. Are these businesses using cross-subsidies or other mechanisms to continue to reach these target groups? Question 2. Is there demand and local ownership for the SWFF innovations (individually and across all innovations)? (H2) FINDING: SWFF has made progress in further localizing its support to awardees. There is widespread local need in developing and emerging economies for the innovations and potential for local ownership. The challenge is the ability to pay. Innovators are generating effective demand as well as meeting existing demand from better off farmers for their infrastructure, products and services. The ability to pay in the local context often includes the availability of inexpensive locally available financing options for poor or very poor farmers. RECOMMENDATION: Further efforts to localize WE4F support would involve greater engagement with funder embassies and other donor programs with NGOs operating in related activities. Rather than treating awardees as operating in isolation, this could create synergies among funded activities allowing for greater visibility, reach and impact. Embassies and other donors are likely to also be able to make the links to policy and national government programs that would support the awardees in positioning and marketing their innovation in the local context. More effort could be placed on mapping the kinds of partnerships that would help innovators strengthen their offering and expand their customer base. For example, WE4F should consider the most effective ways of partnering with or incorporating programs that offer locally available low-interest finance without over-burdening the innovations. SECTION 2: EFFECTIVENESS Question 7. To what extent did the SWFF TA Facility provide SWFF innovators with timely and appropriate technical assistance that led to the creation or enhancing of 107 a viable business centered around an innovation that saved water / generated more agricultural product? FINDING: The SWFF support structure is understood by innovators as helping them advance towards viability, and the support is widely acknowledged as critical to an innovation’s uptake by users. The constraints of working in a tight system of planning, targeting and reporting are less than the expansion of horizons and capacity development. The country visits are highly appreciated and can lead to the clarification of key aspects of targeting and reporting. RECOMMENDATION: None Question 7a. Does the SWFF TA support help the awardee to overcome organizational capacity barriers? (These could include strategy development, financial forecasting capabilities, improved manufacturing approaches or product design, barriers to accessing funding, etc.). FINDING: SWFF support is focused on helping innovations to scale up and meet targets; the vending system has been considerably improved to meet innovators’ needs but some innovators would like to explore further the engagement of TA support providers to be more flexible and useful. By years 4 and 5 in the SWFF program, there is a reduction in private funding and an increase in public funding, compared to non-SWFF innovators, potentially highlighting the need for a review of barriers to diversifying funding sources prior to SWFF ending. RECOMMENDATION: More TA support providers from developing and emerging economies should be listed and other aspects of choice explored such as contributions from innovators themselves to match funds to engage TA support providers. There should be a strengthened focus in the WE4F program on identifying barriers to diversifying funding sources and supporting innovations in access to private capital or income-generation (through directly related revenue or other fundraising) prior to SWFF ending. Given the existence of many synergistic or related USAID and other Founding Partner programs, projects, and activities in the countries in which these innovations operate more can be done to identify and coordinate with existing programs that work in areas of low-interest finance, private sector engagement, women’s saving groups, agricultural and market systems development, and gender. Given the current operating environment, there may be changes post-COVID that need to be explored. Question 7b. Did SWFF TAF enable certain (new) (combinations of) expertise to be deployed that would otherwise likely not have been deployed/used (by the individual partners)? FINDING: Finding the appropriate combination of expertise in support of an innovation is intrinsically difficult; while most feel SWFF has helped decisively, some innovators want to further explore ways of prioritizing their needs prior to engaging TA. RECOMMENDATION: Further experimentation with the vendor system in matching funds or another funding mechanism would be rewarded. 108 Question 7c. Were there additional barriers that were not addressed by the SWFF TAF? FINDING: Additional barriers are reported by innovations but not all expected a resolution from SWFF itself. Some barriers (such as the lack of human resources) need longer time and iterative work to overcome. RECOMMENDATION: WE4F should further engage with innovators to list the barriers experienced and undertake online exchanges on the most common. Question 7d. Were there positive or negative unintended consequences of the support? FINDING: From the interviews SWFF support has mostly positive and not negative unintended consequences directly for the innovations or in a wider social context. An unintended negative consequence is that support has not been extended; another consequence of good performance may be continued dependence on available assistance. RECOMMENDATIONS: None. Question 7e. To what extent did the SWFF TA Facility provide SWFF innovators with technical assistance that innovators deemed useful? FINDING: Technical Assistance from service providers as well as direct support from the TAF are both key features of the SWFF design and are highly appreciated; many innovators consider it more important than the actual funding. Possibly because of its importance there is some concern to expand its parameters where possible. RECOMMENDATION: WE4F should use demand-driven ttechnical aassistance as well as support and training based on identified business capacity gaps to address them should be made available in the new initiative. Question 8. How effectively have investment risks been managed by the program? (number of failed projects, timeliness of reaction on problems observed etc.) (H3) Has the milestones-based tiered grant structure led SWFF to continue funding only the most promising innovations over time? (H3) FINDING: As explained above, The Program is effectively run, ensures close attention to detail, and has a hands-on approach with the awardees. SWFF innovations have a more sustainable period of commercialization, even if it takes them longer on the innovation runway. The evaluation team finds that the SWFF risk has been effectively managed by the program in its selection of innovations, their tiering of funding, and their ultimate sustainability in success. RECOMMENDATION: WE4F should foster more exchange of information on the barriers that innovators are facing and efforts to overcome them. Initiating wider links with related embassy and other programs would further enhance SWFF’s effectiveness. 109 Question 9. How well did founding partners interact within SWFF and what lessons should be taken from their interactions? FINDING: The founding partners bring complementary agendas to the SWFF partnership, which meets the various needs and capacities of each agency. USAID’s effective management of the GC is highly regarded. RECOMMENDATION: Greater emphasis should be given to broadening developing and emerging economies membership base of the Founding Partners in a new initiative and linking these partners more effectively to the Program. Such partners could help locate and support suitable innovators particularly from emerging and developing economies. Prospects with regional organizations in Africa and Asia should be explored. SECTION 3: EFFICIENCY Question 10. To what extent were the SWFF results to date in balance with the level of effort and resources (funds, human resources including by the FPs, TAF, interns, consultants)? FINDING: Respondents across the board recognize the higher costs and levels of effort required to administer SWFF, but overwhelmingly suggest that the resultant impacts from the attention to detail and tailored support are in line with or well exceed the costs. Question 10a. To what extent was the TA Facility efficiently set up, organized and managed? FINDING: At no point during the evaluation did questions emerge around the efficiency or effective management of the TAF to deliver support to SWFF awardees and respond to requests from the founding partners or IIAC. Indeed, the TAF is widely appreciated as providing exemplary service from a small, dedicated team of professionals. Question 10b. To what extent are the administrative costs for managing SWFF above, below, or on par with the cost of similar challenge funds? (Special consideration should be made for funds that provide technical assistance to their innovators.) FINDING: SWFF is found to have a highly intensive mode of management. Its administration and fund management costs are generally higher as a function of the total value of support to awardees than other challenge funds, but such comparisons need to be considered with care. The significant level of support provided to awardees and the impacts need to be considered. Support activities are well regarded by innovators as providing valuable assistance. RECOMMENDATION: Swedish International Development Cooperation Agency (Sida), USAID and other funders should agree on a framework for cross comparison to enable a more 110 rigorous assessment of the efficiency and effectiveness of different models of intervention. A comparative review should include the impact that the support has on the enterprise and also on the customers / users. Question 10c. To what extent is the level of effort and resources spent by applicants/innovators in balance with the added value SWFF brings? FINDING: While the burden SWFF places on awardees appears to be higher than most other grand challenge programs, these same processes allow for SWFF to tailor its funding and technical support more directly to achieving the aims and objectives of the awardees as well as those of the founding partners. RECOMMENDATIONS: Comparing incubation and accelerator support for emerging businesses with challenge funds that support civil society or public sector efforts is misleading and unhelpful. If not already in process at the inception of WE4F, efforts should be made to undertake further comparative cost and impact analysis of challenge funds. The Founding Partners should, however, establish a methodology focused on consistent definition of terms, leading to comparisons of like-with-like. Further analysis and a webinar with Tripleline, the Evaluation Team and the Founding Partners would be a useful step towards identifying a unified approach and methodology. SECTION 4 IMPACT Question 3. Were solutions sourced through SWFF adopted and utilized at scale? FINDING: In interviews conducted during 2018-19 most innovations reported a plan to scale up (growth and expansion in new markets) and were able to strengthen existing partnerships and establish new partnerships with SWFF support. Subsequent developments are to be found in the Section on Effectiveness and Sustainability. RECOMMENDATION: WE4F should define realistic target goals for scaling up and develop a clear plan for strengthening existing partnerships or establishing new partnerships. Regular assessment of targets and plan should be adapted to the socio-economic environment in which innovations operate. Question 4. To what extent has SWFF contributed to the outcomes and results outlined by the indicators in the SWFF PMEP during project implementation and post project implementation? FINDING a: In relation to SWFF’s contribution to outcomes; there is a strong indication that SWFF made a decisive contribution throughout the innovation’s operational life to the outcomes and results in terms of (in this order); financial, organizational strengthening, and technical contributions. The financial contribution of SWFF was reported as playing the greatest role (64%, n= 21) in reaching outcomes and results, followed by the contribution to organizational strengthening and to technical aspects. 111 FINDING b: Customers and End-Users highly benefited from the use of the different innovations and their overall quality of life was substantially improved. There is an overall positive impact of the innovations on the respondents’ income. However, benefit from the innovations and improvement of income from the use of innovations are lower for women and poor than for all respondents. This is particularly true in the case of improved income which is substantially lower for women and poor. RECOMMENDATION a: WE4F should establish clear baselines on a well-defined set of indicators that will carefully track SWFF’s or other programs’ support and hold constant support (financial and technical) from all other donors. Set up a robust counterfactual along a set of clearly defined impact indicators to validate the contribution of WE4F and other programs to outcomes and results of funded innovations. RECOMMENDATION b: Further investigation is needed into uneven benefits from innovations for women and poor. Gender Integration should be at the forefront of innovations and programs to help reduce the systemic inequality that exists in communities. The greater inclusion of women will assist in overcoming the limitation in the opportunities of one-half of its population. In addition, attention needs to be given to poor end-users to ensure greater benefit from innovations and improved income. Question 4a. Did SWFF-supported projects increase or make water more accessible? Did SWFF projects meet their water efficiency/availability targets? Overall, across all innovators, did the program meet the water efficiency/availability targets? FINDING: Most of the SWFF-supported projects increased water access for their Customers / End-Users, some quite decisively. In some innovations (e.g. MNP) this is not an explicit goal. This is an important point which could explain a number of variances: unlike other outcomes there is a wide range of recorded impact. The more efficient use of water is a critical aspect of SWFF’s mission. In terms of water efficiency, across all innovators, decided evidence of success (as reported by users) can be seen by users in relation to a few innovations. Although detailed analysis has not been undertaken on the planned targets and results for each innovation, it may be concluded that the results in water efficiency are more modest than the expected targets for a number of innovations. RECOMMENDATION: Focused monitoring and evaluation of this indicator is needed to deepen understanding of the variance in responses. Special attention also needs to be given understanding the variance between the average and levels of water accessibility to vulnerable groups such as women and the poor. More consideration needs to be given on how to ensure innovations achieve better results in terms of improvement in water access attainment and/or improvement for communities. Possibly those innovations such as WGI (Aquaponics), Hydroponics Africa, WASTE, Skyfox and Bhungroo (when it is fully functional) which are or could be the most successful in this aspect need greater support. Given the lower levels recorded overall for water efficiency, a strong recommendation would be to develop novel methods to make quantitative estimates for this important variable for WE4F. 112 Question 4b. Did SWFF-supported projects lead to more agricultural productivity and resilience to climate change? Did SWFF projects meet their agricultural productivity targets? Overall, across all innovators, did the program meet the agricultural productivity targets? FINDING: Results established evidence of noticeable reported increases in farmers’ agricultural productivity and resilience to climate change as being a definite byproduct of the use of SWFF￾supported projects. A high level of farmers (81%) are making 2 or more changes in their farming practices. Innovations offer farmers an array of solutions that overall transcend geographical boundaries as they appear to be in one way or another a combination of the same practices. Hence, a high proportion of farmers report they are taking advantage of innovations to make two or more changes to their farming practices repertory. RECOMMENDATION: More research is needed to increase the effectiveness of SWFF innovations while using a minimal set of inputs as most farmers cannot always afford to buy the necessary inputs. WE4F needs to focus on mitigation of climate change (and thus increased resilience) as a key direct objective by integrating Climate Smart Agriculture in the design and implementation of the supported innovations. Climate-smart agriculture (CSA) is an approach that helps to guide actions needed to transform and reorient agricultural systems to effectively support development and ensure food security in a changing climate. Question 4c. Have vulnerable groups (the poor, women, ethnic minorities) been positively and/or negatively impacted (through income, employment, water/environmental) from SWFF supported innovations? Specifically, has SWFF reduced the number of people in poverty as a result of supporting SWFF-supported innovations and has SWFF increased the number of women benefitting from SWFF￾supported innovations?24 FINDING: Overall, women and vulnerable groups have been positively impacted from SWFF￾supported innovations but not to the same levels as the average impact. This is found, for example, in relation to agricultural productivity, in income, in access to low interest rate loans and in water availability. Women and the poor do not access low interest rate loans at the same level as most customers. Though these impacts are not as striking as expected, they cannot negate the stride that has been made in gender-integration by SWFF-supported innovations. More emphasis was placed round after round on integrating women at all levels of innovations, whether as Customers/End-Users or as Managers and Executives of the innovations themselves. RECOMMENDATION: Provide consistent technical support and guidance to each variation of innovator to balance both the business (profitability) and social (vulnerable groups) model side 24 This section relates gender in relation to access to the innovations and female-specific impacts; in Question 11 gender issues are explored in relation to women’s access as Customers and to participation in management and ownership of innovations. 113 of the equation to allow them to achieve both goals simultaneously and without sacrificing quality or performance. Cross subsidies need to be considered and included in the design of innovations. Gender Integration (or mainstreaming) should be a key consideration in innovation design. This strategy for integrating gender concerns in the analysis, formulation and monitoring of policies, programs and innovations to promote gender equality and the empowerment of women and the vulnerable in population and development activities should be considered for all innovations because different communities’ groups demand different approaches. Question 4d. What is the balance between public/social engagement and private/public engagement? To what extent have private funds been generated that contribute to the developmental objectives of the program both during and following SWFF awards? FINDING: SWFF-funded innovations have been able to secure different types of funding, therefore finding a balance between public/social engagement and private/public engagement. The most important sources are public funding and innovators’ own funds. Although private funds are the least prevalent source of funding, it is worth noting that the novelty of SWFF-funded innovations as well as their worthy cause appeal more and more to “Impact Investors” who are driven not only by profit but also by a social engagement towards vulnerable groups. RECOMMENDATION: Develop with innovators a customized funding strategy that will allow them to appeal to a larger and more diverse pool of funding sources particularly in expanding access to private sector funding. Innovations should not only be financially viable, for-profit entities but also socially engaged and sustainable. Some effort should also be devoted in monitoring SWFF graduates to measure continued public and private engagement in innovations and inform the design of novel and sustainable potential donors engagement strategies. Question 5. How much of the measured change (outcome or result indicator) in the SWFF program can in fact be attributed to the SWFF-supported projects? That is, what portion of the result is not explained by the projects examined by the evaluation? FINDING: Although it is difficult to assess the extent to which the measured outcomes and results can be in fact attributed to SWFF-supported projects alone, end-users/customers reported that SWFF-funded innovations made a high-level contribution in the form of overall benefit from innovations, mostly in terms of improvement in agricultural output, income, water access and availability, and resilience of crops to climate change. RECOMMENDATION: None. 114 Question 6. To what extent are there differences between the planned SWFF￾supported projects and what was actually delivered in Year 1 and then Years 2-3 of the projects? FINDING: Targets, milestones and the reporting system were all valued by most innovations. These helped facilitate a greater reach of their potential, the expansion of their market access, the improvement of their business model or the redesign of the whole innovation when needed. They also improved the innovations’ national and international credibility and better their internal practices leading to viability. RECOMMENDATION: Develop a standard set of targets, milestones and reporting system that could however be adapted to specific innovations as needed. Overall, results show that results for women and poor are lower than the average for all respondents. The reasons for this variance need to be explored. Apart from the key aspect of access to land and other resources there are those of education and social emancipation. Women may also prioritize production for domestic consumption rather for markets. A gendered division of labor and allocation of assets may affect the accurate estimation of variables such as income or agricultural production. OVERALL FINDINGS: SWFF strongly contributed to outcomes and results. Impacts in terms of improvement in agricultural output, income, water efficiency and in the use of inputs (labor, time, pesticides, fertilizers, etc.) are reported. Impact of innovations on adaptation and resilience in the face climate change is more evident through greater adaptation and preparedness. A high level of farmers (81%) made 2 or more changes in their farming practices. These practices are embedded in better adaptation to climate change due to a greater availability of water, better information in adaptation to rainfall agriculture, targeted irrigations, and appropriate farming practices. Targets, milestones, and reporting system were valued by most innovations. They facilitated the greater reach of their potential, the expansion of their market access, the improvement of their business model or the redesign of the whole innovation when needed. They also improved the innovations’ national and international credibility and better their internal practices leading to viability. OVERALL RECOMMENDATIONS: Gender integration should be at the forefront of innovations and programs to help reduce the systemic inequality that exists in communities. The greater inclusion of women will assist in overcoming the limitation in the opportunities of one￾half of its population. At the inception of a new program baselines and robust counterfactual along a set of clearly defined impact indicators need to be established. These will constitute the definitions and data to validate the contribution of WE4F and other programs to outcomes and results of funded innovations. 115 A focus on the adaptation and mitigation of climate change as a key direct objective of future programs, and therefore the resilience to climate change, is proposed by integrating Climate Smart Agriculture in the design and implementation of the supported innovations. Consistent technical support and guidance is needed for each innovator to balance both the business (profitability) and social (vulnerable groups) model to achieve impact which is evenly shared between men and women and better off and poor. Section 5: Sustainability Question 11. What is the likelihood that SWFF-supported innovations will have a sustained market presence (i.e. are financially and socially sustainable)? FINDING: There is clear evidence of a high likelihood that SWFF-supported innovations will have a sustained market presence (i.e. financially and socially sustainable). Although the SWFF’s evaluation process which measures the overall success of an innovation was deemed at times rigorous, exacting and even overwhelming by innovations, it revealed necessary changes. Some of these are incremental, others necessarily more decisive. It is regarded as helping to approach sustainably with more rounded indicators such as gender integration and rigorous financial indicators such as marketability. The external perspective from IIAC as well as openness to flexibility from SWFF proved beneficial in the objectively examining promising innovations. The Failures/Pivot Report indicates the willingness of SWFF to learn both from success and failure in a methodical and well structured way. The identified failures range across the program itself to the details of innovations and provide a feedback loop which provides an active rectification and fresh turns through the pivots and results. RECOMMENDATION: In future endeavors such as WE4F the spirit of inquiry, openness, identifying failure should lead on to continuous improvement. The adoption of the Failures/Pivot Report shows there is the possibility of drawing on the accumulated documentation and review to long term advantage. This approach has which has shorter or quicker feedback loops than that of a midterm or final monitoring and evaluation report, provides a solid foundation for similar programs. ss Future programs should continue to capitalize on the experience gained from SWFF to build and keep momentum to achieve greater success in the sustainability (in the dimensions of gender, institutional stability, finance, technology and other indicators) of funded innovations by: • Developing, in collaboration with innovations, novel funding strategies adaptable in the functioning environment of emerging economies. These could include a combination of private and public funds to offset the potential loss that will arise in the attempt to make innovations equally affordable to all members of the communities. Examples of various sources of funding to consider include: subsidization, crowdfunding, grants, donations, private and public investments, corporate matching-gifts, non-cash donations and peer- 116 to-peer loans. In addition the incorporation of innovations into agricultural extension services (as a number of innovations in India are considering) needs to be reviewed. • Identifying the most promising innovations in terms of 1) gender inclusion, growth, and marketability and 2) key components that are at the nexus of water, food and energy to continue to support in WE4F. Such innovations should already demonstrate their potential. • Working with program funded innovations at developing strategies and skills to secure a higher proportion of a range of private funds to match the start-up public funding to secure their expansion. Question 12. Are SWFF-supported projects environmentally sustainable (i.e., did they provide positive environmental benefit, or did they do more environmental harm than good)? FINDING: Negative impacts are not linked to environmental issues and regarded as minor; they include issues such as changed gender roles and increased burden on women. Among many innovators there is a commitment to green technologies, organic pesticide-free agriculture, and improved soil quality. As well as this commitment, several products (such as NewSil) contribute to reducing or eliminating fertilizers and pesticides. There is a keen interest among innovators to achieve organic farming through the technologies they promote. Unintended negative impacts arise from the use of the innovations, but they are mainly reported as minor. RECOMMENDATION: Governments are increasingly interested in declaring agricultural zones or entire states as organic. Future programs should consider such a target in selection criteria. WE4F should monitor closely as well unintended negative impacts and work at mitigating them as soon as they arise. 117 ANNEX 1: EVALUATION STATEMENT OF WORK STATEMENT OF WORK Performance Evaluation of Securing Water for Food Grand Challenge for Development I. PURPOSE OF THE EVALUATION II. SUMMARY INFORMATION Strategy/Project/Activity Name Securing Water for Food: A Grand Challenge for Development Implementer USAID; The Kaizen Company; various awardees Cooperative Agreement/Contract # USAID Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $35M Life of Strategy, Project, or Activity Sept. 2013- May 2020 Active Geographic Regions Global; Projects in 38 total countries Development Objective(s) (DOs) LAB 1.1 Number of Beneficiaries of LAB-Supported Innovations III. BACKGROUND Four founding partners have embarked on the Securing Water for Food: A Grand Challenge for Development (SWFF) program: USAID, Sweden through the Swedish International Development Cooperation Agency (Sida), Ministry of Foreign Affairs of the Kingdom of the Netherlands (MFA-NL), and the South African Department of Science and Technology (DST). SWFF is an innovation and acceleration initiative that aims to source, select and accelerate innovations that will enable the production of more food with less water. The goal of Securing Water for Food: A Grand Challenge for Development is a multi-donor funding program that aims to produce more food using less water, or make more water available for food by supporting innovations in developing countries. As stated in the partner’s program approval document of SWFF, a final program evaluation is required for Securing Water for Food: A Grand Challenge for Development. SWFF already had a mid-term evaluation that is publicly available on the SWFF website and in the USAID Development Exchange Clearinghouse. The final performance/ pseudo-impact evaluation should assess progress in the achievement of the objectives of the program and distill lessons on strengths and weaknesses of SWFF. The evaluation will also to a certain extent examine the efficiency of the SWFF program’s use of funding as well as the sustainability of SWFF innovations. The final program evaluation will inform policy and design decisions of later Grand Challenges for Development as well as to the evidence available on the US Global Development LAB’s portfolio of innovations. 118 SWFF is to source, incubate and accelerate high-potential technical solutions and/or business models that find new and sustain existing water supplies as well as lower overall water demands in the food value chain to reduce water scarcity and poverty. SWFF innovators scale innovations designed to support the program’s overall goal. The focus areas of SWFF were chosen after months of extensive research and discussions involving a broad range of experts, and include: a) water efficiency and reuse, especially targeted at the food value chain; b) water capture and storage, in particular in regions where rain occurs at limited times; and c) saltwater intrusion, especially in coastal aquifers or deltas and estuaries. The innovations in these areas could include (but were not limited to) improved technologies for irrigation, real-time water quantity monitoring, post-harvest water demand reduction, salinity reduction and water reuse/efficiency storage activities within the food value chain. SWFF also seeks to support business and financial innovations that enable the increased adoption and dissemination of science and technology solutions. The SWFF Founding Partners (USAID, Swedish International Development Cooperation Agency (Sida), DST and MFA-NL) share the common goal of advancing international development through improved access to sustainable water sources for agricultural applications. Values that are common to the Founding Partners include sustainability, efficiency, sourcing of market-based solutions, gender inclusion, climate adaptation and mitigation, benefit to poor people, and a commitment to avoid negative effects (particularly with regard to water-related services). The SWFF Founding Partners recognize that often 100 investments are made in the private sector to get one successful innovation to wide-scale adoption. SWFF aims to have at least 10-20% of the innovations it supports reach some level of wider-scale adoption (greater than 10,000 customers/end-users). In order to do so, USAID (the lead program implementer) has contracted the SWFF Technical Assistance (TA) Facility to provide a number of the TA services and assist USAID in the implementation and help innovators accelerate their progress in order to reach wider-scale adoption. The Innovation Investment Advisory Committee (IIAC) provides advice on selection of innovators, assessment of progress and determination of benchmarks to make sure that the Founding Partners support those innovations that most likely will achieve success. To date, four calls for proposals have been announced and selection of innovators completed. Description of the Problem, Development Hypothesis(es), and Theory of Change Through SWFF, the Founding Partners hope to source and accelerate high potential solutions that will have multiplier effects at various levels of a country’s economy. The following three hypotheses are both meaningful and practical measurements of SWFF’s development impact potential: 1. By investing in science and technology innovations at the water and agricultural nexus, the pace of development in both sectors will be substantially faster than if we relied on “traditional” development programming alone. The basis for this hypothesis is that science and technology play key roles in creating economic growth opportunities through entrepreneurism, investment, research and development, partnership, technology 119 commercialization, and widespread technological adoption – including adoption by the poor. Adoption of SWFF innovations will either have direct economic benefit on end users by increasing efficiency and/or profitability, or indirect economic benefit by improving food security writ large. At this point, we have been comparing some key metrics (e.g. USD spent per customer/end-user) comparing SWFF to “traditional” development programming. 2. By sourcing technologies and business model innovations that have already demonstrated potential at the pilot stage, Tier 2 SWFF-supported innovators have greater likelihood of being brought to scale (reaching at least 1 million people) and Tier 1 SWFF-supported innovators have a greater likelihood of reaching wider scale adoption (reaching more than 10,000 customers/end-users). The basis for this hypothesis is that a sufficiently large number of proven technologies already exist, and many are already on the market. These technologies require adaptation and/or validation for local markets and/or in-depth support for wide scale growth and distribution. Early stage innovations require a higher level of support, testing, and – most importantly – time. Based on our analysis, SWFF investments are best placed at the post pilot stage. In addition, a lesson learned from previous Grand Challenges, only 10-20% of innovations supported from previous Grand Challenges have strong potential for wide-scale adoption, even after having been taken through a rigorous evaluation process. Using this lesson learned, SWFF has designed a milestone-based tiered grant structure so that we only continue to fund the most promising innovations over time. SWFF has undertaken a survey with 3 groups (SWFF innovators that received 2+ years of funding; SWFF innovators that received 1 year of funding; SWFF non-innovators) to help determine whether or not there is evidence to support this hypothesis. 3. By investing in acceleration-oriented technical assistance and facilitating partnerships, we will substantially increase the likelihood that innovators will have the knowledge, tools, and resources to bring their innovations to scale. The basis for this hypothesis is that grant financing alone will not be enough to bring any innovation to scale. Partnerships with the private sector, government, NGOs (for distribution), and others are necessary to (among other things) accelerate business-to-business linkages, catalyze investment, improve distribution, and ultimately stimulate adoption. SWFF has undertaken a survey with 3 groups (SWFF innovators that received 2+ years of funding; SWFF innovators that received 1 year of funding; SWFF non-innovators) to help determine whether or not there is evidence to support this hypothesis. Summary Strategy/Project/Activity/Intervention to be evaluated The SWFF program will in the end have supported at least 40 projects in 30 eligible OESO/DAC 1-4 countries. Expected outcomes of SWFF: 4. At least 8 proposals/innovations that improve water availability and efficiency in the 120 food chain have been adopted, brought to scale and/or commercialized by businesses in least 8 developing and low-to-middle income countries (see p 58 of partners PAD); 5. Demand for and availability of these innovations have increased. 6. More food has been produced with less water or more water has been made available for food production in the eligible countries (NL aim in line with NL policy is 25% resource efficiency improvement by the program innovations as compared to standard practice in the implementation countries; Swedish International Development Cooperation Agency (Sida) aims for 20% resource efficiency improvement by the program innovations as compared to standard practice in the implementation countries). 7. This program will also contribute to increased water-related resilience to climate change (climate change adaptation). 8. Impact indicators include (see SWFF PMEP for specific program indicators and indicator targets): a) Percentage food productivity / volume water increase; b) Volume of water saved through efficiency-increasing innovations in the food value chain c) Volume of water captured and stored for food production d) Percentage increase in agricultural yields/farmed area due to SWFF innovations e) Number of direct beneficiaries of the program (disaggregated by gender) f) Number of innovations adopted, brought to scale, and/or commercialized g) Number of poor people reached (in some cases, individual SWFF awardees had expected outcomes regarding poverty). SWFF will provide the external evaluator with all SWFF awardee, Technical Assistance Facility, and overall program data that were collected on a semi-annual basis. The SWFF program directly feeds into the overarching Global Development LAB (LAB) Monitoring & Evaluation Framework. SWFF’s M&E processes enable timely and consistent collection of comparable performance data in order to make informed program management decisions. In line with the GCD M&E Operational Plan, the SWFF PMEP fully incorporates the LAB Results Framework and tracks and reports on indicators at three levels: 1) the meta-level (enabling analysis across GCDs); 2) the program-level (SWFF level); and 3) awardee-level. To facilitate awardee reporting, each awardee works with the SWFF TA Facility on overall compliance, data collection, quality control, and to facilitate the collection of results data from innovators and entry of validated results into the DevResults online platform. In some cases, the TA Facility conducts field visits to verify awardee achievements and assist with addressing any shortcomings. A milestone is an intermediary result on the way to achieving the overall objective. In the case of SWFF, milestones are used to determine – among other things – the awardee’s ability to achieve wide scale with their innovation. Each awardee works with the Founding Partners to establish quantifiable milestones (based on select indicators from Table 2 and/or additional specific indicators). Importantly, the achievement of milestones determines future SWFF funding. 121 As mentioned in the introduction, SWFF innovators are not required to develop individual M&E Plans. However, awardee’s Acceleration Workplans have an M&E section, and also describe the quantifiable milestones by which the Founding Partners can evaluate performance. These milestones have been tracked against timelines and resources. The TA Facility has been instrumental in helping innovators achieve their milestones. However, as with any innovation programming, SWFF knows that some innovations fail. Therefore, if at any annual review an innovation is no longer meeting agreed upon milestones and matching fund requirements, Securing Water for Food does not continue to fund it. SWFF will also provide the external evaluator with all awardee Acceleration Workplans. VIII. EVALUATION QUESTIONS This final program evaluation should determine whether or not the SWFF program as a whole led to the production of more food using less water based on the performance of the innovations that SWFF funded. During this evaluation, the evaluator will examine evidence provided to SWFF by SWFF innovators in order to determine the impact of the program around the following questions: A.SWFF Innovator’s Impact 1. To what extent has SWFF contributed to the outcomes and results outlined by the indicators in the SWFF PMEP during project implementation and post project implementation? In answering this question, the evaluator should take into consideration the following: a. Is there demand and local ownership for the SWFF innovations (individually and across all innovations)? b. Does the SWFF TA overcome organizational capacity barriers for innovators? Were there additional barriers that were not addressed? c. Have vulnerable groups (the poor, women, ethnic minorities) been positively and/or negatively impacted (through income, employment, water/environmental) from SWFF supported innovations? Specifically, has SWFF reduced the number of people in poverty as a result of supporting SWFF-supported innovations and has SWFF increase the number of women benefitting from SWFF-supported innovations. d. Did SWFF-supported projects increase water efficiency/make water more accessible? Did SWFF projects meet their water efficiency/availability targets? Overall, across all innovators, did the program meet the water efficiency/availability targets? e. Did SWFF-supported projects lead to more agricultural productivity and resilience to climate change? Did SWFF projects meet their agricultural productivity targets? Overall, across all innovators, did the program meet the agricultural productivity targets? f. Were SWFF-supported projects environmentally sustainable (i.e., did they provide 122 positive environmental benefit, or did they do more environmental harm than good)? g. Were SWFF-supported projects likely to be financially and socially sustainable by the organizations supporting the innovation? h. What is the balance between public/social engagement and private/public engagement? To what extent have private funds been generated that contribute to the developmental objectives of the program both during and following SWFF awards. 2. How much of the measured change (outcome or result indicator) in the SWFF program can in fact be attributed to the SWFF-supported projects? That is, what portion of the result is not explained by the projects examined by the evaluation? (SWFF recognizes that this is a difficult question to answer but wants the evaluator to make the best effort to answer this question). 3. To what extent are there differences between the planned SWFF-supported projects and what was actually delivered in Year 1 and then Years 2-3 of the projects? B. SWFF Program’s Impact and Efficiency 1. To what extent were the SWFF results to date in balance with the level of effort and resources (funds, human resources including by the FPs, TAF, interns, consultants)? 2. To what extent is the level of effort and resources spent by applicants/innovators in balance with the added value SWFF brings? 3. How effectively have investment risks been managed by the program? (number of failed projects, timeliness of reaction on problems observed etc.) 4. To what extent was the TA Facility efficiently set up, organized and managed? 5. To what extent did the SWFF TA Facility provide SWFF innovators with technical assistance that led to an immediate success (a support engagement is defined as an immediate success if deliverables formally agreed to by the awardee in the work plan were delivered as the awardee expected) and long term success (a support engagement is defined as a long-term success if the product or advice delivered is actually adopted by the innovator and results in recognized value, such as a shift in strategy, an effective partnership, additional funding, new financial forecasting capabilities, or an improved manufacturing approach or product design)? To what extent did the SWFF TA Facility provide SWFF innovators the technical assistance that innovators deemed useful? Did SWFF enable certain (new) (combinations of) expertise to be deployed that would otherwise likely not have been deployed/used (by the individual partners)? 6. To what extent are the administrative costs for managing SWFF above, below, or on par with the cost of similar Challenge funds? (Special Consideration should be made for funds that provide technical assistance to their innovators.) 7. How well did founding partners interact within SWFF and what lessons should be taken from their interactions? IX. EVALUATION DESIGN AND METHODOLOGY 123 SWFF expects that the evaluator will conduct the following in this performance/partial-impact evaluation: It is expected that the final evaluation will be performed in 2 states because of the number of SWFF awards that end by September 2018. The first stage (March 2018-March 2019) will review awards made in Rounds 1-3 to validate/invalidate evidence presented by SWFF innovators/the SWFF program. The second stage (December 2019-July 2020) will review Rd. 4 awards to validate/invalidate evidence presented by SWFF innovators/the SWFF program. The evaluation should include a description of what types of innovations did the SWFF program support including the following information: a. Technically, what types of innovations have been supported in these projects? How many innovations have been awarded funding? What are the basic demographics/descriptors of innovators? b. What types of enterprises are participating and with what kind of (financial) interests? In addition, the second stage will determine if there is any further evidence that Rd. 1-3 innovations were sustainable and continued to reach wider scale adoption after the SWFF award ended. The final evaluation will draw on SWFF awardee site visits examining program implementation, organizational strength, as well as the effective water productivity and efficiency gains by the awarded innovations. These and other findings will be matched with secondary data from research on similar innovations. Data used for the final evaluation may include information from rejected proposals and their applicants. In addition, SWFF has recently conducted a survey to gather data from SWFF applicants that did not receive award funding with comparable results from SWFF innovators so that we can see if there is a difference in outcomes between the 2 groups. The evaluation shall conform to the OECD DAC Quality Standards for Development Evaluation (2010). The external evaluator is expected to use a combination of qualitative and quantitative methods, including: 1. Desk review of all awardee documents to be provided by USAID and the SWFF TA Facility. These documents will provide the basis for validation activities in field performance evaluation site visits. Documentation may include previously performed external evaluations provided either by innovators or the SWFF program. 2. In-depth, semi-structured interviews with stakeholders (in a few instances, the external evaluator may be able to interview ultimate beneficiaries through stakeholders but will most often use site visit documentation for that effort). These should in any case include relevant staff from founding partners, TA Facility, members of the IIAC, and a majority of SWFF innovators and alumni. Results from these interviews should be provided in the Evaluation Final Report. 3. Questionnaire surveys; 4. Video and/or teleconferencing; 5. Interviews with customers (minimum 200) and non-customers (minimum 50) 6. Observations; 7. Evaluation team is expected to conduct as many site visits as possible to innovators that received 3 years of funding. In addition, the Evaluation Team will conduct site visits for some of the SWFF projects that only received 1 year of funding. At a minimum, the Evaluation Team will visit at least 8 SWFF innovators. SWFF suggests the following SWFF innovators: 124 a) S. Asia and S.E. Asia: Adaptive Symbiotic Technologies, Practical Action Bangladesh, Lal Teer Seed Ltd., Naireeta Services, aQysta, Project Alba, MimosaTEK b) LAC: Climate Stations ICU Peru c) Sub-Saharan Africa: TAHMO, Reel Gardening, Conservation S. Africa, Green Heat Ltd., IRK Sunculture, Skyfox, Aybar Engineering d) Middle East/North Africa: International Center for Biosaline Agriculture 8. After 6 weeks, based on interviews with key informants and initial reviews of documentation, the external evaluator will draft an evaluation plan showing the information required to answer the review questions, including how this information will be collected and discuss this with the Founding Partners. The final evaluation should incorporate the outcome harvesting (OH) method to examine some of the unintended outcomes (positive and negative) from the GCD {OH uses a specific meaning of the term “outcome” that refers to a change in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution}. The Founding Partners recognize that the Evaluation Team may not able to perform a full counterfactual for the SWFF program as a whole given that SWFF does not have a control group of innovations that are closely paired/matched with the awarded innovations to truly determine the effectiveness of the program. However, SWFF has conducted a survey of applicants that did not receive funding along with comparison data among SWFF award winners and hope that the Evaluation Team will use those results and also conduct a follow-up survey to validate/invalidate SWFF’s own findings. The Founding Partners also recognize that the Evaluation Team may not able to perform counterfactuals for most of the SWFF-supported projects. However, we hope that the evaluators will be able determine impact from a mixed methods approach and use existing information where relevant and appropriate. Data collection and analysis USAID requests that the evaluator complete the following table as part of its detailed design and evaluation plan.1 Evaluation question Data source Data collection method (including sampling methodology, where applicable) Data analysis method 125 Evaluation question Data source Data collection method (including sampling methodology, where applicable) Data analysis method 1 Another format may be used if the table is not preferred, but any chosen format should contain all the information specified for each question. X. DELIVERABLES AND REPORTING REQUIREMENT 1. Evaluation Work plan: Within 1 month of the award of the contract, a draft work plan for the evaluation shall be completed by the lead evaluator and presented to the SWFF Team Lead. The work plan will include: a. The anticipated schedule and logistical arrangements. b. A list of the members of the evaluation team, delineated by roles and responsibilities. c. A description of the key issues to be evaluated including the evaluation questions. d. Criteria and (proxy) indicators for assessing the impact and efficiency of SWFF. e. A final proposal for the methodology and sample selection. f. List of specific questions and concerns relating to the evaluation to which Founding Partners and TA Facility will respond. g. List of key documents and resource people for the evaluation. 2. Evaluation Design: Within 1 month of approval of the work plan, the evaluation team must submit to the SWFF Team Lead an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design. 3. The team should also expect to have an in-briefing with the SWFF Team Lead, SWFF Founding Partner representatives, and SWFF TAF M&E manager to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. 4. In-briefing/inception report: Within 5 days of arrival at any site visit location the evaluation team will have an in-briefing with the SWFF innovators receiving the site visit for introductions and to receive background and additional documents and materials from 126 SWFF innovators. 5. Mid-term Briefing Report and Interim Meetings: The evaluation team is to create a Mid- Term report that includes draft summary findings of the first stage (March 2018- March 2019) of the review of awards made in Rounds 1-3 to validate/invalidate evidence presented by SWFF innovators/the SWFF program. The Evaluation Team is expected to hold a mid-term briefing with the SWFF Team Lead, Founding Partner representatives, and the SWFF TAF M&E Manger the status of the evaluation, including a discussion of the draft summary findings) of the review of awards made in Rounds 1-3, potential challenges and emerging opportunities. The team will also provide the evaluation SWFF Team Lead, Founding Partner representatives, and the SWFF TAF M&E Manger with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, monthly briefings by phone can be arranged. 6. Final Presentation: The Evaluation Team is expected to hold a final presentation of 90 minutes including Q&A in person/by virtual conferencing software to discuss the summary of findings and recommendations to SWFF. This presentation will be scheduled as agreed upon during the in-briefing. 7. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with SWFF. The Draft Final Report should include a detailed and in depth analysis of SWFF innovator’s impact overall and individually, as well as detailed and in- depth analysis of the SWFF Program’s impact and efficiency. While the report is being finalized, the Founding Partners will: (1) Meet with the Evaluation Team to debrief and discuss results or findings and provide feedback on any factual errors; (2) Review the key findings, conclusions, and recommendations systematically; The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial Draft Evaluation Report is submitted, the SWFF team lead, Founding Partner representatives, and SWFF TAF M&E Manager will have 10 business days in which to review and comment on the initial draft and send consolidated comments to the evaluation team. The Evaluation Team will then be asked to submit a revised Final Draft Report 10 business days hence, and again the SWFF team lead, Founding Partner representatives, and SWFF TAF M&E Manager will have 10 days to review and send comments on this final draft report. 8. Final Evaluation Report: The Evaluation Team will be asked to take no more than 10 business days to respond/incorporate the final comments from SWFF. The evaluation team leader will then submit the final report to the SWFF Team. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. In addition, USAID would like to have a one-page infographic with summaries included in the report that summarizes the reports findings XI. EVALUATION SCHEDULE 127 The final evaluation will start on March 28, 2018 with preparation of the evaluation plan. A kick off meeting will be organized in the second week of May 2018 with a presentation of the draft evaluation plan. Monthly feedback sessions, chaired by USAID, attended by the Founding Partners will be organized to keep track of progress. The review is to be completed by August 1, 2020. Sample Format: Illustrative Schedule Timing (Anticipated Months or Duration) Proposed Activities Important Considerations/Con straints 1 month Preparation of the work plan and evaluation design 1 month USAID review of the work plan and evaluation design Take into account availability in the 2 weeks In-Briefing 6 months Data Collection There will be 2 data collection stages. The first stage will require 4 months of data collection. It is expected that the 3 months Data Analysis There will be 2 data analysis stages. The first stage will require 2 months of data analysis. It is expected that the second stage will 4 months Report writing There will be 2 report writing stages. The first stage will require 1 months of report writing. It is 1 month USAID review of Draft Report 1 month Incorporate SWFF comments and prepare Final Report XII. FINAL REPORT FORMAT The Evaluation Final Report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being 128 evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: 1) The Evaluation SOW; 2) Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; 3) All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; 4) All sources of information, properly identified and listed; and 5) Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. 6) Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. 7) Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. XIII. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.2 1) Evaluation reports should represent a thoughtful, well-researched, and well￾organized effort to objectively evaluate the strategy, project, or activity. 129 2) Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. 3) The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. 4) Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. 5) Evaluation methodology should be explained in detail and sources of information properly identified. 6) Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). The consequences of the limitations should be discussed as well. 7) Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. 8) Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. 9) If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. 10) If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. XIV. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine￾readable, non-proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the SWFF Team Lead. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. END OF SECTION C 130 ANNEX 2 SWFF EVALUATION QUESTIONS BY OECD DAC CRITERIA This final program evaluation should determine whether or not the SWFF program as a whole led to the production of more food using less water based on the performance of the innovations that SWFF funded. The following overarching questions (from Section C of the Description / Specifications / Statement of Work and the Dexis contract) with key supporting / sub-questions have been used to assist in framing and organizing the analysis along the OECD-DAC standard evaluation categories. [The reader should note that the questions were organized in a different order for the purposes of interviewing in order to allow for a more logical conversational flow of information.] SWFF Program’s Relevance (Meta and Program Levels) (Basic Premise and Approach) 1. Technically, what types of innovations have been supported? How many innovations have been awarded funding? What are the basic demographics/ descriptors of innovators? 2. Is there demand and local ownership for the SWFF innovations (individually and across all innovations)? SWFF Program’s Effectiveness (Program Levels) (Achieved what set out to achieve) 3. To what extent did the SWFF TA Facility provide SWFF innovators with timely and appropriate technical assistance that led to the creation or enhancing of a viable business centered around an innovation that saved water / generated more agricultural product? a. Does the SWFF TA support help the awardee to overcome organizational capacity barriers? (These could include strategy development, financial forecasting capabilities, improved manufacturing approaches or product design, barriers to accessing funding, etc.). b. Did SWFF TAF enable certain (new) (combinations of) expertise to be deployed that would otherwise likely not have been deployed/used (by the individual partners)? c. Were there additional barriers that were not addressed by the SWFF TAF? d. Were there positive or negative unintended consequences of the support? e. To what extent did the SWFF TA Facility provide SWFF innovators with technical assistance that innovators deemed useful? [To what extent did the SWFF TA Facility provide SWFF innovators with technical assistance that led to an immediate success (a support engagement is defined as an immediate success if deliverables formally agreed to by the awardee in the work plan were delivered as the awardee expected)?] [To what extent did the SWFF TA Facility provide SWFF innovators with technical assistance that led to a long-term success (a support engagement is defined as a long-term success if the product or advice delivered is actually adopted by the innovator and results in recognized value, such as a shift in strategy, an effective partnership, additional funding, new financial forecasting capabilities, or an improved manufacturing approach or product design)?] 4. How effectively have investment risks been managed by the program? (number of failed projects, timeliness of reaction on problems observed etc.) 5. How well did founding partners interact within SWFF and what lessons should be taken from their interactions? 131 SWFF Program’s Efficiency (Program Levels) (Value for Money, etc.) 6. To what extent were the SWFF results to date in balance with the level of effort and resources (funds, f resources including by the FPs, TAF, interns, consultants)? a. To what extent was the TA Facility efficiently set up, organized and managed? b. To what extent are the administrative costs for managing SWFF above, below, or on par with the cost of similar Challenge funds? (Special consideration should be made for funds that provide technical assistance to their innovators.) c. To what extent is the level of effort and resources spent by applicants/innovators in balance with the added value SWFF brings? SWFF’s Impact (Innovator and Program Levels) 7. Were solutions sourced through SWFF adopted and utilized at scale? 8. To what extent has SWFF contributed to the outcomes and results outlined by the indicators in the SWFF PMEP during project implementation and post project implementation? a. Did SWFF-supported projects increase water efficiency/make water more accessible? Did SWFF projects meet their water efficiency/availability targets? Overall, across all innovators, did the program meet the water efficiency/availability targets? b. Did SWFF-supported projects lead to more agricultural productivity and resilience to climate change? Did SWFF projects meet their agricultural productivity targets? Overall, across all innovators, did the program meet the agricultural productivity targets? c. Have vulnerable groups (the poor, women, ethnic minorities) been positively and/or negatively impacted (through income, employment, water/environmental) from SWFF supported innovations? Specifically, has SWFF reduced the number of people in poverty as a result of supporting SWFF-supported innovations and has SWFF increased the number of women benefitting from SWFF-supported innovations? d. What is the balance between public/social engagement and private/public engagement? To what extent have private funds been generated that contribute to the developmental objectives of the program both during and following SWFF awards? 9. How much of the measured change (outcome or result indicator) in the SWFF program can in fact be attributed to the SWFF-supported projects? That is, what portion of the result is not explained by the projects examined by the evaluation? 10. To what extent are there differences between the planned SWFF-supported projects and what was actually delivered in Year 1 and then Years 2-3 of the projects? SWFF Sustainability (Meta and Program Level) 11. What is the likelihood that SWFF-supported innovations will have a sustained market presence (i.e. financially and socially sustainable)? 12. Are SWFF-supported projects environmentally sustainable (i.e., did they provide positive environmental benefit, or did they do more environmental harm than good)? 132 ANNEX 3 KEY INFORMANT INTERVIEWS BY COUNTRY SOURCE COUNTRY NAME KII 1 Visit Ghana Ignitia 2 Visit Ghana SKYFOX 3 Visit India WASTE 4 Visit India NewSil/Si Tek 5 Visit ------- IIAC, Mikael Medelberg 6 Visit Holland DGIS 7 Visit Sweden Sida 8 Visit Ghana FGD SkyFox 9 Visit South Africa DST 10 Visit South Africa Vuyisile Phehane 11 Visit South Africa CUT ITIKI 12 Visit South Africa CUT ITIKI 13 Visit South Africa EcoRangers 14 Visit India SWAR 15 Visit India Naireeta 16 Visit India Naireeta/Govt official 17 Visit USA USAID, Ku and Deidre 18 Visit India NewSil 19 Visit Kenya Hydroponics Kenya 20 WWW USA TAF 1 and 6 21 WWW ------- IIAC WWW GROUP 1 22 WWW ------- IIAC WWW GROUP 2 23 WWW South Africa South Africa: FP 24 WWW India AST 25 Visit Kenya CUT ITIKI 26 WWW USA MIT 27 Visit Bangladesh Lal Teer Seed 28 Visit Uganda Water Governance Institute 29 Visit Uganda Green Heat Uganda 30 WWW Peru ICU Peru 31 WWW Pakistan MetaMeta 32 Visit Bangladesh Practical Action 33 WWW Vietnam MimosaTek 34 Visit Egypt Egypt ICBA Cairo 35 Visit Egypt Egypt DRC New Valley 36 WWW Cambodia Project Alba 133 SOURCE COUNTRY NAME KII 37 WWW Mozambique Future Water 38 WWW Mozambique World Hope 39 WWW Nepal Aqysta 134 ANNEX 4 SURVEYS OF INNOVATIONS # Innovation Country Respondents Male Percentage Female Percentage 1 Aquaponics Farming (WGI), Rd 3 Uganda 10 3 70% 7 70% 2 CUT ITIKI, Rd 4 Kenya 4 2 50% 2 50% 3 CUT, Rd 4 South Africa 23 5 22% 18 78% 4 Ecorangers, Rd 1 South Africa 27 21 78% 6 22% 5 Green Heat, Rd 3 Uganda 8 4 50% 4 50% 6 Hydroponics Africa, Rd 3 Kenya 13 2 15% 11 85% 7 ICBA, Rd 1 Egypt 24 24 100% 0 0% 8 Ignitia, Rd 3 Ghana 14 8 57% 6 43% 9 Lal Teer, Rd 3 Bangladesh 25 20 80% 5 20% 10 Naireeta, Rd 3 India 25 1 4% 24 96% 11 NewSil, Rd 3 India 10 8 80% 2 20% 12 Skyfox, Rd 4 Ghana 10 8 80% 2 20% 13 SWAR, Rd 3 India 6 6 100% 0 0% 14 WASTE, Rd 4 India 22 5 23% 17 77% Total 221 112 95 Average 16 9.3 57% 7.9 43% 135 ANNEX 5 METADATA To be included with the final report 136 ANNEX 6 DATA COLLECTION INSTRUMENTS SWFF Evaluation Phase 1 Data Collection 1. Questionnaire - SKYFOX 2. Questionnaire - MNP Matatiele 3. Questionnaire -SWAR 4. Questionnaire - CUT 5. Questionnaire - -WASTE 6. Questionnaire - SNEWSIL 7. Questionnaire - Naireeta 8. Questionnaire - Ignitia 9. Questionnaire - ICBA-DRI 137 1. Questionnaire - SKYFOX Interview Questions for Farmers using the SkyFox AquaCulture Ponds Q1 What is the name of your community? Q2 Names: (Optional) Q3 Gender? □ Male □ Female Q4 Age bracket? □ Under 18 □ 18-35 □ 36-45 □ 46-55 □ 56-65 □ above 66 Q5 Highest Education Level: □ None □ Primary □ Secondary □ Post-Secondary Q6 How many people are in your household? Q7 How large is your farm/plot? (Plots)(Hectares) Q8 What was the biggest challenge you faced in farming during last season? □ Unexpected rainfall □ Less (than expected) rainfall - Drought □ Cost of inputs (seeds, fertilizer etc.) □ Cost of Labour□ availability of machinery□ knowledge of new cost￾effective methods □ Transportation to market □ Spoilage □ Attacks by pests (e.g. army warms) □ Other (PLEASE SPECIFY) Q9 Is access to water a problem in your area? (Yes / No) Q10 Have weather patterns changed in recent years? (Yes / No) If yes, how? Q11 How has the SkyFox Pond helped you? Ask the question without prompting and score these responses, ranking by importance 1, 2, 3: DO NOT READTHIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. □ Conserve water Y/N □ Make water reusable Y/N □ This helped women farmers as well as helping the men .Y/N □ They made a special effort to include woman farmers. Y/N □ It helped in producing more of our most important crop □ Grow cash crop(s) in saline water Y/N □ Grow more produce with less water Y/N □ Increase my yield through accurate and timely forecasts Y/N □ Help with cost of inputs (seeds, fertilizer etc.) Y/N 138 □ Improved health, strength and value of livestock Y/N □ Helping reduce time used, saved labour Y/N □ Reduce amount of water use for irrigation Y/N □ Reduce cost of production through reduced input cost, labour etc. Y/N □ Reduce crop wastage □ Other (PLEASE SPECIFY) Q12 Does the pond save water or are you using more water? (Save / Using more) Percentage change Q13 Are you producing more food with the pond? (Yes / No) Across more seasons? (If yes, how do measure this increase?) Q14 Are you growing different crops with the ponds? (If yes, what kinds? ) Q15 Why did you start using the pond? Q16 When did you start using the innovation? (less than 6 months ago, between 6 months and 1 year, between 1 and 2 years, more than 2 years) Q17 Does it save you time? (Yes / No) Q18 Are some families in the area not using the pond? (Yes / No) If no, why not? Q19 Is it affordable for you? (Yes / No) Q20 Is it affordable for poorer households in the community? (Yes / No) Q21 Are you more secure in terms of food availability? Q22 Are there any negative impacts on poor households in the community? Q23 Are there any negative impacts on women? Q24 Are there any negative impacts on other vulnerable members of the community? Q25 Have there been special efforts to engage women to use the ponds? Q26 Is there anything that has resulted from your use of the ponds that you didn’t expect? Q27 What other benefits would you expect after investing in the pond? __ Training ___Maintenance ___ Repairs ___ Others (Explain _______) Q28 Are there any harmful environmental effects that you are noticing after using the ponds for a period of time? Q29 Has the use of the pond improved your family income? □ Very much □ Significantly 139 □ Very little □ Not at all □ Other benefits (PLEASE SPECIFY) Q30 Do you think you will still be using the pond in the long term (5 to 10 years)? □ No□ Yes If no, why? If yes, why? Q31 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Respondent Photograph GPS Coordinates 140 2. Questionnaire - MNP Matatiele ODK survey, July 5, 2018 1 NAME 2 GENDER 3 NAME OF COMMUNITY 4 NO IN HH 5 SOURCE OF INCOME 6 What are positive or negative changes in your life (in agriculture)? 7 How many cattle do you own? Categories: 0; 1-2; 3-5; 6-9; 10 and above 8 Cattle are stronger. YES/NO 9 Can get a good price when selling cattle. YES/NO 10 How do you get a good price when selling your cattle? 11 Can grow more maize and vegetables. YES/NO 12 Meat Naturally helped achieve these things. YES/NO 13 Does your community experience water scarcity? 14 What is the effect of water scarcity? 15 Have you used the auctions? 16 Why have you not used the auction? 17 Will poorer households in the community use the auctions or benefit from the herders? YES/NO 18 How would poorer families benefit from using the auctions and benefit from the herders? 19 Have the auctions and herders increased your family income? 20 Will the families have more food security? 21 Why are people using the auctions? 22 For how long do you see yourself using auctions into the future? SHORT TERM/LONG TERM 23 Have women benefited from the auctions? NO/A LITTLE/VERY MUCH/ONLY THE FEW WOMEN WITH CATTLE 24 How have women benefited from the auctions? 141 25 Are there any negative impacts/responses on women from the introduction of herders and the auctions? 26 What are the negative impacts of herders and auctions? 27 What are the positive benefits from herders and auctions? 28 Has your own income has increased since starting using the auctions? 29 How often have you used the auctions? 30 30 What are the positive and negative effects on crops and fields? 31 GPS Coordinates 32 GPS Coordinates-altitude 33 GPS Coordinates-accuracy 34 Photograph 142 3. Questionnaire -SWAR INNOVATION IMPACT EVALUATION SURVEY (SWFF) Q1 What is the name of your community? IFTHEREASUB-VILLAGE PUTTHE NAME OFTHE MAIN VILLAGE FIRSTAND THE SUB-VILLAGE SECOND Q2 Names: (Optional) Q3 Gender? □ Male □ Female Q4 Age bracket? □ Under 18 □ 18-35 □ 36-45 □ 46-55 □ 56-65 □ above 66 Q5 Highest Education Level: □ None□ Primary□ Secondary□ Post-Secondary Q6 How many people are in your household? Q7 How large is your farm/plot? (Plots)(Hectares) Q8 What crops do you grow? [Rank the first three profitable or highest volume by number, 1, 2, 3] □ Maize □ Millet□ Beans □ Pumpkins □ Green beans □ Onions □ Other vegs □ Sorghum □ Cow peas□ Green Grams Q9 What was the biggest challenge you faced in farming during last Season? □ Unexpected rainfall □ Less (than expected) rainfall - Drought □ Cost of inputs (seeds, fertilizer etc.) □ Cost of Labour □ availability of machinery □ knowledge of new cost￾effective methods □ Transportation to market □ Spoilage □ Attacks by pests (e.g. army warms) □ Other (PLEASE SPECIFY) Q10 Have you heard of the INNOVATION (name it)? YES/NO IF NO THEN THANK THE RESPONDENT AND END THE INTERVIEW Q11 Would you be prepared to pay for INNOVATION NAME produce or service? YES/NO/THERE SHOULD BENOCHARGE Q12 If yes for the above question, how much would you be willing to pay for this service LOCALCURRENCY/ EQUIVLENT IN USD 143 Q13 On what crop(s) do you use the innovation? □ ALL □ Maize □ Millet □ Beans □ Pumpkins □ Green beans □ Onions □ Other vegs □ Sorghum □ Cow peas □ Green Grams Q14 How has INNOVATION NAME helped you? Ask the question without prompting and score these responses, ranking by importance 1, 2, 3: DO NOT READTHIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. □ Conserve water Y/N □ Make water reusable Y/N □ This helped women farmers as well as helping the men .Y/N □ They made a special effort to include woman farmers. Y/N □ It helped in producing more of our most important crop □ Grow cash crop(s) in saline water Y/N □ Grow more produce with less water Y/N □ Increase my yield through accurate and timely forecasts Y/N □ Help with cost of inputs (seeds, fertilizer etc.) Y/N □ Improved health, strength and value of livestock Y/N □ Helping reduce time used, saved labour Y/N □ Reduce amount of water use for irrigation Y/N □ Reduce cost of production through reduced input cost, labour etc. Y/N □ Reduce crop wastage □ death/wilting of crops/fruit trees could be avoided □ Other (PLEASE SPECIFY) Q15 Has this improved your family income? □ Very much □ Significantly □ Very little □ Not at all □ Other benefits (PLEASE SPECIFY) Q16 How much water increase did you have available for agriculture this year compared to a year in which you did not use the innovation? Q17 How do you compare innovation of SWAR systems with the normal drip irrigation system? Please tick whatever option is applicable. 144 a) I do not see this system to be any different from that of drip with respect to utilization of water b) Usage of water is definitely lesser than that used by drip, but the difference is marginal (less than 10%) c) Usage of water is 11% to 30% than that used by the drip d) Usage of water is 31% to 50% than that used by drip e) Usage of water is 51% to 80% than that used by drip c) Usage of water is 80% to 99% than that used by drip Q18 Do you find any difference in the health of the soil after the usage of innovation of SWAR systems? Yes/No/Not sure Q19 If the answer is Yes, please ask them to enlist the difference that they have seen in the health of the soil and tick whatever may apply and mark 1,2,3 as per the priority. a) The quality of soil seems to be improved b) Quality of soil seems to have deteriorated c) Soil has become more porous and crumbles easily when taken in hand d) Soil has become very hard and has lost its porosity c) Organic matter content seems to have improved d) Organic matter content seems to have decreased e) Soil seems to be absorbing more water f) Soil is getting more washed away with rains when compared to what it did earlier Q20 What other benefits would you expect after buying the INNOVATIOIN? □ Training □ Maintenance □ Repairs □ Other (please specify) Q21 Do you think you’ll use the innovation on the long term (5 to 10/ years)? □ No□ Yes If no, why? If yes, why? Q22 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Q23 PHOTO Q24 GPS 145 4. Questionnaire - CUT INNOVATION IMPACT EVALUATION SURVEY (SWFF) Q 1 What is the name of your community? Q 2 Names: (Optional) Q 3 Gender? □ Male □ Female Q 4 Age bracket? □ Under 18 □ 18-35 □ 36-45 □ 46-55 □ 56-65 □ above66 Q 5 Highest Education Level: □ None □ Primary □ Secondary□ Post-Secondary Q 6 How many people are in your household? Q 7 How large is your farm/plot? (Plots)(Hectares) Q 8 What crops do you grow? [Rank the first three profitable or highest volume by number, 1,2,3] □ Maize □ Milet □ Beans □ Pumpkins □ Green beans □ Onions □ Other vegs □ Sorghum □ Cowpeas □ Green Grams Q 9 What was the biggest challenge you faced in farming during last Season? □ Unexpected rainfall □ Less (than expected) rainfall - Drought □ Cost of inputs (seeds, fertilizer etc.) □ Cost of Labour □ availability of machinery □ knowledge of new cost￾effective methods □ Transportation to market□ Spoilage □ Attacks by pests (e.g. army warms) □ Other (PLEASE SPECIFY) Q 10 Have you heard of the INNOVATION (name it)? YES/NO IF NO THEN THANK THE RESPONDENT AND END THE INTERVIEW Q 11 Would you be prepared to pay for INNOVATION NAME produce or service? YES/NO/THERE SHOULD BENOCHARGE Q 12 If yes for the above question, how much would you be willing to pay for this service LOCALCURRENCY/EQUIVLENT IN USD 146 Q 13 On what crop(s) do you use the innovation? □ ALL □ Maize □ Milet □ Beans □ Pumpkins □ Green beans □ Onions □ Other vegs □ Sorghum □ Cowpeas□ Green Grams Q 14 How has INNOVATION NAME helped you? Ask the question without prompting and score these responses, ranking by importance 1,2,3: DONOT READ THIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. □ Conserve water Y/N □ Make water reusable Y/N □ This helped women farmers as well as helping the men. Y/N □ They made a special effort to include woman farmers. Y/N □ It helped in producing more of our most important crop □ Grow cash crop(s)in saline water Y/N □ Grow more produce with less water Y/N □ Increase my yield through accurate and timely forecasts Y/N □ Help with cost of inputs (seeds, fertilizer etc.) Y/N □ Improved health, strength and value of livestock Y/N □ Helping reduce time used, saved labour Y/N □ Reduce amount of water use for irrigation Y/N □ Reduce cost of production through reduced input cost, labour etc. Y/N □ Reduce crop wastage □ Other(PLEASE SPECIFY) Q 15 Has this improved your family income? □ Very much □ Significantly □ Very little □ Not at all □ Other benefits (PLEASE SPECIFY) Q 16 How much did your water use change in agriculture compared to last year? Percent Q 17 What other benefits would you expect after buying the INNOVATIOIN? □ Training □ Maintenance □ Repairs □ Other(please specify) Q 18 Do you think you’ll use the innovation on the long term (5 to10p/ years)? 147 □ No □ Yes If no, why? If yes, why? Q 19 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Q 20 GPS Q 21 Photo See excel file 148 5. Questionnaire - -WASTE INNOVATION IMPACT EVALUATION SURVEY (SWFF) Q1 Names of the respondent: (Optional) Q2 Type of location: □ Homestad □ Farm □ Home□ Community center □ Office of innovator □ Others Q3 Village: Q4 Block: Q5 District: Q6 Gender? □ Male □ Female Q7 Age bracket? □ Under 18 □ 18-35 □ 36-45 □ 46-55 □ 56-65 □ above 66 Q8 Highest Education Level: □ None □ Primary □ Secondary □ Post-Secondary Q9 How many people are in your household? Q10 How large is your farm/plot? (Plots)(Hectares) Q11 What crops do you grow? [Rank the first three profitable or highest volume by number, 1, 2, 3] □ Potato □ Cabbage □ Carrot □ Cauliflower□ Peaches □ Pears □ Plums □ Strawberry □ Flowers □ Beans □ Beet root □ Radish □ Paddy □ Coffee □ Tea □ Any other Q12 What was the biggest challenge you faced in farming during last Season? □ Unexpected rainfall □ Less (than expected) rainfall - Drought □ No rainfall (seeds, fertilizer etc.) □ Cost of Labour □ availability of machinery □ knowledge of new cost￾effective methods □ Transportation to market □ Spoilage □ Attacks by pests (e.g. army warms) □ Disease attack □ Nutritional deficiency □ Other (PLEASE SPECIFY) 149 Q13 Have you used recycled (grey) water? YES/NO Q14 If yes, from which date Q15 What is the result of use of Grey water? □ Could provide more number of irrigations than what could be done before □ Could cultivate additional land □ Could cultivate fodder and take up diary □ Any other Q16 Have you used Co-compost? YES/NO Q17 If yes, from which date Q18 Has this led to increase in productivity of land? YES/NO/Cannot say Q19 Has this reduced the purchase of inputs from the market? YES/NO/Cannot say Q 20 Percentage reduction in external input because of Co-compost (%) Q21 How has innovation of WASTE supported you? Ask the question without prompting and score these responses, ranking by importance 1, 2, 3: DO NOT READ THIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. □ Conserve water Y/N □ Make water reusable Y/N □ This helped women farmers as well as helping the men. Y/N □ They made a special effort to include woman farmers. Y/N □ It helped in producing more of our most important crop □ Grow cash crop(s) in saline water Y/N □ Grow more produce with less water Y/N □ Increase my yield through accurate and timely forecasts Y/N □ Help with cost of inputs (seeds, fertilizer etc.) Y/N □ Improved health, strength and value of livestock Y/N □ Helping reduce time used, saved labour Y/N □ Reduce amount of water I use for irrigation Y/N □ Reduce cost of production through reduced input cost, labour etc. Y/N □ Reduce crop wastage 150 □ death/wilting of crops/fruit trees could be avoided □ Other (PLEASE SPECIFY) Q22 Has this improved your family income? □ Very much □ Significantly □ Very little □ Not at all □ Other benefits (PLEASE SPECIFY) Q 23 What other benefits would you expect after buying irrigation system from SWAR? □ Training □ Maintenance □ Repairs □ Other (please specify) Q24 Do you think you’ll use the innovation on the long term (5 to 10/ years)? □ No □ Yes Q25 Please explain your previous answer Q26 Are you willing to pay for the innovation? YES/NO Q27 If yes for the above question, how much would you be willing to pay for this service LOCALCURRENCY / EQUIVLENT IN USD Q28 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Q29 GPS Coordinates 151 6. Questionnaire - SNEWSIL INNOVATION IMPACT EVALUATION SURVEY (SWFF) Q1 Names: (Optional) Q2 What is the name of your village?……………………………………………….. Q3 Phone number :……………. Q4 Gender? □ Male □ Female Q5 Age bracket? □ Under 18 □ 18-35 □ 36-45 □ 46-55 □ 56-65 □ above 66 Q6 Village Q7Block Q8 Taluka Q9 District Q10 State Q11 Primary occupation □ Agriculture □ Animal husbandry □ Poultry □ Goat/Sheep rearing □ Agriculture labour □ Manual labour □ Migrant labour □ Government job □ Private job □ Other Q12 Any other, please specify…………………… Q13 How large is your farm/plot? (Plots)(Hectares) Q14 What is the ownership status of your plot? □ Entirely owned by self/family □ Entirely leased by Self /family □ Partly owned and partly leased □ Any other Q15 What crops did you grow in the last three years? □ Paddy □ Sugarcane □ Sorghum □ Indian bean □ Ragi □ Mango □ Sapota □ Banana □ Flowers □ Vegetables □ Plantation crops □ Any other Q16 Any other, please specify ……………………………………………………….. Q17 How much crop was harvested during last year? ……………………………. 152 Q18 Crop was harvested during which season?……………………………………… Q19 How much crop was retained at home? …………………………………………… Q20 How much crop was sold in the market?………………………………………… Q21 How much income was made from farming?……………………………………. Q22 How much income was made from other crops? ………………………………….. Q23 How much money did you spend on farming last year?. ………………………….. Q24 From where did you get money for farming? □ Own savings □ Bank loan □ Loan from SHG □ Friends/Family □ Agri input shop owners/traders □ Money lender□ Any other Q25 Any other, please specify …………………………………………. Q26 Do you use synthetic pesticides? Yes/No Q27 From when did you start using Newsil? Q 28 Have you taken up new crops because of Newsil? Yes/No Q 29 Have you earned more after using Newsil? Yes/No/Cannot say Q 30 If yes, how much have you earned from the crops? ………………… Q 31 How much did you spend on Newsil?…………………. Q 32 Are there any problems with Newsil?………………. Q 33 If yes, what are your suggestions in overcoming the same? ............... Q 34 What are your sources of irrigation? □ Canal □ River □ Pond □ Rainwater □ Dug well/Bore well □ Any other Q 35 What are the methods of irrigation? □ Flooding □ Drip □ Sprinkler □ Any other Q36 Any other, please specify ……………………….. Q37 Do you buy water? Yes/No 153 Q38 If yes, how much do you pay?……………… Q39 How did you hear about Newsil?……………………………….. Q40 Why did you buy Newsil?……………………………….. Q41 Do you know how Newsil works? □ Yes □ No □ Partly aware □ Cannot say Q42 Did you neighbors ask you about Newsil? Yes/No Q43 Did you inform them about Newsil? Yes/No Q44 Any other comments? ………………………………………….. Q45 Contact number………………………………………………….. Q46 Photograph ………………………………………………….. Q47 GPS location ………………………………………………….. 154 7. Questionnaire - Naireeta We are here about the bhungroo / naireeta services, to ask some questions to help with the project. Thank you so much for taking the time to meet and interview us. The answers that you give are very useful for Naireeta Services and will allow them to help you and other villages better. We now ask for your consent to collect the answers which you give to us. These may be used in the future for research and communications needs of Naireeta Services and their funding organization. *Wait for them to give consent* Please remember to give the most honest and accurate answers that you can as this will be the most helpful for us. This should take about 20 minutes. 1. What is your name? a. What is your education level? b. How many members do you have in your household? 2. What is your primary job? a. For how long have you had this job? 3. Do you have any other livelihoods / jobs? If so, what? a. Examples: Day worker, keeps buffalo b. For how long have you had this job? 4. How big is your farm in vigha? a. How much do you own? b. How much do you lease? c. Do you share any with family or community? 1. Did you farm this past year? 2. What crops did you grow? 3. How much of each crop did you collect? (Maund/kilos/etc) 4. Which seasons did you collect? 5. How much of the crops do you sell? 6. How much do you consume inside the home? 155 7. How much of your crops do you share outside the home? 8. How much income did you make from farming? 9. How much income did you make from other jobs? 10. How do you pay for the farm? For seeds, tools, fertilizer, pesticide, etc? a. Examples: From income/savings, loans, government assistance, family 11. How much money did you spend for the farm this year? 12. Do you use synthetic pesticides? 1. When was the bhungroo installed? a. Which season? 1. Do you expect to get any benefits from the bhungroo? *If no, skip following questions. If yes: * a. Do you expect to grow different or new crops? b. Do you expect to grow more crops? c. Will you sell more crops? Do you expect more income? i. If yes, how will you spend the extra income? d. Would you spend more, less, or the same amount of time farming? e. Is there anything else you will do differently if you get benefits from the bhungroo? 2. How much money did you spend on the bhungroo? 3. Will you invest in the maintenance of the bhungroo in the future? a. Answers: Yes, No, Not sure yet 1. Are there any problems with the bhungroo? 2. How could it be improved / do you have any suggestions? 1. What water source are you using for your crops now? Answers: Rain, rainwater harvesting, ground (well), surface water (pond, lake, river), commercial purchase 2. What method did you use to irrigate? Answer: Rain-fed, flood, drip, hand pump, underground diffuse, other 156 3. How much do you pay for water? 4. If the bhungroo can give you L of water during the dry season, would this be enough water for your use? 5. If not, will you keep using the other source of water or change? 6. Would you use the same, less, or more water with the bhungroo? 7. How would you spend additional income if gained from the bhungroo? 1. How did you hear about Naireeta/ bhungroo? 2. Why did you want one? 3. Do you know how the bhungroo works? 4. Has anyone (neighbors, visitors) asked about the bhungroo? a. Did you explain it to them? Any other comments or questions or requests? That is the end of the survey. Thank you so much for taking the time to speak with us. Can we please ask your permission to take a photo and a photo of the farm? Your photo will not be used to publicly identify you by name, but may be used to share your general successes and challenges with the bhungroo. 157 8. Questionnaire - Ignitia Interview Questions for Farmers using the Iska / Ignitia SMS service Q1 What is the name of your community? Q2 Names: (Optional) Q3 Gender? □ Male □ Female Q4 Age bracket? □ Under 18 □ 18-35 □ 36-45 □ 46-55 □ 56-65 □ above 66 Q5 Highest Education Level: □ None □ Primary □ Secondary □ Post-Secondary Q6 How many people are in your household? Q7 How large is your farm/plot? (Plots)(Hectares) Q8 What was the biggest challenge you faced in farming during last season? □ Unexpected rainfall □ Less (than expected) rainfall - Drought □ Cost of inputs (seeds , fertilizer etc.) □ Cost of Labour □ availability of machinery □ knowledge of new cost￾effective methods □ Transportation to market □ Spoilage □ Attacks by pests (e.g. army warms) □ Other (PLEASE SPECIFY) Q9 Is access to water a problem in your area? (Yes / No) Q10 Have weather patterns changed in recent years? (Yes / No) If yes, how? Q11 How hastheIska/IgnitiaSMSservicehelpedyou?Askthequestionwithoutpromptingand score these responses, ranking by importance 1, 2, 3: DO NOT READTHIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. □ Conserve water Y/N □ Make water reusable Y/N □ This helped women farmers as well as helping the men. Y/N □ They made a special effort to include woman farmers. Y/N □ It helped in producing more of our most important crop □ Grow cash crop(s) in saline water Y/N □ Grow more produce with less water Y/N 158 □ Increase my yield through accurate and timely forecasts Y/N □ Help with cost of inputs (seeds, fertilizer etc.) Y/N □ Improved health, strength and value of livestock Y/N □ Helping reduce time used, saved labour Y/N □ Reduce amount of water use for irrigation Y/N □ Reduce cost of production through reduced input cost, labour etc. Y/N □ Reduce crop wastage □ Other (PLEASE SPECIFY) Q12 Does the SMS service save water or are you using more water? (Save / Using more) Percentage change Q13 Are you producing more food by using the SMS service? (Yes / No) Across more seasons? (If yes, how do measure this increase?) Q14 Are you growing different kinds of crops by using the SMS service? (If yes, what kinds?) Q15 Why did you start using the Ignitia SMS service? Q16 When did you start using the SMS service? (less than 6 months ago, between 6 months and 1 year, between 1 and 2 years, more than 2 years) Q17 Does Ignitia SMS Service save you time? (Yes / No) Q18 Are all families in the area using the SMS service? (Yes / No) If no, why not? Q19 Are you paying for the SMS service yourself? (Yes / No) Q20 Is Ignitia SMS service affordable for you? (Yes / No) Q21 Is it affordable for poorer households in the community? (Yes / No) Q22 Are you more secure now with the SMS service in terms of food availability? (Yes / No) Q23 Are there any negative impacts on poor households in the community? (Yes / No) (If yes, please explain) Q24 Are there any negative impacts on women? (Yes / No) (If yes, please explain) Q25 Are there any negative impacts on other vulnerable members of the community? (Yes / No) If yes, please explain) Q26 Have there been special efforts to engage women to use the SMS service? (Yes / No) (If yes, please explain) 159 Q27 Is there anything that has resulted from your use of the SMS service that you didn’t expect? (Yes / No) If yes, please explain Q28 What other benefits would you expect from the SMS service provider? Training Maintenance Repairs Others (Explain ) Q29 Has the use of the SMS service improved your family income? (Please explain) □ Very much □ Significantly □ Very little □ Not at all □ Other benefits (PLEASE SPECIFY) Q30 Do you think you will still be using the SMS service in the long term (5 to 10 years)? □ No□ Yes If no, why? If yes, why? Q31 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Respondent Photograph GPS Coordinates 160 9. Questionnaire - ICBA-DRI Interview Questions for Farmers using the ICBA/DRI Seeds Drafted 8 August 2018 Q1 What is the name of your community? Q2 Names: (Optional) Q3 Gender? □ Male □ Female Q4 Age bracket? □ Under18 □ 18-35 □ 36-45 □ 46-55 □ 56-65 □ above 66 Q5 Highest Education Level: □ None □ Primary □ Secondary □ Post-Secondary Q6 How many people are in your household? Q7 How large is your farm/plot? (Plots)(Acres) Q8 What top three crops do you grow (ranked by profitability) (percentage)? Q9 What top three crops do you grow (ranked by volume)(percentage)? Q10 Whatwasthebiggestchallengeyoufacedinfarmingduringthelastgrowingseason? □1 Unexpected rainfall □ 2 Less (than expected) rainfall-Drought □ 3 Cost of inputs (seeds, fertilizer etc.) □ 4 Cost of Labour □ 5 availability of machinery □6 knowledge of new cost-effective methods □ 7 Transportation to market □8 Spoilage □ 9 Attacks by pests (e.g. army warms) □ 10Other(PLEASESPECIFY) Q9 Is access to water a problem in your area? (Yes / No) Q10 Have weather patterns changed in recent years? (Yes / No) If yes, how? Q11 How has the DRI seed innovation helped you? Ask the question without prompting and score these responses,rankingbyimportance1,2,3: DONOTREADTHISLIST:ratherlistenandbeopentopositiveandnegativeresponses;scorethefirstTH REE most important points positively or negatively. □ 1 Conserve water Y/N □ 2 Make water reusable Y/N □ 3 This helped women farmers as well as helping the men. Y/N □ 4 They made a special effort to include woman farmers. Y/N 161 □ 5 It helped in producing more of our most important crop □ 6 Grow cash crop(s) in saline water Y/N □ 7 Grow more produce with less water Y/N □ 8 Increase my yield through accurate and timely forecasts Y/N □ 9 Help with cost of inputs (seeds ,fertilizer etc.) Y/N □ 10 Improved health, strength and value of livestock Y/N □ 11 Helping reduce time used, saved labour Y/N □ 12 Reduce amount of water use for irrigation Y/N □ 13 Reduce cost of production through reduced input cost, labour etc. Y/N □ 14 Reduce crop wastage □ 15 Other (PLEASE SPECIFY) Q12 Where does the water for your farming come from? (Please specify ) Q13 Since using the DRI seeds, have you been able to save water or do you use more water? (Save / Using more) Percentage change Q14 If you use more water, is this to grow higher volumes of crops? Q15 Are you producing more food with the seeds? (Yes / No) Across more seasons? (If yes, how do measure this increase?) Explain Q16 Are you growing different crops with the seeds? (If yes, what kinds? ) Q17 Do you have access to ready markets for all of your produce you are trying to sell? Yes / No Please explain Q18 How did you learn about the seed program? Q20 When did you start using the seeds? (less than 6 months ago, between 6 months and 1 year, between 1 and 2 years, more than 2 years) Q21 Does using the DRI seeds save you time? (Yes / No) Q22 Does using the seeds improve your farming practices? (Yes / No) (Please explain) Q23 Are all families in the community using the seeds? (Yes / No) If no, why not? Q24 Is it affordable for you? (Yes / No) Q25 Is it affordable for poorer households in the community? (Yes / No / Not sure) 162 Q26 Are you more secure now with the seeds in terms of your family’s food availability? (Yes / No) Q27 In terms of the seed program, are there any specific benefits to women? (Please explain) Q28 Are there any negative impacts on poor households in the community? (Yes / No / Not sure) (If yes, please explain) Q29 Are there any negative impacts on women? (Yes / No / Not sure) (If yes, please explain) Q30 Are there any negative impacts on other vulnerable members of the community? (Yes / No / Not sure) If yes, please explain) Q31 Have there been special efforts to engage women to participate in the seed program? (Yes / No / Not sure) (If yes, please explain) Q32 Is there anything that has resulted from your use of the seeds that you did not expect? (Yes / No) If yes, please explain Q33 Are there any other benefits that you receive from the company? (Yes / No) If yes, please explain. (Could include credit, discounts on other inputs like fertilizer, etc.) Q34 Are there any other benefits you would like to see from the company? Training Maintenance Repairs Others (Explain ) Q35 Are there any harmful environmental effects that you are noticing after using the seeds for a period of time? (Yes / No) If yes, please explain Q36 Are you noticing any differences in the health of the soil after using the seeds for a period of time? (Yes / No) (If yes, please explain) Q37 Has the use of the seeds improved your family income? □ Very much □ Significantly □ Very little □ Not at all □ Other benefits (PLEASE SPECIFY) Please explain (for all responses) Q38 Do you think you will still be using the seeds in the long term (5 to10 years)? □ No □ Yes If no, why? If yes, why? Q39 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. 163 Respondent Photograph GPS Coordinates 164 DATA COLLECTION TOOLS USED IN REVISIT AND FIELD EVALUATORS SURVEYS, 2019 Generic Innovation Impact Evaluation Questionnaire Innovation Impact Evaluation, ICBA/DRI (Egypt) Innovation Impact Evaluation, CUT, SWAYIMANE Innovation Impact Evaluation, MNP, MATATIELE Online Impact Survey of Innovations: InternationalSample of SWFF and Non-SWFF Innovations The questionnaires used in the field by the Dexis local consultants and the SWFF Field Evaluators were calibrated in the generic tool to ensure the compatibility of data collected by both teams. This generic tool was modified for purpose in being applied to very different contexts; for example, MNP (Meat Naturally) is a pastoral innovation does not have the crops associated with other innovations. Three of such modifications to the open source survey (ODK) software used by the Dexis team. 165 GENERIC INNOVATION IMPACT EVALUATION QUESTIONNAIRE DEXIS+FIELD EVALUATORS Q 1 What is the name of your community? _____________________________________________________ Q 2 Site of location _____________________________________________________________________ Q 3 Names: _______________________________________________________ (Optional) Q 4 Gender ❑Male ❑Female Q 5 Income of household (annual)____________________Local currency___________USD ❑Wealthy ❑ Not poor ❑ Poor ❑Very poor Please see guidance in Dexis notes/manual Q 6 How many people are in your household? ________________________________________________ Q 7 How large is your farm/plot? _________________________ (Hectares) ❑ Large ❑ Medium ❑ Small ❑Very small Please see guidance in Dexis notes/manual Q 8 Do you own this land or rent?____________Own ___________Rent_____Both Q 9 If you rent, how much do you pay annually?___________Local currency ______USD Q 10 What crops do you grow? [Rank the first most important by number, 1, 2, 3] Crop 1 ❑ Benefitted ❑ Did not benefit from innovation Crop 2 ❑ Benefitted ❑ Did not benefit from innovation Crop 3 ❑ Benefitted ❑ Did not benefit from innovation Crop 4 ❑ Benefitted ❑ Did not benefit from innovation Crop 5 ❑ Benefitted ❑ Did not benefit from innovation Q 11 How much ofyour produce do you consume in your household? ___________% Q 12 (Credit) Using this innovation have you: a) Do not need to access credit b) Access to credit not improved c) Improved access to credit 166 d) Improved access to credit and able to repay over a short period Q 13 (Income) Has this innovation improved your household income? ❑ Not at all ❑ Somewhat ❑ Significantly ❑ Very significantly ❑ Other benefits (PLEASE SPECIFY) Q 14 (Satisfied) Would you recommend this innovation? a) Would not recommend b)Would recommend d)Would strongly recommend Q 15 (Access to water) Using this innovation have you: a) Had no change in your access to water b) Improved your access to water c) Fundamentally improved access to water Q 16 (Crop yield) Using this innovation have you: a) Yield has declined b) No change in crop yield c) Increased crop yield d) Substantially increased crop yield Q 17 (Crop/drop) Using this innovation have you: a) Used more water and achieved increased crop yield b) Had no change in use of water for same crop yield c) Used less water and achieved increased crop yield Q 18 Has there been a difference in the survival rates of crops before/after the innovation? Less resilient Same resilience More resilient Very resilient Other Q 19 How, if at all, have you changed your farming practices due to innovation? 167 No change Introduced new crops Changed irrigation system Reduced water usage It helps me to decide when to plant It helps me decide which crops to plant Other___________(specify) Q 20 Have changes in rainfall or temperature affected your farming practices or crop yields compared to your historical rainy/dry seasons? Y/N Please explain _______________________________________ Q 21 Are there any negative impacts from this Innovation in the community? ❑Yes, Major (Please explain – probe for issues, drawing, for example, on those below) ❑Yes, Minor (Please explain) ❑No (Increased debt levels, Created conflict within families about gender roles, Increased burden on women’s workload, Created divide in the community between those who can afford to invest and those who cannot, Created divide in the community between landowners and others in the community, Created environmental concerns through waste, Created conflict with buyers, Any other __________) Q 22 If there have been any negative impacts has the innovator helped to resolve these? ❑Yes ❑No ❑N/A Q 23 (Crop/drop) How has this innovation helped you? Ask the question without prompting and score these responses: DO NOTREAD THIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. ❑Makes water reusable Y/N ❑Helps women farmers as well as men. Y/N ❑They made a special effort to include woman farmers. Y/N ❑Helps in producing more of our most important crop Y/N ❑Increases my yield through accurate and timely forecasts Y/N ❑Helps by lowering cost of inputs (seeds, fertilizer, etc.) Y/N ❑Improves health, strength and value of livestock Y/N ❑Helping reduce time used, saves labor Y/N ❑Allows me to use previously unusable land Y/N ❑Reduces crop wastage Y/N 168 ❑Other (PLEASE SPECIFY) Q 24 (Sustainability) Do you think you’ll use the innovation on the long term (5 to 10 years)? ❑No ❑Yes If no, why? If yes, why? Please make a note of the main point. Q 25 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Q 26 GPS Q 27 Photo 169 INNOVATION IMPACT EVALUATION, ICBA/DRI (Egypt) Q 1 What is the name of your community? Village: _____________________________________________________________________ Q 2 Site of location: Block, District: _____________________________________________________________________ Q 3 Names: _______________________________________________________ (Optional) Q 4 Gender ❑Male ❑Female Q 5 Income of household ❑Wealthy ❑ Not poor ❑ Poor ❑Very poor Please see guidance in Dexis notes/manual Q 6 How many people are in your household? ________________________________________________ Q 7 How large is your farm/plot? _________________________ (Plots)(Hectares) ❑ Large ❑ Medium ❑ Small ❑Very small Please see guidance in Dexis notes/manual Q 8 What crops do you grow? [Rank the first most important by number, 1, 2, 3] Please see guidance in Dexis notes/manual; the customized questionnaires will propose more specific lists of crops. Crop 1 ❑ Benefitted ❑ Did not benefit from innovation Crop 2 ❑ Benefitted ❑ Did not benefit from innovation Crop 3 ❑ Benefitted ❑ Did not benefit from innovation Crop 4 ❑ Benefitted ❑ Did not benefit from innovation Crop 5 ❑ Benefitted ❑ Did not benefit from innovation Not applicable Q 9 How many cycles of seed have you used and harvested? ❑None so far ❑1-2 ❑3-4 ❑More than 4 Q 10 Have you made use of the DRI services for threshing? ❑Yes, and DRI purchases seed back from me ❑Yes, I take all the seed back ❑No, a buyer comes to purchase the grain from me 170 ❑No, I use the DRIseeds but process the grain and generate seeds on my own Q 11 Are there any negative impacts of the DRI seed program in the community? ❑Yes, Major (Please explain – probe for issues noted below) ❑Yes, Minor (Please explain) ❑No (Increased debt levels, Created conflict within families about gender roles, Increased burden on women’s workload, Created divide in the community between those who can afford to invest and those who cannot, Created divide in the community between landowners and others in the community, Created environmental concerns through waste, Created conflict with buyers, Any other __________) Q 12 If there have been any negative impacts of the seed service in the community (Q12), has DRI helped to resolve these? ❑Yes ❑No ❑N/A Q 13 (Credit) Using this innovation have you: a) Not improved access to credit b) Improved access to credit c) Improved access to credit and able to repay over a short period Q 14 (Income) Has this innovation improved your family income? ❑ Not at all ❑ Somewhat ❑ Significantly ❑ Very significantly ❑ Other benefits (PLEASE SPECIFY) Q 15 (Satisfied) Would you recommend this innovation? a) Would not recommend b)Would recommend d)Would strongly recommend Q 16 (Access to water) Using this innovation have you: a) Had no change in your access to water b) Improved your access to water c) Fundamentally improved access to water Q 17 (Crop yield) Using this innovation have you seen: 171 a) Yield has declined b) No change in crop yield c) Increased crop yield d) Substantially increased crop yield Q 18 (Crop/drop) Using this innovation have you: a) Used more water and achieved increased crop yield b) Had no change in use of water for same crop yield c) Used less water and achieved increased crop yield Q 19 How has this innovation helped you? Ask the question without prompting and score these responses, ranking by importance 1, 2, 3: DO NOTREAD THIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. ❑Makes water reusable Y/N ❑Helps women farmers as well as men. Y/N ❑They made a special effort to include woman farmers. Y/N ❑Helps in producing more of our most important crop Y/N ❑Increases my yield through accurate and timely forecasts Y/N ❑Helps by lowering cost of purchased inputs (seeds, fertilizer, etc.) Y/N ❑Improves health, strength and value of livestock Y/N ❑Helps reduce time used, saves labor Y/N ❑Reduces crop wastage Y/N ❑Allows me to use previously unusable land Y/N ❑Other (PLEASE SPECIFY) Q 20 (Sustainability) Do you think you’ll use the innovation on the long term (5 to 10 years)? ❑No ❑Yes If no, why? If yes, why? Please make a note of the main point. Q 21 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Q 22 GPS Q 23 Photo 172 INNOVATION IMPACT EVALUATION, CUT, SWAYIMANE Q 1 What is the name of your community? _____________________________________________________________________ Q 2 Site of location _____________________________________________________________________ Q 3 Names: _______________________________________________________ (Optional) Q 4 Gender ❑Male ❑Female Q 5 Income of household (annual)__________________Local currency________USD ❑Wealthy ❑ Not poor ❑ Poor❑Very poor Please see guidance in Dexis notes/manual Q 6 How many people are in your household? ________________________________________________ Q 7 How large is your farm/plot? _________________________ (Plots)(Hectares) ❑ Large ❑ Medium ❑ Small ❑Very small Please see guidance in Dexis notes/manual Q 8 Do you own this land or rent?____________Own___________Rent_____Both Q 9 If you rent, how much do you pay annually?___________Local currency ______USD Q 10 What crops do you grow? [Rank the first most important by number, 1, 2, 3] Crop 1 ❑ Benefitted ❑ Did not benefit from innovation Crop 2 ❑ Benefitted ❑ Did not benefit from innovation Crop 3 ❑ Benefitted ❑ Did not benefit from innovation Crop 4 ❑ Benefitted ❑ Did not benefit from innovation Crop 5 ❑ Benefitted ❑ Did not benefit from innovation Q 11 How much of your produce do you consume in your household? ___________% Q 12 Has there been a difference in the survival rates of crops before/after the innovation? Less resilient Same resilience More resilient Very resilient Other 173 Q 13 How, if at all, have you changed your farming practices due to innovation? No change Introduced new crops Changed irrigation system Reduced water usage It helps me to decide when to plant It helps me decide which crops to plant Other___________(specify) Q 14 Have changes in rainfall or temperature affected your farming practices or crop yields compared to your historical rainy/dry seasons? Y/N Please explain _______________________________________ Q 15 Would you be prepared to pay for INNOVATION NAME produce or service? YES/NO/THERE SHOULD BE NO CHARGE Q 16 If yes for the above question, how much would you be willing to pay for this service ______________LOCAL CURRENCY / EQUIVLENT IN USD Q 17 Q11Howdo you participate in the intervention? a) Subscribeto the cellphone messaging service b) Do not subscribe; get messages and advice from group c) Not a member of a group; follow what others are doing in my area Q 18 Are there any negative impacts from this Innovation in the community? g ❑Yes, Major (Please explain – probe for issues, drawing, for example, on those below) ❑Yes, Minor (Please explain) ❑No (Increased debt levels, Created conflict within families about gender roles, Increased burden on women’s workload, Created divide in the community between those who can afford to invest and those who cannot, Created divide in the community between landowners and others in the community, Created environmental concerns through waste, Created conflict with buyers, Any other __________) Q 19 If there have been any negative impacts has the innovator helped to resolve these? ❑Yes ❑No ❑N/A Q 20 (Credit) Using this innovation have you: 174 a) Do not need credit b) Access to credit not improved c) Improved access to credit d) Improved access to credit and able to repay over a short period Q 21 (Income) Has this innovation improved your family income? ❑ Not at all ❑ Somewhat ❑ Significantly ❑ Very significantly ❑ Other benefits (PLEASE SPECIFY) Q 22 (Satisfied) Would you recommend this innovation? a) Would not recommend b)Would recommend d)Would strongly recommend Q 23 (Access to water) Using this innovation have you: a) Had no change in your access to water b) Improved your access to water c) Fundamentally improved access to water Q 24 (Crop yield) Using this innovation have you: a) Yield has declined b) No change in crop yield c) Increased crop yield d) Substantially increased crop yield Q 25 (Crop/drop) Using this innovation have you: a) Used more water and achieved increased crop yield b) Had no change in use of water for same crop yield c) Used less water and achieved increased crop yield Q 26 (Crop/drop) How has this innovation helped you? Ask the question without prompting and score these responses, ranking by importance 1, 2, 3: DO NOTREAD THIS LIST: rather listenand be open to positive and negative responses; score the first THREE most important points positively or negatively. ❑Makes water reusable Y/N 175 ❑Helps women farmers as well as men. Y/N ❑They made a special effort to include woman farmers. Y/N ❑Helps in producing more of our most important crop Y/N ❑Increases my yield through accurate and timely forecasts Y/N ❑Helps by lowering cost of inputs (seeds, fertilizer, etc.) Y/N ❑Improves health, strength and value of livestock Y/N ❑Helping reduce time used, saves labor Y/N ❑Reduces crop wastage ❑Other (PLEASE SPECIFY) Q 27 (Sustainability) Do you think you’ll use the innovation on the long term (5 to 10 years)? ❑No ❑Yes If no, why? If yes, why? Please make a note of the main point. Q 28 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Q 29 GPS Q 30 Photo See excel file emerging on ………….. (Google Drive); please download at the earliest opportunity. 176 INNOVATION IMPACT EVALUATION, MNP, MATATIELE Q 1 What is the name of your community? Village_ ____________________________________________________________________ Q 2 Site of location, sub-village _____________________________________________________________________ Q 3 Names:_______________________________________________________ (Optional) Q 4 Gender ❑Male ❑Female Q 5 Income of household (annual)_________________local currency___________USD ❑Wealthy ❑ Not poor ❑Poor❑Very poor Please see guidance in Dexis notes/manual Q 6 How many people are in your household? ________________________________________________ Q 7 How large is your farm/plot? _________________________ (Plots)(Hectares) ❑Large ❑Medium ❑Small ❑Very small Please see guidance in Dexis notes/manual Q 8 Do you own this land or rent?____________Own ___________Rent_____Both Q 9 If you rent, how much do you pay annually?___________Local currency ______USD Q 10 Do grow more maize and vegetables BECAUSE OF THE INNOVATION. YES/NO Q 11 How many cattle do you own?--------------------------Number Categories: 0; 1-3, 4-7,8-9; 10-14,15-19, 20 and more Q 12 How have you participated in MNP: Tick all categories which apply. ❑My cattle are herded by EcoRangers ❑Have sold cattle at MNP auction ❑My crops in the field are protected against cattle ❑I benefit by improved grass cover and improved water retention Q 13 (Cattle) I have benefited from herding and auction: Skip if no cattle or do not use herding or auctions Applicable. ❑ Yes ❑ No 177 My cattle are healthier and stronger. ❑ Yes ❑ No I can get a good price when selling cattle. ❑ Yes ❑ No Have increased my herd of cattle. ❑ Yes ❑ No Q 14 How much o f your produce do you consume in your household? Crops ___________% Livestock ___________% Q 15 (Credit) Using this innovation have you: a) Do not need credit b) Not improved access to credit c) Improved access to credit d) Improved access to credit and able to repay over a short period Q 16 Has there been a difference in the survival rates of CATTLE before/after the innovation? Less resilient Same resilience More resilient Very resilient Other Q 17 Have changes in rainfall or temperature affected your CATTLE REARING practices or crop yields compared to your historical rainy/dry seasons? Y/N Please explain _______________________________________ Q 18 Are there any negative impacts from this Innovation in the community? ❑Yes, Major (Please explain – probe for issues, drawing, for example, on those below) ❑Yes, Minor (Please explain) ❑No (Increased debt levels, Created conflict within families about gender roles, Increased burden on women’s workload, Created divide in the community between those who can afford to invest and those who cannot, Created divide in the community between landowners and others in the community, Created environmental concerns through waste, Created conflict with buyers, Any other __________) Q 19 If there have been any negative impacts has the innovation helped to resolve these? ❑Yes ❑Nos ❑N/A Q 20 (Income) Has this innovation improved your household income? 178 ❑Not at all ❑Somewhat ❑Significantly ❑ Very significantly ❑Other benefits (PLEASE SPECIFY) Q 21 (Satisfied) Would you recommend this innovation to your neighbors? a) Would not recommend b)Would recommend d)Would strongly recommend Q 22 Does your community experience water scarcity? ❑ No ❑ Yes ❑ Yes, severely Q 23 (Access to water) Recently have you: a) Had no change in your access to water b) Improved your access to water c) Fundamentally improved access to water Q 24 (Access to water) Using this innovation have you: a) Had no change in access to water for your cattle b) Have improved access to water for cattle c) Fundamentally improved water sources community and for cattle Q 25 How has this innovation helped you? Ask the question without prompting and score these responses, ranking by importance 1, 2, 3: DO NOTREAD THIS LIST: rather listen and be open to positive and negative responses; score the first THREE most important points positively or negatively. ❑Makes water reusable Y/N ❑Helps women farmers as well as men. Y/N ❑They made a special effort to include woman farmers. Y/N ❑Helps in producing more of our most important crop Y/N ❑Increases my yield through accurate and timely forecasts Y/N ❑Helpsby lowering cost of inputs (seeds, fertilizer, laboretc.) Y/N ❑Improves health, strength and value of livestock Y/N ❑Helping reduce time used, saves laborY/N 179 ❑Reduces crop wastage Y/N ❑Other (PLEASE SPECIFY) Q 26 (Sustainability) Do you think you’ll use the innovation on the long term (5 to 10 years)? ❑No ❑Yes If no, why? If yes, why? Please make a note of the main point. Q 27 Are you willing to share your contact with us? We guarantee that your information will not be shared and will only be used to eventually contact you for any further questions we might have. Q 28 GPS Q 29 Photo See excel file emerging on ………….. (Google Drive); please download at the earliest opportunity. 180 ONLINE IMPACT SURVEY OF INNOVATIONS: AN INTERNATIONAL SAMPLE OF SWFF AND NON-SWFF INNOVATIONS 1. What is the name of your organization? Name and country 2. How many people are in your organization (country 1) Country Male Female Females in Management/Executive 3. How many people are in your organization (country 2) Country Male Female Females in Management/Executive 4. Please select up to five keywords that best describe your objectives and innovation a. Water efficiency b. Green technology c. Infrastructure d. Water access e. App (mobile or other) f. Irrigation g. Improved water access h. Gender integration i. Access to credit j. Women customers k. ICT l. Green energy m. Extremely vulnerable/extreme poor 5. How would you characterize your organization? a. Non-profit organization (consider these elements: tax exempt, social enterprise, prioritize benefits to a specific group, community development, cross-subsidization to include the poorest households, etc.) b. For-profit organization (consider these elements: registered as a company, give priority to return on capital, expand markets where-ever available, etc.) 181 6. Have you received SWFF support? Yes/No 7. Please indicate how you would characterize your innovation (using a maximum of two relevant codes) a. Sells a product to customers b. Sells a service to customers c. Sell IT/ICT service to customers d. Provides agriculture or related infrastructure for ongoing use e. Other (please specify) 8. How has your innovation developed over the past 5 calendar years? (Please tick the one box for each column that your innovation has been operating) Please indicate from the following for each year. a. Development stage establishing a functioning example, no customers b. Initial pilot stage (limited testing): some but less than 100 customers c. Multiple pilots: 100-500 customers d. Early commercialization: widening the customer base, >500-1000 customers e. Commercialization: broadened customer base, >1000-100,000 Please any additional information about the evolution of your innovation 9. For each area of technical support received, from various entities, how effective was each support? Please respond for all types of support received from SWFF, non-SWFF, and internal entities using these categories: Very effective, Effective, Slightly effective, Not at all, N/A, a. Product design and production b. Communications and/or branding c. Partnerships and networking d. Business Practices/Organizational capacity e. Access to finance 10. In the last five years, how much funding have you received for your innovation? Indicate 1) No funding 2) Less than 25k, 3) 25-50k, 4) 50-100k, 5) 100-200k, 6) 200+k from each of the following: a. Public Funding (Government, donor or foundation grants, concessional loans, etc.) b. Private funding (commercial loans, equity investments, etc.) c. Self (savings) 11. As a result of SWFF or other innovator/accelerator technical support (i.e. not funding), have you been able to: 182 a. Significantly increase your female customer base b. Increase your production capacity c. Enhance your business and financial practices in terms of managing a company d. Leverage private or public and foundation funding e. Significantly improve access to water for customers/beneficiaries f. Significantly improve access to credit for customers/beneficiaries g. Significantly improve water efficiency of customers/beneficiaries h. Significantly improve agricultural productivity of customers/beneficiaries 12. Overall, compared to other innovator support programs with which you have worked, the added value that SWFF’s technical support brings is: a. Significantly less b. Somewhat less c. About the same d. Somewhat higher e. Significantly higher f. N/A 13. If you have received SWFF support, in comparison to other programs, what was the level of effort in undertaking the following: Use these criteria in assessing their effectiveness in terms of the following activities below: a. Significantly more b. Somewhat more c. About the same d. Somewhat less e. Significantly less Activities: a. Application process b. Financial reporting c. Activity reporting and monitoring d. Building/achieving partnerships e. Agreeing technical assistance/working with TA providers f. Hosting visits g. Other forms of support h. Other (please specify) 14. Are there any additional points you would like to make, your contribution will be valued: 183 ANNEX 7 TABLES OF FIELD VISIT KIIS Table 1. Gender Repartition by Innovations (in-countries surveys), 2019 # Innovation Respondents Male Percentage Female Percentage 1 Aquaponics Farming (WGI), Rd 3 10 3 70% 7 70% 2 CUT ITIKI, Rd 4 4 2 50% 2 50% 3 CUT, Rd 4 23 5 22% 18 78% 4 Ecorangers, Rd 1 27 21 78% 6 22% 5 Green Heat, Rd 3 8 4 50% 4 50% 6 Hydroponics Africa, Rd 3 13 2 15% 11 85% 7 ICBA, Rd 1 24 24 100% 0 0% 8 Ignatia, Rd 3 14 8 57% 6 43% 9 Lal Teer, Rd 3 25 20 80% 5 20% 10 Naireeta, Rd 3 25 1 4% 24 96% 11 NewSil, Rd 3 10 8 80% 2 20% 12 Skyfox, Rd 4 10 8 80% 2 20% 13 SWAR, Rd 3 6 6 100% 0 0% 14 WASTE, Rd 4 22 5 23% 17 77% Total 221 112 95 Average 16 9.3 57% 7.9 43% 184 Table 2. Stage of Development of Innovations (Scale, in-countries surveys) # Innovations No plan scale up place to in Initial plan in place but evidence largely anecdotal Early plan in place with quantitative evidence collected Matured plan in place for scale Replication and expansion in place with clear evidence 1 Aquaponics Rd 3 Farming (WGI), X 2 CUT, Rd 4 X 3 CUT ITIKI, Rd 4 X 4 Ecorangers (MNP), Rd 1 X 5 Green Heat, Rd 3 X 6 Hydroponics Africa, Rd 3 X 7 ICBA, Rd 1 X 8 Ignitia, Rd 3 X 9 Lal Teer, Rd 3 X 10 Naireeta, Rd 3 X 11 NewSil, Rd 3 X 12 Skyfox, Rd 4 X 13 SWAR, Rd 3 X 14 WASTE, Rd 4 X Total 0 3 8 2 1 Percentage 0% 21% 57% 14% 7% 185 Table 3. Revisit surveys undertaken by Dexis (in-countries surveys), 2019 # Innovation Country Revisit (12 innovations) Respondents N Male% (N) Female% (N) 1 CUT ITIKI Kenya 31 35% (11) 65% (20) 2 CUT Swayimane South Africa 31 3% (1) 97% (30) 3 Greenheat Uganda 35 40% (14) 60% (21) 4 Hydroponics Africa Kenya 34 20% (7) 80% (27) 5 ICBA Egypt 19 100% (19) 0% (0) 6 Lal Teer Bangladesh 31 94% (29) 6% (2) 7 Naireeta Bhungroo India 25 32% (8) 68% (17) 8 Newsil India 21 57% (12) 43% (9) 9 Practical Action Bangladesh 30 0% (0) 100% (30) 10 Skyfox Ghana 34 76% (26) 24% (8) 11 Waste India 26 58% (15) 42% (11) 12 WGI (Aquaponics) Uganda 30 47% (14) 53% (16) Overall 347 156 191 Average 29 47% (13) 53% (16) 186 Table 4. Gender Repartition by Innovations (in-countries surveys), 2019 # Innovation Country FE (11 innovations) Respondents N Male% (N) Female% (N) 1 Aqysta Nepal 52 85% (44) 15% (8) 2 AST India 50 88% (44) 12% (6) 3 Aybar Ethiopia 50 90% (45) 10% (5) 4 CUT ITIKI Kenya 62 27% (17) 72% (45) 5 CUT Swayimane South Africa N/A N/A N/A 6 Greenheat Uganda 58 52% (30) 49% (28) 7 Hydroponics Africa Kenya 48 22% (11) 77% (37) 8 ICBA Egypt N/A N/A N/A 9 Lal Teer Bangladesh N/A N/A N/A 10 MNP (Ecorangers) South Africa 65 75% (49) 24% (16) 11 Naireeta Bhungroo India 62 2% (1) 98% (61) 12 Newsil India N/A N/A N/A 13 Practical Action Bangladesh N/A N/A N/A 14 Skyfox Ghana 59 59% (35) 40% (24) 15 Waste India 56 58% (33) 41% (23) 16 WGI (Aquaponics) Uganda 48 72% (35) 27% (13) Overall 610 344 266 Average 55 58% (31) 42% (24) 187 Table 5. Stage of Development of Innovations (Targets Met, in-countries surveys) # Innovations Not meetin Early stage, i.e. pre￾scalin Meetin g some targets in Meetin g all targets in Exceedin g targets g up stage scaling up scaling up g scaling targets 1 Aquaponics Farming (WGI), Rd 3 X 2 CUT, Rd 4 X 3 CUT ITIKI, Rd 4 X 4 Ecorangers (MNP), Rd 1 X 5 Green Heat, Rd 3 X 6 Hydroponics Africa, Rd 3 X 7 ICBA, Rd 1 X 8 Ignitia, Rd 3 X 9 Lal Teer, Rd 3 X 1 0 Naireeta, Rd 3 X 1 1 NewSil, Rd 3 X 1 2 Skyfox, Rd 4 X 1 3 SWAR, Rd 3 X 1 4 WASTE, Rd 4 X Total 1 3 5 3 2 Percentage 7% 21% 36% 21% 14% 188 Table 6. Stage of Development of all Innovations (Scale, data from surveys & KII interviews) # Innovations No plan to scale up in place Initial plan in place but evidence largely anecdotal Early plan in place with quantitative evidence collected Matured plan in place for scale and expansion in place with clear evidence 1 Aquaponics (WGI), Rd 3 Farming X 2 Aqysta, Rd 1 X 3 AST, Rd 1 X 4 CUT ITIKI, Rd 4 X 5 CUT, Rd 4 X 6 Ecorangers (MNP), Rd 1 X 7 FutureWater, Rd 1 X 8 Green Heat, Rd 3 X 9 Hydroponics Africa, Rd 3 X 10 ICBA, Rd 1 X 11 Ignitia, Rd 3 X 12 Lal Teer, Rd 3 X 13 MimosaTek, Rd 4 X 14 Naireeta, Rd 3 X 15 NewSil, Rd 3 X 16 Practical Action, Rd 1 X 17 Project Alba, Rd 4 X 18 Skyfox, Rd 4 X 19 SWAR, Rd 3 X 20 WASTE, Rd 4 X 21 WorldHope, Rd 1 X Total 0 5 9 4 3 Percentage 0% 24% 43% 19% 14% Replication 189 Table 7. Stage of Development of all Innovations (Targets Met, data from surveys & KII interviews) # Innovations Not meetin Early stage, i.e. pre￾scalin Meetin g some targets in Meetin g all targets in Exceedin g targets g up stage scaling up scaling up g scaling targets 1 Aquaponics Farming (WGI), Rd 3 X 2 Aqysta, Rd 1 X 3 AST, Rd 1 X 4 CUT ITIKI, Rd 4 X 5 CUT, Rd 4 X 6 Ecorangers (MNP), Rd 1 X 7 FutureWater, Rd 1 X 8 Green Heat, Rd 3 X 9 Hydroponics Africa, Rd 3 X 10 ICBA, Rd 1 X 11 Ignitia, Rd 3 X 12 Lal Teer, Rd 3 X 13 MimosaTek, Rd 4 X 14 Naireeta, Rd 3 X 15 NewSil, Rd 3 X 16 Practical Action, Rd 1 X 17 Project Alba, Rd 4 X 18 Skyfox, Rd 4 X 19 SWAR, Rd 3 X 20 WASTE, Rd 4 X 21 WorldHope, Rd 1 X Total 1 4 8 6 2 Percentage 5% 19% 38% 29% 10% 190 ANNEX 8 SUSTAINABILITY OF INNOVATIONS: EVIDENCE FROM PERFORMANCE EVALUATIONS To provide further insight into the nature of sustainability the performance evaluations of innovations which have now graduated from SWFF have been examined. These include the following 8 innovations which were evaluated between 2-3 years following the conclusion of SWFF funding. These evaluations were made of the innovations in the table below. Unfortunately, there is little available data to make a similar analysis of the sustainability of the Alumni. Innovation Current status Years post-SWFF funding 1 Aybar Graduate/Rd. 1 3 2 aQysta Graduate/Rd. 1 3 5 AST Graduate/Rd. 1 3 6 World Hope Graduate/Rd. 1 3 7 Future Water Graduate/Rd. 1 3 3 Green Heat Graduate/Rd. 3 2 4 Meat Naturally Graduate/Rd. 3 2 8 WGI Graduate/Rd. 3 2 There are a number of questions relating to sustainability which have been raised in this evaluation. Why have these innovations flourished or stagnated? Have the innovations which graduated continued further on a course towards financial sustainability or have they simply sustained their operations? Has the additional project life of the graduates moved them on to maturity with a broader base of users with a greater willingness to pay? The following concepts have been used to extract the data and analyze the evidence from the evaluations: Affordability and Willingness to pay, Commitment to use and advice and Profitability and Durability. Each of the evaluations were thorough but they did not systematically examine the qualities of profitability and financial flows and relate these to the size of the user base. Profitability or financial sustainability was not generally mentioned while sustainability was mentioned to some extent in relation to environmental factors. However, in each evaluation the main emphasis is on affordability and use of the innovation which is covered in other columns. The concepts are set out as follows: Affordability and Willingness to pay The entries here include indications of whether the innovation is free or readily accessible and/or the ability and willingness to pay. Free provision complicates the analysis as it may indicate high use but also no indicates no revenue stream and limited expansion of users as each user adds to the cost. “High” would indicate a high level of commitment to use and pay for its use (particularly of users sharing cooperatively); “moderate”, some reporting of lack of willingness to pay but continual use and “low”, low levels of willingness to pay and uncertain future use. 191 “Free” would indicate that no revenue was being accrued from users; revenue may, however, be the form of contracts or other forms of government assistance. Commitment to use and Promote Entries here include indications of commitment to continue using the innovation and to encourage other farmers to use the innovation. ‘High” would include mention of group participation to pay for the innovation, put forward suggestions for improvement and supporting initiatives associated with the innovation; “low” a lack of these indications. Profitability and Financial Sustainability A key result of the SWFF program is the planned achievement of sustainability. The performance evaluations are found not to include environmental sustainability nor thorough examination of financial sustainability, but they do capture the responses of end-users. The financial status of innovations is, however, constantly assessed by SWFF TAF in reports and exchanges. The Social Rate of Return Analysis provides a rigorous examination of the interweaving of committed funding and social and agricultural returns. An innovation has an improved probability of achieving financial sustainability where there is high level of paid use and the innovation achieves progress towards a larger market. An innovation here is financially sustainable if it has a reasonably high level of demand, has users who are committed to use the innovation well into the future, are also willing to pay and encourage others to use the innovation. When the innovation has stimulated and secured demand and increased supply it has the basis for financial sustainability. This is accomplished when the revenue generated is more than covers costs without needing subsidization. In reality this would be a high-performance financial sustainability or mature commercialization. To assess the range of financial sustainability, the term durability is used. Durability is possibly a more practical measure of performance of an innovation, which may range between being very unsure to achieving financial sustainability. The time duration between being fully funded and continued operation is a robust and simple measure of durability. A lack of sustainability would be indicated particularly by weak demand and willingness to pay, more certain sustainability by a combination of expanding demand and profitable returns. Somewhere between these poles would be those “durable” situations where, despite poor revenue, there are prospects for survival through, for example, on-going subsidies or the possibility of becoming absorbed in agricultural extension services. Since these supporting features are not reported in evaluations, where there no evidence, the term “unsure” is used. Where these innovations have continued for some time even under these conditions, then the innovation is described as “durable”. A table setting out the assessments of the 8 graduate innovations under the three key headings is presented below. These assessments are summed up as High, Moderate or Low and a short version of the supporting evidence is presented immediately below. 192 Table 8. Analysis of Affordability, Use and Financial Sustainability with supporting evidence Affordability/Willingness to pay Commitment to use/Promote Profitability/durability AQYSTA HIGH HIGH DURABLE/MEETING COSTS Users make down-payment and pay in installments. Government subsidy for remainder who pay. 80% committed to continue into the future and proposals for improvement. 94% strongly recommend. Operational 3 years post-SWFF but possible net neutral revenue. AST HIGH/MODERATE HIGH BEYOND Seed cheaper than conventional. Price for poorest discounted. Some willingness to pay a minority is prepared to continue to pay if price is halved. Long term recurring us: same or higher yield than conventional seed. Overwhelming majority used for more than one season. 84% report increased yield and 86% improved income. DURABLE/PROFITABLE Continued revenue stream based strong demand and durable in 14 countries 3 years post-SWFF. on AYBAR HIGH/MODERATE Willingness to pay evident with considerable expansion with users paying. Some only at a lower cost and with modification. HIGH Strong commitment among users to continue suggested modifications. More than half users borrow or share use of the innovation. DURABLE/CONTINUING Increasing demand and innovation operational 3 years post-SWFF funding. FUTURE LOW LOW/NOT WILLING TO NOT DURABLE/UNSURE WATER Innovation free. It informs users PAY Operational 3 years post-SWFF but of problems in their farms, but solutions not affordable. 29% of users state they cannot afford the service at any price. Users do not pay and unsure what they could afford. 66% want innovation to continue while remainder are against or unsure whether innovation was really beneficial. Affordability not an individual attitude. no revenue stream and does not appear sustainable. GREEN HEAT HIGH/MODERATE Innovation provides infrastructure; requires sufficient land and animals. Addition of human waste. Clear willingness to pay. HIGH/PROMOTE Clear Improvement and adaptation proposed by users. Promote innovation. BEYOND DURABLE/PROFITABLE Durable 2 years post-SWFF has expanded to 2 other countries and enjoys continuing contractual funding. MNP HIGH HIGH/MODERATE BEYOND Users access community grazing and accept fee charged for cattle sold at auction. ACCURATE Users accept the MNP approach: healthy rangelands will produce increased quantity and quality of livestock, which, with improved market access, DURABLE/PROFITABLE MNP is a for-profit company that is well established. Further innovation with rural cattle slaughterhouse will improve profitability. 193 Affordability/Willingness to pay Commitment to use/Promote Profitability/durability will improve returns for rural livelihoods. Users do not share costs but, individually, pay a fee for cattle sales. WGI MODERATE/AVAILABLE FREE Down-payment for 90% including fish tank, pipes and taps, half drums and stock of fingerlings. Innovation available in various sizes for poorer/richer farmers. Return on investment over one year. HIGH/IMPROVED CREDIT Users use their own savings in building a shelter, paying for water and fish feed. Access to credit had improved due to the innovation. Fish farming highly valued in community. Fish deaths, dirty tanks, and problems with stocks are reported. Self-sufficiency: most (more than 80 percent) farmers are eating much of what is produced and selling very little. DURABLE/NOT PROFITABLE Durable 2 years post-SWFF, has revenue stream in continuing demand. Continuing operation. WORLD LOW HIGH VERY UNSURE HOPE Provided free for some. 25% greenhouses not maintained and repaired. 20% pay full price. Recommendation selection criteria for farmers getting greenhouses and confirmed commitment to use. 92% of users share information on greenhouses. Strong interest in bigger greenhouses and more agricultural advice and training. Promotion leading to farmers building themselves. No revenue stream identified unlikely to be durable. Off shoot business continuing to make greenhouses. Frequent monitoring to ensure the project’s aims are being achieved. Source: SWFF Performance Evaluations, Social Rate of Return study and Annex A, 2019 Annual Report. The analysis of affordability, commitment and financial sustainability has been undertaken. The higher level of affordability (Aqysta and MNP) and high/moderate levels (WGI; AST, Aybar and Green Heat) are scored from high commitment to use and willingness to pay. In some innovations like Green Heat there is relatively limited access to the biogas tank infrastructure, but the use is high. Unfortunately, there is also evidence of the destruction of a proportion of the installed infrastructure. Low levels of affordability (World Hope and Future Water) are linked to uncertain or no revenue stream and are linked to unsure or very unsure financial sustainability. Virtually all innovations score continued commitment to use and promote the innovation among other farmers (Aqysta, Aybar, Green Heat, MNP, World Hope, and WGI). The only exception is Future Water which has a low commitment to use. Free access to the innovation is not necessarily linked to high levels of use or willingness to pay in the future, as indicated by Future Water. In relation to financial sustainability, the following innovations were found to be both durable and profitable: AST, Green Heat and MNP. Another group were durable but not necessarily 194 profitable: Aybar and Aqysta. Finally, there were three with unsure financial sustainability (World Hope, WGI and Future Water) were also found to unsure durability and profitability (despite still being operational 2 or 3 years after receiving SWFF funding). The general conclusion that can be drawn is that with some exceptions, those innovations which have graduated from SWFF, that is completed 3 years receiving funding and technical assistance, have continued operating 2-3 years after this support. Virtually all have attracted and held the commitment of users to the innovation and most have further expanded their user base. A number have continued as durable and profitable innovations, a further section as durable and meeting their costs but not necessarily profitable. The concluding 3 are assessed to be of unsure financial sustainability; largely, it appears, because of a low willingness to pay. 195 ANNEX 9 STATEMENT OF DIFFERENCE Final Performance Evaluation of Securing Water for Food Grand Challenge for Development Responses to Comments by Swedish International Development Cooperation Agency (Sida) and MFA Netherlands on the Final Report Dexis Consulting Group David Hemson, Ken Caplan, Stephanie Monschein and Nohemi Voglozin. August 13, 2020 196 Swedish International Development Cooperation Agency (Sida) Feedback 1. In the sustainability section it is mentioned that, although minor, some negative effects on women and poor have been reported, including increased burden on women, increased debt levels and creation of conflict within family about gender roles or creation of conflict within the community on the order of use of innovation or jealousy from other members of the community who don’t have access to the innovations or who cannot afford them. For Sida a do￾no-harm approach in all our interventions are essential, so we would welcome any recommendations on how we could try to prevent these negative effects. As for potential negative effects on environment/climate, we have already taken considerable precautionary measures in the new WE4F programme, but not so much with regards to gender and the poor/communities. OUR RESPONSE: Unfortunately, it is not possible to anticipate unintended negative consequences of an innovation or activity which may contradict the spirit of the intervention. Performance evaluations can help in providing feedback on such consequences to identify, address, and pivot innovations as needed to ensure all rounded positive impact. They also help in revealing the reasons for such consequences once they are identified. To address the concrete issue: the evidence from surveys was that farmers reported an additional burden of work, especially for women. Although this was not an intended consequence it does follow from the priorities of SWFF to both include women in production and to achieve higher agricultural productivity. The following points should be noted: firstly the negative impacts are stated by respondents to be minor, secondly these are recorded in surveys and the reasons are not directly requested, and thirdly that these negative impacts are also associated (and counter-balanced) with a number of positive impacts. The reasons for the additional burden of work for women is identified as “minor” but can be explained by the course of the innovation. Since users generally report 2 or more adaptations in farming practices (such as a wider range of crops grown and the acquisition of cattle) there is likely to be additional work. An increase in agricultural productivity by these methods is likely to make for an overall increase in work by women. How could WE4F mitigate such negative impact? Firstly, it should be understood that there is some counter-balance; the additional burden leads to higher agricultural productivity, increased income and a probable diversification of income sources. Secondly, WE4F could prepare better by anticipating such impact being associated with the positive impact and include this subject to be reported on in monitoring and evaluation. Thirdly, feedback on any additional burden on women (or any other negative impacts) could be better understood and responded to if there were more direct inter-relationship between survey methods (where such impacts are measured) and qualitative methods (where the reasons for such impacts could be further explored). 197 2. In terms of financing, the report mentions that with time SWFF innovators showed greater use of public funds and lower use of private and own funding compared to non-SWFF innovators. Does this possibly imply that SWFF has not been additional but rather contributed to crowing out private capital (in the “mature” phase)? OUR RESPONSE: From the analysis of sources of funding we found that innovations have been able to secure funding from all sources, but more from public and self-funding. We measured the proportion of funding from each of these sources and found that private funding is lower than the others. This raises the question of why this should be; has private funding been “crowded out” by the availability of public funds? We report that since private donors are more interested not only in innovations that are “mature” (assessed in terms of a widening base of users) but also by profit. Although by the end of their award years, a number of innovations had switched from “not for profit” to “for profit”, this had not attracted more funding from private donors. Indeed on graduation many innovations, including for profit innovations, still receive more funding from public sources than private. This appears as something of a contradiction and it is. The following observations could shed more light to the subject. Firstly, in analysis and in future work WE4F needs to define what the private sector is. USAID has useful policy on the question of engaging the private sector; this sets out a strategy to consult, strategize, collaborate with the private sector to achieve development goals. 25 This may provide ideas for WE4F to explore even though it may not directly answer the question of private sector funding as directly posed in the SWFF program. Part of the challenge is to understand the different dimensions of the private sector which potentially includes the following: private venture capital, banks, impact investors, corporate social responsibility programs (CSR), philanthropic agencies, marketing agencies, etc. In addition, there are also various elements of the “third sector” or large NGOs which may be relevant. It is not always obvious or unambiguous to know which funding is “public” or “private” and in what dimension. Secondly, WE4F needs to think through engaging private sector funding from the perspective of the various types of private sector and the innovations themselves. It is certain that, with the exception of CSR programs, the private sector will expect a positive return on investment. This raises two particular questions: what terms would the private sector want to exact in funding and whether innovations would be prepared to concede equity, a share in the ownership, to the private sector. 25 USAID. Private sector engagement policy. https://www.usaid.gov/sites/default/files/documents/1865/usaid_psepolicy_final.pdf 198 The recommendations on page 84 suggests ways to improve engagement of the private sector in future programs: “Develop with innovators a customized funding strategy that will offer a balance between profit and social engagement. It will allow them to appeal to a larger and more diverse pool of funding sources. Innovations should not only be financially viable, for-profit entities but also socially engaged and sustainable.” This paragraph includes mention of elements which could be pulling in different directions e.g. social engagement and profit. Possibly attracting more impact investors which intend to generate a measurable social or environmental impact along with a financial return could SWFF-funded innovations. Thirdly, there is the question of innovations being open to sharing equity. The attraction of making equity available to private capital is likely to receive differing views from innovators but it would be an interesting possibility to explore for technical support. In a previous evaluation undertaken by Dexis for USAID in innovation (HESN), the team suggested that this avenue be explored. The WE4F needs to get the Technical Support arm to support innovations in thinking through a funding strategy, preparing pitch materials or grant proposals, and practicing pitches for equity, debt or grant funding. This could be added as an option as it is a modality typically used by innovations in funding strategies. 3. It is mentioned that financing was the biggest contributor to outcomes and results in the programme, but also that in some instances TA was considered more important. It would be interesting to know more about in which cases TA was considered more decisive in achieving results? OUR RESPONSE: The relative contribution of Technical Assistance (TA) to achieved results in comparison to that of funding has been widely canvassed in the evaluation. TA was considered more important by SWFF respondents compared to funding in the KII and online survey. In both the quantitative and qualitative responses, the TA was identified as the key determining factor for innovation success, not funding outright. Contrasting the comparative contribution of financing to TA is difficult. Funding is essential to an innovation and TA accompanies this basic component. It is possibly better to think of in terms of the added value of the TA to added funding after the inception stage; in achieving the organizational strengthening in breaking into a new level of operation, technical support contributes most heavily to future sustainability. Many innovation can succeed while receiving money, but it’s the technical support that gives innovators the tool to do so independently. The online survey allowed a comparative perspective on SWFF TA and Non-SWFF TA. The SWFF TA were found significantly beneficial than non-SWFF TA: 78% of respondents with both SWFF and non-SWFF experience, rated SWFF technical support (not funding) as somewhat and significantly higher than non-SWFF. 199 The survey data and the interviews provide a slightly different sample, however, with the survey including alumni and graduates. This could paint a slightly different picture since pages 63-64 focus on the KIIs conducted at World Water Week and through site visits. 4. When it comes to including more partners from the South in the new programme, it would also be good to know a bit more what types of partners the evaluators have in mind (donors, governments, academia, networks, etc.?). OUR RESPONSE: SWFF and now WE4F faces a wide range of possible international and local partners which could help advance the work and provide support to innovations. It is understood that the WE4F initiative might not have the bandwidth to go country by country to deliver on its goals. Rather than survey the landscape anew, it is proposed to draw in existing USAID missions to identify or and help make connections to local actors by sector or focus of the innovations. USAID Missions have such knowledge and sometimes also conduct mapping exercises of networks and actors in government, academia, and in civil society (NGOs and CSOs,) etc. Other types of partnerships would be with organizations that have been operating in developing and emerging countries for some time and that already have established partnerships with different stakeholders along the agricultural spectrum (farmers, extension agents, research institutes, academia, NGOS, civil society, etc.). Such knowledge could be paired with contacts of the Swedish International Development Cooperation Agency (Sida) and Netherlands in country. This could provide a wide-ranging network of partners from which to draw expertise and match against the specific challenges of innovations. In the spirit of CLA, Journey to Self Reliance, and Local Systems foci for USAID (and other priorities for other donors) this would be a step towards partnerships. That said, the embassies and missions could bring a lot of exposure and connect WE4F to relevant activities in country. Knowledge of such potential partnerships would help WE4F build a field of support and expertise for the TAF and for the innovations. There is a particular challenge to embed WE4F in countries and regions in emerging regions. WE4F could liaise with the African and Asian Development Banks, regional organizations like SADCC, associations of regulators, and groups like the African Ministers Conference on Water (AMCOW) to provide information, seek alignment, help get the word out through their networks to help identify possible innovators to apply for the program. In addition to these regional bodies, there are agriculturally based agencies such as CGIAR (Consultative Group on International Agricultural Research) centers, AGRA (Alliance for a Green Revolution in Africa), FARA (Forum for Agricultural Research in Africa), etc. These bodies have been working on issues important to SWFF/WE4F such as inclusive finance, climate 200 adaptation and resilience, extension and capacity building, etc. Such partnerships would benefit WE4F and the innovations funded through the program and help promote innovative and sustainable growth in the targeted communities’ agriculture through improved resource use and market links. 5. In the executive summary on p. 3 it says that the World Water Week took place in March 2019. If you mean the World Water Week in Stockholm, maybe you refer to the one in August 2018? OUR RESPONSE: The current Report reflects the correction. 6. On p. 4 in the executive summary the fund “Sustainability and Reliance” should be changed to “Sustainability and Resilience”. OUR RESPONSE: The current Report reflects the correction. 7. When citing Sida in this partnership, please always use the term “Sweden through the Swedish International Development Cooperation Agency (Sida)”. OUR RESPONSE: The current Report reflects the correction. MFA-NL (The Netherlands) Comments General Comments 1. The ability to pay (like mentioned on page 29) is a critical factor in the SWFF. How has been dealt with it to increase affordability and what works best, as lesson for WE4F? OUR RESPONSE: The “affordability” of innovations is related to the ability and willingness to pay; issues which were not specifically identified in the surveys although they were explored in the performance evaluations (See Annex 8). The surveys did, however, measure an associated indicator which was the access to the innovation of women and poor or very poor. It is obvious this could not be achieved if the innovation was highly priced and users unable to access low interest finance. However, since the social dimension (particularly widening access to all income levels) is a main priority for SWFF and WE4F, the innovations should be made affordable while also promoting profit and sustainability of innovations. We offer the following recommendations: 201 • Innovators could offer a diversified product range across income levels with add-ons for wealthier customers. If appropriate, innovations could establish cross subsidies to charge some customers more in more affluent areas. • Innovators should consider how to work with institutions that provide low interest loans for more expensive innovations. Waste’s FINISH project in India and Kenya has had some success in liaising with banks to arrange for low interest loans for sanitation. In a competitive environment where the government had capped interest rates, banks were looking for more customers. The incentive for the sanitation loans was thereby to get customers to open savings accounts. • Tiered pricing strategies based on income and ability to pay, and a “one-for-one” model in which every actual purchase provides one at no cost to someone inside a vulnerable population) could also be considered. There are always issues with these, but more so if it’s coming from outside the host country. • Financing directly by the innovator instead of outside loans to purchase the innovation (WGI Aquaponics, Hydroponics Africa, maybe others) seems to be positive for beneficiaries. There was an issue with repayment, but it’s something to be explored. 2. Maturity of innovators is one of the guidance’s, for instance on page 30, and it is purely related to commercialization. But in a programme such as SWFF or WE4F, social criteria are as important as commercialization. How are these incorporated in your concept of maturity? OUR RESPONSE: WE4F could place an added emphasis on applications from social entrepreneurs who bring a blended social mission vis-a-vis a more traditional commercial outlook. Social criteria are not fully integrated into our definition of maturity because in looking at development stage and sustainability, we were assessing the current level and trajectory of current SWFF innovators and whether they were achieving their own stated objectives (in line with SWFF criteria, milestones, and indicators). The evaluation team does feel that social criteria is incredibly important given the mandate, objective, and mission of SWFF and now for WE4F for all Founding Partners in supporting vulnerable populations. However, for the purpose of maturity in this regard, the team used SWFF’s own selection of socially-conscious innovators and stages of development, understanding that innovators were selected who already had a social mission and SWFF’s technical support was contributing to that, and continued funding is contingent on their meeting objectives and targets. We did not use “level of social consciousness” as a criterion for assessment of the development stage, but it is an interesting point and should be tracked for innovators with indicators and performance management tools (perhaps in an index) for the WE4F program. 3. Several sources of funding are discerned in the analysis, like on page 36. And private and public funded innovators are compared. SWFF innovators rely on public funds, but was that an omission in the SWFF design, what were the real lessons, and trade-offs? How do you analyse different funding sources in different stages and what happened over a period of time to the differently funded innovators in terms of longer-term continuity? This has not been analysed, but perhaps lessons can be learned. 202 OUR RESPONSE: SWFF’s efforts to introduce innovators to private sector funders did not come through strongly in the evaluation. While efforts were clear and well done on supporting the innovators to “tell their story”, it was clearly more difficult to provide opportunities for awardees to engage with possible investors. This could be unpacked further as to whether investors were looking for more on the path to scale, how investments would be used and other elements that clearly suggest an investment opportunity. Could SWFF have done more to test whether innovations were ready for market? Would a different configuration of the IIAC help to strengthen this element even further? Some suggestions on the sources and types of funding are made throughout the text. Specifically we have the following on page 93: “suggestions to attract various types of investors such as for example offering end-users a one-time subsidy so they can get acquainted with the innovation and test it and mitigate the willingness of end-users to pay for the innovation or the affordability of the innovation, while increasing the demand of the product and the level of adoption of the innovation”. “We do what we measure” is a saying apropos to monitoring, evaluation, and learning in development programs especially in the sphere of USAID reporting requirements. One way WE4F could better address the composition of innovator funding is to firstly make it part of innovators needs assessment and capacity building/technical support strategy to look at their current levels of funding and have each of them early on develop a strategy for attracting more and more diverse funding (private sector, other donors and foundations) and secondly include this as an indicator that looks for innovators to over time increase the number of funding streams (showing strong sustainability potential and diversification of “income”) and disaggregate that data by funding type (private, public). If this is undertaken in the name of sustainability for the innovator to support them in diversifying funding sources and living a long, healthy innovation life, (including these areas as areas for technical support) and not as an auditing tool or micromanagement, then this could be a positive move forward for WE4F. 4. The TAF can be further analysed, on the one hand it is highly valued on the other hand it reduced risk taking. What is the overall analysis in terms of performance of the innovators, not only of the TAF system as such? OUR RESPONSE: The balancing of risk in performance and continued operation is a keen question addressed throughout the Report. There was a balanced assessment by innovators in their negotiations around the milestones. This process could have encouraged more risk taking but if milestones were more ambitious and the innovators did not meet them, then the funding was jeopardized. Perhaps the TAF could have encouraged more and different kinds of partnerships (see response to other question on potential partners). Ultimately could the kinds of risk taking to be encouraged be unpacked further and the kinds of failure that are unacceptable to continue in the program defined on a case-by-case basis? Risk taking has to be balanced and analyzed through risk assessment tools in comparison to trade-offs. This should be done on an innovator-by-innovator basis through individualized risk analyses, and with concrete definitions of what is meant by risk (contextual, programmatic, 203 reputational, fiduciary). These need to align with Founding Partners risk management approaches (such as those of USAID). Assessing risks comprehensively and comparatively is important at the business and strategic planning stages- to help identify what to prioritize + high-level partnerships – and innovation design to help inform selection of beneficiaries, local partners, and markets. 5. One of the issues that come up is comparing SWFF with a commercial fund manager. But the comparison between a Grand challenge fund and a commercial Fund is not really made in depth, while it would be worthwhile to do so. Also, for lessons that can be learned for WE4F, because for investor readiness, fund manager’s approach might be helpful. And combination with cross cutting might strengthen ESG and add value. Can some more comparison be made between these, based on experience of SWFF? OUR RESPONSE: It is difficult to compare SWFF with a commercial fund manager as the goals are different – SWFF is designed with a social and environmental purpose. Efforts to compare with other challenge funds are to ensure that they are learning from each other but we have to recognize that comparisons made may not be benchmarking against the same things. Some challenge funds are aimed at increasing participation and accountability, some to develop new products and services, etc. For those interested in products and services, they may operate at different points in the process – from concept development to scaling. This issue does need to be further explored, it is recommended: • There could be a high level of learning from a roundtable between donors, commercial fund managers and perhaps innovators to understand what aspects of the fund management model could be applied to an initiative like SWFF/WE4F. • WE4F could examine the For profit innovations that are already “mature” and that are just starting up, to examine maximum success and impact. In addition a focus on innovations that are not only “for profit” but also have a social component (such as tending to include vulnerable groups and women) would enable the identification of factors making a profit and being socially sustainable. 6. The cost of operation is compared, is a kind of starting point and it comes back various times. What are the overall lessons to learn, especially in view of the new hubs that might be more costly, as each hub has its overhead. This topic could perhaps be elaborated a bit more. See also page 46 and recommendation page 50, making a framework for cross comparison. But would that really be of help, thinking of for instance how to get a better WE4F, and not thinking in terms of high-level frameworks? OUR RESPONSE: We were not really tasked with conducting a Value for Money exercise and were not privy to the costings. Again, in terms of comparisons, it would be important to ensure that any comparative work be done using like with like. Specifically, comparisons should be made between models that operate in a similar way to and offer a similar package to innovators as SWFF. We have recorded in the Report the high value accorded to TAF by innovators but have not had a full set of comparable metrics to confirm this conclusion. Since comparing SWFF with other kinds of challenge funds can ultimately be unhelpful, we have recommended in the Report that a framework be developed that standardizes measures to know where comparison is helpful and 204 where not. We recognize that the TAF was run with a lean team that was largely consistent throughout in terms of personnel. The TAF personnel were reflexive, making changes throughout when they found that systems were ineffective or inefficient. Their compact size allowed them to make those changes swiftly. 7. Why is it some innovators perform very poorly on end user’s income and impact, while SWFF was focusing on supporting this. Where was the mismatch, for instance: (P 57 Table): What is behind the low percentage of Naireeta, as it seems strongly focusing on gender and poverty, but the impact is low. See also what is mentioned later on, page 64 for instance. What reasons and what can be learned? The same holds true for table 30? This comes back up page 75-77: targeting the poor. But how, what we learned from SWFF as do’s and don’ts? OUR RESPONSE: There are a number of unanticipated anomalies revealed after analysis. The examples identified generally include successful innovations reporting lower levels of favourable outcomes over time. Some of these can be explained by closer examination of the available evidence, others not. We can only report how users responded to our questions and subsequent surveys may not record the high levels of satisfaction than the initial survey. Some of these anomalies may result from somewhat different sampling methods in earlier and later surveys, others could be attributed to changing environmental conditions. Since SWFF Field Evaluators had more time on the ground and often greater samples we have used their data to understand impact at the household level. More of a longitudinal study would help to gauge impact across different stages of uptake – from piloting to scaling. Different responses are likely between the first year of farmers using a new product or service and later years. Once the innovation is more fully bedded in evaluators may likely get different responses. The results for Naireeta on some measures appear as “outliers”. In assessing impact in farmers lives and their agricultural practices, some results may appear erratic because crop production was disturbed by highly variable monsoon conditions. We anticipate much elevated results after salinity in irrigated water is considerably reduced by several recharges. We have made a rigorous review of the survey results together with the COR and the local evaluator on the basis of data collected during rigorous sampling on the original visit, revisit, and subsequent Field Evaluator surveys. In addition, we have added a paragraph on page 87 to explore possible reasons behind some of the differences computed especially for women and the poor, mainly the level of education, gender relations and division of assets, and control over decision-making and autonomy. 8. Climate resilience is mentioned in some parts of the report, but gets little attention. Nevertheless, the nexus gains are in function of climate adaptation. Little evidence it says on page 72. Why is that the case? OUR RESPONSE: All innovations deal with climate resilience to a greater or lesser extent. Since we adopted a set of surveys over time rather than an extended set of case studies, it is difficult to trace the internal linkages which led to the high proportion of farmers changing their practices. We explored these issues throughout the text but also in length when referring to changes in farming practices in Table 41, and subsequent text on pages 77 and 78. 205 Shocks and stressors (droughts, floods, pests and disease outbreaks, market prices and other externalities: environmental degradation, soil fertility decline, poor infrastructure and social issues) can undermine the outcomes achieved by SWFF innovations. The frequency or likelihood of occurrence and the severity of the effect of these on smallholder farmers and systems may vary from country to country or within countries. We would recommend that, in addition to the methodology utilized in this evaluation of surveys, there could be greater synchronization, quicker turnaround in analysis and initial reporting, and reciprocal exchanges between surveys and case studies to trace the linkages leading to practices which enhanced climate resilience. More detailed observations 1. Page 25- first paragraph: Why is it surprising not more SWFF enterprises are focused on some form of innovation in the infrastructure… OUR RESPONSE: WE4F could exercise more systematic prioritization in determining which innovations should receive support. Infrastructure innovations require a different business model and more complex relationships than buying a product. Examples of SWFF infrastructure innovations include the Skyfox fish ponds and Naireeta’s bhungroo but these seem to be the two outstanding cases. An important item of consideration is that of the type and depth of investment in infrastructure as compared to other innovations: the former is more capital intensive for the user than buying a one-off product or subscribing to a service. In irrigation, for instance, in comparison to product, service or information systems, there is a longer￾term investment and different kinds of capital investment needed (alongside potential negotiation across users). There is also more burden on the user or groups of users with more low cost finance potentially required. Innovations need to seek out combinations of funding which do not depend on a complete subsidy and also require the user to invest as well. Innovations for infrastructure may not include the ultra poor unless they band together as a community (like for weir or storage technology). We would recommend the following: • Past applications could be examined to consider why fewer infrastructure innovations succeeded at the first stage. Were applications for such innovations submitted and, if not, why did they not respond to the SWFF calls? If infrastructure is to become a focus of the WE4F program, then applications for innovations in infrastructure may need to be specifically solicited so that We4F can identify any that fit within the model and goals of WE4F. • WE4F could give attention in future calls to identify a few categories in which such innovations would fit; for example “innovations focused on water retention and effective delivery”; “Innovations focused on efficient use of inputs”, etc. Subsequent performance evaluations could compare results in each category to identify those with maximum impact and sustainability to focus on in future programs. 2. Page 25: What infrastructure and would there be enough customers, as that type of products might not be affordable. How to analyse this? 206 OUR RESPONSE; Unfortunately we did not collect data on the different surveys and this question on type of infrastructure as type of products that might be affordable was not the focus of the evaluation questions. A comparative examination of products and depth of users was unfortunately outside the scope. It is also difficult to speculate on given the data collected by the evaluation team and the observations made. A suggestion to further analyze this would be to do or find a market study on infrastructure through another donor project, a local partner, in a specific set of countries or other secondary research. If none exists, it could be the focus of a WE4F consultant to identify the most relevant or effective sectors for innovations. The answer to whether there would be customers needs to be assessed on a case-by-case basis relevant to the type of infrastructure, the country, the demand, and the market. WE4F could consider a case by case study of innovations that are focused on infrastructure could be undertaken to come up with the best solutions in managing effectiveness of innovations and affordability. 3. Page 26 mid page: Social character of the innovation and table: what difference does profit and non￾profit make to the poverty goals SWFF wanted to support? Has that been analysed, it does seem to be part of this performance evaluation. It could be interesting for future selection: non-profit perhaps has less scale potential but more social potential? OUR RESPONSE: If we could identify poverty goals and associated indicators (even those that SWFF collects on itself) then it would be possible to make a comparison with the Online Impact survey on the basis of profit vs non-profit innovations assess whether those with a stronger social mission (not for profit) were able to reach more vulnerable people. Unfortunately we can only speculate on this and assume that innovations that are non-profit reach more vulnerable people, but we could also imagine that unless they pair with a profit model or very strong funding streams, they won’t last and act more as a typical development program rather than an innovation. 4. P 27 Recommendations: What is meant with the question on ‘Are these businesses using cross￾subsidies or other mechanisms to continue to reach these target groups?’ OUR RESPONSE: The question of cross-subsidies is closely related to the affordability of the innovation and indicates the use of full pricing of richer customers to subsidize the pricing of a product in relation to poorer customers. The question is posed to raise the question of whether an innovation was acting to broaden its affordability to the poorest users. In the review of performance evaluations conducted by SWFF Field Evaluators in Annex 8 there is description of forms of cross subsidization; in WGI, for instance, a cheaper variant of the fish tank is available to poorer farmers. This is a weaker form of cross-subsidization, in other innovations greater affordability is achieved by use of subsidies drawn from external funds. 207 5. P 28 mid page: Awardees remain very small firms for the moment ….. How do we need to interpret this and what does it mean for scale? How with these small awardees scale can be realized? Because here the evaluation concludes there is not a lot of spare cash and scope for expanding the employee base. But is that the only barrier? And has been analysed what would be the best options to support? OUR RESPONSE: This is, unfortunately, a very broad set of questions (even though they are also highly relevant) for which we do not have appropriate data to analyse. In the evaluation we have broadly followed the SWFF adopted model of innovation growth from early pilot to commercialization. In doing so we did not specifically examine the employee base as our focus was on broadening the customer or user base. It is not immediately clear that that a lack of investment is the primary barrier to innovations scaling up. Despite this, NewSil explained in a KII that there were major technical issues to be resolved in moving from small-scale to large-scale production. In other words, scaling up is not necessarily incremental but sometimes involves a qualitative leap upwards; a particularly challenging moment. 6. Page 29 Recommendation on low interest finance: How realistic is low interest finance in the WE4F countries and is that available in the local finance sector, or do you need to involve development banks for instance? OUR RESPONSE: We have not been able to make a comprehensive review of the availability of low interest finance but we have had valuable exchanges during visits with innovations. There appears to be growing challenges in low interest funding. The indication from innovations in India is that major private banks have withdrawn from funding small farmers. Recently it was reported that state banks had similarly withdrawn. Since this is a broad national position it is proposed that USAID in India raise the question of widening access to users to provide support to WE4F. Responses from Africa indicate that low interest loans to small farmers are very limited and that high interest rates are being expected. Innovations in Africa generally were without low interest funding except where specific arrangements were made between innovations and credit agencies. This is an area where the evaluation team recommends coordination with USAID and founding partner POCs at the beginning of WE4F in expected innovator countries to identify ongoing work in the availability of low-interest finance and access to finance through existing activities working on market systems development, workforce development, agriculture writ large, private sector engagement, or access to finance specifically. In Africa, Asia, and Latin America especially, USAID has existing Economic Growth and Resilience and Food Security projects that are doing work to identify financing options for micro, small, and medium enterprises (especially farmers, entrepreneurs) to reduce collateral requirements or identify credit and loan products designed with the low-income beneficiary in mind. This is a challenging area, however, and some of these producers take years to build trust with banks, more time to develop the products themselves, and then actually have enough availability of these products to make a difference. While we can discuss this anecdotally and potentially point WE4F in the direction of some examples where this is happening (Rwanda’s Huguka Dukore workforce development activity, others), the team did not have a related evaluation question and therefore data collected to buttress this overview. 208 7. Page 42 last part: Relation between founding partners is interesting, but in recommendation on page 43 visibility and creating synergy is not mentioned. What roles you see founding partners in a new constellation could play to optimize the WE4F for all of us. Are there some strengths to build on and opportunities not used? Or perhaps other issues? OUR RESPONSE: Broadly we would recommend that the general overarching synergy of the Founding Partners is an important strength in SWFF and should carry over into WE4F. In the section on partnerships we recommend that the Founding Partners (particularly those in emerging economies) could take on the promotion of WE4F and the identification of potentially high performance innovations. The diplomatic links between Founding Partners and regional organizations could yield rewarding results particularly with the African and Asian Development Banks, the Southern African Development Community (SADC) and AMCOW which includes Water Ministers in Africa. The synergies of the combined efforts of the Founding Partners could also be focused to resolving the key obstacle of widening access to low interest finance particularly in Africa. In reference to partnerships above we discuss how WE4F could support the networking via existing partnerships, identify synergies with existing programs, local partners, and other key actors. Considering a country or region as a system is a key component of identifying leverage points or opportunities for innovations to address critical challenges of the country/community /region (unit of analysis or support needs to be selected. We suggest that if the WE4F program thinks systemically and require that in applying the innovations identify the key challenges in the community or country and the key local actors. This would enable the Founding Partners to identify the existing partnerships and networks that can help support the cadre of innovators selected. This would enable a positive role for the Founding Partners each bringing different relationships and partnerships to bear where they may not otherwise be drawn on. 8. P 53 Mid page: SWFF helped developing partnerships. But the type and performance and added value of these partnerships is not analysed here. Is more information available and more analysis done to get a better feel of these issues? Especially, taking in mind lessons for the partnership approach of WE4F. Could be helpful to get lessons from SWFF. OUR RESPONSE: Assessing the most productive combinations of partnerships opens a fruitful avenue for future research. In our KII and the online survey unfortunately we did not request responses in relation to a classification of types and sources of partnership and this issue was not raised in comments on the first report. It is clear that, in WE4F, partnerships would be vitally productive, for instance, a communication partnership could get ideas out. We offer some suggestions: • Partnership between innovations. The products and services could be shared; for instance, an innovation could buy NewSil coating and use a set of seeds from Reel Gardening. There are further possibilities of aggregating sets of products and services across innovations. • Partnerships between innovations and community NGOs: Community partnerships could be explored to build capacity at the community level to best use the innovation e.g. Skyfox works with a local NGO to galvanize communities. Meat Naturally also has a set of complementary community groups. 209 • Partnerships with market intermediaries: SWFF’s prime focus has been on production, WE4F could make a useful exploration of improving partnerships with agencies to get products to market. Farmers have, for instance, the challenge of getting produce to market or restaurants. There are some examples like with Skyfox taking on the role of intermediary or liaising with third parties to get fish and produce to markets and restaurants. • Partnerships with research institutions. Research institutions could be useful with scientific review and advice on modifications to the innovations such as occurred with NewSil moving from small scale to mass production. • Partnerships with commercial and financial institutions: Since low interest finance is a crucial question to expanding the scale of innovations, partnerships to provide such loans as with Waste innovation (through the FINISH program in India and Kenya, a non-SWFF innovation) could be invaluable. • Partnerships to advise regulatory standards: There is considerable potential for partnerships with regulatory agencies and local government to advise on standards for types of service or products. In response to the extended requirements of USAID on waste management, WASTE developed policy for waste management in India which has gained considerable recognition and been utilized. • Partnerships between innovations. Innovators have identified partnership among SWFF awardees to profit from working together: lessons learned, results from one innovation could be an answer to another innovation’s challenges, etc. Also, there should be some opportunities for innovations themselves to identify partners they’d like to collaborate with, and WE4F could facilitate these collaborations. 210 ANNEX 10 EVALUATION TEAM Evaluation Advisor/Team Leader Christopher David Hemson Dr. Hemson has demonstrated expertise in the fields of water services and resource management, water policy, climate change, clean energy policy, social policy, and gender analysis over the past 21 years. His research has included public private partnerships, performance and impact evaluation, gender analysis, rural development, infrastructure, labor market analysis, and disaster risk reduction. As a senior evaluator, he has developed qualitative methodologies, designed analytical framework, prepared templates for qualitative data collection, trained team members to code qualitative data, written the reports, and published peer reviewed analysis. He has been the Director of the Social Policy Program at the University of Durban-Westville and the Research Director with the Human Sciences Research Council, a statutory research institution in South Africa. He has written numerous research reports, published peer-reviewed articles and book chapters, reviewed manuscripts on WASH issues for publication and edited a book on poverty and water. He recently published on water resources and public health in South Africa, on training and integrating rural women into technology in Bangladesh, and published an evaluation on integrating women into Grameen Shakti’s renewable energy value chain in Bangladesh project for peer reviewed journals including gender, technology, and development. His country experience includes Ethiopia, Kazakhstan, Bangladesh, India, South Africa and Indonesia. He has a Ph.D in Sociology from the University of Warwick. As the Team Leader, he is the primary interlocutor with the COR, the SWFF Founding Partners, and SWFF TAF. Overall, he is responsible for the quality and timely execution of the Evaluation Report and all deliverables. The Team Leader will conduct site visits, lead the evaluation, and present the findings. Evaluation Methods Specialist Ken Caplan Mr. Caplan brings almost 20 years of comprehensive experience working in the WASH sector. He has supported programs across a wide range of contexts and sub-thematic areas in both water and sanitation in urban, peri-urban, and rural contexts. Mr. Caplan has extensive experience with specific institutional arrangements of PPPs, and numerous reviews of partnerships for innovations in the WASH sector. Mr. Caplan was on a three-person team for the mid-term review of the SWFF Grand Challenge Fund, and on the mid-term review of the GIZ International Water Stewardship Program, and mid-term review of the Sustainable Water Fund. He was also the team lead for a partnership review for Plan International UK and Unilever under their DFID Payment for Results WASH funding. He led the 2014 global program review for Sanitation and Water for All (SWA) for UNICEF (the Secretariat host for the global partnership); was a team member on the IRC global review for DGIS Directorate-General for International Cooperation (DGIS), an agency of the Netherlands' Ministry of Foreign Affairs); led the global partnership and a project review of the USAID-The Coca-Cola Company Water and Development Alliance; partnered with AguaConsult for a review of the USAID-Rotary Foundation’s H2O Partnership in three countries; and was a member of a two-person team 211 reviewing a major PPP for DGIS in Indonesia. He has a MA in International Development from American University. As one of the two Evaluation Methods Specialists, he will advise on the description of outcomes, conduct site visits, and lead the writing of meta level and GCD Evaluation Report sections. Additionally, he will contribute to the design of data collection tools and instruments for KIIs and FGDs Evaluation Methods Specialist Stephanie Monschein: Ms. Monschein is an M&E, Research, and Learning Specialist with 12 years’ experience in performance management, large-scale program management, learning approaches, facilitation, capacity building, and evaluation in international development. She has completed as a team member, methodologist, or co-team lead more than 15 USAID, State, and IFC evaluations of projects focused on innovation, agriculture, countering violent extremism, public diplomacy, donor cooperation, health capacity building, and trade. Of note are two USAID portfolio-wide meta evaluations on which she participated: a mixed-methods evaluation and meta-analysis of Trade Capacity Building Projects that received an Evaluation Excellence Award in 2011, and a meta evaluation of USAID evaluation quality before/after the change in the Evaluation policy that is still cited in international development policy literature. For USAID’s Global Development Lab, she completed an evaluation of the Higher Education Solutions Network (HESN), which included a focus on grand challenges, as well as an evaluation for the Department of State on the Lower Mekong Initiative supporting education, health, water, energy and environment, and a sustainable economy. She has an MA in Latin American Economic Development and Political Economy from Georgetown University and is proficient in Spanish. As one of the two Evaluation Methods Specialists, she will conduct site visits, lead the writing of the Methodology, Survey, and Innovator Impact Evaluation Report sections, as well as contribute to the design of data collection instruments. Additionally, she will lead the design of data collection tools (Desk Review matrix, FCR matrix, data collection database), organize quantitative and qualitative data for team use in NVivo and SPSS, and lead the survey efforts. Agriculture Technical Advisor, Nohemi Voglozin Dr. Voglozin’s background is in agriculture, rural development and food security developing and implementing solutions for the challenges of economic growth and cross-cutting topics in the agricultural sector, international development and related fields. Additionally, she is in the process of completing her MPS in Geographic Information Systems from the University of Maryland. While pursuing her PhD she conducted consultancies with Africare as a consultant for the Office of Agriculture & Food Security as well as for the Food and Agriculture Organization of the United Nations & CIRAD, as an Agriculture and Food Security consultant. Additionally, as a petrostate fellow for the University of Maryland, she led a team to collect socio-economic and agro-ecological data from 300 rice growing farmers in Beninese agricultural regions, designed and conducted experiments with seed varieties, as well as produced maps 212 using ESRI ArcGIS to identify hot-spots of diversity for conservation, develop a base-line understanding of rice genetic resources and areas for conservation in Benin. As the Agriculture Technical Advisor, she will provide agriculture subject matter expertise to the SWFF team throughout the evaluation’s period of performance. Dexis Project Management Unit, Adelaide Bryan Ms. Bryan has five years of experience with Monitoring, Evaluation, and Learning projects operating in conflict, transitional, or post conflict environments throughout the Middle East and North Africa region. She worked in Egypt India, Iraq, and Turkey, as well as worked remotely based in Washington DC on projects in Jordan, South Sudan, and Syria. Ms. Bryan has a Masters degree in Arab Studies from the School of Foreign Service at Georgetown University with a focus on political economy, regime structures, and ideologies. She has non-native fluency in Modern Standard Arabic, is conversational in Egyptian Arabic, and has basic conversational skills in Lebanese and Iraqi Arabic dialects. Ms. Bryan has experience with a wide range of donors such as, USAID, USDA, State Department, and DIFID. Ms. Bryan provides project, deliverable, and budget management for the Evaluation Team. She is responsible for maintaining communication with the COR throughout the evaluation on the contractual and technical deliverables as well as any programmatic or recruitment needs. South Africa Evaluation Methods Specialist, Mutsa Prudence Mambo Dr. Mambo's background is in Environmental Biotechnology and she recently consulted with the World Bank focusing on Water and Agriculture Global Practices to achieve the Bank’s goals of eliminating poverty and increasing shared prosperity in developing countries. In the past year she was also a participant in the California Irrigation Study Tour- California Polytechnic State University, CA and her doctoral research looks at the ways to inform and facilitate the implementation of sustainable practices, infrastructure and technology choices resulting in clean and resilient development. Her academic background includes a Ph. D. and a M. Sc. In Environmental Biotechnology from the Rhodes University. As the local South Africa Evaluation Methods Specialist, she will work with the Evaluation team and conduct site visits, engage with the innovators and end users, as well as contribute to the analysis of the data with the Agriculture Technical Advisor and Evaluation team members. Ghana Evaluation Methods Specialist, Ussif William Ayinga Mr. Ayinga has extensive knowledge and experience in agriculture monitoring and evaluation. He is well versed in USG indicators and especially with the Feed the Future indicators. As the Monitoring and Evaluation Coordinator for USAID/FtF/ Agriculture Development Value Chain Enhancement Project (ADVANCE), a four year Feed the Future flagship project in Ghana, he developed several tools and instruments to collect, organize and analyze performance indicators, annual gross margins, private sector investments, agricultural loans, and improved technologies. He also ensured that the Outgrower Business Model procedure was adhered by all field teams and farmers. As the M&E Coordinator for the USAID/Ghana Agriculture Maize 213 Seed Adoption Project (GAMSAP) in the Ashanti and Brong Ahafo regions of Ghana, he conducted yield estimation and margin analysis in over 100‐demonstration fields to assist farmers to make informed seed decisions. As the Director of Monitoring and Evaluation for the USAID/Agriculture and Natural Resource Management Project (AgNRM), a Feed the Future program, he led the development of systems to monitor land use, water bodies, wildlife and forest by working closely with eight Community Resource Management Associations (CREMA). Additionally, he was central to the implementation of alternative livelihood interventions provided to beneficiary communities and households. As the M&E Consultant for the Akoma Women Cooperative, he worked on a trans ‐border intervention to increase shea oil production among the women. As the local Ghana Evaluation Methods, he will work with the Evaluation team and conduct site visits, engage with the innovators and end users, as well as contribute to the analysis of the data with the Agriculture Technical Advisor and Evaluation team members. India Evaluation Methods Specialist, Phani Priya Nandula Ms. Nandula’s experience is in Livelihoods and Microfinance within the agriculture sector. Her background covers impact assessments, agriculture value chain analysis, and market studies throughout India. Most recently she was part of a feasibility study for the establishment of a spice processing unit as well as a study on the decline of orange cultivation for the Institute of Livelihood Research and Training. She has also facilitated trainings on Rural Development and on Agriculture Value Chains as well as studied the market linkages and value chain for agriculture development and its products to support small and marginal farmers. Her academic background is in Agriculture and Business Management. As the local India Evaluation Methods Specialist she will support the Evaluation team in the data collection and analysis. India Evaluation Methods Specialist, Reshma Dixit Ms. Dixit has sector specific experience within agriculture, resource management, and water and sanitation. Her background covers monitoring and evaluation, assessing impact, gender, and both qualitative and quantitative research methods. She is currently pursuing her PhD in Social Work from the University of Delhi. Her research thesis focuses on Gender and Water and Sanitation in the slums of Bhopal. While working for WATERAID, she managed the urban program on water concertation and environmental sanitation. Additionally, while with Catholic Relief Services, she was responsible for the program’s M&E and assessing the program’s impact. Ms. Dixit worked on water conservation and recharging groundwater for agricultural purposes. She holds a certificate in research methodology, participatory monitoring and evaluation, and SPSS. As the local India Evaluation Methods Specialist she will support the Evaluation Team in the data collection and analysis. Uganda Evaluation Methods Specialist, Robert Gensi 214 Mr. Gensi’s background is in advising on agriculture initiatives in several capacities. For the past several years, he has served as the Agriculture Advisor for Self Help Africa. Previously, he also worked with FHI 360, World Food Program, Bill and Melinda Gates Foundation, Grameen Foundation, and ACDI VOCA in technical roles on agricultural focused programs. He has also done some agribusiness consulting and has experience with evaluations. His academic background includes an M.Sc. in Postharvest Food Technology and a B.Sc. in Agriculture (Rural Economy) from Makerere University. As the local Uganda Evaluation Methods Specialist, he will work with the Evaluation team and conduct site visits, engage with the innovators and end users, as well as contribute to the analysis of the data with the Agriculture Technical Advisor and Evaluation team members. Egypt Evaluation Methods Specialist, Ahmed Taha Mr. Taha’s academic background is in Agricultural Science from the Faculty of Agriculture from the Cairo University and he is certified as an agro organic trainer from the Center Lab for Organic Agriculture (CLOA). He has extensive experience in organic agriculture practices, cultivation, best practices for adaptation to climate change, and on bringing organize products to market. He is currently an Agro-Ecology Technical Advisor and has provided consultant expertise on dairy value chains for the Ministry of Agriculture with Cultivating New Frontiers in Agriculture (CNFA Association) and Empowering Women in Fayoum Rural Areas (EWFRA project). He also has experience with evaluations, baseline surveys, and data collection methodologies. With CNFA, he also provides technical assistance on data collection methods and practices to improve the income of small farmers throughout Egypt. As the local Egypt Evaluation Methods Specialist he will support the Evaluation team in the data collection and analysis. Bangladesh Evaluation Methods Specialist, Md. Harun-Ar-Rashid Mr. Ar-Rashid has a strong academic and professional background in the agriculture sector, research methods, and analysis. He has a Ph.D in Agriculture from the American Independent University as well as a Masters in both Crop Physiology from the Reading University and in Agriculture from the Bangladesh Agricultural University. He is the Chairman of Agricultural Advisory Society (AAS) and Partner NGO networks as well as a member of the Bangladesh Society of Agronomy. He provides technical expertise on a wide range of Agriculture fields such as value chains, irrigation, horticultural, seed production, and agronomic research. As the local Bangladesh Evaluation Methods Specialist he will support the Evaluation team in the data collection and analysis. Kenya Evaluation Methods Specialist, Lewis Mwenda Aritho Mr. Aritho’s background is as an agriculture economist with sector experience in agriculture, youth, and gender programming. He holds a Masters in Economics from the University of Bombay and a Masters in Agricultural Economics from the University of London. Highlights of his agriculture experience include his role as the Program Principal Investigator for the Lasting 215 Infrastructures for Northern Karamoja Promoting Agro-Pastoral Growth and Enterprise (LINKAGE) End-line Evaluation with Mercy Corps in Uganda as well as the Learning and KM Advisor for the USAID Pastoralist Areas Resilience Improvement through Market Expansion (PRIME) project in Ethiopia. Additionally, while he was the Monitoring and Verification Officer with the USAID Performance, Analysis and Evaluation (PACE) in Kenya, he completed the collation of key performance indicators for the Economic Growth Component with a focus on the Feed the Future program. Since 2015, he has been the Senior M&E Specialist for the USAID Youth Employment and Skills program in Kenya for RTI International. As the local Kenya Evaluation Methods Specialist, he will work with the Evaluation team and conduct site visits, engage with the innovators and end users, as well as contribute to the analysis of the data with the Agriculture Technical Advisor and Evaluation team members. 216 ANNEX 11 CONFLICT OF INTEREST FORMS 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523