MIDTERM EVALUATION OF THE FEED THE FUTURE STRENGTHENING AGRICULTURE AND NUTRITION EXTENSION ACTIVITY (SANE) July 2019 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by David Soroko, Beatrice Mtinumi, Charles Masangano, William Fiebig, James Phillips, and Abigail Price of International Business and Technical Consultants Inc. (IBTCI). MIDTERM EVALUATION OF THE FEED THE FUTURE STRENGTHENING AGRICULTURAL AND NUTRITION EXTENSION ACTIVITY (SANE) Originally Submitted December 18, 2018 Final Submitted October 31, 2019 AID-OAA-I-15-00022/AID-612-TO-17-00001 International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive, Suite 400 Vienna, VA 22182 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover Photo: Farmers march for farmers’ rights during MaFAAS Extension Week, Lilongwe, June 2018 Table of Contents ACRONYMS.............................................................................................................................................. ii EXECUTIVE SUMMARY........................................................................................................................ 1 FINDINGS AND CONCLUSIONS .................................................................................................. 1 RECOMMENDATIONS...................................................................................................................... 3 PROJECT BACKGROUND.................................................................................................................... 5 EVALUATION PURPOSE, METHODS, AND LIMITATIONS..................................................... 8 FINDINGS CONCLUSIONS AND RECOMMENDATIONS......................................................... 9 GENERAL RECOMMENDATIONS...............................................................................................25 EXIT STRATEGY RECOMMENDATIONS .................................................................................26 Annex I: Strengthening Agriculture and Nutrition Extension: Mid-Term Evaluation Scope of Work.....................................................................................................................................................29 Annex II: KIIs and FGDs........................................................................................................................39 Annex III: Data Collection Instruments............................................................................................41 ANNEX IV: Statement of Differences...............................................................................................46 ANNEX V: Evaluation Team Response to Statement of Differences .......................................57 ACRONYMS AMEP Activity Monitoring and Evaluation Plan ADD Agricultural Development Division AEDO Agricultural Extension Development Officer AEDC Agricultural Extension Development Coordinator AES Agriculture Extension Specialist Afikepo/Kulima EU program supporting agriculture and nutrition ANCC Area Nutrition Coordinating Committee AWP Annual Work Plan ASFNS Agriculture Sector Food and Nutrition Strategy ASP Area Stakeholder Panel ASWAp Agricultural Sector Wide Approach AEMO Agricultural Extension Methodologies Officer CAEO Chief Agricultural Extension Officer CISANET Civil Society Agriculture Network CoAg Cooperative Agreement COP Chief of Party CRS Catholic Relief Services DADO District Agricultural Development Officer DAECC District Agricultural Extension Coordinating Committee DAES Department of Agricultural Extension Services DAESS District Agricultural Extension Services System DARS Department of Agricultural Research Services DC District Council DCOP Deputy Chief of Party DEC District Executive Committee DNCC District Nutrition Coordinating Committee DSAP District Strengthening Action Plan DSP District Stakeholder Panel EPA Extension Planning Area E ZOI Extended Zone of Influence FGD Focus Group Discussion FNO Food and Nutrition Officer FSU Farm Services Unit FTF Feed the Future GOM Government of Malawi GVH Group Village Headman HSA Health Surveillance Assistant IFAD ICT International Fund for Agricultural Development Information, Communication and Technology IFAD/SAPP International Fund for Agricultural Development/Sustainable Agricultural Production Project IFPRI International Food Policy Research Institute KII Key Informant Interview LOP Life of Project LUANAR Lilongwe University of Agriculture and Natural Resources MaFAAS Malawi Forum for Agricultural Advisory Services MISST Malawi Improved Seed Systems and Technologies MoAIWD Ministry of Agriculture, Irrigation, and Water Development MSU Michigan State University MTE Mid-term Evaluation NAEASS National Agricultural Extension and Advisory Services Strategy NAPAS New Alliance Policy Acceleration Support: Malawi activity NASFAM National Association of Smallholder Farmers in Malawi NGO Non-governmental organization NES Nutrition Extension Specialist PD Project Director (Activity staff) PD Program Description PM Program Manager SANE Strengthening Agricultural and Nutrition Extension SCAT System Capacity Assessment Tool SOP Standard Operating Procedures SDS System Development Specialist TA Traditional Authority UBALE United in Building and Advancing Life Expectations UIUC University of Illinois at Urbana-Champaign USAID United States Agency for International Development VAC Village Agricultural Committee VNCC Village Nutrition Coordinating Committee VSL Village Savings and Loan ZOI Zone of Influence 1 EXECUTIVE SUMMARY The Feed the Future (FTF) Malawi Strengthening Agriculture and Nutrition Extension (SANE) was intended to strengthen agricultural and nutrition extension services through policy work, capacity development and extension service provider coordination, and improved research extension linkages. SANE Cooperative Agreement (CoAg) implementation, with an overall budget of $15 million, was scheduled for the November 2015-2020 period. USAID commissioned this mid-term evaluation (MTE) to determine: 1. If SANE “learnings” were being applied outside the FTF Extended Zone of Influence (E￾ZOI); 2. If SANE was achieving its policy, capacity development and coordination strengthening, and research-extension linkage objectives; 3. What implementation challenges SANE faced, how they were addressed, and potential alternatives to how the challenges were addressed; 4. What SANE outputs and outcomes are sustainable, what steps is SANE taking to ensure sustainability, risks to activity and benefits sustainability, recommendations for overcoming sustainability risks, and whether SANE activity integration with other donors or USAID activities enhances sustainability; and 5. Recommendations on how to ensure greater adoption and scaling up in non-ZOI areas of SANE promoted practices. The evaluation was conducted using three methods which included document review, key informant interviews (KII) and focus group discussions (FGDs). FINDINGS AND CONCLUSIONS The evaluation results show that SANE has made good achievements regarding strengthening District Agricultural Extension Services System (DAESS) as a system. SANE has piloted an effective model for DAESS upper echelon development (i.e. the District Agricultural Extension Coordinating Committees (DAECCs) and Area Stakeholder Panels (ASPs), and less so for the District Stakeholder Panels (DSPs). SANE’s Strengthening and Capacity Assessment Tool (SCAT) 2018 survey shows that SANE easily surpassed its year 4 platform functionality targets in year 3; that in the E-ZOI, DAECC functionality is now at 100%; and, that “The overall objectives of the DAESS system are being met”. KII results show an improved capacity of DAECC and ASP in coordination and collaboration in 6 of the 10 E-ZOI districts. Government, NGOs and the private sector were actively involved in DAECC activities such as meetings, joint field activities, hosting of informative intervention presentations and many others. They co￾financed or self-financed their participation in the DAECC activities and as a result, managed to implement activities such as district agricultural fairs and the manure launch in a coordinated manner. Despite the fact that sustainability was not emphasized in its objectives, SANE has identified a number of exit strategy actions which include; cost sharing for some activities with its stakeholders, strengthening of DAESS platforms such as DAECCs and ASPs which are likely to continue their activities after the project period, and working closely with MoAIWD structures such as DAES and ADDs. 2 SANE has to improve on its progress in promoting research/extension/farmer linkages. The primary activity it seems to have actively implemented is mobilization of stakeholder participation at MISST field days, which demonstrate that the Department of Agricultural Research Services (DARS) released varieties. In the few cases where there is a research station in the district, researchers from DARS participate in DAECC activities. With limited time and ability to travel outside of the SANE ZOI, the evaluation team through KIIs and FGDs did not find evidence that SANE “learnings” were applied outside the E-ZOI. SANE developed and published materials, such as the (DAESS) platform Standard Operating Procedures (SOP), that were distributed outside the E-ZOI through WhatsApp networks1 . When the National Agricultural Extension and Advisory Services Strategy (NAEASS) is completed and distributed, then these materials will probably be utilized outside the E-ZOI. SANE contributed to NAEASS and the Agricultural Sector Food and Nutrition Strategy (ASFNS) review. Although the CoAg design described an aggressive NAEASS development and finalization approach, SANE found that the Department of Agricultural Extension Services (DAES) was receiving support under the Agricultural Sector Wide Approach (ASWAp). Thus, SANE scaled back and put NAEASS support on hold. The MTE team found that SANE’s ASFNS support was substantial and appreciated by DAES leadership. Neither strategy has been finalized nor disseminated. Key Informant Interviews (KIIs) reflected near unanimous agreement that strategy finalization and effective dissemination would significantly improve DAESS functionality as stakeholders embrace government “rules of the game”. KIIs and FGDs both reflected improved DAESS platform capacity due to SOP pamphlet and poster distribution and follow up Area Stakeholder Panel (ASP) training. SANE supported stakeholder mapping which is gaining use as a tool for enhancing DAESS platforms coordination of extension service providers. SANE has strengthened District Agricultural Extension Coordinating Committees (DAECC) and ASPs in 6 or less E-ZOI districts (i.e. 4 E-ZOI districts have not received platform capacity building)2 . Where SANE has intervened, issues remain on level/consistency of service provider and researcher DAESS platform participation; District Stakeholder Panel (DSP) functionality; and, the effectiveness of “Training of Trainers” or “Training of Facilitators” in platform capacity building, especially with Village Agricultural Committees (VACs). The MTE team was concerned with the appropriate mix of “systems strengthening” (platform capacity building, linkages) and traditional extension interventions (extension methodologies training, Lead Farmer training, Fall Army Worm training). SANE’s extension strengthening interventions are not sustainable, risk “projectizing” SANE support, and need to better reflect the National Agricultural Extension Policy’s (NAEP) “farmer demand led service delivery” approach so scarce resources are not diverted away from “systems strengthening”. The MTE team found that SANE needs to focus more on improving research/extension linkages so that farmer input would be reflected in on-farm or on-station research. With SANE assistance Malawi Improved Seed System and Technologies Project (MISST) field days demonstrating Department of Agriculture Research System (DARS) released crop varieties are better attended by farmers, agribusiness and researchers with more active networking, input 1 KII with AEMO. 2 SANE has only trained some ASPs in those 6 districts. For example according to KIIs and FGDs there are 19 EPAs in Mangochi but only 3 ASPs in 3 EPAs received training. 3 supplier/farmer contact and researcher/farmer interaction. This interaction, however, is not sufficient to attain results for the performance indicator: “Number of research protocols that incorporate farmers’ involvement in research design and implementation”. In some DAECCs researchers are regular participants. This exposes them, in principle, to farmer technology needs and demands. While KIIs described how difficult it was to strengthen research/extension linkages the team found no specific examples SANE strengthened linkages. In pursuing output/outcome sustainability, SANE supported NAEASS and Agriculture Sector Food and Nutrition Strategy (ASFNS) finalization. SANE cut NAEASS funding at a critical time that could hurt sustainability if the NAEASS is not finalized and effectively disseminated in 2019. In some districts SANE facilitated or inspired DAECC Agricultural Fairs that encourage non￾state actors, especially agribusinesses, to participate in DAESS platforms. Agribusiness DAESS platform participation is essential to DAESS sustainability. While sustainability is not a frequently stated SANE objective (CoAg Program Description, Annual Work Plans (AWP)) SANE’s small staff pushes implementation onto DAES, in theory building capacity and sustainability. SANE has teamed with IFAD’s Sustainable Agriculture Production Project (IFAD/SAPP) and, potentially, Afikepo/Kulima in supporting DAESS structures. This could contribute to outcome/output sustainability. SANE integration with other USAID activities, especially the MISST activity, improved resource use efficiency, but likely had little impact on long term sustainability. There is a significant risk that there will be inadequate political will and financing to distribute the finalized NAEASS and ASFNS and no time for SANE to support strategy implementation. This risk is exacerbated by a Ministry of Agriculture, Irrigation and Water Development (MAIWD) perception that SANE works directly with Chief Agricultural Extension Officers (CAEO) without Agricultural Development Division (ADD) Program Manager (PM) knowledge. There was an insufficient level of awareness of SANE within non-DAES MAIWD departments, and linkages with DAES leadership that may affect SANE district resource allotments needed strengthening. RECOMMENDATIONS To improve SANE output/outcome sustainability, SANE should develop an exit strategy. The exit strategy should focus on SANE’s remaining Length of Project (LOP) on strengthening DAESS “software”. SANE implementation of traditional extension interventions (extensionist training, FAW training, etc.) should stop, as should District Strengthening Action Plans (DSAP) use. As a result, SANE could more easily focus on DAESS software strengthening. The D/COP position should be staffed with a Systems Development Specialist (SDS) with multi-country management information system, customer service relations and/or engineering training and experience. The D/COP/SDS’s primary role will be exit strategy implementation while focusing SANE on strengthening DAESS platform functionality and linkages with nutrition and local government structures. SANE should cease activities in Blantyre and Balaka given International Fund for Agricultural Development/Sustainable Agricultural Production Project (IFAD/SAPP) presence and initiate activities, from the ground up (i.e. beginning with VAC strengthening) in two or three of the remaining underserved districts. 3 SANE’s exit strategy objective should be 3 In Nsanje SANE should return overall DAESS support to CARE while assisting CARE with VAC strengthening in line with SANE’s exit strategy. 4 to pilot VAC and ASP functionality strengthening as a first step in DAESS strengthening (and to generate VAC strengthening models). Specific recommendations related to the exit strategy are grouped together under “Exit Strategy” at the end of the report. 5 PROJECT BACKGROUND The livelihoods of approximately 80% of Malawi’s population depends directly on smallholder agriculture, the majority farming at a subsistence level using traditional methods. However, Malawi’s agricultural context is faced with increasing challenges of reduced soil fertility, erratic weather patterns due to climate change and changing markets. Smallholders must urgently diversify what they produce, change what they produce and how they produce it. However, farmers lack the skills, knowledge and support to improve their productivity. Potential smallholder productivity is constrained by poor education and low functional literacy levels, with 28% of men and 51% of women being illiterate. Poverty is widespread throughout the farming population reducing quality farm input access and affordability. Smallholders lack planning and financial management skills and an appreciation of the business approach to farming. Individual smallholders lack the organization and voice to employ economies of scale, exploit collective bargaining power, and influence policies. Democracy was officially adopted in 1994 and traditional power structures still retain much influence. For farmers to effectively work together in groups, democratic processes and structures must be encouraged and maintained. The lack of strong farmer organizations increases technology development and dissemination costs. Government extension supports farmers, but lacks quality and impact due to inadequate staffing and funding. Farmers must be equipped to respond to the changing agricultural environment, through extension support, training on techniques and sharing of best practices. Farmer groups formally organized into an extension structure, can assists this process, and increase group productivity.4 The FTF Malawi Strengthening Agricultural and Nutrition Extension Services Activity was awarded via cooperative agreement (CoAg) (No. AID-OAA-L-15-0003) between the University of Illinois at Urbana-Champaign (UIUC) and USAID/Malawi. The Agreement was signed for a five-year period November 10, 2015-November 9, 2020 with the Life of Project (LOP) budget was $15,000,000. SANE was to be implemented by UIUC, Catholic Relief Services Malawi (CRS), and Michigan State University (MSU). UIUC is the lead institution, with CRS and MSU as sub-awardees. The Activity is being implemented in partnership with DAES. The SANE CoAg Program Description (PD) did not include a Theory of Change, Development Hypothesis, or Logical Framework. It included the introduction to this section “Approach: Actions and Results”: “The overall outcome of this activity is to improve access to agricultural and nutrition extension services in Malawi, particularly in the FTF ZOI. To produce this outcome, the Activity will work to improve the policy environment, to strengthen coordination and capacity of extension service providers, and to strengthen institutional linkages between research and extension in Malawi.” 4 NASFAM http://www.nasfam.org/index.php/farmer-capacity-building 6 Figure 1. Original AMEP Theory of Change SANE’s first Activity Monitoring and Evaluation Plan (AMEP), submitted May 31, 2016, included the above Theory of Change5 . The Results Framework provided in SANE AMEP’s 2nd revision, submitted to USAID/Malawi on January 31, 2018, with no USAID approval date. This is recreated below in figure 2. 5 This “Theory of Change” was also included in the “The District Agricultural Extension Services System (DAESS) Implementation Guidelines”, Revised July 2017. 7 Figure 2. AMEP 2nd Revision: Results Framework SANE operates in the 10 USAID/Malawi FTF E-ZOI districts: Mchinji, Lilongwe rural, Dedza, Ntcheu, Balaka, Machinga, Mangochi, Blantyre rural, Chikwawa, and Nsanje. 8 EVALUATION PURPOSE, METHODS, AND LIMITATIONS USAID commissioned this mid-term evaluation (MTE) to determine: • Whether SANE “learnings” were being applied outside the FTF E-ZOI; • Whether SANE was achieving its policy, capacity development and coordination strengthening, and research-extension linkage objectives; • What implementation challenges SANE faced, how they were addressed, and potential alternatives to how they were addressed; • Whether SANE outputs and outcomes are sustainable, what steps SANE is taking to ensure sustainability, activity and benefit sustainability risks, recommendations for overcoming sustainability risks, and whether SANE has integrated its activities with other donor or USAID activities to enhance sustainability prospects; and • Recommendations on how to ensure greater adoption and scaling up in non-ZOI areas of SANE promoted practices. The evaluation employed documentation review, KIIs, and FGDs. FGD respondents were selected using a combination of purposive and random selection. SANE identified the vast majority of KIIs and the FGDs were selected "randomly" based on their and supporting MAIWD staff availability. The MTE was hampered by significant administrative constraints. Due to delayed the MTE initiation, 6 Level of Effort (LOE) days intended for the in-country MTE Team, SANE and USAID consultations, data collection, instrument development and planning and wrap up discussions were transferred to US based and virtual consultations and planning. Virtual consultations were not as effective as they could have been due to delays in Malawian Extension Governance (EGS) and Nutrition Education (NES) Specialists contracting. USAID staff were unavailable at the MTE start up and as a result there was no in-country brief. This issue with the addition of EGS and NES contracts not being finalized in a timely manner, created significant challenges on information access, methodological planning, evaluation implementation, and MTE team performance. The shortage of time in the field was a limitation for the evaluation. Nevertheless, the team was able to conduct field work with a large number of KIIs and FGDs with beneficiaries and stakeholders. FGDs were held with DAECC, District Nutrition Coordinating Committee (DNCC), DSPs, ASPs and VACs in Lilongwe Rural, Mchinji, Blantyre, Balaka and Mangochi. Planned FGDs in Chikwawa and Nsanje were cancelled by the DOA due to time conflicts with Malawi’s Fertilizer Input Subsidy Program (FISP). Not knowing whether the EGS and NES would participate in (Chichewa) FGDs caused FGDs to be scheduled haphazardly and at the last minute with DOA staff assistance. This could be viewed as another limitation of this evaluation. FGDs were held with DAES and DOA staff, DAECC members, DNCC members, VAC members, ASP members, ASP representatives on a DSP, combined ASP and VAC members (who were attending a Lead Farmer Network meeting), and combined ASP, and Area Nutrition Coordinating Committee (ANCC)/Health Surveillance Assistants (HAS). Other FGDs were conducted with staff of the nutrition as well as extension methodologies sections of DAES, Blantyre ADD, as well as Machinga ADD staff. 9 KIIs with DAECC leadership did take place in the same districts of Lilongwe, Mchinji, Balaka, Blantyre, Mangochi, Chikwawa and Nsanje. KIIs were also held with SANE COP, SANE NES, Director of DAES, Programme Manager of Lilongwe ADD and Machinga ADD, District Agricultural Development Officers (DADOs) and Agricultural Extension Methodology Officers (AEMOs) for Lilongwe West, Lilongwe East, Mchinji, Blantyre, Balaka and Mangochi districts. Choice of respondents to participate in FGDs and KIIs in the districts were based on their roles and involvement with DAESS as well as involvement in the implementation of SANE Project. KIIs were also held in Mulanje to gauge whether SANE learnings were applied outside the E￾ZOI. In the end there were 40 KIIs and 15 FGDs with over 100 participants. See detailed list of all KIIs and FGDs conducted in the appendix as well as a list of all key documents reviewed. It is possible that the MTE team findings from one non-SANE district do not apply to the other non-SANE districts. This is indeed a limitation. KIIs and FGDs were recorded with participant permission and used for content analysis and accuracy. In addition to the recordings, the MTE team took detailed notes as a way of triangulating with the recorded content and written reports. The MTE team shared the recordings and listened to them several times in addition to reading transcriptions made from the recorded discussions and interviews. As with most performance evaluations teams have a limited amount of time to gather information and have to conduct the evaluation under time constraints. These constraints prevented the team from visiting the large list of EPAs that work with SANE, and limited the amount of work the team could work outside the ZOI. This evaluation is constrained by this time limitation. FINDINGS CONCLUSIONS AND RECOMMENDATIONS The results of the MTE show that SANE has made some good progress in promoting coordination and collaboration at the upper echelons of the DAESS in some of the E-ZOI districts it is operating. This is more conspicuous in the DAECC and ASP platforms. Government, NGO and the private sector organizations are working together in implementing a number of activities such as meetings and coming up with joint plans and joint field visitations. On a number of occasions, the organizations participated in joint financing or self-financing the activities. Most of the districts have been able to conduct such activities like district agricultural fairs which demand a lot of effort and commitment from each of these organizations and the benefits of such activities to the smallholder farmers is enormous. The capacity of the DAECC and ASP platforms on the DAESS system has been enhanced. From the KIIs and FGDs conducted by the MTE it appears that the training which has helped to enhance the capacity of these platforms has not effectively cascaded down to lower platforms such as the VACs. The capacity of DSPs needs improvement. SANEs attention needs to focus on these platforms. There is need to ensure that DAESS is made operational in the districts if SANEs efforts are to be sustainable. Sustainability is another area where SANE appears to some extent to be going in the right direction. The production and printing the SOP materials on DAESS is a good step for promoting sustainability in that those materials will continue to be available and useful even after SANE project period is over. Although not SANE’s mandate, the materials can actually be transferred to non E-ZOI districts so that they can also start using them. KIIs with AEMO for Mulanje district revealed that a number of these materials are already on WhatsApp and being 10 shared among AEMOs even outside the E-ZOI districts. SANE has also made efforts to link with other programmes like IFAD/SAPP, EU funded Kulima and AFIKEPO programmes where they share lessons and experiences and through this one expects that lessons on the DAESS will be extended to other areas. Many of SANE’s implementation challenges sprang from activity design. While agricultural extension capacity constraints were well researched, SANE implementation mechanisms were inadequately informed and agriculture-nutrition linkages poorly articulated. For example, the design described step-by-step DAESS capacity building in all E-ZOI districts for the “life of the activity”. This process would “be overseen by District Activity Liaisons” who would be “identified from the Activity team, DAES, the Ministry of Agriculture, Irrigation, and Water Development (MoAIWD), and Lilongwe University of Agriculture and Natural Resources (LUANAR) and assigned to each of the 10 Districts”. Substantial capacity building was planned to enable the Malawian Forum for Agriculture and Advisory Services (MaFAAS) “to assume the role of the national stakeholder panel and to strengthen its advocacy capacity”. District Nutrition Coordinating Committees (DNCC) were to be “strengthened to improve+ functioning and enhance linkages to DAESS and other relevant actors”.6 These interventions never materialized. SANE also suffered from extreme levels of staff turnover7 and USAID budget cuts while addressing contextual challenges such as inadequate non-state actor DAESS participation, Malawi’s “incentive culture” suppressing farmer DAESS participation, and general organizational “stove piping” that detracted from activity, local and national government agency coordination and collaboration. KIIs showed that SANE and DAESS progress depends almost entirely on individuals which is not unusual with donor funded projects. If a District Agricultural Development Officer (DADO), AEMO or DAECC Chairperson is effective in supporting DAESS then SANE and DAESS will succeed in that district. SANE recognized this when selecting districts with DAESS “champions” as priority capacity building targets. However, SANE, Department of Agriculture (DOA), Non-Government Organization (NGO) and agribusiness staff are transient. The real determinant of SANE output/outcome sustainability, and of DAESS, will be Village Agricultural Committee (VAC) and ASP capacity in elucidating and satisfying farmer extension service demands. District VAC and ASP capacity is more permanent than DOA or DAECC capacity. VAC and ASP capacity building is thus key to SANE output/outcome sustainability. While DAECC strengthening and service provider coordination improves chances of SANE output/outcome sustainability, if this does not result in tangible 6 SANE Cooperative Agreement Program Description 7 The FY 2017-18 Annual Work Plan states that the COP resigned in year 2; USAID advised against hiring a replacement due to pending budget cuts; the Agricultural Extension Specialist (AES) was elevated to Acting COP, then COP; the Research-Extension Linkages Specialist position, one of two Monitoring, Evaluation and Learning Specialist positions, and administrative staff at UI were eliminated due to budget cuts. Key Lessons: Successful implementation of DAESS hinges on functional platforms at grassroots level (strong Clusters, VACs and ASPs) Full commitment and investment on these structures is therefore key. The District Agricultural Extension Services System: Experiences from Balaka (undated) 11 improvements to small holder food security farmer food security, livelihoods, and resilience DAESS will not be sustainable8 . KIIs indicated that if DAESS mobilized and coalesced farmer demands for extension services are not satisfied farmers will likely lose interest in DAESS platform participation9 . Inadequate NGO and agribusiness DAESS participation increases this risk. If farmers are forced to pay for low quality services (“those who benefit pay”) and VAC, ASP, DSP participation10 (i.e. to cover transport, refreshment costs, Lead Farmer (LF) training) they may refuse to participate, hurting DAESS sustainability, and the political attractiveness of DAESS to district government. SANE works at the “top” of the DAESS structure (DAECC, DSP, ASP) and expects strengthening to “cascade down” to VACs, clusters and community members11. As there is significant information loss in the cascade, inadequate quality monitoring and training supervision reduces this approach’s effectiveness and risks results sustainability. Intended or not, SANE has been “projectized” at the VAC, ASP and DSP level. When SANE leaves, if farmers don’t feel VAC and ASP participation provides benefits above costs, the results will not be sustainable. This risk is exacerbated by SANE resource dissipation by implementing “non-core business function” activities (i.e. extensionist training, FAW training, etc.). These activities detract from SANE strengthening of DAESS “software”12: VAC ability to elucidate farmer service demands and hold service providers accountable; consistency and effectiveness of ASP and DSP meetings; more effective DAESS platform linkages with nutrition coordinating committees and local government structures. SANE needs to improve its strategy for promoting research-extension-farmer linkage. So far the efforts that SANE has in this area seem not to be producing the results expected. Integration of agriculture and nutrition platforms is not adequate. Evaluation Question 1: To what extent do we see evidence that the DAES is taking learnings from SANE and applying them in non-ZOI areas? Although not a direct SANE mandate, the SANE MTE team observed no hard evidence of SANE learnings application outside the E-ZOI. However, SANE materials have been distributed outside the ZOI. This finding is the result of KIIs and FGDs along with project reports as the team had limited opportunity to look for application outside of the SANE ZOI. As the NAEP was launched in 2000, with implementation guidelines published in 2006, it is difficult to isolate SANE “learnings” outside the E-ZOI from NAEP implementation prior to or concurrent with 8 This isn’t to say DAES, a government agency, will not be sustainable. 9 Or, if demands are perceived as being satisfied outside DAESS channels, (i.e. through more traditional DADO￾AEDO-farmer or LF-farmer connections). 10 “Systems Strengthening to Improve Agricultural and Nutrition Extension Services: An Assessment of SANE and DAES Efforts in Malawi”, joint DAES-SANE authorship, October 2018. 11 SANE’s unsatisfactory experience with ASP/ANCC TOF training of VACs/VNCCs is documented in 2018/19 AWP. 12 Dr Wilkson Makumba, Director of Agricultural Research Services, chairing the June, 2017 SANE Project Steering Committee: “The Chairperson thanked members…were reminded that Strengthening Agricultural Nutrition and Extension (SANE) project was conceptualized to strengthen the extension system in Malawi. The project is rather complex as it requires to understand it from the systems thinking or software perspective. Participation of the stakeholders in agricultural extension and advisory services in the platforms for engagement is critical to the success of the project.” 12 SANE. For example, United in Building and Advancing Life Expectations (UBALE) initiated DAESS platform strengthening in three districts (Blantyre, Chikwawa, Nsanje) prior to SANE; Harvest Help Find Your Feet13 strengthened DAESS structures in Rumphi circa 2006; the MTE Team interviewed one KI who, as an Agricultural Extension Development Officer (AEDO), initiated VAC and ASPs without external support; and IFAD/SAPP support to decentralization strengthening began in 201114 in Blantyre, Chiradzulu, Balaka, Nkhotakota, Lilongwe and Chitipa. Kulima/Afikepo was designed and is initiating implementation with some knowledge of the SANE experience in Chiradzulu, Chitipa, Karonga, Kasungu, Mulanje, Mzimba, Nkhata-Bay, Nkhotakota, Salima, and Thyolo.15 Finally, WhatsApp forums (of DAECC chairpersons, AEMOs, Agricultural Extension Development Coordinators (AEDCs) and AEDOs) created with and without SANE involvement have distributed SANE materials (SOP booklets, poster pictures) to other than E-ZOI districts. While the extent of material application was impossible to assess with available MTE resources the MTE team believes that non-ZOI material application will take place after NAEASS completion and dissemination16. Thus, SANE “scaling” may come via a DAESS development continuum beginning as early as 2006 (with DAESS implementation guideline publication) and gaining momentum with EU, IFAD, NGO and USAID investments; DADOs and DAO staff recognizing DAESS as a solution to crippling resource constraints; and NGOs and non-state actors seeing that coordinated, collaborative and pluralistic agricultural and nutrition extension is good for their beneficiaries, their donors and their institutions’ longevity. Evaluation Question 2: What would they recommend that SANE do to ensure greater adoption and scaling up in non-ZOI areas of the practices that they are promoting? Interviews and project reports show that altogether IFAD-SAPP, Kulima/Afikepo and SANE support DAESS strengthening in 26 districts, with overlap in 6 districts: Blantyre, Balaka, Lilongwe (SANE, SAPP) and Nkhotakota, Chiradzulu, and Chitipa (SAPP, K/A). So, 20 of Malawi’s 28 districts currently enjoy external DAESS strengthening investments. The 3C (coordination, collaboration, co-location) theory would seem to advise against SANE scaling up in non-ZOI areas where IFAD￾SAPP and/or Kulima/Afikepo are already supporting DAESS. 13 Decentralization in The Agricultural Sector In Malawi: Policies, Processes and Community Linkages, A Status Report Prepared by: Randi Kaarhus and Ramji Nyirenda, Noragric Report No. 32 May 2006 14 IFAD: Malawi Sustainable Agricultural Production Programme Supervision Report, October 31, 2018 15 “USAID will also implement in parallel from 2016 the 'Malawi Strengthening Agricultural and Nutrition Extension Services' project aiming to support implementation of the District Agricultural Extension Services System (DAESS) in other 10 districts than AFIKEPO”: Action Document for 'AFIKEPO' Nutrition programme in Malawi, https://ec.europa.eu/europeaid/sites/devco/files/ad2-annex-2-malawi-2016_en.pdf 16 KIIs indicated that AEMOs outside the E ZOI were impressed with the materials and wondered why they weren’t distributed officially by DAES or MAIWD. “Implementation of new resources should follow the DAES chain of command so that junior officers are not responsible for instructing their supervisors on how to use extension materials.” (DAO staff quoted in “The 2017 Malawi Agriculture Extension Field Notebook as a Capacity Building Tool, October 2018) 13 SANE has piloted an effective model for DAESS upper echelon development (i.e. the DAECCs and ASPs, less so the DSPs). SANE’s Strengthening and Capacity Assessment Tool (SCAT) 2018 survey shows that SANE easily surpassed its year 4 platform functionality targets in year 3; that in the E-ZOI DAECC functionality is now at 100%; and, that “The overall objectives of the DAESS system are being met”.17 Thus, SANE’s upper echelon model has effectively mobilized DAECC and ASP capacity in a little more than 18 months. Supporting cross visits by DAES teams from ZOI and Non ZOI districts to learn and share knowledge and lessons learned from their DAESS would facilitate the non ZOI adoption of new practices. Evaluation Question 3: Are Interventions on course to result in improved access to extension services for smallholder farmers in the zone of influence and beyond, particularly with respect to the key result areas: a) Improved policy environment for delivery of extension services. SANE’s ASFNS and NAEASS contributions may assist in strategy finalization and distribution in 2019. KIIs were virtually unanimous in thinking strategy dissemination will lead to “improved access to extension services” by showing government commitment to DAESS and agriculture-nutrition linkages. Unfortunately, SANE had a role in delayed strategy finalization. In response to 2017 funding reductions, SANE pulled back on its NAEASS financial support. .18 Concurrent with the end of ASWAp, funding this halted progress on strategy finalization. More importantly, this decision was not made with all stakeholder consultations19. SANE’s reduction of strategy completion funding, while continuing less impactful extensionist and FAW interventions, ignored SANE’s comparative DAESS strengthening advantage and core business function. Policy implementation is fundamental to DAESS impact in the E-ZOI, and nation-wide. On the other hand, extensionist and FAW training are locally focused, short-term interventions implemented by a wide range of donor and NGO activities. SANE should resume funding for policy finalization. In the absence of policy finalization, UBALE, IFAD/SAP, NGO, and SANE DAESS capacity building are improving the agriculture and nutrition extension policy environment to better reflect NAEP guidelines, pluralistic extension, and farmer demand led services as seen in Balaka, 17 Draft slide presentation on 2018 SCAT survey results provided by SANE. 18 KII with Director, DAES. 19 “The workplan was presented highlighting key activities to be implemented. Attention was drawn to the reduction in budget. The annual budget was reduced from the original $3 million to $1.14 million for the year. Due to the budget cut, the project made some adjustments which included: reduction of activities on objectives 1 and 3, and removal of Michigan State University as lead for Result 3, increased focus on establishing partnerships; integration and cost sharing; and reducing two staff members.” 3rd Project Steering Committee Minutes, December 2017. Even though SANE works to implement the DAESS – which is part of the National Agricultural Extension Policy – many stakeholders do not recognize the influence of SANE’s policy work on service delivery at the field level. SANE Year 3 | Annual Report 1 October 2017 – 30 September 2018 14 Blantyre, Mchinji, Rumphi, Nsanje, and Mangochi. However, a strong national or regional DAESS policy and a coordination champion is lacking, due perhaps, to SANE not supporting MaFAAS as planned. Additionally, there is tepid support for SANE’s approach among some MoAIWD national and ADD offices due to SANE’s implementation approach20. Evaluation Question 4: Interventions on course to result in improved access to extension services for smallholder farmers in the zone of influence and beyond, particularly with respect to the key result areas: b) Improved institutional and field level capacities for service delivery. KIIs reflected improved DAECC and ASP capacity. DAECC’s in at least 6 of SANE’s 10 E-ZOI districts have improved coordination and collaboration of government, NGO and private sector agricultural extension service providers. DAECC’s mounted coordinated interventions (“Manure Launch” in Balaka, agricultural fairs), documented and shared meeting notes, hosted informative intervention presentations, and self-financed. In all interviewed DAECC’s DADO support and secretariat capacity were key to effectiveness. Nutrition extension provider coordination has improved around the Integrated Homestead Farming /Household Approach (IFH/HH) while DNCC participation at DAECCs is uneven. Several DAECCs embraced stakeholder mapping and WhatsApp networks as coordination tools and several DAECC‘s routinely review and reorient, when needed, new NGO or project entrants into the district agricultural extension landscape. DAECCs have reduced extension service redundancy, spread extension capacity more evenly geographically, and eliminated some contradictory extension messages. DAES FNOs, as DOA technical staff, may be the only nutritionists on the DAECC. While FNOs are also DNCC members this level of nutrition representation needs reinforcement by other stakeholders during policy, budget or technical discussions. There may also be nutritionists in ASPs or VACs who are not adequately informed and less than effective agricultural representation in ANCCs and VNCCs. Hence, nutrition capacity building, to compliment and broaden SANE’s IHF/HH focus, is critical at DAECC, DNCC, ASP, ANCC, VAC, and VNCC levels. The issues that ASPs elevated to higher platforms were crop production, and marketing based on practically all FGDs conducted. Nutrition issues were hardly ever mentioned in FGDs and KII conducted by the team. Likewise, SANE rapid assessment (2018) reported that the two most frequently mentioned SANE trainings were related to the DAESS system itself, and IHF/HHA, believed to be the approach to link agriculture, nutrition, and family decision-making. However, IHF/HH extension is a DAES front line staff responsibility and not part of SANE’s system development core business function. Given the limited number of 20 KIIs with four current or former ADD program or assistant program managers. Key Lessons: A functional DSP provides the most direct link between smallholder farmers and service providers. It is therefore a crucial platform in achieving quality pluralistic demand driven agricultural advisory services in Malawi The District Agricultural Extension Services System: Experiences from Balaka 15 community structures that might bridge agriculture-nutrition gaps SANE’s approach to systemic agriculture-nutrition linkage capacity building should spring from ASP and VAC functionality capacity building that includes ANCC, VNCC and Care Group Promoter participation. The vast majority of DAECC work is not farmer led. While DAECC members may say they know demands because they work closely with farmers, this reflects the top down extension approach the NAEP wishes to reorient. There is evidence that DSPs, with DAECC member participation, do recognize farmer demand. Unfortunately, almost all E-ZOI DSPs function sporadically due to transport and allowance issues21 that force DSPs to meet quarterly or not at all. DSP non￾functionality defeats farmer voice inclusion in service provider accountability, policy dialogue, and extension needs identification. If the DSP effectively aggregated ASP farmer demand contributions, then DAECC member DSP participants could respond and, with adequate ASP capacity, farmers can hold service providers accountable at DSP or ASP level. If the DSP is inconsistent then the DAECC may periodically invite an AEDC to participate and present farmer demands (for example). This is hardly in line with the NAEP or SANE’s intent. The Blantyre DAECC Vice-Chair stressed that as the DAECC and DSP meet quarterly, and the VACs and ASPs meet monthly and SANE is working to address this issue. The DAECC and DSP are choke points in responding to farmer demands (due to the lack of meeting synchronization). His solution was to strengthen VAC and ASP functionality so that farmer demand satisfaction takes place at the Extension Planning Area (EPA) or Group Village Headman (GVH) level, via the ASP. Less time sensitive issues that the ASP cannot resolve can be passed to the DSP and DAECC. SANE has improved some ASP capacity in 6 E-ZOI districts while other ASPs need further training. Also VACs have not been trained adequately and hardly figure in SANE implementation. VAC capacity is to be addressed through ASP level or AEDO “Training of Trainer” or “Training of Facilitator” training, and “cascade down” to VAC and community members. In some districts VACs are referred to as “Steering Committees” composed of representatives from the village’s entire farming community (in Blantyre the community is grouped into “clusters” that feed into the VAC Steering Committee). This adds additional complexity in communicating and training how the NAEP and DAESS are to change farmer￾extension service interface.22 SANE’s main institutional and DAESS capacity building tool is the District Strengthening Action Plan (DSAP). DSAPs“….outline priorities and guide the work of the Activity…” and “….identify district-specific extension priorities derived through a collaborative and decentralized process”23. Over the three AMEP revisions two performance indicators that: 1) track the number of districts that have developed DSAPs; and, 2) the number of districts that have 21 All KIIs and FGDs. 22 KII with Blantyre DAECC representative. 23 SANE Cooperative Agreement, Attachment B: Program Description From the Cooperative Agreement: “The development of these District Strengthening Plans will use stakeholder input from the stakeholder platforms at the District level and it will build on identified farmer-expressed needs and opportunities (MEAS, 2012; Swanson & Rajalahti, 2010).” 16 completed at least 75% of DSAP activities” have been monitored. As could be expected DSAP content varies with some DSAPs emphasizing DAESS platform strengthening while others emphasize more traditional extension (LF capacity, FAW prevention, etc.). In 2018/19 SANE will develop DSAPs in Chikwawa, Dedza, Machinga and Mchinji. The AWP excerpted table below shows that SANE has already programmed support to those districts24 (M&E Results Dissemination, Extension Methodologies Training, FAW training and monitoring, ICT based extension roll out, etc.). These “activities by district” raise questions about farmer input in SANE activity identification; the DSAP’s top down quality; and whether activities identified prior to DSAP completion illustrate past DSAP implementation reality. 25 Understandably, most donor projects would have difficulty responding to unstandardized farmer identified service demands in 10 or more agro-ecological areas (i.e. E-ZOI districts)26. 24 Table from SANE’s Annual Work Plan 1 October 2018 – 30 September 2019 25 At least one district KII decried how SANE staff must be physically present to manage DSAP implementation resources. This makes implementation inefficient and not what they expected after SANE’s DSAP approval. 26 IFPRI’s 2017 “State of AEAS Provision in Malawi” (pages 15-16) said this about what advice farmers wanted from extension: “It was surprising that most of the advice requested is…. on older agricultural practices-minimum tillage, pit planting, manure or fertilizer making, composting, soil cover or mulching,….” and “A third of farmers requested food, health and nutrition related advice.” 17 Table 1. Activities by District for FY 2018/19 There is opportunity for SANE to employ more resources uniquely on the DAESS system (or “software”) strengthening. This would allow DAECC strengthening, but broaden and deepen ASP strengthening and, most importantly, strengthen large numbers of VACs. (SANE might also have focused on only 2 or 3 districts.27) This would position SANE to inform, with more experience, other donor DAESS strengthening interventions and national DAESS roll out. In this way SANE would have given real form to NAEP, DAESS and SANE objectives of extension that responds to small holder food security farmer demands. Only 41% of the 2018-19 activities listed above focus on DAESS strengthening and a much smaller percentage focus on farmer demands IFPRI identified in their survey.28 Evaluation Question 5: Interventions on course to result in improved access to extension services for smallholder farmers in the zone of influence and beyond, particularly with respect to the key result areas: c) improved research/extension linkages. The team found limited evidence that SANE has improved research/extension linkages as defined by the AMEP performance indicator. SANE helps mobilize stakeholder participation at MISST field days that demonstrate DARS released varieties. SANE support results in a more diverse audience (including farmers, researchers, agribusinesses, extensionists); more active networking (between input suppliers and farmers seeking demonstrated variety seed), and researcher/farmer interaction. SANE uses this collaboration with MISST to report on research/extension linkage performance indicator achievement: “SANE worked with the MISST project to ensure farmers’ involvement in research on the five focal value chains (soybean, groundnut, pigeon pea, maize, and orange￾fleshed sweet potato).”29 However, MISST is not a research project.30 It is an activity that demonstrates and distributes DARS released crop varieties to strengthen technology delivery. Farmer participation at SANE supported MISST demonstration field days does not constitute “farmer involvement in research design and implementation”. Some DAECCs enjoy frequent researcher participation. This exposes researchers, in principle, to farmer input and demands. This may lead to greater farmer input in research agenda development. All KIIs were asked the question of SANE performance in strengthening research/extension linkages. All described how difficult it was to strengthen research/extension linkages but none gave an example of SANE strengthened linkages. Recommendation: Change indicator wording to eliminate “research” references (“responsible for agricultural and nutrition research”, “research actors”, “research protocols”, 27 KII with the DAES Director 28 Even less if you consider that half or more of DSAP implementation activities are not demand led and may be more traditional “top down” extension. 29 SANE Year 3 Annual Report 1 October 2017 – 30 September 2018 30 KII with MISST. 18 research design and implementation”) and focus on technology demonstration or dissemination linkages. Evaluation Question 6: What challenges has SANE faced in the course of implementation and how have they been addressed? What recommendations can be made on alternative ways of addressing these challenges? Table 2 provides a list of the challenges identified, how SANE has addressed them and the alternative ways such challenges could have been addressed. The challenges identified include shortage of staff, budget reductions, inadequate participation of non-state actor in the DAESS platforms, inadequate integration of agriculture and nutrition extension activities. As can be seen from the Table, SANE has made efforts to address the challenges and is also providing alternative ways of responding to the challenges. Table 2. Implementation Challenges, How Addressed, Alternatives Implementation Challenge How Addressed by SANE Alternative Ways to Address Staffing challenges (COP, DCOP, M&E, governance specialist, communications, activity liaisons); Agriculture Extension Specialist becomes DCOP then COP; DCOP, AEREL, MEALs, activity liaisons remain vacant or are eliminated; Staff COP (or DCOP) position with “Systems Development” capacity; recognize DADO or AEMO as de facto “activity liaison”; focus activities on “systems strengthening”; Financial challenges – 3rd year budget reduction; how to prioritize activities with reduced funding; Reduced/eliminated activities in 4 districts; rolled back NAEASS finalization commitments; eliminate farmgate focused performance indicators REL sub￾grantee; Consult with DAES stakeholders in adjusting budget; fund strategy commitments; shrink districts to 6, or 4, permanently; eliminate non-systems development interventions; encourage UBALE implementers to resume DAESS strengthening; develop required 5-year work plan after award; “Incentive culture” suppresses DAESS platform functionality; Promote internal resource mobilization (maize trading, gardening, local food purchase and preparation); jawboning self￾reliance to DAESS participants (while paying lodging, DSA, transport for SCAT results dissemination, other); Promote DAECC member funded “pot of money” to support DSPs/ASPs; train DSP/ASP/VAC in proposal development to source DAECC or other funding; capitalize VSLs as reward for platform participation and enable future self-financing; Inadequate non-State DAESS platform participation; No evidence this was formally recognized; SANE inspired agricultural fairs resulted in increased NGO, private sector interest and participation in Finalize/disseminate NAEASS; employ non-participation “shaming”; develop VAC capacity to elucidate real demands making DAECC an attractive source of 19 DAESS platforms; market and activity design information; Integration of agriculture and nutrition inadequately articulated in activity design; SANE promotes IHF/HH extension approach; Adopt a “systems approach” to promoting DAESS-Nutrition Coordinating Committee￾Harmonized Care Group Model collaboration or coordination; Stove piping suppresses broader MoAIWD and/or local government DAESS ownership. No evidence that lack of MoAIWD or local government support was formally recognized; SANE conducted extensive stakeholder consultations in the beginning, perhaps less after initiation. More “systems analysis” during design; focus on DAESS linkages with local government31; develop broader national level MAIWD understanding of SANE and DAESS. Evaluation Question 7: What steps has SANE taken to address sustainability of its outcomes/outputs? Sustainability is not a stated SANE objective (in design, AWP, reports). SANE’s 4th year AWP mentions sustainability three times: policy dialogue, DSAPs, and environmental sustainability. The majority of CoAg PD sustainable and sustainability references are to environmental sustainability. Nonetheless, SANE identified these exit strategy actions32: a) cost sharing of resources (e.g. finances, fuel to activities); b) DAESS platforms: DAECC, DSP and ASP leading development of activities; c) DSAPs are co-created and funded with district partners with SANE mentoring on how to support systems strengthening; d) Working more closely with DAES and ADDs; e) Build trainer teams for DAECCs and ASPs to facilitate training in IHF, methodologies, SOPs; e) Backstopping DAECCs in capacity building of VACs by ASPs; f) Identifying and mentoring champions to advocate for DAESS; g) Linkage with local government platforms to recognize and value DAESS platforms. These constitute the most evident steps SANE is currently taking towards sustainability. (See above MTE issues on DSP functionality, DSAP content, and the effectiveness of ASP’s building VAC capacity.) 31 For limits on this approach see “Africa Works: Disorder as Political Instrument”, Patrick Chabal and Jean Pascal Deloz, September 2010 and http://pdfproc.lib.msu.edu/?file=/DMC/African%20Journals/pdfs/Journal%20of%20the%20University%20of%20Zimba bwe/vol27n2/juz027002008.pdf 32 November 22, 2018 E-mail response from SANE COP to the MTE Team Leader question: “Has SANE begun thinking, even in an initial sense, on what is going to happen to SANE activities at the end of the cooperative agreement?” 20 SANE supported ASFNS and NAEASS finalization. These policies will guide agricultural and nutrition extension providers in increasing farmer access to quality, demand led extension services. SANE also promoted demand driven DAESS policy by facilitating farmer participation in policy discussions. ASFNS finalization consultations included ASP and ANCC member contributions. SANE enabled DAESS platform representatives – including ASP farmer members – to attend MaFAAS Extension Week where extension and advisory service policy issues were discussed. SANE also facilitated ASP and ANCC member policy dialogue participation at the National Agriculture Fair. Farmer participation like this may ensure that extension service policies are farmer demand led. KIIs were consistent in believing that the effective satisfaction of farmer identified extension needs is key to DAESS sustainability. SANE has facilitated or inspired DAECC’s Agricultural Fairs. These have stimulated non-state actor participation, especially private sector participation, at the DAECC and other DAESS levels. In Balaka, where the DAECC has held three consecutive annual agricultural fairs, the National Small Holder Farmers Association (NASFAM) holds the DAECC Chairperson position. KIIs with three Agora and Farmers World Farm Services Unit (FSU) staffers illustrated how agribusinesses view DAESS platforms as sources of marketing information and customer access. FSU currently has 90 extension staff attached to its national network of input supply stores33. Given agribusinesses input and output marketing, technology and information dissemination, and policy dialogue participation private sector participation in DAESS platforms will buttress sustainability. However, DAESS should not become over reliant on private sector support for DAESS platforms or farmer demand led extension services34. DAESS platform strengthening has resulted, in some cases, in greater visibility and consideration of agricultural development investment needs by district government. Nonetheless, KIIs stressed that district government still favors more visible construction projects over agricultural projects.35 SANE has teamed with IFAD/SAPP and, potentially, Afikepo/Kulima in supporting DAESS structures. This could contribute to outcome/output sustainability as long as Malawi’s overall governance performance does not lead to reduced multi-lateral donor funding. A number of recommendations are given at the end of this report under “exit strategy” that address the sustainability issue. Recommendation: SANE, DAES, and USAID should formally discuss the 3C’s with IFAD/SAP, Afrikepo/Kulima, and other donors or NGOs to identify how greater SANE/SAP/Afrikepo/Kulima co-location, coordination and collaboration can contribute to NAEASS and ASFNS finalization and distribution, DAESS strengthening and expansion; 33 KIIs with FSU national manager and 2 FSU regional managers. 34 For example, NASFAM’s role in encouraging but then not buying burgeoning small holder pigeon pea supplies was noted as a problem during two FGDs 35 “Systems Strengthening to Improve Agricultural and Nutrition Extension Services: An Assessment of SANE and DAES Efforts in Malawi”, joint DAES-SANE authorship, October 2018. NASFAM is involved in lobbying Government to ensure that the smallholder perspective is considered when developing any policies that impact on the potential of smallholder farming. NASFAM website http://www.nasfam.org/index.php/advoca cy-and-lobbying 21 agriculture nutrition linkages, and sustainable small holder food security farmer demand led extension services nationwide36. The SANE CoAg PD describes MaFAAS involvement with and contributions to DAESS strengthening and SANE implementation at the national, district and local level; in each SANE policy, institutional capacity building and research/extension linkages objective; and with cross cutting themes such as gender, environment and learning by doing. Aside from small financial grants and promoting farmer participation in MaFAAS functions this SANE support has not materialized. As a result SANE appears to lack a significant non-state national champion. This has the potential for hurting sustainability prospects. Recommendation: SANE should make a concerted effort to fill the apparent “DAESS national champion” void, either through MaFAAS capacity building or identifying another potential champion (National Stakeholder Panel, CISANET) and working with other donors (IFAD/SAPP, etc.) and DAES to see what is possible. This may warrant short-term technical assistance in determining and planning what is possible with SANE’s relatively short remaining LOP. DAECC strengthening and DAECC service provider coordination and collaboration improves chances of SANE outcome/output sustainability. That being said, collaboration and coordination alone will not ensure sustainability if not accompanied by tangible improvements in small holder subsistence farmers’ food security, livelihoods, and resilience. Given the transient nature of DAES, NGO and donor project staff, and the observed importance of individuals (DADO, AEMO, DAECC Chairperson) sustainability will hinge on VAC and ASP capacity and appreciation for DAESS structures. As suggested by KIIs one way to build sustainable VAC and ASP capacity to contribute to DAESS objectives is to scale up their “stakeholder mapping” (Venn diagram) and other participatory farmer needs identification capacity. There is opportunity for SANE to focused on this in a determinant way. (See recommendation on VAC strengthening.) While SANE’s reliance on DOA staff in implementation theoretically contributes to sustainability SANE may have mobilized DOA staff in a way that creates institutional jealousy and resentment (“cherry picking”) as opposed to employing existing bureaucratic structures (i.e. ADD, DAES, other MoAIWD structures) and building broad MoAIWD capacity, awareness and support (overcoming MAIWD ‘stove piping’) for DAESS implementation. In addition, the Project Steering Committee (PSC) where broader SANE ownership could be achieved is inconsistently attended, not attended by MoAIWD leadership, and not convened on a consistent basis. Recommendation: SANE’s exit strategy should include a communications sub-strategy aimed at broad MoAIWD understanding and ownership of SANE’s approach, components, accomplishments and short falls. A specific focus should be on winning the support of ADD program managers and staff. This could entail passing some SANE activities to ADD staff (overseeing more aggressive ASP and VAC capacity building) or jointly supervising ASP and 36 The IFAD SAPP Supervision Report November 2018 includes this action in the “Exit Strategy: Main Issues” section: “(iii) continuation of the District Agricultural Extension Service System (DAESS) structure;” 22 VAC capacity building with ADD offices. SANE focus on “core business” DAESS “software” in 2-3 districts should free up capacity for pursuit of broader institutional SANE ownership. Evaluation Question 8: What are the key risk factors for longer-term sustainability of the results? What additional steps need to be taken in order to improve the chances for sustainability of the activities and benefits derived from the activities? The following risks discussed below were obtained through Key Informant interviews and Focus Group Discussions. Risk: Inadequate non-state actor participation in DAESS increases the risk that farmer demands, once mobilized and coalesced, will not be satisfied. Significant failures could destroy the DAESS brand, increasing farmer demands for “incentives” to participate in DAESS structures. This risk is higher if DAESS platform financial requirements are pushed onto small holder farmers (at least 50% of platform participants). Failure will also affect district government perception of DAES and DAESS. Recommendation: DAECC’s should use stakeholder mapping to ensure that District Executive Committee (DEC) or District Council (DC) agricultural and nutrition actor inventory list is comprehensive and up to date. If an NGO or agribusiness does not participate in DAECC meetings or DAESS platforms local government could: 1) publicize a list of non-participants to “shame” offender into more consistent participation; 2) communicate with the DAESS non￾participant on DAESS platform participation importance. DAECC’s can also stage agricultural fairs to stimulate DAECC and DAESS participation. Risk: Delayed ASFNS and NAEASS completion; inadequate strategy publication/distribution; inadequate financial and human resources for strategy implementations; inadequate SANE LOP to support strategy implementation. (See recommendations 9 and 16). Risk: Inadequate 3C approach to DAESS institutionalization reduces higher level MoAIWD support. There is evidence that that SANE is by-passing ADD Program Managers, hurting regional DAESS ownership and coordination. MoAIWD “stove piping” leads to non-DAES departments not supporting DAESS. (See recommendation 18). Risk: Seemingly unique agriculture-nutrition focus on IHF/HHA37 model dissemination and weak SANE and DAESS linkages with VNCC, ANCC, DNCC, or DAHN’s “Harmonized Care Group Structure”. Given DAES capacity, SANE is hesitant to push expanded agriculture￾nutrition linkages, relying on IHF/HH dissemination to VACs, Village Development Committees (VDC), ASPs and Area Development Committees (ADC). Model dissemination, if successful, could buttress linkages between DAESS and DNCC/ANCC/VNCC structures. However, to date SANE’s “TOT” approach to model dissemination has not worked as anticipated.38 (See recommendations 1 and 20 below). Recommendation: SANE’s renewed focus on DAESS “software” and systems development should encourage greater attention to DNCC, ANCC and VNCC linkages. Some of this will be 37 IHF is an intervention that DOA frontline workers promote hence not the core business of SANE. 38 SANE’s unsatisfactory experience with ASP/ANCC TOF training of VACs/VNCCs is documented in 2018/19 AWP. 23 possible as SANE provides direct VAC capacity building (i.e. SANE will more effectively strengthen VAC-VNCC and ASP-ANCC linkage with direct training.) Risk: SANE resource dissipation by implementing “non-core business function” activities (i.e. more extension training/interventions, FAW interventions, video clip dissemination without alignment with DAEP, DAESS objectives, etc.) spreads resources too thin, hurts capacity building effectiveness and sustainability. (See recommendations 1 and 8). Risk: SANE Removal of outcome level performance indicators focusing on extension service access and quality may divert SANE’s attention from DAESS raison d’etre. (See recommendation 11). Risk: Inadequate political, administrative and financial linkage between DAESS structures and district government offices and political decision makers may result in lack of funding for DAESS platforms or lack of overall political/administrative support for DAESS. (See recommendation 1). Risk: Poorly targeted focus on “those who benefit pay” may push DAESS platform operation costs onto smallholder food security farmers who make up 50% of platform membership. If farmers are forced to pay for low quality services and VAC/ASP/DSP participation they may refuse to participate, hurting DAESS sustainability. (See recommendation 6). Risk: Lack of SANE training quality monitoring. SANE assumes training at ASP level will trickle down to ASPs, VACs, clusters, farmer groups. ASP members point out that “we are farmers, not trainers”. SANE doesn’t follow up with training impact monitoring or quality control. (See recommendations 2, 4 and 12). Risk: Farmer DAESS platform participation and voice drive DAESS performance. SANE’s lack of VAC capacity building risks longer term results sustainability. Intended or not, SANE has been “projectized” at the VAC, ASP and DSP level. When SANE leaves, if farmers don’t feel VAC participation provides benefits above the costs, results will not be sustainable.(See recommendations 2, 3, 4 and 12). Risk: SANE works predominantly at the “top” of the DAESS structure (DAECC, DSP, ASP) while expecting upper echelon DAESS strengthening to “cascade down” to VACs, clusters, farmer group members and ordinary Malawians. There is no follow up quality control to this capacity building approach. SANE misses the importance of VAC capacity in policy dialogue participation, needs identification, research/extension linkages, and DAESS success. (See recommendations 8) Risk: SANE, NGOs, and donor projects continue to rely on DAES front line staff in project implementation, overtaxing AEDC/AEDO capacities and reducing ability to fulfill their “…role of being primarily responsible for strengthening farmer organizations”. (See recommendation 21 below). Recommendation: USAID should engage with other donors, DAES, Bunda and Natural Resources Campuses of LUANAR and others during NAEASS and ASFNS finalization and roll out to discuss how to increase NGO development and use of own extension staff. NAEP and associated strategies have codified pluralistic extension service provision. Now NGOs and donor projects are encouraged to develop their own front-line extension staff. The Government has recognized non-state actor extension services as bona fide agricultural 24 development participants. DAES front line staff should focus on, among other things, VAC, cluster and farmer group development and support. Risk: SANE work planning out of sync with activity monitoring and results achievement. Untimely SCAT assessment results do not inform SANE work planning (as shown by 3rd year results significantly exceeding 4th year AWP targets). This risks scarce SANE resources ineffectively deployed and opportunities to “deepen” DAESS sustainability lost. (See recommendation 21). Recommendation: USAID should conduct a Data Quality Assessment of SANE’s AMEP implementation. Risk: Inadequate national DAESS non-state champion strengthening support hurts ASFNS and NAEASS implementation and this SANE activity and benefit sustainability and DAESS effectiveness. While reduced SANE support for MaFAAS has enable MaFAAS to attract other donor support that support appears piece meal and not based on strategic planning. (See recommendation 5). Evaluation Question 9: To what extent did SANE integrate with other activities and those supported by USAID and other development partners to help increase chances for sustainability of impact? SANE has integrated or pursued integration with USAID’s Agricultural Diversification Activity, MIIST, Local Government Accountability Project (LGAP), New Alliance for Policy Acceleration and Support, Malawi Early Grade Reading Improvement Activity, Health Communication for Learning, Peace Corps, Protecting Ecosystems and Restoring Forests in Malawi, Ubale and Njira.39 These different integration initiatives are in different phases of development. NAPAS (that collaborated with SANE on DAESS related policy development) could contribute to impact sustainability (i.e. if/when the ASFNS and NAEASS are finalized and disseminated). SANE worked with LGAP to raise District Councils’ awareness of DAESS platforms’ role in providing quality and demand-driven agricultural extension services to support rural growth. This has led to better coordination between the DAESS platforms and Local Government structures, especially at area and village levels. If this improved coordination contributes to greater Local Government operational and policy support for DAESS it would contribute to SANE impact sustainability. SANE has integrated its DAESS strengthening activities with IFAD/SAPP and Afikepo/Kulima. As previously discussed, this integration increases the chances of SANE impact sustainability. EU and IFAD resources are used to support DAESS structures in 20 of Malawi’s 28 districts (including three districts where SANE operates and three districts where SAPP and Afikepo/Kulima both operate). 39 SANE Year 3 Annual Report 25 GENERAL RECOMMENDATIONS Recommendation: SANE “direct to VAC” capacity building (i.e. not through ASP member “training of trainers”) should establish effective and replicable VAC strengthening models. Capacity building should focus on participatory farmer needs identification; stakeholder and health facility/service mapping40; VAC ability to coordinate and hold accountable agriculture and nutrition service providers; youth and female member empowerment; leadership; action planning; proposal writing; using WhatsApp for DAESS communications; and resource mobilization (including group marketing or Village Savings and Loan (VSL) operations tied to DAESS platform participation). VNCC representatives (Care Group Promoters) should be involved in VAC training to build VAC￾VNCC linkages.41 VAC capacity building skills should be mobilized through non-state DAECC members (preferably), short term outsourcing42, academic institutions (LUANAR), MAIWD interns, CRS, or others. The harmonized materials on training methods (role play, encouraging participation, simulation, feedback, building self-esteem, etc.) should be simplified and distribute for VAC capacity building. VAC pilots would explore with DAES how best to train VACs when rolling out the NAEASS and ASFNS nationwide. Recommendation: The following definition of “demand”, from the November 2006 DAESS Implementation Guidelines, should be translated into Chichewa and distributed to all VACs and ASPs (so that it is readily available to small holder food security farmers): “Demand represents the need articulated for services (e.g. training, farmer organization development) by farmers and/or their organizations. The better the quality of articulated demand by farmers, the more responsive service systems and providers need to be, hence the use of terms such as demand-led (more appropriate than demand driven), client orientation and customer satisfaction. The more informed and emancipated farmers are in relation to their rights and farming problems, the more capable they are to demand specific responses or services they need. Among rural communities that have not benefitted from sound basic or civic education, there is a special need to foster and develop their abilities to voice and assert their demands as empowered individuals and or groups.” Recommendation: National non-state DAESS champion strengthening is needed as foreseen in SANE design. This champion might be MaFAAS, Civil Society Agriculture Network 40 Stakeholder mapping should include nutrition infrastructure including village clinics, Positive Deviance/Hearth sites, and communal food supplement cooking sites. by the District Councils and partners working in the district. 41 Lirangwe EPA FGD revealed that recent SANE ASP training included ANCC (1) and VNCC (1) members. 42 Extensive VAC capacity building will be a medium-term need after NAEASS roll out. University leavers or other youth (currently serving as MAIWD interns) may see VAC capacity building as a career building opportunity. They, of course, will require their own capacity building to refresh skills they learned at LUANAR. “If you go to the VACs they will express as if they have not learned because they didn’t see the SANE people. They usually think, ASP members, “they are used to me” and as an extension worker “today I talk about SANE, tomorrow I talk about SAP”. What they get is that an extension worker came and talked to us…but the quality of the content usually gets low because the way I can understand it is not like the way somebody else can understand it…and we weren’t given materials on how to train…just on roles.” -AEDO 26 (CISANET), or a national stakeholder, trade association or NGO forum. SANE provides limited MaFAAS support (ASP farmers to participate in national MaFAAS functions) and SANE support attracts other donor support for MaFAAS. During ASFNS and NAEASS roll out stakeholder discussions SANE (and USAID) should encourage strategic donor MaFAAS support and indicate how SANE will support MaFAAS during SANE’s remaining LOP. Recommendation: Until farmer access to quality extension services improves, SANE should encourage DAESS non-state participants to innovate in paying “incentives”. If accepted by farmers SANE could develop financial transfer mechanisms (such as VSL capitalization) to encourage farmer VAC, ASP and DSP participation. This will focus “those who benefit pay” on service providers. Ensure that all service providers pay to participate in DAESS platforms, with those funds used to support VAC, ASP, and DSP farmer participation and capacity building and DAECC operations. Recommendation: All capacity building training should include pre and post-training surveys that identify trainee needs, whether the needs were addressed, trainers’ competence, setting appropriate, and training objectives realistic. These surveys must be culturally appropriate, implemented verbally or in writing, etc. EXIT STRATEGY RECOMMENDATIONS Recommendation: Address MoAIWD interests that feel bypassed by SANE implementation. SANE’s system strengthening should be approached institutionally, with less reliance on individuals or champions. SANE should approach ADD and other staff and, referring to the MTE findings; ask them: 1) Is what the MTE found about SANE’s approach true? 2) If so, how can we address this misstep? and, 3) How can SANE build broad MoAIWD understanding and appreciation for what SANE is doing and trying to accomplish? ADD and MoAIWD staff may offer SANE valuable support with the exit strategy implementation. Recommendation: Over the remaining LOP SANE could eliminate all “non-DAESS software strengthening” interventions, cede responsibility for DAESS platform development in Balaka, Blantyre, Chikwawa, Nsanje and Mchinji to DAES and committed non-state entities (including UBALE); and employ their upper DAESS echelon model in kick starting DAECC and ASP functionality development in the 10 districts not currently enjoying IFAD-SAPP, Kulima/Afikepo or SANE investments. SANE’s upper echelon DAESS strengthening approach will leave these districts with less developed VAC functionality. This must be remedied with non-SANE resources VAC functionality will drive DAESS success and sustainability. Recommendation: SANE could support translation into Chichewa and other languages, and adequate publication and distribution of the NAEASS, the AFNS, and associated revised implementation guidelines (if necessary) to all districts, extension planning areas EPA), sections and communities. Condensing and focusing content on actionable steps relevant to DAECC, DSP, ASP, VAC, AEDC, and AEDO capacity in strategy implementation is also recommended. The DAESS has clearly defined platforms at district and area levels to foster coordination and collaboration of extension stakeholders. However, the system did not have a national platform for engagement of national-level stakeholders. In FY16/17, SANE, in collaboration with DAES, intends to build the capacity of MaFAAS to represent the interests of both service providers and farmers. SANE Annual Work Plan FY 2016-2017 27 SANE should not heavily brand the distributed documents to promote national and local DAESS ownership, and distribution should be well organized and closely tracked to promote application. District level IFAD-SAPP, Kulima/Afikepo and SANE investments encourage DADO’s, AEMOs and Food and Nutrition Officers (FNO) to have a positive DAESS experience. The DADOs AEMOs and FNOs then serve as DAESS champions when assigned to districts lacking external DAESS support. Recommendation: To buttress this DAESS championing, SANE could use its convening power (during strategy and implementation guideline finalization and dissemination discussions, for example) to discuss DADO and AEMO Job Descriptions (JD) and performance assessment (PA) indicators that would include DAESS platform strengthening as a core work/career advancement objective.43 Recommendation: SANE must show that DAESS system strengthening increases small holder food security farmer access to quality extension services to ensure non-ZOI adoption and scaling. SANE must: 1. Reinstate the two AMEP performance indicators that measure percentage of households with increased access to quality extension services; 2) Develop and document a VAC capacity building approach that empowers farmers to express real extension service needs and hold service providers accountable to satisfy those needs. Recommendation: Consider transferring SANE agricultural extension (AES) and nutrition extension specialists’ (NES) DAECC, DSP, and ASP strengthening responsibilities to an appropriate MAIWD, ADD, MaFAAS or other capacity. This can be done while standing ready to advise this capacity on SANE “lessons observed” and work on functionality strengthening challenges. The AES and NES should focus on piloting effective VAC functionality strengthening. Recommendation: As part of the final report, SANE should suggest that future USAID/Malawi awards include a “DAESS platform strengthening” budget line item and performance indicators requiring awardee participation on DAESS platforms. Recommendation: When designing and implementing ASP and VAC functionality training SANE should emphasize CoAg design content with particular focus on this approach: “….training programs will not be one-off events but a series of events with a cohort of participants, which lends itself to accountability and group encouragement. Through this approach we can identify and further coach and support those organizations or individuals that demonstrate passion and commitment to increasing access to extension services for small￾holder farmers”.44 SANE should continue to encourage, and expand, its inclusion of VNCC and ANCC representatives in ASP and VAC functionality training to cross fertilize and build systematic agriculture-nutrition linkages. 43 Mangochi DADO stated that under the Mangochi Local Authority Performance Assessment, introduced in 2017, DAESS strengthening was one of nine performance targets to which he would be held accountable. 44 SANE Cooperative Agreement 28 29 Annex I: Strengthening Agriculture and Nutrition Extension: Mid-Term Evaluation Scope of Work BACKGROUND The purpose of The Feed the Future Malawi Integrating and Strengthening Agriculture and Nutrition Extension Services (SANE) activity is to strengthen the capacity of the Government of Malawi’s (GoM’s) Department of Agricultural Extension Services (DAES) to mobilize and work with other service providers to deliver agricultural and nutrition extension and advisory services more effectively and in a coordinated manner. SANE’s activities are concentrated in 10 districts in central and southern Malawi: Mchinji, Lilongwe Rural, Dedza, Ntcheu, Mangochi, Balaka, Machinga, Blantyre Rural, Chikwawa, and Nsanje. PROGRAM COMPONENTS AND DESCRIPTION In 2000, in response to the Malawi Government’s Decentralization Policy, the Ministry of Agriculture, Irrigation and Water Development (MoAIWD) adopted a new extension policy, the National Agricultural Extension Policy (NAEP) which among other things, calls for Accountability, Pluralism, Demand-driven, and Coordination in extension services delivery. During this time period, a District Agricultural Extension Services System (DAESS) was put in place to provide a framework for the provision of extension services in line with the Decentralization Policy. The Decentralization Policy seeks to devolve administration and political authority to the district level resulting in the transferring of powers and functions of governance and development to elected Local Government Units. While action was taken on implementation of the new policy, many key elements of the NAEP remained largely unimplemented due to capacity constraints and other challenges. USAID/Malawi in collaboration with DAES commissioned two separate studies to assess the status of agricultural and nutrition extension services delivery to identify specific interventions that would help improve agricultural extension and nutrition education. Key findings and recommendations from the two studies informed the design of an extension strengthening activity, SANE. An award was made in 2015 to the University of Illinois to implement the activity. SANE has three expected results. These three results, along with their sub-results and interventions, are discussed below. Result 1: Policy environment improved Sub-Result 1.1: Enabling policy adopted: The following interventions were perceived key to achieving this Sub-Result: • Review of the current extension policy. • Providing support to the Malawi Forum for Agriculture Advisory Services (MaFAAS) to assume the role of a national level stakeholder panel, and to strengthen its advocacy capacity. • Increasing participation of farmers in policy dialogue platforms to push for implementation 30 of favorable policies. Sub-Result 1.2: DAES nutrition strategy developed and implemented: To help DAES intensify its food and nutrition programs SANE was to assist in the development of a strategy that integrates agricultural and nutrition extension. In this strategy, DAES would play a central role in facilitating coordination of extension service delivery at district level across the array of service providers. Sub-Result 1.3: Implementation of the pluralistic and demand-driven extension policy improved: The following interventions were to be included to achieve this Sub-Result: • Implementing the DAESS as a dialogue platform at district level for provision of nutrition and agricultural extension services. • Developing strategies to establish and support stakeholder committees to operationalize the DAESS. • Enhancing awareness of the DAESS implementation guidelines among all stakeholders to ensure that the DAESS is implemented in accordance with the guidelines. • Increased participation of farmers in stakeholder panels to ensure that they are able to effectively demand services and strengthening farmers’ skills to articulate their needs and demand accountability from service providers. Result 2: Strengthened coordination and capacity of extension service providers Sub-Result 2.1: Mechanisms for coordination and harmonization of service delivery strengthened: The following interventions were expected to be included to achieve this Sub-Result: • Strengthening capacity of DAES to coordinate Malawi’s pluralistic, decentralized extension system and building the elements of the DAESS, particularly the stakeholder panels in which farmers articulate their needs and demand accountability from service providers. • Facilitating coordination among service providers at the district level. • Developing modalities for ensuring harmonization of extension messages and training approaches. Sub-Result 2.2: Capacity of extension service providers increased: The following interventions should be included to achieve this Sub-Result. The Recipient may propose additional Sub-Results to USAID/Malawi with justification of their relevance: • Improving capacity of extension workers in delivering agriculture and nutrition extension messages. • Strengthening the capacity of DAES to coordinate public and private sector agriculture and nutrition extension service delivery at district and community levels. • Increasing implementation of proven, effective, and efficient models for agriculture and nutrition extension outreach that increase extension-farmer contacts. • Strengthening the capacity of agricultural and nutrition training institutions to train staff to deliver integrated extension and nutrition messages. • Making available appropriate educational technologies and curriculum to improve students’ 31 learning and knowledge of agriculture and nutrition. • Improving extension workers’ access to relevant climate, pest, and disease information and ability to disseminate to smallholder farmers. • Improving district and community level capacity to do accurate and timely climate, pest, and disease forecasting. Result 3: Strengthened institutional linkages between research and extension Sub-Result 3.1: Extension-research linkages improved: The following interventions were to be included to achieve this Sub-Result: • Strengthening modalities for institutionalizing linkages between extension service providers and research institutions to ensure that extension agents transfer appropriate messages to smallholder farmers. • Strengthening capacity of research organizations to train extension workers in the application of new technologies and behaviors. Sub-Result 3.2: Capacity of research organizations to support technology transfer and behavior change improved: The following interventions were considered key to achieving this Sub-Result: • Increasing farmers’ access to knowledge and skills for adapting and using Indigenous Knowledge Systems (IKS). • Strengthening participatory research and extension approaches so that both research and outreach systems in research organizations are in concert with technology transfer and behavior change methodologies pursued by extension service providers. EVALUATION PURPOSE This mid-term evaluation (MTE) is viewed as a tool for learning and accountability and provides an excellent opportunity to examine the progress and relevance of activities, the indications of potential impact, and to discuss and recommend mid-course corrections, if necessary. The MTE will evaluate the relevancy, effectiveness, efficiency, impact, and sustainability of the activity against the objectives. In addition to the evaluation of programmatic inputs, outputs, and outcomes, SANE intends to utilize this MTE as an opportunity to revise its M&E Plan. The evaluator(s) will be expected to conduct a thorough review and revision of the evaluation plan, its methodology, and the activity’s current M&E activities prior to conducting the fieldwork for the MTE. This will take place with a review of the M&E plan and an interactive meeting to discuss the theory of change and appropriateness of indicators, roles and responsibilities. EVALUATION QUESTIONS Taking into account the implementation status of the program and the resource disbursements made to date, the following questions will be evaluated: 1) To what extent do we see evidence that the DAES is taking learnings from SANE and applying them in non-ZOI areas? Interventions on course to result in improved access to 32 extension services for smallholder farmers in the zone of influence and beyond, particularly with respect to the key result areas: a) Improved policy environment for delivery of extension services; b) Improved institutional and field level capacities for service delivery; and c) improved research/extension linkages. What would they recommend that SANE do to ensure greater adoption and scaling up in non-ZOI areas of the practices that they are promoting? 2) What challenges has SANE faced in the course of implementation and how have they been addressed? What recommendations can be made on alternative ways of addressing these challenges? 3) What steps has SANE taken to address sustainability of its outcomes/outputs and what are the key risk factors for longer-term sustainability of the results? What additional steps need to be taken in order to improve the chances for sustainability of the activities and benefits derived from the activities? To what extent did SANE integrate with other activities and those supported by USAID and other development partners to help increase chances of sustainability of impact? EVALUATION METHODOLOGY The consultant will be expected to design the MTE methodology in consultation with USAID and SANE program management. However, it must employ a combination of qualitative and quantitative methods, it must be participatory in nature, and it must include a variety of data collection methods as appropriate, including structured questionnaires administered to representative beneficiary samples, focus group discussions with stakeholders, interviews with beneficiaries/stakeholders, direct observations of activities and consultations with SANE and DAES with USAID Mission. The MTE methodology must be approved by MELS and USAID prior to commencing work. USAID will provide the valuation team with: ● All pertinent documents for the consultants to review, including the Program Description, PMP, work plans, the baseline assessment, technical reports, and other relevant reports. ● Project staff and beneficiaries will be available for interviews and consultations. Mixed methods should be utilized, including both qualitative and quantitative methods. Direct and indirect impacts should be assessed as well as intended and unintended impacts. The evaluator(s) will use external as well as internal records, reports, and data for conducting the evaluation. The evaluators should utilize several different, yet complementary and inter-related forms of gathering information/data such as those described below. The contractor is expected to utilize its expert judgment and evaluation best practices in selecting which methodological components to include in the evaluation design. 33 Document Review. Evaluation team members will review documents throughout the evaluation process including program reports, relevant studies to ensure that comprehensive and grounded best practices will be identified. Key Informant Interviews. The team will conduct one-on-one interviews with a variety of stakeholders including the leads for each of the participating research centers, most relevant GOM ministries and agencies, bi- and multilateral development partners supporting development and seed production and dissemination activities in Malawi. This will be done administering a semi-structured or structured questionnaire. Self-assessment. The IPs will respond to a self-assessment either through a questionnaire or standard interview checklist put together by the evaluation team and approved by USAID before use. Expert Opinion Survey. Utilizing expert opinion is a technique is used increasingly in evaluations. The evaluation team, with approval of USAID, can apply this method as well. Focus Group Discussions (FGDs). FGD (small group of 6 to 10 people) will be used to lead open discussion through a skilled moderator to gather semi-structured qualitative data. The preselected participants will discuss issues and concerns based on a list of key themes drawn up by the moderator. No more than 10 questions will be addressed by a group. These sessions will encourage free flowing discussion about the activity. Mini Survey. This type of survey is small (30-40 participants) and can be performed rapidly in the field without analytical software such as SPSS or a large questionnaire. The sample size is not statistically significant; however, this type of analysis can be used to triangulate with other methods. Because of the small sample size this type of survey can be implemented quickly when time and resources are constrained. All the methodological strengths and weaknesses should be explicitly described in the evaluation report. Prior to the start of the evaluation, the evaluation team shall meet with EG office to refine the evaluation methodology, and address any other concerns the EG office may have. Limitations to Data Collection. Due to the compressed timeline of the evaluation, the evaluation team will not have adequate time to conduct large surveys that are statistically significant. Wherever possible the team will triangulate findings using multiple sources. EXISTING SOURCES OF INFORMATION The evaluation team should consult a broad range of background documents apart from project documents provided by USAID /Malawi. These should include, but are not limited to, documents on the agricultural context in Malawi, the seed commercialization process, requirements for bringing seed to market and improved seed varieties both in and outside of Malawi. The evaluation team also should review relevant GOM national strategies and policies, as well as relevant projects and strategies of other aid agencies active in Malawi. USAID, MELS, and the SANE project will provide the assessment team with a package of briefing materials, including: 34 ● The agreement for SANE activity; ● M&E plan for SANE; ● Project quarterly and annual reports, work plans and management reviews developed as part of routine monitoring; ● USAID/Malawi Country Development Cooperation Strategy 2014-19 (Public version); ● USAID Malawi DO2 PMP; ● Training & Beneficiary database; ● USAID Evaluation Policy; ● Checklist for Assessing USAID Evaluation Policy DELIVERABLES The consultant is responsible for preparing and submitting the following deliverables: ● An inception report with proposed methodology and work plan, based on reviewing key documents and consultations with key staff and stakeholders, including USAID. The report must include roles & responsibilities, methodology, and time-specific work plan. ● Draft Evaluation Report and Evaluation Summary: The consultant is responsible for writing a draft report that takes into consideration comments received at the evaluation wrap-up meeting with USAID/Malawi, to produce a coherent Draft Evaluation Report and Evaluation Summary. The Draft Report and Summary is to be submitted electronically to the Feed the Future Team Leader and MELS Contract Officer’s Representative according to the agreed schedule. ● Final Evaluation Report and Evaluation Summary: Based on comments received on the Draft Evaluation Report, the consultant will finalize the report and summary and submit the Final Evaluation Report (and Summary) to the Feed the Future Team Leader within 7 days of the evaluation debriefing, or by the agreed date. The findings and conclusions of the report should be evidence-based and well-supported with evidence. The Evaluation Team’s contractual obligations are complete once USAID/Malawi has reviewed and approved the Final Evaluation Report for quality and completeness as per the SOW. Evaluation plan. Specifically, the evaluation will comprise the following stages: a. Review of relevant documentation: A list of key reference documents will be provided. b. Finalization of evaluation work plan: Prior to arriving in country the Evaluation Team will review the draft evaluation work plan and make any adjustments they see fit, taking into account practical and logistical considerations. c. In-country briefing: The Evaluation Team will be briefed on the first day of the evaluation mission by USAID/Malawi. All relevant documentation not already sent in advance to the Evaluation Team will be provided. d. The evaluation: The Evaluation Team will design and conduct the evaluation employing 35 best practice evaluation planning and methodologies. As far as possible the Evaluation Team will triangulate findings using multiple sources/methods. The evaluation should include all key stakeholders, and a sample of districts and communities in which SANE is operating, aiming for the most representative sample feasible. Whilst this mid-term evaluation does not necessarily focus on achievement of outcomes or impact, indications of such should be sought using qualitative methods, including consultations with the intended clients of the Activity. As far as possible, the Evaluation Team should discuss findings with beneficiaries and stakeholders at each stage of the evaluation and obtain their feedback. During the evaluation, the consultant will be required to provide regular updates to USAID through the MELS COR. e. In-country evaluation out-brief: There will be a joint wrap-up meeting of the Mid-Term Evaluation. At the wrap-up meeting the Evaluation Team will present their key findings and recommendations for discussion. f. Draft evaluation report and Evaluation Summary: The consultant will submit a Draft Evaluation Report and Evaluation Summary not exceeding 30 pages (including annexes) to USAID, which will be circulated to key stakeholders for comment. g. The Final Evaluation Report and Evaluation Summary: will be submitted electronically to USAID. EVALUATION TEAM COMPOSITION At the minimum the evaluation team should comprise: Evaluation Team Leader, Agricultural Extension Specialist, Nutrition Education Specialist, and Extension Governance Specialist. Evaluation Team Leader (International). This expert will serve as Team Leader for the SANE evaluation. The Team Leader will provide overall leadership for the team, and s/he will finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations into a high-quality document. S/he will lead the preparation and presentation of the key evaluation findings and recommendations to the USAID / Malawi team and other major partners. The TL should have the following attributes/qualifications and experience: • S/he should have a postgraduate degree in an appropriate field, with area of expertise in evaluation, agriculture extension, agricultural value chains, nutrition and agricultural economics, with at least 10 years of international experience in the agriculture sector or with experience conducting USAID evaluations, ideally for agricultural extension projects. Relevant experience in Malawi or Eastern/Southern Africa preferred. • S/he should have extensive experience in conducting quantitative and qualitative assessments. • The Team Leader must be familiar with USAID regulations and systems including performance monitoring, gender policies and guidance, project management, budgeting and 36 financial analysis, and reporting. • Experience in international donor development program management and overseeing multiple program areas simultaneously is preferred. • Should be experienced in preparing documents that are objective, evidence-based, and well organized. • Excellent oral and written skills in English are required. Senior Agricultural Extension Expert (International). The Senior Agricultural Extension Expert will be responsible for assessing the effectiveness of project implementation according to the M&E plan, while addressing the evaluation questions above. S/he will provide technical assistance and analytical ideas with respect to the agricultural extension sector in Malawi. S/he will participate in team meetings, and in the absence of the team leader, lead key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The Senior Agricultural Extension Expert will have: • A postgraduate degree in agriculture, agriculture economics or a related field. • A minimum of 8 years of international field experience in agricultural development with an understanding of agricultural extension and nutrition in southern Africa. • M&E experience with at least one similar USAID project, or that of another international donor, and have familiarity with USAID regulations and systems. Nutrition Education Specialist (Local). The local Nutrition Specialist will have experience in the Malawi agricultural sector preferably in the nutrition sector and understand the needs of the Malawi smallholder farmer. S/he will participate in team meetings, key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he also will participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The local Nutritional Specialist should: • Have an advanced degree in nutrition or related field. • Be a Malawi national with at least 5 years of field experience in evaluation, preferably with some sectoral experience in nutrition, agricultural and improved seed. Extension Governance Specialist (Local). The local Extension Governance Specialist will participate in team meetings, key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he also will participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The local Extension Governance Specialist should: • Have a degree in monitoring and evaluation, international development, governance, extension systems, economics, agriculture, or a related field. 37 • Be a Malawi national with at least 2 years of field experience in in governance, extension systems, preferably with some experience in evaluation. Logistics Specialist (Local). The local Logistics Specialist will be responsible for providing program logistics support, arranging appointments and maintaining the schedule, providing interpretation and translation, and assisting with the preparation of project reports, as needed. Conflict of Interest. All evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project being evaluated. USAID will provide the conflict of interest forms. See Annex 1 for the Template. LEVEL OF EFFORT Below is an estimate of the evaluation level of effort (LOE). Level of Efforts of Team Members by Task Deliverables Task/Deliverable Duration / LOE in Days Team Leader/ Sr. Ag. Ext. Expert Nutrition Education Expert (Local) Ext. Governan ce Specialist (Local) Review background documents and home-based preparation work 5 5 2 2 Review SANE Results framework (Theory of Change) and M&E Plan including indicators 2 0 0 0 Travel to Malawi 2 2 0 0 Team planning meeting and meeting with USAID 2 2 2 2 Development of Evaluation Work Plan (concurrent with document review and initial meetings) 2 2 2 2 Participatory Review of Results Framework including SANE and USAID including roles and responsibilities 3 0 0 0 Development of data collection instruments 2 2 2 2 Information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits 20 20 20 20 Discussion, analysis, and preliminary draft evaluation report in country including discussion with USAID 3 3 3 3 Debrief meetings with USAID (preliminary draft report due to USAID) 1 1 1 1 Team Leader meets with Technical Specialists and USAID to synthesize findings/discussion 1 1 1 1 Depart Malawi/Travel to U.S. 2 2 0 0 Finalization of draft and internal review (IBTCI) remote 4 4 2 2 USAID provides comments on draft report 0 0 0 0 Team revises draft report and submits final to USAID (out of country) 10 4 4 4 USAID completes final review 0 0 0 0 38 Editing and formatting of report 0 0 0 0 Total Estimated LOE 59 48 39 39 SCHEDULING AND LOGISTICS Funding and Logistical Support. USAID/Malawi’s MELS project will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate local staff. It will take care of arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. The MELS Logistics Specialist will arrange field visits, local travel, hotel, and appointments with stakeholders. Scheduling (see above table). Include a tentative evaluation schedule showing all the timings for the key tasks to be undertaken as part of this evaluation. Work is to be carried out over a period of approximately 10 weeks. At this point in time, we anticipate that the evaluation would begin on/about August 1, 2018, with work in Malawi commencing on/about August 15th, field work completed on/about September 10 and final report and close out before the end of September. A six-day work week is authorized for the evaluation team while in Malawi, however, no overtime or premium pay is authorized. The evaluation team will submit a work plan with timeline as part of the evaluation methodology proposal and develop a Gantt chart displaying the time periods during which activities occur. Team mobilization will include: travel approval; airline tickets; visa; lodging; work facility and vehicle transport arrangements; dates for meetings with USAID/Malawi EG staff and key contacts; in-country travel agenda; and accommodations. 39 Annex II: KIIs and FGDs Name Position Dr. Clodina Chowa COP/SANE Silvia Mapanje Assistant Chief Agricultural Extension Officer (CAEO) (Food and Nutrition) Hastings Yotamu DADO Lilongwe West Sheila Mahata Kang'ombe Acting Program Manager/Lilongwe ADD Yamikani Fiwa FNO/Lilongwe West Paul Fatch MaFAAS Secretary General Boaz Mandula Agricultural Extension Specialist, SANE Stacia Nordin Nutrition Extension Specialist, SANE Grace Mzumara MEAL Specialist, SANE Shelix Munthali Agriculture Diversification Project Caitlin Shaw Farm Services Unit Manager, Farmers World Dr. Jerome Chim’gonda Nkhoma Director/DAES Agnes Mangwela Dean of Human Sciences, LUANAR, SANE PSC Enford Kanyimbo DADO/Lilongwe East V Anderson Total Land Care Manual Milz Interaid Blessings Susuwele Lilongwe ADD Maureen Maguza Tembo D/DIR/FN/DAES Joel Suzi DCOP, HC4L Abel Mwenechanya FISD, Lilongwe/DAECC Chairperson Jim Ashman Program Manager, CRS Diamond Kambewa Ass. Prof. Ag. Ext. LUANAR, SANE PSC Noel Limbani SANE Desk Officer, DAES Dr. Naomi Kamanga MISST COP Kumbukani Munthali Farm Services Unit, Farmers World 40 Alfred Tsitsi AEMO/DAO/Balaka Fortunes Kapingiri Total Land Care, DAECC/Blantyre Vice-Chair Friday Mulenge Farm Services Unit, Agora Erick Haraman Program Manager, Blantyre ADD Ms Anily Msukwa Deputy Program Manager Blantyre ADD Ms Kumwenda Acting CAEO and AEMO Blantyre ADD Mphatso Mbulukwa AEMO, DAO, Blantyre Lester Kacheche FNO, DAO Blantyre Precious Chautsi DADO, Mchinji Raymond Chimsale CADECOM, Chikwawa DAECC Chairperson Mictor chawola CRS, UBALE, Blantyre Mary Kandikole Mpinda CRS, Blantyre Jackson Kachidede CARE Nsanje, DAECC Chairperson Isaac Chipeta ADD DPM. Machinga Adrick Benati ADD PM Machinga Mr. Manure ADD FNO Machinga Mr Aggrey Mfune SEEED Malawi, DAECC/Mangochi, Chair Adam Kabango AAEMO/DAO Mangochi Francis Liati Red Cross, SUN, Mulanje Winston Mtambwe AEDC/Chigodi EPA, Lilongwe East M. Kimwenda DADO/DAO Mangochi James Lichapa AEMO Mulanje Balaka DAECC Mchinji DAECC Mangochi DAECC DAES Food and Nutrition as well as Extension Methodologies Officers Blantyre DAECC/DNCC Msako VAC Mazengera EPA, Lilongwe East Katulusa Model Village Chitekwele EPA, Lilongwe East Msitu ASP Msitu EPA, Mchinji Chibwana VAC Amidu Traditional Authority (TA), Balaka Kalembo ASP Ulongwe EPA, Balaka Lunzu DSP Blantyre Rural Chileka ASP/VAC Blantyre Rural Mkata VAC, Kumtaja EPA, Blantyre Rural Lirangwe ASP/ANCC Lirangwe EPA, Blantyre Rural Mbwadzulu ASP Mbwadzulu EPA, Mangochi 41 Annex III: Data Collection Instruments Please describe your VAC or ASP’s operations: ASP/VAC Name: Location: When did the ASP/VAC begin meeting? Why does it meet that often? How many members do you have? Female Male Participating Today How were the members selected? By Vote Are there NGOs/agribus on the ASP/VAC? Is there a Chairperson/Secretary? How is the Chairperson/Secretary selected? Why are the meetings held? What takes place at the meetings? Is there a written meeting agenda? Why or why not? How is the agenda established? Is it distributed before the meeting? To who? How long do meetings usually last? Are meetings taken during the meeting? Why are meeting minutes taken? Are the minutes distributed to members after the meeting? Why or why not? Give three examples of issues discussed or decisions made at meetings? 1) 2) 3) Have these issues been presented to the ASP or DAECC? Issue 1) Why or why not? 42 Issue 2) Issue 3) Has the DAECC responded to your issues? Why or why not? Have the ASP/VAC meetings helped you farm better? How? Have the ASP/VAC meetings helped improve the nutrition of your family? How? What training have you received on ASP/VAC roles and responsibilities? What are ASP/VAC responsibilities? How many of you are Lead Farmers? How many of you are VSL members? Have you ever spoken about agricultural research at ASP/VAC meetings? What did you discuss? Have you ever spoken about allowances at ASP/VAC meetings? What did you discuss? SANE MTE: Questions for DAES and other State/Non-State Partners (DARS, LUANAR, etc) 1. How familiar are you with USAID’s Strengthening Agricultural and Nutrition Extension (SANE) activity? 2. How were you introduced to SANE? 3. Does your activity implement interventions in the same districts as SANE? 4. What has been your/your institution’s interaction or role with SANE? 5. Why did you/your institution decide to interact with SANE? 6. Are you familiar with SANE’s “3 Cs approach” (co-location, collaboration, coordination)? 7. Please explain which of the 3 Cs is most important to you/your institution and how it came about? 8. How does (co-location, collaboration, coordination) contribute to you/your institutions success? (i.e. What benefits do you or your institution receive from 3C with SANE?) 43 9. What human, financial or other resources did you/your organization have to mobilize to (co-locate, collaborate, coordinate) with SANE? (i.e. What were the costs to you/your institution for participating in 3C with SANE?) 12. Has the (co-location, collaboration, coordination) increased your contributions or work with DAESS platforms? 13. How has it increased your work with DAESS platforms and what benefits has your institution received from working with DAESS platforms? 14. Have you begun collaborating with other institutions through the DAESS platforms? Which institutions? 15. Is working with DAESS platforms important in delivering improved agricultural and nutrition services to rural families? Why or why not? 16. What challenges did you/your institution face in (co-locating, collaborating, coordinating) with SANE? 17. Is the (co-location, collaboration, coordination) sustainable (or does it contribute to benefits that are potentially sustainable)? Why or why not? 18. How could the co-location, collaboration, and coordination be improved? SANE MTE: Questions for Senior SANE Staff (IUCI, CRS, MSU) 1. How long have you been working with SANE? 2. In general, how do you think SANE is progressing? 3. To give a background to this discussion, please identify SANE’s three most important achievements to date. 4. What has SANE done to achieve an improved policy environment for extension service delivery? (EQ2a) 5. What has SANE done to improve institutional and field level capacities for extension service delivery? (EQ2b) 6. What has SANE done to promote improved research/extension linkages? (EQ2c) 6. Of these three objectives, which is the most important? Why? 7. Of these three objectives, which will be the most likely to have a sustainable impact on rural family farm productivity and improved nutrition? 8. What has been SANE’s three most important interventions to improve DAES functionality at the DAECC, DSP and ASP level? (EQ2b) 9. Why are these interventions important? How were they chosen as best ways to improve functionality? 10. What is a key variable that affected the intervention success or failure? 11. What are the 3 most important challenges SANE has faced? 44 12. How were these challenges addressed or overcome? (EQ5) 13. What might SANE have done differently in addressing these challenges? 14. How would you define sustainability for SANE? Was sustainability an important part of SANE design? (EQ6) 15. How much of a focus has sustainability been for SANE during the first half of its implementation? 16. What steps has SANE taken to address sustainability? 17. In project documents it is stated that “Collaboration with DAES assisted in devising workable and cost-effective approaches that have potential to be carried on by DAES beyond the life of the SANE project”. What are those approaches? 18. What are the key risks to long-term results sustainability? (These would include risks SANE can and cannot address.) (EQ7) 19. How is SANE addressing risks to long-term sustainability? (EQ8) 20. Which risks are outside of SANE’s control? 21. Is activity integration with USAID and other donor projects an important SANE strategy to ensure sustainability? Why or why not? (EQ9) 22. To what extent have you seen evidence that the DAES is taking learnings from SANE and applying them in non-ZOI areas? Which non-ZOI areas? (EQ1) 23. What non-ZOI conditions favored application of those SANE learnings? 24. What contextual circumstances are most important for SANE’s delivery of NAES improvements that affect rural family agricultural productivity and nutrition? SANE MTE: Questions for SANE Integrators (MISST, Ubale, NAPAS, LGAP, etc.) 1. How familiar are you with USAID’s Strengthening Agricultural and Nutrition Extension (SANE) activity? 2. Does your activity implement interventions in the same districts as SANE? 2. How were you introduced to SANE? 3. What has been your/your institution’s interaction or role with SANE? 4. Why did you/your institution decide to interact with SANE? 5. Are you familiar with SANE’s “3 Cs approach” (co-location, collaboration, coordination)? 6. Please explain which of the 3 Cs is most important to you/your institution and how it came about? 7. How does (co-location, collaboration, coordination) contribute to you/your institutions success? (i.e. What benefits do you or your institution receive from 3C with SANE?) 8. What human, financial or other resources did you/your organization have to mobilize to (co-locate, collaborate, coordinate) with SANE? (i.e. What were the costs to you/your institution for participating in 3C with SANE?) 45 9. Are there other institutions involved with you/your institutions (co-location, collaboration, coordination) with SANE? 10. How could the co-location, collaboration, coordination) be improved? 11. Is the (co-location, collaboration, coordination) sustainable? Why or why not? 12. Has the (co-location, collaboration, coordination) increased your contributions or work with DAESS platforms? 13. How has it increased your work with DAESS platforms and what benefits has your institution received from working with DAESS platforms? 14. Is (co-location, collaboration, coordination) important in delivering improved agricultural and nutrition services to rural families? Why or why not? 15. Is working with DAESS platforms important in delivering improved agricultural and nutrition services to rural families? Why or why not? Guiding Questions for KIIs on “SANE learnings transferred to non-ZOI areas” 1. Have SANE “lessons learned” or DAESS strengthening processes or tools been introduced in non￾Feed the Future Zone of Influence areas? 2. Please describe the areas and, if possible, the individual DAESS strengthening process (standard operating procedures, district strengthening action plans, research/extension linkages, DSP meeting notes, etc.)? 3. How were these SANE “lessons learned” transferred to these non-ZOI areas? 4. For each area, what were the conditions that led to the lessons learned transfer? 5. For each area, has the lesson learned transfer been beneficial for DAESS operations? 6. How has it been beneficial? 7. Were there any area specific conditions (staff, leadership, location, partners) that led to the lesson learned transfer? 8. What additional costs has the DAESS platform had to cover to implement the SANE lesson learned? 9. How have these costs been covered? 10. Do you expect that additional SANE lessons learned will be transferred to this area? Why or why not? 11. Do you expect that SANE lessons learned will be transferred to other areas? Why or why not? 46 ANNEX IV: Statement of Differences Statement of Difference for the Midterm Evaluation Report of the Strengthening Agricultural and Nutrition Extension (SANE) Activity Evaluator: IBTCI The University of Illinois at Urbana-Champaign (UIUC) recognizes the significant work accomplished in preparing the Midterm Evaluation Report of the Strengthening Agricultural and Nutrition Extension Activity (SANE) Activity. UIUC and SANE appreciate the revisions made to the Report after back-and-forth feedback with the Evaluation team and the flexibility of the evaluators to date to incorporate certain feedback into the Report itself. However, concerns still remain about: (1) how the chosen methodology and Scope of Work constraints affected conclusions drawn in the report, and (2) the inclusion of statements that are inaccurate and potentially misleading. These issues lead to a presentation of SANE that requires amendment. Comments on Methodology and Scope of Work Limitations The SANE team questions whether the methodology used was sufficient and appropriate to arrive at some of the findings presented in the report. Findings are presented as being determined from 40 Key Informant Interviews and 15 Focus Group Discussions, plus document review. However, it does not appear the MTE team adequately consulted existing project documents – particularly those that contain key M&E data – in presenting their findings. This 47 led to many findings not matching or outright contradicting evidence generated from other data sets. For example, the Y3 Annual Report and Rapid Assessment report provide data that directly show: (1) learnings from the project in non-SANE districts45, determined from SCAT interviews with 150+ DAESS platforms SCATs in these districts, and (2) examples of sustainability at the platform level from the Rapid Assessment46, which the MTE repeatedly questioned. Other documents provide similar contradictions. Additional cross-checking of the MTE data against project data before drawing conclusions would have addressed these issues. In addition, the MTE team appeared to generalize its findings on the project primarily from the KIIs and FGDs. This leads to additional concerns. Qualitative data is not meant to be generalized beyond the sample47 . Other validity and reliability approaches – member checking, auditing/debriefing for transferability48 – were not mentioned in the report nor the lack thereof cited as potential limitations. Despite these qualitative research conventions, the KIIs and FGDs were generalized to the entire FTF region based on limited interactions with stakeholders. Conclusions about all non-FTF districts were also drawn from a single district (Mulanje). This – and lack of clarity on how respondents were selected, qualitative data analyzed, etc. – represent methodological concerns not sufficiently cited as limitations. Findings presented could also have been better supported with data to address this concern, without which there remain questions about the findings presented and their validity/reliability. The text provided few quantified (ie. X respondents said…) or attributed statements, direct quotes to back up assertions, or data from SANE reports and other documents. Presenting the actual data alongside findings would have alleviated concerns about researcher interpretation and/or bias. It is acknowledged that the MTE team faced time constraints and shifting schedules that made completing its full Scope of Work difficult. This likely contributed to fewer-than-desired interviews and less time for document review. However, given this constraint, more time should have been dedicated to consultation with the SANE Team. SANE did not have adequate time with the MTE team during their evaluation, despite at least three planned meetings: (1) briefing/inception, (2) clarification meetings during the process, and (3) debrief for questions and answers. The debriefing in particular could have been an opportunity for clarification and triangulation of data, yet the SANE team was debriefed without discussion and email clarifications were not included in the draft. 45 SANE FY2018-19 Annual Work Plan, p. 12-15, p. 32-49 (available on the Development Experiences Clearinghouse) 46 SANE FY2017-18 Annual Report, p. 17 47 Flick, U. (2009). An introduction to qualitative research. London, UK: Sage. 48 Creswell, J. W. (2013). Qualitative inquiry and research design (3rd ed.). Thousand Oaks, CA: Sage. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. 48 Clarifications to Specific Content As a result, there are areas of the Report that remain misrepresented and have the potential to mislead the reader in absence of clarifications. This Statement of Difference intends to clarify contextual matters overlooked in the process of conducting the evaluation exercise and due to the lack of sufficient debriefing with the SANE team and USAID/Malawi regarding preliminary findings. A major basis of the misrepresented areas originates from inadequate understanding of SANE’s origins and mandate to support the Malawi’s National Agricultural Extension Policy (NAEP) (2000) and the resulting District Agricultural Extension Services System (DAESS) Implementation Guidelines (2006). After failing to function effectively for almost 10 years, USAID/Malawi awarded UIUC the SANE Activity to help realize the potential of the system to mobilize stakeholders and strengthen platforms to better provide agricultural and nutrition extension services in a harmonized and coordinated manner. SANE’s mandate is therefore to strengthen implementation of the NAEP and DAESS, and to provide proof of concept of how a functional decentralized extension system can support improved access and quality of extension services. Neither the pillars of the I improvement of farmer voice, coordination of service providers, accountability in service provision, and access to and quality of services. This makes SANE a somewhat unique project that was difficult for the Evaluation team to fully understand during the short period of its Scope of Work. On this understanding, SANE wishes to make the following clarifications to misrepresentations presented in the Report: 49 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification Page 2 paragraph 4 Page 11 paragraph 1 E ac This generalization shows inadequate understanding of SANE’s activities and general approach. SANE supports platform agricultural strengthening so these platforms can better demand needed services, as outlined in the National Agricultural Extension Policy which established DAESS. Better organized/articulated demands and improved coordination within stronger platforms then can lead to better services, as outlined in SANE’s Theory of Change below. Failure to include this Theory of Change in the report highlights this misunderstanding. Through supporting the delivery of services – either by brokering linkages with service providers to meet needs or piloting specific activities requested but not yet available – SANE also works to create a feedback 50 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification loop that reinforces the potential of and generates buy-in to the DAESS. Doing so with beneficiaries from VACs, ASPs, and DSPs is crucial to these strategic goals. However, the report argues that traditional extension activities do not contribute to system strengthening, while evidence from SANE’s implementation show that systems (or platform) strengthening and service provision do play complementary roles. To say traditional extension activities in DAESS are not sustainable suggests poor understanding of the Malawian agricultural extension system and its implementation modalities. Page 3- recommendations SANE should cease activities in Blantyre and Balaka given International Fund for Agricultural Development/ Sustainable Agricultural Production Project (IFAD/SAPP) presence and initiate activities, from the ground up (i.e. beginning with VAC strengthening) in two or three of the remaining underserved districts This text is highlighted in red in the report. There is a difference between activities of IFAD and SANE. IFAD is using the platforms to provide agricultural extension services while SANE is strengthening the DAESS platforms to function effectively and improve access to quality extension services (for IFAD and others to use). While the DAESS platforms are expected to have a coordination role under the National Agricultural Extension Policy, many stakeholders bypass the platforms because the system is not capacitated to carry out its intended functions. In its capacity development efforts, SANE has developed 51 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification informational materials and trained/coached stakeholder panels on implementation of such materials to improve platform functionality, thereby improving coordination with service providers and access/quality of services farmers receive. This is not within the scope of IFAD/SAPP. Page 3, paragraph 2 Page 13 Evaluation Question 3, paragraph 2 SANE had a role in delayed strategy finalization. In response to 2017 funding reductions, SANE pulled back on its NAEASS financial support. SANE cut NAEASS funding at a critical time that could hurt sustainability if the NAEASS is not finalized and effectively disseminated in 2019. The Evaluation team seems to misunderstand the context of the decision. SANE had originally planned to provide resources for the review of the Agriculture Sector Food and Nutrition Strategy (ASFNS) and National Agricultural Extension and Advisory Services Strategy (NAEASS). Support to consultants writing the NAEASS was provided by the Multi-Donor Trust Fund (to which USAID contributes) through ASWAp. SANE’s support was limited to technical assistance through participating in policy advisory meetings and supporting dissemination of the finalized NAEASS to stakeholders and within the districts. To date, the ASFNS and NAEASS have not yet been approved by relevant Government Authorities for dissemination. SANE has no influence on policy processes beyond participation in technical meetings but will support dissemination as planned when the Strategies are ready. Page 11, SANE resource dissipation by SANE was designed to support the strengthening of a 52 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification paragraph 1 implementing “non-core business function” activities (i.e. extensionist training, FAW training, etc.). These activities detract from SANE strengthening of DAESS “software” demand-driven extension system. Thus, the Activity is responsive when capacity strengthening needs for extension workers are requested either by the Government of Malawi or, in the case of Fall Armyworm (FAW) support, by USAID. SANE view these activities as complementary to system strengthening activities (as noted in point 1 above) and conducts them in a way that also strengthens the DAESS platforms. As a Cooperative Agreement with one of its three Result Areas as Strengthened Coordination and Capacity of Extension Service Providers, making these adaptations to locally-expressed needs fall within core business functions. This is especially true when the donor specifically requests activities not in the original proposal design, as was the case for extension methodologies training and Fall Armyworm support. However, the report cites Fall Armyworm and extension worker training as examples of non-core business activities. The FAW monitoring training serves to: (a) strengthen community capacity to identify and address a community need, and (b) institutionalize the FAW response mechanism through the DAESS platforms. Building the training/teaching capacity of extension workers and Lead Farmers (who 53 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification are typically DAESS platform members) again falls under Result Area 2 and was emphasized due to requests from the GoM and USAID. It is also important to note that this particular intervention also contributes to research/extension linkages by way of strengthening community “pest forecasting” capabilities. Page 14, paragraph 3 Unfortunately, almost all E-ZOI DSPs function sporadically due to transport and allowance issues that force DSPs to meet quarterly or not at all. DSPs, not SANE, determine their own meeting schedules. This quarterly meeting schedule is an agreement between DAECCs and DSP members, who are also farmer representatives from ASPs. ASPs are encouraged to support travel of their representative to the district headquarters for the meeting to build self-reliance and not rely on service providers to bear all the costs. This is key to sustainability. Their decision to meet quarterly is to avoid overburdening ASPs in looking for their own resources to travel to the district for meetings every month. However, DSPs representatives are at liberty to engage with the DAECCs any time in case of issues that require urgent attention. Some DSPs have formed WhatsApp groups for regular communication with the district-level service providers. All of these decisions were made by the DSPs themselves (it is not a SANE ask). Page 15 These “activities by district” The District Strengthening Action Plan (DSAP) process 54 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification last paragraph raise questions about farmer input in SANE activity identification; the DSAP’s top down …” is misunderstood by the Evaluation team (and some national stakeholders) to be top-down and led by SANE when in actuality it is stakeholder-led and bottom-up. Being that is it a new process, it will take time for all members of DAESS platforms, ADDs, and national staff to grasp the DSAP approach. DSAPs are developed by representatives from ASPs, DSPs, and DAECCs in their respective districts. Briefly, the DSAP process is as follows: 1. DAESS representatives participate in a DSAP development workshop, facilitated by SANE a. SANE’s platform-level assessment (SCAT) results and any other available data and experiences are presented to DAESS platform representatives at a workshop b. DAESS representatives discuss strengths and weaknesses in small groups c. DAESS representatives prioritize areas to address to strengthen extension services and their platforms d. A template for documenting the DSAP is shared by SANE, which includes timelines, budgets, and M&E approaches e. Each district develops their proposed 55 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification strengthening activities using the template 2. After the workshop, DAESS representatives who participated present draft DSAPs to their respective platforms for input 3. Workshop participants refine the DSAP at district level 4. Revised DSAPs are submitted to SANE (or indeed any interested donors) for funding As shown, stakeholders who make up the DAESS platforms lead the DSAP process and design their own plans, not SANE. The DSAPs are increasingly used by DAECCs and DSPs for implementation, even without SANE’s direct involvement. Page 25 – recommendation 1 During ASFNS and NAEASS roll out ……. SANE support to MaFAAS during SANE’s remaining LOP… The report questions SANE’s support to MaFAAS throughout the project. However, the discussion about MaFAAS requires understanding of SANE’s mandate to build the capacity of a national-level platform recognized by the Ministry of Agriculture. Strengthening a national stakeholder panel (NSP) was intended to continue the DAESS system up to the national level. At the time the Program Description for SANE was being developed, USAID, the Department of Agricultural Extension Services, and other stakeholders 56 Page Number Area Misrepresented in the SANE MTE Report SANE Clarification identified MaFAAS for the national stakeholder panel role. Later, DAES decided to form a separate NSP, and presently the role of that NSP and that of MaFAAS as the pinnacle national stakeholder panel remains unresolved. Given this confusion, SANE (with the consent of USAID) decided to hold off engaging MaFAAS in the NSP role until the matter was resolved. The formation of DAES’s NSP was not addressed in the report however, and should be noted. We hope the above information will serve to clarify differences with the Midterm Evaluation Report of the Strengthening Agricultural and Nutrition Extension (SANE) Activity. Sincerely, Clodina Chowa Paul McNamara Chief of Party Project Director Strengthening Agricultural and Nutrition Extension Strengthening Agricultural and Nutrition Extension University of Illinois at Urbana-Champaign University of Illinois at Urbana-Champaign 57 ANNEX V: Evaluation Team Response to Statement of Differences Evaluation Team response to the Statement of Differences for the Midterm Evaluation Report From the Strengthening Agricultural and Nutrition Extension Management Team The Evaluation Team for the Mid Term Evaluation of the Strengthening Agricultural and Nutrition Extension (SANE) Activity was made up of four professionals who each had more than 20 years’ experience in Agricultural/Nutrition Extension work. The team recognizes that the evaluation process is often an imperfect process with the evaluation team having a limited amount of time to visit project sites, interview stakeholders, conduct large surveys and write up their findings. Despite the fact that this process is imperfect, the evaluation team stands by their findings. Below are additional responses from the team to the Statement of Differences. • The methodology used during this evaluation consisting of document review, direct observation, key informant interviews and focus group discussions is a methodology utilized in the majority of USAID evaluations all over the world. The team stands by this methodology although realizing it has its limitations which is pointed out in the report. • SANE Management suggested that the team did not cross check findings against reported results. However the team did this and found that some of the SANE reports were at odds with selected evaluation findings. (For example the spillover to non SANE districts.) Since SANE reports are written by SANE Management, these documents may contain bias so the team relied more on the KIIs and FGDs in cases where there was conflicting information. This was rare however. Perhaps a DQA (recommended by the Evaluation team) on the reported results will yield additional usable information on data quality reported from the field. • During the KIIs and FGDs the evaluation team inquired about SANE learnings spilling over in to non-SANE districts. Review of the written up notes from these interviews will point out the fact that at the time of the evaluation there was no evidence of spillover to non SANE districts even though reported results show that some learnings may have spilled over to non SANE districts. • The Evaluation team was somewhat surprised by the responses from SANE Management on the evaluation report. During the debrief with SANE management, there were few substantive issues raised by SANE management regarding the team’s findings. There were also many other opportunities to respond to the findings before the report was written while the team leader was in country, and during phone calls with the DCOP that took place after the team departed Lilongwe. • The SANE theory of change underwent 3 changes during the first 3 years of implementation and at the time of the evaluation the third iteration of the theory of change was still unapproved by USAID. Each version of the AMEP contained a different 58 theory of change. This may have been the reason why the team did not understand the project’s theory of change. • The team agrees that there is a difference between IFAD activities and those of SANE so the recommendation to cease activities in Blantyre and Balaka should be dropped. • The evaluation team agrees with SANE Management that the DSP quarterly meetings, although infrequent, are necessary due to the financial and time constraints of more frequent meetings and may be an important part of sustainability. James Phillips Team Leader MELS