MID-TERM PERFORMANCE EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR FOOD SECURITY POLICY October 14, 2017 MID-TERM PERFORMANCE EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR FOOD SECURITY POLICY This publication was produced at the request of the United States Agency for International Development Developed under Contract/Task Order Number: AID-OAA-I-15-00024/ AID-OAA-TO-16-00008. It was prepared independently by: Dr. Della E. McMillan, Team Leader Professor James L. Seale, Jr., Technical Specialist Contractor: Program Evaluation for Effectiveness and Learning (PEEL) Mendez England & Associates 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 www.mendezengland.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT The Feed the Future (FTF) Innovation Lab for Food Security Policy (FSP IL) consists of three consortium members—Michigan State University (MSU) as prime, and the International Food and Policy Research Institute (IFPRI) and University of Pretoria (UP), as subcontractors—that seek to contribute to Feed the Future’s goal of significant reductions in poverty and hunger among poor people through facilitating host-government policy changes that expand inclusive economic growth and reduce malnutrition. The purpose of the FSP IL mid-term performance evaluation is to assess whether FSP IL is on track to achieve these expected outcomes and to identify what, if any, adjustments are needed to improve implementation and better achieve FSP IL outcomes at the Sub-Intermediate Result level. The evaluation was conducted over a seven-month period, from January to July 2017. The evaluation team used a mixed-methods approach to assess the performance of the FSP IL, including: document review; field visits to Tanzania and Myanmar; focus group discussions and key informant interviews with stakeholders; an online questionnaire in Tanzania, Myanmar, Malawi, Nigeria, Zambia, and South Africa; and analysis of existing project databases. This report summarizes the principal findings of the evaluation, and includes 20 priority recommendations for strengthening implementation of the project for the length of the project (LOP), and 14 recommendations for future USAID/Bureau for Food Security (BFS) initiatives to strengthen the national, regional, and global policy environment in countries where USAID has Feed the Future projects. ACKNOWLEDGEMENTS The team would like to express its deep appreciation to the FSP IL Management Office for all the solid back up at every stage of the report. International Development Specialist Steve Longabaugh was the main contact for project documents and global information. FSP IL Director Mywish Maredia provided strong back up and helped develop simple tables that distilled the complexity of the project’s management structure and monitoring and evaluation systems. The team also appreciates the Chairperson of the MSU Department of Agriculture, Food and Resource Economics, Titus O. Awokuse, and the Co-Director of the MSU Food Security Group, Eric Crawford, making the long trek to Washington, DC to be interviewed. The assistance of the FSP IL Deputy Director, Xinshen Diao, and Administrative and IFPRI Project Monitoring Coordinator, Shirley Raymundo, with planning the field work was critical to getting the evaluation started on the right foot. Xinshen provided useful contact with IFPRI’s core Washington, D.C. staff as well as the IFPRI staff working at the country level. She also helped identify the stakeholder lists for the Washington, DC-based interviews of FSP IL and Mission-funded project staff. IFPRI’s hosting of the all-FSP IL meeting in March 2017 provided a well-organized forum for the team to conduct its first round of interviews with the component teams. The UP’s Lead Principal Investigator, Sheryl Hendriks, played a critical role in helping the evaluators understand how FSP IL worked from the perspective of an African university and the African governments. Especially important, she helped the evaluators better understand the Comprehensive Africa Agriculture Development Program (CAADP) processes, the role of ReSAKSS in helping the African governments support these processes, and how the FSP IL is feeding into both the CAADP processes and ReSAKSS. The team would also like to express its deep appreciation to the country directors and senior FSP IL staff that helped develop the stakeholder lists that provided the basis for the online survey. These include: George Mavrotas, Saweda Liverpool-Tasie, and Oyinkan Tasie for Nigeria; Flora Nankhuni for Malawi; Thom Jayne and Nicholas Sitko for Zambia; Sheryl Hendriks for South Africa; David Nyange for Tanzania; and Duncan Boughton for Myanmar. The richest data on FSP IL are from the two countries where the team conducted field visits. These field visits would not have been possible without the strong, forthright support received from the Mission￾funded Associate Award (AA) and Buy-in projects in those countries. Special thanks must be given to: the FSP IL Buy-in ASPIRES Chief of Party David Nyange and Professor Isaac Minde for hosting the team; Edith Lazaro, Lorna Yoyo, and Claire Ijumba who organized the field visits; and Elizabeth Mwambulukutu, who provided the team with background information before, during, and after the field visits. The team would also like to thank David Tschirley, Thom Jayne, and David Mather for explaining the history of the FSP IL core component activities in Tanzania, as well as Grace Mwangwa who translated and assisted the team with note taking in the field. In Myanmar, the team was co-hosted by CESD Director Zaw Oo, Food Security Policy Project (FSPP) (FSP IL AA) Chief of Party Duncan Boughton, and Deputy Chief of Party Ben Belton. Special thanks are due to the CESD staff that accompanied and facilitated the field visits: Ngu Wah Win, Kyan Htoo, Aung Hein, and Myat Thida Win. The team is also grateful to FSP IL/FSSP Trainer Ellen Payongayong, as well as all of the CESD management unit staff that helped organize the cars, hotels, translators, and enumerators for the field visits. None of this would have been possible without the help of the Activity Manager David Atwood (USAID/BFS), who worked in close collaboration with the two M&E specialists charged with overseeing this evaluation for USAID/Washington—M&E Specialist at BFS Lesley Perlman, and M&E Consultant Grace Fisiy. The team especially wants to express its deep appreciation to the PEEL project team for its backstopping. Its in-depth knowledge of the USAID guidance, rules, and regulations and its creativity in both design and write-up helped push the team to do its best. This includes: Susan Pietrzyk, David Hughes, Phelisha Midy, Mirela McDonald, Michelle Ghiselli, Isadora Vigier de Latour, Kendell Brown, and Florence Nyangara. Finally, we wish to thank Lynn Hurtak for her assistance with organizing the interviews, setting up the database, and editing the first draft of the concept paper, protocol, and final evaluation document. CONTENTS EXECUTIVE SUMMARY…………………………………………………………………………………….i EVALUATION PURPOSE............................................................................................................................................ i PROJECT BACKGROUND ........................................................................................................................................ i EVALUATION DESIGN, METHODS, AND LIMITATIONS ............................................................................. ii FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS......................................................................... iii 1.0 EVALUATION PURPOSE AND QUESTIONS…………………………………………………...1 1.1 EVALUATION PURPOSE............................................................................................................................. 1 1.2 AUDIENCE ....................................................................................................................................................... 1 1.3 EVALUATION QUESTIONS ....................................................................................................................... 1 2.0 PROJECT BACKGROUND………………………………………………………………………3 2.1 MECHANISM DESCRIPTION ..................................................................................................................... 3 2.1.1 Context .................................................................................................................................................... 3 2.1.2 Goals and Objectives............................................................................................................................ 3 2.1.3 Target Areas and Target Population Groups.................................................................................. 5 2.1.4 Program Implementation Plan............................................................................................................. 5 3.0 EVALUATION METHODS AND LIMITATIONS………………………………………………...7 3.1 EVALUATION METHODOLOGY............................................................................................................. 7 3.1.1 Qualitative Methods.............................................................................................................................. 7 3.1.2 Quantitative Methods........................................................................................................................... 9 3.1.3 Other Data Sources............................................................................................................................ 10 3.2 LIMITATIONS ................................................................................................................................................ 10 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS…………………………………11 4.1 POLICY ANALYSIS AND RESEARCH (EQ CLUSTER #1) ............................................................... 11 4.1.1 Findings................................................................................................................................................... 11 4.1.2 Conclusions........................................................................................................................................... 16 4.2 CAPACITY STRENGTHENING (EQ CLUSTER #2)........................................................................... 17 4.2.1 Findings................................................................................................................................................... 17 4.2.2 Conclusions ........................................................................................................................................... 31 4.2.3 Recommendations................................................................................................................................ 34 4.3 PROGRESS TOWARD POLICY OUTCOMES (EQ CLUSTER #3) ................................................ 35 4.3.1 Findings................................................................................................................................................... 35 4.3.2 Conclusions ........................................................................................................................................... 47 4.3.3 Recommendations................................................................................................................................ 48 4.4 MANAGEMENT AND IMPLEMENTATION (EQ CLUSTER #4)..................................................... 50 4.4.1 Findings................................................................................................................................................... 50 4.4.2 Conclusions ........................................................................................................................................... 55 4.4.3 Recommendations ................................................................................................................................ 56 ANNEX I....................................................................................................................................................................... 58 ANNEX II...................................................................................................................................................................... 68 ANNEX III .................................................................................................................................................................... 73 ANNEX IV.................................................................................................................................................................... 79 ANNEX V..................................................................................................................................................................... 91 ANNEX VI................................................................................................................................................................. 129 ANNEX VII................................................................................................................................................................ 139 ANNEX VIII ............................................................................................................................................................... 174 LIST OF FIGURES AND MAPS Figure 1. Results Framework for the FSP IL................................................................................................................ 4 Map 1. Locations of Key Capacity Building/Analysis/Research Sites, Including Global and Regional Interventions .......................................................................................................................................................... 7 LIST OF TABLES Table 1. FSP IL Mid-Term EQs and EQ Clusters....................................................................................................... 2 Table 2. Countries with Activities Supported Under the FSP IL Core, AAs, and Buy-ins............................... 6 Table 3. Sources of Qualitative and Quantitative Data for the Seven Categories of FSP IL Stakeholders in the Mid-Term Performance Evaluation ....................................................................................................... 8 Table 4. Respondents Who Agreed with the Statement That the Coordination of FSP IL Capacity Strengthening Activities is Effective and Improving the Use of Evidence to Advocate for Policy Change (x=number who agreed/y=number who responded to the question) ............................... 18 Table 5. Areas of Collaboration Between the FSP IL Component C4A Global Team and its Principal Local Research Partners Associated with ReNAPRI.............................................................................. 23 Table 6. Examples of Capacity Building Coordination Between ReSAKSS-Supported Activities and the FSP IL Component and AA Teams in Myanmar, Senegal, and Malawi............................................... 26 Table 7. Examples of Best Practice in Capacity Strengthening Coordination, Organization, and Implementation in Target Countries.......................................................................................................... 27 Table 8. Respondents Who Agreed with the Statement that the Organization and Implementation of Capacity Strengthening Activities (i.e., Policy Research, Analysis, Strategy, Communication, and Advice) is Highly Effective at Promoting Policy Change at the Following Levels............................ 28 Table 9. Respondents Who Agreed that the FSP IL Works Closely with the Following Partners and Entities to Affect More Rapid Policy Change at the National, Regional, and Global Levels......... 36 Table 10. Respondents with an Opinion Who Agreed That the Following FSP IL Components Have Contributed to Rapid Change in Policy at the Country Level and Their Rankings of Components .................................................................................................................................................... 36 Table 11. Different Organizational Models That the FSP IL Has Used to Support Policy Change at the Country Level, the Link to the Lead Ministries of Agriculture and Challenges Moving Forward ............................................................................................................................................................................. 45 Table 12. Ranking of the Different Types of Teams (Expatriate, Local, Mixed) in Terms of Effectiveness for Policy Analysis and Dialogue ................................................................................................................. 47 Table 13. Strengths and Weaknesses of the FSP IL Management and Coordination Systems...................... 50 Table 14. Summary Strengths and Weaknesses of the FSP IL Funding Arrangements................................... 53 Table 15. Summary Strengths and Weaknesses of the FSP IL Monitoring and Evaluation Systems............ 55 LIST OF TEXT BOXES Text Box 1. Stakeholder Feedback on the question: “Did the design and conduct of the FSP IL policy analysis and research influence policy review?”............................................................................... 14 Text Box 2. Stakeholder Feedback on the question: “Did the design and conduct of the FSP IL policy analysis and research influence policy formulation?”..................................................................... 15 Text Box 3. Case Study—Tanzania: FSP IL Coordination with a University-Affiliated Food Policy Research Institutes.................................................................................................................................. 20 Text . Box 4. Sample Cases of FSP IL Collaboration with Africa Lead and AGRA in Tanzania .................... 21 Text Box 5. Coordination of Capacity Building Between the FSP IL and the African Union Commission’s (AUC’s) CAADP .....................................................................................................................................25 Text Box 6. Stakeholder Responses to the Question: “In what ways can each partner's capacity strengthening approaches be improved in this country?” ................................................. 32 Text Box 7. Myanmar Case-study: Type of FSP IL-Facilitated Policy Influence................................................ 38 Text Box 8. Tanzania Case-study: Type of FSP IL-Facilitated Policy Influence................................................ 39 ACRONYMS A4NH Agriculture for Nutrition and Health AA Associate Award AAEA American Agricultural Economics Association AAPC Agricultural Annual Policy Conference ADB African Development Bank ADS Agricultural Development Strategy ADMARC Agricultural Development and Marketing Corporation AFRE-FSG Department of Agricultural, Food, and Resource Economics – Food Security Group, MSU AGRA Alliance for a Green Revolution in Africa AOR Agreement Officer’s Representative APA Associate Policy Analysts ARP Agricultural Research Policy ASDP Agricultural Sector Development Program ASPIRES Accelerating Strategies for Practical Innovation and Research in Economic Strengthening AUC African Union Commission BFAP Bureau for Food and Agricultural Policy BFPR Bureau for Food Policy and Agricultural Research, South Africa BFS Bureau for Food Security C Component CAADP Comprehensive Africa Agriculture Development Program CARD Centre for Agricultural Research and Development CEPPAG Research Centre for Agricultural and Food Policies and Programs CESD Centre for Economic and Social Development CGIAR Consultative Group on International Agricultural Research CILSS The Permanent Interstate Committee for Drought Control in the Sahel CKM Communications Knowledge Management, IFPRI CLA Collaborative, Learning, and Adaptation COE Center of Excellence COMESA Common Market for Eastern and Southern Africa COP Chief of Party COR Contracting Officer’s Representative DAEERD Department of Agricultural Economics, Extension and Rural Development DAPSA Direction de l'Analyse, de la Prévision et des Statistiques Agricoles DFID Department for International Development DC District of Columbia (Washington) DGO Director General’s Office, IFPRI DQA Data Quality Assessment DSGD Development Strategy and Governance Division, IFPRI ECOWAS Economic Community of West African States EOI Expression of Interest EPTD Environment and Production Technology Division, IFPRI EQ Evaluation Question ESSP Ethiopia Strategy Support Program ET Evaluation Team EU European Union FAO/MAFAP United Nations Food and Agriculture Organization/Monitoring and Analyzing Food and Agricultural Policies Program FGD Focus Group Discussion FISP Farm Input Subsidy Program FLO First-level Objective FSRP-III Food Security Research Project III FSP IL Feed the Future Innovation Lab for Food Security Policy FSPP Food Security Policy Project FTF Feed the Future GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GISAIA Guiding Investments in Sustainable Agricultural Intensification in Africa GOT Government of Tanzania HICD Human Institutional Capacity Development IAPRI Indaba Agricultural Policy Research Institute IDIQ Indefinite Delivery Indefinite Quantity IFAD International Fund for Agricultural Development IFDC International Fertilizer Development Center IFNuW Institute of Food, Nutrition and Well-being, UP IFPRI International Food Policy Research Institute IL Innovation Lab IR Intermediate Result ISP Input Subsidy Policy IT Information Technology JRC Joint Research Commission KII Key Informant Interview KM Knowledge Management LGFA Local Government Finance Act LGA Local Government Agency LIFT Livelihoods and Food Security Trust Fund LOP Length of Project LWA Leader with Associate M&E Monitoring and Evaluation MAAS Myanmar Agriculture and Aquaculture Survey MALF Ministry of Agriculture, Livestock and Fisheries MAPI Malawi Agricultural Policy and Implementation MGD Millennium Development Goal MI Michigan MITI Ministry of Industry, Trade and Investment MIU Market Intelligence Unit MOALI Ministry of Agriculture Livestock and Immigration MS Master of Science MSU Michigan State University MTID Markets, Trade, and Institutions Division, IFPRI NAPAS:Malawi New Alliance Policy Acceleration Support – Malawi NAIP National Agricultural and Food Security Investment Plan NBS National Bureau of Statistics NGO Non-Governmental Organization NSSP Nigeria Strategy Support Program PAC Partnership Accountability Committee PAG Policy Analysis Group PAPA Projet d’Appui aux Politiques Agricoles PAPAC Platform for Agricultural Policy Analysis and Coordination PE Partial Equilibrium PEEL Program Evaluation for Effectiveness and Learning – USAID task order contract PHND Poverty, Health, and Nutrition Division, IFPRI PI Principal Investigator PIM Policies, Institutions, Markets PIRS Performance Indicator Reference Sheets PMCA Policy, Management, Constraints, Action PMP Performance Monitoring Plan PRC Policy Resource Center PS3 Public Sector Strengthening ReNAPRI Regional Network of Agricultural Policy Research Institutes ReSAKSS Regional Strategic Analysis and Knowledge Support System RFA Request for Application SAKSS Strategic Analysis and Knowledge Support System SAGCOT Southern Agricultural Corridor of Tanzania SGD Sustainable Development Goal SMART Specific, Measurable, Accurate, Realistic, and Time bound SME Subject Matter Expert SR Strategic result SUA Sokoine University of Agriculture Sub-IR Sub-intermediate Result SWOT Strengths, Weaknesses, Opportunities, and Threats TEC Total Estimated Cost TMCF The MasterCard Foundation TNBS Tanzania Bureau of Statistics UP University of Pretoria U.S. United States USAID United States Agency for International Development. USG United States Government VUNA Vuna (local name for water in southern Africa; a southern African NGO i EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of the Feed the Future Innovation Lab for Food Security Policy (FSP IL) mid-term performance evaluation was to assess: 1) whether the FSP IL is on track to achieve its intended results; and 2) what, if any, adjustments are needed to improve implementation and better achieve FSP IL outcomes at the Sub-Intermediate Result (Sub-IR) level in the targeted countries.1 The evaluation was also expected to extrapolate important lessons learned for future USAID/Bureau for Food Security (BFS) initiatives. Because a significant part of FSP IL is related to country-level policy analysis and capacity building, the mid-term evaluation focused on results at the country level, including where Missions are funding Associate Awards (AAs) or Buy-ins. Although the country-level AAs and Buy-ins were taken into consideration when evaluating the core-funded in-country work, they were not evaluated since they are Mission-funded projects. PROJECT BACKGROUND FSP IL seeks to contribute to Feed the Future’s goal of significant reductions in poverty and hunger among poor people through facilitating host-government policy changes that expand inclusive economic growth and reduce malnutrition. It is implemented by a consortium of three members: Michigan State University (MSU) as prime, and the International Food and Policy Research Institute (IFPRI) and University of Pretoria (UP) as subcontractors. Since the FSP IL started, it has adopted an activity-work program, organized into five activity-component teams: ● ● ● ● ● Component 1 (C1): Country/regional-level collaborative research (on farms, firms, and markets) and formulation/analysis of policy options; Component 2 (C2): Country/regional-level capacity building for policy (data, analysis, advocacy, formulation, consultation, coordination, and implementation); Component 3 (C3): Global collaborative research on support to the policy process and policy capacity; Component 4 (C4): Engagement in global policy debates on food and nutrition security; and Component 5 (C5): Strategic analytical agenda and support to donor policy and strategy. During the first year, FSP IL implemented activities to support country-level policy analysis and capacity building in six countries—Nigeria, Mali, Malawi, Tanzania, Senegal, and Myanmar—and global-level research and policy engagement in Myanmar. In subsequent years, the FSP IL Leader with Associate (LWA) award was amended several times and received Mission support through three Mission Buy-ins. The FSP IL has also received support through six Mission AAs, which are standalone agreements with country Missions. Since the program started, these combined initiatives have supported activities in 12 countries in Africa and two in Asia. It is currently active in 10 countries—Ghana, Malawi, Mali, Myanmar, Nigeria, Rwanda, Senegal, South Africa, Tanzania, and Zambia. Each of the AAs and Buy-in projects: ● ● Is “owned” by the Mission that decided to fund and implement the project as part of a larger Mission Feed the Future portfolio in that country that is linked to the larger FSP IL LWA award; and Has a separate set of indicators, which includes some of the FSP IL indicators, as well as a number of standard Feed the Future indicators that the Mission-level projects track as part of their routine Performance Monitoring Plans (PMPs). 1 USAID/BFS. 2016. PEEL Task Order. Expression of Interest – Performance Evaluation. Washington, DC: USAID/BFS. Pg.6. ii At the same time, for the purposes of the core planning and management of its principal donor, the USAID/Bureau for Food Security (BFS), the FSP IL functions as a coherent program with potential funding up to $70 million in which the country programs (funded by the Missions) were designed to be implemented in a larger framework and with many of the same staff that work on the core USAID/BFS￾funded activities. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation team (ET) collected the data needed to answer 12 evaluation questions (EQs) through: ● ● ● ● Thirty-four key informant interviews (KIIs) and 24 focus group discussions (FGDs) executed during two 12-day field visits to Tanzania and Myanmar during May 2017; Forty-seven self-administered online questionnaire surveys completed by a stratified random sample of staff and stakeholders in six of the 14 countries where the FSP IL intervenes—Tanzania, Myanmar, Malawi, Nigeria, South Africa, and Zambia—between April and July 2017; Ten FGDs and five KIIs with FSP IL core staff during the FSP IL meeting in Washington, DC in March 2017; and A review of the basic FSP IL project documents and monitoring and evaluation (M&E) data as well as reports for the two case-study countries, Tanzania and Myanmar. Altogether, the two-person ET interviewed 260 people in person and 47 people through the anonymous online survey. There were several limitations for this evaluation. They include: 1. Focus on the two case-study countries of Tanzania and Myanmar. The ET attempted to minimize the impact of this limitation on the evaluation’s independence by combining the case studies with an online questionnaire administered to a stratified random sample of staff and stakeholders in six non-case-study countries. 2. Reliance on the FSP IL Management Office staff to coordinate with the senior staff in the Mission-funded AA and Buy-in projects in developing the country stakeholder lists, which were then used to draw the sample for the online questionnaire. The ET sought to minimize this limitation by reviewing the draft stakeholder lists with multiple staff to make sure they were complete, and by making a random choice of the staff and stakeholders. 3. Inability to clearly differentiate policies influenced by FSP IL and those influenced by the Mission￾funded AAs or Buy-ins. In almost all cases, FSP IL’s research and capacity strengthening are only part of a much broader effort by national governments and their regional and international partners. It is for this reason that the EQs ask the evaluation to focus on FSP IL’s contribution to these processes rather than its policy “impact.” 4. The ET was unable to make a campus visit to MSU nor to have any phone interviews with any business management staff to get MSU’s perspective on this large and complex LWA. To address this issue, MSU made available two of its senior administrators to be interviewed by the evaluators during the FSP IL meeting in Washington, DC. 5. Response rate of the online survey was 52 percent (47 out of a sample size of 90). Of those that completed the online survey, 51 percent of the respondents were either FSP IL staff (34 percent) or AA/Buy-in staff (17 percent), while the other 49 percent consisted of FSP IL partners (17 percent), stakeholders (19 percent), and USAID Mission staff (13 percent). Of the nine responding stakeholders, only two were government staff. It should be noted that the FSP IL and AA/Buy-in staff were in some cases less positive in their responses about policy influence than the other respondents (partners, stakeholders, and Mission staff). In anticipation of this issue, the evaluation design included two intensive case studies, which allowed the evaluators to have better access to the policy research “users” as well as those generating the research. iii FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Based on this analysis, the ET determined that there is a great deal of qualitative and quantitative evidence that the FSP IL has exceeded its original intended results in terms of the two Feed the Future Sub-IRs and Sub-Sub IRs.2 Furthermore, it has worked in a larger number of countries than the initial IL was designed to intervene.3 This successful record has also galvanized a great deal of interest within USAID and other donors about the potential merits of linking a major land grant university in a developed country with an international agricultural research center and the university-based research centers in developing countries. For this reason, the team has focused its recommendations at two levels. The first level of recommendations focuses on helping to consolidate and coordinate the results that have already been achieved in collaboration with the FSP IL’s local partners, including the Mission￾funded AAs and Buy-ins. The second level focuses on providing feedback to the USAID/BFS for future Leader with Associate Awards (LWA) that may seek to extend the FSP IL model for applied policy research. 2 Sub-IR 1.1: Enhanced institutional capacity development; Sub-IR 1.3. Better policy environment (inclusive and stable incentive and opportunities; strengthened regulatory and enforcement infrastructure); Sub-Sub IR 1.3: Better policy formulation process (inclusive, stable, timely, transparent evidence-based, mutually accountable). 3 When the FSP IL was designed, it was expected that the project would intervene in a relatively small number of focus countries although the original cooperative agreement did not specify which countries. The IL currently has signed Associate Awards and Buy-in agreements in nine countries and has supported more isolated activities in five additional countries. iv Findings Conclusions Recommendations EQ Cluster 1: Policy Analysis and Research EQ 1.1. “In what ways, if any, has the selection and conduct of policy analysis and research influenced policy review, formulation, or change by host country governments?” The literature review and interviews show that each of Although the concept of demand-driven research was Future LWAs: Ensure a participatory process that: 1) the FSP IL component teams participated in two broad one of the six core principles of the FSP IL, some of the selects research topics in close collaboration with types of activities: supply driven and demand driven. biggest success stories emerged when the local country-level partners; and 2) conducts the research Both types of activities influenced policy review, stakeholders agreed to support new diagnostic studies in close collaboration with these same partners formulation, or change by host governments. or to conduct in-country research on some of the whenever possible. Conduct of policy analysis and research was heavily influenced by the component leader’s and lead topics that the USAID/BFS wanted FSP IL to examine on a comparative basis. This participatory selection process should include research topics that are both demand driven (by the institution’s previous work in conjunction with their By selecting and conducting policy analysis and research government’s and Mission’s requests) as well as contacts (stakeholders) in the countries of engagement. in a participatory manner with stakeholders including supply driven (on topics that USAID/BFS and the Early evidence of policy influence was that 81 percent of online survey respondents indicated that the highly participatory manner in which the initial FSP IL studies were selected, designed, and conducted affected their policy influence. host-country government, FSP IL policy research and analysis was able to influence policy review, formulation, or change by host-country governments because this participatory process led to ownership of FSP IL-led policy analysis and research findings by stakeholders including host-country governments. researchers consider to be important). EQ 1.2. “What areas of policy analysis and research, if any, need more attention or should be included to bring about outcomes related to a better policy environment?” To date, few of the FSP IL research studies appear to be Policy changes can have unforeseen positive and Length of Project (LOP): The FSP IL component monitoring the impact of policies that they helped negative impacts, making it important to evaluate their teams need to collaborate with local partners on the create or reform (e.g., Step 6 of the six-step model that short-term and medium-term impacts on client groups design and execution of rigorous studies that monitor 4 most of the FSP IL component teams follow). This and vulnerable households (Step 6 of the six-step model the short and medium-term impacts on local monitoring is needed to identify what, if any, that was implicit in most of the FSP IL component stakeholders of some of the policy reforms they adjustments might be needed to capitalize on the early team’s long-term research agendas). influenced. achievements of a policy change. It may also help identify some unintended negative consequences that could result from new policy recommendations being adopted. In Tanzania, for example, the partial adoption of some of the FSP IL research recommendations is having a 5 negative effect on local farmers. Indeed, the need for 4 Step 1. Identify conventional wisdoms underlying current policies and policy making/change processes; Step 2. Test those as hypotheses in field survey and empirical analysis; Step 3. Show governments how empirical findings do or do not support the underpinning conventional wisdoms of their policies; Step 4. Strategize policy changes that fit the findings and the realities unearthed by field work; Step 5. Build medium-term strategy to form the enabling environment to implement the policy changes by extension of capacity building started in steps1-4 with collaborators, local institutions, government, and donors; and Step 6. Track impact and identify where new changes make it necessary to repeat steps 1-5 again. 5 See discussion of partial adoption of recommendations of the local government agency (LGA) cess tax study in main text. v Findings Conclusions Recommendations better monitoring of the long-term consequences of policies influenced by FSP IL research was one of the principal needs of local stakeholder groups and staff affiliated with the Mission-funded Buy-in in Tanzania. The impact of international trade rules, regulations, and standards was identified as the top issue by all of the private-sector producer group and individual farmers in both case-study countries. The issue of international and inter-regional trade on paddy rice emerged over and over in the Tanzania stakeholder meetings, even at the village level. In Myanmar, the issue of international trade is extremely important because of the country’s proximity and easy access to China’s huge fruit and vegetable import market and to India’s vast import market for pulses and beans. Currently, none of the component teams are focused on trade. Having trade as an explicit and incorporated part of each separate subcomponent would allow the project to address trade policy issues in a cost-effective manner without disrupting current upstream and downstream policy research activities. It would also help the project to better capitalize on the emerging regional and international debates on this topic. LOP: Given the critical importance of this issue, the FSP IL should consider adding policy research and analysis on trade issues as a new subcomponent, C4C, in the remaining years of the project. Future LWA: Encourage future requests for application (RFAs) that provide the basis for new program designs to use an “open economy”6 paradigm to inform the design of their global and country-level activities. EQ Cluster 2: Capacity Strengthening EQ 2.1. “How effective has the coordination of capacity strengthening activities been among the FSP IL components, FSP IL partners, USAID Missions, and related entities at the country level?7 In what ways could this be improved?” The initial investment by FSP IL quickly created a wide range of partners that have very different capacity￾strengthening needs, which increases the difficulty of coordinating capacity-strengthening activities through a single ministry-based policy office or USAID Mission￾funded program. It was this issue that led Tanzania’s Ministry of Agriculture to request FSP IL’s assistance in the development of the Policy Analysis Group (PAG), whose mission is to better harmonize the policy Two promising new mechanisms that are widely recognized by the USAID/BFS staff and AA/Buy-in staff and identified as examples of best practice by a wide range of stakeholders for facilitating cost-effective policy coordination are the FSP IL-facilitated AAPC and PAG in Tanzania. LOP: Work with bilateral USAID Missions and Africa Lead to facilitate government partners of FSP IL in other countries to attend the next AAPC and, while in Tanzania, to meet with the FSP IL-facilitated PAG. 6 To effectively and comprehensively study and analyze downstream and even upstream activities within a country, it is often imperative to understand the driving forces for those activities that occur from outside the national boundary. An open-economy paradigm starts with the premise that countries trade goods and services with other countries, often with their adjacent neighboring countries. The consequence of using an open-economy paradigm in the design of regional, global, and even country-level activities is to ensure that the effects of trade in goods and services is taken into account. For example, in Myanmar, China’s demand for specialty crops such as melons and India’s demand for pulses are two of the largest stimuli for Myanmar’s agricultural sector outside of the rice sector. In Tanzania, its implicit trade ban on maize has numerous effects on the maize sector that are potentially not identified and understood if the paradigm is a more closed-economy model, one that does not account for the many effects that trade (or trade bans) in goods and services have on the local agricultural sector—such as prices of commodities, availability of goods, availability of agricultural inputs, profits and losses in the agricultural processing sector, transportation of agricultural commodities, and even innovations in agriculture. 7 Such as Africa Lead, which supports capacity building under the Feed the Future Initiative and the African Union Commission’s Comprehensive Africa Agriculture Development Program (AUC CAADP); Alliance for a Green Revolution in Africa (AGRA); Food and Agriculture Organization of the United Nations/Monitoring and Analyzing Food and Agricultural Policies Program (FAO/MAFAP); Regional Strategic Analysis and Knowledge Support System (ReSAKSS); Futures Agriculture Consortium; International Fertilizer Development Center (IFDC); or other policy analysis groups. vi Findings Conclusions Recommendations research and recommendations of the different policy actors in Tanzania. The PAG started an Agricultural Annual Policy Conference (AAPC), which provides a useful forum for the FSP IL global teams and other national and regional stakeholders to share the results of their work with a large number of national, regional, and global actors in southern and eastern Africa. Even though there was no contractual obligation to report on its country-level activities that were funded by the FSP IL core budget, FSP IL was willing and able to respond to the Missions’ desire for close coordination because, in most cases, it had in-country chiefs of party (COPs) and experienced Contracting Officer’s Representatives (CORs). This coordination helped facilitate collaboration with the wider donor community and, in many cases, helped leverage additional capacity￾strengthening resources. Unfortunately, only one of the Mission-funded AA or Buy-in projects adopted all of the indicators in the FSP IL PMP, which has made it more difficult for the Missions to track the country-level impacts of the Mission-funded AAs and Buy-ins and their coordination with the FSP IL component teams. Some initial challenges for coordination and reporting existed because the cooperative agreement did not spell out the USAID/BFS’s expectations for coordination or joint reporting with the Mission. Future LWAs: Future LWA cooperative agreements need to spell out USAID/BFS’s expectations for coordination with the Mission and joint coordination of activities and M&E with any Mission-funded AAs or Buy-ins that result from the leader award. Coordination between the researchers on the different FSP IL teams and between the teams, and the FSP IL country-level partners and related entities tended to be stronger in the early years before the Missions agreed to fund the AAs and Buy-ins. Once the Buy-ins and AAs were created, these country programs developed their own strategy and staff and focused on their own project’s objectives, which tended to respond to more specific Mission needs. Once the Mission-funded projects started, many of the original C1/C2 staff migrated to the other component teams (C4A, C4B, C3) and to Mission-funded projects. While the Mission-funded projects have strengthened the impact of the FSP IL, they have created new challenges for coordination with some of their country￾level partners and the USAID country-level Missions. Future LWAs: The proposals for new Mission-funded AAs and Buy-ins need to be written with explicit linkages between the AAs and Buy-ins and the LWA to facilitate joint capacity building, coordination, strategic planning, M&E, and reporting with the component teams associated with the leader award. vii Findings Conclusions Recommendations EQ 2.2. “How effective has FSP IL been in the organization and implementation of capacity strengthening activities covering different content such as policy research, analysis, strategy, and communication as well as providing advice to improve the use of evidence to advocate for policy change at the following levels: a) individuals8 who work at the country level (including ministers, parliamentarians, and other host government staff engaged in the policy process); b) individuals who work for organizations that support food policy at the national and local level (including farmer groups, non-governmental organizations [NGOs], and other civil society groups); and c) Individuals in key positions of influence (such as journalists); as well as d) the global development process (which engages and coordinates each or several of these levels) in specific countries?” Responses to the online survey corroborated the evaluators’ findings from the case studies that: ● FSP IL has organized a series of activities to build the capacity of its most important government and national research partners to design, implement, execute, and write-up specific studies, and this is widely appreciated by USAID and other donors, NGOs, and private-sector organizations that were interviewed in the two case-study countries; and ● Increased institutional capacity for research, analysis, advocacy, and proposal development stemming from FSP IL activities has successfully promoted the types of dialogue among different levels of policy actors needed to affect policy change and better policy development process. To date, however, there has been no systematic attempt to track this increased capacity or the contribution of specific core and Mission-funded activities to its development. Since its inception, the FSP IL was intended to coordinate with two major regional entities, Regional Strategic Analysis and Knowledge Support System (ReSAKSS) and Regional Network of Agricultural Policy Research Institutes (ReNAPRI), which were designed to build the capacity of some of the leading university and non-university-based research institutions to conduct, analyze, and use evidence-based research to improve national food policies. To date, however, the FSP IL has not developed a coordinated capacity-strengthening While coordination, organization, and implementation of capacity-strengthening activities by the FSP IL have been effective in developing the initial capacity of their key government, private-sector, and civil-society partners, the approach in general has been ad hoc and based on the personal experiences and professional networks of the lead researchers in specific countries. This type of generalized capacity strengthening was more appropriate in the early exploratory phase of the FSP IL, but going forward a more structured strategy and approach for capacity strengthening would be more effective. University and non-university-based food policy institutes have been some of the FSP IL’s most reliable local research partners, but it is hard to determine whether they have the core capacities needed to sustain their role going forward. While FSP IL’s documentation shows that individual FSP IL researchers and Mission-funded projects are coordinating with ReSAKSS and ReNAPRI, these coordination activities have been largely driven by individuals and not a conscious FSP IL strategy. LOP: Develop a comprehensive capacity strengthening strategy for FSP IL that informs and is informed by a complementary set of national-level strategies in the Mission-funded AAs and Buy-ins. Specific sub-recommendations for achieving this include: 1. Conduct a comprehensive mapping of the current capacity strengthening activities supported by FSP IL and the Mission-funded AAs and Buy-ins in each of the target countries that identifies which capacities the key partners need in order to affect more rapid policy change; 2. Utilize the information generated by the mapping to work with local partners (including the AAs and Buy-ins) to develop country-specific and stakeholder/capacity building strategies; 3. Focus the strategy on a limited number of critical partners in each country including: a) its key ministry partners; and b) the university-based think tanks and promising non-university think tanks in countries like Myanmar where the universities do not have the legal independence and autonomy that they do in other countries where the FSP IL is engaged; 4. Pay careful attention to conducting a detailed baseline assessment of key capacities that the critical partners will need to sustain their involvement in evidence based policy research, analysis, and advocacy; 5. Encourage: a) the AA and Buy-in Projects to report on the execution of their country-level 8 In response to feedback on the first draft of this report, the formulation of this EQ was changed slightly (see Table 2 for the original formulation and the revisions, which were added in August 2017 based on USAID/Washington feedback). viii Findings Conclusions Recommendations strategy to link its activities to either the regional or national-level ReSAKSS and African Union Commission’s Comprehensive Africa Agriculture Development Program (AUC CAADP) activities. capacity strengthening strategies to the Mission at least once a year; and b) FSP IL to report on each country’s execution of its capacity strengthening strategy in its semi-annual and annual reports to USAID/BFS; and 6. Ask MSU to co-lead the capacity strengthening strategy with part of the UP team (as envisioned in the original FSP IL cooperative agreement proposal). Future LWA: Future cooperative agreements based on the FSP IL LWA model need to develop a systematic approach to human and institutional capacity development. Specific sub-recommendations include: 1. Give priority to key ministry partners and university-based policy institutes whenever possible; 2. Have a planned strategy to build the core capacity of these institutions in collaboration with other regional and donor-funded programs that support them; and 3. Track current and project synergies among these partners and some of the most critical regional entities and programs (like ReSAKSS, ReNAPRI, AGRA, Africa Lead). EQ 2.3. “In what ways could the capacity strengthening component be improved for greater policy impact—including support to local stakeholder groups?” All six of the community-based stakeholders who were interviewed by the ET stated that they were aware that they and their communities had been studied and were anxious to learn about the results of the studies. Although none of the FSP IL country programs have made the NGO community a major focus of their intervention, they have sometimes forged strong working relationships with the NGOs that intervene in certain topical areas or geographical areas. This collaboration has had a major impact on the global conceptualization of the NGO’s programs as well as To date, FSP IL has not developed a focused strategy for how it will strengthen the capacity of local stakeholder groups as partners in food policy reform, but there is a great deal of evidence from the field visits that local stakeholders want to be informed and have access to social media applications and NGO programs that would facilitate this communication. LOP and Future LWA: ● Given the cost and security clearance problems associated with sending an enumerator back to debrief a surveyed village on the results of a survey, consider presenting and discussing some of the results of the other previous surveys conducted by FSP IL when initiating a new survey. ● Given the cost and difficulty of building effective engagement strategies with local stakeholder groups on a five-year IL, future programs should consider ways to reach local stakeholder groups ix Findings Conclusions Recommendations some of its interventions that could be documented in the FSP IL online publication series. with appropriate summaries of their major research findings through the NGOs that are already working with these groups. EQ 2.4. “What capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities? What types of synergies, if any, have there been among the varied MSU, IFPRI, and UP capacity strengthening approaches? In what ways can each partner’s capacity strengthening approaches be improved?” The mid-term case studies in Tanzania and Myanmar show that: 1) the major policy stakeholder groups in each country vary widely in terms of which capacity delivery methods they found most useful; and 2) the preference of one individual stakeholder or group of stakeholders for one type of capacity-delivery method was likely to evolve as they increased their familiarity with a particular area of research findings. To date, however, the decisions about what types of capacity strengthening an institution or group needs have been ad hoc and based on the individual researcher’s or COP’s understanding of the situation and personal contacts. It is possible to see strong synergies between the three, senior consortium leads in terms of their approach to strengthening the capacity of and collaborating with national policy and research centers such as ReNAPRI and RESAKSS that are tasked with building country-level capacity for evidence-based food policy research and analysis. While there is anecdotal evidence that different stakeholder groups responded to different delivery methods better than others, FSP IL has not developed a system for tracking stakeholder feedback on this issue. With a few notable exceptions, there was a tendency for each consortium partner to plan and execute its principal activities on its own with its local partners. This has reduced potential synergies. LOP: ● To capitalize on potential synergies, encourage the three senior FSP IL partners to develop joint country-level work plans when feasible. ● Develop a simple mechanism for stakeholders to give better feedback on which capacity-delivery methods have been more or less effective for delivering the best capacity-strengthening activities. EQ Cluster 3: Progress Toward Policy Outcomes EQ 3.1. “To what extent have the five different activity components (Figure 1) contributed toward policy change at the global, regional, and country levels? In what ways could these components be expanded or improved to better support policy influence or policy change? What is country-level policy change/influence and how could this be expanded? How have C3, C4, and C5 contributed to more effective or rapid policy change at the country level?” While there is evidence that the five component teams have contributed to policy change at the global, regional, and country levels, there is weak coordination of research, capacity building, and advocacy amongst the component teams themselves, for example, among C3, C4A and C4B. This weak coordination has lessened FSP IL’s ability to backstop the country programs in a holistic The FSP IL activity components have been effective in building the initial capacity of the country programs and in strengthening the linkages between these programs and some of the most important global and regional initiatives that support evidence-based food policy like ReSAKSS, ReNAPRI, and Africa Lead. However, a better coordination amongst the components is needed in order for the FSP IL to better support policy influence LOP: Going forward, the component teams need to capitalize on the increased capacity of the Mission￾funded country programs by developing a more cohesive model for coordinating their county-level activities as well as those directed at influencing regional and global policy debates. Specific sub￾recommendations include: x Findings Conclusions Recommendations manner. or change. 1. Strengthen the coordination of research between the FSP IL regional-global teams and the Mission￾funded AA and Buy-in projects; 2. Strengthen the cohesion, coordination, and leadership among and within the FSP IL global component teams themselves; and 3) Build better evidence-based systems for joint planning between the component teams and the most important regional USAID-funded and supported regional initiatives like Africa Lead and ReSAKSS. EQ 3.2. “How effectively has FSP IL communicated policy analysis, recommendations, and options to policymakers or others in a position to influence them, and in what ways could this communication be improved (including content, target, etc.)?” Most of the stakeholders interviewed felt that FSP IL has been effective in communicating policy analysis and recommendations but that the project needs a formal communication strategy. To date, FSP IL does not have a fully integrated strategy that includes a mechanism for regular monitoring of, reaction to, and use of messages received by each of the FSP IL’s major national, regional, and global partners. This is important because the effectiveness of different communication channels shifts as stakeholders strengthen their capacity to use the materials.9 Since the end of the third year, the FSP IL has made a concerted effort to develop a more focused communication plan for all of its core-funded activities. This same initiative has encouraged many of the Mission-funded projects to hire full-time communication officers. The project, however, lacks an integrated strategy, and a mechanism to monitor key stakeholder groups use and needs. Interviewees asked for, in addition to briefs, access to reports that provide detailed information about the results of the projects. Briefs and full reports meet different needs of the stakeholders. LOP: Develop a comprehensive communication strategy for FSP IL that informs and is informed by a complementary set of national-level strategies in the Mission-funded AAs and Buy-in projects. Specific sub￾recommendations for achieving this include: 1. Include a tracking table that identifies the top key informants and backup key informants for each stakeholder group in each country;10 2. Determine which mechanisms and individuals are necessary to ensure quality feedback from different key stakeholder groups about how effective FSP IL’s communication with them has been and how it could be improved; 3. Encourage the Mission-funded AAs and Buy-ins to conduct a disaggregated analysis of this information at least once a year, and FSP IL to include this information in its annual reports to USAID/BFS; and 4. Ask IFPRI to co-lead the design and monitoring of this communication strategy in collaboration with the UP FSP IL team. Future LWA: Make communication capacities and experience a core competency requirement in future 9 Many of the staff noted, for example, that although they liked the FSP IL briefs, they also needed access to the bigger, more detailed reports once they shifted to active policy engagement and advocacy since these provided the types of details needed to support new program designs. 10 If the strategy is clarified to this level, it can reduce duplication and make it easier to monitoring the evolution of the strategy. xi Findings Conclusions Recommendations LWA proposals for food policy research. EQ 3.3. “In what ways has FSP IL worked closely with other partners and stakeholders at the country, regional, or global level to effect more rapid policy change AUC CAADP focal points, and Africa Lead)? What opportunities exist to affect more rapid policy change with existing stakeholders or new stakeholders?” (for example ReSAKSS, Currently, the best source of information about how To date, the most reliable sources of information on the LOP: Build better evidence-based systems for joint FSP IL has worked with some of its key regional FSP IL’s collaboration with its major regional partners planning between the component teams and the most partners is its annual reports. It is also clear from the (e.g., ReSAKSS, the AUC CAADP focal points, and important USAID-funded and supported regional reports that over the previous three years, certain Africa Lead) are the FSP IL’s annual reports and the 11 initiatives like Africa Lead, ReSAKSS, and ReNAPRI. senior staff and institutions have focused on some component teams’ presentations at the all-FSP IL specific regional groups more than others. FSP IL’s meeting in March 2017. This involvement was extensive collaboration with its key regional partners is not and needs to be better profiled and tracked in the tracked by the FSP IL M&E system, nor by any of the AA project’s PMP and reports. or Buy-in tracking systems. The only source of this data is the annual reports, which makes it hard to integrate these activities into the project’s strategic planning for the LOP. Without better data on what types of collaboration are occurring with these different groups, it is hard to strengthen the collaboration with existing regional and global stakeholders. This is a missed opportunity since these regional organizations are the principle organizations tasked with backstopping the national governments on policy review and reform. EQ 3.4. “What MSU/IFPRI/UP FSP IL organizational models appear to be more or less effective in supporting policy change at the country level? 3.4.a) Is a separate country office more or less effective than an office integrated with a host country institution? 3.4.b) Is an expatriate, local, or mixed team most effective in credible policy analysis and dialogue? 3.4.c) How can these organizational models be strengthened to better support policy change?” Most country programs have started with one of two The chief advantage of being fully embedded is that it Future LWA: As a policy-oriented LWA matures and basic models in terms of their relationships with their increases the likelihood that the government will own where the circumstances allow, it should move lead host-country government partner. The first is one the initial results of the FSP IL component teams as well toward forming the hybrid organizational model and in which the FSP IL activities, and the activities of the AA as the activities of any expansion of these activities to register as a type of unit that has some financial and Buy-in awards that they support, are literally through an AA or Buy-in. autonomy. embedded in a host government institution (Tanzania, Malawi, Zambia, and South Africa). A second model is one in which FSP IL attempted to have a more independent project identity from the start (Nigeria, Senegal, Myanmar, Mali, and Rwanda). Currently, many At the same time, this high degree of embeddedness can be a disadvantage if there is a history of regional or political division in a country where aligning a program with the government reduces the researchers’ willingness and ability to conduct unbiased, rigorous, In addition to lead agricultural ministries as principal policy partners, encourage future LWA to work with policy research institutes that are affiliated with national agricultural universities to be their principal local research and outreach partners unless there is a 11 One USAID reviewer suggested that the USAID Collaboration, Learning, and Adapting (CLA) Framework could be a useful tool for carrying out this recommendation. USAID LEARN. 2016. Collaborating, Learning, and Adapting (CLA) Framework and Maturity Matrix Overview. Washington, DC: USAID (October 2016). https://usaidlearninglab.org/library/collaborating percent2C-learning percent2C-and-adapting-cla-framework-and-maturity-matrix-overview. xii Findings Conclusions Recommendations of the country programs that have been most successful in generating active stakeholder engagement in policy change are moving toward a more hybrid organizational model in which: ● The project has certain activities and staff embedded under its principal local government partner; and ● Other activities that are more independent, where the project has some degree of autonomy from the larger host institution while at the same time maintaining its embeddedness with their original partner and also adding new partners. About 60 percent of the respondents to the online survey question on this topic felt that a mixed team that included both expatriate and national staff was the most effective. This sentiment echoed the strong preference for mixed teams that was expressed in the case-study countries. evidence-based policy research and analysis. It also makes it difficult for the project to be seen as unbiased when it is time to disseminate the findings. A mixed-program team that links local experts with individuals from strong institutions like IFPRI, MSU, and UP can open the door to strong two-way learning that benefits both sets of partners and wider acceptance of the research findings by the national government as well as the larger donor community that supports them. political reason not to do so. EQ Cluster 4: Management and Implementation EQ 4.1. “How does FSP IL define its customers, and who are they (Missions, BFS, ministries)? To what extent is FSP IL meeting or not meeting customer needs and why?” The FSP IL was designed to create a need and to strengthen the country, regional and global-level systems for responding to that need in the FTF countries. Its principal customers are the USAID bilateral missions, lead ministries of agriculture, and USAID/BFS. These customers are agencies who have a vested interest in facilitating food security policy review and revision. There is clear evidence from the online survey, the document review, and the case studies that: ● FSP IL has been successful in creating high levels of demand for evidence-based research in a variety of the countries where it has focused those activities; and ● This demand has generated a new set of demands for FSP IL and Mission-supported or country, regional, and global-level policy research, analysis, The increased demand for FSP IL evidence-based policy research and advocacy in the nine countries where the FSP IL is most active has created national, regional, and international actors with very different demands and expectations for FSP IL and Mission-supported research, analysis, and advocacy activities. LOP: Build better systems for tracking the shifts in stakeholder demand into the routine FSP IL and Mission-funded AA and Buy-in M&E processes. xiii Findings Conclusions Recommendations and advocacy. EQ 4.2. “What are some of the strengths and weaknesses of the following components and in what ways can the following be improved: 2.a) Management and coordination; 2.b) Funding arrangements; and 2.c) M&E?” A major strength of the current decentralized￾management model is its flexibility, which allows FSP IL to respond quickly and efficiently to emerging needs and opportunities in the target countries where it works. The team found that while this model was appropriate when the FSP IL was just starting, it is less appropriate for the LOP when the FSP IL is shifting gears to develop a more long-term plan for sustaining its achievements. Three issues raised in the online survey and interviews that were not raised in the FSP IL internal mid-term review12 were: ● The negative impact that the current system of single-year budgeting has on the component team work plans and collaboration with their country￾level partners. ● The weak definition of the role of the component team leaders in the global hierarchy of the project, which makes it difficult for these leaders to play their critical role in determining the appropriate balance of research, capacity building, and policy advocacy for all of the activities being supported by the FSP IL component teams. ● Although the current systems for funding the FSP IL’s collaborative activities in the countries where it is engaged are not ideal, they work. One problem is that, in some cases, the staff, particularly the field staff, did not initially receive adequate training in the financial rules and regulations. As a result, some of the expenses which they thought were justified, were rejected as inadmissible, which created delays or holes in reimbursement and support for program activities. Shifting from single-year budgets to multi-year work plans and budgets and better defining the role, time commitment, and responsibilities (reporting, M&E, strategic planning, and coordination) of the component team leads would contribute to more effective and efficient planning of component team activities. This would increase FSP IL’s ability to make significant contributions toward policy change at the global, regional, and country level by January 2020. LOP: ● Allow for multi-year work plans with associated budgets, but have actual budgets allocated annually based on a formal progress review. ● Better define the management roles of the component team leaders and others that play a critical role in determining an appropriate balance of research, capacity strengthening, and policy advocacy. ● Develop a manual of simple financial guidelines for AAs and Buy-ins, and hold regular financial training sessions for all their staff, including field staff. Future LWA: ● Budget a start-up meeting at the beginning of the project to clearly articulate the project’s goals, objectives, and financial procedures and discuss the purpose of the proposed M&E system to all staff and faculty working with or in conjunction with the IL. ● Budget an annual meeting to ensure that coordination and communications remain strong between the consortium partners and any Mission-funded projects that develop. 12 This internal mid-term evaluation included numerous recommendations for improving the FSP IL’s management structure that are already being executed by the FSP IL. Frank Young. 2016. Independent Review of Food Security Policy Innovation Lab: Internal Mechanisms and Processes and External Linkages. East Lansing: Michigan State University (December 11, 2016). xiv Findings Conclusions Recommendations There is widespread consensus among the partners— especially senior staff with experience in M&E and reporting—that the FSP IL M&E system has many strengths. The indicators were developed in close collaboration with all of the major FSP IL partners and USAID; they are SMART—specific, measurable, accurate, realistic, and time bound—and comply with the highest standards of M&E for policy projects. However, the PMP does not include: 1) any indicators that track FSP IL’s impact on its key institutional partners at the country level; or 2) indicators that track FSP IL’s coordination with the major regional organizations that support evidence-based policy reform. The indicators in the FSP IL PMP are not being uniformly used by the Mission-funded programs, which limits the utility of the data for joint planning with the Missions and strategy development. There is no full-time M&E specialist, although the current FSP IL manager is an experienced M&E and impact assessment specialist with over 10 years of experience in the World Bank’s Consultative Group on International Agricultural Research (CGIAR) system. The fact that only two of the 17 current indicators in the FSP IL PMP are included in all of the Mission-funded projects’ PMPs (except in Myanmar) limits the utility of the data for strategic planning and coordination. LOP: ● Encourage AAs and Buy-ins to collaborate with FSP IL in conducting a country-level analysis of its existing outcome and output indicators for capacity strengthening for all of its major partner institutions. ● Develop a simple capacity self-assessment tool that the FSP IL global teams and country programs can use for tracking capacity building for specific institutions and/or units in order to promote evidence-based joint planning. ● Develop a simple self-assessment tool that can be used to track the FSP IL’s coordination with some of the most important regional Feed the Future and USAID initiatives that support evidence-based food policy (like ReSAKSS, ReNAPRI, and Africa Lead). ● Strengthen the capacity of the FSP IL and AA/Buy-in staff responsible for the design and execution of the M&E systems to support this new system of joint reporting on their individual and co-sponsored capacity-building activities. Future LWA: Future LWA cooperative agreements for food policy need to require all Mission-funded AAs and Buy-ins to adopt a core set of outcome and output indicators in addition to any Mission-specific indicators they might be required to adopt. 1 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE In contrast to the conventional USAID-funded Innovation Labs (ILs), which focus only on research, the Feed the Future Innovation Lab for Food Security Policy (FSP IL) undertakes research, analysis, and capacity building in a context where influence on concrete policy changes is one of the expected outcomes of the FSP IL investments. These expected outcomes are formulated as the two Feed the Future sub-intermediate results (Sub-IRs) and a Sub-Sub IR (see Figure 1), which include: ● ● ● Sub-IR 1.3: Better policy environment (inclusive and stable incentives and opportunities, strengthened regulatory and enforcement infrastructure); Sub-IR 1.1: Enhanced institutional capacity development; and Sub-Sub-IR 1.3: Better policy formulation process (inclusive, stable, timely, transparent, evidence￾based, and mutually accountable). The purpose of the FSP IL mid-term performance evaluation is to assess: ● Whether FSP IL is on track to achieve its intended results, as described above; and ● What, if any, adjustments are needed to improve implementation and better achieve project outcomes. Because a significant part of FSP IL is related to country-level policy analysis and capacity building, the mid-term evaluation focuses on assessing results at the country level, including where Missions are funding Associate Awards (AAs) or Buy-ins. The expression of interest (EOI), which defined the scope of the evaluation, emphasized that although the AAs and Buy-ins are not to be evaluated since they are Mission programs, they must be taken into account when evaluating the core-funded in-country work.13 Although the evaluation focuses on FSP IL’s progress toward the achievement of its Sub-IRs and Sub-Sub IR, its findings and recommendations will be positioned to: 1) maximize the potential contribution toward intermediate results (IRs) and first-level objectives (FLOs) at higher levels in the results framework; and 2) extrapolate important lessons learned for future USAID/Bureau for Food Security (BFS) initiatives designed to strengthen the national, regional, and global policy environment for agriculture, nutrition, and food security in countries where USAID has Feed the Future projects. 1.2 AUDIENCE The primary audience and intended users for the FSP IL evaluation are the USAID/BFS/Agricultural Research Policy (ARP)/Policy Division and the BFS/ARP IL management team; however, Missions with ongoing or planned AAs or Buy-ins are also intended users, as is the FSP IL leadership team. The evaluation is being undertaken as one piece of a larger set of evaluation reviews of leading activities and approaches in the BFS/ARP/Policy Division portfolio that are expected to inform future USAID/BFS strategic planning and program development, including two internal mid-term evaluation studies. 1.3 EVALUATION QUESTIONS Only two minor changes14 were proposed to the 12 evaluation questions (EQs) that were identified in the EOI. The final revised questions are in Table 1. 13 USAID Bureau of Food Security. 2017. Expression of Interest—Performance Evaluation. Feed the Future Innovation. Bethesda, Maryland: Mendez England & Associates for USAID/BFS. Pg. 6. 14 These changes included replacing the original request for the evaluation to assess the research as part of the management and implementation section with a section on monitoring and evaluation (M&E) (Cluster 4, Table 2). 2 Table 1. FSP IL Mid-Term EQs and EQ Clusters Cluster 1: Policy Analysis and Research 1. In what ways, if any, has the selection and conduct of policy analysis and research influenced policy review, formulation, or change by host country governments? 2. What areas of policy analysis and research, if any, need more attention or should be included to bring about outcomes related to a better policy environment? Cluster 2: Capacity Strengthening 1. How effective has the coordination of capacity strengthening activities been among the FSP IL components, FSP IL partners, USAID Missions, and related entities at the country level?15 In what ways could this be improved? 2. How effective has FSP IL been in the organization and implementation of capacity strengthening activities covering different content such as policy research, analysis, strategy, and communication as well as providing advice to improve the use of evidence to advocate for policy change at the following levels: ● ● ● ● Country (including ministers, parliamentarians, and other host government staff engaged in the policy process); Organization (including farmer groups, non-governmental organizations [NGOs], and other civil society groups); Individual (including journalists); and Policy development process (which engages and coordinates each or several of these levels)?16 3. In what ways could the capacity strengthening component be improved for greater policy impact—including support to local stakeholder groups?17 4. What capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities? What types of synergies, if any, have there been among the varied Michigan State University (MSU), International Food Policy Research Institute (IFPRI), and University of Pretoria (UP) capacity strengthening approaches? In what ways can each partner’s capacity strengthening approaches be improved? Cluster 3: Progress Toward Policy Outcomes 1. To what extent have the five different activity components (Figure 1) contributed toward policy change at the global, regional, and country levels? In what ways could these components be expanded or improved to better support policy influence or policy change? What is country-level policy change/influence and how could this be expanded? How have C3, C4, and C5 contributed to more effective or rapid policy change at the country level? 2. How effectively has FSP IL communicated policy analysis, recommendations, and options to policymakers or others in a position to influence them, and in what ways could this communication be improved (including content, target, etc.)? 3. In what ways has FSP IL worked closely with other partners and stakeholders at the country, regional, or global level to effect more rapid policy change (for example, Regional Strategic Analysis and Knowledge Support System [ReSAKSS], African Union Commission’s Comprehensive Africa Agriculture Development Program [AUC CAADP] focal points, and Africa Lead)? What opportunities exist to affect more rapid policy change with existing stakeholders or new stakeholders? 4. What MSU/IFPRI/UP FSP IL organizational models appear to be more or less effective in supporting policy change at the country level? 4.a. Is a separate country office more or less effective than an office integrated with a host country institution? 4.b. Is an expatriate, local, or mixed team most effective in credible policy analysis and dialogue? 4.c. How can these organizational models be strengthened to better support policy change? 15 Such as Africa Lead, which supports capacity building under the Feed the Future Initiative and the African Union Commission’s Comprehensive Africa Agriculture Development Program (AUC CAADP); Alliance for a Green Revolution in Africa (AGRA); Food and Agriculture Organization of the United Nations/Monitoring and Analyzing Food and Agricultural Policies Program (FAO/MAFAP); Regional Strategic Analysis and Knowledge Support System (ReSAKSS); Futures Agriculture Consortium; International Fertilizer Development Center (IFDC); or other policy analysis groups. 16 Some reviewers noted that the correct formulation of this EQ should have been “How effective has FSP IL been in the organization and implementation of capacity strengthening activities covering different content such as policy research, analysis, strategy, and communication as well as providing advice to improve the use of evidence to advocate for policy change at the following levels: • Individuals who work at the country level (including ministers, parliamentarians, and other host government staff engaged in the policy process); • Individuals who work for organizations that support food policy at the national and local level [including farmer groups, non￾governmental organizations (NGOs), and other civil society groups]; and • Individuals in key positions of influence (such as journalists); as well as the global development process (which engage and coordinates each or several of these levels) in specific countries?” 17 Groups such as associations, farmer groups, NGOs, and other civil society groups, journalists, parliamentarians, and others. 3 Cluster 4: Management and Implementation 1. How does FSP IL define its customers, and who are they (Missions, BFS, ministries)? To what extent is FSP IL meeting or not meeting customer needs and why? 2. What are some of the strengths and weaknesses of the following components and in what ways can the following be improved: 2.a. Management and coordination? 2.b. Funding arrangements? 2.c. M&E? Source: USAID/BFS. 2016. EOI—Performance Evaluation. Feed the Future Innovation Lab for Food Security. 2.0 PROJECT BACKGROUND 2.1 MECHANISM DESCRIPTION 2.1.1 Context The country, regional, and global policy environment for agriculture, nutrition, and food security is a major determinant of the success and sustainability of more local or project-level investments in these areas. This was recognized early in Feed the Future, when interagency reviews of the initial country and regional strategies identified little or no engagement at the policy level or in policy dialogue with host governments and institutions. The result was a determination to: engage explicitly in policy dialogue, analysis, and capacity strengthening of policy partners; and task every Feed the Future focus-country Mission (except Haiti) to do the same. FSP IL18 was designed shortly after the determination that an explicit focus on food security policy is an integral factor in the success of Feed the Future project investments. In the FSP IL design stage, a series of developments highlighted—and indeed increased—demand for FSP IL engagement and outputs, resulting in a major increase in the project ceiling. FSP IL is implemented by Michigan State University (MSU), in collaboration with the International Food Policy Research Institute (IFPRI) and University of Pretoria (UP). The project’s expected end date is January 14, 2020. 2.1.2 Goals and Objectives The overall goal of FSP IL is to promote inclusive agricultural productivity growth, improved nutritional outcomes, and enhanced livelihood resilience for men and women through improved policy environments. This goal is to be achieved by fostering credible inclusive, transparent, and sustainable policy processes at country and regional levels, and filling critical policy gaps. The results framework describes the pathway that FSP IL uses to achieve its development goals and objectives (Figure 1). 18 The official title of the award is “USAID’s Global Hunger and Food Security Research Strategy: Climate Resilience, Nutrition, and Policy – Feed the Future Innovation Lab for Food Security Policy (FSP IL).” 4 Figure 1. Results Framework for the FSP IL ● ● ● ● ● Source: USAID/BFS. 2016. EOI—Performance Evaluation. Feed the Future Innovation Lab for Food Security. Since FSP IL started, it has adopted a consistent activity work program, organized into five components:19 Component 1 (C1): Country/regional-level collaborative research (on farms, firms, and markets) and formulation/analysis of policy options; Component 2 (C2): Country/regional-level capacity building for policy (data, analysis, advocacy, formulation, consultation, coordination, and implementation); Component 3 (C3): Global collaborative research on support to the policy process and policy capacity; Component 4 (C4): Engagement in global policy debates on food and nutrition security; and Component 5 (C5): Strategic analytical agenda and support to donor policy and strategy. 19 In the initial award, the role of the component teams was described as Component 1 (C1): Field-Level Collaborative Research (on Farms, Firms, and Markets) and Formulation/Analysis of Policy Options; Component 2 (C2): Capacity-Building for Policy (Data, Analysis, Advocacy, Formulation, Consultation, Coordination, and Implementation); Component 3 (C3): Global Collaborative Research on Support to the Policy Process and Policy Capacity; Component 4 (C4): Engagement in Global Policy Debates on Food Security; and Component 5 (C5): Strategic Analytical Agenda and Support to Donor Policy and Strategy. The description used in the EOI used the term country/regional-level for C1 and C2 but was otherwise the same as the one listed here. The description of the component teams that is used in this document is taken from the most recent annual report. FSP-IL Annual Report: October 1, 2015 – September 30, 2016. (Award AID-OAA-L-13-00001). Revised version: February 15, 2017. Department of Agriculture, Food and Resource Economics, Michigan State University, East Lansing MI 48840. http://foodsecuritypolicy.msu.edu/uploads/files/About/Project_Reports/FSP_Workplan-Oct2016Sept2017.pdf 5 The above components are designed to achieve five strategic results (SRs). The SRs are essential to the achievement of the two Feed the Future Sub-IRs, which are critical to the project’s attainment of five IRs and two FLOs (Figure 1). During the project’s first year the decision was made to: ● Merge the C1 and C2 teams into one single C1/C2 team since FSP IL researchers decided the ground-level research and capacity building had to go hand-in-hand; and ● Sub-divide the C4 team into two sub-teams: - C4A: One global component team focused on “upstream” agricultural issues such as land and agricultural input (e.g., seed, fertilizer); and - C4B: A second global component team focused on “downstream” agricultural issues such as food processing, shifts in food consumption patterns, regional and national trade, and market information systems. 2.1.3 Target Areas and Target Population Groups FSP IL seeks to contribute to Feed the Future’s goal of significant reductions in poverty and hunger among poor people through facilitating government policy changes that expand inclusive economic growth and reduce malnutrition in the countries that have Feed the Future investments. Facilitating policy changes involves building the capacity of a wide range of actors in the formulation and implementation of policy, as well as providing direct support for advocacy and analysis. The actors that need capacity building include government ministries—including but not limited to the lead Ministries of Agriculture—as well as parliamentary committees, regional and global food-security research institutions, civil societies, farmers’ associations, private-sector lobby groups, think tanks, universities, and multiple donors and foundations involved in policy support or advocacy. A major focus of FSP IL is to influence global and regional policy and strategy through improved policy research and advocacy. The focus of the research to accomplish these ends is on farms, firms, and markets in the agriculture, food security, and nutrition space. 2.1.4 Program Implementation Plan FSP IL was expected to affect the productivity and impact of Feed the Future interventions through activities in a sub-set of countries (through C1 and C2) as well as the lessons learned from its global research work (through C3 and C4), combined with short-term, on-demand, strategic analytical support to the USAID/BFS staff (through C5). During the first year, FSP IL implemented activities to support country-level policy analysis and capacity building in six countries—Nigeria, Mali, Malawi, Tanzania, Senegal, and Myanmar—and global-level research and policy engagement in Myanmar. In subsequent years, the FSP IL Leader with Associate (LWA)20 award was amended five times and received Mission support through six Mission AAs and two Mission Buy-ins (Table 2, Map 1, next page). The total estimated cost (TEC) for the FSP IL LWA award is $70 million over the life of the grant. Toward this TEC, the total estimated amount of commitment received by FSP IL, aggregated across all three channels of funding—core, AA, and Buy-in—is $58.6 million. These combined initiatives currently support activities in 12 countries in Africa and two in Asia (Table 2, Map 1). Each of the Mission-funded AAs: 20 The issuance of a LWA award covers a specified worldwide activity. The LWA award includes language that allows a Mission or other office to make one or more separate awards, called AAs, to the LWA award recipient without using restricted eligibility. The AA must be within the terms and scope of the program description of the LWA award and support a distinct local or regional activity. LWA awards are not Indefinite Delivery Indefinite Quantity (IDIQ) contracts that are used in acquisition nor use any of the procedures under an IDIQ. As examples, a LWA award must have a program description and a budget with sufficient funds to carry out the program description in the LWA award. AAs are not made using fair opportunity procedures (ADS Chapter 303.3.26, Grants and Cooperative Agreements to Non-Governmental Organizations), and, in contrast, a Buy-in award refers to Mission funding into the LWA or centrally managed AAs. 6 ● ● Is owned by the Mission that decided to fund it and is implemented as part of a larger Mission Feed the Future portfolio; and Has a separate set of indicators that include some of the FSP IL indicators, as well as a number of standard Feed the Future indicators, which are tracked as part of their routine Performance Monitoring Plans (PMPs). At the same time, FSP IL functions as a coherent $70 million program in which the country programs funded by the Missions were designed and implemented in a larger framework and with many of the same staff that work on the core USAID/BFS-funded activities. Thus, the country programs—funded by the AAs and Buy-ins—already benefit from and provide benefits to the core-funded activities in major ways. Table 2. Countries with Activities Supported Under the FSP IL Core, AAs, and Buy-ins Countries/Regions Where At Least One FSP IL-Funded Activity Has Taken Place to Date (1/2017) Contract with USAID/Washington (7/15/2013-1/14/2020) Contracts with USAID Core Activities and Country Missions: AAs Components Intensity21 Buy-ins22 1. Bangladesh X (C3, C4A*) 2 2. Ethiopia X (C4B) 2 3. Ghana X (C3, C4A) 2 4. Malawi X (C1/C2, C3, C4A, C4B) 4 X (11/24/2014-11/24/2017) 5. Mali X (C1/C2, C4B) 3 X (2/17/2016-2/15/2021) 6. Mozambique X (C1/C2, C4B) 2 7. Myanmar X (C1/C2) 5 X (9/24/2014-9/23/2019) 8. Nigeria X (C4A, C4B) 3 X (7/1/2015-6/30/2020) 9. Burundi23 N/A 0 10. Rwanda*24 N/A 0 X (7/20/2015-7/19/2018) 11. Senegal X (C4A, C4B) 3 X (7/27/2015-7/26/2018) 12. South Africa X (C3) 2 13. Tanzania X (C1/C2, C3, C4A, C4B) 5 X 14. Zambia X (C1/C2, C3, C4A) 4 X West Africa region X25 China26 X (C4A*) 1 India X (C4A*) 1 Kenya X (C4A) 1 Nepal X (C4A*) 1 21 Subjective grades on intensity scale: 1 (low intensity) to 5 (high intensity) based on evaluator discussions with FSP IL management team. January 16-18, 2017. 22 Support through Mission Buy-in added over the years does not necessarily correspond with the overall contract start-end date. 23 As of January 2017, FSP IL has not planned or initiated activities in Burundi. 24 Neither Burundi nor Rwanda has core funding. Rwanda, however, has intensive activity through the AA. See Column 5. 25 Funding is through the West Africa Regional Bureau to specifically look at regional issues. It is difficult to identify this project with specific countries. 26 The four countries with an intensity level of 1 are countries where the FSP IL has supported a variety of ad-hoc activities, many of which were relatively short in duration. They are not considered to be in the main sample of countries where the project has been active as of January 2017. For this reason, they are not counted as mainstream FSP IL countries. 7 Source: FSP IL, based on FSP IL Mid-Term Performance Evaluation team discussion with MSU FSP IL management team. East Lansing, Michigan: MSU for FSP IL. January 17, 2017. *The C4A activities in Bangladesh, China, India, or Nepal is built mostly on mechanization work that IFPRI is doing, with little expenditure of FSP IL C4A resources. Map 1. Locations of Key Capacity Building/Analysis/Research Sites, Including Global and Regional Interventions Source: FSP IL, based on discussions with the FSP IL Mid-Term Performance Evaluation team. East Lansing, Michigan: MSU for FSP IL; January 18, 2017. 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY Based on its initial understanding of the project, the evaluation team (ET) adopted a mixed-methods approach to assess the performance of FSP IL pertaining to the EQs, especially in terms of how the issues broached in the EQs affected the achievements of the FSP IL’s principal outcomes at the country level. This approach included qualitative and quantitative methods. 3.1.1 Qualitative Methods The ET used a series of key informant interviews (KIIs) and focus group discussions (FGDs) to collect qualitative data from FSP IL core staff, USAID/BFS field and headquarter staff, and local stakeholders in two case-study countries—Tanzania and Myanmar. These interviews were conducted through telephone, Skype, and field visits (Table 3). In an effort to standardize the data collection between informant groups in the different interview sites, the team developed a list of interview guides that had certain questions overlapping with the questions in the team’s online survey instrument.27 Altogether, the team interviewed 260 people by conducting 47 KIIs and 34 FGDs (Table 3). 27 These survey instruments were included in the revised evaluation protocol that was submitted to USAID on March 19, 2017. A series of steps were followed in the design and execution of the qualitative interviews to try to guarantee that the interviews were as independent as possible and to protect the identity of the people being interviewed. 8 Table 3. Sources of Qualitative and Quantitative Data for the Seven Categories of FSP IL Stakeholders in the Mid-Term Performance Evaluation Category of Stakeholder Country Case Studies In Conjunction with the All-FSP IL Meeting in Washington, DC Six-Country Online Survey Tanzania Myanmar FSP IL Core Staff 28 USAID Respondents Sample Size of Stakeholder Category Category 1: FSP IL staff 2 4 38 7 (50%) 14 Category 2: AA and Buy-in staff 5 12 4 (57%) 7 Category 3: Local partners 3 0 8 (53%) 15 University-based partners 4 (57%) 7 Civil society (NGOs/private) 4 (50%) 8 Category 4: Local stakeholders 55 109 9 (32%) 28 4.a. Local government (7)29 (0) 4.b. Community-level stakeholders (13) (68) 4.c. University-based stakeholders (8) (4) 2 (67%) 3 4.d. Government stakeholders (5) (19) 2 (17%) 12 4.e. Civil society (NGOs/private) (10) (18) 3 (33%) 9 4.f. Other donor-funded projects (1230) See 5c 2 (50%) 4 Category 5. USAID 6 (60%) 10 5.a. USAID/BFS 9 5.b. USAID Mission 2 1 6 (60%) 10 5.c. Other donors that co￾fund31 0 932 5.d. USAID-funded projects (total) 6 2 Categories 6 and 7. FSP IL and AA staff involved in management 12 (75%) 16 FSP IL staff 8 (89%) 9 AA/Buy-in staff 4 (57%) 7 Total survey 47 (52%) 90 Actual interviews (#) 17 41 15 8 KII (#) 5 2933 5 8 28 Number is based on interviews at the all-FSP IL meeting. Additional follow-up interviews were conducted with many staff. To avoid double counting, these follow-up interviews are not counted here. 29 These numbers indicate the number of people interviewed in each category of local stakeholders (i.e., seven representatives of local government were interviewed as part of the 55 local stakeholders who were interviewed. 30 In conjunction with the monthly Policy Analysis Group (PAG) meeting in Dar es Salaam, May 2017. 31 Leveraged funding sources. 32 Seven from staff and contractors associated with the Livelihoods and Food Security Trust Fund (LIFT) multi-donor trust fund and three from Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), European Union (EU), and Department for International Development (DFID) that support FSP IL /FSP AA through the LIFT multi-donor trust fund. 33 In contrast to Tanzania, there was a strong preference in Myanmar for KIIs over FGDs, both in terms of privacy and logistics. 9 Category of Stakeholder Country Case Studies In Conjunction with the All-FSP IL Meeting in Washington, DC Six-Country Online Survey Tanzania Myanmar FSP IL Core Staff 28 USAID Respondents Sample Size of Stakeholder Category FGD (#) 12 12 10 0 Total people interviewed 73 137 38 12 Summary 260 interviewed in KII and FGD vs. 198 anticipated 47 (52%) respondents out of sample size of 90 Source: FSP IL Mid-Term Performance Evaluation. 3.1.2 Quantitative Methods In an effort to provide a quantitative cross-check of the qualitative data, the evaluators proposed a six￾country online survey. It was anticipated that the survey would be completed by five categories of FSP IL stakeholders (later expanded to seven) in six of the 14 countries where FSP IL has worked—Tanzania, Myanmar, Malawi, Nigeria, Zambia, and South Africa. The six countries represented a reasoned choice from three broad categories of FSP IL countries identified during the design of the evaluation protocol:34 ● ● ● Category 1: Countries that have had the highest level of activity and policy influence—Myanmar and Tanzania (intensity ranking 5, Table 2); Category 2: Countries where the activities started later or where it is not yet possible to see extensive policy impact—Senegal, Mali, Nigeria, Malawi, and Zambia (intensity ranking 3-4); and Category 3: Countries where most of FSP IL activities have been global research and/or where the country-level research and capacity building have been more recent—Bangladesh, Ethiopia, Mozambique, Rwanda, Ghana, and South Africa (intensity ranking 2). The ET proposed using a seven-step process35 for the execution of the online survey in the evaluation protocol. During the initial pilot test, it was discovered that individuals who were identified for the pilot test in two countries not included in the broader survey were confused by the EQs, which referred to the formal FSP IL activities. Even the two chiefs of party (COPs) that took the survey were primarily familiar with their own AAs and Buy-ins, and had little, if any, understanding of how these activities were linked to the FSP IL Leader Award. Based on this information, the evaluators: ● ● Simplified the survey and launched a second pilot test, which precipitated a second major revision of the stakeholder questionnaire to further simplify it; and Worked in close collaboration with the COPs in each of the six countries to develop a: - - Detailed stakeholder list for the six countries that were targeted by the survey; and Cover letter from the COP explaining the connection between FSP IL and the AA/Buy-in project with which they were familiar. This revised stakeholder list, 90 stakeholders36 from the six countries—Tanzania (15), Myanmar (16), Malawi (15), Nigeria (16), Zambia (14), and South Africa (14)—was selected from a stratified random 34 These categories are based on the level of intensity of FSP IL activities in the country (Table 2). 35 Della McMillan and James Seale. 2017. Evaluation Protocol. Mid-Term Performance Evaluation of FSP IL. Bethesda, MD: Program Evaluation for Effectiveness and Learning (PEEL) for USAID. Pg. 15. 36 In the original protocol, the team had expected to focus the online survey on FSP IL, Mission-funded AA and Buy-in staff, and staff associated with the USAID bilateral Missions chosen for the survey because these were the groups that were expected to be most willing to respond. After the protocol, the evaluators decided to also include at least four national stakeholders in each country. In fact, the response rate from 10 sample of the master stakeholder lists that the ET developed based on interviews with the FSP IL administration and staff in each of these counties. The invitations to take the survey were sent on May 3, 2017, through the Qualtrics Software email system. A reminder through the Qualtrics Software email system was sent to all non-respondents on May 10, 2017. It was learned thereafter that most of these emails with the survey instrument and the reminders did not reach the intended respondents, but were treated as junk mail. A second mailing of the survey instrument was necessary and was sent through one of the ET’s email accounts. That emailing on June 22-24 increased the number of respondents from a total of 19 on June 22 to 47 responses (i.e., a response rate of 52 percent [47/90] of the full stakeholder list or 76 percent [47/62] response rate for the original stakeholder list minus the national stakeholders; see footnote 34 above). 3.1.3 Other Data Sources The information from the KIIs, FGDs, and online survey was complemented by: ● ● A review of the FSP IL, AA/Buy-in literature and secondary data sources identified by FSP IL AA/Buy-in and stakeholder partners that focused on the two case-study countries of Myanmar and Tanzania and, to a lesser degree, on the four countries targeted by the online questionnaire (Ghana, Zambia, Nigeria, and Malawi); and A re-analysis of some of the existing FSP IL and AA/Buy-in monitoring and evaluation (M&E) data sets for Malawi and Myanmar. 3.2 LIMITATIONS There were several limitations for this evaluation. They include: 1. Focus on the two case-study countries of Tanzania and Myanmar. The ET attempted to minimize the impact of this limitation on the evaluation’s independence by combining the case studies with an online questionnaire administered to a stratified random sample of staff and stakeholders in six non-case-study countries. 2. Reliance on the FSP IL Management Office staff to coordinate with the senior staff in the Mission-funded AA and Buy-in projects in developing the country stakeholder lists, which were then used to draw the sample for the online questionnaire. The ET sought to minimize this limitation by reviewing the draft stakeholder lists with multiple staff to make sure they were complete, and by making a random choice of the staff and stakeholders. 3. Inability to clearly differentiate policies influenced by FSP IL and those influenced by the Mission￾funded AAs or Buy-ins. In almost all cases, FSP IL’s research and capacity strengthening are only part of a much broader effort by national governments and their regional and international partners. It is for this reason that the EQs ask the evaluation to focus on FSP IL’s contribution to these processes rather than its policy “impact.” 4. The ET was unable to make a campus visit to MSU nor had any phone interviews with any business management staff to get MSU’s perspective on this large and complex LWA. To address this issue, MSU made available two of its senior administrators to be interviewed by the evaluators during the FSP IL meeting in Washington, DC. 5. Response rate of the online survey was 52 percent (47 out of a sample size of 90). Of those that completed the online survey, 51 percent of the respondents were either FSP IL staff (34 percent) or AA/Buy-in staff (17 percent), while the other 49 percent consisted of FSP IL partners (17 percent), stakeholders (19 percent), and USAID Mission staff (13 percent). Of the nine responding stakeholders, only two were government staff. It should be noted that the FSP IL and AA/Buy-in staff were in certain cases less positive in their responses about policy influence than the other respondents (partners, stakeholders, and Mission staff). In anticipation of this issue, the evaluation design included two intensive case studies, which allowed the stakeholders was low (2 [17 percent] of the 12 invited government staff completed the survey; 7 [44 percent] of the 16 other nongovernment stakeholders completed the survey). 11 evaluators to have better access to the policy research “users” as well as those generating the research. 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 POLICY ANALYSIS AND RESEARCH (EQ CLUSTER #1) 4.1.1 Findings 4.1.1.1 EQ 1.1: In what ways, if any, has the selection and conduct of policy analysis and research influenced policy review, formulation or change by host country governments? Stakeholder Involvement in the Selection and Conduct of the Studies: Several questions on the online survey dealt with the selection, design, or conduct of the FSP IL research topics or policy analyses.37 A careful reading of the respondents’ comments indicates that most FSP IL component teams started with a list of thematic areas endorsed by USAID/BFS and the USAID country Missions. From these thematic areas, research topics and policy analyses were selected by the three consortium members jointly. Each member of a component team then used their in-country contacts and previous in-country experience to select topics and activities within these areas. This participatory selection process was achieved through one-on-one meetings with stakeholders and, in a few cases, through team workshops and stakeholder workshops.38 The exact process seems to have been highly influenced by the component leader’s and lead institution’s previous work and contacts in the country. Most staff noted that once the broad topic areas were agreed upon by MSU, IFPRI, and USAID, each partner designed the research that they wanted to do. It is noteworthy that this information was corroborated by the FSP IL and AA/Buy-in staff during the FGD and KIIs that the ET conducted in Washington, DC and the two case-study countries. All FSP IL studies were executed in close coordination with local food policy research institutes, many of them university based. Staff interviewed during the case studies noted that FSP IL was very different from many earlier USAID-funded projects in these countries, which tended to hire local experts as consultants. This decision to base the research in local institutes was made in order to provide them with the on-the-job training necessary to be policy actors, as outlined in the final technical proposal, which was attached to the cooperative agreement.39 Almost all of the initial studies, with the exception of Myanmar,40 involved junior staff from the lead ministries. Stakeholder Feedback on Whether the Studies were Supply or Demand Driven: The literature review and interviews show that each of the FSP IL component teams participated in two broad types of activities: 37 In response to a question on the selection of FSP IL research topics, 14 (70 percent) out of 20 respondents from stakeholder categories 1, 2, 3, and 6 indicated that FSP IL research topics were selected by the three consortium members jointly. Of these, 11 (55 percent) respondents wrote comments on how FSP IL made decisions on design of policy analysis and research (Annex III.A). Eleven (55 percent) of the same 20 respondents indicated that they were involved in selection of FSP IL research topics and wrote comments on their involvement (Annex III.B). Further, 15 (75 percent) of the same 20 respondents indicated that they were involved in the design or conduct of FSP IL research topics or policy analyses. In addition, 15 of the same 20 (75 percent) respondents stated that they were involved in the design or conduct of FSP IL research topics or policy analyses, and 11 (55 percent) of them wrote descriptions of how they were involved (Annex III.C.3). 38 From online survey and two-country case-study. 39 FSP IL Cooperative Agreement Proposal. 40 During the first two years of the FSP IL, staff from the Ministry of Agriculture were constrained from working with the project by the military government. 12 ● ● Supply driven, in that the impetus behind them was a set of research concerns generated by the USAID/BFS office or the personal experience of the researchers; and Demand driven, by the needs requested from the local government and field Missions. Although the concept of demand-driven research was one of the six core principles of FSP IL, some of the biggest success stories (with success defined in terms of policy influence) emerged when the local stakeholders agreed to support new diagnostic studies or to conduct in-country research on some of the topics that the USAID/BFS wanted FSP IL to examine on a comparative basis. One example of “demand driven” studies that were identified as having influenced national-level policy review and analysis by a wide range of national government, international donors, private sector, and NGO stakeholders was the four initial FSP IL-sponsored diagnostics studies in Myanmar, which (see Annex V.A): ● ● ● ● ● Provided government with its first evidence-based overview of major agricultural systems; Determined that a major constraint to smallholder incomes and agricultural growth was emphasis on paddy rice; Identified critically underfunded public goods such as research, extension, and agricultural statistics; Encouraged the opposition party (now in power) to include agriculture as a pillar in their manifesto; and Provided USAID/Burma with justification to make agriculture a major component of the Mission’s portfolio. Something similar was observed in Nigeria where the FSP IL’s initial diagnostic studies cast doubt on the popular misconception that Nigeria was importing large amounts of processed chicken and instead identified the country’s rapidly growing smallholder poultry sector as an important topic that had been largely ignored by food policy research.41 A third example of demand-driven research that influenced national policy was the “upstream” land input and farm productivity study (C4A) and the “downstream” agri-food system transformation study (C4B) in Tanzania. According to project reports and some of the field-interviewed government and Sokoine University of Agriculture (SUA) staff that were interviewed in the field, the national government used the results from (Annex V.B.): ● ● FSP IL land studies to initiate dialogue on land-policy reforms and collaboration with partners in advocacy for land-policy reforms; and Agri-food system study to finalize the national agro-processing strategy in Tanzania. Six-Stage Participatory Process Used to Design and Execute the Studies: Whether the study was demand or supply driven, most of the senior FSP IL researchers generally followed the same six-step model:42 1. Identify conventional wisdoms underlying current policies and policy making/change processes; 2. Test those as hypotheses in field survey and empirical analysis; 3. Show government how empirical findings do or do not support the underpinning conventional wisdoms of their policies; 4. Strategize policy changes that fit the findings and the realities unearthed by field work; 5. Build medium-term strategy to form the enabling environment to implement the policy changes by extension of capacity building started in steps1-4 with collaborators, local institutions, government, and donors; and 41 Source: KII with senior FSP IL staff. 42This six-step model was identified by one of the senior FSP IL staff; it is not outlined in the cooperative agreement. When discussed with other staff, many of them agreed that this was the de facto model followed by most of the global component team. Their point was that it did not matter whether the research was demand or supply driven rather that the process and level of participation helped support the successful ownership of the results. 13 6. Track impact and identify where new changes make it necessary to repeat steps 1-5 again. Unfortunately, the ET found that, to date, very few of the research studies appear to be monitoring the impact of policies that they help inform. This is important in cases like Tanzania where some of the initial FSP IL research recommendations ended up having a negative impact on smallholders’ incomes and living standards.When the FSP IL Local Government Finance Act (LGFA) cess tax study recommended that the government was unlikely to eliminate some of the cess taxes on local produce crops unless they could increase the efficiency of the collection process on the lower cess taxes, the government adopted the recommendation for increasing the efficiency of the collection process but failed to lower the tax rate. Although FSP IL is still working with the government and other stakeholders to lower the tax, this partial adoption of the researchers’ recommendation is having a very negative effect on many low￾income farmers’ income and livelihoods. Indeed, the need for better monitoring of the long-term recommendations of FSP IL studies was one of the principal recommendations mentioned by the local stakeholder groups and many of the FSP IL Buy-in [Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES) Project] staff that the team interviewed in Tanzania. How Selection and Conduct Influenced Policy Review, Formulation, or Change: Most FSP IL studies included a large component of “stakeholder consultation” and “policy debate,” both prior to and after the studies.43 The ET was struck by FSP IL’s commitment to involving the local and regional government officials in the initial diagnostic studies and report-outs (Annexes III.A.1-3; Annexes V.A-2 and V.B). Analysis of the online survey shows that: ● ● ● Seventy-five percent of respondents from stakeholder categories 1, 2, and 6 answered in the affirmative to the question:“Did the design and conduct of the FSP IL support to policy analysis and research influence policy review?”44 Seventy percent of the same respondents answered in the affirmative to the question “Did the design and conduct of the FSP LAB support policy analysis and research influence policy formulation?”45 and Forty-five percent of the same respondents answered in the affirmative to the question “Did the design and conduct of the FSP LAB support policy analysis and research influence policy change?”46 Some of the online respondents identified ways that the selection and conduct processes of the project helped influence policy review and formulation (Text Boxes 1 and 2). The same responses from the online survey confirmed the ET’s observations in the case-study countries that the projects which were most successful provided a good foundation for participation and potential policy impact (Annex III.D): ● ● Built on the senior FSP IL researchers’ relationships with established research partners that were familiar with their previous research so they trusted the researchers’ recommendations to conduct an initial participatory review of the major policy assumption to determine which hypotheses needed more investigation; Involved staff from the key local government partners as equal research partners; 43 These stakeholder consultation and policy debate processes are illustrated for the case-study countries in Annexes V.A. and V.B. 44 Fifteen of the 20 respondents from stakeholder categories 1, 2, and 6 answered this question “yes,” two answered “no,” and three did not answer the question. 45 Fourteen of the 20 respondents from stakeholder categories 1, 2, and 6 answered this question “yes,” two answered “no,” one answered “Don’t know or N/A,” and three did not answer the question. 46 Nine of the 20 respondents from stakeholder categories 1, 2, and 6 answered this question “yes,” three answered “no,” three answered “Don’t know or N/A,” and five did not answer the question. 14 Text Box 1 keholder Feedback on the question: “Did the design and conduct of the FSP IL policy analysis research influence policy review?” “Absolutely yes in Tanzania...Lots of policy influence also in Malawi...In Nigeria policy influence is in a more nascent phase.” “An example that carried on from Food Security Research Project (3rd phase, from 2010 to 2015) (FSRP-III) is the increase in debates on the mode of delivery of the farmer input support program, from the traditional way to one based on the electronic vouchers systems.” “By continually reminding government of the list of conventions, agreements, and commitments to be included in [national] policy processes [and documents].” “Execution of C3 has impacted on the Malawi policy review that preceded drafting and pre-finalization of the Malawi National Agriculture and Food Security Investment Plan 2 (NAIP2).” “FSP IL support for policy analysis and research provided important knowledge [about] other countries’ experiences, which informed the policy makers reviewing policy.” “I cannot say well. So far nothing has started in Senegal except the very start of the surveys as that took a long time to start. For Nigeria, we are at the middle of chicken value chain surveys. For my part of Tanzania work we are just starting the surveys. For Myanmar, I personally had more effect after I left the project, as I started the fish work with [name] and then we put that into policy realm. But then I drafted the whole ag policy statement of the government for Nathan (basing a lot on what I had learned at start of Myanmar project) and I also worked with LIFT to get the main African Development Bank (ADB) grant to the government. These had results. I have no idea if the project after I started it had effect. The C4B food systems work, on policy? Who knows? I think this ‘direct and immediate effect on policy’ is not a good criterion. What really affects general policy is build-up of evidence and ‘zeitgeist’ over time and then that affects all. If one says there was some immediate effect on policy I am suspicious and doubtful as that is usually not how things work. It takes years.” “Research on crop cess and crop boards in Tanzania has led to the review of the Local Government Finance Act (LGFA) and the review of charges and levies for cash crops in the country.” “Some of the activities that I was involved [in] were directly used to inform the development of new government policies and strategies.” “Speaking for New Alliance Policy Acceleration Support—Malawi (NAPAS: Malawi), we were able to actively engage in supporting the government with policy analyses on their priority policy reforms. Review of existing policy was a component of this.” “The question does not make sense. If it is meant to say ‘Did FSP influence policy review,’ then yes. In Tanzania, there has been review of the crop cess as a result of MSU research.” “The three policy economy case studies [C3] are influencing design of the follow-on Malawi Agricultural Policy and Implementation (MAPI) project that USAID wants to fund as a follow-on to the current Malawi Associate Award (NAPAS: Malawi). It will most likely also influence USAID’s policy engagement in Malawi on several policies including the three-case studied: Farm Input Subsidy Progamme (FISP); Seed Policy; and Grain Marketing/Agricultural Development and Marketing Corporation (ADMARC) reforms.” “National consultations after the research were useful to get the message across to the policy makers.” “To the extent that FSP works closely with local policy units to liaise with governments on a daily basis, FSP has/can be effective in influencing policy review. But this is not always done. In some cases, members of the consortium have their own local field offices in the country that are perceived as competing with national policy research institutes. Where this happens, the influence on policy review is impeded.” “Yes, and vice versa.” Source: FSP IL online survey, Q20, respondents from stakeholder categories 1, 2, 3, and 6. Sta and ● ● ● ● ● ● ● ● ● ● ● ● ● ● 15 Text Box 2 “ASPIRES on several occasions has been requested to assist in policy analysis on different matters by the Ministry of Agriculture.” “Yes, several items in the country programs are meant to respond directly to policy makers’ technical needs. FSP support in Malawi helped in the design of the new agricultural policy. In Burma, FSP has worked with the Permanent Secretary to develop an agricultural strategy and white paper. Support has also been provided to USAID directly to help their programming and investments.” “Execution of C3 has impacted on the Malawi policy formulation as regards drafting of Malawi’s new Growth and Development Strategy and NAIP2.” “FSP IL support [for] policy analysis and research are likely to have provided greater assurance for policymakers regarding the roles of private sectors in providing the mechanization.” “It assisted in the preparation of the new National Food and Nutrition Security Plan.” “Same comments as above.” “Speaking for NAPAS: Malawi, policy and strategy formulation was a central component of the scope of work of the project.” “The change from the traditional way of implementing the farmer input support program in Zambia to one based on the electronic vouchers systems.” “The gender policy dialogue workshop was used to identify some oversights to Malawi’s Multi-sectoral Nutrition Policy 2016-2020. The dialogue report was used to inform the forthcoming strategy.” “The land research work (component C4A of the FSP research) that was presented at the 2016 and 2017 Malawi Land Symposiums (organized by the NAPAS: Malawi project) helped the Ministry of Land to decide to refine their Landlord Tenant Bill (going forward). It also helped define what research should be conducted on land in Malawi going forward). One research topic that stakeholders want to see conducted is the impact of the rise of medium sale farmers on productivity, employment, equity and other welfare gains.” “Preparation of new country level agricultural policy and development strategy.” “Same answer as above.” Source: FSP IL online survey, Q20, respondents from stakeholder categories 1, 2, 3, and 6. ● ● Stakeholder Feedback on the question: “Did the design and conduct of the FSP IL policy analysis and research influence policy formulation?” ● ● ● ● ● ● ● ● ● ● ● ● Developed mechanisms to continue the collaboration with these key government partners in the resulting outreach and strategy sessions with Mission-funded AAs/Buy-ins and other types of complementary donor funding; and Continued backstopping and monitoring of these in-country partners that were using the data to advocate for policy change. Some of the best evidence for how these processes worked is from the case-study countries (Annex III.D). 4.1.1.2 EQ 1.2: What area of policy analysis and research, if any, need more attention or should be included to bring about outcomes related to a better policy environment? Identification of New Research Activities. One strength of the LWA model that FSP IL adopted was its ability to identify new policy areas based on research and policy responses. Some of the best evidence of this is a comparison of the component work plans between years. This comparison showed that some lines of research morphed into new type of complementary research and capacity- 16 strengthening activities while others stopped and new research themes were started. Other new areas were identified by the respondents of the online survey, such as gender and climate change, integration of youth as a cross-cutting issue, and off-farm segments of the food value chain. Several senior staff interviewed argued that once there was an emerging demand from local stakeholders for evidence￾based research, the biggest problem was deciding how to select the topics with the highest potential for affecting policy change and not spinning off into too many directions. The Missing Topic: International Trade. The issue of better understanding international trade and its current and projected influence on food policy outcomes was identified by almost all the major categories of stakeholder respondents in Myanmar and Tanzania, even in the local villages. Although both projects included a focused program of downstream research, these studies tended to focus on: 1) post-market harvesting and processing and value-added technologies; and 2) the local and regional marketing issues associated with value chains.47 The impact of international trade rules, regulations, and standards was identified as the top issue by all of the private-sector producer groups and individual farmers in both case-study countries. It was also identified as a key issue by most of the government, NGO, and donor representatives that the team met in both countries, as well as by the people who expressed an opinion in the online survey. In Tanzania, the issue of international and inter-regional trade on paddy rice emerged over and over during the stakeholder meetings, even at the village level. In Myanmar, the issue of international trade is extremely important because of the country’s proximity and easy access to China’s huge fruit and vegetable import market, and to India’s vast import market for pulses and beans. The need for international trade and regulation research was strongly voiced by a wide sample of stakeholders that included two village groups and the principal consultant for the nation’s pulse and bean private-sector association, as well as almost all the top officials in the lead Ministry of Agriculture that were interviewed by the ET. 4.1.2 Conclusions EQ 1.1. Although the concept of demand-driven research was one of the six core principles of FSP IL, there are several well-documented examples of supply-driven studies that created a strong demand for additional review and/or stakeholder review, which influenced policy analysis and review. The highly participatory way in which the FSP IL studies were conducted helped build local ownership of the research results and their influence on policy review, formulation, and change by the host-country governments. If research is not conducted in a participatory manner, the results tend to sit on a shelf. If a supply-driven study is not conducted in a participatory manner, it does not become demand driven. EQ 1.2. Although several of the FSP IL researchers are working on regional (southern and eastern Africa) and national trade issues as part of the C4B (downstream) and C1/C2 East African global component teams, there is no one cluster of researchers focused on trade. Having trade as a separate subcomponent (such as C4C) would allow the project to add this activity without deleting any of the additional strengths of the project’s existing C4B’s downstream activities. It would also help the project to better capitalize on the emerging regional and international debates on this topic. 47 When queried about why trade had not been a focus of the FSP IL, one senior researcher attributed it to the fact that: “…in the past we spent more time dealing with ‘traditional’ issues like government intervention in cereals markets, including on the trade side. So, trade, both regional and global, was central to that kind of issue. Now, while we haven’t abandoned those issues, we are much more focused – appropriately, we think – on the transformations we see going on in the upstream and downstream of these food systems. And there is no question that, in C4b, we have explicitly wanted to stress the fact that the vast majority of the food economy is domestic – 80 percent to 90 percent in the countries we look at. This because so many are concerned that food imports are swamping these economies, and we believe they are not. At the same time, we emphasize the need for much more open regional trade, if companies are to have the scope for expansion and the learning opportunities that they need to compete in the medium- and long-run” (email communication to James Seale and Della McMillan, September 21, 2017). 17 Policy change can have unforeseen negative and positive impacts, making it important to evaluate its long-term and medium-term impact on client groups and vulnerable households. Although the concept of monitoring the long-term impact of their research is implicit in the six-stage model that most of the teams seem to be following, there is very little discussion of this topic. 4.1.3 Recommendations EQ 1.1. Future LWA: Ensure a participatory process that: 1) selects research topics in close collaboration with country-level partners; and 2) conducts the research in close collaboration with these same partners whenever possible. Future LWA: This participatory selection process should include research topics that are both demand driven (by the government and Mission’s requests) as well as supply driven (on topics that USAID/BFS and the researchers consider to be important). EQ 1.2. Length of Project (LOP): FSP IL’s component teams need to collaborate with local partners on the design and execution of rigorous studies that monitor the short- and medium-term impact of some of the policy reforms they influenced on local stakeholders. LOP: Given the critical importance of trade, FSP IL should consider adding policy research and analysis on trade issues as a new subcomponent, C4C, in the remaining months of the project. Future LWA: Encourage future Requests for Application (RFAs) that provide the basis for new program designs to use an “open economy48” paradigm to inform the design of their regional, global, and even country-level activities. 4.2 CAPACITY STRENGTHENING (EQ CLUSTER #2) 4.2.1 Findings 4.2.1.1 EQ 2.1: How effective has the coordination of capacity strengthening activities been among FSP IL components, FSP IL partners, USAID Missions and related entities at the country level? In what ways could this be improved? Coordination Among FSP IL Activities and Partners and Among FSP IL and Country Stakeholders: Overall, the responses to the online survey confirmed the ET’s impressions from the case studies and the literature review that the increased capacity stemming from FSP IL activities has successfully promoted the types of dialogue among different levels of policy actors needed to affect policy change and better policy development process. This policy influence has been achieved not by FSP IL alone. It has been achieved through FSP IL’s coordination with its local government, private sector, and NGO partners as well as other USAID and donor-funded national, regional, and global partners (Table 4). Coordination between the researchers on the FSP IL teams and among the teams and the FSP IL country-level partners (including the AA and Buy-in projects) and related entities tended to be stronger 48 To effectively and comprehensively study and analyze downstream and even upstream activities within a country, it is often imperative to understand the driving forces for those activities that occur from outside the national boundary. An open-economy paradigm starts with the premise that countries trade goods and services with other countries, often with their adjacent neighboring countries. The consequence of using an open-economy paradigm in the design of regional, global, and even country-level activities is to ensure that the effects of trade in goods and services is taken into account. For example, in Myanmar China’s demand for specialty crops such as melons and India’s demand for pulses are two of the largest stimuli for Myanmar’s agricultural sector outside of the rice sector. In Tanzania, its implicit trade ban on maize has numerous effects on the corn sector that are potentially not identified and understood if the paradigm is a more closed-economy model, one that does not account for the many effects that trade (or trade bans) in goods and services has on the local agricultural sector such as prices of commodities, availability of goods, availability of agricultural inputs, profits and losses in the agricultural processing sector, transportation of agricultural commodities, and even innovations in agriculture. 18 in the early years before the Missions agreed to fund the AAs and Buy-ins. Once the AAs and Buy-ins were created, these country programs developed their own strategies and staff, and focused on their own project’s objectives, which tended to respond to more specific Mission needs. As a result, many of the original C1/C2 staff migrated to the other component teams (C4A, C4B, C3) and to Mission-funded projects. This rapid growth in the number of AA and Buy-in projects has created new challenges for coordination that were identified and discussed in the FSP IL’s internal mid-term evaluation. 49 Table 4. Respondents Who Agreed with the Statement That the Coordination of FSP IL Capacity Strengthening Activities is Effective and Improving the Use of Evidence to Advocate for Policy Change (x=number who agreed/y=number who responded to the question) Coordination of FSP IL Capacity Strengthening Activities Effective Among FSP IL Coordination of Capacity Strengthening Activities Effective in Promoting Policy Change at the Following Levels Among FSP IL activities 80% (12/15)a FSP IL partners (UP, IFPRI, MSU) 71% (12/17)a Mission-funded Buy-ins (respondents from countries with Buy-ins) 77% (10/13)a Mission-funded AAs (respondents from countries with AAs) 73% (8/11)a In-country government agencies 88% (15/17)a 91% (10/11)b Level of FSP IL In-Country Stakeholder % of Respondents with an Opinion Who Said They Agreed 87% (26/30)c In-country civil organizations 74% (14/19)a 82% (8/12)b 88% (22/25)c Private sector and individuals 96% (25/26) c Policy development process 94% (31/33) c Source: FSP IL Mid-Term Performance Evaluation online survey. a The respondents were from stakeholder categories 1, 2, 5, and 6. b The respondents were from stakeholder categories 3, 4, and 5. c The respondents were from stakeholder categories 1, 2, 3, 4, 5, and 6. Coordination with National Policy Research Institutes: FSP IL’s coordination with the university and non-university-based think tanks, which were always intended to be one FSP IL’s principal local research partners for C2, was perceived to be very effective by the USAID Agreement Officer’s Representative (AOR), private sector, and government stakeholders that the evaluators interviewed in Tanzania.50 It was also mentioned as an example of best practice for future donor-funded programs by some of the respondents to the online survey as well as the university faculty who were interviewed.51 In the original cooperative agreement, it was expected that the FSP IL’s capacity-strengthening activities would focus on three research centers in three focus countries that would be “selected jointly with USAID, based on assessments of the capacities and strengthening needs of the national food policy systems conducted under C3 and on USAID country mission interest” (Annex IV.A). In fact, FSP IL has expanded this model of working with national policy centers to every country where it has been actively involved. FSP IL’s record for working with university and non-university based research policy centers was greatly appreciated by most of the USAID staff who were interviewed in Washington, DC, as well as a wide 49 Frank Young. 2016. Independent Review of Food Security Policy Innovation Lab: Internal Mechanisms and Processes and External Linkages. East Lansing: Michigan State University (December 11, 2016). 50 Given the FSP IL and AA’s extensive collaboration with SUA in Tanzania, the “university” was added to the list of rankings (on a scale of 1 to 5 with 5 being “strongly agree”) in some of the FGD and KII. 51 The evaluators conducted a FGD with nine staff and faculty in the SUA Department of Agricultural Economics and most of the senior research and management staff in the non-university-based Centre for Economic and Social Development (CESD) research center in Myanmar. 19 range of stakeholders in both of the case-study countries.52 There was also a strong sense that: 1) FSP IL did not allocate enough funds to capacity strengthening (in general);53 and 2) too much of the FSP IL on￾the-job and formal training had focused on strengthening the capacity of the individuals who were involved in specific studies rather than the long-term institutional capacity of the institution to execute this type of study, especially in the first two years of the program when they were under pressure to show “results.”54 In Myanmar, where the opportunities to work with the local Ministry of Agriculture staff and the Agricultural University were limited during the first three years of the IL, FSP IL focused its capacity strengthening activity on the independent think tank Centre for Economic and Social Development (CESD). This included formal training in research methods, sampling, and database management as well as on-the-job mentoring to help staff build their presentation, writing, and project management skills (Annex IV.G; Annex V.A). This consistent training over two and a half years has produced a nationally recognized cadre of food security policy researchers. Some of the best evidence of the quality of the coordination has been the high rate of CESD-trained personnel leaving for higher, better-paying jobs in the NGO sector or for Ph.D. training in the United States, Australia, and Korea. Although many of the researchers in the SUA Department of Agricultural Economics had extensive experience doing policy research, this was one of the first times they had had the opportunity to present their findings to the government through the FSP IL debriefings at the lead Ministries of Agriculture and the Agricultural Annual Policy Conference (AAPC) (Text Box 3, next page). Joint trainings in Stata, database management, and case-study preparation with junior ministry staff provided another forum for building these critical policy linkages to the Ministry. Staff reported that the training and support for publication would increase their chances of promotion, while at the same time improving their potential roles in Tanzania food policy. A recurrent theme in the interviews with senior FSP IL staff and USAID staff in Washington, DC and in the field was that: “although the short-term costs of working with the national policy centers were often higher than working with consulting firms, the long-term returns on this investment would far outweigh the costs.” With the USAID Field Missions: All three USAID AORs that were interviewed in the case-study countries agreed that FSP IL/AA/Buy-in has been effective in coordinating capacity-strengthening activities with its key national partners and other USAID projects. They stated that the principal vehicles for this capacity strengthening were FSP IL’s reports, briefing papers, presentations at donor meetings, and one-on-one informal communication and technical backstopping. The AORs stressed that these strong relationships did not happen overnight, but were the result of commitment and good management on both sides of the in-country USAID and FSP IL management teams. 52 In Tanzania, this included two USAID AORs and all FSP IL and Buy-in staff as well as the private sector and government staff who were interviewed in the FGD and KII. In Myanmar, this view was expressed by the USAID AOR, three donor representatives, and all of the LIFT multi-donor trust fund staff who were interviewed, the senior ministry staff and all of the AA staff and leaders who were interviewed. 53 Sources: KII with some of the lead FSP IL researchers in Washington, DC. FSP IL Online Evaluation Survey. Q 33 Text Response. Sample responses: 1)”Some people in FSP think of capacity building as identifying the best researchers in local institutions and hiring them as consultants. Very little attention to long-term institution building.” 2) “We have not found the best capacity delivery methods because…the capacity trainings were short lived [maximum of five days trainings mostly to journalists [for example]. There were not enough funds devoted to long lasting capacity building efforts. In Zambia, it took at least 12 years of continuous support where MSU staff and local Zambian/African staff are being capacitated to do quality and useful policy research. The first two years of FSP engagement in Malawi did not have that vision/element.” 54 Sources: FGD and KII with some of the lead FSP IL researchers in Washington DC. FSP IL Online Evaluation Survey Q35 Text Response. Sample response: “Serious partnership between local institutes and FSP from the very beginning, not bringing in local partners as sub￾contractors under terms dictated by FSP.” 20 Text Box 3 Case-Study -Tanzania: FSP IL Coordination with a University-Affiliated Food Policy Research Institutes In an FGD with nine of the SUA55 professors and students that were the local research partners with the FSP IL C4A and C4B teams, participants ranked their respective global team’s activities with them highly. When asked about the most important activities that had contributed to this ranking, they cited: The global teams’ hands-on, in-the-trenches-with-you approach to coordination and capacity building with their local university counterparts; Short training module courses that the FSP IL facilitated on various topics (like sampling, database management, new methodologies for data collection and entry) on which the researchers needed to improve their capacity; The long bi-weekly Skype meetings and the field supervision during which they discussed any problems and received coaching and direct assistance with analysis, write up, and publications; and Their participation in various research progress-report meetings at the ministries, as well as other FSP IL and Mission-funded Buy-in activities (through the ASPIRES project), that allowed them to meet for the first￾time colleagues in the Lead Ministries for Agriculture, as well as major donors and NGOs. At the same time, the evaluators observed a lack of coordination between the regional-global components of FSP IL and the Mission-funded Buy-in, ASPIRES. Specifically, by the middle of Year 3, the evaluation team observed that there was: Little coordination with the ASPIRES Buy-in COP except for local logistics; Almost no overlap between these three global team-sponsored research programs and the main ASPIRES Buy-in staff or the COP’s strategic planning/coordination with the Ministry; and Almost no attempt to get the three research teams to develop into a more cohesive group of inter-related sub-teams. One result of this disconnect was that the evaluators had to contact the global team leads to obtain information on the research, financing of the activities, and the current year’s research, since the ASPIRES COP’s involvement in the team was mostly focused on logistical support. Source: FSP IL case-study research in Tanzania. See also Annex IV.G and Annex V.B. While USAID field Missions were identified from the start as key stakeholders that would potentially join FSP IL as co-financiers, the initial FSP IL agreement did not include a clear strategy for ensuring that the Mission priorities and investments were clearly considered and reported. Specifically, the initial FSP IL agreement put the main project administration under no contractual obligation to provide any country-level analyses or to develop routine debriefings for the field Missions in the countries where they worked. Fortunately, FSP IL was able to overcome this oversight in the original design by having: 55 Note: The Department of Agricultural Economics at SUA is the official university-based policy research center that is affiliated with ReNAPRI in Tanzania. ● ● ● ● ● ● ● 21 ● ● ● Strong and experienced in-country managers with USAID experience who were willing to develop this type of country-level reporting on a case-by-case basis in countries where FSP IL intervened;56 A series of experienced AORs and Contracting Officer’s Representatives (CORs) in most of the countries where FSP IL was most active who were willing to work on the relationships; and An experienced full-time AOR at the level of USAID/BFS who understood both the Mission-level concerns and those of FSP IL consortium partners. Once the AAs or Buy-ins were in place, the Missions reported benefitting from both the FSP IL core￾funded, and AA/Buy-in co-funded or direct-funded capacity-building activities in ways that helped strengthen their strategic planning and management. The most appreciated capacity-strengthening mechanisms were: ● ● ● The structured feedback on other donor-supported policy initiatives received through participation in the FSP IL/ASPIRES Buy-in co-facilitated Policy Analysis Group (PAG) and AAPC in Tanzania, and through the FSP IL/FSP AA’s participation in the Livelihoods and Food Security Trust Fund (LIFT) multi-donor trust fund’s routine debriefings; Periodic one-on-one debriefings of staff and Mission-level representations by senior FSP IL staff; Consistent one-on-one communication with the in-country FSP IL Country Director and/or AA or Buy-in COP. With Other In-Country USAID-Funded/Supported Initiatives: One of the major strengths of the FSP IL capacity-building model is its willingness and mandate to encourage FSP IL component teams, AAs, and Buy-in projects to coordinate with other Missions and regional initiatives funded by USAID. This type of coordination and co-funding has helped the project to leverage additional funds for USAID￾funded initiatives like Alliance for a Green Revolution in Africa (AGRA) and Africa Lead in Tanzania, and the LIFT multi-donor trust fund in Myanmar. One direct impact of this type of co-funding and coordination with other in-country USAID-funded initiatives has been: 1) building the capacity of co￾funding institutions like the LIFT multi-donor trust fund in Myanmar to use FSP IL-generated data to strengthen the impact of the FSP IL studies on national policy processes; and 2) helping expand funding for FSP IL and Buy-in project’s stakeholder consultation and outreach activities in Tanzania (Text Box 4). Text Box 4 Sample Cases of FSP IL Collaboration with Africa Lead and AGRA in Tanzania Africa Lead, which supports capacity building under the Feed the Future initiative, has worked with several of the FSP IL, AA, and Buy-in activities.57 From the beginning, for example, they collaborated with AGRA to support the FSP IL-created AAPC in Tanzania. Both AGRA and Africa Lead are founding members of the FSP IL-facilitated PAG that meets monthly to ensure better coordination of different donor and government-orchestrated research efforts for the Ministry of Agriculture in Tanzania. More recently, both scaled up their collaboration with FSP IL by co-funding the initial scale up of the stakeholder consultation phase of the new e-payment system that is a direct outcome of some of the initial FSP IL research in Tanzania. In addition, they have supported numerous small activities that have helped widen the policy impact of the FSP IL’s and Mission-funded Buy-ins’ initial investment in research as well as capacity building. Source: FSP IL documentation and FSP IL case-study research in Tanzania. 56 All three AORs signaled the critical importance of having an empowered in-country director with appropriate support staff to ensure good coordination. Although coordination was strong at the beginning in Tanzania, it waned as the activities grew until the COP got appropriate support staff. In Myanmar, this coordination was made more difficult by a one-year gap without an in-country director in the second year. 57 Source: FSP IL. 2017. Feed the Future Innovation Lab for Food Security Policy Team Meeting, March 28-30, 2017. Country Program Summaries. Washington, DC: IFPRI for FSP IL. 22 With Regional Capacity Strengthening Initiatives: Since its inception, FSP IL coordinated with two major regional entities designed to build the capacity of some of the leading university and non￾university based research institutions to conduct, analyze, and use evidence-based research to improve national food policies. These entities include: ● ● The Regional Network of Agricultural Policy Research Institutes (ReNAPRI), which was created at the initiative of the national agricultural policy institutes in 2012 to achieve two objectives: ­Assist the national agricultural policy institutes with building their capacities to carry out high-quality policy analyses, outreach, and capacity-building activities through collaboration and coordination amongst themselves; and Establish an effective platform for providing policy guidance to national policy makers and stakeholders based on cross-country learning of policy experiences. The Regional Strategic Analysis and Knowledge Support System (ReSAKSS), which IFPRI helped to create in 2006 to support efforts to promote evidence and outcome-based planning and implementation as part of the Comprehensive Africa Agriculture Development Program (CAADP) agenda through the national Strategic Analysis and Knowledge Support System (SAKSS)58 nodes. The coordination with ReNAPRI embraces a wide range of activities (Table 5): ● Most of the C4A team’s research in East and Southern Africa has been sub-contracted through six of ReNAPRI’s nine university-based think tanks. The Bureau for Food and Agricultural Policy (BFAP) at UP has supported the maintenance and expansion of the core analytical capacity of three of the ReNAPRI networks through national and regional-level training in partial equilibrium (PE) modeling and database management, which was funded by C1/C2.59 Through its collaboration with the national institutes in the ReNAPRI network, FSP IL has supported institutional capacity strengthening – going well beyond strengthening the capacity of individuals—in at least several countries (Table 5). In Zambia, for example, FSP IL has continued to build the institutional capacity of Indaba Agricultural Policy Research Institute (IAPRI) with consultations on the strategic direction of the institute, including how evolving megatrends affecting Africa may influence the topic areas where IAPRI should focus on building long-term expertise and capacity to respond to rising demand. These areas include climate-smart agriculture/resilience and sustainable intensification. FSP IL has introduced IAPRI management to other partners in the region working on such issues, including the Center of Excellence (COE) Programme in South Africa, the NGO VUNA60/Southern Africa, etc. In March 2017, IAPRI partnered with VUNA and the South African COE to co-sponsor the regional Climate Change/Resilience Summit. FSP IL financially supported this effort in various ways, including research that was presented by IAPRI researchers at this Summit. The conference helped develop IAPRI’s strategic position in the region as a local research institute with expertise in this topic area, which has generated a number of follow-up research and policy outreach collaborations involving IAPRI and other partners and is clearly building IAPRI’s institutional capacity. 58 The SAKSS date back to before 2000. ReSAKSS was meant to empower the region in owning and priority setting for the SAKSS research. 59 The expansion of the ReNAPRI partial equilibrium model as well as the databases is also partially funded by the EU commission in Tanzania. FSP IL has made a significant contribution in maintaining and strengthening the momentum of the ongoing partial-equilibrium modeling, outlook/projections, and database development in these countries. This forms part of the annual outlook event that ReNAPRI is hosting for stakeholders in the sector. 60 Vuna is the name for “water” in one of the South African local languages. VUNA is a registered Southern African NGO. 23 Table 5. Areas of Collaboration Between the FSP IL Component C4A Global Team and its Principal Local Research Partners Associated with ReNAPRI Category Tanzania Zambia Kenya Malawi Mozambique University of Pretoria in Senegal Contact Charles Jumbe Prof. Emilio Tostao Ward Anseeuw Name of Principal Implementing Partner SUA IAPRI Tegemeo Institute Centre for Agribusi ness and Rural Develop ment (CARD) Research Centre for Agricultural and Food Policies and Programs (CEPPAG) BFAP Type of Institution University Policy institute Policy institute Policy institute Policy institute Policy institute Member of ReNAPRI Yes Yes Yes Yes Yes Yes Year Founded n/a (but a member) 2012 1997 1990s 2014 2005 Years Working with Global Team 3 3 3 3 2 1 Broad Categories of Collaboration Land research and outreach work between SUA, MSU/FSP and Tanzania Bureau of Statistics (TNBS).61 Land, fertilizer, inputs, Markets, price policies, climate change Inputs, climate change Land, medium￾scale farms Land, medium-scale farms Multifaceted , but mainly on land, megatrends, and economic transformati on themes under C4A Research Design Multifacet ed 62 Multifacet ed Yes Yes Yes Yes Research Implementation Yes Multifacet ed Yes Yes Yes Yes Data Entry and Analysis Yes Yes Yes Yes63 Yes64 TBD Debriefings/ Presentations At least three already in 2016 and 2017 Yes Yes Yes Yes Yes Write-Up/Publication Policy Research Yes Yes Yes Yes In progress Yes 61 The work also involves training of SUA staff in econometric skills and Stata as well as discussions between the Dean, Faculty of Agriculture/SUA and MSU about building an Agricultural Policy institute within the Faculty of Agriculture in SUA to strengthen SUA’s institutional capacity to undertake agricultural policy work in Tanzania and to strengthen SUA’s role in the ReNAPRI network. 62 There are seven studies underway under this collaboration between SUA, TNBS, and FSP IL/MSU. 63 The FSP IL C4A financed the survey of medium-scale farms in Malawi, implemented by CARD. The results of the survey are available for wider use by CARD on other issues. 64 FSP IL C4A financed CEPPAG’s survey of medium-scale farms; it also supported research design, sampling, implementation of the survey, etc. This data set is CEPPAG intellectual property rights, and can be used by CEPPAG for their independent use. 24 Category Tanzania Zambia Kenya Malawi Mozambique University of Pretoria in Senegal Academic Publication of Food Policy Research One paper finalized so far, others in the pipeline Yes Yes Yes65 Yes66 Database Design and Management Yes Yes Yes Yes Grants Management Yes Yes Yes67 Proposal Development Yes68 Yes69 Facilitate Graduate Training Staff Yes Yes70 Networking Yes Yes Yes a) Ag. Ministries Yes Yes71 b) Potential Donors Yes c) Other National and Regional Leaders Yes Yes Source: Source: Interviews and Feedback from National Staff Associated with the Global C4A Local Partners. August 2017. There are other examples where FSP IL has supported initial work in a particular area with one of the local institutes in a way that synergistically engages with other public and NGO partners, plugs the local institute into new opportunities, and supports their credibility and name recognition in particular areas. FSP IL also works closely to build the capacity of individuals in these institutes so that they can carry out the technical work once the local institute experiences increased demand for their engagement. The coordination with ReSAKSS is also strong (Text Box 5). The Kaleidoscope Model developed under C3 and the methodology for reviewing national commitments related to C3, Activity 4, have been integrated into the Toolkit for experts to be recruited to support the review of the first-generation National Agricultural and Food Security Investment Plans (NAIPs) and the drafting and review of the second-generation NAIPS. To date, 24 experts from 10 countries have been trained in food security and nutrition analysis as part of the network. 65 2016 publication involving CARD, FSP IL, and UP researchers. Also, the CARD staff among the MSU team won the 2017 AAEA Bruce Gardner Memorial Award for policy analysis, for research on input subsidy programs in Africa funded partially under FSP IL C4A. 66 FSP IL has supported BFAP for past five years in its national BFAP annual outlook conferences and linking BFAP to wider networks in the wider Africa region. 67 FSP IL has worked with BFAP on a number of proposals that have received funding from other sources to strengthen on-going FSP IL activities and ensure continuity of the research activities. 68 FSP IL helped SUA to successfully obtain a grant under the PIM (Policies, Institutions, Markets) mechanism ($60,000) to build a research program within SUA on land-policy issues in Tanzania. FSP IL also provided over $100,000 to SUA to enable it to collect unique survey data on medium/large-scale farms in Tanzania that has attracted several new sources of funding, including $30,000 from UP/BFAP. 69 FSP IL supported the institutional develop of IAPRI in diverse ways. In fact, IAPRI was created out of a long-term MSU project in Zambia, converted into an autonomous Zambia-led institute in 2012, receiving support from FSP/MSU ranging from technical to financial management, personnel policies, policies toward individual consultancies, outreach, strategic plan, operations manual, etc. 70 Served as visiting professor in 2014 and 2015 at UP teaching graduate classes and guiding four masters (MS) graduate students. 71 Several debriefs of Ministry of Agriculture. 25 Text Box 5 Coordination of Capacity Building Between the FSP IL and the African Union Commission’s (AUC’s) CAADP Numerous statements, decisions, and declarations commit Africa’s leaders to realizing the continent’s aspiration for equitable growth and socio-economic development through improving food security and human nutrition. One of the most important of these is the African Union’s Agenda 2063 (AU 2015a), the policy framework for the continent’s development over the next 50 years. The “Malabo Declaration” sets out the plan of action for achieving the vision of Agenda 2063 and Sustainable Development Goal 2 in terms of agriculture, food security, and nutrition.72 The Declaration contains seven priority areas for development and implementation of the Sustainable Development Goals (SDGs) in a manner that adequately supports the broader development of the continent. Since 2017, the CAADP process has entered a second stage in which, starting in 2018, countries will review their first-generation CAADP NAIPs and design and implement second generation NAIPs. One senior FSP IL stakeholder called it “an unprecedented policy window for the project” to intensify its existing formal and informal connections to the CAADP policy processes. ● Many of the senior national experts working on the Mission-funded AAs and Buy-ins have already been involved in the review of the first generation and the design of the second generation NAIPs; and ● Some are members of the NAIP Task Team established by ReSAKSS/IFPRI in 2016 to support country teams in the review of the first-generation NAIPs, the design of the second-generation NAIPs, and the AUC’s review of the second-generation NAIPs. This Task Team was recently (June) integrated into the CAADP Technical Network. In her role on the task force, one of the senor UP researchers—who is also the UP Lead on the FSP IL—has been responsible for the development of training materials and training experts from across the continent in food security and nutrition to carry out this role and she will backstop the experts when they are in-country. In conjunction with this training, FSP IL C3 tools that were developed during the second and third year have been integrated into the training and technical toolkit, namely: 1) the Kaleidescope Model; and 2) a database of international, continental and national commitments on food security. Once the online library is uploaded to the ReSAKSS website (at no cost to FSP IL), it will be used to back stop other African countries in the review of their first generation NAIPs and the design of the second generation NAIPs. The gender in nutrition assessment tool (developed by a Ph.D. student on the project) will be made available to the experts in the Network as well. To date, 24 experts have been trained in food security and nutrition to participate in the 20 country support initiatives through the CAADP Technical Network.73 None of these activities were supported by the FSP IL core component funds. The activities did, however, translate the work done in the FSP IL into products that are supporting country-level processes through one of the most important regional entities that is supporting these processes (IFPRI/ReSAKSS). Source: FSP IL documents and FSP IL Mid-Term Performance Evaluation interviews. There is also strong collaboration between the IFPRI-facilitated ReSAKSS network and FSP IL component teams, AAs, and Buy-ins (Table 6). The original FSP IL cooperative agreement identified this coordination as a top priority for FSP IL. A casual reading of the FSP IL component teams’ annual reporting and their presentations at the all-FSP IL meeting in March 2017, show at least a hundred74 examples of coordinated capacity building either through joint trainings, joint research projects, or formal training sessions and presentations (many of them with national colleagues) at learning events with the Food and Agriculture Organization of the United Nations/Monitoring and Analyzing Food and Agricultural Policies Program (FAO/MAFAP), the Economic Community of West African States’ 72 African Union (AU). (2014a). Malabo Declaration on Accelerated Agricultural Growth and Transformation for Shared Prosperity and Improved Livelihoods. http://pages.au.int/sites/default/files/Malabo%20Declaration%20 2014_11%2026-.pdf. 73 Hendriks’s membership on the NAIP Task Force combined and close working relationship with ReSAKSS has facilitated the uploading of the materials and database to the ReSAKSS website as part of a public good as well as in support of the CAAPD processes in countries and as part of the toolkit for the CAADP Technical Network. 74 Based on what was reported in the March presentations—which overlaps with the figures reported in the annual and semi-annual report— the actual figure is probably in the hundreds. In each case, however, it is hard to determine what can be directly attributed to the FSP IL and what is co-contributed by the FSP IL. Source: FSP IL. 2017. Feed the Future Innovation Lab for Food Security Policy Team Meeting, March 28- 30, 2017. Country Program Summaries. Washington, DC: IFPRI for FSP IL. 26 Permanent Interstate Committee for Drought Control in the Sahel (ECOWAS/CILSS), and other groups. Table 6. Examples of Capacity Building Coordination Between ReSAKSS-Supported Activities and the FSP IL Component and AA Teams in Myanmar, Senegal, and Malawi Country Collaboration Between ReSAKSS and FSP IL Early Evidence of How This Impact Is Affecting the Policy Process in the Host Country Myanmar AA and FSP IL Both the Myanmar AA and FSP IL teams backstopping the AA are contributing empirical analysis of the rural economy and policy analysis to ReSAKSS. ReSAKSS can now include Myanmar in its analysis whereas it could not previously. Senegal AA and FSP C4A and C4B Facilitate the set up and operation of an interactive information technology-based (IT-based) knowledge management system through the national ReSAKSS node under Direction de l'Analyse, de la Prévision et des Statistiques Agricoles (DAPSA). This collaboration has helped share climatic data and statistics of the agricultural sector through a system of information on Senegalese agriculture. Senegal AA and FSP C4A and C4B Mobilize local research to guide emerging agricultural policy agenda. In this collaboration, Project d’Appui aux Politiques Agricoles (PAPA) has requested to fund a NAIP expert to collaborate with the Ismael team in charge of conception of analytical tools for Senegal NAIP 2.0 M&E. Conception of analytical tools for monitoring and assessing NAIP 2.0 for Senegal under CAADP. Malawi AA -The NAPAS: Malawi team supported the Ministry of Agriculture, Irrigation and Water Development in producing the joint-sector review reports for 2014/15, 2015/16, and 2016/17. The NAPAS: Malawi team also supported the Ministry of Agriculture in producing the Malawi New Alliance report in 2015/16. -The COP of Malawi AA (NAPAS: Malawi) was one of the resource persons at the 2015 ReSAKSS meeting in Addis Ababa, Ethiopia. -Ministry of Agriculture staff are better able to submit their reports to CAADP on time. -This collaboration lead to the Malawi government sharing its experiences with using the FSP IL C3 Kaleidoscope Model for the policy formulation process in Malawi with the delegates at the annual ReSAKSS meeting. As a result of this presentation, the Malawi government was invited to share their experiences at an AGRA-Organized Technical Convening on Policy for the Implementation of Malabo Goals in 2016 in Zambia. -The NAPAS: Malawi team supported the Ministry of Agriculture, Irrigation, and Water Development in developing the Malabo Biennial Report for Malawi that will be submitted to the African Union through the Common Market for Eastern and Southern Africa (COMESA). -The NAPAS: Malawi team provided technical support to inform the design of the National Agricultural Investment Plan for Malawi that will be submitted to the African Union after country-level validation. Source: FSP IL documentation. 27 In each case, however, this coordination has been the product of strong leadership from specific FSP IL component team leaders and researchers rather than a focused FSP IL strategy.75 To date, FSP IL has not developed a coherent strategy for coordinating its capacity strengthening activities with ReSAKSS that would help capitalize on the existing and potential synergies between the two partners. Many of the senior staff that were interviewed in the FGDs and KIIs in Washington, DC feel that minor changes in the way the data are presented and communicated to the national governments and ReSAKSS could dramatically increase the governments’ willingness and ability to use this data to inform their second￾generation national agricultural investment strategies. As one senior researcher stated in a KII during the evaluation, “this is a short-term policy window that could dramatically increase the use of the FSP IL component team, AA, and Buy-in studies to inform both regional and national policy.” Examples of Coordination Best Practice: One direct consequence of the initial investment by FSP IL in policy research, capacity strengthening, and communication was to create a wide range of partners that have different capacity strengthening needs. This, in turn, increased the difficulty of coordinating capacity strengthening activities through a single ministry-based policy office or USAID Mission-funded program. The initial requests for building a better system for channeling donor advice on food policy came from the Ministry of Agriculture in Tanzania. To respond to this request, the FSP IL COP worked closely with the USAID/Tanzania AOR to identify other donors and USAID and non-USAID projects potentially interested in supporting this type of stakeholder information sharing event. This core planning group transformed into PAG in 2013 (Table 7). Since 2013, PAG has met and held monthly round table breakfasts for it members and is the body responsible for planning and executing the AAPC (Table 7). Although the ASPIRES Buy-in COP chairs the PAG meetings, the bulk of PAG’s work in terms of planning the program, logistics, and funding is done by PAG’s members. It is this participatory approach to planning and fund raising that has enabled the small FSP IL and later the FSP IL ASPIRES project team to manage sizable amounts of information from PAG partners and still have a coordinated, multi-project approach to PAG and AAPC. Table 7. Examples of Best Practice in Capacity Strengthening Coordination, Organization, and Implementation in Target Countries Example Relationships to FSP IL and Why an Example of Best Practice Early Evidence of Impact on National Policy Review and Analysis AAPC (Tanzania) -Started in Year 1 with FSP IL core funds. -Now in its third year with sustainable funding from a diverse group of USAID and non-USAID projects and other donors. -Provides a mechanism for FSP IL researchers and country partners that work in Tanzania to share their results with private sector, government, civil society, and donors. Increased the flow of information between all the major food policy actors and helped harmonize the food policy recommendations being transmitted to the national government. PAG (Tanzania) Organized and originally funded by the FSP IL, it was formed in 2013. The PAG holds monthly round table breakfasts for its members and is the body responsible for planning and executing the AAPC. Source: FSP IL Mid-Term Performance Evaluation. 75 The senior leader of the C4A component team serves as a member of the Board of Directors of ReNAPRI as well as the IAPRI that is a member of ReNAPRI. A senior researcher from UP serves as Director of the BFAP that is a member of ReNAPRI. 28 Now that the AAPC is established, it helps to share the policy research agenda of PAG members to ensure their policy work is demand-driven. AAPC also provides a platform for inclusive public policy debate by creating a more closed workshop between the PAG and the government of Tanzania— through the Partnership Accountability Committee (PAC), which is tasked with: 1) reviewing the policy issues that emerged from AAPC; and 2) assigning roles among PAG members in policy research or support for the government as it implements reforms; and 3) tracking progress in policy reforms through an annual New Alliance report to which PAG contributes to its preparation. Because of this strong alliance between PAG, PAC, and the government, the pace of policy reforms in agriculture has accelerated in recent years, as reflected in the 2016 Annual Report on the New Alliance in Food Security and Nutrition. PAG has helped promoting mutual trust between policy researchers and the government as well as improve coordination of policy research in Tanzania and, hence, minimize duplication of effort. 4.2.1.2 EQ 2.2: How effective has FSP IL been in the organization and implementation of capacity strengthening activities covering different content such as research, analysis, strategy and communication as well as providing advice to improve the use of evidence to advocate for change at the following levels: 1) Country; 2) Organization; 3) Individual; and 4) Policy development process? In general, survey respondents agreed that FSP IL’s organization and implementation of capacity strengthening activities had been effective in promoting policy change at different levels (Table 8). Table 8. Respondents Who Agreed with the Statement that the Organization and Implementation of Capacity Strengthening Activities (i.e., Policy Research, Analysis, Strategy, Communication, and Advice) is Highly Effective at Promoting Policy Change at the Following Levels Level of Policy Change % of Respondents That Agreed with Statement National government (ministers, parliamentarians, other staff) 87% (26/30) Organizations (farmer groups, NGOs, other civil society groups) 88% (22/25) Individuals 96% (25/26) Policy Development Process 94% (31/33) Source: FSP IL Mid-Term Performance Evaluation online survey, Q31. Respondents were from stakeholder categories 1-6. Although most of the stakeholders felt that FSP IL has been very effective in developing the initial capacity of their key government, private sector, and civil society partners, they were equally vocal in expressing their impression that, to date, the organization and implementation of these activities has been: 1) mostly ad hoc; and 2) based on the personal experiences and professional networks of the lead researchers in specific countries. 4.2.1.3 EQ 2.3: In what ways could the capacity strengthening component be improved for greater policy impact including support to local stakeholder groups? Even though staff generally had a strong commitment to the idea that the rural farmer was the principal focus of their work,76 neither FSP IL nor its AAs or Buy-ins have developed much of a strategy for communication with these farmers.77 All four of the community groups who were interviewed, stated 76 Source: KII and FGI in Washington, DC and the two case-study countries. 77 Source: FSP IL Mid-Term Evaluation Case-study, Community-level FGD in Tanzania and Myanmar. See summary in Annex IV.G. 29 that they were aware that they and their communities had been studied and were anxious to learn about the results of the studies. In Myanmar, the community leaders wanted to be sure that their elected parliamentarian had both the reports and local language briefing papers to assist him in speaking up for their rights. To date, however, none of them had attended any sort of community briefing on the research results nor had they seen any of the reports. Although none of the FSP IL country programs have made the NGO community a major focus of their intervention, they have sometimes forged strong working relationships with the NGOs that intervene in certain topical areas and/or geographical areas. The evaluators observed a very successful collaboration between the FSP IL/AA researchers and a large national NGO that worked in all but one (Mon State) of the region/states where the FSP IL/AA has conducted its research. To date, this collaboration has had a major impact on the global conceptualization of the NGO’s programs as well as some of its interventions that could be documented in its online publication series.78 The respondents to the online survey made recommendations for improving the capacity-strengthening component (Annex IV.B). 4.2.1.4 EQ 2.4: What capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities? What types of synergies if any have there been among the varied MSU, IFPRI and UP capacity strengthening approaches? In what ways can each partner’s capacity partnership approaches be improved? Which Method was Most Effective? The respondents to the online survey identified many examples of what they considered to be the “most effective” capacity-strengthening activity (Annex IV.C). One of the strengths of the evaluation case-study research in Myanmar and Tanzania was to highlight that no one capacity-strengthening activity was effective in delivering the best capacity-strengthening activity (Annex IV.G, case-study). Instead different capacity-strengthening activities worked with different groups and worked differently with different groups when the government demand for evidence-based research was low (i.e., at the start of the FSP IL intervention in a country) and at the current stage where the demand is growing fast (Annex IV.G, case-study). To illustrate this point it is useful to examine how every one of the research projects used a mixture of capacity-strengthening methods to build the capacity of their principal government partners and their principal research partners—a non-university-based policy center (CESD) in Myanmar and a university￾based policy center at SUA in Tanzania [see Annexes IV.E79 (summary table) and Annex IV.G. (case￾study)]. ● ● Lead Agriculture Ministry Staff in Tanzania and CESD Staff in Myanmar: ­In Myanmar, where the opportunities to work with the local Ministry of Agriculture staff were limited when the project started, FSP IL focused its initial on-the-job training and formal training courses on the CESD staff. In contrast, the Tanzania program was able to involve the national partner staff in on-the-job training as well as formal training courses from the start. To sustain this training and the contacts between the junior staff in the different lead agricultural ministries, FSP IL facilitated the creation of the voluntary working group, the Platform for Agricultural Policy Analysis and Coordination (PAPAC). Senior Ministry Staff in Myanmar: In contrast, all of the senior ministry staff80 interviewed in Myanmar81 highlighted the critical importance of the one-on-one mentoring they received from the 78 Sources: FSP IL Mid-Term Evaluation Case-study, Community-level FGD and KII in Myanmar. See summary in Annex IV.G. 79 Table 9 attempts to rank the relative importance of different broad categories of capacity strengthening activities for different audiences based on the stakeholders’ observations. 80 No senior staff in the lead ministries of agriculture were interviewed; only middle level technical staff associated with the PAPAC. 81 The team interviewed about 10 senior ministry managers (i.e., department director and above) in the lead Ministry of Agriculture and one senior manager in the President’s Office. 30 FSP IL’s in-country82 Country Director. Especially important, they relied on the Country Director/COP for information on new data sets, documents, policy briefs, and talking points for presentations. ● NGO/Private-Sector Stakeholders in Myanmar and Tanzania: Most of the non-government/civil society/private-sector actors highlighted the great benefits they achieved from attending the FSP IL knowledge dissemination and learning events sponsored by the government, donors, or donor￾funded projects. Most stated that these events “opened the door” to their being interested in the FSP IL and AA/Buy-in’s online documentation in English and local languages. The one capacity delivery method that was effective with every major group of stakeholders in the two case-study countries was the AAPC and PAG (Annex IV.E; Table 7). To date, however, FSP IL has not collected any data from stakeholders that would enable them to measure the relative effectiveness of different stakeholder strategies in addressing the most important capacity gaps in key stakeholders training and knowledge. Synergies: If one compares the capacity-strengthening contribution of the different lead partners (MSU, IFPRI, and UP) one is struck by the high degree of overlap. There are certain areas where each partner has excelled (Annexes IV.D and IV.F). There are also subtle differences in their approaches to local institutional development and regional integration and institutional capacity building at the national level that sometimes overlapped and reinforced one another—especially in terms of complementing one another in particular sub-areas of capacity strengthening (Annex IV.F). One respondent to the online survey stated that (Annex IV.D):83 “1) IFPRI [has] contributed significant public policy skills; 2) UP contributes significant deep knowledge of policy; 3) MSU has the deepest understanding of field conditions in African agriculture and in agribusiness transitions.” FSP IL and AA staff in Myanmar reported strong synergies between IFPRI and MSU’s capacity building activities in the design and execution of the livelihood surveys84 as well as in the NGO training.85 The media training in Malawi was also cited as an example of good synergy between UP, MSU, and IFPRI.86 The staff of the local FSP IL implementation partner (CESD) and in-country FSP IL staff were unanimous in their appreciation of the synergies between IFPRI and MSU’s capacity strengthening approach for research. To date, however, the majority of the MSU, IFPRI and UP’s country-level activities have been executed in parallel and or sequentially.87 One good illustration of this involves the partner activities that link the country programs to ReSAKSS and the CAADP processes. IFPRI helped create ReSAKSS in 2006 and continues to oversee its main office, capacity building, and knowledge exchange activities through a separate project that USAID supports. This connection has helped create a venue for national and continent-wide dissemination of FSP IL results. Many FSP IL researchers (from MSU, IFPRI, and UP) have co-presented papers at the ReSAKSS events and helped backstopped their national-level colleagues in 82 Note: Some of the FSP IL Country Directors are based in the U.S. 83 Source: FSP IL Mid-Term Performance Evaluation online survey Q34 text (Annex IV.D). 84 Source: FSP IL Mid-Term Performance Evaluation Online Survey FGD and KII with FSP IL and AA staff in Myanmar. Specifically, they contended that the: 1) The Senior and mid-level MSU researchers who provided short-term technical assistance tended to be better on the initial design and sampling of the household-level livelihood surveys, which have been the background of FSP IL’s policy research in Myanmar, and had a more long-term pedagogical approach to capacity building of the local staff; while 2) IFPRI staff were more up to date (than the MSU staff) on new cutting-edge data entry and database management innovations, and were more product-oriented on data analysis and reports. The staff reported that the two perspectives were highly complementary and recommended that the evaluators make a strong recommendation for continuing the existing “marriage of convenience” that mated IFPRI to an experienced university partner for future USAID/BFS LWA awards designed to strengthen the national, regional, and global policy environment for agriculture, nutrition, and food security in counties where USAID has Feed the Future projects focused on food security policy. 85 Source: FSP IL Mid-Term Performance Evaluation Online Survey. Q34.Text (Annex IV.D). 86 Source: FSP IL Mid-Term Performance Evaluation Online Survey. Q34.Text (Annex IV.D). 87 Source: KII and FGD with senior FSP IL staff in Washington, DC; FGD and KII in the two case-study countries. 31 providing information to the SAKSS and Joint Research Commission (JRC) reports that each country must produce as part of its annual reporting (Table 6, above; Also, Annex V.C). UP is a member of ReSAKSS. The FSP IL Consortium Lead for UP was recently appointed to the ReSAKSS task force and is using her position to funnel FSP IL-generated data and the C3 module into the ReSAKSS training programs (Text Box 5, above). Although each partner plans these activities through a joint planning process and work plan, the partners have tended to execute the activities on their own. To date, however, FSP IL has not developed a strategy for capturing some of the potential synergies of these partner activities with ReSAKSS at either the country or regional level. One cross-cutting theme in the online survey (Text Box 6), the KIIs and FGDs in Washington, DC, and the case-study countries, was the need to encourage more joint planning between the partners at the national and regional level to enhance the potential synergies between the capacity-strengthening activities of the three strong consortium partners. 4.2.2 Conclusions EQ 2.1 FSP IL Coordination with In-country Stakeholders: One major conclusion is that when FSP IL teams were successful in generating high levels of local demand for evidence-based research, the demand for capacity building quickly outstripped their ability—and the ability of the lead Ministries of Agriculture that they supported—to keep up with the demand. This, in turn, has quickly generated a demand for new mechanisms for coordinating speedy and effective responses to that national demand. One of the most promising mechanisms for promoting better in￾country coordination is the FSP IL facilitated AAPC which is currently being managed by the FSP facilitated PAG in Tanzania. Since 2013, PAG and AAPC have provided effective models for triple purpose capacity strengthening that: ● ● ● ● Communicate new national and global research; Build stakeholders’ capacity to understand and use the data; Help different categories of stakeholders meet one another to share information and best practice; and Help leverage additional resource beyond the FSP IL to support knowledge policy research, analysis and advocacy from other government and donor-funded projects. FSP IL Coordination with the USAID Country Missions: Even though there was no contractual obligation to report, FSP IL was willing and able to respond to the Missions’ desire for close coordination because they had an in-country COP and strong high-level support for coordination. This coordination helped facilitate their collaboration with the wider donor community, and helped them leverage additional resources, which had a positive impact on the FSP IL’s results in both case-study countries. FSP IL Coordination with the AA/Buy-in Projects: While the Mission-funded projects have strengthened the impact of FSP IL, they have created new challenges for coordination. FSP IL Coordination with the Agricultural Policy Research Institutes: Although the university and non-university based food policy institutes have been some of the FSP IL’s most reliable local research partners, it is hard to determine whether they have the core capacities that they need to sustain their role going forward. This is because FSP IL never developed a good system for tracking the results of its collaboration with these institutions or its principal government partners at the national level. FSP IL Coordination with two of the FSP IL’s Principal Regional Partners (ReSAKSS and ReNAPRI): Given the critical importance of ReSAKSS and ReNAPRI in promoting regional level capacity strengthening for 32 Text Box 6 Stakeholder Responses to the Question: “In what ways can each partner's capacity strengthening approacn this country?” MSU ● ● ● ● “Serious partnership between local institutes and FSP from the very beginning of project design, not bringing in local partners as sub-contractors under terms dictated by FSP.” “Tanzania is largely an MSU effort. Our work in this area is highly coordinated with government and with other USAID development partners. With UP, we just needed to have more communication about the training and how other elements of the work could take advantage of it.” “Well I don’t know what ‘this country’ is as I work in Senegal, Nigeria, Tanzania. In Nigeria and Tanzania. I am working closely with local universities doing research together. In Senegal, who knows what is happening?” Funding constraints currently limit capacity building activities: “we would need significant resource boost to become more effective”; 2) “better needs assessment”; 3) “development of a multi-year capacity building plan for a set of organizations and individuals.” IFPRI ● ● ● “Consider modalities for true collaboration with local units. As a policy research institute, IFPRI has not capacity building mandate, but it needs to seriously develop one if it is to be regarded positively in the countries in which it operates. Increasingly, national governments want to rely on their own homegrown policy institutes, not 1980s models of international technical assistance that does not engage productively with local units.” “Development of a multi-year capacity-building plan for a set of organizations and individuals.” “More effective use of IFPRI’s research and capacity in the field.” UP ● ● ● ● ● ● “1) Provision of information on all other FSP activities in Malawi; 2) Arrangement of meetings with all other FSP researchers working on/in Malawi; 3) Establishment of comprehensive database of all FSP-collected information in Malawi.” “1) Expanding capacity strengthening to various groups including parliamentarians, government officials and private sector; 2) Continued engagement with various stakeholders.” “Development of a multi-year capacity building plan for a set of organizations and individuals.” “As an African organization, UP is more sensitive to these issues and has experienced them first-hand itself. But as it develops its partnerships in the rest of Africa, it will also have to figure out how to add value to the activities of local units rather than compete against them.” “The entire food policy analysis development model needs to change, realizing that the current ‘ecosystem’ involving international researchers, local research institutes, African governments, bilateral and international donors, foundations, and civil society is not operating as effectively as it could be.” “Use the local knowledge more effectively.” Source: FSP IL Mid-Term Final Evaluation online survey. Q35. Respondents were stakeholder categories 1 (FSP IL staff) and 6 (FSP IL and AA/Buy-in management staff). These should be subdivided by MSU, IFPRI, and UP. 33 evidence-based policy research, there is a need for FSP IL to identify clear avenues for coordination and collaboration with these institutions for two reasons: 1) the type of data and analysis that the project is prioritizing is what the governments need to develop their second-generation national agricultural investment plans that each government with or without a CADDP compact must report on in the next two years; and 2) to ensure that the national leadership is in place to lead these improved policy processes over time. Best Practice: The ET observed one example of best practice in Tanzania, which it feels could be usefully scaled up to the other countries where there are FSP IL Mission funded AA’s and Buy-ins. The key ingredient to its success seems to be high levels of investment of the different development partners in the process. Some other ingredients of its success have been the presence of a strong FSP IL COP who is “embedded” in the lead Ministry for Agriculture half-time; a Minister who understands and is interested in the concept; and a strong USAID Mission AOR and co-AOR who support the concept by encouraging the other projects they manage and other donors they work with to attend but keep a low profile at the PAG and AAPC meetings. The model is likely to be most relevant and replicable in countries where there is a certain level of trust between the government and the development partners and private sector. It is less likely to be replicable in a situation where the NGO and private sector partners are suspicious of government control. EQ 2.2 There was a widespread appreciation among the people the ET talked to that FSP IL and its AAs and Buy-ins have been highly effective in developing the initial capacity of their key government, private sector, and civil society partners. To date, however, FSP IL’s organization and implementation of this capacity strengthening has been ad hoc and based on the personal experiences and professional networks of the lead researchers in specific countries. While this type of generalized capacity building was quite appropriate in the early exploratory phase of the FSP IL, it is less appropriate going forward. EQ 2.3 To date, FSP IL has not developed a focused strategy for how to strengthen the capacity of local stakeholder groups as partners in food policy reform. However, there is a great deal of evidence from the field visits that these local stakeholder groups are quite interested in being informed and have access to social media applications and NGO programs that would facilitate this communication. EQ 2.4 Capacity Delivery Methods: There is wide variation between stakeholders in terms of which capacity delivery methods worked most effectively for them; this effectiveness changed as they became more familiar with the data and the concept of evidence-based policy research. A feedback mechanism for stakeholders to give better feedback on which capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities is needed from all levels of stakeholders, including local stakeholder groups like farmer groups, NGOs, parliamentarians, and other civil society groups. During the evaluation, the team observed many new methodologies that are being used to communicate with farmers via smart phones and social media that could easily be called up to incorporate simple online feedback surveys to better link FSP IL to some of its key stakeholder groups in its focal countries. Synergies Between Partner Strategies: It is possible to see strong synergies between the three consortium partners in terms of their approach to building the capacity and collaboration with the national policy research centers and their approach to two major regional groups who are tasked with building country level capacity for evidence-based food policy research and analysis (ReNAPRI and ReSAKSS). To date, however, each partner has tended to execute capacity strengthening activities on its own. Two areas 34 where these potential synergies could be better linked for better policy influence are the current and projected outcomes of their capacity building approaches vis à vis: ● The national policy research institutes; and ● The regional capacity building programs – ReSAKSS and ReNAPRI. 4.2.3 Recommendations EQ 2.1 LOP: FSP IL needs to consider the feasibility of working with bilateral USAID Missions in the countries where it has Mission-funded AAs and Buy-ins to help some government partners related to all of the FSP IL’s Mission-funded projects attend the next FSP IL-facilitated AAPC in Tanzania and, while there, to meet with the members of the FSP IL facilitated PAG. This is an activity that might be usefully coordinated with the USAID-funded regional AGRA and Africa Lead projects, which have contributed to the success of the Tanzania AAPC and PAG model. Given that the USAID bilateral Mission helped the first AAPC garner support from other donors, it might be wise to encourage some of the USAID Mission staff to also attend if resources permit. Future LWAs: Future LWA cooperative agreements need to spell out USAID/BFS’ expectations for coordination with the Mission, joint coordination of activities, and M&E with any Mission-funded AAs or Buy-ins that result from the LWA. Future LWAs: The proposals for new Mission-funded AAs and Buy-ins need to be written with explicit linkages between the AA and Buy-ins and the LWA to facilitate joint capacity building, coordination, strategic planning, M&E, and reporting with the component teams associated with the LWA. EQ 2.2 LOP: Develop a comprehensive capacity strengthening strategy for FSP IL that informs and is informed by a complementary set of national-level strategies in the Mission-funded AAs and Buy-ins. Specific sub￾recommendations for achieving this include: 1. Conduct a comprehensive mapping of the current capacity strengthening activities supported by FSP IL and the Mission-funded AAs and Buy-ins in each of the target countries that identifies which capacities the key partners need in order to affect more rapid policy change; 2. Utilize the information generated on capacity needs identified by the comprehensive mapping to work with local partners (including the AAs and Buy-ins) to develop country-specific and stakeholder/capacity building strategies; 3. Focus the strategy on a limited number of critical partners in each country including: a) its key ministry partners; and b) the university-based think tanks and promising non-university think tanks in countries like Myanmar where the universities do not have the legal independence and autonomy that they do in other countries where the FSP IL is engaged; 4. Pay careful attention to conducting a detailed baseline assessment of key capacities that the critical partners will need to sustain their involvement in evidence based policy research, analysis and advocacy; 5. Encourage: a) The AA and Buy-in Projects to report on the execution of their country-level capacity strengthening strategies to the Mission at least twice a year; and b) FSP IL to report on each country’s execution of its capacity strengthening strategy in its semi-annual and annual reports to USAID/BFS; and 6. Ask MSU to co-lead the capacity strengthening strategy with part of the UP team (as envisioned in the original FSP IL cooperative agreement proposal). Future LWA: Future cooperative agreements based on the FSP IL LWA model need to develop a systematic approach to human and institutional capacity development. Specific sub-recommendations include: 35 1. Give priority to key ministry partners and university-based policy institutes whenever possible; 2. Have a planned strategy to build the core capacity of these institutions in collaboration with other regional and donor-funded programs that support them; and 3. Track current and project synergies among these partners and some of the most critical regional entities and programs (like ReSAKSS, ReNAPRI, AGRA, Africa Lead). EQ 2.3 LOP and Future LWA: Given the cost, security, and clearance problems that can be associated with sending an enumerator back to debrief a village on the results of a survey conducted in that village, consider discussing some of the results of previous surveys conducted in other communities. LOP and Future LWA: Given the cost and difficulty of building effective engagement strategies with local stakeholder groups on a five-year IL, future programs should consider ways to reach local stakeholder groups with appropriate summaries of their major research findings through the NGOs that are already working with these groups. EQ 2.4 LOP: Encourage the three consortium FSP IL partners to develop joint work plans for their FSP IL activities with the national policy research centers in the countries where they work in order to encourage potential capacity building synergies (led by MSU/UP) and how these activities could better support and be supported by ReSAKSS. LOP: Encourage the three consortium partners to develop a joint work plan for their FSP IL activities with ReSAKSS and the national and regional CADDP processes (led by IFPRI/UP) and how these activities could better support and be supported by the national policy research centers. LOP: Develop a simple mechanism for stakeholders to give better feedback on which capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities. 4.3 PROGRESS TOWARD POLICY OUTCOMES (EQ CLUSTER #3) 4.3.1 Findings 4.3.1.1 EQ 3.1: To what extent have the five different activity components contributed toward policy change at the global, regional, and country level? In what ways could these components be expanded or improved to better support policy influence or policy change? What is country level policy change/influence and how could this be expanded? How have C3, C4, and C5 contributed to more effective or rapid policy change at the country level? EQ 3.1.1: To what extent have the components contributed toward policy change at the country-level? Stakeholder Feedback from the Online Survey: The majority of the stakeholder respondents to the online survey (15 out of 17, categories 1-6) agreed with the statements that: ● ● “FSP IL works closely with national governments (89 percent), regional partners (79 percent), and civil society and local stakeholder groups (90 percent) to promote rapid policy change at the national level;” and “FSP IL works closely with regional partners and stakeholders (79 percent), national governments partners (88 percent), and civil society and local stakeholders (90 percent) to promote rapid policy change at the national level” (Table 9). 36 Table 9. Respondents Who Agreed that the FSP IL Works Closely with the Following Partners and Entities to Affect More Rapid Policy Change at the National, Regional, and Global Levels Partners and Entities National Level Regional Level Global Level Regional Partners and Stakeholdersa 79% (26/33) 71% (20/28) 80% (20/25) National Government Partnersb 88% (29/33) 80% (20/25) 72% (13/18) Civil Society and Local Stakeholders (Framer Groups, NGOs and Other Civil Society Groups)c 90% (27/30) 68% (15/22) 63% (10/16) a Source: FSP IL Mid-Term Performance Evaluation online survey, Q42. b Source: Q43. c Source: Q44. It is not surprising that the majority of the online survey respondents felt that the C1/C2 team, which is the combined team tasked with building country-level research and capacity, agreed with the statement that the team has contributed toward rapid change in policy at the national level (Table 10).88 Seventy￾eight percent (7/9) of these respondents agreed that C5 had contributed toward rapid change in policy at the national level, while about 70 percent agreed that C3 (9/13) and C4A (7/10) had. When asked to rank the six FSP IL components in terms of their contribution toward rapid change in policy, 74 percent (7/11) of the respondents ranked the C1 team as number one or two in terms of its contribution to effective or rapid policy change at the country-level. Forty-six percent (6/13) of the respondents ranked the C2 team as number one or two.89 Table 10. Respondents with an Opinion Who Agreed That the Following FSP IL Components Have Contributed to Rapid Change in Policy at the Country Level and Their Rankings of Components Component Teams Respondents Who Agreed Percent That Ranked Component First or Second C1 100% (12/12) 74% (7/11) C2 83% (10/12) 46% (6/13) C3 69% (9/13) 23% (3/13) C4A 70% (7/10) 10% (1/10) C4B 55% (6/11) 17% (2/12) C5 78% (7/9)a 13% (2/16) Source: FSP IL Mid-Term Performance Evaluation online survey, Q38 and Q39. Respondents were from stakeholder categories 1, 3, and 6. One indicator of the successful contribution of the C1/C2 activities to national-level policy influence is the number of Mission-funded AAs and Buy-ins it has generated as well as a disaggregated analysis of the official FSP IL indicators that were collected by each team for the PMP; the column that tracks the measurable outputs and outcomes of the C1/C2 team outstrips all of the others (see Annex IV.B). The ET contends, however, that a simple focus on the achievements of any one team with “core” fund overlooks that: 88 The C1 and C2 teams also had the largest number of online survey respondents: stakeholder categories 1-6. 89 Source: FSP IL Mid-Term Performance Evaluation Online Survey, Question 39. Respondents were from stakeholder categories 1 (FSP IL staff), 2 (AA/Buy-in staff), 3 (FSP IL partners), and 6 (FSP IL and AA/Buy-in management staff). 37 ● ● The activities that were attributed to the C1/C2 team were often supported by funds and researchers that are now attached to the C4A, C4B, and C3 teams (Text Box 6; Annex V.A); that backstop the Mission-funded AA and Buy-in projects; and Many of the follow-up activities that contributed to the C1/C2 activities influencing policy were funded by the Mission-funded projects that they help create (not C1/C2). Project Documentation: To date, the principal source of information about the current and projected policy influence of the FSP IL component teams are FSP IL’s semi-annual and annual reports and the component teams’ presentations at the all FSP IL meeting in March 2017. This documentation provides a great deal of anecdotal evidence that some of the activities supported by FSP IL’s regional and global component teams (C3, C4A, C4B) have influenced or are likely to influence the priorities and investments of the national governments in every country with a Mission-funded AA (Malawi, Mali, Myanmar, Nigeria, and Senegal) or Buy-in project (Tanzania and Zambia), plus South Africa and Ethiopia (Annex V.C). In some countries—like Myanmar and Tanzania—it is already possible to see situations where this research is influencing new investments by international donors (including USAID) and the government (Annexes V.A, V.B, and V.C). Unfortunately, it is hard to fully document this influence with the project’s existing M&E data, semi￾annual and annual reports. This is because, except for two outcome indicators, FSP IL M&E data focuses on activities that have been totally or partially funded by the FSP IL core funding. Since most policy influence is the result of a combination of FSP IL core funding and the FSP IL facilitated AA and Buy-in projects—which the FSP IL core component teams helped create—the current M&E data is not an accurate reflection of either the direct or the indirect influence of the project. Country Case Studies: The evaluators were able to better document the broad categories of policy influence through research, analysis and reporting, capacity strengthening, stakeholder consultation/policy debate, and policy review and formulation for the two case-study countries (Text Boxes 7 and 8). EQ 3.1.2: To what extent have the components contributed toward policy change at the regional and global level? According to the online survey: ● ● ● Seventy-one percent (20/28) and 80 percent (20/25) of the respondents agreed that FSP IL component teams work closely with regional partners and stakeholders to affect more rapid change at the regional and global level, respectively. Eighty percent (20/25) of the respondents agreed that FSP IL woks actively with national government partners to affect more rapid change at the regional level and 72 percent (13/18) at the global/international level. Sixty-eight percent (15/22) agreed with the statement FSP IL worked closely with civil society and local stakeholders to affect more rapid change at the regional level and 63percent (10/16) at the global level. Documentation: The project documentation corroborates the perception (from the online survey results) that the component teams and Mission-funded AA and Buy-in projects have played in helping the national governments work with ReSAKSS and comply with their Malabo commitment (Table 6, Section 4.2). Unfortunately, these regional activities and the resulting regional influence are not currently being tracked by the FSP IL’s M&E system so it is difficult to provide any more fine-tuned analysis of this influence by FSP IL or the Mission-funded AAs or Buy-ins than what is presented in Annex V.C, which is based on the FSP IL component team’s presentations at the all-FSP IL meeting in March 2017. 38 Text Box 7 Myanmar Case-Study: Type of FSP IL-Facilitated Policy Influence Phase I (October 2012-September 2014): Initial phase funded by a USAID Burma Associate Award to Food Security III Cooperative Agreement and deepened through FSP IL core-funded (bridge) activities in Myanmar: Research, Analysis, and Reporting: Executed a series of 6 background studies and a synthesis paper that provided the first up-to-date information on Myanmar’s agricultural sector in 30 years. Capacity Strengthening: Helped strengthen the capacity of its local research partner (CESD) to execute the studies and undertake stakeholder consultations; and Stakeholder Consultation/Policy Debate: Conducted a series of debriefings and presentations to donors that build their understanding of the research and its implications for policy review and formulation. Policy Review and Formulation: The principal result of these first-generation studies was to demonstrate that rice production was a poverty trap for small holders.90 These initial studies (Annex V.A): Provided the Myanmar government with first evidence-based overview of major agricultural systems; Identified a major constraint to smallholder incomes and agricultural growth was the current government’s emphasis on paddy rice; Identified critically underfunded public goods such as research, extension, agricultural statistics; Encouraged the opposition party (now in power) to include agriculture as a pillar in its manifesto; Provided USAID/Burma with the justification to make agriculture a major component of its Mission portfolio; Built on the initial diagnostic by providing evidence on the current situation and constraints the of pulses and beans sector (Myanmar’s most important export sector by value); Provided regional comparative analysis of the potential role of smallholders/contract farming (aquaculture) as an alternative model to large scale plantations; Built the institutional capacity of the CESD’s policy research center for agricultural policy and research; and Created a pool of FSP IL research that has caught the attention of civil society, the donor community, and agribusiness, and made USAID a major player91 in food policy research in Myanmar. Phase II: September 201492-April 2016: In the face of the government’s recognition that the research studies “might possibly be correct but [be] unwilling to take action on it,” [Centre for Economic and Social Development] CESD and FSP IL researchers backed off and turned their attention to developing a second generation of: 1) more detailed research analysis and reporting at regional level (Mon State) as well as more in-depth value chain studies especially aquaculture); combined with 2) additional capacity strengthening; and 3) stakeholder consultation and policy debates that were primarily focused on the international donor community (Annex V.A). It was in this same time period that these activities, which had been supported by the FSP IL, were intensified and expanded through a Mission-funded AA and additional funding from the LIFT multi-donor trust fund. Phase III: April 2016-December 2016:93 The landslide victory for the NLD party in November 2015 opened the door to a new generation of policy reforms to support more diversified small-holder agriculture. The new government’s mandate started in April 2016 which explains the startup date of April 2016 for this next period. In preparation for the transition, the USAID Mission asked the FSP IL researchers to participate in a multi-donor working group tasked with developing a series of policy briefing papers for whichever political party took power in the up-coming government election in November 2015. (Annex V.A). Instead of focusing on a series of briefing papers, this working group—led by one of the most respected economic advisors in the country under the previous regime as well as today—ended up focusing exclusively on agriculture and produced the country’s first evidence-based white paper on agricultural policy in 30 years. All major stakeholder participants in this process stated that this paper would not have been possible without FSP IL’s leadership. FSP IL contributed: 1) by bringing in experts familiar with various examples of best practice from other Southeast Asian countries; and 2) through findings of the high-quality FSP IL diagnostic and value chain studies in Year 1 and the first household livelihood survey in Mon State which was completed in December 2015., This new agricultural strategy white paper was completed in April 2016 helped inform a new generation of policy reforms to support more diversified smallholder agriculture. This period was characterized by more active engagement of the FSP IL researchers with the union government on a host of new policy issues including a new agricultural policy, published in January 2016, and a draft Agricultural Development Strategy, currently undergoing regional consultations. During this same time period, the AA and FSP IL helped advise the government on the creation of the first agricultural policy unit in the new Ministry of Agriculture, Livestock and Irrigation formed from three separate ministries under the previous government (Annex V.A). Phase IV: January 2017-Present. The approval of a new agricultural policy and Agricultural Development Strategy (ADS) marks the most recent period of FSP IL work in Myanmar during which the FSP IL and AA activities are focusing on a fourth generation of studies that are being executed in close collaboration with Ministry of Agriculture, Livestock and Irrigation (MOALI) (Annex V.A). In contrast to the earlier 90 The military government in power at that time had a heavy bias towards paddy rice production. 91 Source: FSP IL Mid-Term Performance Evaluation KII interviews with non-U.S. donor staff related to the LIFT multi-donor trust fund. 92 Receipt of funding from AA marks beginning of this phase. 93 The landslide victory for the NLD party, November 2015, opens the door to a new generation of policy reforms to support more diversified small-holder agriculture. The new government’s transition started in April 2016 which explains the startup date of April 2016 for this next period. ● ● ● ● - - - - - - - - - 39 Myanmar Case-Study: Type of FSP IL-Facilitated Policy Influence periods, all of the new FSP IL and AA sponsored policy studies are being executed in close collaboration with the MOALI staff as well as those from CESD. During this same time period the FSP IL has supported a series of studies as well as training programs to help build the capacity of the MOALI staff to support food policy reform including a MOALI requested study on reform of the agricultural research system. Source: FSP IL Mid-Term Performance Evaluation Case Studies and Annex V.A. Text Box 8 Tanzania Case-Study: Type of FSP IL-Facilitated Policy Influence uring the first and second year of the FSP IL in Tanzania, the C1/C2 team used a series of diagnostic studies and stakeholder consultations to explore some of the “conventional wisdoms” underlying the Tanzania government’s food policies. In Tanzania, where the government was already highly open to policy reform, this influence was quicker. During the two years of the FSP IL, the component team (Annex V.B): ● Research, Analysis, and Reporting: Executed a series of studies that explored some of the “conventional wisdoms” underlying current policies for the crop levees (aka nuisance taxes), land policies, and post-harvest policies in Tanzania; ● Capacity Strengthening: Helped strengthen the capacity of the consultants and faculty and students from the Department of Agricultural Economics at SUA who were tasked with executing the study; ● Stakeholder Consultation/Policy Debate Stakeholder Consultation: Conducted extensive stakeholder consultation with local governments, the major private-sector actors, the principal international donors and donor-funded projects which included: ­ A series of stakeholder start-up and debriefing presentations that the researchers usually organized around the studies; ­ Regular one-on-one debriefings by the component team researchers with their ministry colleagues; and ­ Presentations/discussions of the research to the AAPC, the PAG, and the PAPAC which got its initial start-up funding from the FSP IL; ● Policy Review and Formulation: Based on this initial work,94 the FSP IL component teams argued that: ­ The local governments would never agree to the national government lowering the crop levees until the LGAs had access to a better method for collecting the taxes; ­ That the pace of change for small holder and mid-sized producers far outpaced the “land grab” by foreign governments that had the greatest visibility in the national press; and ­ That there was very little processed food being consumed in Tanzania’s rural areas. These results fed into a series of informal and formal ministry debriefings that were usually facilitated by the senior FSP IL staff that headed the research teams along with their junior colleagues from the lead agricultural ministries. They were also presented at the country’s first national policy conference, which was initially supported by the FSP IL and then scaled up by additional funding from two USAID-funded regional projects (AGRA and Africa Lead) and other donors. The results were also used to inform the one-on-one technical assistance that the FSP IL COP provided to senior management through his post as a half-time “embedded” advisor to the Ministry. The evaluators have identified six important areas where these results have influenced and/or are likely to influence key policy debates. (Annex V.B): Since FY 2013, the FSP IL FSP IL’s C4A team has conducted a series of studies that were executed in close collaboration with their local partners (SUA and Tanzania National Bureau) that have influenced the ongoing land policy reforms process in Tanzania that is trying to address youth access to land, employment, and migration (Annex V.B). A series of pre-research consultations and studies by the FSP IL’s C4B team I informing the revision of the country’s agro-processing strategy and how it fits into the overall industrialization strategy, which in turn (Annex V.B): - Is expected to influence the content and spatial location of investment incentives to promote agro-processing investment; which also - Led to a recent request from the Ministry of Finance to conduct a study of trade policy for the oilseeds sector, which has seen a boom in small- and medium-scale investment in processing and is now affected by conflict between these firms and large firms reliant on important of crude vegetable oils. One of the most visible (in terms of press coverage) success stories of policy influence is the initial start-up research that the FSP IL supported on the LGA cess producer, tax which included a one-district pilot study to determine the feasibility of using a new e-payment technology to strengthen the transparency and rigor of local tax collection as a basis for lowering the overall levy on key crops while at the same time maintaining revenue for the local government agencies (LGAs). Since 2014 when this research got additional support from the Mission-funded AA ASPIRES project, this research has (Annex V.B): 94 Which was funded under Components C1, C3, and C4A and a small AA from the Mission. ● ● ● 40 Tanzania Case-Study: Type of FSP IL-Facilitated Policy Influence - Led to the scale up of the new technology from one district to over 30 districts in FY 2017; and - Set in motion a huge cross-country and cross-donor debate on how the successful dissemination of this new technology must be linked to reforming the national policy on cess agricultural tax levies by lowering the levies on key crops, especially those like paddy rice that directly affect small holders. This information continues to be used to inform ministry and private sector lobbying to reform the Local Government Finance Act (LGFA), which is currently before the Tanzania parliament. A fourth area of policy influence involves the C4B/ASPIRES project’s joint research and stakeholder interviews with crop boards which revealed strong demand among stakeholders for continued board action but with different objectives and done more efficiently. This information, which was shared with the lead ministries through a series of stakeholder consultations and one-on-one meetings with the ASPIRES COP, led to a broad consensus to: 1) consolidate boards; 2) reduce overhead by sharing office space and other infrastructure; 3) reduce spending by reducing the number of representatives on each board; and 4) revisit the issue in five years to decide whether further reforms are needed (Annex V.B). The ASPIRES COP and C4A team leader led 15 of the mid-level ministry staff—who had participated in earlier FSP IL and ASPIRES sponsored trainings and who had (with FSP IL and ASPIRES support) organized themselves into the Associate Policy Analysts (APA) group—in a critical review of the second draft of the 10-year Agricultural Sector Development Strategy II. Because most of the APA associates had participated in the FSP IL-sponsored trainings and conferences, they were able to better defend some of the new strategies that the ADSII was promoting, which were based on FSP IL and ASPIRES sponsored and co-sponsored research (Annex V.B). A sixth area of policy influence involves a series of FSP IL and Buy-in co-facilitated studies (with the SERA project) that led to the establishment of the new Market Intelligence Unit (MIU) in the Ministry of Agriculture, Livestock and Fisheries (MALF). The ASPIRES project (with backup from C4B) is continuing to help the unit get operational by hiring qualified staff, launching market research as part of the MIU, and using the MIU as a platform for influencing trade and other marketing policies and for integrating innovative technology or crop forecasts (depending on funding and collaboration with others at MSU) (Annex V.B). A seventh area of documented policy influence is related to the use of the PE modeling that the FSP IL has supported with a faculty member at SUA to model the impact of the drought in 2015/16 on regional trade of maize and the impact of a trade ban by the Zambia government on regional prices. This is a good example of how the regional PE training that the FSP IL is conducting with ReNAPRI can potentially influence decision makers (Annex V.B). The team is expecting to run an export-ban scenario for Tanzania to influence trade policies by illustrating the impact on prices and trade flow within the region. Although it is still too soon to tell, the ASPIRES team is excited about the potential impact of using the C3-developed PMCA (Policy, Management, Constraints, Actions) tool to accelerate the rate of policy review and formulation on several issues (Annex V.B).95 Source: FSP IL Mid-Term Performance Evaluation Case Studies and Annex V.B. The same documentation shows that the FSP IL has been very active in presenting and co-presenting (with local partners) the results of its country-level and regional research findings (Annex V.C). Some of these results have been incorporated into important donor documents for AGRA and other regional USAID and non-USAID projects (Annex V.C). The numbers of workshop and conference presentations are currently being tracked but they are not broken down by international organization. Without this type of disaggregated analysis, it is difficult to track the influence of these presentations or other types of information sharing at the international level. However, interviews with senior researchers and emails from international donors that individual staff members shared with the ET indicate there is some evidence that this is occurring. 95 The output of this exercise was a guidebook which is currently being tested. This guidebook, and its associated tools, is designed to help agricultural policy practitioners improve the effectiveness of their engagement with policy systems in developing countries. It provides practitioners with conceptual tools and practical strategies designed to improve the quality of policy practitioners’ efforts to influence complex agricultural policy systems. It develops a strategic approach to policy engagement termed the PMCA approach, and provides concrete applications and examples of this approach to help users apply the approach to their own policy domains of interest (Source: http://foodsecuritypolicy.msu.edu/uploads/resources/FSP_Research_Paper_49.pdf) 41 Case-Study Evidence-Myanmar: One under-documented area of international policy impact is the influence that FSP IL and Mission-funded projects are having on “development thinking” in the national development community. This point was underscored in the ET’s interviews with three senior staff related to the Department for International Development (DFID), the European Union (EU), Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), and four staff related to the LIFT multi-donor trust fund in Myanmar. In separate interviews, three non-USAID donors and two representatives of USAID-funded projects cited the impact of these studies on evidence-based policy change at the country level as nothing short of revolutionary. This was reflected in the associated rankings of country level impact (usually very high). One donor representative cited how the studies offered a “ray of hope that woke us up to how we needed to shift our thinking from disaster recovery to new options for growth and development.” Another donor representative stated, “The [author’s name redacted] paper was super important. You could not do a literature review. That was the first paper that tried to pull things together. For the larger community—e.g., for the donors and the Burmese—it was also very, very important for USAID. It is a tough game when you go up with the World Bank to a planning paper. This paper established USAID as a serious contributor to the national [and donor] discussions on agricultural policy.” EQ 3.1.3: In what ways could these components be expanded or improved to better support policy influence or policy change? What is country level policy change/influence and how could this be expanded? The respondents to the online survey identified a number of ways that FSP IL’s country and regional￾level support could be improved to better support policy influence and policy change (Annex V.D). Currently, at least two of the major component teams (C4A and C4B) function as an amalgam of sub￾groups, not as cohesive teams.96 The six activities for C4A, for example, are executed by three separate sub-groups: one from IFPRI, one from MSU, and one from UP, with little overlap. Certain aspects of this structure are good, in that it encourages high levels of investment by the sub-team leader in his or her work. Other aspects are less advantageous, in that it reduces the potential synergies among the different sub-teams, like linking the insights from one sub-team—such as the C4A land research and advocacy team that works within Nigeria and Ethiopia—with the C4A land research and advocacy team that works under Activity 3 in Tanzania, Malawi, Kenya, and Zambia on land and youth. Many staff that were interviewed in person and responded to the online survey felt that a more cohesive structure for the teams—with adequate time allocated to the team leader—might make it easier for these groups to be better integrated into the Mission-funded Buy-ins and AAs and vice versa. The C3 team is already operating cohesively and should be encouraged to maintain a more evidence￾based system for strategic planning and coordination with the Mission-funded AAs and Buy-ins as it moves forward with its plans to scale up its applied country-level research over the LOP. Many of the key informants argued that the C3 team is the strongest team in that it functions with greater horizontal integration among its members in terms of having a core strategy, jointly conceived work plan, and regular meetings to update one another on activities. Even here, however, the evaluators have the impression that its horizontal integration could be strengthened by having a written strategy that better aligns with each of the other countries that has a Mission-funded AA or Buy-in. The cooperative agreement outlined four activities that C3 would lead. Three of them are already well developed and much appreciated by the other FSP IL team members, if not yet by a wide range of country-level stakeholders, due to their focused interventions in only four countries. 4.3.1.2 EQ 3.2: How effectively has FSP IL communicated policy analysis, recommendations and options to policymakers or others in a position to influence them, and in what ways could this communication be improved (including content, target, etc.)? 96 Source: FSP IL Mid-Term Performance Evaluation FGD and KII in Washington, DC and in the two case-study countries. 42 Most of the stakeholders interviewed felt that FSPI IL has been very effective in communicating policy analysis and recommendations but that the project needs a formal communication strategy; they made a number of recommendations for what could be included in that strategy. Although the concept of communication was a key theme running through the original project proposal, it was not identified as a key component of the project nor accorded an institutional “lead” as were the other critical components such as capacity building. To address this issue (which was identified by the internal mid￾term evaluation), the project hired its first full time Communication Specialist in December 2016. Some of the Mission-funded programs (like Tanzania) have also hired communication specialists. While these new hires appear to be improving FSP IL’s communication processes, the project does not have a fully integrated strategy that includes a mechanism for regular monitoring of and how these messages are being received and used by each of the FSP IL’s major national, regional, and global partners. Current Patterns of Communication with Donors: Most donors interviewed seemed to be satisfied with the way the component teams, AAs, and Buy-ins were communicating with them. Washington, DC-Based Staff: Although the Washington, DC-based USAID staff emphasized the importance of the FSP IL-generated articles in international refereed journals and briefing papers, they said that their principal ports of communication were (Annex IV.G): ● ● ● Periodic presentations by the team leaders during debriefings at USAID Missions or USAID/BFS; Other workshops and conferences where the team presented; and New briefing papers and paper summaries promoted by the FSP IL Communication Officer as part of her global strategy to improve communication. When asked to rank the importance of different communication outlets, most of the Washington, DC￾based staff ranked FSP IL’s workshop presentations and briefing papers as numbers one and two. Field-Based Donor Staff: In contrast, most of the field-based donor staff ranked their personal relationship with the in-country FSP IL Country Director highest, followed by researcher presentations, briefing papers, and more detailed reports. When staff were asked to discuss or justify their rankings, they shared that the types of communication that worked and was most effective depended on where an individual was in terms of understanding the issues. When staff are new to the material, they are most likely to be influenced by a presentation at a donor-funded workshop or, in the case of Tanzania, through the AAPC. Staff also emphasized that it was important to understand that what happened in the presentation was far more than the simple transfer of information: ● ● ● First, the fact that the presentation was occurring in a policy context (i.e., a workshop or conference) provided a certain group validation of the information that was being presented so that it was not just them getting the information, but also their colleagues from the wider donor community; Second, this type of presentation provided a social connection to the researcher as well as to the COP of the Mission-funded AA or Buy-in; and Third, in the process of connecting with the COP of the Mission-funded AA or Buy-in and the researcher, the donor expert usually got handed a briefing paper as well as information on how to access other documents on the study through FSP IL’s central website and the local AA and Buy-in websites. The same staff noted that once they became more actively engaged in policy review and advocacy, their communication channels tended to shift. At this juncture, their chief conduits of communication with the project tended to continue to be the in-country COP or deputy COP, followed by some of the inter￾donor/inter-ministerial working groups that appeared in both case-study countries. Since the in-country 43 COPs were members of these groups, they continued to provide a social matrix for maintaining communication with FSP IL component teams as well as the AAs and Buy-ins about specific topics. Many of the staff noted that although they liked the FSP IL briefs, they also needed access to the bigger, more detailed reports once they shifted to active policy engagement and advocacy since these provided the types of details needed to support new program designs. Current Patterns of Communicating with Policy Makers at the Country Level: The most frequently cited communication channels to people in positions to change policy according to responses in the case-study countries were (Annex V.E): ● ● ● ● Formal presentations of interim results and final reports to senior government partners; Formal presentations at workshops, conferences, or a university general meeting; In-country working groups or task forces, small group meetings, and one-on-one meetings; as well as Direct and indirect distribution of briefs, talks, and articles that the teams had published in refereed journals. These meetings and presentations were almost always done in concert with a local research partner and the COP or deputy COP of the Mission-funded AA or Buy-in. It should be noted that FSP IL is just at the beginning of developing a system to measure different categories of stakeholder feedback regarding the relative effectiveness of different communication channels in specific countries. Current Patterns of Communicating with Policy Makers at the Sub-State Level: One under￾documented impact of FSP IL and Mission-funded AAs and Buy-ins has been their impact at the sub￾national level of states and regions and in the private sector. In Myanmar, the principal channel for communicating the Burmese-language version of the FSP IL briefing papers and reports has been the staff of the national partner CESD, which is often solicited by the states to assist with the development of their agricultural strategies. These tend to mirror the national strategies but are written in local languages and through a completely separate consultative process that involves state and sub￾state/regional and sub-region-level actors. In contrast to the major stakeholders involved in the national￾level policy analysis and review, the state strategies depend heavily on short briefing papers in the local languages as well as the larger English-language versions of key documents. When queried about this ranking, some of the key stakeholders underscored the importance of having documents in both languages since the translations were often confusing. 4.3.1.3 EQ 3.3: In what ways has the FSP IL worked closely with other partners and stakeholders at the country, regional, or global level to effect more rapid policy change (for example, ReSAKSS, AUC CAADP focal points, and Africa Lead)? What opportunities exist to affect more rapid policy change with existing stakeholders or new stakeholders? As discussed in Section 3.2 of this report, several of the senior FSP IL staff are using FSP IL materials in conjunction with some of the leadership roles they have been elected to in ReSAKSS and ReNAPRI. To date, however, this collaboration has tended to be personality driven rather than something that is institutionalized through FSP IL. It is also clear from the reports that over the previous three years, certain senior staff and institutions have focused on some specific regional groups more than others (see Section 3.2). For example: ● UP has showed strong leadership in providing a bridge between FSP IL and AUC CAADP through its Director. She works in close collaboration with the IFPRI Director for Africa, who was very instrumental in the creation of ReSAKSS; 44 ● ● ● UP C3 Senior Researcher and Director of BFAP, Ferdi Meyer, is a board member of ReNAPRI and through FSP IL has supported training and capacity building in PE and policy impact modeling, foresighting, and the maintenance and expansion of databases; MSU C4A Team Leader, Tom Jayne, has developed a close working relationship with ReNAPRI and is a member of the Board of Directors for IAPRI in Zambia; and Certain country directors and COPs have been proactive in reaching out to the regional offices of certain groups like Africa Lead through their regional and national-based offices. While there is a great deal of anecdotal evidence from the semi-annual reports and the March presentations that this regional collaboration is occurring (Annex V.C), it is very hard to track for specific regional organizations at the regional level or by country. Without this information, it is hard for the FSP IL administration to develop a strategy to affect more rapid policy change with existing stakeholders or new stakeholders. 4.3.1.4 EQ 3.4: What MSU/IFPRI/UP FSP IL organizational models appear to be more or less effective in supporting policy change at the country-level? Is a separate country office more or less effective than an office integrated with a host country institution? Is an expatriate, local, or mixed team most effective in credible policy analysis and dialogue? How can these organizational models be strengthened to better support policy change? FSP IL has been willing to accommodate different institutional models for its in-country interventions. Consequently, there has been a great deal of interest in getting stakeholder feedback on whether one model has been more effective than another in making sure that policy change occurs. Most of FSP IL country programs started with one of two basic models in terms of their relationship with their lead host-country government partner (Table 11). The first is one in which the FSP IL activities, and the activities of the AA and Buy-in awards that they support, are literally embedded in a host government institution. This is the case for Tanzania and Malawi, where the initial activities were all embedded in the Ministry of Agriculture; Zambia, where the initial project office was embedded in an independent think tank, IAPRI; and South Africa, where the project is embedded in the Bureau for Food Policy and Agricultural Research. A second model is one in which the FSP IL attempted to have a more independent project identity from the start, as in Nigeria, Senegal, Myanmar, Mali, and Rwanda. Which model was chosen for an AA/Buy-in in the first three years of FSP IL was usually the result of a wide variety of factors, the most important of which was usually related to the institutional home of the previous policy project being built on in terms of relationships with the key government partner (Table 11). An Embedded Model: The consequence of being fully embedded is that the Ministry in which they are embedded becomes the face of the project during its first two years (Table 12). Once the Tanzania AA ASPIRES project was funded at the end of the second year, the project acquired a separate off-site support office, but it was the Ministry that was still the face of the activity vis à vis the government. In Zambia, IAPRI was the face both before and after getting Mission funding. The chief advantage of being fully embedded is that it increases the likelihood that the government will own the initial results of the FSP IL component teams as well as the activities or any expansion of these activities through an AA or Buy-in. At the same time, this high degree of embeddedness can be a disadvantage if there is a history of regional or political division in a country where aligning a program with the government reduces the researcher’s willingness and ability to conduct unbiased, rigorous, evidence-based policy research and analysis. It also makes it difficult for the project to be seen unbiased when it comes time to disseminate the findings. 45 Table 11. Different Organizational Models That the FSP IL Has Used to Support Policy Change at the Country Level, the Link to the Lead Ministries of Agriculture and Challenges Moving Forward Country Institutional Host Why This Arrangement Was Chosen Link to Lead Ministry Strengths Challenges Going Forward Embedded Tanzania Ministry of Agriculture FP IL built on an earlier MSU￾managed Gates Foundation Project COP is embedded advisor in Ministry Facilitated early, consistent communication with government, accelerating update of results FSP IL (with Ministry backstopping) efforts to register as an independent NGO in Tanzania Malawi Ministry of Agriculture FSP IL activities and AA were designed to capitalize on IFPRI’s previous projects COP is embedded advisor in Ministry Facilitated early consistent communication with government which accelerated update of results The FSP IL needs to strengthen its activities with the independent university-based food policy research institute, CARD Zambia Independent policy research institute related to the University of Zambia FSP IL activities and Buy-in were designed to capitalize on MSU’s past collaboration with IAPRI IAPRI97 is and independent policy research instituted related to the major agricultural university Facilitated early, consistent communication with government, accelerating update of results None; just ensure institutional capacity of IAPRI to continue South Africa Semi￾independent institute within the Institute for Food, Nutrition and Well-being (IFNuW)98 at the UP Both IFPRI and MSU had worked with UP in the past, and USAID was familiar with the quality of their work on other projects IFNuW Center is embedded in the major agricultural university Facilitated collaboration with South African government offices and elected officials in UP The IFNuW may need to develop the types of more autonomous financial systems they are likely to need in order to manage and increasingly diverse set of international partnerships Independent Myanmar CESD When the FSP IL started, CESD was embedded in an organization that had worked with MSU and IFPRI in the past CESD director was the official advisor to the President in his first years99 Facilitated access to the Ministry of Agriculture during a time when the government was not open to new ideas and policy changes FSP IL AA (with CESD backing) is considering registering as an international NGO to make it easier to manage an increasingly diversified portfolio with multiple funding sources and partnerships Nigeria IFPRI country office FSP IL and AA model built on earlier relationships that IFPRI and MSU had with strong university All FSP IL and AA activities are routed through the national universities Since the agricultural universities have a high degree of autonomy in Nigeria this makes them independent The universities that FSP IL and the AA are working with may need to strengthen their institutional capacity to manage outside funds in order to sustain the 97 MSU helped create IAPRI and it became a locally independent policy research institute (rather than an MSU project) in 2012-2013 at the insistence of the USAID Mission in Zambia. This was a 20-year investment in African agricultural policy institutional building and it is not sure that this can be done in a much shorter time period. 98 Most staff engaged in C3 belong to the Institute for Food, Nutrition and Well-being—a virtual institute with the capacity of mobilizing human capacity to address specific issues across the food security domain. 99 Since January 1, 2017 current COP is one of the three official advisors to the ministry. 46 Country Institutional Host Why This Arrangement Was Chosen Link to Lead Ministry Strengths Challenges Going Forward partners before FSP IL100 but related to the government types of activities they have started under FSP IL, the AA, and IFPRI Source: FSP IL Mid-Term Performance Evaluation interviews. An Embedded Model: The consequence of being fully embedded is that the Ministry in which they are embedded becomes the face of the project during its first two years (Table 12). Once the Tanzania AA ASPIRES project was funded at the end of the second year, the project acquired a separate off-site support office, but it was the Ministry that was still the face of the activity vis à vis the government. In Zambia, IAPRI was the face both before and after getting Mission funding. The chief advantage of being fully embedded is that it increases the likelihood that the government will own the initial results of the FSP IL component teams as well as the activities or any expansion of these activities through an AA or Buy-in. At the same time, this high degree of embeddedness can be a disadvantage if there is a history of regional or political division in a country where aligning a program with the government reduces the researcher’s willingness and ability to conduct unbiased, rigorous, evidence-based policy research and analysis. It also makes it difficult for the project to be seen unbiased when it comes time to disseminate the findings. An Independent Organizational Model: The chief advantage of being totally independent is a greater degree of control over staff, strategic planning, and finances. However, even the most independent FSP IL programs are embedded in some non-government partner (Table 11). This quickly creates another set of challenges, which include: ● How to build a bridge to the government; and ● How to differentiate the project’s financial systems from the NGO within which it is embedded. It is important to note that the most successful FSP IL country models based on the independent model have been the ones in which the independent program had some sort of embedded organic link to the lead ministry that facilitated its connection to the government. In Myanmar, for example, that embedded organic link was the CESD Director. During the first two years of the FSP IL activities in Myanmar, he was the official Economic Adviser to the President. This direct connection between the Myanmar AA and the government made it possible for the FSP IL component teams to conduct the types of forward￾thinking research that took issue with the conventional wisdom and planted the seed for more broad￾based policy analysis to develop better, more appropriate policies. In Tanzania, this embedded organic link occurred because FSP IL chose to work with an in-country director who was already embedded in the Ministry of Agriculture through another project funded by the Bill and Melinda Gates Foundation. This director worked for 10 years at the USAID/Tanzania Mission office, which facilitated FSP IL’s contacts with the USAID country Mission. A Hybrid Organizational Model: Taking a longer-term view of the strengths and weaknesses of the different models over time, many of the country programs that have been most successful in generating active stakeholder engagement in policy change are moving toward a more hybrid organizational model in which the project has: 100 IFPRI had a strong relationship with Ahmadu Bello’s previous project Nigeria Strategy Support Program (NSSP), which was funded; MSU had a strong relationship with the University of Ibadan through its GISAIA (Guiding Investments in Sustainable Agricultural Intensification Project in Africa) Project, which was funded by the Bill and Melinda Gates Foundation. 47 ● Certain activities and staff that are embedded under its principal local government partner, which is usually the lead Ministry of Agriculture; and ● Other activities that are more independent. Some of the MSU staff reported that a similar situation happened on one of their oldest, most successful projects in Mozambique. Their project started off being completely embedded in the Ministry of Agriculture, then started a second office that was embedded in one of the government’s research organizations while still maintaining a small core office in the Ministry. Some of the most successful hybrid models have combined an embedded office with a university or non￾university-based policy institute. In each of these cases, the national universities had a high degree of intellectual and financial autonomy from the national government but were still seen as key government partners. This model is less feasible in countries like Myanmar, where the agricultural university has not yet acquired the same degree of independence. Relative Effectiveness of an Expatriate, Local, or Mixed Team: About 60 percent of the respondents to the online survey felt that a mixed team that included both expatriate and national staff was the most effective (Table 12). This sentiment echoed the strong preference for mixed teams that was expressed in the case-study countries, followed by local teams and expatriate teams (as the least favorite) (Table 12). Table 12. Ranking of the Different Types of Teams (Expatriate, Local, Mixed) in Terms of Effectiveness for Policy Analysis and Dialogue Least Middle Best Total with an Opinion Expatriate 12 (46%) 8 (31%) 6 (23%) 26 Local 9 (28%) 14 (44%) 9 (28%) 32 Mixed 5 (14%) 9 (26%) 21 (60%) 35 Source: FSP IL Mid-Term Performance Evaluation online survey. The expatriate FSP IL stakeholders cited the tremendous benefits that accrue from working with local researchers who have first-hand familiarity with the local context and who can continue to support the activities once the FSP IL funding ends. Other benefits include helping to validate the research in the eyes of the local people. Most of the local government, university, NGO, and private-sector stakeholders cited the same advantages that would accrue to the expatriates but also the personal benefits that accrue from mixed teams from their perspective, which include: ● ● ● ● ● Opening the door to more open and consistent collaboration with the wider donor community; Having an opportunity to learn about other countries in the sub-regions’ experiences; Developing personal relationships with some of the top international experts in international policy; Learning new methodologies and ways of working with data; and For university professors, getting a better understanding of how to publish in the international journals needed for promotion and tenure, as well as getting known beyond the frontiers of their own country. 4.3.2 Conclusions EQ 3.1. The FSP IL activity components have been very effective in building the initial capacity of the country programs and in strengthening the linkages between these programs and some of the most important global and regional initiatives that support evidence-based food policy. The type of assistance that the Mission-funded projects need to support policy change at the country and regional level (e.g., CADDP processes) was very different when they were just starting to develop than today. However, a 48 better coordination amongst the activities is necessary in order for FSP IL to backstop the country programs in a holistic manner. EQ 3.2. Since the end of the third year, FSP IL has made a concerted effort to develop a more focused communication plan for all of its core-funded activities. This same initiative has encouraged many of the Mission-funded projects to hire full-time communication officers. The project, however, lacks an integrated strategy, and a mechanism to monitor key stakeholder groups use and needs. Many interviewees, however, asked for more reports that provide more information about the results of the projects and are tailored to the needs of some of their key stakeholder groups at the national, regional, and global levels. EQ 3.3. To date, the most reliable source of information on FSP IL’s collaboration with its major regional partners (ReSAKSS, the AUC CAADP focal points, and Africa Lead) is FSP IL’s annual reports and the component teams’ presentations at the all-FSP IL meeting in March 2017. This involvement was extensive and needs to be better profiled and tracked in the project’s PMP and reports. Without better data on what types of collaboration are occurring with these different groups, it is hard to strengthen the collaboration with existing regional and global stakeholders. This is a missed opportunity since these regional organizations are the principle organizations that are tasked with backstopping the national governments on policy review and reform. EQ 3.4. Initially, an office integrated with a host country institution, such as the host government or an independent think tank, is more effective than a separate country office in affecting policy dialogue, review, formulation, or change. In all cases, the perception of independence and objectivity of the LWA is upmost, and the hybrid model of embeddedness and having a separate office facilitates this perception, making it an even more effective organizational model then the embedded model in terms of affecting the policy process. As a policy-oriented, LWA matures and, where the circumstances allow, moving toward the hybrid model is ideal. A mixed program team that links local experts with individuals in strong institutions like IFPRI, MSU, and UP can open the door to: 1) strong two-way learning that benefits both sets of partners; and 2) wider acceptance of the research findings by the national government as well as the wider donor community that supports them. 4.3.3 Recommendations EQ 3.1. LOP: Going forward, the component teams need to capitalize on the increased capacity of the Mission￾funded country programs by developing a more cohesive model for coordinating their country-level activities as well as those directed and influencing regional and global policy debates. This includes: 1) strengthening the coordination of research between the FSP IL regional-global teams and the Mission￾funded AA and Buy-in projects; and 2) Strengthening the cohesion, coordination, and leadership within the FSP IL global component teams themselves by better defining and recognizing the roles and responsibilities of the global component team member and their leader. EQ 3.2. The ET’s recommendations focus on developing a more integrated strategy and assigning a firm institutional leadership to support the communication offices in the lead institution and field programs in the design and execution of that strategy. A number of very concrete recommendations surfaced in the course of the KIIs, FGDs, and online surveys for some of the key elements that need to be included in that communication strategy. They include (Annexes V.F): ● Continue to support the full-time FSP IL Communication Officer; 49 ● ● ● ● ● ● Continue improving the appearance and functionality the FSP IL website and its links to any of the Mission-funded AA and Buy-in websites; Expand the number of briefing papers and their translations; Film more of the FSP IL component team and AA/Buy-in staff’s presentations, which could be posted on the websites as well as distributed through some of the channels that farmers and community leaders watch on television, Facebook, and their smartphones;101 Strengthen the project’s involvement with journalists to encourage better media coverage; Organize personal meetings with people of influence at least twice a year, ideally when FSP IL researchers are visiting; and Examine examples of best practice in communication that could be potentially scaled up to other countries, including the: ­PAPAC, PAG, and AAPC mechanisms in Tanzania; and Journalist training model that was initially tried in Malawi through a joint collaboration between UP, IFPRI, and the Mission funded AA Project (NAPAs) under C3 with follow-up sessions in Malawi (coinciding with the workshop on gender and nutrition) and in South Africa (linked to the C3 validation of the C3 case studies). LOP: Develop a comprehensive communication strategy for FSP IL that informs and is informed by a complementary set of national-level strategies in the Mission-funded AAs and Buy-ins. Specific sub￾recommendations include: 1. Include a tracking table that identifies the top key informants and backup key informants for each stakeholder group in each country;102 2. Determine which mechanisms and individuals are necessary to ensure quality feedback from different key stakeholder groups about how effective FSP IL’s communication with them has been and how it could be improved; 3. Encourage the Mission-funded AAs and Buy-ins to conduct a disaggregated analysis of this information at least twice a year, and FSP IL to include this information in its semi-annual and annual reports to USAID/BFS; and 4. Ask IFPRI to co-lead the design and monitoring of this communication strategy in collaboration with the UP FSP IL team. Future LWA: Make communication capacities and experience a core competency requirement in future LWA proposals for food policy research. EQ 3.3. LOP: Build better evidence-based systems for joint planning between the component teams and the most important regional USAID-funded and supported regional initiatives like ReNAPRI, Africa Lead, the AUC CADDP processes, and ReSAKSS. EQ 3.4. Future LWA: Anticipate that as LWA matures, it may wish to move from a fully “embedded” organizational model to a more “hybrid” model in which: 1) certain activities and staff are embedded under its principal local government partner; and 2) other activities that are more independent. 101 The evaluators were surprised by the fact that a majority of the community leaders—both male and female—had smartphones in both villages where they interviewed. In the village without electricity, the famers liked watching video spots and programs on one particular channel. Facebook was widely used by all of the staff associated with the private-sector associations as well as all of the input and tractor supply dealers the team interviewed. One of the input dealers said that over half his 28 colleagues who were licensed to sell inputs by the regional Ministry of Agriculture had their own computers and smartphones. 102 If the strategy is clarified to this level, it can reduce duplication and make it easier to monitoring the evolution of the strategy. 50 Future LWA: Consider encouraging future LWA awards to work with university-based policy research institutes as their independent research and outreach partners unless there is a political reason not to do so. This is recommended because it provides the LWA with country-level partners that are both independent but related to the government. 4.4 MANAGEMENT AND IMPLEMENTATION (EQ CLUSTER #4) 4.4.1 Findings 4.4.1.1 EQ 4.1: How does FSP IL define its customers, and who are they (Missions, BFS, ministries)? To what extent is FSP IL meeting or not meeting customer needs and why? Asking the evaluators to assess “to what extent is FSP IL meeting or not meeting customer needs” raises an important issue, which is what constitutes client needs (Annex VI.D). FSP IL was designed to create a need and to strengthen the country, regional, and global-level systems for responding to that need in the countries targeted by the Feed the Future program. There is clear evidence from this study that: ● ● FSP IL has been successful in creating high levels of demand for evidence-based research in a variety of the countries where it has focused those activities; and This demand has generated a new set of demands for FSP IL and Mission-supported or country, regional, and global-level policy research, analysis, and advocacy. Most of the problems—i.e., areas where FSP IL is not meeting customer needs—are client requests for additional assistance in meeting the explosive demand that these first-generation FSP IL activities helped catalyze. This demand has put pressure on FSP IL’s existing management and implementation systems, which the project is trying to address. That same pressure has highlighted a number of important lessons learned for future programs that may try to emulate this LWA model. 4.4.1.2 EQ 4.2: What are the strengths and weaknesses of the following components and in what ways can the following be improved: a) Management and coordination; b) Funding arrangements; and c) Monitoring and Evaluation? Management and Coordination: FSP IL is a consortium of three members: MSU, IFPRI, and UP. MSU is the lead institution contracting directly with USAID/BFS and subcontracting with IFPRI and UP. MSU and IFPRI have had country offices on the African continent, while UP is located in Africa. Both MSU and IFPRI have several decades of experience of fieldwork in Africa or Asia. MSU and IFPRI have historically considered each other as competitors not collaborators, but the two have similar as well as different strengths that can potentially lead to positive synergies. Each partner brought into the consortium its own culture and management style as well as different financial systems. Although the management systems are unorthodox, in the opinion of most stakeholders that the ET interviewed in person and through the online survey, the current systems have evolved and are working better (Table 13). Table 13. Strengths and Weaknesses of the FSP IL Management and Coordination Systems Strengths Weaknesses 1. Flexibility to respond quickly to changing needs. 2. Starting in third year, improved communication and coordination. 3. Extended reach through consortium membership, wider expertise upon which to draw, wider contacts, and broader country experiences. 1. Lack of overall oversight for capacity strengthening activities. 2. Annual work plans and budgets in a multi-year project. Source: FSP IL Mid-Term Performance Evaluation. 51 MSU’s management style grew out of the implementation of three 10-year projects. These projects were led by the same group of self-organized faculty, and the faculty that ran the projects consisted of tenure stream and fixed-term. The fixed-term faculty did not teach but only did research, while the tenure-stream faculty taught and did research. Also, the Department of Agricultural, Food, and Resource Economics-Food Security Group (AFRE-FSG) collaborates at MSU with the geography department, crop science department, and community sustainability department. A strength of this type of organization and management style lies in its flexibility to respond quickly to changing needs. Further, an effective self-monitoring system by the faculty entailed little overhead costs of management. IFPRI is a large international institution consisting of six divisions, of which the Development Strategy and Governance Division (DSGD) is one. It is DSGD, not IFPRI as a whole, that is a member of the FSP IL consortium. These divisions are semi-autonomous, and staff and researchers are generally assigned to one division led by a division director. A strength of IFPRI is its wide array of expertise. However, while possible, it is not seamless for a person working in a division other than DSGD to contribute to FSP IL. The Institute for Food, Nutrition and Well-being (IFNuW) is the institutional home for the FSP IL at the UP, supported by the University’s Departments of Research and Innovation and Finance.103 Staff from the Department of Agricultural Economics, Extension and Rural Development (DAEERD) and BFAP at UP collaborate on FSP IL components. DAEERD is a leading Department in the country and region. BFAP, founded in 2004, is an independent non-profit organization initiated at and associated with UP.104 The management of the consortium and coordination among the three member institutions has not been without problems and growing pains. Selection of policy research topics was done in concert by the three consortium members with input from USAID/BFS, but implementation was less coordinated in some instances. IFPRI and MSU worked together in implementing the Mon State household income survey in Myanmar. C3 activities are also good examples of IFPRI, MSU, and UP members working and planning together. Upstream work on land issues between MSU and UP is relatively coordinated, but work on land has been mostly independent with little coordination among MSU, IFPRI, and UP. FSP IL consortium members are fully aware of these issues and are currently making a concerted effort to improve management, coordination, and communication, which are essential for a successful consortium-run project. At first, there were misunderstandings in terms of activities between IFPRI and the MSU Department of Agricultural, Food and Resource Economics (AFRE), but those were eventually worked out through open communication. The sharing of information is important, because the activities being executed by the three consortium members should not conflict. The all-FSP IL meeting in March 2017 in Washington, DC, was a positive step in improving coordination and communication. Further, the consortium now has a monthly meeting among the three consortiums’ higher-up representatives, and management to complement the bi-weekly meetings between principal investigators (PIs) and the business office which have always been part of the FSP IL management model. One question arises: are the challenges of managing a consortium worth the costs? In every case when asked, KIIs responded that the advantages outweigh the costs. Advantages of the consortium have been extended reach through consortium membership, wider expertise upon which to draw, wider contacts, and broader country experiences. Funding Arrangements: Funding for FSP IL core activities is from USAID/BFS, while funding for AA/Buy-in activities is from the USAID Mission in which the AA/Buy-in resides. In all cases, the monies 103 UP is one of a few research-intensive universities in Africa. In 2011, the focus on food security research was strengthened through a strategic investment by the University that established the UP Institute for Food, Nutrition and Well-being (IFNuW). The virtual institute coordinates research in this area across 35 academic departments at UP. The transdisciplinary Institute is unique in the continent and globally. 104 BFAP provides independent, rigorously tested, research based market and policy analyses. BFAP consists of a network of associates and researchers at universities spanning the African continent. BFAP is South Africa’s partner institution in ReNAPRI. 52 first are appropriated to MSU, and then MSU allocates the monies to itself, other consortium partners (i.e., IFPRI and UP), or other country-specific partners (e.g., CESD in the case of Myanmar AA). MSU AFRE-FSG has its own business office and administers the funding of grants awarded to FSG faculty, including the FSP IL and the associated AAs and Buy-ins. FSP IL core funds are allocated among the consortium members based on submitted and approved annual work plans. In the initial two years, MSU and IFPRI received allocations of similar size, while UP only received 10 percent of the total annual FSP IL core funds. This became an issue for UP that was remedied in the third year, with UP receiving a larger share of FSP IL core funding. Each of the three consortium members has its own financial system that differs from those of the other members. This presented challenges in the initial stage of FSP IL, particularly for UP. MSU has traditionally not given advances to subcontractors. UP, however, had no mechanism to give start-up monies to its DAEERD. As pointed out above, this delayed activities by UP until MSU went against its long-held policy and advanced start-up monies to UP. Invoices with documentation for monies spent by IFPRI and UP on FSP IL activities under the core award agreement or the AA agreements are submitted for reimbursement to MSU AFRE-FSG’s financial office. Its finance people go over the invoices and documentation prior to sending them to the concerned grant PIs for review and approval, and then to MSU’s Contracts and Grants Office. Upon approval, reimbursements are sent to IFPRI and UP. In the case of AAs or Buy-ins for which MSU faculty members are based in-country to manage the activities (e.g., Tanzania), MSU advances monies to MSU faculty based in-country through a personal imprest account to cover cash expenses in support of the AA or Buy-in activities. In cases where in￾country activities are led by a partner (e.g., IFPRI, UP, or a local partner) monies are channeled through a sub-contract with that partner institution. All the sub-contracts are written as cost reimbursable. However, depending on the institution, MSU determines if it needs or receives an advance of funding or not. Currently, the invoices with documentation are sometimes submitted every two months. The risk of this practice is that the project could run out of money for project activities (Table 14). At the present time, however, FP management and in-country Buy-in leadership agree that the size of the Tanzania imprest is sufficient to avoid this problem. Originally in Tanzania, in every quarter a work plan and financial forecast would be developed by the country staff for approval by MSU based PI. Now, these are done every month. Transferring money from MSU campus to country based faculty member managing the in-country activities under an AA or Buy-in can be a challenge. Monies to Myanmar, for example, must be transferred first to a bank in Singapore before being transferred to the AA’s account in Myanmar. The AA in Burma does not have a bank account, nor does the COP have an imprest account, as all local expenses are processed through the sub-award with the local partner CESD. When the AA started working with CESD in 2012, direct wire transfers to CESD accounts in Burma were not possible due to sanctions and transfers had to be made indirectly (with due approval of course). Now that sanctions have been lifted, direct wire transfers are possible through some local banks. 53 In Tanzania, the ASPIRES project is not yet a legal entity and cannot have its own account. Transfer of money to cover ASPIRES project activities involves advancing funds from MSU to the COP’s personal account via the imprest account mechanism. Since this is a dollar account, funds received through this mechanism have to first be exchanged into Tanzania schillings. This involves transaction costs but, since money flows through personal bank accounts. These costs can be recovered by including the receipts (or bank statements if the charge is shown there) in the subsequent imprest voucher. For money to reach those working in the field, it is advanced from the COP to the Buy-in’s fiscal manager, who then advances it to those going into the field. Monitoring and Evaluation (M&E): Within one month of signing the agreement, the FSP IL Senior M&E and Impact Assessment Specialist, drafted an initial concept paper that she used to inaugurate an initial discussion with a number of key stakeholders, including the Feed the Future M&E specialists, the USAID/BFS line supervisors, and senior staff at IFPRI, MSU, and UP.105 The output of this exercise was a detailed PMP that was submitted to USAID with the first annual report. The FSP IL PMP identified: ● ● Seventeen106 indicators that each of the FSP IL component teams committed to using to track and report on their activities (see Annex VI.B for an updated list of indicators; Figure 1 shows which indicators are used to measure which output and which outcome in the FSP IL results framework); and Clear, very concise Performance Indicator Reference Sheets (PIRS) that described the process that would be used to collect each indicator, including who would collect the data and how it would be analyzed and disaggregated. Table 14. Summary Strengths and Weaknesses of the FSP IL Funding Arrangements Strengths Weaknesses 1. It works in all cases, for the consortium members, the AAs, and the Buy-ins. 2. There is strong oversight over all expenditures and expenses. 3. The financial officers of MSU AFRE-FSG respond quickly to financial submissions involving reimbursements. 4. Budgets and work plans are submitted annually to USAID. 5. Country-based fiscal managers of AAs and Buy-ins have rapid access to campus-based MSU AFRE-FSG fiscal officers via Skype. 6. Members of the MSU AFRE-FSG business office came to Tanzania and helped the country financial person put together financial spread sheets with activity codes for different projects (i.e., FSP IL, Buy-in, and Gates Foundation). 1. Funding consortium members and local partners through a reimbursement-only system can put financial burdens on these entities and can delay project activities.107 2. Initially, financial rules and guidelines were not clearly explained to the country staff supported by the AAs or Buy-ins. 3. While invoices with documentation are expected by MSU AFRE-FSG business office monthly or more frequently to ensure a healthy cash flow, this is often difficult in practice for the country staff to accomplish as many researchers may be in the field for a long period of time and unable to submit timely receipts. 4. Often scanned receipts are illegible, many receipts fade in a short time, and it is difficult if not impossible to get receipts for some expenses occurred in the field. 5. In some cases, country based personnel are expected to bare the risk of carrying project monies in their personal accounts. Source: FSP IL Mid-Term Performance Evaluation with feedback from Mywish Maredia, David Tschirley, and Duncan Boughton, August 2017. 105 FSP IL. 2013. Monitoring and Evaluation Framework for the ‘Feed the Future’ Policy Agenda: Some Preliminary Thoughts and Proposed Ideas for Consideration. East Lansing: FSP IL Management unit on behalf of the MSU-IFPRI-University of Pretoria Food Security Policy (FSP) Consortium (November 20, 2013). 106 The original PMP included 18 indicators but one of them was merged into 16 and 17. 107 Since USAID’s contracts with MSU are also cost reimbursable, there is no way around this weakness. In fact, in cases (such a UP and the imprest account mechanism), when MSU advances cash, it is doing it from its internal resources and taking the risk, because they get paid by USAID only on cost-reimbursement basis (i.e., after costs are already incurred and MSU submits invoices to USAID). 54 In preparation for the first-year report, the M&E Specialist developed a series of user-friendly Excel spreadsheets that the teams were expected to use to report on the indicators. The majority of the indicators in the FSP IL PMP (Annex VI.B) are custom indicators in that they are not official Feed the Future indicators.108 Once the FSP IL started generating a new generation of Mission-funded AAs and Buy-ins, the FSP IL M&E Specialist encouraged all the new programs to at least consider putting these indicators in their tracking table. Her argument was that the indicators are sufficiently generic that they could be compatible with whatever custom or standard indicators that the Missions needed to report on in their Mission PMP. Unfortunately, the only AA that took that message to heart was Myanmar (Annex VI.C). Each of the other AAs and Buy-ins adopted only a few of the FSP IL indicators from the tracking table. This accounts for the fact that the Mission-funded programs are currently tracking only 2 of the 17 indicators in the PMP that the FSP IL M&E Specialist refused to back down on (indicator 17 and 18) (Annex VI.B). The only exception is Myanmar, which is tracking all 17 of the FSP IL indicators as well as its own Mission-mandated indicators. This has made it much easier to see the results of the AA’s joint collaboration with the AA in Myanmar (Annex VI.C). There was widespread consensus among most of the partner staff the team interviewed, as well as the USAID officials, that the FSP IL M&E system has many strengths (Table 15): ● ● ● ● ● FSP IL has had a very experienced M&E Specialist on the management team since the start of the program who continues to oversee the system; The indicators are SMART—specific, measurable, accurate, realistic, and time bound—and comply with the highest standards of M&E for policy projects;109 The data has been collected in a consistent manner; The IL has a highly functional database; and There has been rigorous quality control by the FSP IL administrative specialist to ensure that that the data is accurate and only reflects activities that the core funds have supported or co-supported. The chief weaknesses of the system are that (Table 15): ● ● The data on the FSP IL indicators are only collected consistently for the activities that are funded or co-funded by the FSP IL core funds; The only “shared” indicators that all of the FSP IL AAs and Buy-ins have with FSP IL are indicators 17 and 18 (Table 15; see Annex VI.B for an updated list of indicators). This limits the utility of the data to inform strategic planning to strengthen policy research, analysis, and advocacy with country, regional, and global partners. Another weakness is that there is no shared indicator in the PMP that tracks either the output or outcomes of the project’s substantial investment in institutional capacity strengthening. A third set of weaknesses relates to the way the global component teams collect and use the data. Each researcher who is responsible for the execution of a specific activity is responsible for collecting the data on that specific activity (or set of activities) and feeding it to the FSP IL M&E Advisor. Because the global component team leaders do not oversee the data collection process or quality assessments of that data, and have almost no M&E training, they are not using the data to inform either the development or monitoring of their work plans. This is a huge missed opportunity which is demonstrated by how easily this type of disaggregated analysis can be done using the existing indicators and data collection tools (See Annex VI.C). 108 FSP IL. 2014. Performance Monitoring Plan. Innovation Lab for Food Security Policy, East Lansing: FSP IL. Version Nov. 17, 2014 (approved by USAID on 10/16/14). 109 USAID/BFS identified the FSP IL outcome indicators 17 and 18 as an example of best practice that is being studied by a DAI consultant who is helping identify a new generation of Feed the Future indicators. 55 A fourth weakness of the system is that there is very little analysis or write-up of the PMP data in the semi-annual and annual reports. This is because FSP IL continues to use the report writing templates that it developed in the first year of the project, before it had an approved PMP. Were the team to adopt a more conventional post PMP model that uses the PMP to structure each component team’s presentation of its results in the semi-annual and annual report, it would be easier for USAID, the component teams, and evaluators to see the cumulative effect of the project for the different component teams, and to compare and contrast these activities between component teams and between countries. 4.4.2 Conclusions EQ 4.1. The increased demand for FSP IL evidence-based policy research and advocacy in the nine countries where FSP IL is most active has created national, regional, and international actors with very different demands and expectations for FSP IL and Mission-supported research, analysis and advocacy activities. EQ 4.2. Management, Coordination, and Funding Arrangements. Three issues emerged from the online survey that were also raised in the KIIs and FGDs, but were not raised by the internal management review: Table 15. Summary Strengths and Weaknesses of the FSP IL Monitoring and Evaluation Systems Strengths Weaknesses 1. The FSP IL has had a very experienced M&E specialist on the management team since the start of the program who continues to oversee the system 2. SMART indicators that comply with the high standards of M&E best practice 3. Good systems in place for collecting data 4. Staff have been diligent about collecting the data for the indicators 5. The FSP IL administration has exercised rigorous quality control and internal data quality assessments (DQAs) 1. Data on the FSP indicators are only collected consistently for activities that were funded or co-funded by the FSP IL core funds. 2. Only two indicators (17 and 18) take into account the joint results of the FSP IL component team activities and the activities of the Mission-funded AAs and Buy-ins that the component teams backstop. 3. There are no indicators in the tracking table that track enhanced institutional capacity or communication outreach. 4. The FSP IL component teams are collecting the data on the indicators in a disaggregated manner and not using it to inform the development or monitoring of their work plans. 5. Most of the senior researchers have little or no understanding about how they can use the M&E data to improve coordination, collaboration, and results. 6. There is very little analysis or write up of the PMP data in the annual report which makes it compare the project’s achievements on indicators between years and countries or between component teams. Source: FSP IL Mid-Term Performance Evaluation. ● ● ● The current system of single-year work plans makes it difficult to plan multi-year activities in a multi￾year project, but allocating budgets annually gives more fiscal oversight than multi-year budgeting. Lack of an explicit FSP IL organizational scheme makes it difficult for the component team leaders to play their critical role in ensuring an appropriate balance of research, capacity strengthening, and policy advocacy for the activities being supported by the FSP IL component teams. While the current systems for funding the FSP IL’s collaborative activities in the different countries where it is engaged generally work, they could be improved by increasing the knowledge of field staff concerning allowable and unallowable expenses. 56 Monitoring and Evaluation. The fact that only 2 of the 17 current indicators in the FSP IL PMP were included in all of the Mission-funded projects’ PMPs (except in Myanmar) limits the utility of the FSP IL’s M&E data for strategic planning and coordination. 4.4.3 Recommendations EQ 4.1. LOP: Build better systems for tracking the shifts in stakeholder demand into the routine FSP IL and Mission funded AA and Buy-in M&E processes. These systems should be an integral part of the FSP IL communications or capacity strengthening strategies that are recommended by the recent internal FSP IL management review110 as well as under EQ 2.2 and EQ 3.2 above. EQ 4.2. Management, Coordination, and Funding Arrangements. LOP: Allow for multi-year work plans with associated budgets, but have actual budgets allocated annually based on a formal progress review. LOP: Better define the management roles of the component team leaders and others that play a critical role in determining an appropriate balance of research, capacity strengthening, and policy advocacy. LOP: Develop a manual of simple financial guidelines for AAs and Buy-ins and hold regular financial training sessions for all their staff including field staff. Future LWA: Budget a start-up meeting at the beginning of the project to clearly articulate the project’s goal, objectives, and financial procedures and discuss the purpose of the proposed M&E system to all staff and faculty working with or in conjunction with FSP IL. Future LWA: Budget an annual meeting to ensure that coordination and communications remain strong between the consortium partners and any Mission-funded projects that develop. Monitoring and Evaluation (Annex VI.A)111 Given the critical importance of joint planning with the Mission-funded AA and Buy-in projects to sustain some of the FSP IL achievements, the ET feels that the IL should consider investing the time and funding that are needed to: 1) better use the data they are already collecting.; and 2) help the Mission-funded programs to develop the types of comparable data that will facilitate joint planning. LOP: Encourage AAs and Buy-ins to collaborate with FSP IL in conducting a country-level analysis of its existing outcome and output indicators for capacity strengthening for all of its major partner institutions. LOP: Develop a simple capacity self-assessment tool that the FSP IL global teams and country programs can use for tracking capacity building for specific institutions and/or units in order to promote evidence￾based joint planning. LOP: Develop a simple self-assessment tool that can be used to track FSP IL’s coordination with some of the most important regional Feed the Future and USAID initiatives that support evidence-based food policy (like ReSAKSS, ReNAPRI, and Africa Lead). LOP: Strengthen the capacity of the FSP IL and AA/Buy-in staff that are responsible for the design and execution of the M&E systems to support this new system of joint reporting on their individual and co￾sponsored capacity building activities. Future LWA: Future LWA cooperative agreements for food policy need to require all Mission-funded AAs and Buy-ins to adopt a core set of outcome and output indicators in addition to any Mission￾specific indicators they might be required to adopt. 110 Frank Young. 2016. Independent Review of Food Security Policy Innovation Lab: Internal Mechanisms and Processes and External Linkages. East Lansing: Michigan State University (December 11, 2016). 111 See Annex VI.A. for a more detailed explanation of the four priority recommendations for the remaining LOP. 57 ANNEXES 58 ANNEX I 59 ● ● ● ● ● ● PEEL TASK ORDER EXPRESSION OF INTEREST – PERFORMANCE EVALUATION I. BACKGROUND INFORMATION A) Identifying Information 1. Feed the Future Innovation Lab for Food Security Policy 2. Award Number: OAA-L-13-00001 3. Award Dates: 7/15/2013 to 1/14/2020 4. Project/Activity Funding: $70 m LWA; including $15 m. core support to Leader Award plus up to $13.5 Buy-ins to Leader Award. 5. Implementing Organization(s): Michigan State University prime, in consortium with International Food Policy Research Institute (IFPRI) and University of Pretoria 6. Project/Activity COR/AOR: Courtney Buck (Activity Manager: David Atwood) B) Development Context 1. Problem or Opportunity Addressed by the Project/Activity Being Evaluated The country, regional and global policy environment for agriculture, nutrition and food security is a major determinant of the success and sustainability of more local- or project-level investments in these areas. This was recognized early in Feed the Future, when initial country and regional strategies reviewed by the interagency demonstrated little or no engagement at the policy level or in policy dialogue with host governments and institutions. The result was a determination to engage explicitly in policy dialogue, analysis, and capacity building and task every FTF focus country Mission to do the same. This project was designed shortly after the determination that an explicit focus on food security policy is an integral factor in the success of FTF project investments. As the project was being designed, a series of developments highlighted, and indeed increased, demand for project engagement and outputs, occasioning a major increase in the project ceiling. These developments include: An expanded and formal USAID and interagency focus on food security policy. Emphasis on policy by the G8 (now G7) African-Private Sector New Alliance for Food Security and Nutrition. Almost 300 policy commitments by African heads of state to implement the New Alliance. Growing interest from Missions to build host-country policy capacity. Major opening for food security and agricultural policy in Burma within the new political relationship between the U.S. and Burmese governments. The 2014 African Union commitment to double productivity, eliminate hunger, and cut poverty in half in a decade, with a particular focus on policy and institutional capacity as drivers of change. Missions in 18 FTF Focus Countries and five Regional Missions have developed interagency policy agendas to pursue with host country governments, and a significant number of these countries are now seeking, or are likely to soon seek, FTF Innovation Lab for Food Security Policy engagement in pursuing this agenda. Ten countries have New Alliance policy commitments in which USG engagement is essential. Several USAID 60 Missions in New Alliance countries are already seeking assistance for host governments from the FTF Innovation Lab for Food Security Policy, and more are expected to do so in the future. 2. Target Areas and Groups The FSP IL’s seeks to contribute to Feed the Future’s goal of significant reductions in poverty and hunger among poor people through facilitating host government policy changes that expand inclusive economic growth and reduce Figure 1. Locations of key capacity building/analysis/research sites (omitting global and regional interventions): 61 malnutrition. Facilitating policy changes involves both capacity building of a wide range of actors in the formulation and implementation of policy, as well as providing direct support to advocacy and analysis. These actors include government ministries, including but not limited to agriculture and nutrition, parliamentary committees, regional and global food security research, policy and strategy groups, civil society, farmers’ associations and private sector lobby groups, think tanks, universities and other policy research organizations, and multiple donors and foundations involved in policy support or advocacy. A major focus of the project is to influence global and regional policy and strategy through improved policy research and advocacy. The focus of the research to accomplish these ends is on farms, firms, and markets in the agriculture, food security, and nutrition space. Direct in-country partner or collaborating institutions include: ● ● ● ● ● ● ● ● ● ● ● Ethiopia o o o o o o o o o o o o o o o o o o o o o o o o o o o Ethiopian Development Research Institute (EDRI) Kenya Kenya Land Alliance Malawi Civil Society Agriculture Network (CISANET) – Malawi Malawi’s Ministry of Agriculture, Irrigation and Water Development (MoAIWD) MoAIWD Department of Agricultural Planning Services (DAPS) Mali Institut Polytechnique Rural (Mali) Myanmar Food Security Working Group Myanmar Development Resource Institute-Centre for Economic and Social Development (MDRI￾CESD) Nigeria University of Ibadan Senegal Senegal’s Ministry of Agriculture, Directorate for Analysis, Forecasting and Statistics (DAPS) Senegalese Institute for Agricultural Research (ISRA) University Cheikh Anta Diop of Dakar. Tanzania Department of Agricultural Economics and Agribusiness (DAEA), Sokoine University Monitoring & Evaluation Directorate- MAFC Zambia Zambia Land Alliance USAID USAID/Mali USAID/West Africa Africa Rising KSU Sustainable Intensification Innovation Lab International AFAP Bill and Melinda Gates Foundation CGIAR PIM (CRP2) CIMMYT ECOWAS IFDC ICRISAT ReNAPRI 62 o ReSAKSS-Asia C) Intended Results of the Project/Activity Being Evaluated The overall goal of the FSP program is to promote inclusive agricultural productivity growth, improved nutritional outcomes, and enhanced livelihood resilience for men and women through improved policy environments. The goal will be achieved by fostering credible, inclusive, transparent and sustainable policy processes at country and regional levels and filling critical policy evidence gaps. The Results Framework illustrated in Figure 2 provides the pathway for FSP to achieve its development goals and objectives and as such, is central to FSPs management, monitoring and evaluation approach. The various tiers of the results framework, from top to bottom, are aligned with the USAID’s Feed the Future initiative’s overall goal and objectives, five intermediate results (IR) and two sub-intermediate results (represented by boxes with solid black lines). The framework also includes one custom sub-sub-IR and five strategic results (SR) across project components that contribute to the sub-intermediate results. These tiers are arranged to illustrate the casual relationship between FSP project’s planned efforts and impact, and identify those intermediate results critical to achieving the objectives. Specifically, the framework conveys the development hypothesis implicit in the FSP program strategy and demonstrates how planned activities and deliverables will lead to expected outputs, outcomes, results, and eventual impact. Project Goal, Intermediate Results, Sub-Intermediate Results and Strategic Results: The ultimate goal of the FSP project, appearing at the top of the results framework, and as mentioned before, is to promote inclusive agricultural productivity growth (FTF’s First Level Objective –FLO 1), improved nutritional outcomes (FLO 2), and enhanced livelihood resilience for men and women. FSP has identified five Intermediate Results (IRs), which will be targeted for different involvement in focused countries/regions where FSP works. These include:112 IR 1: Improved Agriculture Productivity IR 2: Expanding Markets and Trade IR 3: Increased Investment in Agriculture and Nutrition related Activities IR 4: Increased Employment Opportunities IR 5: Increased Resilience of Vulnerable Communities and Households FSP has also identified two sub-Intermediate Results and one custom sub-Sub-Intermediate Results to further narrow down its technical priorities. Sub-IR 1.1: Enhanced human and institutional capacity development for increased agricultural sector productivity Sub-IR 1.3: Improved Agricultural Policy Environment Sub-Sub IR 1.3: Better policy formulation process Five custom Strategic Results (SR) will contribute towards the Sub-IRs. These include: SR1: New information, knowledge, and practices SR2: New datasets SR3: Knowledge dissemination and learning events SR4: Policies, programs, and regulations reviewed and analyzed SR5: Human resource capacity building Together the IRs, sub-IRs and SRs listed above provide the framework for identification and implementation of activities designed to achieve the required results. 112 The numbering of the FLO, IRs and Sub-IRs follows USAID’s FTF indicator sequence where they are aligned with that framework (i.e., in boxes identified with solid black line or black font). 63 Figure 2. Results Framework for the FSP Project D) Approach and Implementation FSP accomplishes the SRs, IRs and project objectives through an activity work program organized into five components: C1: Country/Regional Level Collaborative Research (on Farms, Firms, and Markets) and Formulation/Analysis of Policy Options. C2: Country/Regional Level Capacity-Building for Policy (Data, Analysis, Advocacy, Formulation, Consultation, Coordination, and Implementation). C3: Global Collaborative Research on Support to the Policy Process and Policy Capacity C4: Engagement in Global Policy Debates on Food and Nutrition Security 64 C5: Strategic Analytical Agenda and Support to Donor Policy and Strategy II. EVALUATION RATIONALE A) Evaluation Purpose This mid-term evaluation is to assess whether the activity is on track to achieve intended results, and what, if any, adjustments need to be made to improve implementation and better achieve project outcomes. Outcomes of and progress toward the sub-IRs and SRs listed in the FSP Results Framework above will be the primary focus of the evaluation, however, findings and recommendations should be positioned to maximize the potential contribution towards the IRs and FLOs at higher levels in the results framework. B) Audience and Intended Uses The primary audience and intended users are the BFS/ARP/Policy Team and the BFS/ARP Innovation Lab management team; Missions with ongoing or planned associate awards or buy-ins are also intended users as is the MSU/IFPRI/Pretoria leadership team implementing FSP. C) Evaluation Context: This evaluation will be undertaken as one piece of a larger set of evaluations and reviews of leading activities and approaches in the BFS/ARP/Policy Division portfolio as well as at Michigan State, including: a. Internally-funded MSU evaluation of some FSP activities prior to the implementation of this external USAID evaluation. b. October 2015-February 2016 BFS/ARP External Review of Policy Division Portfolio c. BFS/ARP/Policy Division Review and options paper (planned for summer 2016) setting out future approaches options to achieve joint policy and institutional architecture/capacity goals of Policy Division, especially those for which ReSAKSS, AfricaLEAD and FSP activities are key instruments. This review and options paper will look in a coherent, integrated way at whether and possibly how to continue three of the most extensive and core ARP/Policy Team functions: the intellectual and strategic contribution of an FSP–like organization, the Africa–implementation of ReSAKSS, and the flexible support functions of Africa Lead? d. USAID external evaluation of ReSAKSS, 2015 e. USAID AfricaLEAD external performance evaluation, end CY2016 (similar or a bit later timing than the FSP evaluation) Because a significant part of FSP is related to country level policy analysis and capacity building, the evaluation needs to be conducted taking account of country level programs, including where Missions are funding Associate Awards (AAs) or buy-ins. Those AAs and buy-ins will not be evaluated, since they are Mission programs, but they do need to be taken into account when evaluating the core funded in-country work. All three aspects of FSP should be examined in the external evaluation. The latter two aspects should receive the bulk of attention and resources in the evaluation, with approximately 20% of evaluation resources and attention going to finance/management/partnership arrangements: ● Finance/management/partnership arrangements ● Actual research and capacity building ● Imputed policy influence. An assessment of the quality and/or competence of the actual conduct of research is not a primary concern or focus, but relevance of research topics and use of research as it relates both to policy capacity and policy influence/dialogue/change will be important aspects of the evaluation. N.B., This IL is very different from all other ILs which are focused only on research (not on taking research into action/development/operations); this IL undertakes research/analysis and capacity building in 65 a context where influence on concrete policy changes and outcomes are expected and are part of program investments. D) Evaluation Questions The four most important clusters of questions cover policy analysis and research; capacity strengthening; policy change, influence, and communication; and management/administration/consortium relationships. 1. ● ● 2. ● ● ● ● 3. ● ● ● ● Policy Analysis and Research: In what ways, if any, has the selection and conduct of policy analysis and research influenced policy review, formulation, or change by host country governments? What areas of policy analysis and research, if any, need more attention or should be included to bring about outcomes related to a better policy environment? Capacity-strengthening: How effective has the coordination of capacity strengthening activities been among the FSP components, FSP partners, USAID missions, and related entities at the country level (such as Africa LEAD, AGRA, FAO/MAFAP, ReSAKSS, Futures Agriculture Consortium, IFDC, or other policy analysis groups). In what ways could this be improved? How effective has FSP been in the organization and implementation of capacity strengthening activities covering different content such as policy research, analysis, strategy, and communication as well as providing advice to improve the use of evidence to advocate for policy change at the following levels: o o o o Country (including ministers, parliamentarians, other host government staff engaged in the policy process) Organization (including farmer groups, NGOs, and other civil society groups) Individual (including journalists) Policy development process, which engages and coordinates each or several of these levels. In what ways could the capacity strengthening component be improved for greater policy impact – including support to groups such as associations, farmers groups, NGOs and other civil society groups; journalists; Parliamentarians and others, and content to use evidence to advocate for policy changes? What capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities? What types of synergies, if any, have there been among the varied MSU, IFPRI, and University of Pretoria capacity strengthening approaches? In what ways can each partners’ capacity strengthening approaches be improved? Progress toward policy outcomes: To what extent have the five different activity components (see diagram above) contributed towards policy change at the global, regional, and country levels? In what ways could these components be expanded or improved to better support policy influence or policy change? What is country level policy change/influence and how could this be expanded; how have C3, C4, C5 contributed to more effective or rapid policy change at country level? How effectively has FSP communicated policy analysis, recommendations, and options to policy makers or others in a position to influence them and in what ways could this communication be improved (including content, target, etc)? In what ways has FSP worked closely with other partners and stakeholders at the country, regional or global level to effect more rapid policy change (for example, ReSAKSS, AUC CAADP focal points, and AfricaLEAD)? What opportunities exist to effect more rapid policy change with existing stakeholders or new stakeholders? What MSU/IFPRI/University of Pretoria FSP organizational models appear to be more or less effective in supporting policy change at the country level? (For example, is a separate country office more or less effective than an office integrated with a host country institution? Is an expatriate, local, or mixed team most effective in credible policy analysis and dialogue?) How can these organizational models be strengthened to better support policy change? 66 4. Management and implementation: ● How does FSP define its “customers” and who are they (Missions, BFS, Ministries?) and to what extent is FSP meeting/not meeting customer needs and why? ● What are some of the strengths and weaknesses of the following components and in what ways can the following be improved: o Management and coordination; o Research; o Funding arrangements III. TIMEFRAME & TRAVEL A) Timeframe Start date: November-December 2016 End date: April 2016 B) Travel Travel to two field sites, possibly Senegal, Tanzania, Malawi, or Burma is essential. IV. DELIVERABLES & DESIGN A) Deliverables 1. Concept Note on preliminary evaluation design 2. Evaluation Design (at least 2 drafts pending USAID approval) 3. Mission outbriefs: short presentation of country-wide findings for USAID mission for each country where site visits were conducted, pending mission agreement 4. Presentation of preliminary findings: utilizing web based interface, including partners and missions who bought in to the activity, as desired. 5. Evaluation report (at least 2 drafts pending USAID approval, including evidenced based findings and short-term and longer term actionable recommendations, relating to the evaluation question and activity implementation. B) Evaluation Design Due to the nature of the activity, this IL undertakes research/analysis and capacity building in a context where influence on concrete policy changes and outcomes are expected and are part of program investments, the evaluation team is encouraged to be innovative and creative with the evaluation methodology, while providing robust and evidence based results, geared toward the evaluation questions. V. TEAM COMPOSITION Evaluation Team Lead: A senior-level evaluator with a minimum of 10 years of experience designing, managing and/or evaluating multifaceted international development teams, involving agriculture, policy education, or sector development. The candidate will also have: a) a demonstrated capacity to conduct independent program evaluation; b) an understanding of USAID’s foreign assistance goals, and its particular objectives related to agricultural development and food security; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Experience in sub-Saharan Africa, Latin America, and/or South Asia is necessary. Economist/Agriculture and/or Food Policy Specialist: Must be experienced expert (5-10 years or more) in economics as applied to food and/or agricultural policy. Field experience using research, analysis, data in helping policy makers formulate, choose among, implement and assess results of specific policies. Experience related to agricultural sector institutional development and policy. Experience working in sub￾Saharan African or Asian policy contexts; experience in effectively conducting outreach and dissemination to policy makers, development practitioners and/or the private sector; and the ability to analyze issues 67 and formulate concrete recommendations orally and in writing. Experience in strengthening the capacity of institutions and individuals responsible for policy advice and/or policy making and implementation. VI. SUGGESTED LOE Provide information about the suggested LOE, using the sample template below: Task/Deliverable Team Leader Team Member Conference Call/ Desk Review 10 8 Evaluation Plan & revisions 4 3 Data Collection & Travel 18 22 Presentation of findings 2 2 Draft Report 6 4 Revisions of Final Report 3 2 Total 43 41 LOE: Level of Effort in days 68 ANNEX II 69 70 71 72 73 ANNEX III 74 Annex III.A Respondent feedback on the question: How did FSP LAB make decisions on design of policy analysis and research? Please provide maximum of three bullets. “1) The Micronutrient study was guided by the Kaleidoscope Model/theory article (2015); / 2) For the latest work on Political Economy Analysis (of the FISP, Seed Policy and Grain markets), it was USAID￾Malawi that wanted the study to be conducted. My role was to identify an [….] FSP colleague [name] to lead it and to employ consultants to help with the data collection and write up. However, the methodology was designed by [consortium member] through [name] Political Economy tool. 3) Through [consortium member]/[name] research themes (e.g., on the CAADP case-study or the gender and nutrition policy study).” “1. Initial meeting of team from MSU and IFPRI to lay out research themes and plans / 2. Mostly MSU then deciding (among several of us) on themes for cross-country data analysis, as we had been engaged in that for some time already and continued. / 3. FSP then coordinating with country leadership for specific research activities there.” “For those I have been part of (except Tanzania) I co-designed or designed all in the proposal. For Tanzania, I just got the final proposal from lead PI and then within that I suggested a specific activity that went with the general.” “I only speak here of NAPAS: Malawi. As noted earlier, the policy priorities from which we selected were the policy reform commitments made by the government of Malawi in the New Alliance document for Malawi, signed in late-2013. We prioritized these and brought those selected into our work plan.” “I think I've addressed this in earlier comments.” “I'm not involved in the main FSP LAB project in a major way (I am involved in the Zambia Mission Buy￾In) so I don't have good insights on the decision-making process for the main FSP LAB project.” “USAID priorities / / Researcher priorities / / consultation within component team members and between component team, program management, and USAID AOR.” “We were given a mandate by USAID to come up with a conceptual framework on policy processes. We subsequently identified gaps in policy process/political economy literature with respect to food security policies. We then derived our model and wanted to test it on a set of very different countries and policy domains (input subsidies and micronutrients).” “a) Joint decision-making, by consensus, among all team members ([name], [name] from [consortium member]; [name], [name] from [consortium member]; [name] from [consortium member]).” “Based on researchers’ interest / what is needed in the countries from their knowledge / what funding is available. Broad topic areas are generally jointly agreed on by MSU, IFPRI and USAID. IFPRI designs the research they want to do, MSU designs theirs.” Source: FSP IL Mid-Term Performance Evaluation online survey, question q21, respondents from stakeholder categories 1, 2, and 6. Categories 1, 2, and 6 (Table 3) are FSP IL staff, AA and Buy-in staff, and FSP IL and AA staff involved in management. 75 Annex III.B Respondent feedback on the question: Were you involved in selecting FSP LAB research topics or policy analyses? If yes, please indicate how. “In consultation with USAID/BFS in all cases / / in consultation with USAID Missions in some cases / / in consultation with local/in-country research institutes/university faculty in all cases.” “By participating in team workshops” “C3” “Heavily involved in C4B and how the research gets carried out in countries” “I am a researcher involved with the project and identified topics in collaboration with IAPRI colleagues.” “I co-drafted the C4B (food systems) section of proposal and helped decide implementation. I wrote the proposals for the Myanmar work. I designed the Tanzania AA work on processed food from urban into rural areas. I designed the survey part of the value chain segments for the Senegal AA work and had input into proposal that way.” “I had to select appropriate research topics and questions within the FSP thematic areas in Zambia” "I was responsible for drafting the proposal for the NAPAS: Malawi Associate Award activity, finding staff for the project, and have participated in the project since its launch. The policy analyses for NAPAS: Malawi were defined by the New Alliance Country Cooperation Agreement for Malawi, but I was responsible for prioritizing these analyses." “I've been involved in the FSP work on policy process (C3) and in the Nigeria NAPP program” “As a component lead” “Policy analysis, capacity development, policy process” Source: FSP IL Mid-Term Performance Evaluation online survey, question q18, respondents from stakeholder categories 1, 2, and 6. Categories 1, 2, and 6 (Table 3) are FSP IL staff, AA and Buy-in staff, and FSP IL and AA staff involved in management. Annex III.C Respondent feedback on the question: Did you have a role in design OR conduct of any of the FSP LAB research topics or policy analyses? If yes, please indicate how. “In some cases, new work was determined based on USAID FTF priorities, e.g., climate-smart agriculture, resilience / / In some cases based on strong guidance from USAID/BFS, e.g., USAID expected that FSP would continue work on fertilizer policy and input subsidy programs, reflecting Mission demand for such work in many countries / / in some cases based on national government demand, based on consultations with government / / In some cases based on pre-existing research activities that were carried forward into the FSP IL work plan. / / It seems to me that in almost all cases, the work being done in each country and under C3 and C4 has its advocates and proponents, and is being appreciated by some elements within the FSP countries. I am not aware of any work being done under FSP that might be considered irrelevant to the needs of USAID Missions or African governments. However, I am not 76 Respondent feedback on the question: Did you have a role in design OR conduct of any of the FSP LAB research topics or policy analyses? If yes, please indicate how. convinced that much of the work under C3 is going to be very useful to either USAID or to governments, though there is great interest in learning whether there are ways to more effectively relate to governments as a way to make them more receptive to using policy guidance and technical analysis. I am not sure there is any magic bullets to be found here, other than just doing good work that is written in an accessible way and engaging civil society more explicitly in our outreach work.” “1) C3 conceptual framework: participated in joint development by MSU, IFPRI and UP of the Kaleidoscope Model (KM); 2) participated in the three micronutrient case studies; 3) participated in the Tanzania Big Results Now case-study currently underway” “C4B, and how it was then implemented in Tanzania, to some extent in Mozambique, and in supporting role for Nigeria. Also related to Ethiopia but [name] largely managed that.” “Designing a case-study on gender and nutrition to address cross-cutting issues.” “FSP C3/A4 Malawi—assisted in conceptualizing and implementing project” “Finding the right stakeholders to interview in Malawi (on the Micronutrient Policy study), conducting several interviews together with other colleagues and helping with the writing and editing of the report” “I am a researcher in FSP Zambia” “I am responsible for conducting specific activities and leading specific studies for the ‘agricultural mechanization’ component under ‘C4A-Upstream/Promoting Food Systems Transformation (with particular focus on policies toward farm inputs, mechanization, land and labor markets, land policies, youth employment)’” “I'm involved in the Zambia Mission Buy-in to the FSP LAB” “With colleagues on NAPAS: Malawi, I worked on several policy analyses that were on the project work plan: National Agricultural Policy, Contract Farming Strategy, Agricultural Zoning Strategy, Fertilizer Policy, and a few other activities.” “Yes, I took the lead in developing the Kaleidoscope Model for the C3 work.” “Yes, in design, already noted. I forgot to put that for the FSP C4B work (along with FSP AA and World Bank project) on chicken value chains in Nigeria I also co-designed that project. In Senegal, Nigeria, Tanzania I am a researcher under the lead PI. I am not a PI. For Myanmar I used to be PI and COP but I dropped out of it after 3 months in the field as COP and after I designed the proposals.” “Component lead” “Conducting research on various topics with different components” “Policy process and capacity development part” Source: FSP IL Mid-Term Performance Evaluation online survey, question q20, respondents from stakeholder categories 1, 2, and 6. Categories 1, 2, and 6 (Table 3) are FSP IL staff, AA and Buy-in staff, and FSP IL and AA staff involved in management. 77 Annex III.D Key Factors That Influenced the Selection and Conduct of the FSP IL Projects and Their Subsequent Influence on Policy Analysis and Review in Tanzania and Myanmar What distinguished the rankings that the evaluators got during the case-study interviews was the evaluators’ ability to solicit the stakeholders’ opinions about which factors had contributed to or detracted from these high-level rankings. Capitalization on Previous or Ongoing Projects: All of the staff associated with the FSP IL and AA/Buy-in Mission-funded projects emphasized that the initial selection of research was heavily influenced by the FSP IL researchers’ knowledge of the issues from earlier projects, and that the local COPs in each country were highly respected individuals with whom the FSP IL researchers had worked with on these earlier projects. This pre￾existing knowledge and pre-connected in-country leadership enabled them to connect quickly the FSP IL’s global agenda to the most pressing issues on the front burner of their principal local partner, the Lead Ministry/Ministries of Agriculture. The same knowledge helped the FSP IL leaders get small amounts of bridge funding (through Mission-funded AAs associated with their ongoing projects) that they used to jump start a series of pre-planning studies in FY 2012 in Myanmar and the first LGA study in Tanzania. Choosing National-Level COP and/or Partner: The fact that MSU had ongoing projects in both Tanzania and Myanmar helped identify two national leaders who had strong direct relationships to the FSP IL’s principal local government partner. These relationships helped facilitate the active engagement of the FSP IL’s chief ministry partners in two radically different political contexts: One in which the Tanzania government was actively involved in reviewing some of its inherited agricultural policies that were seen as constraining small-holder growth and development; and The other in which the reformist military government in power was interested in getting more information on Myanmar’s highly diversified cropping systems but not yet willing (or able) to commit to co-partnering on this research. Close Coordination with the Major Government and Private-Sector Stakeholder Groups: Having a strong COP/local partner made it possible for the FSP IL researchers to adapt their processes used to select and conduct the FSP IL policy analysis and research to these quite different socio-windows into the administration. In Tanzania, where the government was open to reform, the FSP IL elicited the active participation of the lead ministries into the initial design and execution of its initial policy analysis and research by recruiting some of the most promising junior technical staff to participate in the research teams during the first and second year.113 Given the political constraints on working with the lead agricultural ministries in Myanmar, the FSP IL focused on:114 The recruitment and training of qualified researchers from the country’s leading food policy think tank at the time, CESD; and Involving the three most active producer associations—beans and pulses, fish, and rice—in both the design and execution of the FSP IL’s initial diagnostic and livelihood surveys and value-chain studies. 113 In Tanzania, all first-year policy research projects were: 1) conducted by a mixed team of consultants and young technical officers from the lead agricultural ministries; 2) led by a respected national researcher funded by the project who was embedded in the Ministry of Agriculture; and 3) funded lightly by FSP IL and some bridge funding from the ongoing USAID project through an AA/Buy-in that was written by the FSP IL staff. One strength of the AA bridge funding was to enable the project to launch a rapid start up before the full funding from the FSP IL could be dispersed. The embedded leadership facilitated close coordination with the lead ministries in the selection of the research topics and research conduct. The same embedded leadership helped: Facilitate a series of highly participatory debriefings of the results to the lead ministries and parliament; and Lead the government to turn to FSP IL for help in harmonizing the debates around these issues through the creation of the first annual food policy workshop in FY 2013. 114 In Myanmar, each of the preplanning and Year 1 research projects were: 1) conducted by a team of researchers from CESD, an organization with which both MSU and IFPRI had worked previously; 2) led by a respected CESD researcher and team of FSP IL researchers; and 3) funded by some transitional funding in the preplanning year (FY 2012) and the FSP IL Asia C1/C2 support in Year 1. Although no Ministry staff were involved in this initial research, the FSP IL invested heavily in the communication of its major findings to lead policy makers in the Ministry through direct communication from the CESD director, as well as at multiple presentations to the wider donor community that were attended by the NGO leaders related to the three key producer associations—fishery, pulses, and rice. 78 Key Factors That Influenced the Selection and Conduct of the FSP IL Projects and Their Subsequent Influence on Policy Analysis and Review in Tanzania and Myanmar Continued Strong Linkages with the Wider Donor Community (including USAID): Both country programs were blessed by strong Mission leadership that: Provided a vehicle to connect them to the wider donor community; and Participated actively in the FSP IL debriefings and field activities. This close coordination had a direct and measurable impact on the selection and conduct of the scope and depth of the FSP IL policy analysis and research in both countries from the very first year. Tanzania: This coordination with the local Mission: Helped facilitate the transition of COP from one USAID project to the FSP IL, which ensured good focused leadership and coordination with the lead ministries from Year 1; Ensured that the FSP IL COP was invited to represent USAID on the country’s most important inter-donor coordination committee; Facilitated the design of a Buy-in to expand these activities and an interim bridge grant to support the activities until the Buy-in became available; and Provided some initial logistical and networking support for creating the PAG and the AAPC. Since Year 1, the AAPC, which the FSP IL helped create with strong assistance from the local Mission, has emerged as one of the chief mechanisms for: Harmonizing food policy analysis and research in Tanzania; and Ensuring that evidence-based research (including that which was facilitated or co-facilitated by the FSP IL) influenced policy review, formulation, or change by the host national government. Myanmar: The project’s strong coordination with the USAID local Mission: Helped facilitate the speedy design of the first FSP IL AA on September 24, 2014; and Provided an entrée to LIFT, a large multi-door trust created to coordinate multilateral and bilateral aid in the wake of a devastating earthquake that had been the principal focus of foreign donor support in Myanmar. This entrée, which was solidified by a series of debriefings on the FSP IL research during the FSP IL’s first year, had four important impacts on the FSP IL-facilitated research and policy analysis: It provided additional funds through a second AA grant from the LIFT multi-donor trust, which helped expand the project’s research and capacity strengthening activities starting in the second year; It affected site selection for some of the studies, since the LIFT donors wanted to see a stronger overlap with their intervention areas in the Delta and Drylands agro-ecological zones. This shift in siting and the donors increased access to evidence-based research on these priority areas helped strengthen the likelihood that FSP IL research would influence donor funding; and It connected the research results to the actual funders through LIFT and a new generation of bilateral funding that is just starting through GIZ, DFID, the EU, and USAID. Source: FSP IL Performance Evaluation case studies. 79 ANNEX IV 80 Annex IV.A. Implementation of C2: Capacity Building for Policy—Data, Analysis, Advocacy, Formulation, Consultation, Coordination, and Implementation115 Overview: Consortium members will team with local partners to develop and implement customized policy system strengthening programs in three focus countries over the life of the program. Countries will be selected jointly with USAID, based on assessments of the capacities and strengthening needs of national food policy systems conducted under C3 and on USAID country Mission interest. These assessments will identify key actors in each country’s policy system, evaluate the quality of their work and the degree of linkage they have with other components of the system, and identify potential points of entry to improve the quality and amount of information flow to strengthen policy and program design. Capacity building will be supported by competitive and/or collaborative grants for policy analysis in conjunction with C1, as well as expert advice on strengthening data systems. Capacity building in the selected focus countries will be further supported by the development of training modules on policy system and policy process, as well as modules on specific policy topics. These modules will also be used in regional workshops that bring together representatives of national policy institutes who choose to work together on a common theme. Methodology: FSP IL will develop customized strengthening activities for each building block and/or linkage in the system identified by the country assessment as needing improvement. Particular attention will be given to improving the effectiveness of data collection and policy analysis in support of policy formulation. Attention will be given to both organizational and human capacity dimensions of analytical building blocks. Based on the consortium’s experience in many countries, there are two fundamental requirements for success of a policy research unit or center and for its ability to strengthen the system in which it operates: ● ● First, a policy center must have autonomy to provide necessary work-related and financial incentives to attract and retain quality researchers and to emphasize applied policy research and policy outreach, as opposed to day-to-day administrative and bureaucratic activities. In many cases, this will require that the center be located outside the government in a university or private sector think tank, such as those supported by a consortium of U.S.-based foundations through the International Development Research Centre (IDRC). Other models do exist, however. In Bangladesh, for example, the Food Planning and Monitoring Unit (FPMU) within the Ministry of Food, and the Agricultural Policy Support Unit (APSU) in the Ministry of Agriculture, both show evidence of high-quality work. FSP IL will identify centers that will benefit most from FSP IL assistance and that provide the best prospects for driving improved policy outcomes. Second, a national policy center must be linked institutionally to government, the private sector, national universities, civil society, and donors. Though the particulars of each investment will depend on the country, the following provides a menu of instruments the consortium anticipates drawing from to support these centers: ● ● f ● Fellows Program: This program facilitates seconding people from a research center to key units within the policy system with a mandate to help that unit: 1) prepare effective policy memoranda and use other traditional internal mechanisms to reach policy makers; 2) articulate their assessment of policy research priorities to the research center to influence the focus of their research; and 3) with modest funds and basic methods, do additional qualitative analysis as needed to contribute directly to the units’ policy advisory/design mandate. Mentoring Program: Research personnel from a center, which could be independent of the university or an autonomous unit within the university, and a consortium member institution will work with the local university to help both male and female students conceive research topics related to FSP IL policy priorities, provide funding for the research, and serve as research supervisors for the students. This activity will be designed to provide incentives for the best students in this program to either join the center or serve in the ellows program (out-posted to a government or civil-society node), thus helping to strengthen the policy system. Policy Research Grants: These will be competitive grant funds for applied research carried out nationally and regionally. The consortium has several years of experience managing this kind of competitive grant in Zambia and other countries. 115 Source: FSP IL Technical Proposal. 2013. Pp. 11-12. 81 ● ● ● ● ● Policy Outreach Grants: These competitive grants will support innovative policy outreach initiatives. They will be awarded directly to researchers or to individuals focused on interpreting and using research results. The emphasis will be on teaming with researchers who have compelling research findings, and on using innovative communication tools and processes to get the message out to those who can make decisions. Strengthening of Data Systems: National units responsible for data collection will be able to draw on consortium expertise in all aspects of data collection and management to improve their systems. Linkages will be made with donors to obtain complementary resources for data system improvement. Policy Stakeholder Prioritization Workshops: These workshops will bring together representatives of key stakeholders to identify priority policy agenda topics and the specific questions for which additional analysis of options or additional empirical evidence is needed to formulate, monitor, or revise policies or programs. Short-Term Training: The consortium will design and implement training programs for stakeholders and researchers in the policy system to strengthen their ability to promote sound, inclusive, and environmentally sustainable policy and program design. Training will range from basic concepts (e.g., the difference between a policy and a strategy) to technical analytical topics (e.g., impact evaluation methods, gender-disaggregated data collection, and both quantitative and qualitative analysis methods), as well as the political economy of policy change and how stakeholders—especially women—can be most effective in articulating their voices. Some training sessions will be explicitly designed to give stakeholders from different national organizations the opportunity to collaborate in analysis and outreach activities on specific policy topics that would continue after the initial training is completed. Regional Policy Analysis Network Engagement: Bringing national policy analysis centers together in a regional network for joint participation in policy research activities of interest to several countries, training in appropriate analytical frameworks, and other forms of capacity building. 82 Annex IV.B In what ways could the capacity strengthening component be improved for greater policy impact (i.e., support to local associations, farmer groups, NGOs, journalists, government, and others)? Please provide maximum of three bullet points. “For the response I gave, there is an exception for journalists because the journalist trainings that took place in Malawi were effective in building capacity of journalists but not NGO's farmer groups and local associations in general. It was also effective in building capacity of several government employees. To improve its effectiveness, the trainings need to have a long-term view, e.g., towards building institutions as was the case in Zambia where USAID support through MSU led to development of IAPRI, now a strong and well-respected think tank in the region. This is not happening in Malawi. The support is very short sighted and short lived.” “1. Provision of information on in-country status quo to government officials / 2. Provision of implementable options to government officials / 3. Assistance in preparing draft sections (if requested by government officials).” “A more explicit focus on developing policy change strategies. This has started, but only recently and not in all countries.” “Among the areas of capacity strengthening could include host governments’ being trained on monitoring and reporting of results. This could be an inclusive process.” “By providing financial and technical support to capacity building initiatives and their impact assessment to better design the programs.” “Don’t know or N/A” “Focus on applied research collaboration with local universities. PERIOD.” “Increased engagement with various stakeholders to identify areas of need in terms of capacity building / Increased engagement with various stakeholders during and after policy research, in particular, stakeholder involvement in research findings dissemination / More engagement, particularly with government / Research dissemination and validation workshops with various stakeholders.” “To my knowledge (and as reflected in the annual reports) there has been no policy influence as a result of capacity strengthening.” “Very limited engagement by partners in Malawi as to what their capacity strengthening needs were. In consequence, the training provided was strongly supply-driven. Effort is needed to better identify capacity needs.” “University does not reward this work effectively; land grant university needs to embrace this mission.” “Alleviating the trade-offs between research, policy change in the short run, and capacity building (which is less visible).” “Encourage new permutations of FSP to work with national policy research institutes as prime consortium partners, not as small sub-contractors.” “Give serious attention from the beginning; conduct a needs assessment in the beginning; have a strategic plan.” Source: Online survey question q33. Answers EQ 2.3: page 13, Sec. 3.2. Respondents were stakeholder categories 1 (FSP IL staff), 2 (AA/Buy-in staff), 3 (FSP IL partners), and 6 (FSP IL and AA/Buy-in management staff). 14/19 respondents answered question with comments. 83 Annex IV.C What capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities in this country? Please provide maximum of three bullet points. “Depends largely on the views and proclivities of the individual lead people in FSP, which provides guidance to the rest of the team about what effective collaboration with local partners really means. It is my view that FSP could be much more effective in building local capacity as joint products of the research and outreach work that FSP does. / Some people in FSP realize this keenly. / Some people in FSP think of capacity building as identifying the best researchers in local institutions and hiring them as consultants. Very little attention to long-term institution building.” “1) Seminars/workshops / 2) One on one trainings / 3) Online/through electronic devices.” “1) We have not found the best capacity delivery methods because as I said above, the capacity trainings were short lived (maximum of 5 days trainings, mostly to journalist). There was not enough funds devoted to long lasting capacity building efforts. In Zambia it took at least 12 years of continuous support where MSU staff and local Zambian/African staff are being capacitated to do quality and useful policy research. The first 2 years of FSP engagement in Malawi did not have that vision/element.” “1. In depth meetings with government officials / 2. Attendance of, and participation in, meetings, workshops, etc.” “A lot of attention has been given to FSP’s journalist training initiatives.” “Don’t know or N/A.” “Engaging implementing partner staff in all aspects of research design, implementation, and analysis” “Focus on applied research collaboration with local universities. PERIOD.” “Participation of various stakeholders in training, policy dialogues, and research findings dissemination and validation workshops / Continuous engagement with stakeholders on a recurring theme / Training Workshops with practical components.” “Training of appropriate post graduate students.” “a) Collaborative research / b) direct contracting through local partners, with MSU as subcontractor / c) long-term training does not work unless local conditions of service improve significantly.” “Consultations; policy dialogues; analytical training; institutional strengthening.” “In-service formal and on-the-job training of counterparts” Source: Online survey question q33. Answers EQ 2.4: page 13, Sec. 3.2. Respondents were stakeholder categories 1 (FSP IL staff), 2 (AA/Buy-in staff), 3 (FSP IL partners), and 6 (FSP IL and AA/Buy-in management staff). 13/18 respondents answered question with comments. 84 Annex IV.D What types of synergies have there been among the various MSU, IFPRI, and University of Pretoria capacity strengthening approaches in this country? Please provide maximum of three bullet points. “Collaboration in capacity building in the media and researchers in three countries/Collaboration on training.” “Don’t know or N/A” “In Nigeria, I guess nothing. Tanzania? Who knows? Senegal? Who knows?” “Not many. UP has done some training at SUA, and it was well regarded, but MSU was not much involved as far as I know.” “a) IFPRI has contributed significant public policy skills / b) UP contributes significant deep knowledge of nutrition policies / c) MSU has the deepest understanding of field conditions in African agriculture and in agribusiness transitions.” “Joint implementation of in-service and on-the-job training.” “Media training in Malawi was effective / NGO training in Myanmar brought the best synergies / regional consultation on policy process was effective.” Source: Online survey question q34. Answers “Evidence of Capacity Strengthening Synergies Between MSU, IFPRI, and UP.” Page 14, Sec. 3.2. Respondents were stakeholder categories 1 (FSP IL staff) and 6 (FSP IL and AA/Buy-in management staff). 6/11 respondents answered question with comments. 85 86 87 88 Annex IV.G. Case-Study of the Organization and Implementation of Capacity Strengthening Activities Covering Different Content for Different Target Groups in Tanzania and Myanmar Local Execution Partners: All of the major stakeholder groups recognized that both projects were extremely successful in building the capacity of their local execution partners. In Myanmar, where the opportunities to work with the local Ministry of Agriculture staff when the project started were very limited, these activities focused on the national think tank, CESD. This training involved formal training in research methods, sampling, and database training, as well as on-the-job mentoring to help staff build their presentation, writing, and project management skills. This consistent training over two and a half years has produced a nationally recognized cadre of food security policy researchers. Some of the best evidence of the quality of the capacity building has been the high rate of CESD trained personnel leaving for higher, better-paying jobs in the NGO sector or for Ph.D. training in the United States and Australia.116 Junior Ministry Staff: In contrast, the Tanzania country program has been able to involve its national partner staff in the design and execution of its research programs from the first year and to expand this collaboration to work effectively with the country’s major agricultural university in Year 2. Once the staff were trained, they stayed in close contact with the FSP IL researchers and helped with the analysis and initial ministry debriefings. During the first year, this close communication with the trained staff morphed into an informal network of trained ministry staff that refer to themselves as the Associate Policy Analysts (APA). Through their listserv and blog, the APA members keep one another informed about the FSP IL’s activities in their country as well as new initiatives and training opportunities by other programs. Most have continued to be actively involved in the PAG and Annual Policy Conference, where they have been an active voice that has promoted the studies with which they were affiliated. Staff reported that their participation in the APA and other project events has helped them stay abreast of the new FSP IL and ASPIRES (Buy-in) research findings as well as donor trends. Senior Ministry Staff: All nine senior Ministry staff interviewed in Myanmar highlighted the critical importance of the one-on-one mentoring they received from the FSP IL’s in-country directors/COPs. This assistance included one-on-one debriefings of specific studies and eventually grew to helping the Ministries with document review and revision and preparation of presentations and speeches. In general, they cited relying on the COPs/country directors for advice on documents and networking opportunities. University-Based Researchers: One major strength of the FSP IL—which was anticipated in the original design— was its willingness to invest in building the capacity of certain departments and institutes in the countries that it served. Although the short-term costs of working with the national universities were often higher than working with an autonomous think tank since most of them were not used to conducting applied research, it was conceived that the longer term returns on this investment would far outweigh the costs. Although the concept was good, the reality of working with the universities varied widely. In Tanzania, for example, the FSP IL global teams were able to develop three research programs with three separate teams of researchers in the Department of Agricultural Economics that were funded through three separate funding mechanisms. Although many of the Tanzania faculty had extensive experience doing policy research, this was one of the first times they had had the opportunity to present their findings to the government through the FSP IL￾facilitated ministry debriefings and APC sessions. Staff reported that, if this involvement continues, it would dramatically increase the role of the SUA faculty in Tanzania food policy. In Myanmar, the immediate prospects for working with the main agricultural university were dimmed by the fact that the university is still legally part of the Ministry of Agriculture. As a result, any professor’s research needs to be approved by the Ministry of Agriculture and probably focused on rice or some technical issue related to rice. 116 Out of 16 CESD personnel trained, eight have left the project [two have enrolled in international Ph.D. programs (another is planning to enroll in a Ph.D. program), two have left for NGO work, two left for the U.S. embassy, one left to work for the LIFT multi-donor trust fund; one left for the Myanmar Development Institute]. As of the time of the final revision of this evaluation report, eight trained CESD staff remain. The MSU faculty working with the Mission-funded AA provided letters of recommendation for all university applications made by CESD staff working with the project. They also provided guidance and follow up on GRE preparations, especially exam practice (source: email communication, October 3, 2017). 89 Once the agricultural university is placed under a new ministry of higher education, it may be easier to build joint research programs. In the short-term, therefore, the FSP IL and AA focused their capacity building on providing two-week technical training programs for some of the staff as well as mid-level technical specialists from the Ministry and the FSP IL local partner. Both groups of professors were very interested in knowing which studies had been published in refereed journals, especially the SUA professors who were required to meet international standards for publication and tenure. Development NGOs: Even the work with NGOs can vary widely between countries and between different periods of a country program. In Myanmar, the NGO sector is still dominated by a series of NGOs that focus on civil rights, democratic reforms, and emergency response. It is thus not surprising that most of the stakeholder interviewed ranked the FSP IL/AA low on its coordination of capacity with international and national NGOs. This perspective overlooks the program’s development of a strategic alliance with one of Myanmar’s largest national NGOs, Priority Designs, which has strong multi-donor funding to support activities that overlap with most of the areas where the FSP IL/AA research is being conducted. Though the relationship is still being developed, the staff highlighted a number of areas where their institution had been directly impacted by the FSP IL/AA’s publications, policy briefs, inclusion of staff in some of their research, and informal technical backup. Private Sector: In Myanmar, the FSP IL/AA focused their capacity building on the three NGO associations that represented rice, pulses and beans, and aquaculture. This capacity building focused on: doing evidence-based presentations on food policy issues at the bi-weekly meetings that each associate organized at the Chamber of Commerce; involving them in the design and review of the scopes of work for specific studies; and, more recently, working on the design and execution of the regional-level agriculture strategies that the FSP IL/AA are facilitating in about half of the regions/states where it currently intervenes or has intervened in the past. Two of the three farmer groups interviewed in Myanmar ranked these capacity delivery mechanisms very high.117 The Myanmar fisheries associates stated that this support had completely transformed their sector by attracting a new generation of donor support slated to take off in the near future. In Tanzania, the FSP IL and the Buy-in have also focused their private-sector activities on the major professional associations that represent the different private-sector farmer and merchant groups. In this case, the principal formal vehicle for building capacity was their participation in the PAG and the Annual Policy Conference, and the principal informal vehicle for building their capacity has been the informal linkages to other state and donor actors that their participation in these formal knowledge dissemination and learning events has forged. Parliamentarians: The FSP IL, AAs, and Buy-ins have worked directly with parliamentarians in both countries, in most cases in the form of informal meetings to debrief them on particular topics coming up for a vote. One unique synergy between the FSP IL/AA research programs and the AA was the organization of a training workshop for the 19 mid-level technical staff affiliated with the AA, during which they received assistance in writing briefing papers on 19 value chains. These value chain papers were loaded up on tablets that were given on long-term loans to some of the key parliamentarians affiliated with the major committees associated with agricultural policy and to senior ministry officials that were critical to food policy. To date, the communication with parliamentarians in Myanmar has focused on formal debriefings on specific policy issues at the national level with more extensive debriefing through the local partner’s policy advisor of the state-level parliamentarians in the regions where the FSP IL and AA have worked. Wider Public/Media: Although the FSP IL was successful in pilot testing the new Push Mobile technology for communicating with farmers in Tanzania during the first year, the fact that there was no wider stakeholder consultation affected farmers’ willingness to participate in the pilot. Based on this lesson learned, the FSP IL was careful to include stakeholder consultation in their initial pilot test of Maxicom E-payment technology in Year 2. That consultation-focused work included a media component using radio and telephone spots, loudspeakers, flyers, and a consultation process through the local government leaders to promote the pilot test. During the same period, several researchers took the initiative of contacting journals and seeding articles on their work to build national and international appreciation of the results. Once the ASPIRES Project developed a formal 117 Due to an issue on timing, the evaluators were not able to meet with the staff in the rice producer association that had worked most closely with the FSP IL, so this was not a representative interview. 90 communication strategy in 2016 and hired a full-time communication specialist, it has developed better-targeted materials and media linkages for each of its broad categories of activities. To date, the FSP IL/AA in Myanmar has not developed a formal strategy for coordinating with the press. When stakeholders were interviewed about this, most of them agreed that this was a wise political decision given the project’s need to be viewed as politically independent in a context where most newspapers and media outlets are not. There have, however, been a number of articles about their research that were picked up by local journalists who attended some of the FSP IL/AA debriefings at the national chamber of commerce. One possible avenue of strengthening this communication would be to route the FSP IL materials through the journalist training courses that are being conducted by some of the projects in the USAID/Myanmar Democracy and Governance portfolio. Local Stakeholder Groups: Although the stakeholders interviewed in both Myanmar villages knew they had been studied, they were never debriefed on the survey results. Both Myanmar communities expressed a strong interest in learning about the global results of the research and requested that they and their regional-level elected parliamentarians be given copies of the summary results in both English and Burmese to help them better lobby for some of the local-level supports that they feel their communities need and deserve. The local producers and millers interviewed in Tanzania were also interested in getting information from the studies related to the use of the e-payment system to increase the efficiency of collecting the FSP LGA (Local Government Authority) Crop Cess tax study. Many of the farmers, merchants, and local government officials with whom the team met in Kilombero District reported that they had been interviewed and/or consulted during the: ● Beginning phases of the initial research that led the team to conclude that it was unlikely that the local governments would agree to changing the agricultural levies unless there were some parallel investment in developing more efficient systems of tax collection; and ● Initial pilot tests of the e-payment system in Kilombero District that are currently being scaled up through a joint initiative with Africa Lead. What concerned these stakeholders was that they were no longer being consulted—by the FSP IL/ASPIRES researchers or the higher-level officials of the government—about of the consequences of the resulting reports, some of which they deemed highly negative. Given the high cost of hiring enumerators, one option for improving the support to this group would be to extend the contracts of the enumerators so that they could return to the villages that they studied in order to debrief the full 100 communities or at least a sample of these communities (e.g., 25 percent). Other Donors: Most of the donor and NGO representatives interviewed stated that they were strongly influenced by the-high quality FSP IL presentations that they attended at various donor workshops and meetings, which was their first contact with the FSP IL and the concept of evidence-based policy research. Once this door was opened, they reported benefitting from the consistent follow-up they had from the in-country representative and COP, which included making sure they were aware of how to download the FSP IL’s published papers and reports. They also reported a number of instances in which this information affected their global policy documents and the types of agricultural programs they were supporting through their bilateral country programs as well as the multilateral support they contributed to LIFT.118 In Myanmar, this consistent backup from the COP was assessed even higher than the papers generated by the project. All of them reported using the online documentation for policy papers and new proposals and seemed to have no difficulty in navigating between the FSP IL web materials and the local Mission-funded materials that were posted. 118 Source: FGD and KII with donor representatives and LIFT staff. FSP IL Mid-Term Performance Evaluation. 2017. 91 ANNEX V 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 Annex V.D Annex V.D. In What Ways Could the Country and Regional-Level Support Be Improved to Better Support Policy Influence or Policy Change? “A midterm review may be useful in taking stock of what has worked and what can be done to enhance/influence policy change.” “Better identification of policy research that involves agreement of topics before the research is started rather than trying to find relevance of that research to a country after the research has already been developed.” “Ensure that they are demand driven. Has a single country asked for policy processes research? How about the farm-size research? It is interesting research but no policy maker (except maybe USAID) requested it.” “We would need to identify key opportunities then seek them out; I sense that research has been driven by publication interests rather than interest in changing policy.” “Focus on applied research with local universities. Period.” “I think the coordination has been quite good. Global work could not proceed without country-level content. And since key people on C4B were also key people on two major AAs/buy-ins, the coordination was good.” “Include the regional [United Nations] UN officers.” “Policy reform is gradual process, it requires sustained investment and time. [T]here [is] a need to design projects with a realistic time lines and enough finding to actually achieve policy reform.” “Regional-global activities in the case of Malawi were quite divorced from what was going on in-country. Due primarily to work planning on different time horizons that prevented better integration. Country-level work not too pertinent for regional and global discussions.” “The most successful approaches to influencing policy change in country have been when the FSP team has involved local partners including government and civil society.” “[Pay more] attention to process and how the various institutions involved need to relate to one another in a way that aligns incentives toward the goal. As important as technical research quality is the most effective research can go nowhere in a country unless the eco-system is favorable. Bottom line: partnerships and process are decisive. Parts of FSP still conceive of supply-driven research quality (and it is generally very strong) as the key ingredient. In the long run, really good global or regional research can influence perceptions and the engagement of other researchers in important ways. But, in the short and medium term, this work will not have impact unless there is serious attention to working hand in hand with local units that are generally much closer to governments.” “We would need to identify key opportunities then seek them out; I sense that research has been driven by publication interests rather than interest in changing policy.” “Continuous interaction with local researchers/consultation with policy makers/capacity building/policy communications workshops.” “Stronger emphasis on understanding and strengthening policy change processes.” Source: FSP IL Mid-Term Performance Evaluation online survey. Question 37_text. 127 Annex V.E How Has FSP IL Communicated Policy Analysis, Recommendation, and Policy Options to Policy Makers and Others in a Position to Influence Policy? “Tangible impacts of C4A on reforms of government agricultural input subsidy programs / / * tangible impacts of C1/C2 on same / / * tangible impacts of C4A activities on land policy formulation in some countries.” “Through reports/publications and workshops.” “1) Workshop and seminars / 2) Dissemination of policy briefs / 3) Provides technical support on easing access to information by policy makers” “1) Many policy presentations to many organizations at global and regional levels; 2) strong participation in country-level policy conferences; 3) feeding information to in-country leaders to use in their own outreach (policy briefs, researcher reports, informal communication).” “1) Meetings with individual officials; 2) participation in meeting and workshops; 3) sharing of insights during progress and close-out meetings.” “Consultative stakeholder engagement/ participatory stakeholder workshops, Annual Agricultural Policy Conferences (AAPC), Presentations to key decision makers from the Agricultural Sector Line Ministries (ASLMs), Agricultural Sector Development Program (ASDP 2) Socialization and Capacity Building Workshop” “1) Formal presentations; 2) joint working groups or task forces; 3) small group meetings; 4), one-on-one meetings.” “1) In-country workshops and seminars with policy makers and civil society; 2) DC-based policy seminars with donors and global practitioners.” “In the case of NAPAS: Malawi by engaging in the Ministry of Agriculture on a day-to-day basis and interacting with other Ministries, donors on agriculture, and to other stakeholders in the agricultural sector.” “Policy dialogues/workshops / / Policy briefs / / Media / - Collaboration with various stakeholders” “Policy documents, seminars and events, direct contact.” “1) Through in-country collaborators; 2) events; 3) journal articles and discussion papers.” “1) Tanzania cess example is the one I know of; 2) UP has had some success in south Africa by virtue of their interaction with policy makers there.” “1) Articles; 2) briefs; 3) talks.” “1) Discussion papers; 2) consultative workshops; and 3) capacity development workshops.” Source: FSP IL Mid-Term Performance Evaluation online survey, Question 40_text. 128 Annex V.F In What Ways Could FSP IL Communication Improve? “1) Production of briefs that are relevant to policy makers; 2) distribution of the briefs through personal contact rather than having them posted on a website somewhere; 3) personal meetings with people of influence at least twice a year when FSP IL researchers are visiting a particular country where their research can be relevant.” “1) By reaching people through current communication medium.” “Greater use of local media; contributing op-eds to local newspapers and radio interview—as well as Africa￾focused blogging outlets (e.g., The Conversation, Mail and Guardian Africa).” “Improved engagement and collaboration with governments and other stakeholders, in particular, the media/Strengthening partnerships and involvement of various stakeholders in-country.” “Possibly through more proactive designed networking activities with stakeholders. Those which took place were quite issue-specific and ad hoc to the demands of the issue in question.” “Translation of all policy outputs to local languages.” “1) Website revamp has helped; 2) good at targeting international audiences with recent research findings.” “1) Better engagement with policy makers from the beginning; 2) communicate using less academic language.” “More briefs and articles.” “Regular (e.g., 3/6 monthly) brief progress reports using a prescribed template.” “1) Research summaries in local language; 2) engagement with press.” “We should put out more policy briefs.” Source: FSP IL Mid-Term Performance Evaluation online survey. Question 41_text. 129 ANNEX VI 130 Annex VI.A. FSP IL Mid-Term Recommendations for Strengthening the Capacity of the FSP IL M&E System to Inform Strategic Planning Recommendation #1: Encourage all AAs and buy-ins to collaborate with the FSP IL in conducting a country-level analysis of its existing outcome and output indicators for capacity strengthening for all of its major partner institutions. The FSP IL did create a very good set of output and outcome indicators to track the number of institutions affected by this capacity building, as well as the number of USAID-related institutions and private-sector entities affected that received some sort of training or capacity strengthening. This includes (Annex 6.B): ● ● ● FSP IL Outcome Indicator 6: Number of institutions where trained individuals are applying new practices; FSP IL Output Indicator 7: Number of USAID operating units supported; and FSP IL Output Indicator 8: Number of private sector and civil society organizations/entities assisted to participate in policy formulation process. Currently these three indicators are only being tracked for activities funded by the FSP IL directly and/or co￾funded by the FSP IL. This way of tracking ignores the large number of activities that are being funded by the AAs and Buy-ins, as well as other leveraged funds. Since the vast majority of the capacity activities are being funded by the AAs and Buy-ins, this distorts the utility of the data for strategic planning. It would be very useful: ● If each level of the FSP IL—i.e., the core team and the Mission-funded Buy-ins—were convinced that it would help them to report on the same three indicators for each of the major institutions that they work with (i.e., executing agencies for activities as well as collaborating institutions); and ● If these disaggregated indicators were used for strategic planning by the FSP IL global teams as well as the Mission-funded programs. This type of disaggregated analysis of the combined FSP IL and Mission-funded AA and Buy-in award data on the same group of indicators would provide all of the key central (FSP-IL, USAID/BFS) and national-level (AA/Buy￾in/USAID bilateral mission) actors with comparable data on: ● ● ● Which institutions have been and continue to be the focus of the FSP IL/AA/Buy-ins’ capacity building (Outcome Indicator 6) in each country; Which USAID operating units (including key programs like Africa Lead, CAADP, AGRA, ReSAKSS, and RENAPRI) have been and continue to be supported by the FSP IL/AA/Buy-ins’ capacity building in each country (Outcome Indicator 7); and Which categories of private sector and civil society organizations/entities have been and continue to be assisted by the FSP IL/AA/Buy-ins’ capacity in each country (Outcome Indicator 8). For this information to be useful, this tracking needs to focus on the principal partners going forward over the next two and a half years, rather than every actor they have consorted with during the first three years. This information is critical to informed strategic planning by the FSP IL, the AAs and Buy-ins, and the USAID bilateral missions and headquarters units that support FSP IL, as well as the other regional capacity building programs like Africa Lead and AGRA with which the program will be working in FY 2018 and FY 2019. Recommendation #2: Develop a simple capacity self-assessment tool that the FSP IL global teams and country programs can use for tracking capacity building for specific institutions and/or units. Based on their initial discussions with some of the key “trained individuals [who are] applying new practice” referred to in Outcome Indicator 7 (Annex VI.B), the evaluators are recommending that one of the leading FSP IL M&E specialists, who is the current FSP IL director, pilot test the development of a simple self-assessment tool such as the ones that USAID has endorsed for use with NGOs in developing countries and to track core community capacity. Most of these tools use a simple four-step process that can be executed during a single focus group session and updated periodically (Text Box 1 below). 131 Annex VI.A. Text Box 1 Sample Steps for Developing a Capacity Self-Assessment Tool Sample Steps for Developing a Capacity Self-Assessment Tool Step 1: Participatory Review: The FSP IL/AA/Buy-in staff member who has worked most directly with the unit/institution engages in a participatory review of their joint capacity building activities to date and the impact that these activities have had on that unit/institution and the individuals within that unit/institution. Step 2: Identification: Based on this participatory review, the staff member guides the team through a participatory identification of a small number of key capacities that these activities have helped develop (5-10). Step 3: Ranking: During the same session, the staff member asks the FGD members to rank their capacity for each of the key capacities identified in Step 2 during three different time periods. A retroactive baseline indicator of the beneficiary institution’s capacity before they started working together; A current assessment of these critical capacities today; and A series of targets for where the two partners would like to see this capacity going over the next two and a half years in order to increase the changes that these institutional achievements in capacity strengthening could be sustained once the FSP IL funding ends in January 2020. Step 4: Joint Planning: The team brainstorms on what types of activities will be needed to help the institution/unit achieve some of the targets it identified in Step 3, which ones can be funded by the FSP IL/AA/Buy-in and which cannot. Source: FSP IL Mid-Term Performance Evaluation based on the consultants’ experience with NGO self￾assessment tools for food security projects. May 2017. The genius of this tool is that it provides a simple mechanism for more-informed joint planning of capacity building and periodic updates on how that capacity strengthening is proceeding. Recommendation #3: Work with USAID central-funded projects to look at ways the capacity self-assessment tool can be used to track the FSP IL’s collaboration with important regional initiatives that USAID supports. Given: 1) USAID’s interest in ensuring that the FSP IL coordinates with these regional initiatives;119 and 2) the passionate interest of the FSP IL staff that believe strongly that these regional initiatives are critical to ensuring that the core capacities that were developed under the FSP IL are maintained120 once the FSP IL ends, the evaluators feel that the FSP IL administration needs to make this a priority for its M&E system over the next two and a half years of the project. This type of joint planning would be facilitated by the development of a self-assessment tool to facilitate: ● A better understanding of how these regional initiatives have worked with the FSP IL and the Mission AAs and Buy-ins in the past; and ● What the most critical challenges they are likely to confront during the next two and a half years are. Recommendation #4: Strengthen the capacity of the FSP IL and AA/Buy-in staff responsible for the design and execution of the M&E systems to support these new activities using the existing FSP IL M&E systems. Although the basic FSP IL M&E systems are very good, they are understaffed. For the FSP IL and field programs to be able to strengthen their ability to use these systems for strategic planning, they need training and technical backstopping. Currently, almost all of the M&E activities are being conducted by individuals who have only 5-10 percent of their full-time employment (FTE) devoted to M&E and reporting. One of the first steps to executing this indicator (for M&E in general and for capacity building in particular) will involve: ● Increasing the amount of time (that is, FTEs) that key staff specialized in M&E staff can devote to the design and execution of this better harmonized M&E system; 119 FSP IL Mid-Term Performance Evaluation interviews with Washington, DC-based BFS staff, February-March 2017. 120 FSP IL Mid-Term Performance Evaluation FGDs and KIIs with the individuals leading these initiatives. 132 ● ● Using these FSP IL specialists to conduct light on-the-job trainings to help the senior FSP IL staff understand the new system and use it to track their country-level interventions; and Building the capacity and FTE devoted to M&E of the AA and Buy-in-level staff that are tasked with supporting the country-level M&E systems. Once these new systems are put in place, the AA and Buy-in-level staff tasked with tracking these indicators will become the de facto local capacity building experts of the project, which will be critical to planning the FSP IL’s exit strategies during the next two and a half years.121 121 The FSP IL’s scheduled end date is January 14, 2020, which is three months into FY 2020 so effectively the time remaining on the project is FY 2018 and FY 2019 plus the remainder of FY 2017, which ends September 30, 2017. 133 134 135 136 Source: FSP IL Administrative unit at the request of the FSP IL Mid-Term Performance Evaluation team. June 3, 201 137 Annex VI.D. Stakeholder Perspectives on How FSP IL Is Meeting or Not Meeting Customer Needs (ERQ 4.1) Stakeholder Perspectives on How FSP IL Is Meeting Customer Needs Stakeholder Perspectives on How FSP IL Is Not Meeting Customer Needs USAID Missions/AAs/Buy-ins Good reporting (after slow start) Helped design Mission-funded AAs/Buy-ins Initial FSP IL supply-driven/diagnostic work helped re￾examine conventional wisdom and identify new policy options Increased the USAID field Mission’s visibility as food policy actors In-country COP and/or country director facilitates coordination Building in-house capacity of key national partners Policy briefing papers Strong synergies between lead institutions Now realize they might need more assistance on reporting country-level impacts (CS, TZ, & M) Current research by the global teams not well connected to the priorities and activities of the Mission￾funded Buy-ins (online survey, TZ, M) Mission AORs sometimes needed quick advice/policy papers that did not fit with the workplan Lead institution do not always collaborate, which increases management burden for field Mission Wider Donor Community No information in the online survey Consistent high-quality presentations of FSP IL/AA and Buy-in work in the case-study countries are affecting new donor designs and budgets Donors like presentations and briefs when they are just learning about research studies Donor would like full studies USAID/BFS Appreciate reports, debriefing at USAID, publications, willingness to respond to questions Strong full-time AOR in USAID/BFS facilitated getting a response to any request for information Would like more regular updates from teams on global research teams Would like more information about country-level activities Would like more information on how they collaborate with Africa Lead and other USAID-funded projects by country In-Country SAKSS Nodes for CAADP Process FSP IL/AA and Buy-in AAs are helping countries increase the quality of the data they are reporting This information is not being tracked by the project M&E system In-Country Ministries FSP IL has hired experienced national staff and linked with experienced, well-connected partners that are known and trusted The high-quality expatriate staff has worked well with the national staff and partners Some portion of the activities of the FSP IL/Buy-in or Buy-in have been embedded in the lead ministry with other portions independent FSP IL/Mission-funded Buy-in support for the creation of policy units within ministries very positive FSP IL’s willingness to build capacity of junior staff through formal training and affiliation with its research Senior staff conduct regular debriefings that are co-organized with the in-country COP/field director and junior staff who participated in the work (from universities and the ministry) In-Country Universities Meeting needs in countries where they collaborate with university-based research institutes Not working with universities in all countries 138 Stakeholder Perspectives on How FSP IL Is Meeting Customer Needs Stakeholder Perspectives on How FSP IL Is Not Meeting Customer Needs Works with six of the eight universities in the regional ReNAPRI network in Eastern and Southern Africa On-the-job and formal training has tended to focus on building capacity to execute specific studies that the institution was contracted to execute Collaboration with these universities has not followed the model outlined in the proposal, which was to be based on an initial assessment and more targeted assistance to build the core capacity of the institution Civil Organizations Tanzania has developed a highly successful model for outreach to civil organization through the AAPC and PAG Policy briefs in local languages much appreciated Presentations in local languages at the regular meetings of the different sub-groups at the national chambers of commerce much appreciated Strong in country COPs/country director Relatively little documentation on stakeholder consultation Limited collaboration to date with Mission-funded Buy￾ins to determine most effective means of reaching the general public and most important civil organizations In-Country NGOs Most effective linkages to the wider NGO community are in Tanzania through the APC and PAG Strong connection of FSP IL and AA to one of the leading national NGOs that works in all of the areas where FSP IL and the AA have done field studies In general, this has not been a priority for FSP IL in most counties, though most of the AAs and Buy-ins work closely with the NGO community Private Sector Major impact on private sector in Tanzania through the APC and PAG Major impact on three private-sector value chains (pulses/beans, fisheries, rice) through presentations in local languages and briefing papers at producer association meetings at the national chamber of commerce Emerging impact on regional-level value chains due to collaboration with producer associations at the state level in Myanmar through presentations and briefing papers in local languages Producer groups expressed strong interest in getting more information on how international trade rules, regulations, and standards affect their business Producer groups want more presentations and briefing papers in local languages and more exchange with researchers State Governments Evidence that the data is being used for state-level agricultural strategies No direct information since state government officials were not interviewed, only document review and interviews with the staff working with these groups Source: FSP IL Mid-Term Performance Evaluation FGD and KII. 139 ANNEX VII 140 Annex VII.A. Key Informant Interview Guides Annex VII.A.i. KII Guide for Senior USAID, FSP IL Consortium Management Staff, Country Leads and Co-Leads, and Relevant USAID Mission Staff in the Two Case-Study Countries Identifier Response Interview number Date Name/s122 Sex and age Country/location Duration of interview Evaluation Question Cluster 1: Policy Analysis and Research 1. How did FSP IL make decisions on policy analysis or research? (ERQ 1.1) 2. Did you have a role in selecting, designing, or conducting of any of the FSP IL or Mission-supported research topics or policy analyses? (ERQ 1.1) 3. How did the design and conduct of these policy analyses influence policy review, policy formulation, or policy change? (ERQ 1.1) 4. What areas of policy analysis and research need more attention for better policy environment outcomes? (ERQ 1.2) 5. What could be could be done to improve coordination of research and policy analysis activities at the country level? (ERQ 1.2) Evaluation Question Cluster 2: Capacity Strengthening 1. What could be done to improve coordination of capacity strengthening activities? (ERQs 2.1-2.2) 2. In what ways could the capacity strengthening activities be improved for greater policy impact? (ERQ 2.3) 3. What capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities? (ERQ 2.4) 4. What types of synergies have there been among the consortium members? How could these synergies be improved? (ERQ 2.4) Evaluation Question Cluster 3: Progress toward Policy Outcomes 1. In what ways could the country and regional-level support provided by the FSP IL be improved to better support policy influence or policy change? (ERQ 3.1) 2. Please rank the following FSP IL components in terms of contribution to effective or rapid policy change at the country level (1 is best; 6 is least) (ERQ 3.2) ● C1: Country/regional-level collaborative research (on farms, firms, markets, and formulation/analysis of policy options); ● C2: Country/regional-level capacity building for policy (data, analysis, policy, advocacy, consultation, coordination, and implementation); ● C3: Global collaborative research on support to the policy process and policy capacity; 122 A few key KIIs will be with two people not just one. Names will be deleted from database, for use by evaluation team only for initial data entry. 141 ● C4A: Engagement on global policy debates on food security and upstream agri-food systems transformation; ● C4B: Engagement on global policy debates on food security and downstream agri-food systems transformation; and ● C5: Strategic analytical agenda in support to donor policy strategy. 3. How has FSP IL communicated policy analysis recommendations and policy options to policy makers and others in a position to influence policy? In what ways could this communication be improved? (ERQ 3.2) a. To what extent do you agree/disagree with the statement “FSP IL works closely with regional partners and stakeholders to affect more rapid policy change?” (ERQ 3.3) b. How effective has this coordination been? How could this be improved? (ERQ 3.3) 4. In your opinion, how effectively is FSP IL working with these different stakeholder groups? (ERQ 3.3) a. USAID Missions and Mission funded buy-ins and AAs? b. USAID/BFS? c. Other USAID-funded regional initiatives like ReSAKSS, AUC CAADP, Africa Lead? d. In-country ministries? e. In-country NGOs (international and national)? 5. The FSP IL has different organizational models for its programs. Which country models do you consider to be the most effective? Which model do you consider to be the least effective? (ERQ 3.4) a. Most effective: b. Least effective: 6. How could these FSP IL organizational models be improved? 7. In your opinion, how important is the issue of publication in achieving the FSP IL’s stated goal of influencing policy change? (ERQ Cluster 3; no specific ERQ) a. What types of publications for which audiences are encouraged? b. Does the FSP IL encourage publication in national, regional, and international journals? c. Is there a mechanism/strategy for publishing in the popular press in the countries where the project works? d. In the major journals/newspapers/news outlets that policy makers read in the U.S.? 4.0. Evaluation Question Cluster 4: Management and Implementation 1. Customers (ERQ 4.1): In your opinion, how should FSP IL define its customers; who are they (Missions, BFS, ministries); and to what extent is FSP IL meeting/not meeting customer needs and why? 2. Management and Coordination (ERQ 4.2): Please explain in your own words the principal mechanisms that you have used to coordinate between the consortium partners for: a. Strategic planning; b. Setting the research and policy engagement and policy implementation agendas in general and in specific countries; and c. Determining the level of effort of effort/full-time equivalents (FTEs) of each person working on the project. d. Is there a project steering committee for the consortium? If so, how often does it meet and what does it do? e. What are the strengths, weaknesses, opportunities, and threats (SWOTs) of the FSP IL’s principal coordination mechanisms? f. How can these coordination mechanisms be improved in future projects? 3. Funding Arrangements (ERQ 4.2): a. Please explain in your own words how the contractual structure between the consortium members works. For example, do the contractors get a lump sum, or is it reimbursed as they go? b. What are the SWOTs of this contractual mechanism and how can these be improved in future projects? 142 4. M&E Systems (ERQ 4.2): a. What were USAID’s M&E expectations/legal requirements for the Innovation Labs when the FSP IL started? Have these expectations changed in the last three years? b. How did the M&E and reporting systems of the FSP IL measure up to these expectations? c. What aspects of the FSP IL M&E systems have been most useful? (ERQ 4.2) d. Are there any lessons learned from this experience that could help inform future policy research projects and ILs? 5. Summary SWOT (ERQ 4.2): Could we please review what you have told me about the major strengths and weaknesses of the FSP IL management and implementation systems? Management/Implementation ERQ Cluster Strengths Weaknesses Opportunities Threats/ Risks Management and finance Coordination M&E systems 143 Annex VII.A.ii. KII Guide for Global FSP IL Research Leads and Co-Leads Identifier Response Interview number Date Name/s123 Sex and age Country/location Duration of interview Evaluation Question Cluster 1: Policy Analysis and Research 1. How were the activities in the regional and global component that you supervise chosen? (ERQ 1.1) a. What percentage of these activities was supply driven (based on skill sets, interests, and past experiences of the component team)? b. What percent was demand driven (i.e., requested by BFS, USAID/Missions, civil advocacy groups, in￾country governments, or international/national NGOs? 2. What needs to be done to make these policies more demand driven? (ERQ 1.1) 3. How did the design and conduct of these policy analyses influence policy review, policy formulation, or policy change? (ERQ 1.1) 4. What areas of policy analysis and research need more attention for better policy environment outcomes? (ERQ 1.2) 5. What could be could be done to improve coordination of research and policy analysis activities at the regional and global level that directly affect country-level policy engagement? (ERQ 1.2) Evaluation Question Cluster 2: Capacity Strengthening 1. Do the regional and global components of the FSP IL have a specific strategy for building the regional or country-level capacity to address these issues? (ERQ 2.1) 2. How have the specific regional and global component activities you support built institutional capacity at the country and regional level? What could be done to improve the use this research to build regional and country-level capacity? (ERQs 2.1-2.2) 3. What types of synergies has your global research program had with other consortium members? How could these synergies be improved? (ERQ 2.4) Evaluation Question Cluster 3: Progress toward Policy Outcomes 1. Have your regional and global component activities directly affected the policy environment, debate, or become actionable policies? Please give actual examples. (ERQ 3.1) 2. To date, how has your activity team extended its regional and global component research findings into the policy realm? Please give specific examples. (ERQ 3.2) 3. How does you research team interface with policy engagement at the country level? Please give specific examples. (ERQ 3.2) 4. What new strategies are being considered to improve or better extend your team’s global research findings 123 A few key KIIs will be with two people not just one. Names will be deleted from database, for use by evaluation team only for initial data entry. 144 into the policy debates and actionable policy at the regional and country level? (ERQ 3.3) 5. What methods and structures have worked most effectively for your global research team for the extension of its research findings into the policy realm at the country level? Which ones have not worked well? (ERQ 3.3) 6. What were the reasons for breaking component 4 into 4.a (upstream) and 4.b (downstream)? (ERQ 3.4) a. Should these sub-components be better integrated and coordinated? (ERQ 3.4) b. If so, how? (ERQ 3.4) 7. In your opinion, how important is the issue of publication in achieving the FSP IL’s stated goal of influencing policy change? (ERQ Cluster 3; not a specific question) a. What types of publications for which audiences are encouraged? b. Does the FSP IL encourage publication in national, regional, and international journals? c. Is there a mechanism/strategy for publishing: ● In the popular press in the countries where the project works? ● In the major journals/newspapers/news outlets that policy makers read in the U.S.? Evaluation Question Cluster 4: Management and Implementation 1. Customers (ERQ 4.1): a. In your opinion, how should the FSP IL global research program define its customers and who are they (Missions, BFS, ministries)? b. To what extent are the research programs meeting/not meeting customer needs and why? 2. Management and Coordination (ERQ 4.1): How do you coordinate the different activities in your component with regard to: a. Strategic planning; b. Setting the research and policy engagement and policy implementation agendas in general and in specific countries; c. Determining the level of effort of effort/full-time equivalents (FTEs) of each person working on the activity; and d. What are the SWOTs of these coordination mechanisms, and how can these be improved in future projects? 3. Funding Arrangements (ERQ 4.2): Please explain in your own words how the contractual structures that support your research. For example, does your team get a lump sum, or is it reimbursed as they go? 4. M&E Systems (ERQ 4.2): a. How are your activities evaluated and reported on by the FSP IL M&E system in: ● The FSP IL reports to USAID? ● The Mission-funded projects’ reports to USAID? b. Which indicators are currently being used to report on your activities? How could they be improved? 6. Summary SWOTs (ERQ 4.2): Could we please review what you have told me about the major strengths and weaknesses of the management and implementation systems for the global activities that you support? Management/Implementation ERQ Cluster Strengths Weaknesses Opportunities Threats/ Risks Management and finance Coordination M&E systems 145 Annex VII.B. Focus Group Discussion Checklists Annex VII.B.i. FGD Checklist for the Country/AA/Buy-in Staff Attending the All-FSP IL Workshop Identifier Response Interview number Date Country/location of interview Duration of interview # Name124 Age Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 125 Spell Out Code 126 1 2 3 4 5 6 7 Overview: This FGD will pilot test some of the most important ranking exercises being used in the online questionnaire by encouraging a free-flow discussion among staff about what the rankings mean. Groups will be pre-assigned with the sign-up sheet completed at the start of each group. This focus group will be organized in conjunction with the all￾FSP IL meeting in Washington, DC on March 30, 2017.127 Fourteen registered participants in the all-FSP IL workshop have been identified for as fitting in this category. The 14 participants represent eight Mission-funded AA and buy-in projects as well as the FSP IL’s support for the Great Lakes Initiative in East Africa will be divided into two separate focus groups that will be facilitated by one of the evaluation team members. The analysis will focus on the six FSP IL countries that the evaluation team has chosen to focus on—Myanmar, Tanzania, Nigeria, Malawi, Zambia, and Ghana. The six countries represent a reasoned choice from three broad categories of FSP IL countries: ● ● ● Category 1: Countries that have had the highest level of activity and policy influence (Myanmar and Tanzania); Category 2: Countries where the activities started later or where it is not yet possible to see extensive policy impact (Senegal, Mali, Nigeria, and Malawi); and Category 3: Countries where most of the FSP IL activities have been global research and/or where the country￾level research and capacity building has been more recent (Bangladesh, Ethiopia, Burundi, Mozambique, Rwanda, Zambia, Ghana, and South Africa). 124 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 125 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 126 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 127 Estimated to be 14 staff from eight Mission-funded AA and Buy-in projects (Malawi, Senegal, Myanmar, Mali, Nigeria, Tanzania, AGLC, and Zambia) and one staff person who supports the FSP IL C1/C2 support to Mozambique. 146 These countries were chosen to represent three broad categories of countries where the FSP IL intervenes. Each focus group will include one country from each of these three categories. Any discussion of another country will be listed as “other.” Instructions: Each focus group will be asked to appoint a co-facilitator to work with the team leader in pilot￾testing the questions and in recording the responses. Each group’s responses will be noted separately and projected on the wall with a PowerPoint project for the group as a tool for encouraging debate. Expected output: Staff feedback on the rankings should provide the evaluation team with qualitative information on why staff are assigning certain rankings and the global validity of these rankings in addressing the questions. Evaluation Question Cluster 1: Policy Analysis and Research Question 1: Coordination of FSP IL research activities at the country level is highly effective: (Please refer to the country where you work.)128 Country 1 Country 2 Country 3129 Other 4 Among FSP IL components, partners, and in-country government agencies __ __ __ __ Among FSP IL components, partners, and in-country civil organizations __ __ __ __ 128 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 129 Note: Since there will be two parallel FGDs of the AA and buy-in staff at the all-FSP IL workshop, each group will have staff who represent one Category 1 country, one Category 2 country, and one Category 3 country, as well as at least one other country (there are three others). In general, however, the analysis should focus on the three countries that are included in the online questionnaire, which is why only three are noted here. 147 Evaluation Question Cluster 2: Capacity Strengthening Question 2: Coordination of capacity strengthening activities at the country level is highly effective. (Please refer to the country where you work)130 Country 1 Country 2 Country 3 Other 4 Among FSP IL components, partners, and in-country government agencies __ __ __ __ Among FSP IL components, partners, and in-country civil organizations __ __ __ __ Question 3: What could be done to improve coordination of capacity strengthening activities in this country where you work? Question 4: FSP IL organization and implementation of capacity strengthening activities (i.e., policy research, analysis, strategy, communication, and advice) is highly effective at promoting policy change at the following levels: (Please refer to the country where you work)131 Country 1 Country 2 Country 3 Other 4 National government (i.e., ministers, parliamentarians, other staff) __ __ __ __ Organizations (i.e., farmer groups, NGOs, other civil society groups) __ __ __ __ Individuals __ __ __ __ Policy development process __ __ __ __ Question 5: In what ways could the capacity strengthening component be improved for greater policy impact (i.e., support to local associations, farmer groups, NGOs, journalists, government, and others)? 130 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 131Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 148 Evaluation Question Cluster 3: Progress toward Policy Outcomes Question 6: The different FSP IL components (e.g., C1, C2, C3, C4A, and C4B) have differently contributed toward policy change at the: 132 Country 1 Country 2 Country 3 Other 4 Country level __ __ __ __ Regional level __ __ __ __ Global level __ __ __ __ Question 7: The different FSP IL components (e.g., C1, C2, C3, C4A, and C4B) ARE LIKELY TO CONTRIBUTE toward policy change at the: 133 Country 1 Country 2 Country 3 Other 4 Country level __ __ __ __ Regional level __ __ __ __ Global level __ __ __ __ Question 8: How has FSP IL communicated policy analysis, recommendations, and policy options to policy makers and others in a position to influence policy? Question 9: In what ways could this communication be improved? Question 10: FSP IL works closely with regional partners and stakeholders to affect more rapid policy change:134 Country 1 Country 2 Country 3 Other 4 At the country level __ __ __ __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ __ __ __ At the global level __ __ __ __ Question 11: FSP IL works closely with national government partners to affect more rapid policy change:135 Country 1 Country 2 Country 3 Other 4 At the country level __ __ __ __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ __ __ __ At the global level __ __ __ __ 132 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 133 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 134 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 135 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 149 Question 12: FSP IL works closely with civil society and local stakeholders (farmer groups, NGOs, and other civil society groups) in this country:136 Country 1 Country 2 Country 3 Other 4 At the country level __ __ __ __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ __ __ __ At the global level __ __ __ __ Question 13: In the country where you work, FSP IL is meeting the needs of the following customers:137 Country 1 Country 2 Country 3 Other 4 USAID Missions and Mission-funded AAs and Buy￾ins __ __ __ __ BFS __ __ __ __ In-country ministries __ __ __ __ NGOs __ __ __ __ In-country universities __ __ __ __ Civil organizations __ __ __ __ Journalists __ __ __ __ 136 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 137 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 150 Annex VII.B.ii. FGD Checklist for the Global Research Staff Identifier Response Interview number Date Country/location of interview Duration of interview # Name138 Ag e Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 139 Spell Out Code 140 1 2 3 4 5 6 7 Overview: This FGD is designed to facilitate group discussions with the members of the FSP IL C3, C4A, and C4B global research teams. Some of these discussions will occur during the all-FSP IL meeting in Washington, DC; others will occur at MSU or online (through Skype). When a group is large as the Component 3 group is expected to be at the all-FSP IL meeting in March (eight people), it will be subdivided into two smaller focus groups (four each). Instructions: Each group is being asked to respond to the questions posed for the countries they cover and to compare and contrast their experiences. They will also be asked to brainstorm about an initial list of cross cutting lessons learned. The analysis will focus on the six FSP IL countries that the evaluation team has chosen—Myanmar, Tanzania, Nigeria, Malawi, Zambia, and Ghana. The six countries represent a reasoned choice from three broad categories of FSP IL countries: ● ● ● Category 1: Countries that have had the highest level of activity and policy influence (Myanmar and Tanzania); Category 2: Countries where the activities started later or where it is not yet possible to see extensive policy impact (Senegal, Mali, Nigeria, and Malawi); and Category 3: Countries where most of the FSP IL activities have been global research and/or where the country level research and capacity building has been more recent (Bangladesh, Ethiopia, Burundi, Mozambique, Rwanda, Zambia, Ghana, and South Africa). These countries were chosen to represent three broad categories of countries where the FSP IL intervenes. Each focus group will include one country from each of these three categories. Any discussion of another country will be listed as “other.” Expected output: Staff feedback on the questions, which mirror the questions asked in the KIIs of the regional research leads (i.e., KII 2.b), should provide the evaluation team with qualitative information on how a broader 138 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 139 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 140 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 151 group of regional staff feel about a sub-sample of the questions that will be asked in KII 2.b. The same feedback should help the evaluators to fine-tune some of the rankings/questions in the online questionnaire. Evaluation Question Cluster 1: Policy Analysis and Research Question 1: Regional and global FSP IL component (i.e., C3, C4A, C4B) research activities are coordinated with FSP IL activities at the country level (please refer to the countries where you work):141 Country 1 Country 2 Country 3 Country 4 Country 5 Country 6 Other Among FSP IL components, partners, and in-country government agencies __ __ __ __ __ __ __ Among FSP IL components, partners, and in-country civil organizations __ __ __ __ __ __ __ Question 2: What could be could be done to improve coordination of research and policy analysis activities at the regional and global level that would directly lead to country level policy engagement? (ERQ 1.2) 141 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 152 Evaluation Question Cluster 2: Capacity Strengthening Question 3: The activities of the regional and global FSP IL components contribute directly to capacity strengthening activities (i.e., policy research, analysis, strategy, communication, and advice) to improve and promote policy change at the following levels (please refer to the country where you work):142 Country 1 Country 2 Country 3 Country 4 Country 5 Country 6 Other National government (i.e., ministers, parliamentarians, other staff) __ __ __ __ __ __ __ Organizations (i.e., farmer groups, NGOs, other civil society groups) __ __ __ __ __ __ __ Individuals Policy development process __ __ __ __ __ __ __ Question 4: How have the specific regional and global component activities you support built institutional capacity at the country and regional level? What could be done to improve the use of this research to build regional and country-level capacity in these specific countries? In general? (ERQs 2.1-2.2) Question 5: What types of synergies have your global research activities had with other consortium members in these specific countries? How could these synergies be improved in these specific countries? How could they be improved across the board? (ERQ 2.4) 142 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 153 Evaluation Question Cluster 3: Progress Toward Policy Outcomes Question 6: Activities of the regional and global FSP IL components have contributed to the policy environment, debate, or become actionable policies at the country level:143 Country 1 Country 2 Country 3 Country 4 Country 5 Country 6 Other C3: Global collaborative research on support to the policy process and policy capacity __ __ __ __ __ __ __ C4A: Engagement on global policy debates on food security and upstream agri-food system transformation __ __ __ __ __ __ __ C4B: Engagement on global policy debates on food security and downstream agri￾food system transformation __ __ __ __ __ __ __ Question 7.a: How has FSP IL communicated global and regional policy analysis, recommendations, and policy options to policy makers and others in a position to influence policy? Question 7.b: How could this communication be improved? Question 8: Regional and global FSP IL component teams work closely with national government partners to affect more rapid policy change:144 Country 1 Country 2 Country 3 Country 4 Country 5 Country 6 Other At the country level __ __ __ __ __ __ __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ __ __ __ __ __ __ At the global level __ __ __ __ __ __ __ 143 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 144 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 154 Question 9: In the country/countries where you work, regional and global FSP IL component activities are meeting the needs of the following customers:145 Country 1 Country 2 Country 3 Country 4 Country 5 Country 6 Other USAID Missions and Mission funded AAs and Buy-ins __ __ __ __ __ __ __ BFS __ __ __ __ __ __ __ In-country ministries __ __ __ __ __ __ __ NGOs __ __ __ __ __ __ __ In-country universities __ __ __ __ __ __ __ Civil organizations __ __ __ __ __ __ __ Journalists __ __ __ __ __ __ __ 4.0. Evaluation Question Cluster: Management and Implementation Question 10: Summary SWOT Analysis (ERQ 4.2): Please develop a SWOT analysis for four major aspects of the regional and global research (i.e., C3, C4A, C4B) activities. In contrast to the other tables, this one is not focused on specific countries. Management/Implementation ERQ Cluster Strengths Weaknesses Opportunities Threats/ Risks Definition of who their customers are (Missions, BFS, ministries)? Management and finance Coordination M&E systems Question 11: Lessons Learned The FSP IL mid-term performance evaluation is expected to generate a series of lessons learned for future USAID/BFS programming for this type of multi-country, multi-partner (consortium). Please help us identify a series of global lessons learned for future global research initiatives. What do you feel are the most important lessons learned from your team’s experience with the FSP IL? 11.a. Lessons Learned: TBD ● ● ● ● ● ● Summary observations Recommendations for future programs 11.b. Lessons Learned: TBD Summary observations Recommendation for future programs 11.c. Lessons Learned: TBD Summary observations Recommendations for future programs 145 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 155 Annex VII.B.iii. FGD Checklist for Management Staff Identifier Response Interview number Date Country/location of interview Duration of interview # Name146 Ag e Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 147 Spell Out Code 148 1 2 3 4 5 6 7 Overview: Two broad categories of management staff have been identified for the FSP IL mid-term performance evaluation: 1) the senior management of each consortium partner; and 2) the management support staff for each consortium partner. The evaluation team will interview the senior manager, and in some cases the deputy manager (at the same time), for each consortium using the KII guide in Annex VII.A. In addition, the evaluators will conduct one FGD with the management support staff for each consortium member (MSU, IFPRI, UP) using this checklist. Some of the interviews will be scheduled in conjunction with the all-FSP IL workshop in Washington, DC; other FGDs are likely to be scheduled at MSU or through Skype. Instructions: The members of each focus group will be asked to express their opinion on the questions. At the end of the FGDs, the group will be asked to identify of a list of summary lessons learned and recommendations for future multi-country USAID/BFS Innovation Labs and/or projects using a consortium model. Evaluation Question Cluster 4: Management and Implementation Question 1: Management and Coordination (ERQ 4.1): How do you coordinate the different FSP IL activities that are supported by your institution with regard to: a. Strategic planning; b. Setting the research and policy engagement and policy implementation agendas in general and in specific countries; and c. Determining the level of effort/FTEs (full-time equivalents) of each person working on the activity. d. What are the SWOTs of these coordination mechanisms and how can these be improved in future projects? Question 2: Funding Arrangements (ERQ 4.2): 146 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 147 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 148 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 156 Please explain in your own words the contractual structures that support your institution’s activities under the FSP IL. For example, does your team get a lump sum, or is it reimbursed as they go? Is the contractual arrangement that supports your group similar to the others? If not, how so? Question 3: M&E Systems (ERQ 4.2): a. How are your institution’s activities evaluated and reported on by the FSP IL M&E system in: ● The FSP IL reports to USAID? ● The Mission-funded projects’ reports to USAID? b. Which indicators are currently being used to report on your activities? How could they be improved? Question 4: Summary SWOT Analysis (ERQ 4.2): Could we please review what you have told me about the major strengths and weaknesses of the management and implementation systems for the global activities that you support? Management/Implementation ERQ Cluster Strengths Weaknesses Opportunities Threats/Risks Management and finance Coordination M&E systems Question 5: Lessons Learned The FSP IL mid-term performance evaluation is expected to generate a series of lessons learned for future USAID/BFS programming for this type of multi-country, multi-partner program (consortium). Please help us identify a series of global lessons learned for (1) management and coordination; (2) funding arrangements; and (3) M&E systems. 5.a. Lessons Learned: Management and Coordination ● ● ● ● ● ● Summary observations Recommendations for future programs 5.b. Lessons Learned: Funding Arrangements Summary lessons learned Recommendation for future programs 5.c. Lessons Learned: M&E Systems Summary observations Recommendations for future programs 157 Annex VII.B.iv. FGD Checklist for Relevant USAID/BFS and IFPRI Staff About FSP IL Component 5 Activities Identifier Response Interview number Date Country/location of interview Duration of interview # Name149 Age Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 150 Spell Out Code 151 1 2 3 4 5 6 7 Context: The evaluation plan anticipates interviewing interview two broad categories of staff in a single focus group in Washington, DC: 1) USAID/BFS staff who have been involved the management, conceptualization, and evaluation of the FSP IL C5 activities; and 2) the IFPRI staff who have been the frontline managers of the C5 activities to date. Instructions: Each group is asked to provide insights into the questions. These individual insights will be noted. At the end of the focus group, the group will be asked to collaborate with the evaluation team leader facilitator/facilitators on the identification of a list of summary lessons learned and recommendations for future multi-country USAID/BFS-funded Innovation Labs and/or projects using a consortium model. Evaluation Question Cluster 1: Policy Analysis and Research Question 1: How are most requests for C5 support developed? Does the way a request is generated have effect the policy impact of the activities? Evaluation Question Cluster 2: Capacity Strengthening Question 2: Has the knowledge generated by the other components fed into C5 and, if so, how? Should it? Evaluation Question Cluster 3: Progress toward Policy Outcomes Question 3.a: Could you discuss one case in which the FSP IL C5 support had an important impact on policy? Question 3.b: Could you discuss one case in which the FSP IL C5 support had a less successful impact on policy? 149 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 150 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 151 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 158 Question 4: In what ways could the support provided by the FSP IL C5 activities be improved to better support policy influence or policy change? (ERQ 3.1) Evaluation Question Cluster 4: Management and Implementation Question 5: What was the original conceptualization of C5 in terms of budget, activity, and management? (ERQ 4.2) Question 6: How has this evolved over the FSP IL life cycle? (ERQ 4.2) 6.a. In Fiscal Year 1 (FY 2001): 6.b. In FY 2002: 6.c. In FY 2003: 6.d. In FY 2004-2005: Question 7: What is the current management and funding structure for C5? How is the funding handled? Question 8: Is it possible to provide the evaluation team with a disaggregated analysis of the budget spent on C5 activities? For example, what percentage of the C5 budget was spent on (ERQ 4.2): ● ● ● ● ● ● ● ● ● ● ● ● Training? Meetings? Research? Developing draft guidance? Improving other programs tracking and M&E systems? Other areas? Question 9: Is there a mechanism in place to track the short-term impact of these activities on USAID/BFS regional or country programs or any of the FSP IL global or country-level support activities? If not, should there be? If these activities are being tracked, are they useful to planning? Are there ways they could be strengthened? Question 10: How have these activities been reported in the annual report? Is this satisfactory? Cross Cutting Lessons Learned The FSP IL mid-term performance evaluation is expected to generate a series of lessons learned for future USAID/BFS programming for this type of multi-country, multi-partner consortium. Please help us identify a series of global lessons learned for future global research initiatives. What do you feel are the most important lessons learned from your team’s experience with the FSP IL? 10.a. Lessons Learned: TBD Summary observations Recommendations for future programs 10.b. Lessons Learned: TBD Summary observations Recommendation for future programs 10.c. Lessons Learned: TBD Summary observations Recommendations for future programs 159 Annex VII.B.v. FGD Checklist for Senior FSP IL Financial Officers in the Main FSP IL Administrative Office MSU and in Mission-Funded AA and Buy-in Project Offices in the Two Case-Study Countries Identifier Response Interview number Date Country/location of interview Duration of interview # Name152 Age Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 153 Spell Out Code 154 1 2 3 4 5 6 7 Context: The evaluation team will use this checklist to guide FGDs with the chief financial officer and his or her deputy or assistants for: 1) each consortium partner (MSU, IFPRI, UP); and 2) the Mission-funded projects in the two case-study countries of Myanmar and Tanzania. Instructions: Each group is asked to provide insights into the questions. These individual insights will be noted. At the end of the focus group, the group will be asked to collaborate with the evaluation team leader facilitator/facilitators on the identification of a list of summary lessons learned and recommendations for future multi-country USAID/BFS-funded Innovation Labs and/or projects using a consortium model. Evaluation Question Cluster 4: Management and Implementation Question 1: Please describe the financial structure that links the three consortium members. a. What works well in the current financial structure? b. What does not work well? c. How could the current financial structure be changed to better support FSP IL component activities? Question 2: In what ways does the FSP IL contribute to the financial support of the Mission-funded AAs? a. How is the FSP IL linked to the AAs in terms of money transfer and financial management? b. What works well in the current financial links between FSP IL and the Mission-funded AA projects? c. What does not work well? Question 3: In what ways does the FSP IL contribute to the financial support of the Mission-funded buy-ins? a. How is the FSP IL linked to the buy-ins in terms of money transfer and financial management? b. What works well in the current financial links between FSP IL and the Mission-funded buy-in projects? 152 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 153 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 154 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 160 c. What does not work well? Question 4: How could the current financial structure be changed to better support FSP IL components and activities? Question 5: What, if any, cross cutting lessons learned and recommendations can be deduced from the FSP IL’s experience for future programs using a similar implementation structure? a. For the consortium providing technical support to the AAs and buy-ins? b. For the Mission-funded AAs? c. For the Mission-funded buy-ins? Annex VII.B.vi. FGD Checklist for In-Country C1/C2/AA/Buy-in Support Staff in the Two Case￾Study Countries Identifier Response Interview number Date Country/location of interview Duration of interview # Name155 Age Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 156 Spell Out Code 157 1 2 3 4 5 6 7 Overview: All of the in-country C1/C2/AA/buy-in support staff in each of the two case-study countries will be asked to participate in this FGD. This exercise is designed to encourage a free-flow discussion among in-country staff in answering the questions. Instructions: Participants are asked to respond to the questions posed for the case-study country in which they work, and to compare and contrast their experiences. The participants will be asked to appoint a team leader from among the participants to lead the participants in answering and in recording the responses. They will also be asked to brainstorm about an initial list of cross cutting lessons learned. Expected Output: In-country staff feedback on the questions, which mirror the questions asked in the KIIs of two-case-study country directors and co-directors (i.e., KII 2.a), should provide the evaluation team with 155 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 156 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 157 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 161 qualitative information on how a broader group of in-country staff feels about a sub-sample of the questions that will be asked in KII 2.a as well as the global validity in addressing the questions. Evaluation Question Cluster 1: Policy Analysis and Research Question 1. Coordination of FSP IL research activities in this country is highly effective: Code 158 Among FSP IL components (i.e., C1/C2, C3, C4A, C4B) __ Among FSP IL partners (i.e., UP, IFPRI, MSU) __ Among FSP IL components and Mission-funded buy-ins __ Among FSP IL components and Mission-funded AAs __ Among FSP IL components and in-country government agencies __ Among FSP IL components and in-country civil organizations ___ Evaluation Question Cluster 2: Capacity Strengthening Question 2: Coordination of capacity strengthening activities in this country is highly effective: Code 159 Among FSP IL components (i.e., C1/C2, C3, C4A, C4B) __ Among FSP IL partners (i.e., UP, IFPRI, MSU) __ Among FSP IL components and Mission-funded buy-ins __ Among FSP IL components and Mission-funded AAs __ Among FSP IL components and in-country government agencies __ Among FSP IL components and in-country civil organizations __ Question 3: What could be done to improve coordination of capacity strengthening activities in the country where you work? Question 4: FSP IL organization and implementation of capacity strengthening activities (i.e., policy research, analysis, strategy, communication, and advice) is highly effective at promoting policy change at the following levels: Code 160 National government (i.e., ministers, parliamentarians, other staff) __ Organizations (i.e., farmer groups, NGOs, other civil society groups) __ Individuals __ Policy development process __ 158 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 159 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 160 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 162 Question 5: In what ways could the capacity strengthening component be improved for greater policy impact (i.e., support to local associations, farmer groups, NGOs, journalists, government and others)? Evaluation Question Cluster 3: Progress Toward Policy Outcomes Question 6: The different FSP IL components (e.g., C1, C2, C3, C4A, and C4B) have differently contributed toward policy change at the: Code161 Country level __ Regional level __ Global level __ Question 7: The different FSP IL components (e.g., C1, C2, C3, C4A, and C4B) ARE LIKELY TO CONTRIBUTE toward policy change at the: Code162 Country level __ Regional level __ Global level __ Question 8: How has FSP IL communicated policy analysis, recommendations, and policy options to policy makers and others in a position to influence policy? In what ways could this communication be improved? Question 9: FSP IL works closely with regional partners and stakeholders to affect more rapid policy change: Code163 At the country level __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ At the global level __ Question 10: FSP IL works closely with national government partners to affect more rapid policy change: Code164 At the country level __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ At the global level __ 161 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 162 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 163 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 164 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 163 Question 11: FSP IL works closely with civil society and local stakeholders (farmer groups, NGOs, and other civil society groups) in this country: Code165 At the country level __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ At the global level __ Question 12: In this country, FSP IL is meeting the needs of the following customers: Code166 USAID Missions and Mission-funded AAs and Buy-ins __ BFS __ In-country ministries __ NGOs __ In-country universities __ Civil organizations __ Journalists __ 165 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 166 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 164 Annex VII.B.vii. FGD Checklist for In-Country Regional and Global Research Staff in the Two Case￾Study Countries Identifier Response Interview number Date Country/location of interview Duration of interview # Name167 Age Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 168 Spell Out Code 169 In-Country Staff 1 2 3 4 5 6 7 Overview: All of the in-country regional and global research staff in each of the two case-study countries will be asked to participate in this FGD. This exercise is designed to encourage a free-flow discussion among in-country staff in answering the questions. Instructions: Participants are asked to respond to the questions posed for the case-study country in which they work, and to compare and contrast their experiences. The participants will be asked to appoint a team leader from among the participants to lead the participants in answering and in recording the responses. They will also be asked to brainstorm about an initial list of cross cutting lessons learned. Expected Output: In-country staff feedback on the questions, which mirror some of the questions asked in the KIIs of the regional and global research component leaders (i.e., KII 3.b), should provide the evaluation team with 167 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 168 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 169 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 165 qualitative information on how a broader group of in-country staff feel about a sub-sample of the questions that will be asked in KII 3.b, as well as the global validity in addressing the questions. Evaluation Question Cluster 1: Policy Analysis and Research Question 1: Regional and global FSP IL component (i.e., C3, C4A, and C4B) research activities are coordinated with FSP IL activities at the country level: Code 170 Among FSP IL components and in-country government agencies (1) __ Among FSP IL components and in-country civil organizations (2) __ Question 2: What could be could be done to improve coordination of research and policy analysis activities at the regional and global level that would directly lead to country-level policy engagement? (ERQ 1.2) Evaluation Question Cluster 2: Capacity Strengthening Question 3: The activities of the regional and global FSP IL components contribute directly to capacity strengthening activities (i.e., policy research, analysis, strategy, communication, and advice) to improve and promote policy change at the following levels: Code171 National government (i.e., ministers, parliamentarians, other staff) __ Organizations (i.e., farmer groups, NGOs, other civil society groups) __ Individuals __ Policy development process __ Question 4: How have the specific regional and global component activities you support built institutional capacity at the country and regional level? What could be done to improve the use of this research to build regional and country-level capacity in this country? In general? (ERQ 2.1-2.2) Question 5: What types of synergies have your global research activities had with other consortium members in in this country? How could these synergies be improved in this country? In general? (ERQ 2.4) Question 6: FSP IL organization and implementation of capacity strengthening activities (i.e., policy research, analysis, strategy, communication and advice) is highly effective at promoting policy change at the following levels: Code172 National government (i.e., ministers, parliamentarians, other staff) __ Organizations (i.e., farmer groups, NGOs, other civil society groups) __ Individuals __ Policy development process __ 170 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 171 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 172 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 166 Question 7: In what ways could the capacity strengthening component be improved for greater policy impact (i.e., support to local associations, farmer groups, NGOs, journalists, government, and others)? Evaluation Question Cluster 3: Progress toward Policy Outcomes Question 8: Activities of the regional and global FSP IL components have contributed to the policy environment, debate, or become actionable policies in this country: Code173 C3: Global collaborative research on support to the policy process and policy capacity __ C4A: Engagement on global policy debates on food security and upstream agri￾food system transformation __ C4B: Engagement on global policy debates on food security and downstream agri-food system transformation __ 173 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 167 Question 9.a: How has FSP IL communicated global and regional policy analysis, recommendations, and policy options to policy makers and others in a position to influence policy? Question 9.b: How could this communication be improved? Question 10: Regional and global FSP IL component teams work closely with national government partners to affect more rapid policy change at the: Code174 Country level __ Regional level __ Global level __ Question 11: In the country/countries where you work, regional and global FSP IL component activities are meeting the needs of the following customers: Code175 USAID Missions and Mission-funded AAs and Buy-ins __ BFS __ In-country ministries __ NGOs __ In-country universities __ Civil organizations __ Journalists __ 174 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 175 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 168 Annex VII.B.viii. FGD Checklist for the In-Country Stake Holders in the Two Case-Study Countries Identifier Response Interview number Date Country/location of interview Duration of interview # Name176 Age Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 177 Spell Out Code 178 Stakeholder Participant 1 2 3 4 5 6 7 Overview: The FSP IL works with a wide variety of national, regional, and local stakeholder groups in each of the countries where it works. In each case-study country, the evaluation team will organize FGDs with some of the most important national-level stakeholder groups. At least one set of discussions will be with stakeholder groups that have developed sub-contracts with the FSP IL. A second set will bring together a sample of national-level staff from some of the other stakeholder institutions that have worked closely on joint activities. Instructions: Participants are asked to respond to the questions posed for the case-study country in which they work and to compare and contrast their experiences. Evaluation Question Cluster 1: Policy Analysis and Research Question 1: Did you have a role in design OR conduct of any of the FSP IL or Mission-supported research topics or policy analyses? If so, in what capacity? Question 2: What areas of policy analysis and research need more attention for better policy environment outcomes? (ERQ 1.2) 176 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 177 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 178 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 169 Evaluation Question Cluster 2: Capacity Strengthening Question 3: Coordination of capacity strengthening activities at the country level is highly effective (please refer to the country where you work): Code 179 Among FSP IL activities and in-country government agencies __ Among FSP IL activities and in-country civil organizations __ Question 4: What could be done to improve coordination of capacity strengthening activities in this country? Question 5: The activities of the regional and global FSP IL components contribute directly to capacity strengthening activities (i.e., policy research, analysis, strategy, communication, and advice) to improve and promote policy change at the following levels: Code 180 National government (i.e., ministers, parliamentarians, other staff) __ Organizations (i.e., farmer groups, NGOs, other civil society groups) __ Individuals __ Policy development process __ Question 6: In what ways could the capacity strengthening component be improved for greater policy impact (i.e., support to local associations, farmer groups, NGOs, journalists, government, and others)? 179 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 180 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 170 Question 7: FSP IL organization and implementation of capacity strengthening activities (i.e., policy research, analysis, strategy, communication, and advice) is highly effective at promoting policy change at the following levels: Code 181 National government (i.e., ministers, parliamentarians, other staff) __ Organizations (i.e., farmer groups, NGOs, other civil society groups) __ Individuals __ Policy development process __ Evaluation Question Cluster 3: Progress toward Policy Outcomes Question 8: How has FSP IL communicated policy analysis, recommendations, and policy options to policy makers and others in a position to influence policy? In what way could this communication be improved? Question 9: FSP IL works closely with regional partners and stakeholders to affect more rapid policy change: Code182 At the country level __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ At the global level __ Question 10: FSP IL works closely with national government partners to affect more rapid policy change: Code183 At the country level __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ At the global level __ Question 11: FSP IL works closely with civil society and local stakeholders (farmer groups, NGOs, and other civil society groups) in this country: Code184 At the country level __ At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) __ At the global level __ 181 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 182 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 183 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 184 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 171 Question 12: In this country, FSP IL is meeting the needs of the following customers: Code185 In-country ministries __ NGOs __ In-country universities __ Civil organizations __ Journalists __ 185 Strongly disagree (1); disagree (2); somewhat disagree (3); neither agree nor disagree (4); somewhat agree (5); agree (6); strongly agree (7). 172 Annex VII.B.ix. District-Level Intensive Case Studies in Myanmar and Tanzania Identifier Response Interview number Date Country/location of interview Duration of interview # Name186 Age Sex Country Program Represented Position Agency Affiliation (MSU, IFPRI, UP) Group Contact Information (Optional) Spell Out Code 187 Spell Out Code 188 1 2 3 4 5 6 7 Context: In each district, the evaluation team will work through one of the experienced district-level extension agents that the FSP IL has worked with in the past to identify four groups of individuals to interview about the district-level impacts of the FSP IL support to the current Mission-funded AA and Buy-in projects. ● Direct beneficiaries: The local beneficiaries who participated in FSP IL-facilitated activities (e.g., new technology demonstrations, short-term trainings, and/or received or purchased inputs or services that were funded or subsidized by the FSP IL or the Mission-funded project to support these new initiatives). In most cases the direct beneficiaries are members of a farmer group or community-based organization that worked with the project. It is assumed that many of the direct beneficiaries have adopted the new technologies that were advocated by the FSP IL and/or supported by policy reforms that the FSP IL helped create. ● Indirect beneficiaries: Local individuals who have benefitted from some of the policy reforms or new technologies that were pilot tested by the FSP IL activities who have not had direct contact with the FSP IL activities such as trainings or service provisions. ● Local Leaders: Individuals who occupy important civil, elected, and government positions in the community. Not all of these individuals will have participated in the FSP IL-supported activities, but they are likely to be informed about the global impact of the activities on their constituency. Instructions: The evaluation team will work through the local extension expert and community-level government structures to identify a representative sample of individuals and, whenever possible, groups for the focus groups. The team will endeavor to ensure that at least half the people interviewed are women beneficiaries and leaders. Unless instructed otherwise, the team plans to organize separate FGDs with women and with men. All district￾level interviews will be facilitated by a translator who will be recruited from an NGO or government office outside the project. 186 Will be deleted in the notes recorded and entered into the database. To speed the process of roll taking, the names and country programs will be completed then distributed to the participants for review/revision. 187 Codes: Ghana (1); Malawi (2); Myanmar (3); Nigeria (4); Tanzania (5); Zambia (6); Mali (7); Mozambique (8); Senegal (9); Other (10); Other (11); Other (12). 188 Care will be taken to ensure a mixture of staff from different countries. Based on feedback from the FSP IL, the evaluation team will pre￾select the group facilitators who will lead the report-out. 173 Expected Output: The district-level case studies are expected to provide qualitative information on the local￾level impacts of the FSP IL on direct and indirect beneficiaries in areas of the country where the FSP IL has been active. 1. Introduction of the evaluation team and purpose of the discussion. 2. Introduction of the participants. a. What is the purpose of the group? How many members are there? b. Where do the members come from? Are there links with other groups? c. Are there links with the district-level government offices? 3. Linkages between the FSP IL activities or the Mission-funded project (past, present, and future). a. What linkages exist between the FSP IL and Mission-funded AA or buy-in project and the group? b. If linkages exist, what are they and how did they affect the members? c. If there are no formal linkages, are the members of the group aware of some of the FSP IL’s activities that it executed through its local partners? 4. Use follow-up questions in relation to the FSP IL activities to determine how many members of the groups are using the new policy reforms/technologies/activities and what impacts they have had on their day-to-day lives and living standards. a. What was the purpose of the policy reforms/technologies/activities? Why were they being promoted? (The purpose of the question is to determine the level of knowledge/understanding of the beneficiaries). b. How were the policy reforms/technologies/activities promoted? (Include group activities as well as radio ads, extension programs, posters, etc.) c. How did individual group members acquire the policy reforms/technologies/activities? (Were they acquired free or through purchase?) d. Were women faced with any special constraints when attempting to adopt the new policy reforms/technologies/activities? If so, what were these and how were they overcome? e. Show of hands for how many people in the group are currently using the new policy reforms/technologies/activities and plan to use them in the future? f. Show of hands for how many people in the group tried the new policy reforms/technologies/activities but decided not to use them again? g. Has the group or individual members tried to promote the policy reforms/technologies/activities to individuals outside the group? If so, how did this work? 5. What was the global impact of the new policy reforms/technologies/activities on: a. The direct beneficiaries? b. The indirect beneficiaries? c. Women as individuals and women’s groups? d. The average household living standards? e. The most vulnerable low-income households in the community, including handicapped individuals? f. Local government revenues? 6. Were there any negative impacts associated with these policy reforms/technologies/activities? If so, what were they? 7. Is it likely that the positive impacts will be able to be sustained without any direct support/assistance from the FSP IL or from the Mission-funded AA or Buy-in project? 8. End of questions: Does anyone else have any questions or additional points they would like to raise? Closing of the discussion: Thank everyone for participation and encourage them in their future activities. 174 ANNEX VIII 175 FSP LAB Mid-Term Evaluation Online Survey Questions Q1-Q7: Cat 6 and 7. Q1 Please describe the financial structure that links the three consortium members. 2) Description: (1) ____________________ 3) Don’t know or N/A (2) Q2 What are the effective elements in the current financial structure? Please provide bullet points. 4) My list: (1) ____________________ 5) Don’t know or N/A (2) Q3 What are the ineffective elements in the current financial structure? Please provide bullet points. 6) My list: (1) ____________________ 7) Don’t know or N/A (2) Q4 How could the current financial structure be changed to better support FSP LAB activities? Please provide bullet points. 8) My list (1) ____________________ 9) No opinion (2) Q5 Are you involved in any financial aspects of an AA (Associate Award)? 10) Yes (1) 11) No (2) Condition: No Is Selected. Skip To: Are you involved in any financial asp.... Display This Question: Are you involved in any financial aspects of an AA (Associate Award)? If Yes Is Selected Q6 In what ways does the FSP LAB contribute to the financial support of the Mission-funded AAs? Please provide bullet points. 12) My list: (1) ____________________ 13) Don’t know or N/A (2) Display This Question: Are you involved in any financial aspects of an AA (Associate Award)? If Yes Is Selected Q7 How is the FSP LAB linked to the AAs in terms of money transfer and financial management? Please provide bullet points. 14) My list: (1) ____________________ 15) Don’t know or N/A (2) 176 Display This Question: Are you involved in any financial aspects of an AA (Associate Award)? If Yes Is Selected Q8 What are the effective elements in the current financial links between FSP LAB and the Mission-funded AA projects? Please provide bullet points. 16) My list: (1) ____________________ 17) Don’t know or N/A (2) Display This Question: Are you involved in any financial aspects of an AA (Associate Award)? If Yes Is Selected Q9 What are the ineffective elements in the current financial links between FSP LAB and the Mission-funded AA projects? Please provide bullet points. 18) My list: (1) ____________________ 19) Don’t know or N/A (2) Q10 Are you involved in any financial aspects of a Mission-funded Buy-in? 20) Yes (1) 21) No (2) Condition: No Is Selected. Skip To: What if any cross-cutting lessons lea.... Display This Question: Are you involved in any financial aspects of a Mission-funded Buy-in? If Yes Is Selected Q11 In what ways does the FSP LAB contribute to the financial support of the Mission-funded Buy-in? Please provide bullet points. 22) My list (1) ____________________ 23) Don’t know or N/A (2) Display This Question: Are you involved in any financial aspects of a Mission-funded Buy-in? If Yes Is Selected Q12 How is the FSP LAB linked to the Buy-in in terms of money transfer and financial management? Please provide bullet points. 24) My list (1) ____________________ 25) Don’t know or N/A (2) Display This Question: Are you involved in any financial aspects of a Mission-funded Buy-in? If Yes Is Selected Q13 What are the effective elements in the current financial links between FSP LAB and the Mission-funded Buy-in projects? Please provide bullet points. 26) My list (1) ____________________ 27) Don’t know or N/A (2) 177 Display This Question: Are you involved in any financial aspects of a Mission-funded Buy-in? If Yes Is Selected Q14 What are the ineffective elements in the current financial links between FSP LAB and the Mission-funded Buy￾in projects? Please provide bullet points. 28) My list: (1) ____________________ 29) Don’t know or N/A (2) Q15 What if any cross-cutting lessons learned and recommendations can be deduced from the FSP LAB’s experience for future programs concerning finance and funding among the consortium members? Please provide bullet points. 30) My list (1) ____________________ 31) Don’t know or N/A (2) Display This Question: Are you involved in any financial aspects of an AA (Associate Award)? If Yes Is Selected Q16 What if any cross-cutting lessons learned and recommendations can be deduced from the FSP LAB’s experience for future programs concerning finance and funding for the Mission-funded AA’s? Please provide bullet points. 32) My list (1) ____________________ 33) Don’t know or N/A (2) Display This Question: Are you involved in any financial aspects of a Mission-funded Buy-in? If Yes Is Selected Q17 What if any cross-cutting lessons learned and recommendations can be deduced from the FSP LAB’s experience for future programs concerning finance and funding for the Mission-funded Buy-ins? Please provide bullet points. 34) My list (1) ____________________ 35) Don’t know or N/A (2) 178 _____________________________________________________________________________ Questions Q18 – Q25: Cat 1, 2, and 6. Q18 Were you involved in selecting FSP LAB research topics or policy analyses? ❑ Yes (Please indicate how) (1) ____________________ ❑ No (2) ❑ Don’t know or N/A (3) Q19 FSP LAB research topics were selected by: 36) MSU, IFPRI, and U Pretoria jointly (1) 37) MSU (2) 38) IFPRI (3) 39) U Pretoria (4) 40) USAID Mission (5) 41) Other (6) 42) Don’t know or N/A (7) Q20 Did you have a role in design OR conduct of any of the FSP LAB research topics or policy analyses? 43) Yes (Please indicate your role) (1) ____________________ 44) No (3) 45) Don’t know or N/A (4) Q21 How did FSP LAB make decisions on design of policy analysis and research? Please provide maximum of three bullet points. 46) My list: (1) ____________________ 47) Don’t know or N/A (2) Q22 Did the design and conduct of the FSP LAB support policy analysis and research influence policy review? 48) Yes (Please indicate how) (1) ____________________ 49) No (Please indicate why) (2) ____________________ 50) Don’t know or N/A (3) Q23 Did the design and conduct of the FSP LAB support policy analysis and research influence policy formulation? 51) Yes (Please indicate how) (1) ____________________ 52) No (Please indicate why) (2) ____________________ 53) Don’t know or N/A (3) 179 Q24 Did the design and conduct of the FSP LAB support policy analysis and research influence policy change? 54) Yes (Please indicate how) (1) ____________________ 55) No (Please indicate why) (2) ____________________ 56) Don’t know or N/A (3) Q25 What areas of policy analysis and research need more attention for better policy environment outcomes? Please provide maximum of three bullet points. 57) My list: (1) ____________________ 58) Don’t know or N/A (2) 180 Questions Q26 – Q28: Cat 1, 2, 5, and 6 Q26 Coordination of research activities at the country level is highly effective: (Please refer to the country where you work.) Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) Among FSP LAB activities (1) 59) 60) 61) 62) 63) 64) 65) Among FSP LAB partners (i.e., UP, IFPRI, MSU) (2) 66) 67) 68) 69) 70) 71) 72) Among FSP LAB activities, and USAID Missions funded Buy￾ins (3) 73) 74) 75) 76) 77) 78) 79) Among FSP LAB activities and USAID Mission￾funded AAs (4) 80) 81) 82) 83) 84) 85) 86) Among FSP LAB activities and in-country government agencies (5) 87) 88) 89) 90) 91) 92) 93) Among FSP LAB activities and in-country civil organization s (6) 94) 95) 96) 97) 98) 99) 100) Q27 What could be done to improve coordination of research activities in this country where you work? Please provide maximum of three bullet points. 101) My list: (1) ____________________ 102) Don’t know or N/A (2) 181 Q28 Coordination of capacity strengthening activities at the country level is highly effective: (Please refer to the country where you work.) Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) Among FSP LAB activities (1) 103) 104) 105) 106) 107) 108) 109) Among FSP LAB partners (i.e., UP, IFPRI, MSU) (2) 110) 111) 112) 113) 114) 115) 116) Among FSP LAB activities, and USAID Mission￾funded Buy￾ins (3) 117) 118) 119) 120) 121) 122) 123) Among FSP LAB activities and USAID Mission￾funded AAs (4) 124) 125) 126) 127) 128) 129) 130) Among FSP LAB activities and in-country government agencies (5) 131) 132) 133) 134) 135) 136) 137) Among FSP LAB activities and in-country civil organization s (6) 138) 139) 140) 141) 142) 143) 144) 182 Questions Q29 – Q30: Cat 3, 4, and 5. Q29 Coordination of capacity strengthening activities at the country level is highly effective: (Please refer to the country where you work.) Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) Between FSP LAB activities and in-Country government agencies (1) 145) 146) 147) 148) 149) 150) 151) Between FSP LAB activities and in-Country civil organization s (2) 152) 153) 154) 155) 156) 157) 158) Q30 What could be done to improve coordination of capacity strengthening activities in this country where you work? Please provide maximum of three bullet points. 159) My list: (1) ____________________ 160) Don’t know or N/A (2) 183 Question Q30: Cat 1, 2, 3, 4, 5, and 6. Q31 FSP LAB organization and implementation of capacity strengthening activities (i.e., policy research, analysis, strategy, communication and advice) is highly effective at promoting policy change at the following levels: (Please refer to the country where you work.) Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) National government (i.e., ministers, parliamentarians, other staff) (1) 161) 162) 163) 164) 165) 166) 167) Organizations (i.e., farmer groups, NGOs, other civil society groups) (2) 168) 169) 170) 171) 172) 173) 174) Individuals (3) 175) 176) 177) 178) 179) 180) 181) Policy development process (4) 182) 183) 184) 185) 186) 187) 188) Questions Q32 – Q33: Cat 1, 2, 6. Q32 In what ways could the capacity strengthening component be improved for greater policy impact (i.e., support to local associations, farmer groups, NGOs, journalists, government, and others)? Please provide maximum of three bullet points. 189) My list: (1) ____________________ 190) Don’t know or N/A (2) Q33 What capacity delivery methods have been more or less effective for delivering the best capacity strengthening activities in this country? Please provide maximum of three bullet points. 191) My list: (1) ____________________ 192) Don’t know or N/A (2) Questions Q34 – Q35: Cat 1 and 6. Q34 What types of synergies have there been among the various MSU, IFPRI, and University of Pretoria capacity strengthening approaches in this country? Please provide maximum of three bullet points. 193) My list: (1) ____________________ 194) Don’t know or N/A (2) 184 Q35 In what ways can each partner's capacity strengthening approaches be improved in this country? Please provide maximum of three bullet points each institution. ❑ MSU (1) ____________________ ❑ IFPRI (2) ____________________ ❑ U of Pretoria (3) ____________________ ❑ Don’t know or N/A (4) 185 186 187 Q41 In what ways could this communication be improved? Please provide maximum of three bullet points. 195) My list: (1) ____________________ 196) Don’t know or N/A (2) Questions Q42 – Q44 and Q46: Cat 1, 2, 3, 4, 5, and 6. Q42 FSP LAB works closely with regional partners and stakeholders to affect more rapid policy change: Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) At the country level (1) 197) 198) 199) 200) 201) 202) 203) At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) (2) 204) 205) 206) 207) 208) 209) 210) At the global level (3) 211) 212) 213) 214) 215) 216) 217) Q43 FSP LAB works closely with national government partners to affect more rapid policy change: Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) At the country level (1) 218) 219) 220) 221) 222) 223) 224) At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) (2) 225) 226) 227) 228) 229) 230) 231) At the global level (3) 232) 233) 234) 235) 236) 237) 238) 188 Q44 FSP LAB works closely with civil society and local stakeholders (framer groups, NGOs, and other civil society groups) in this country: Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) At the country level (1) 239) 240) 241) 242) 243) 244) 245) At the regional level (e.g., West Africa, East Africa, Southern Africa, Asia) (2) 246) 247) 248) 249) 250) 251) 252) At the global level (3) 253) 254) 255) 256) 257) 258) 259) Q46 Please rank from 1 (least) to 3 (best) the following teams in terms of effectiveness for policy analysis and dialogue: ______ Expatriate team (1) ______ Local team (2) ______ Mixed team (3) 189 Questions Q45 – Q41: Cat 1, 2, 5, and 6. Q45 In the country where you work, FSP LAB is meeting the needs of the following customers: Strongly disagree (1) Disagree (2) Somewhat disagree (3) No opinion/ don't know (4) Somewhat agree (5) Agree (6) Strongly agree (7) USAID Missions and Mission￾funded AAs and Buy-ins (1) 260) 261) 262) 263) 264) 265) 266) BFS (Bureau for Food Security) (2) 267) 268) 269) 270) 271) 272) 273) In-country ministries (3) 274) 275) 276) 277) 278) 279) 280) NGOs (4) 281) 282) 283) 284) 285) 286) 287) In-country universities (5) 288) 289) 290) 291) 292) 293) 294) Civil organization s (6) 295) 296) 297) 298) 299) 300) 301) Journalists (7) 302) 303) 304) 305) 306) 307) 308) Questions Q47 – Q48: Cat 1, 2, and 6. Q47 Please describe the FSP LAB organizational model in the country in which you work or support. Q48 How can these FSP LAB organizational models be improved? Please provide maximum of three bullet points. 309) My list: (1) ____________________ 310) Don’t know or N/A (2) Questions Q49 – Q63: Cat 6. Q49 How do you coordinate the different FSP LAB activities that are supported by your institution with regard to strategic planning? Please provide bullet points. Q50 How do you coordinate the different FSP LAB activities that are supported by your institution with regard to setting the research and policy engagement and policy implementation agendas in general and in specific countries? Please provide bullet points. 190 Q51 How do you coordinate the different FSP LAB activities that are supported by your institution with regard to determining the level of effort/FTEs (full-time equivalents) of each person working on the activity? Please provide bullet points. Q52 Please explain (in your own words) the contractual structures that support your institution’s activities under the FSP LAB. Please provide bullet points. Q53 Does the team that you manage get a lump sum, or is it “reimburse as they go”? 311) Lump sum (1) 312) Reimburse as they go (2) 313) Don’t know or N/A (3) Q54 How could the contractual structures that support the team you manage be improved to better support FSP LAB activities and policy engagement? Please provide bullet points. Q55 How are your institution’s activities evaluated and reported on by the FSP LAB M&E system in the FSP LAB reports to USAID? Please provide bullet points. Q56 How are your institution’s activities evaluated and reported on by the FSP LAB M&E system in the Mission￾funded projects’ reports to USAID? Please provide bullet points. Q57 Which indicators are effectively being used to report on your activities and how could they be improved? Q58 Please help us identify a series of global lessons learned for management and coordination that would be useful in planning for future USAID/BFS (Bureau for Food Security) programming for this type of multi-country, multi-partner (consortium) program. Please provide bullet points. Q59 Please help us identify a series of global lessons learned for finance and funding arrangements that would be useful in planning for future USAID/BFS (Bureau for Food Security) programming for this type of multi-country, multi-partner (consortium) program. Please provide bullet points. Q60 Please help us identify a series of global lessons learned for M&E Systems that would be useful in planning for future USAID/BFS (Bureau for Food Security) programming for this type of multi-country, multi-partner (consortium) program. Please provide bullet points. 191 Q61 Please list a maximum of three recommendations for future USAID/BFS (Bureau for Food Security) programming for this type of multi-country, multi-partner (consortium) program in terms of management and coordination. Q62 Please list a maximum of three recommendations for future USAID/BFS (Bureau for Food Security) programming for this type of multi-country, multi-partner (consortium) program in terms of finance and funding arrangements. Q63 Please list a maximum of three recommendations for future USAID/BFS (Bureau for Food Security) programming for this type of multi-country, multi-partner (consortium) program in terms of M&E Systems.