NOVEMBER 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by a Technical Team under the USAID Monitoring, Evaluation & Learning Activity. EVALUATION NIGERIA AGRICULTURAL POLICY PROJECT FINAL PERFORMANCE EVALUATION FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT Contract No: AID-OAA-I-15-00018 FINAL PERFORMANCE EVALUATION Submitted to: Dr. Chidimma Anyanwu, Contracting Officer Representative (COR), MEL Activity, USAID/Nigeria Copied to: Dr. Samba Kawa, Agreement Officer Representative (AOR), NAPP, USAID Office of Economic Growth and Environment (EGE) Mr. Oladele Kolade, M&E Specialist and Mission Environment Officer (MEO), USAID Office of Economic Growth and Environment (EGE) Prepared by: Professor Aderibigbe Olomola, Team Leader/Agriculture Policy Specialist Professor Olajide Ajao, Agricultural Economist Dr. Muhammad Bello, Agricultural Economist Ms. Toni Akinluyi, Sr. Evaluation Specialist Mr. Olufemi Gisanrin, Sr. MEL Specialist Submitted by: Paul DeLucco Chief of Party, MEL Activity DevTech Systems, Inc. Email: PDelucco@devtechsys.com Cover Photo: Nigeria Agricultural Policy Project Activity Scholars Panel, August 14–16, 2018, Transcorp Hilton, Abuja. Photo by Oyinkansola Tasie. Disclaimer: The views expressed in this report are those of the Evaluation Team and do not necessarily reflect the views of USAID, the Government of Nigeria, or any other organization or person associated with this project. i TABLE OF CONTENTS TABLE OF CONTENTS................................................................................................................................................... i LIST OF TABLES............................................................................................................................................................... iii ACRONYMS...................................................................................................................................................................... iv EXECUTIVE SUMMARY..................................................................................................................................................1 Background..........................................................................................................................................................................................1 Methodology .......................................................................................................................................................................................1 Findings, Conclusions and Recommendations............................................................................................................................1 Constraints And Weaknesses Of The Project...........................................................................................................................8 Lessons Learned………………………………………………………………………....................................................9 Recommendations ...........................................................................................................................................................................10 SECTION 1. EVALUATION OVERVIEW.................................................................................................................11 SECTION 2. BACKGROUND.....................................................................................................................................11 SECTION 3. ACTIVITY OVERVIEW.........................................................................................................................12 3.1 RESULTS FRAMEWORK ...............................................................................................................................................12 3.2 AGRICULTURAL POLICY ACTIVITY’S STRATEGY.............................................................................................15 3.3 Main Activities...................................................................................................................................................................15 3.4 Critical Assumptions........................................................................................................................................................16 3.5 Performance Indicators...................................................................................................................................................17 SECTION 4. EVALUATION PURPOSE AND METHODOLOGY.....................................................................18 4.1 Evaluation Purpose ...........................................................................................................................................................18 4.1.1 Evaluation Questions.............................................................................................................................................18 4.2 Evaluation Methodology..................................................................................................................................................19 4.2.1 Primary Data Collection ............................................................................................................................................20 4.2.2 Sampling Strategy..........................................................................................................................................................20 4.2.3 Sample Selection and Composition of Partners for Key Informant Interviews ...........................................21 4.3 Desk Review ......................................................................................................................................................................22 4.3.1 Focus of Document Review ......................................................................................................................................23 4.4 Key Informant Interviews...............................................................................................................................................23 4.5 Mapping of Evaluation Questions with the Category of Stakeholders ...............................................................24 4.6 Data Analysis......................................................................................................................................................................25 4.7 Quality Control Mechanism for Data Gathering, Analysis and Oversight.........................................................27 4.8 Guiding Principles and Values........................................................................................................................................27 4.9 Study Limitations and Remedial Measures.................................................................................................................27 SECTION 5. FINDINGS AND CONCLUSIONS....................................................................................................28 5.1 EVALUATION QUESTION 1 (EQ1)..........................................................................................................................28 5.1.1 EQ1: Findings...........................................................................................................................................................28 5.2 EVALUATION QUESTION 1(a)..................................................................................................................................30 5.2.1 EQ1(a): Findings......................................................................................................................................................30 5.3 EVALUATION QUESTION 1(b) .................................................................................................................................31 5.3.1 EQ1(b): Findings......................................................................................................................................................31 5.4 EVALUATION QUESTION 1(c)..................................................................................................................................31 5.4.1 EQ1(c): Findings......................................................................................................................................................31 5.5 EVALUATION QUESTION 1(d) .................................................................................................................................31 5.5.1 EQ1(d): Findings......................................................................................................................................................32 5.6 EQ1: CONCLUSIONS....................................................................................................................................................32 5.7 EQ1: RECOMMENDATIONS.......................................................................................................................................33 5.8 EVALUATION QUESTION 2 (EQ2)..........................................................................................................................33 5.8.1 EQ2: Findings...........................................................................................................................................................33 5.9 EVALUATION QUESTION 2(a)..................................................................................................................................35 5.9.1 EQ2(a): Findings......................................................................................................................................................36 ii 5.10 EQ2: CONCLUSIONS....................................................................................................................................................40 5.11 EQ2: RECOMMENDATIONS.......................................................................................................................................40 5.12 EVALUATION QUESTION 3 (EQ3)..........................................................................................................................41 5.12.1 EQ3: Findings............................................................................................................................................................41 5.12.2 EQ3(a): How well are local research partners providing needed support?...........................................42 5.12.3 EQ3(a): Findings......................................................................................................................................................42 5.12.4 EQ3(b): How often do policy actors interact to advance policy processes…………………… …43 5.12.5 EQ3: Conclusions..................................................................................................................................................44 5.12.6 EQ3: Recommendations.......................................................................................................................................45 5.13 Evaluation Question 4 (EQ4).........................................................................................................................................45 5.13.1 EQ4: Findings...........................................................................................................................................................45 5.13.2 EQ4: Conclusions...................................................................................................................................................46 5.13.3 EQ4: Recommendations.......................................................................................................................................46 5.14 Evaluation Question 5 (EQ5).........................................................................................................................................46 5.14.1 EQ5: Findings...........................................................................................................................................................46 5.14.2 EQ5: Conclusions...................................................................................................................................................47 5.14.3 EQ5: Recommendations.......................................................................................................................................47 SECTION 6. CONSTRAINTS AND WEAKNESSES..............................................................................................48 6.1 Issue of Baseline Not Adequately Addressed ...........................................................................................................48 6.2 Use of Cumbersome Custom Indicators ...................................................................................................................48 6.3 Delay in Conduct of Midterm Evaluation ...................................................................................................................49 6.4 Partnership Relationships ...............................................................................................................................................49 6.5 Communication Lapses...................................................................................................................................................50 6.6 Advisors’ Visits to MSU ..................................................................................................................................................50 6.7 Training Logistics..............................................................................................................................................................50 SECTION 7. LESSONS LEARNED & RECOMMENDATIONS ...........................................................................51 7.1 Lessons Learned................................................................................................................................................................51 7.2 Recommendations............................................................................................................................................................52 7.2.1 Way Forward ..........................................................................................................................................................52 7.2.2 Redefine Focus of Follow-On Project...............................................................................................................52 7.2.3 Focus Follow-On on Specific Policy Areas ......................................................................................................52 7.2.4 Ensure proper Targeting of Beneficiaries.........................................................................................................53 7.2.5 Strengthen Process of Establishing and Maintaining Partnerships..............................................................53 7.2.6 Strengthen Implementation Procedure.............................................................................................................53 7.2.7 Ensure efficient Delivery of Project Activities ................................................................................................53 7.2.8 Encourage Use of Stable and “Smart” Performance Indicators..................................................................53 7.2.9 Sustainability.............................................................................................................................................................54 ANNEXES ANNEX 1: Statement Of Work ....................................................................................................................................1 ANNEX 2: Work Plan ...................................................................................................................... ………………..1 ANNEX 3: Results from Descriptive and Inferential Analysis……….…...………………………………..1 ANNEX 4: List of Key Informants Contacted in the North during the Evaluation...........................................1 ANNEX 5: List of Key Informants Contacted in the South during the Evaluation............................................1 ANNEX 6: List of Training Workshop Beneficiaries Contacted and Interviewed ............................................1 ANNEX 7: List of Conference Beneficiaries Contacted and Interviewed...........................................................1 ANNEX 8: References.....................................................................................................................................................1 ANNEX 9: Evaluation Team Biographies………………………………………………………………… 1 ANNEX 10: Conflict of Interest Statements…………………………………………………………….. 1 iii LIST OF TABLES Table 1: Agricultural Policy Project Performance Indicators 15 Table 2: Sample Size by Gender and Category of Participants 19 Table 3: Evaluation Questions and Target Respondents 21 Table 4: Analytical Approach and Data Collection Sources to Answer Each Evaluation Question 23 Table 5: Distribution of Participants in NAPP Workshops by Gender and Type of Training 27 LIST OF FIGURES Figure 1: USAID/Nigeria Economic Growth Results Framework (2015 -2019 CDCS) 11 Figure 2: FTF Nigeria Agricultural Policy Project Results Framework 12 Figure 3: Three-Stage Evaluation Methodology 19 Figure 4: Sampling Strategy 21 Figure 5: Distribution of Training Beneficiaries by Category of Stakeholders 19 Figure 6: Distribution of Conference Beneficiaries by Category of Stakeholders 20 Figure 7: Distribution of respondents by Age 28 Figure 8: Involvement of Training Beneficiaries in Policy Research 31 Figure 9: Application of skills acquired under NAPP training 32 Figure 10: Effect of Training on Decision to Conduct Policy Research 32 Figure 11: Extent of improvement in capacity to conduct policy research under NAPP training 34 Figure 12: Comparison of usefulness of capacity building activity under NAPP 35 Figure 13: Change in capacity to conduct policy research 36 Figure 14: Creation of interest in conducting independent policy research under NAPP 37 iv ACRONYMS ABU Ahmadu Bello University ADAN Association of Deans in Agriculture in Nigeria Universities ADP Agricultural Development Programs ADWG Agriculture Donor Working Group APRNet Agricultural Policy Research Network ARCN Agricultural Research Council of Nigeria ATA Agricultural Transformation Agenda ATLAS Software, Qualitative Data Analysis AU African Union BSU Benue State University C:AVA Cassava: Adding Value for Africa CBA Cost-Benefit analysis CDCS Country Development Cooperation Strategy CSO Civil Society Organization DO Development Objective EQ Evaluation Question ET Evaluation Team FCT Federal Capital Territory FMARD Federal Ministry of Agriculture and Rural Development FSP Food Security Policy FTF Feed the Future FUAM Federal University of Agriculture, Makurdi FUNAAB Federal University of Agriculture, Abeokuta GDP Gross Domestic Product GON Federal Government of Nigeria IFPRI International Food Policy Research Institute IR Intermediate Result KII Key Informant Interview KPPG Key Political Partners Group KSUST Kebbi State University of Science and Technology Landpkts Complete Toolkit for Sustainable Land Management M&E Monitoring and Evaluation MANR Ministry of Agriculture and Natural Resources MDA Ministry, Department, Agency MEL Monitoring, Evaluation and Learning MOU Memorandum of Understanding MSU Michigan State University MTR Midterm Review v NAAE Nigerian Association of Agricultural Economics NAC National Advisory Council NAIC Nigerian Agricultural Insurance Scheme NBS National Bureau of Statistics NEPAD New Partnership for African Development NGO Non-Governmental Organization SR Strategic Results SMART Specific, Measurable, Achievable and Attributable, Realistic and Time-bound/Timely STATA Statistical Analysis Software for Data Visualization TETFUND Tertiary Education Trust Fund UNN University of Nigeria USAID United States Agency for International Development USG United States Government WEF World Economic Forum WTO World Trade Organization YISA Youth Initiative for Sustainable Agriculture 1 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION EXECUTIVE SUMMARY Background This report presents the findings, conclusions, and recommendations of the final performance evaluation of the United States Agency for International Development (USAID)-funded Feed the Future (FTF) Nigeria Agricultural Policy Project (NAPP), a five-year, $12.5-million capacity-building and policy research and dissemination initiative implemented by Michigan State University (MSU) in partnership with the International Food Policy Research Institute (IFPRI). The goals of the activity were to; 1) increase Nigerian national capacity to generate and analyze information in order to formulate evidence-based policy options for consideration by decision-makers and stakeholders; and, 2) improve policy processes driven by empirical evidence and active dialogue among decision-makers and stakeholders. This evaluation, conducted between August and September 2020, is intended to provide an independent analysis of the overall progress and accomplishments of the project to determine whether the assistance provided by USAID through the NAPP achieved its development objectives. Taking into consideration the findings and recommendations of the earlier midterm performance evaluation, this final evaluation will provide comprehensive lessons learned from NAPP to inform the design of future programs. Methodology The evaluation team (ET) applied a mixed-methods approach, leveraging quantitative and qualitative research methods1 and using primary and secondary data to undertake a thorough final performance evaluation. Following desk reviews of the Activity’s background documents, primary data were collected virtually via key informant interviews (KIIs), focus group discussions (FGDs), and telephone interviews - a total of 633 randomly selected respondents. Specifically, FGDs and KIIs were held with USAID, implementing partners - MSU and IFPRI, members of the National Advisory Committee, civil society organizations and associations, project scholars, scholar advisors, the media, the Federal Ministry of Agriculture and Rural Development (FMARD), and state partners - universities and state ministries in Ebonyi, Delta, Kebbi, Kaduna, Oyo, Cross River, Niger, and Benue. In addition, telephone interviews were conducted with project beneficiaries who participated in training workshops and conferences. Findings, Conclusions and Recommendations Evaluation Question 1 (EQ1): Gender/Youth – To what extent have women and youth (15–29 years) benefited from policy intervention undertaken by the activity? EQ1: Findings Participating women and youth reported benefiting to a great extent. Scholars reported that they have been able to apply lessons learned from the project, including new approaches, leading to improvements in the way they undertake their research and work. These scholars also reportedly went on to train others 1 Data generated from the KIIs were coded and analyzed using the ATLAS.ti software tool for qualitative analysis. Themes from the respondents’ answers were coded and analyzed, and afterwards using STATA, cross-tabulations and bar charts generated to show the percentage distribution of responses to quantitative data from women that benefited from national research capacity strengthening, improvement in policy research capacity of next generation of researchers, improvement in capacity to plan and implement effective policy analyses as well as improvement in policy dialogue and communication. 2 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION on analytical methods using R statistics in universities and ministries in some of the FTF focal states. In some cases, the scholars stepped down the training they received at MSU to faculty members and students in their respective schools. In general, however, female participation in project activities - events and capacity-building activities - was low, approximately 30 percent when compared to male counterparts. EQ1(a): Which approaches have been most effective in reaching women and youth? EQ1(a): Findings The scholar program was adjudged to be most effective in reaching women and youths. Scholar advisors and scholars themselves reported that they had incorporated learnings from their experience into their departments, which is reflected either in advisors’ teaching style or by knowledge-sharing with colleagues, helping to reach more women and youth. EQ1(b): How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? EQ1(b): Findings The project’s capacity-building and research activities reportedly helped increase the capacity of Nigerian analysts to undertake evidence-based policy analysis. Nigerian graduate students, i.e. project scholars, their research advisors, and project researchers from MSU or IFPRI, constituted the research teams who worked on specific topics, who in turn transferred the knowledge to the mid-level and junior researchers - the project scholars, of whom the majority, 54 percent, were female. Thus, through the training of scholars and other actors in the use of Stata and R statistical software, women in research are able to apply standard analytical tools to their research. EQ1(c) How have women and youth been part of the dialogue in the agriculture sector in developing national policy systems? EQ1(c): Findings Although one of the scholars, a woman, said youth and female students who benefited from the program are yet to participate in policy dialogue, she, herself, developed a policy brief of her gender-focused research which was presented during one of the sessions at the 2018 NAAE conference, was uploaded on the FTF website, and was subsequently shared on ResearchGate platform where her work has been read over 300 times so far. Other scholars stated that they had participated in policy debates organized by other groups. EQ1(d) Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? EQ1(d): Findings One example of gender-focused research, Gender Dynamics in Agglomeration Economies: A Case Study of Rice Processing Clusters in Kano State, Nigeria, carried out by an M.Sc. scholar, targeted parboilers in rice￾processing clusters. Parboiling is typically women-dominated, but she found that men and youth were also active in that space. Another example is a study by an M.Sc. scholar who examined the climate variables that primarily affected youth and women. Other activities developed by Ph.D. scholars include the provision to communities, particularly to women, of a pamphlet on dietary diversity, how to use available foods to make a rich diet. In addition, training on the “Landpks App,” Complete Toolkit for Sustainable Land 3 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Management, was delivered to farmers for on-the-spot soil testing by another Ph. D. scholar in Benue State. EQ1: Conclusions Women have benefited from the policy interventions undertaken by the NAPP, but their participation was at a disproportionately lower level than men. NAPP activities were open to all and were not discriminatory to women and youth, but as one advisor pointed out, “the challenge is that female participation in this area, that is the agriculture policy sector, is low.” The scholar program has had a remarkable effect on beneficiaries compared to other activities. Beneficiaries of the scholar program have been able to apply the knowledge gained from their participation in the program, including new approaches to undertaking research, and these have led to improvements in their policy research. EQ1: Recommendations To ensure the participation of women in areas where there is a general dominance of male participants, deliberate promotion of female attendance for events is critical. Future activities should, therefore, consider adopting minimum participation levels for females in their interventions. Guidelines to ensure female participation, specifying the cadre of females required to attend program events, should be communicated to stakeholders. Furthermore, replication of the longer-term support offered by the scholar program, including mentoring, participation in research teams, etc., may help increase women’s participation in future capacity-building interventions. Evaluation Question 2 (EQ2): To what extent has the direct training of select graduate students in Nigerian universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? EQ2: Findings The capacity of the next generation of policy researchers in Nigeria has been strengthened to a great extent, resulting in greater involvement in policy research. Post training, over half, 58 percent, reported more involvement in policy research activities, while others reported that they were applying their skills by communicating policies to farmers. Project beneficiaries reported improved skills for undertaking data analysis, drafting policy briefs, and report writing. EQ2(a): How have policy issues chosen by the Activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy process? EQ2(a): Findings NAPP activities, the scholar program, training events, conferences, etc. have helped increase the exposure of the next generation of Nigerian researchers to new approaches, such as the improved teaching methods learnt by scholars’ advisors at MSU, and the adoption of standard analytical tools like Stata. In addition, involvement in NAPP activities has enabled these researchers to interact with policy makers to provide support in the policy process. For example, a scholar’s advisor in Kaduna State noted that the MSU visit afforded advisors the opportunity to acquire new teaching methods that they brought to bear on how they supervise their graduate students, particularly in the area of research methodology. In a similar vein, 4 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION the UNN scholar’s advisor said she learned new ways to teach, to carry out research, to write research grant proposals, and to supervise students, ways that she is effectively applying with success. At the state level, the ministries reported that policy brief training sessions enabled them to improve operations in their ministries. The Benue Ministry of Agriculture and Natural Resources (MANR), for example, reportedly used lessons learned from the training to write a policy brief to the Commissioner promoting the adoption of an agriculture policy for the state. NAPP engagement with the Cross River State MANR also led to the revision of that state’s Agricultural Policy. The ministries reported that the training sessions led to the development of specific policy briefs aimed at developing key sectors of the economy. For example, the Niger State MANR reported that they had developed policy briefs for rice and cowpea value chains, while Kaduna State MANR reported the development of a draft policy brief calling for improved state participation in the small ruminant sector. Cross River MANR reported that although they do not conduct research, themselves, they are now able to commission policy research because NAPP training had helped ministry staff to recognize and articulate the need for evidence-based research. In addition to the enhanced capacity of academia and MDAs through various capacity building programs, the MDAs from Delta and Ebonyi states worked with the university scholars and noted that they would welcome future collaboration. Furthermore, the Benue MANR reported that training delivered by NAPP to the ministry facilitated the collection of key economic indicators needed for the analysis of the livelihoods of people in Benue State, an analysis that is required for the annual food and nutrition security policy metrics computed for Nigeria. In addition, NAPP supported FMARD in their response to the COVID-19 pandemic as it relates to agriculture and food security. For example, in December 2019, as a member of the Agriculture Donor Working Group (ADWG), NAPP co-led the technical preparation of the policy brief for FMARD aimed at supporting the agriculture policy dialogue and review of the Ministry. Subsequently, along with other development partners in the ADWG, NAPP undertook a COVID–19 risk analysis and policy brief on its secondary impacts on the agriculture sector to serve as the context within which, from the agriculture lens, the group would support the GON in its resolve to grow the economy post-COVID. In March 2020, as the pandemic became global in breadth, NAPP conducted an impact analysis of the pandemic as a complementary study to estimate the direct and indirect impact of the pandemic on the Nigerian economy. The study is intended to feed into the FMARD-led assessment and policy formulation for the agricultural sector. Also, in May 2020, in partnership with World Fish, NAPP launched a study aimed at guiding the direction of policy and donor support to the fish and poultry value chains in a post-COVID-19 era. NAPP scholars, for their part, launched a virtual informational series in April 2020 to provide technical information on COVID-19 and its impact on several sectors of the Nigerian economy. The series focused primarily on agriculture and included expert discussions on food security, nutrition, climate change, agricultural research, gender, and livelihoods. NAPP scholars reported that the policy issues chosen by the program has enabled them to interact with policy makers to provide support in the policy process based on their own areas of interest and expertise. One scholar reported that he is part of a technical team working with Kebbi State to review the agricultural roadmap to transform it into a state agricultural policy document. An overall majority, 62 percent, of beneficiaries stated that the training had improved their capacity to conduct policy research; an additional 27 percent claimed much improvement, while only 11percent claimed their capacity had 5 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION remained unchanged (Figure 11). Indeed, before the training, many beneficiaries had little or no knowledge about econometric tools such as STATA and R, but, as a result of the training received, they have been able to conduct policy analysis independently. EQ2: Conclusions The relationship between academia and policy makers is key to disseminating research findings. The involvement in NAPP activities has reportedly enabled researchers to interact more with policy makers, to provide support in the policy process. Also, training provided to state government stakeholders has strengthened the capacity of these beneficiaries to undertake policy development and implementation. EQ2: Recommendations Future policy interventions should engage with actors to identify critical areas of need to provide targeted interventions to support improvements at state ministries, departments, and agencies (MDAs). Also, future policy interventions may benefit from concentrating efforts on a consensus of key policy issues and follow them through, widely publicizing any changes which occur. Evaluation Question 3 (EQ3): To what extent has the capacity of local research partners been strengthened to inform policy debates and implementation with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? EQ3: Findings Capacity of local researchers has reportedly been strengthened to a great extent. NAPP activities have resulted in improved knowledge in data analysis, increased engagement of local researchers in research projects, and better opportunities for research grants. In addition, beneficiaries reported that they have a better understanding of policy research and are confident in their ability to conduct it independently. The motivation to engage in policy research is largely due to improved knowledge of sector issues and improved capacity to undertake research. EQ3 (a): How well are local research partners providing needed support to relevant State Ministries in the formulation and implementation of policy processes? EQ3 (a): Findings Researchers - NAPP scholars, their advisors, and research collaborators - reported some interaction with state MDAs to support the policy process. In Kebbi and Benue States, the MANRs reported that local researchers were instrumental in developing, and for Benue, finalizing their states’ agricultural policy documents. In Niger State, the project scholar and advisor have met with MDAs to identify their needs, including the capacity-building needs of the MANR. Additional studies, such as a feasibility study on the best way to utilize available land in the rice value chain, a sector in which women play a significant role, are being planned to be carried out after the easing of restrictions imposed by COVID-19. These studies will address the different areas of concern identified by the Ministry. The planned research is expected to feed into the rice sustainability policy of Niger State - one study would specifically investigate the comparative advantages of rice produced by the State. Similarly, in Benue State one scholar worked with the Benue MANR on a farmer-herder project during conflict periods. This scholar trained MANR staff and farmers on the use of the LandPKS testing app, saving farmers the trouble of going to a laboratory for soil tests. The LandPKS app which can be used in any location worldwide, is an application which supports 6 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION more sustainable land management decision-making by assisting users to collect site-specific data about their soils, vegetation, and other site characteristics. Another scholar developed a policy brief on the degradation of soil in Taraba State, which was shared with the MANR. Agricultural Policy Research Network (APRNet) reported that they shared seven of their journals with FMARD and in Kogi State; APRnet contributed to Kogi’s rice project and suggestions given by APRNet were adopted. Finally, the Kebbi MANR noted that they no longer have to send out research needs to the research institute in Zaria because of the confidence they have in the capability of Kebbi State University of Science and Technology (KSUST) to carry out the research needed. EQ3 (b): How often do policy actors interact to advance policy processes at national and state levels? EQ3 (b): Findings There is evidence of periodic engagement of actors to advance policy processes in Kebbi State. For example, a “brown bag” series was created including researchers from KSUST, policy makers, i.e. state MANR, and farmers. These meetings were a result of NAPP’s activities; during development of the policy document, the actors perceived the necessity to collect research inputs from stakeholders, such as farmers, and communicate them to policy makers. Stakeholders in this series met three times - April 2019 (inaugural meeting), July 2019, and November 2019. This series indicates that relevant research was carried out and that the policy makers obtained the required evidence for effective policy making. EQ3: Conclusions NAPP activities have reportedly strengthened the capacity of local researchers and improved their ability to participate in policy debates. In addition to increasing their exposure to new techniques, participation in NAPP has given them a platform from which to engage with MDAs, and thus build the confidence of ministries in the policy process. In Niger State, for example, researchers have been able “to engage with MDAs to find out the areas of concern and are working to provide needed support” (e.g., targeted studies) to address the identified issues. In Kebbi State, the agricultural policy document is regarded as a key achievement of NAPP, as it has improved the capacity of the MANR in budgeting and planning activities. EQ3: Recommendations Future policy programs should continue this approach of involving local researchers in interventions targeted at MDAs, as it builds MDAs’ familiarity with and confidence in local research capabilities. Evaluation Question 4 (EQ4): To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? EQ4: Findings The capacity of the media has reportedly been increased to a great extent. Feedback from media respondents indicates that NAPP activities have enhanced knowledge about various reporting outlets, including social media, for their stories. Media actors who attended the workshops in Niger State said that they applied methodologies learned to undertake investigative journalism about food insecurity. Also, knowledge from the training events was shared with other journalists and media students in Niger State. One of the media actors in Kebbi State pointed out that the training enabled her to produce radio call-in programs - which, in turn, revealed some of the challenges farmers face in Kebbi, particularly those who got involved in an anchor borrowers’ scheme. Another media beneficiary in Kebbi State reported that he 7 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION and other colleagues undertook investigative journalism and published reports in various media which led to the inclusion of previously excluded rice farmers in a state loan scheme. Finally, the Delta MANR reported that the training of local reporters boosted their confidence and knowledge on reporting agriculture-related issues. EQ4: Conclusions NAPP contributed immensely to building the capacity of the media to undertake informed reporting of policy issues. The media actors in Niger and Kebbi found the media training, particularly the senior master class training, and workshops they attended to be useful and reported improved capacities for informed reporting to better enlighten the public about agricultural issues. EQ4: Recommendations Future policy programs should build upon NAPP’s work in building the capacity of the communications unit in relevant agriculture agencies, as well as the capacity of mainstream media actors, to ensure improved analysis and reporting of agriculture-related issues. Considering that agriculture is not a typical area covered by media actors, future agriculture policy programs targeting media actors should allocate more time for the training workshop to allow for relevant topics to be covered more in-depth and allow for feedback from participants. ▪ Future policy programs should build upon NAPPs work in growing the capacity of the communications unit in relevant agriculture agencies, as well as the capacity of mainstream media actors, to ensure improved analysis and reporting of agriculture-related issues. ▪ Considering that agriculture is not a typical area covered by media actors, future agriculture policy programs targeting media actors should allocate more time for the training workshop, to allow for relevant topics to be covered in greater depth and to allow for feedback from participants, rather than scheduling the workshop for two or three days. Evaluation Question 5 (EQ5): What measures is the activity putting in place to ensure: sustainability of the interventions and the changing needs of the policy environment at the national/state levels? (ii) policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs? EQ5: Findings The key aspects of sustainability put in place by NAPP, as reported by implementing partners, were collaborative research and the scholars’ program. The reasoning behind this was that if the goal is to strengthen research networks and have “go-to” people to provide an evidence base for the policy process, then one would want those people to have strengthened capacity and be close to policy conversations. Regardless, the most convincing evidence for sustainability identified by the ET is the knowledge gained by the MSU scholars, research collaborators, and capacity-building beneficiaries. Of note, however, is that whereas policy research is collaborative in nature, the leadership of the research teams was restricted to MSU and IFPRI researchers. This limits sustainability. Furthermore, a limited source of funding and lack of political will is identified as a major risk to sustainability and continuity of policy dialogues. EQ5: Conclusions The training-of-trainers’ model promoted under NAPP is regarded as key to the sustainability of the interventions. With the 13 scholars, scholars’ advisors, APRNet, NAAE, etc., there are now a significant 8 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION number of people that can be called upon without MSU or IFPRI being physically present. Although there are several risks to the sustainability of the interventions, the two major risks identified are financial and political. Nonetheless, it is moderately likely that the outcomes and effects of NAPP will be sustained. EQ5: Recommendations Careful attention should be given to exit strategies for interventions during implementation, including alternative funding and resourcing for activities, so sufficient room is provided to ensure interventions can self-sustain after implementation. In this way, institutionalizing learning can be sorted out while the program is still running. It will be important to strengthen promising aspects of NAPP in future policy projects, particularly the capacity-building scholars’ program and the policy development aspects, so that they adequately address the gaps. Future projects should also consider including Nigerian policy researchers in the conduct of policy research. This can be achieved through an entrenched and systematic process of calling for research proposals from Nigerian researchers in already identified thematic policy areas. For instance, the activity can issue requests for proposals from the agricultural policy research community and select the best proposals for funding under the identified research themes. Similarly, beyond project implementers, funds should be provided to local institutions to build partnerships and jointly carry out specific research, training, and dissemination activities. This is an important built-in mechanism to foster the sustainability of similar projects. CONSTRAINTS AND WEAKNESSES OF THE PROJECT The most remarkable constraint was the COVID-19 pandemic, which hit the world during the last year of the project and resulted in the disruption of some planned activities, especially in the last three quarters of the year. The IPs made efforts to continue with implementation of the project to the extent permissible, actively complying with public health measures and protocols in many working places. Over the life of the project, considerable achievements have been made. However, it would have been possible to achieve even greater results but for some inherent weaknesses in the implementation procedure. ▪ In order to fully assess achievements in this project based on the specified performance indicators, baseline information would have been valuable to guide the process. ▪ With respect to the stakeholder evaluation survey to capture levels of satisfaction and confidence, the use of the cumbersome index indicators is ill-conceived and ill-advised. It would have been helpful if the midterm performance evaluation, carried out in 2018, had been strategically scheduled to allow all the midline lessons to be incorporated in the latter half of the Activity, so that necessary corrections could be made before the end of the project in 2020. Whereas some recommendations from the midterm performance evaluation, such as the Activity’s shift from continuous research to using research to engage with stakeholders, as well as the Activity ensuring that its agenda is set by stakeholders instead of the Activity, were incorporated into the Activity, other aspects were not. For instance the Activity failed to address (i) the issue of weak demand for research products through strengthening research infrastructure among local partners and stakeholders, (ii) the disproportionate emphasis on academic-style research that may be seen as advancing the careers of the IFPRI and MSU researchers, rather than on bringing Nigerian collaborators to the forefront, (iii) the provision of financial 9 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION support to execute collaborative policy works with the government in the focal states, and (iv) the introduction of an entrenched and systematic process of calling for research proposals from Nigerian researchers in already identified thematic policy areas to expand the participation of local researchers in the conduct of research under the Activity. The midterm evaluation did not commence until November 2018 and lasted until March 2019, leaving less than two years for project implementation. Also, the last year of the project witnessed the COVID-19 pandemic, which disrupted operations at the level of project implementation. However, some of the recommendations which were implemented helped to smoothen the implementation process. The effects of such changes would have yielded more results if the implementers had had sufficient time to adjust. ▪ Weaknesses in partnership relationships have arisen due to the informal nature of the partnerships or poor identification and definition of roles and responsibilities. ▪ In the case of partnerships with associations, apathy on the part of some facilitators has been experienced due in part to communication lapses between them and NAPP. This has affected the effectiveness of some NAPP activities. ▪ Feedback from one advisor indicated that there should be a clear plan about MSU activities before arriving at MSU, which should then be agreed with advisors. The advisor needed further knowledge or inputs relative to his research, and the parameters changed when he got to MSU. Eventually, he had to give a presentation on land which was preferred by MSU rather than the issue of entrepreneurship and economic analysis which was his priority. ▪ While it was reported that the training was adequate, there were reservations about the welfare of people who participated in training events conducted in some states, e.g., Delta and Niger. These participants noted that though they travelled long distances within their States, often from outside the city where the training was conducted, to attend training, they were not reimbursed or compensated for their transport costs. LESSONS LEARNED ▪ One change the implementing partners noted was made by the activity after the Midterm Review (MTR); there was a shift in project focus from continuous research to using research to engage with stakeholders. In engagements with stakeholders, they often found that further work was needed to tailor the research to the various audiences. As such, they converted the research to non-technical pieces. ▪ Another change, that the partners noted was made after the MTR, was to ensure the agenda was set and research determined by stakeholders in Nigeria and not by MSU/IFPRI. Particularly in Year 5, the IPs consulted stakeholders to understand their research needs. In general, they said they learned that Nigeria needs to set the agenda while the activity supports that agenda, requiring the program to adapt resources to reflect this change. As noted earlier, however, there were other recommendations in the MTR, with potential benefits, which the IPs failed to address for reasons best known to them and quite unrelated to shortage of time or COVID-19 pandemic. ▪ Incentives need to be provided for stakeholders who are not direct beneficiaries of the activity. This refers particularly to the members of the National Advisory Council (NAC). At some point the IPs held back from asking the NAC to do more; the project was conscious of the fact that NAC members were offering their services free of charge. 10 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION RECOMMENDATIONS The findings from the key questions guiding this evaluation show that NAPP has been beneficial in several respects in terms of improving skills and capacity to conduct policy research and widely disseminating results. A continuation of the project is strongly recommended, with a focus on areas of where substantial impact has been made and where benefits can be upscaled. It should be noted, however, that there are a number of design and implementation issues that must be appropriately addressed when a follow-on project is designed. Going forward, there should be changes in the types of activities; beneficiary targeting; implementation procedure, e.g. issues of baseline, NAC membership, etc.; performance indicators; partnership arrangements - and ways of defining partnerships; stakeholders’ roles and responsibilities; and feedback mechanisms. In the light of the foregoing, the following specific recommendations should be helpful. ▪ Properly define the focus of a follow-on project. A follow-on project must target specific policy segments and work with the stakeholders to achieve the desired results. The project must dwell more on tangible activities and results that can generate returns within the short to medium term. ▪ Ensure proper targeting of beneficiaries. In the case of gender-focused capacity building, greater participation by women themselves is necessary. ▪ Strengthen the process of establishing and maintaining partnerships. Greater detail in the memoranda of understanding or other structural steps lead to clarity of responsibilities. ▪ Strengthen the implementation procedure. A baseline should be established in an activity of this nature. Coverage of activities should be narrowed and appropriate performance indicators constructed to monitor progress. Both output and outcome indicators should be articulated to track implementation performance. ▪ Ensure efficient delivery of activities by identifying better ways of carrying out the activities to ensure resources are deployed efficiently. ▪ Encourage the use of stable and “smart” performance indicators. To avoid confusion and reporting challenges, it is important to avoid frequent changes in performance indicators. If there is a follow￾on, there should be agreement on performance indicators with implementing partners at the start of the project. This is particularly important for baselines, because changes in indicators would necessitate baselines to be conducted for each new indicator (in the particular case of outcome indicators, for example). ▪ To ensure sustainability, funds should be provided to the local institutions to build partnerships and jointly carry out specific research, training, and dissemination activities. In addition, establish linkages with other FTF activities. For example, farmers under NAPP might be linked with the USAID-funded Agri-business Activity to enhance their access to modern technology and provide better market linkages and integration. 11 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION SECTION 1. EVALUATION OVERVIEW This report presents the findings, conclusions, and recommendations of the final performance evaluation of the United States Agency for International Development (USAID)-funded Feed the Future (FTF) Nigeria Agricultural Policy Program (NAPP), a five-year, $12.5-million capacity-building and policy research and dissemination initiative implemented by Michigan State University (MSU) in partnership with the International Food Policy Research Institute (IFPRI). MSU and IFPRI began implementing the NAPP contract on July 1, 2015 with oversight from USAID. Although NAPP was originally scheduled to end on June 30, 2020, it was extended by USAID to December 31, 2020. The goals of NAPP are to 1) increase Nigerian national capacity to generate and analyze information in order to formulate evidence-based policy options for consideration by decision-makers and stakeholders and 2) improve policy processes driven by empirical evidence and active dialogue among decision-makers and stakeholders. This dual goal will be achieved: a) with due recognition of gender dimensions, through strengthening the national capacity for greater evidence-based policy processes through collaborative research for informed agricultural policy debate and formulation at the national and regional levels, b) by promoting and fostering interaction and informed policy dialogue among all agricultural sector stakeholders to promote a “think tank” culture within the agricultural policy landscape, and c) through improving federal and state governments’ capacity to plan and implement effective policy analyses and programs—and demand and absorb policy research in their policy process. The purpose of this final performance evaluation is to determine whether the assistance provided by USAID through the NAPP achieved the stated development objectives and to understand the lessons learned from NAPP in Nigeria, including the findings and recommendations of the midterm performance evaluation. It is intended to provide an independent examination of the overall progress and accomplishments of the project. SECTION 2. BACKGROUND With an abundant and diversified agricultural base, Nigeria’s economy has a large agricultural component. Until the early 1970s, Nigeria was self-sufficient in food production, with a small surplus for export; agriculture was the main foreign exchange earner. Post-1970, however, the sector stagnated for a number of reasons, chief among them the discovery, exploitation, and export of oil and a deliberate policy to shift resources from agriculture to industry (Oyejide,1986). Because agriculture employed an overwhelming share of the Nigerian labor force, stagnation of the sector resulted in increased poverty. The poverty rate increased from 28 percent in 1980 to 68 percent in 2012; since 2015, however, the performance of the agricultural sector has reportedly improved (National Bureau of Statistics [NBS], 2017; CBN, 2018). The largest country by population and size of the economy in Africa, Nigeria has abundant and rich agricultural resources, human capital, and a diversified and rich natural resource base. Although the agriculture sector employs about 48 percent of the labor force, in 2017 agriculture contributed around 25 percent to the GDP of Nigeria, suggesting that productivity and incomes in the sector were low (NBS, 2018). Unemployment as well as poverty in rural areas, especially among the youth, tends to be high. According to the National Bureau of Statistics, poverty incidence (using $1.25/day per capita as the measure) in 2011 was 52 percent in urban areas but 66.1 percent in rural areas. Although sectoral poverty data are not available, rural unemployment rates are higher than urban unemployment rates: 33.5 percent in urban areas, compared to 38.2 percent in rural areas. 12 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Recognizing the need to revitalize and transform the agriculture sector to increase rural incomes and grow its economy, in 2011 the Federal Government of Nigeria (GON) embarked on a visionary strategy for the sector through the Agricultural Transformation Agenda (ATA). The goal of the ATA was to increase rural incomes and grow its economy by providing an improved enabling environment for the private sector to lead the agricultural transformation process. More recently, in 2013, Nigeria joined the New Alliance for Food Security and Nutrition led by the G8 and the Grow Africa initiative led jointly by the African Union (AU), the New Partnership for African Development (NEPAD), and the World Economic Forum (WEF) as a way to share knowledge about its own policy and investment commitments through the ATA and the shared goals of promoting greater private sector investments. NAPP is intended to ensure that all partners have the capability to meet the growing national and international research demands that support the capacity, knowledge, and information needs of the Federal Ministry of Agriculture and Rural Development (FMARD). SECTION 3. ACTIVITY OVERVIEW The goals of NAPP are to 1) increase Nigerian national capacity to generate and analyze information in order to formulate evidence-based policy options for consideration by decision-makers and stakeholders and 2) improve policy processes driven by empirical evidence and active dialogue among decision-makers and stakeholders. Under these goals, NAPP has three principal objectives: 1. To strengthen national capacity for greater evidence-based policy processes in agriculture by increasing the capacity of Nigerian analysts to undertake and make widely available relevant evidence-based policy analysis; 2. To promote and foster informed policy dialogue among all stakeholders in the agricultural sector through an inclusive, transparent, and sustainable process at the country level, building blocks for a well-integrated and developed national policy system; and 3. To help federal and state governments improve their capacities to plan and implement effective policy analyses and programs, and demand and absorb policy research in their policy process. 3.1 RESULTS FRAMEWORK To achieve the goals and principal objectives, the overall results framework (Figure 1) for the FTF Innovation Lab for Food Security Policy (FSP) project and the Country Development Cooperation Strategy (CDCS) provide the model for the results framework for the Project. The FSP results framework is itself consistent with that of FTF, the United States Government’s (USG’s) global hunger and food security initiative. The results framework adapted for this activity is illustrated in Figure 2 below. It shows the pathways through which the activity will achieve its goals of promoting more inclusive, private-sector-led agricultural growth (which is the FTF strategy’s First Level Objective) and reducing poverty (which is the goal of the CDCS). This framework is central to the management, monitoring, and evaluation of this project. 13 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Within the USAID CDCS, the FTF Agricultural Policy project is located under Development Objective (DO) 1, Broadened and inclusive growth, Intermediate Result (IR) 1.2: Business Environment Improved, Sub-IRs 1.2.1: Legal and Regulatory Environment Improved. Figure 1: USAID/Nigeria Economic Growth Results Framework (2015–2019 CDCS) 14 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Figure 2: FTF Nigeria Agricultural Policy Project Results Framework The NAPP results framework also includes one customized sub-sub IR and five strategic results (SRs) across the activity components that will contribute to the sub-IRs. These tiers are arranged to illustrate the causal relationship between planned interventions under the activity and their expected impact, and to identify the IRs that are critical to achieving activity objectives. Specifically, the framework demonstrates how planned interventions and deliverables will lead to expected outputs, outcomes, results, and eventual impact. Together the DOs, IRs, sub-IRs and SRs identified in Figure 2 provide the framework for identification and implementation of activities designed to achieve the required results. 15 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 3.2 AGRICULTURAL POLICY ACTIVITY’S STRATEGY NAPP uses a robust approach to enhance skills, training, and institutional capacities to meet the demands for policy analysis by the FMARD and the national food security and agricultural policy processes. In order to achieve this, the project engages in training collaborations with FMARD, academic institutions, and other key stakeholders. It strives to strengthen the links between various actors and to foster the development of a network of institutions that can independently interact during the policy process. The policy-driven collaborative research and analysis component directly supports the knowledge needs of the policy process and is sub-divided into two sub-components: research and policy analysis. The research component involves research teams comprising activity and Nigerian researchers with priority topics identified in consultation with FMARD, the states’ Ministries of Agriculture, development partners, and other key stakeholders. These researchers and their networks provide their expertise to inform policy debates with empirical evidence as they increase interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders). The policy analysis component, on the other hand, is more focused on demand-driven and shorter-term policy analyses that originate from requests by FMARD, the states’ Ministries of Agriculture, and their development partners. Strengthening the evidence‐based policy process and promoting impact will ensure that the activity addresses policy gaps through increased and targeted policy communications and capacity building at universities and FMARD. Strategies to strengthen the evidence-based policy process include: directly contributing to the donor-government policy dialogue on agricultural policy; launching a policy brief series for the activity; promoting a “think tank” culture within the agricultural policy process by organizing seminars and events targeted at all agricultural policy actors; providing training courses for FMARD and scholars; and providing for selected Nigerian graduate students to take advanced courses at MSU through the Scholars’ Exchange. 3.3 Main Activities The Agricultural Policy Activity planned to accomplish the SRs, IRs, and activity objectives through a work program organized around three themes—capacity building, research, and outreach. Examples of activities funded by the project include the following. 1. Support skill-enhancing training of staff from Nigerian partner institutions on methods and applications for research and policy analysis. The content of this training was developed in consultation with FMARD, Ahmadu Bello University (ABU), the University of Ibadan, and other key stakeholders to ensure their appropriateness in addressing the most serious gaps in knowledge, analytical and research approaches, and application tools in Nigeria. 2. Devise and institute a direct-training collaboration between researchers at MSU and IFPRI with graduate student researchers and their supervisors in Nigerian universities. This is to include a competitive annual selection of two or three Nigerian post-graduate students (one at the Masters’ level and one or two at the Ph.D. level). These students and their research advisors form a research team with MSU and IFPRI researchers on the proposed research topics. Where possible (and to be encouraged), the research project will constitute part of the students’ thesis or dissertation. The Nigerian students attend technical courses (e.g., econometrics, research design, 16 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION and/or modeling techniques) at MSU for one semester for Masters’-level students and one year for Ph.D. students. Periodic team meetings are scheduled to discuss team findings and progress and facilitate more learning by junior researchers. 3. Nigerian research supervisors visit the U.S. to participate in various activities, broaden their outlook, and facilitate the provision of feedback for their research students. The visits also facilitate interaction of the Nigerian professors with faculty at the department of Agricultural, Food and Resource Economics, as well as faculty at the College of Agriculture and Natural Resources, to broaden their outlook as well. The Nigerian professors collaborate with senior IFPRI researchers based in Washington, DC. 4. Based on demand, periodic training courses at Nigerian universities on various research methods and tools are organized to reach a broader set of promising graduate students and young research professionals. Lecturers and students in Nigerian universities have access to all course materials developed. Additionally, as part of this periodic training, lectures by visiting MSU and IFPRI staff are encouraged at the project’s collaborating institutions. This is a key long-term effort to facilitate the training of the next generation of Nigerian policy researchers and analysts. The training courses, which cover policy analysis methods and tools, target a broad audience including the research community, practitioners in government (such as those at FMARD and other ministries related to agriculture), CSOs, non￾governmental organizations (NGOs), the private sector, and think tanks. 5. Major conferences are held to promote the visibility of the policy research and analysis being undertaken, including activity workshops and other activities. MSU and IFPRI proposed organizing three major conferences during the tenure of the project (2015–2020), to bring together experts on selected issues in agriculture and economic development to provide guidance to Nigerian graduate students and young research professionals working in these areas. In addition, the project funds select graduate students to attend a national/regional conference and present their work. Such conference participation broadens the reach of these conferences and increases interaction between actors at various institutions interested in agricultural policy. The interactions are intended to strengthen the links between such actors and foster the development of a network of institutions that can independently interact during the policy process. 6. Formal ties are to be established with two or more Nigerian institutions in order to develop appropriate skills, knowledge, and tools to strengthen the credibility and relevance of evidence generated by policy analysis and research for informing policy. In addition to two already identified institutions (ABU and University of Ibadan), additional institutions are to be selected in line with FTF focus states and their interest and/or relevance to activity research topics; the purpose is to broaden the activity’s interaction with institutions across Nigeria’s diverse agro-ecological or geopolitical systems. 3.4 Critical Assumptions There were several critical assumptions made in order for the Agricultural Policy Activity to achieve its targets and objectives. The first assumption was that the policy making environment in Nigeria remains relatively stable and is not subjected to major upheavals, often associated with shifts in political power 17 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION resulting from undemocratic forces. The second is that the government, policy makers, and stakeholders targeted by the activity are receptive to new ideas, responsive to presented evidence, and committed to implementing and enforcing the outcomes of policy research and analysis. In other words, the results framework relies on there being political will and support for the policy change agenda and that the activity receives the cooperation and participation of targeted stakeholders in program activities and its outputs. These factors pose challenges; Nigeria is new to broad policy reform as, until recently, much of Nigeria’s policies were concerned with changing the urban landscape the fortunes of urban dwellers (Ering et al, 2014). The Agricultural Policy Activity will thus provide timely evidence to improve the local capacity to conduct evidence-based policy analysis and advocacy to overcome these challenges. 3.5 Performance Indicators The NAPP performance indicators for monitoring and reporting purposes are listed in Table 1. These 10 indicators were identified through a review of the performance indicators used in the FTF Monitoring System (FTFMS), and those developed for the global Food Security Policy (FSP) project. The indicators in this list track the delivery of outputs and outcomes related to project efforts to strengthen evidence￾based policy processes for agriculture and food security in Nigeria. Table 1: Agricultural Policy Project Performance Indicators # Indicator Title Indicator ID in FTFMS Unit of Measurement 1 Number of high-quality research reports published. Custom Number 2 Number of participants attending project-organized research and policy events. Custom Number 3 Number of agricultural and nutritional enabling environment policies analyzed, consulted on, drafted, or revised, approved, and implemented with USG assistance (RAA). EG.3.1-12 (Custom) Number 4 Number of individuals participating in USG food security programs. EG.3.2 Number 5 Number of individuals who have received USG supported degree-granting agricultural sector productivity or food security training. EG.3.2-2 Number 6 Percent of USG-assisted organizations with improved performance [IM-level]. CBLD - 9 Percentage 7 Number of agriculture policy communications developed and/or written for stakeholder consumption. Custom Number 8 Number of public private advocacy dialogues focused on policy that supports private sector investment Custom Number 9 Index (or scorecard) of quality of agriculture and food security policy processes in Nigeria, as measured by stakeholder evaluation to capture level of satisfaction and confidence. Custom Average Score 18 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 10 Index (or scorecard) of quality of the institutional architecture for agriculture and food security policy processes in Nigeria, as measured by stakeholder evaluation survey to capture level of satisfaction and confidence. Custom Average Score SECTION 4. EVALUATION PURPOSE AND METHODOLOGY 4.1 Evaluation Purpose The primary purpose of this final performance evaluation is to determine whether the assistance provided by USAID/Nigeria through the Agricultural Policy Activity achieved the stated development objectives and to understand the lessons learned from this particular project in Nigeria, including the findings and recommendations of the midterm performance evaluation. It is intended to provide an independent examination of the overall progress and accomplishments of the activities of this project. The evaluation provides a detailed picture of the major accomplishments and weaknesses of NAPP since its inception and determine how its successes can be sustained. This final performance evaluation will provide USAID/Nigeria, its implementing partners, and agriculture sector stakeholders with data on outcomes and achievements. The evaluation also elucidates lessons learned and includes specific recommendations to USAID/Nigeria on how project interventions can be sustained and scaled by other FTF activities, partners, private sector firms, or by the GON. 4.1.1 Evaluation Questions In assessing the achievements of the NAPP objectives, the following key evaluation questions are to be answered. 1. Gender/Youth: To what extent have women and youth (15–29 years) benefited from policy intervention undertaken by the activity? ♦ Which approaches have been most effective in reaching women and youth? ♦ How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? ♦ How have women and youth been part of the dialogue in the agriculture sector in developing national policy systems? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? 2. To what extent has the direct training of select graduate students in Nigerian universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? ♦ How have policy issues chosen by the activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? 3. To what extent has the capacity of local research partners been strengthened to inform policy debates and implementation with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? 19 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ♦ How well are local research partners providing needed support to relevant state ministries in the formulation and implementation of policy processes? ♦ How often do policy actors interact to advance policy processes at national and state levels? 4. To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? 5. What measures has the activity put in place to ensure: ♦ Sustainability of the interventions and the changing needs of the policy environment at the national/state levels? ♦ Policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs? 4.2 Evaluation Methodology The ET utilized a mixed-methods approach, quantitative and qualitative research methods, which involved the use of primary and secondary data to undertake a thorough and effective final performance evaluation. The evaluation was carried out in three stages (Figure 3). 1. Desk reviews, during which the ET carefully reviewed the project’s background documents to develop data collection protocols; 2. Primary data collection, consisting of key informant interviews (KIIs), focus group discussions (FGDs), and telephone interviews, during which virtual interviews were conducted with the activity funding agency, implementation team, stakeholders, and beneficiaries to gather necessary data; 3. Analysis and report writing, wherein the ET analyzed secondary and primary data collected and developed the draft report. Figure 3: Three-stage Evaluation Methodology 20 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 4.2.1 Primary Data Collection Primary data collection took place from August 24 through September 25, 2020. The procedures for primary data collection were twofold. ▪ Virtual meetings were set up with all categories of stakeholders during which the required data were collected through FGDs and KIIs, based on a pre-arranged meeting schedule. Categories of respondents with whom these meetings were held included USAID/Nigeria, implementing partners (MSU, IFPRI), National Advisory Committee, civil society organizations (CSOs)/associations, project scholars, scholars’ advisors, the media, FMARD, and state partners (universities and Ministries from Ebonyi, Delta, Kebbi, Kaduna, Oyo, Cross River, Niger, and Benue States). ▪ Structured questionnaires were used to elicit information from project beneficiaries who participated in training workshops and conferences. This was done to gather information regarding the extent to which the outcomes of the project have been achieved in line with relevant evaluation questions. The questionnaires were administered via telephone. To achieve this, DevTech engaged six experienced enumerators to conduct the telephone interviews, using the questionnaire, following a short period of training. The ET broke up into two groups to carry out the FGD/Virtual meetings at the federal level and across the seven states. DevTech provided each group with necessary technical and administrative support by setting up and managing the meetings. 4.2.1 Sampling Strategy The beneficiary database consisted of 6,610 activity participants stratified into 80 key informants, 431 conference participants and 6,099 training workshop participants (Figure 4). Each category of participants was further stratified by gender with key informants consisting of 58 males and 22 females, while the conference participants comprised 344 males and 87 females. The trainers’ category was the largest, consisting of 4,521 males and 1,578 females (Table 2). A confidence level of 95 percent and 5 percent margin of error was used to select a sample size of 67 key informants, 204 conference participants, and 362 training participants, making a total of 633 respondents included in evaluation interviews (Figure 4). This represents about 10 percent of the total population of activity participants in the database. The sample size for respondents was calculated using RaoSoft Software.2 This statistical approach was used to ensure a random distribution of the sample frame across all targeted states and allow the study to generalize its results to the population. The Excel-based random-sampling procedure was used to select the sample of respondents. 2 http://www.raosoft.com/samplesize.html 21 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Figure 4: Sampling Strategy Table 2: Sample Size by Gender and Category of Participant Gender Beneficiaries in Activity Database Percent Of Total Sample Size* Category (a) Project Partners/Key informants Male 58 72.5 49 Female 22 27.5 18 Total 80 100.00 67 Category (b) Conference Participants Male 344 79.81 163 Female 87 20.19 41 Total 431 100.00 204 Category (C) Training Participants Male 4521 74.13 268 Female 1578 25.87 94 Total 6099 100.00 362 *Based on the 95% confidence interval and 5% margin of error 4.2.2 Sample Selection and Composition of Partners for Key Informant Interviews A total of 362 training beneficiaries comprising 132 from government, 108 from academia, 41 from media, 30 from NGOs and associations, 29 from the private sector, and 22 others, predominantly retired civil servants and those with unspecified affiliation and designation in the database. These six categories represent 36.46, 29.83, 11.33, 8.29, 8.01 and 6.08 percent of the total respondents respectively (Figure 5), seeming to indicate that the highest proportion of beneficiaries is from the government. On the other hand, the total number of interviewees is made up of 46 from academia, 41 from government, 20 from 22 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION the private sector, 17 from NGOs and associations, 15 from the media, and 13 others, representing 30.26, 26.97, 13.16, 11.18, 9.87 and 8.56 percent respectively, implying that academia has the highest proportion (Figure 6). Figure 5: Distribution of Training Beneficiaries by Category of Stakeholders Figure 6: Distribution of Conference Beneficiaries by Category of Stakeholders 4.3 Desk Review Various documents and reports were reviewed at the initial stage of the evaluation (July 20–24, 2020) to enrich the evaluation process technically. The documents reviewed included original project agreement and amendments; USAID/Nigeria strategy document; Feed the Future Project Appraisal Document; activity annual work plans; quarterly and annual progress reports; monitoring and evaluation plan; data 0 20 40 60 80 100 120 140 Distribution of Training Beneficiaries by Categories Number % of total 0 5 10 15 20 25 30 35 40 45 50 Distribution of Conference Beneficiaries by Categories Number % of total 23 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION quality assessment reports; activity technical studies; midterm evaluation report; and other relevant documents such as the specification of the activity deliverables to be provided by USAID/Nigeria and MSU/IFPRI. 4.3.1 Focus of Document Review The ET endeavored to address the following issues in the review: 1. Categorize the available documents and identify gaps. 2. Review the NAPP implementation plan, including the periodic revisions, and determine whether the activities therein were consistent with the evaluation questions. 3. Determine whether the project outcomes/impacts/benefits were adequately captured in the quarterly and annual reports and appropriately reflected in the evaluation questions. 4. Indicate whether the evaluation time frame presented in the SOW document was consistent with the contract documents and prevailing circumstances, especially regarding timing and scheduling of activities. 5. Identify the relevant categories of respondents to be involved in the evaluation for the purpose of data beneficiaries as indicated in various documents. 6. Determine on the basis of 5 above whether the category of respondents to be covered as indicated in the scope of work document is adequate. Indicate other relevant category of respondents if necessary. 4.4 Key Informant Interviews The Team conducted interviews and FGDs with a variety of stakeholders including USAID staff, activity beneficiaries, government staff, implementing partner staff, and other key donor partners. The required data were collected using KIIs. There were 10 main groups of key informants for this evaluation: 1. USAID/Nigeria Staff 2. Implementing partner (MSU and IFPRI) staff 3. National Advisory Committee Members 4. Government partners (Federal includes FMARD, while state partners are drawn from Ebonyi, Oyo, Kebbi, Benue, Delta, Niger, Kaduna and Cross River States) 5. Research collaborators 6. Project scholars from universities involved in the activity in USAID/Nigeria’s focus states 7. Project scholars’ advisors 8. Representatives of CSOs/associations 9. Representatives of the private sector 10. Representatives of the media 24 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 4.5 Mapping of Evaluation Questions with the Category of Stakeholders The five evaluation questions were addressed in accordance with relevance of stakeholders to aspects of the Agricultural Policy Project activity (Table 3). Research instruments were developed to drill the interview exercise down to necessary sub-questions as shown in the collection protocols, based on correspondence between the evaluation issue and category of stakeholders. Table 3: Evaluation Questions and Target Respondents Evaluation Questions Relevant Stakeholders 1 Gender/youth: To what extent have women and youth (15–29 years) benefited from policy intervention undertaken by the activity? ● Which approaches have been most effective in reaching women and youth? ● How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? ● How have women and youth been part of the dialogue in the agriculture sector in developing national policy system? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? USAID MSU IFPRI Advisory Committee members Partner States Project scholars Scholars’ advisors Training participants 2 To what extent have the direct training of select graduate students in Nigerian Universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? ● How have policy issues chosen by the Activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? MSU IFPRI FMARD Partner States Project Scholars Scholars’ Advisors Training Participants Activity Advisory Committee 3 To what extent has the capacity of local research partners been strengthened to inform policy debates with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? How well are local research partners providing needed support to relevant state ministries in the formulation and implementation of policy processes? ● How often do policy actors interact to advance policy processes at national and state levels? Research Collaborators (including Project Scholars and Scholars’ Advisors) MSU, IFPRI, FMARD Partner States Activity Advisory Committee 4. To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? Representative of Media MSU, IFPRI, FMARD Partner States 5 What measures has the activity put in place to ensure: ● Sustainability of the interventions and the changing needs of the policy environment in the national/state levels? USAID MSU/IFPRI FMARD 25 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ● Policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs? Partner States Other Donors 4.6 Data Analysis A mixed-methods approach was used for the analysis of primary data, summarizing interview and group discussion notes in line with the research questions (this served as the framework for the analysis). The data generated from the KIIs were coded and analyzed using ATLAS.ti, a qualitative statistical software program, and analyzed in stages: ▪ First, interview guide questions were mapped to evaluation questions with an intent of aggregating and synthesizing results to answer the research questions. ▪ Second, themes from the responses to the interview questions were generated and coded. ▪ Third, the codes were synthesized and matched to the themes, with aggregation of similar themes thereafter. ▪ Finally, networks that capture the themes, evaluation questions, and key components of the analysis were synthesized for visual representation of the results. The primary data obtained through telephone interviews were coded and subjected to descriptive analysis (means, percentages, and cross tabulations) and graphical illustrations of the type of benefits, relevance, utility, and effects. The variation in information supplied by respondents across groups of stakeholders was also tabulated and described to assess the project’s benefits and the different ways they are distributed among participants. To ensure this, questions posed to proposed respondents were grouped by the relevant evaluation questions. The analytical approach for each evaluation question is highlighted in Table 4. 26 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Table 4: Analytical Approach and Data Collection Sources to Answer Each Evaluation Question Evaluation Question Analytical Approach Data Collection Source 1. Gender/youth: To what extent have women and youth (15–29 years) benefited from policy intervention undertaken by the activity? ● Which approaches have been most effective in reaching women and youth? ● How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? ● How have women and youth been part of the dialogue in the agriculture sector in developing national policy system? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? Both qualitative and quantitative analytical tools used. The ways in which women have benefited from national research capacity strengthening will be analyzed qualitatively. Quantitative analysis of project implementation performance and extent to which project objectives have been achieved was used. Some of the data obtained were subjected to descriptive analysis using suitable descriptive statistics (e.g., mean, percentages, etc.) focusing on performance indicators as variables and the proportion of respondents attesting to the relevance of the activities and the benefits derived from the interventions. The way the information supplied by respondents varied across groups of stakeholders was also tabulated and described as a way of assessing the project’s benefits and the different ways they are distributed among participants. Document review; KIIs with MSU/IFPRI, government (federal and state), private sector, media, CSOs, project scholars, scholars’ advisors, training and conference beneficiaries, and Activity Advisory Committee members 2. To what extent have the direct training of select graduate students in Nigerian Universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? ● How have policy issues chosen by the activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? Qualitative analysis of the improvement in capacity to plan and implement effective policy analyses as well as improvement in policy demand and utilization was analyzed qualitatively. Quantitative analysis of the benefits derived from the training programs including variations in the responses from different categories of stakeholders Document review; KIIs with MSU/IFPRI, government (federal and state), private sector, media, CSOs, project scholars, scholars’ advisors, training and conference beneficiaries, and Activity Advisory Committee members 3. To what extent has the capacity of local research partners been strengthened to inform policy debates with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? ●How well are local research partners providing needed support to relevant State Ministries in the formulation and implementation of policy processes? ●How often do policy actors interact to advance policy processes at national and state levels? Qualitative analysis of improvement in policy research capacity of next generation of researchers Quantitative analysis of the variations in the responses from different categories of stakeholders Document review; KIIs with MSU/IFPRI, government (federal and state), project scholars, scholars’ advisors, and Activity Advisory Committee members 4. To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? Qualitative analysis of improved capacity for policy engagement and communication Document review; KIIs with MSU/IFPRI, media 27 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 5. What measures has the Activity put in place to ensure: ● Sustainability of the interventions and the changing needs of the policy environment in the national/state levels? ● Policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs? Comparison of sustainability measures/strategy developed by the activity with what has been implemented Comparison stakeholders’ sustainability options with reality and with a view to determining how the desired change will be achieved and sustained Document review, Interviews with activity staff; KIIs with government, and other stakeholders on sustainability of the activity results 4.7 Quality Control Mechanism for Data Gathering, Analysis and Oversight Both primary and secondary data were collected for this evaluation. The secondary data were obtained from the activity, activity beneficiaries and periodic reports (e.g., quarterly and annual reports). For ease of access, formal letters requesting such data were obtained from USAID/Nigeria and MSU/IFPRI, via DevTech. Primary data were obtained through KIIs and FGDs using virtual platforms: Google Meet (primary mode), WhatsApp, and via telephone calls if respondents had issues with internet connectivity. As a quality control mechanism, the ET conducted all KIIs and FGDs. Notes were taken by members of the ET during the interviews on emerging key issues relating specifically to the evaluation questions and sub-questions, with the respondents’ consent. This enabled fact-checking during subsequent stages of the evaluation. 4.8 Guiding Principles and Values Ethical considerations are of the utmost importance and the evaluation process ensured that the evaluation was conducted with integrity and sensitivity, showing the highest level of respect to all participants. Informed consent was obtained from all respondents prior to the data collection process. In reporting, respondents were referred to by role/job title and location, if necessary, but only where this information did not reveal the identity of the respondent(s). The ET adhered to the “Do No Harm” principle, placing the protection of human subjects as its highest priority. The ET ensured the confidentiality of data and the respect for the privacy of all individuals concerned, and has made all data collected available to DevTech and USAID in a usable format. 4.9 Study Limitations and Remedial Measures The ET used a qualitative approach, which is widely accepted in conducting performance evaluations; however, there are concerns of biases due to the variations in memory recall, especially from the beneficiaries of the capacity-building component. Various attitudes and emotions may have interfered with an assessment of the activity’s implementation. Also, the behavior and body language of respondents could not be observed during virtual and telephone interviews, particularly for the capacity-building beneficiaries. It is important to note, however, that this did not have any impact on the quality of the data collected, as the only difference between virtual/ telephone data collection and in-person interviews, in addition to the inability to viewing body cues, is physically meetings with respondents. In addition, the definition of beneficiaries as indicated in the implementer beneficiary database imposed considerable challenges to the time and cost of securing responses and obtaining a high respondent response rate. For example, during some interviews the ET discovered that the category of beneficiaries 28 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION as described by the database was wrong, or that some people who provided services during events (such as media coverage/photography) had been wrongly labeled as beneficiaries. To address these limitations: ▪ The ET members personally handled interviews with key informants and assigned the responsibility of copious and diligent note-taking among themselves. ▪ Responses were triangulated among members of same category of respondents placed on different schedules for the virtual meetings. ▪ Follow-up calls were made to validate information. ▪ Information updates were sought and received where necessary using social media channels. ▪ Triangulation was achieved also for respondents from different categories (such as scholars and their advisors, as well as state ministries and research collaborators). SECTION 5. FINDINGS AND CONCLUSIONS 5.1 EVALUATION QUESTION 1 (EQ1) EQ1: Gender/Youth – To what extent have women and youth (15-29 years) benefited from policy intervention undertaken by the activity? 5.1.1 EQ1: Findings Women and youth have benefited to a great extent. Scholars reported that they participated in additional trainings both at MSU and in Nigeria as well as core training at MSU, including on analytical software such as Stata and R. Beneficiaries reported that they have been able to apply the learnings, including new approaches, leading to improvements in the way they undertake their research or work. Another reported benefit was building relationships and networks during collaborative research. Scholars also went on to train more women and youths on analytical methods using R statistics in universities and ministries in some of the FTF focal states and, in some cases, stepped down the trainings they received while at MSU to faculty members and students in their respective schools. They noted that the intervention is very relevant to the career needs of women and youth who took part in the training and that some of the beneficiaries are using the knowledge acquired to conduct research and data analysis for responsibilities assigned to them. Collaborators from Delta State noted that “the training on policy communication and policy advocacy involved small-scale farmers, farmers’ business groups, and various groups where a lot of them were female.” At Ibrahim Badamasi Babangida University, Professor Folorunsho Lawal said, “we aimed for a minimum of 30 percent female participation in NAPP activities”; similarly, a respondent from KSUST said the ratio of female to male attendance was typically 3:7 in Kebbi State. In Ebonyi State, on the other hand, research collaborators said young academics (graduate assistants and lecturers II) were encouraged to attend the trainings and that FGDs were used as a strategy to ensure there was adequate participation of youth and women in NAPP interventions. Participation by women in NAPP activities appeared to be a challenge for the relevant northern MDAs. Although Benue MANR reported that 40 percent of people trained in the ministry and college of agriculture were women; they said this was only possible because there was a deliberate effort to ensure 29 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION this proportion of female participation. Similarly, Kaduna and Kebbi MANR reported challenges with female participation. Kaduna MANR reported that for the monitoring and evaluation (M&E) training, “there are not many women in the M&E department; we had to deliberately target women for inclusion in the training by including women from other departments and agencies.” Kebbi MANR reported that female participation was not “up to the level required by NAPP.” Across both regions (northern and southern state MDA partners) there appear to be more women at the farmer/Agricultural Development Program (ADP) levels. State partners reported that NAPP trainings helped to strengthen the capacity of women in research in the MDAs. The Cross River MANR reported that “in our draft for the agricultural policy, the director of agricultural services was a woman, while two of the stakeholders who participated in the policy review, the directors of fisheries and veterinary, were women.” In addition, the Delta MANR reported that the trainings have helped to sharpen their understanding of data analysis, using data to interpret policy and making policy statements. Ebonyi State MANR reported that, since the ministry is significantly involved in survey work, relevant NAPP trainings have enriched the survey skills of women which can help them advance their careers in the ministry. By and large, the results show that the majority 370 (72 percent) of the 514 total respondents were male, while 142 (28 percent) of them were female. This suggests that gender balance is still a work in progress in the project’s focal states. This can be confirmed from quarterly reports from Year 2 Quarter 1 to Year 5 Quarter 3, where fewer women than men participated in different NAPP activities. As evidenced in the reports, however, the project emphasized female participation/representation in all activities. Most universities, CSOs, and institutions had fewer female staff to represent them in the first place, however— hence, the activity’s keen interest in gender balance did not materialize. In cases where NAPP could influence selection, more women participated, but few such cases existed. With regard to training (workshop) activities in particular, there were more than twice as many male attendees as female (Table 5). Female participation is not up to the standard requirement for gender balance, however, the numbers do reflect an improvement in what used to be the case; NAPP encouraged and emphasized gender balance where possible. Table 5: Distribution of Participants in NAPP Workshops by Gender and Type of Training Training/Workshop Attended Sex Software Testing Policy Training Communication Training Monitoring and Evaluation Agricultural Development and Nutrition Group Dynamics Total Female 32 46 33 28 53 10 202 Male 99 139 84 76 132 35 565 Total 131 185 117 104 185 45 767 Source: Authors’ computation Irrespective of the level of gender participation, the ET found no statistically significant difference in the experience of the beneficiaries of the NAPP’s training activities. This implies that the experience of the participants had nothing to do with their gender and the NAPP activity made useful progress towards gender inclusiveness. 30 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 5.2 EVALUATION QUESTION 1(a) EQ1(a): Which approaches have been most effective in reaching women and youth? 5.2.1 EQ1(a): Findings Capacity-building trainings in the scholar program have been most effective in reaching women and youths. The scholar program provides scholars and their advisors with opportunities for learning new approaches, sharing ideas, presenting, and obtaining feedback on their work; it also offers the added opportunity to apply these learnings to their research work or—in the case of the advising professors—incorporate learnings in their departments. It was reported that the program helps push scholars out of their comfort zones and enables them try and do things they wouldn’t ordinarily do. The scholar program had a remarkable effect on beneficiaries compared to other activities. Beneficiaries in this area have been able to apply the trainings, including new approaches and widened research networks, and these have led to improvements in the way they undertake their research or work. The scholar’s advisor at the University of Nigeria (UNN) noted that a female scholar from UNN has had a very high impact on students through her teaching and interactions after attending an NAPP program. The advisor said, “after her training at MSU, I made sure that she was assigned to teach here so that the knowledge and skills acquired can [impact] others. Unfortunately, it is just her. I know that she has gained a lot, and the department can testify that some value has been added to the department with her here.” The UNN scholar’s advisor went on to say, “I am also a female, and I have benefitted a lot from this programme through peer interactions, collaborations (like organization joint training, fieldwork, and so on), publications. But it is just two of us in Enugu, and this is too small.” With respect to youth participation, the scholars’ training program involved young researchers, young academicians, and graduate students. Apart from these categories of participants, however, the coverage of youths is rather limited. As seen in the figure below, the respondents' average age is 43 years with a standard deviation of 10.6 years. Moreover, 45 percent of participants fall between 20 and 40 years, while 55 percent are even older; they cannot be regarded as youths (see Figure 7). Figure 7: Distribution of Respondents by Age 12.7% 32.9% 29.4% 20.4% 4.3% 0.4% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 20-30 31-40 41-50 51-60 61-70 Above 70 Distribution of Respondents by Age Mean = 43.0 31 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 5.3 EVALUATION QUESTION 1(b) EQ1(b): How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? 5.3.1 EQ1(b): Findings With funding from NAPP, the Agricultural Policy Research Network (APRnet) reportedly gave young scholars scholarships and organized conferences with the agricultural professional society and media to build synergy between the two groups. They ensured the events were attended by both male and females. For one of the conferences in Abuja, 45 percent of the panelists were female. One of the papers presented and discussed by the panelists at the conference was a paper on incentivizing youths. In addition, APRnet has published journals with funding from NAPP in collaboration with IFPRI and other editors; the journals are a platform for all (including youths and women) to publish and disseminate research findings. Eight journals have been produced by APRNet since 2016 and have included issues on Gender. In addition, APRnet engaged stakeholders in the FTF focal states to conduct workshops. In Delta State, after a training on applying for research grants, APRnet young scholars won a research grant from the Global Development Network to develop a paper on “The Role of Aid in Agricultural Development in Africa,” with a focus on Nigeria. 5.4 EVALUATION QUESTION 1(c) EQ1(c): How have women and youth been part of the dialogue in the agriculture sector in developing national policy systems ? 5.4.1 EQ1(c): Findings One scholar noted that youth and female students who benefited from the program are yet to participate in policy dialogue, but she developed a policy brief of her gender-focused research. This was then presented during one of the sessions at the 2018 NAAE conference, as well as uploaded on the FTF website and shared on ResearchGate platform where her work has been read over 300 times. A young scholar from Kogi State said that he has participated in a policy dialogue platform organized by a group of non-NAPP scholars that state, and through this platform has participated in policy debates on agricultural insurance and agricultural finance. A scholar from Federal University of Agriculture, Abeokuta (FUNAAB) said she was involved in policy debates on Cassava: Adding Value for Africa (C:AVA) II projects. “I do more policy debates now, having been exposed to many types of training under NAPP.” Furthermore, the project scholar in UNN said, after scaling her research to the community level, that one of the things she learned from it was that transitional and multidisciplinary research approaches are needed to inform policy; the NAPP program has helped her to modify her approach to inform policy. 5.5 EVALUATION QUESTION 1(d) EQ1(d): Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? 32 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 5.5.1 EQ1(d): Findings One example of gender-focused research was carried out by an M.Sc. scholar (Gender Dynamics in Agglomeration Economies: A Case Study of Rice Processing Clusters in Kano State, Nigeria) which targeted parboilers rice processing clusters. Parboiling is typically a woman-dominated occupation, but she found that men and youth were also active in that space. Another example is a study by an M.Sc. scholar which examined climate variables that primarily affect youth and women. According to her, “Basically, my work looked at how women are disadvantaged by climate issues. It examined how climate variables affect women in Northern and southern Nigeria.” Examples of other work undertaken by these scholars include the provision to communities, particularly women, of a pamphlet on dietary diversity (using available foods to make nutrient-rich diets) developed by a Ph.D. scholar, and training on the Landpks app delivered to farmers for on-the-spot soil tests by another Ph.D. scholar in Benue State. Research collaborators noted that young academics were involved in data collection as enumerators and that specific research was focused on areas relevant to women and youth, such as farming productivity, where these populations are disadvantaged. The collaborators in Ebonyi State noted that “[the] Aflatoxin dissemination and training program was focused mainly on women since women were end-users. This program really helped women maintain household nutrition/health.” There was also an NAPP training on writing publishable articles which was organized specifically for UNN, involving all young academics in the faculty of agriculture. Also, at UNN, the training on research grant writing targeted female academics and they were particularly encouraged to participate. One respondent said, “there was room for encouraging women in the research. Whenever there was limited space, women came first in participation. This was to ensure that the capacity of women in research is strengthened and to increase the 30 percent representation of women in academics.” 5.6 EQ1: CONCLUSIONS ▪ Women have benefitted from the policy interventions undertaken by the program, but their participation was at a disproportionately lower level than that of men. NAPP activities were open to all; there was no discrimination against women and youth. However, as one advisor pointed out, “The challenge is that female participation in this area, that is the agriculture policy sector, is low, so it depends on the number of women who apply.” ▪ Although all NAPP activities were relevant to the career paths of the beneficiaries, training was the most beneficial of all the activities. ▪ The scholar program has had a remarkable effect on beneficiaries compared to other activities. Beneficiaries of the scholar program have been able to apply their knowledge, including new approaches to undertaking research, and these approaches have led to improvements in their policy research. ▪ At the MDA level, female participation was promoted, but due to low numbers of females in relevant departments in the ministry, a deliberate effort to ensure females participation in the training program was needed. (The existing gender policy in some states promoted adequate participation of women in NAPP training activities.) 33 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ▪ Replacement of Stata with R for data analysis has been very effective because access to the latter is free of charge. ▪ Improvement in the capacity to conduct policy research due to the training received by beneficiaries under NAPP is not gender-related. There is no significant difference in the strength of male and female participants’ ability to conduct policy research (Annex 1). ▪ The gender-friendly posture of NAPP was revealed by the experience of the participants in capacity-building activities. There is no significant difference in the experience of male and female beneficiaries of the project’s capacity-building activities (Annex 1). ▪ The improvement in the capacity of beneficiaries to conduct policy research is statistically significantly and different across age groups. Improvement is greater between the ages of 31 and 50, than either those of 51 years and above, or those between 20 and 30 years of age (Annex 1). 5.7 EQ1: RECOMMENDATIONS ▪ Policy makers should be encouraged to adhere to the criteria of female representation in nominating candidates to participate in NAPP activities. ▪ Deliberately targeting female attendance in activity events is critical to ensuring the participation of women in areas where there is a general dominance of male participants. Future programs should, therefore, consider adopting minimum participation levels for females in their interventions. Guidelines to ensure female participation, specifying the percentage of females required to attend program events, should be communicated to stakeholders. 5.8 EVALUATION QUESTION 2 (EQ2) EQ2: To what extent has the direct training of select graduate students in Nigerian universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? 5.8.1 EQ2: Findings Strengthened capacity in the next generation is evident to a large extent, as is discussed below. There is a greater involvement in policy research. Post-training activities showed a majority (58 percent) of beneficiaries are now more involved in policy research activities. Other beneficiaries not involved in policy research have been applying their new skills to other areas, such as communicating the policies to farmers and becoming involved in policy debates/dialogues (Figure 8). 34 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Figure 8: Involvement of Training Beneficiaries in Policy Research There are improvements in various types of skills for conducting policy research. The NAPP training activities benefited participants most especially in terms of an improved capacity to carry out data analysis (42 percent), followed by dissemination of information (18 percent), report writing, (13 percent) improvement in farming practices (13 percent), and policy brief writing (11percent) (Figure 9). This implies increased capacity through new skills acquisition in critical areas like data analysis, research writing, and policy brief writing. The trainings improved the proficiency of scholars in statistical analysis as well as the proficiency of other researchers trained on R. One scholar from Benue State noted that before she joined the program, she was not proficient in Excel and gave out her data to be analyzed by other people. Initially, she did not want to do the statistical analysis training at MSU. Since she was trained on Stata and R statistics by NAPP, however, she does her own research analysis, can better understand her data, and can clearly inform decision-making. Being adept in research analysis also makes her eager to write papers; she does research now with “problem-solving” in mind. She now notes that she wants to know how her research impacts the common farmer with 1.0 ha of land. She has trained several people in R statistics. A scholar from Kogi State said that prior to NAPP, he did not know how to use Stata. He reports that Stata trainings attended in Nigeria were very useful because all his analysis in MSU were done using Stata, and he now confidently teaches others in Stata. Yet another scholar said trainings on R have strengthened the capacity of next-generation policy researchers. Feedback he received from one student trained in R was that the participant could now afford to do his own research analysis, because R is free. That participant previously used to outsource his data analysis. The scholar from Kogi State said he now has a better understanding of research, including analytical tools, linking research to agricultural theories, and developing a problem statement. He said there is a stark difference in the quality of the articles he published before and after his engagements with NAPP/scholar program. He has published articles in Tertiary Education Trust Fund (TETFUND)-sponsored journals and at Nasarawa State University. He said 58% 42% Post-Training Involvement in Policy Research Activities Yes No 35 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION his capacity to contribute to policy issues has been strengthened in the area of evidence-based research. Findings from his study in Kebbi State revealed that farmers’ participation in insurance is low because of the type of product offered by insurance companies and because of low awareness of insurance among farmers. The report was submitted to NAPP in 2019, as well as to the Nigerian Agricultural Insurance Scheme (NAIC). With respect to dissemination, a scholar in Kaduna State recounted her experiences as part of a team that trained university students on R statistics software; she is currently working with two female students, teaching them using the research methodology she learnt from MSU. She is developing a manual on cost￾benefit analysis (CBA) which can be used by researchers and has plans to conduct a training on CBA. She believes that NAPP activities have impacted the agricultural sector. For example, there was a glut in onions in Niger state in 2019, so a policy brief was developed on how to preserve onions. There is also a plan to carry out trainings and a feasibility study for the Federal Ministry of Women Affairs on the best way to utilize land. Figure 9: Application of Skills Acquired Under NAPP Training There has also been influence on the decision of participants to conduct policy research in the future. The training activities under NAPP have directly influenced the majority of the beneficiaries (59 percent) to get involved in policy research. This is consistent with the findings that showed 58 percent of beneficiaries are now involved in the policy research. However, nine percent of the beneficiaries reported that the training had no influence at all on their decision to conduct policy research in the future, while 32 percent found the training attended to have a little influence of their desire to conduct policy research (Figure 10). 5.9 EVALUATION QUESTION 2(a) EQ2(a): How have policy issues chosen by the activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? 42% 18% 13% 13% 11% 3% 0.5% Usefulness of Training 36 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 5.9.1 EQ2(a): Findings NAPP activities, the scholar program, training events, conferences, etc. have helped increase the exposure of the next generation of Nigerian researchers to new approaches, such as the improved teaching methods learnt by scholars’ advisors at MSU, and the adoption of standard analytical tools like Stata. In addition, involvement in NAPP activities has enabled these researchers to interact with policy makers to provide support in the policy process. For example, a scholar’s advisor in Kaduna State noted that the MSU visit afforded advisors the opportunity to acquire new teaching methods that they brought to bear on how they supervise their graduate students, particularly in the area of research methodology. In a similar vein, the UNN scholar’s advisor said she learned new ways to teach, to carry out research, to write research grant proposals, and to supervise students, ways that she is effectively applying with success. A majority of beneficiaries stated that the issues chosen significantly improved their capacity to plan and implement policy analysis. A respondent had this to say: Comparing now (after participating in NAPP) and before I had this opportunity, my understanding of policy debate has increased and my experience and research knowledge has improved, e.g., how to design questionnaires, how to conduct survey interviews to obtain quality data and how best to conduct research. NAPP activities (e.g., the scholar program, trainings, conferences,) have helped increase the exposure of the next generation of Nigerian researchers to new approaches. These include improved teaching methods learned by scholars’ advisors at MSU and the use of standard analytical tools like Stata. In addition, involvement in NAPP activities has enabled these researchers to interact with policy makers to provide support in the policy process. For example, a scholar’s advisor in Kaduna State noted that the MSU visit afforded advisors the opportunity to bring new knowledge to bear on how they supervise their graduate students, particularly in the area of research methodology. A UNN scholar’s advisor said that she learned new ways to teach, do research, write research grants, and supervise students that she is now successfully applying. She explained that her experience at MSU was interactive and the content was quite rich: Presentation of research work was done, and we shared how we carry out research here in Nigeria with them. One-on-one interaction with colleagues was done where we shared notes and learned a great deal from one another. Now I have meetings with my students and we share thoughts regularly on their work, which is a helpful approach I learned from MSU. She went further to say her interactions with colleagues who went through the program indicated that they also learned a lot and went home with value added to teaching, research, and supervision. “The soil scientist from the University of Agriculture Makurdi went home with laboratory equipment from MSU. We still exchange ideas on research experiences.” Similarly, the scholar’s adviser from Kogi State said his participation in the scholar program helped him to establish links and new networks. He said MSU is one of the top centers for agricultural research in the world and he interacted with faculty members at MSU with extensive experience in agricultural research. He also acquired new skills in applying for research grants. Subsequent to his time at MSU, he led the Nigerian team that went around universities and ministries across the seven FTF focal states to train students and staff on Stata and statistical analysis. He is also part of the technical team working with Kebbi State to review the state agricultural road map, transforming it to the state agricultural policy document. Additionally, he has applied the knowledge gained at MSU to prepare his course material on agribusiness. 37 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION The research collaborator in Kebbi State noted that trainings were conducted there for ministry staff, university staff and students, CSOs, and farmers’ groups. Also, a mix of farmers (small-scale to large-scale), through their group representatives, were trained on topics such as group dynamics, effect of climate change, land access, and agricultural policy. Regarding changes in the capacity of people trained, feedback from participant staff at the ministry who were trained on the development of policy briefs indicated that it has improved their performance at a policy level. In regard to researchers, the collaborators noted that policy issues chosen triggered research ideas and revealed different sources of data. For example, one researcher said that participation in the climate-change training has made him start to conduct research on climate change. Figure 10: Effect of Training on Decision to Conduct Policy Research The collaborators reported that research findings were adapted to local communities. For example, the Ebonyi State collaborators said research findings were often presented to both academic and non￾academic bodies. For the non-scientific audience, key findings were translated to local dialects and disseminated through pamphlets to ensure understanding. Collaborators reported that NAPP activities in the focal states have been instrumental in improving states MANRs’ capacity to participate in the policy research. Following NAPP trainings on policy briefs and engagements with the ministries, Benue MANR used the knowledge to write a policy brief to the Commissioner for the adoption of an agricultural policy for the state; NAPP’s engagements with the Cross 9% 32% 59% Influence of Training on Decision to Conduct Policy Research in the Future No influence at all of little influence Highly influential 38 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION River MANR led to the revision to their state Agricultural Policy. Ministries also reported the development of specific policy briefs aimed at developing key sectors of the economy because of the trainings. For example, the Niger MANR reported that they had developed policy briefs for rice and cowpea, while Kaduna MANR reported a development of a draft policy brief targeted at improving the state’s participation in the small ruminant sector. The MDAs reported improved capacities in their operations. Delta MANR reported that the use of graphics in report writing and inferential analysis were skills gained from NAPP. Benue MANR reported that trainings delivered by NAPP to the ministry have facilitated the collection of key economic indicators needed for the analysis of the livelihoods of people in Benue State, which is required for the annual food and nutrition security policy metrics computed for Nigeria. Benue is one of the 16 states and the FCT required to supply this information. Cross River MANR reported that although they do not conduct research, they are now able to commission policy research because NAPP trainings had helped ministry staff better recognize the need for an evidence-based research. The Cross River MANR also noted that “NAPP reviewed and updated our Agricultural policy and it was a rewarding experience. The staff at the Ministry that took part in the capacity building program were encouraged to work more closely with the consultants in academia in developing the revised Agricultural policy for the State”. Ebonyi State MANR reported that “there is a better understanding on the need to consider the policy beneficiaries in the formulation process in order to address the real problem.” On the other hand, Kebbi MANR reported that the conflict-resolution training led the MANR to apply a mechanism for interaction between farmers and herders which enabled the groups to become co-dependent. In addition to the enhanced capacity of the academia and MDAs through various capacity building programs, the MDAs from Delta and Ebonyi worked with the University scholars and noted that they would welcome future collaboration. However, as rewarding as the collaboration was, there is no evidence of any formal agreement being signed between the parties. In addition, the Activity supported FMARD in their response to the COVID 19 pandemic, as it relates to agriculture and food security. For example, as a member of the Agricultural Donor Working Group (ADWG), the Activity co-led the technical preparation of the policy brief for FMARD, aimed at supporting the agriculture policy dialogue and review of the Ministry in December 2019. Following this, along with other development partners, the Activity undertook a COVID–19 risk analysis and policy brief on its secondary impacts on the agricultural sector, to serve as the context within which the group will support the GON in its resolve to grow the economy post-COVID-19, from the agriculture lens. Furthermore, the Activity conducted an impact analysis of the COVID-19 pandemic, as a complementary study. in March 2020, to estimate the direct and indirect impact of the pandemic on the Nigerian economy. The study is intended to feed into the FMARD-led assessment and policy formulation for the agriculture sector. Also, in partnership with WorldFish, the Activity launched a study in May 2020, on COVID-19 and the fish and poultry value chains in Nigeria. The study is aimed at guiding the direction of policy and donor support to these key value chains in Nigeria in a post- COVID-19 era. NAPP scholars, on the other hand, launched a virtual information series in April 2020, to provide technical information on COVID-19 and its impact on several sectors of Nigeria’s economy, particularly agriculture and food security. The series focused primarily on agriculture and included expert discussions on food security, nutrition, climate change, agricultural research, gender and livelihoods. 39 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION An overall majority (62 percent) of the beneficiaries stated that the training had improved their capacity to conduct policy research, an additional 27 percent indicated much improvement. Only 11 percent claimed their capacity had remained unchanged (Figure 11). Indeed, before the training, many beneficiaries had little or no knowledge of econometric tools such as STATA and R, but as a result of the training, they can now independently conduct policy analysis. Figure 11: Extent of Improvement in Capacity to Conduct Policy Research under NAPP Training Almost half of trainees, 48 percent, stated that acquisition of new technical skills was the most useful achievement they have made under the training program; 42 percent reported that they gained new methods of performing their roles and responsibilities; and 10 percent said they gained other things: communication skills, collaboration skills, and the opportunity to network with other professionals in the agricultural sector (Figure 12). 62% 27% 11% Capacity to Conduct Policy Research as a Result of the Training Improved Much improved 40 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Figure 12: Comparison of Usefulness of Capacity Building Activity under NAPP 5.10 EQ2: CONCLUSIONS ▪ NAPP activities (e.g., scholar program, conferences) have helped increase the exposure of the next generation of researchers to new ideas, new approaches in research and knowledge, sharing with peers, and in some cases increased their confidence levels through presentations of their work to various audiences. ▪ The relationship between academia and policy makers is key to dissemination of research findings and the involvement in NAPP activities have enabled these researchers to interact more with policy makers and provide support in the policy process. ▪ At the state level, the training organized for key stakeholders has strengthened the capacity of beneficiaries to engage in policy dialogues, improved their skills to conduct surveys, and increased confidence in data collected for planning and policy purposes. ▪ In Kebbi State in particular, all the trainings delivered by NAPP to the MANR were very useful; however, the most useful was the collection of key economic indicators needed for the analysis of the livelihoods of people in the state (which was required for the annual food and nutrition security policy metrics computed for Nigeria). 5.11 EQ2: RECOMMENDATIONS ▪ Future policy interventions should engage with actors to identify critical areas of need to provide targeted interventions to support improvements at MDAs. 48.0% 42.1% 10.0% What Was most useful about the Capacity-Building Training? New technical skills New methods of performing roles and responsibilities Others (specify) 41 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ▪ Future policy interventions might benefit from concentrating their efforts on a consensus of key policy issues and follow them through, widely publicizing any changes which occur. 5.12 EVALUATION QUESTION 3 (EQ3) EQ3: To what extent has the capacity of local research partners been strengthened to inform policy debates and implementation with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? 5.12.1 EQ3: Findings Again, capacities have been grown to a great extent. NAPP activities have resulted in improved knowledge in data analysis, strengthened capacity of local researchers, increased engagement of local researchers in research project, and better opportunities for research grants among local researchers. The results show that the majority (62 percent) of beneficiaries rated their capacity to conduct policy research due to the training received as improved slightly, while 27 percent reported that the training attended has greatly improved their capacity (Figure 13). Figure 13: Change in Capacity to Conduct Policy Research Moreover, beneficiaries now have a better understanding of policy research and expressed strong interest and confidence in their ability to conduct independent policy research. Nearly 85 percent of the beneficiaries of the NAPP training activities are interested in conducting independent policy research in the future. The motivation to engage in policy research is largely due to improved knowledge of policy issues in the sector and the improved capacity to carry out research (Figure 14). 62% 27% 11% Capacity to Conduct Policy Research as a Result of the Training Improved Much improved 42 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Figure 14: Creation of Interest in Conducting Independent Policy Research under NAPP 5.12.2 EQ3(a): How well are local research partners providing needed support to relevant state ministries in the formulation and implementation of policy processes? 5.12.3 EQ3(a): Findings A collaborator in Ebonyi State said his involvement in the NAPP had huge effects on his capacity to inform policy debates. “Comparing now after participating in NAPP and before I had this opportunity, my understanding of policy debate has increased, and my experience and research knowledge has improved. e.g., how to design questionnaires, how to conduct survey interviews to obtain quality data, and how best to conduct research.” These researchers have initiated COVID-19 response strategy policy research, inspired by data released, to find out which households suffered the impact of COVID-19 the most and how these findings could help target interventions by the state. In Kebbi and Benue States, the MANRs reported that local researchers were instrumental in developing and, for Benue, finalizing their states’ agricultural policy documents. In Niger State, the project scholar and advisor have met with relevant MDAs to identify their needs, including the capacity-building needs of the MANR. Additional studies (such as a feasibility study on the best way to utilize available land in the rice value chain, a sector in which women play a significant role) are planned after restrictions imposed by COVID-19 have been eased. These will address the different areas of concern identified by the Ministry. The planned research is expected to feed into the rice sustainability policy of Niger State; one study plans to investigate the comparative advantages of rice produced by the State. Similarly, in Benue State one scholar worked with the Benue MANR on a farmer-herder project during conflict periods. This scholar trained MANR staff and farmers on the use of the LandPKS testing app, saving farmers the trouble of going to a lab for soil tests. The LandPKS app which can be used in any location worldwide, is an application which supports more sustainable land management decision-making by assisting users to collect site￾specific data about their soils, vegetation and other site characteristics. Another scholar developed a policy brief on the degradation of soil in Taraba State, which was shared with the MANR. Agricultural Policy 84.9% 15.1% Interested in Conducting Independent Policy Research in the Future? Yes No 43 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Research Network (APRNet) reported that they shared seven of their journals with FMARD and in Kogi State; APRnet contributed to Kogi’s rice project and suggestions given by APRNet were adopted. 5.12.4 EQ3 (b): How often do policy actors interact to advance policy processes at national and state levels? 5.12.5 EQ3 (b): Findings A scholar and his adviser have established connections with Kogi State government through the ministries. In collaboration with MSU, he and his supervisor submitted a proposal to support the state on farmer￾herders conflict. They are awaiting a response. In Niger State, a scholar said that “because of our affiliation with NAPP, the ministries are ready to listen and accept our ideas.” She pointed out that the ministries see value in their work and ask for pamphlets to disseminate to farmers on the Aflatoxin training conducted by NAPP. She noted that “prior to NAPP, there was no or limited collaboration between ministries and researchers but now as a Ph.D. student I am able to engage with directors at ministries to identify areas of research needs in the ministries.” In Niger State, local researchers (including a scholar and her advisor) have had meetings with MDAs to identify needs and capacity building of ministry staff for additional studies (such as a feasibility study on best way to utilize available land in the rice value chain—a sector where women play a significant role). These meetings are planned for MDAs (after the restrictions imposed by COVID-19 have been eased) to address the different areas of concern identified by MDAs. It is expected that planned research will feed into the rice sustainability policy of Niger State; one of the studies plans to investigate the comparative advantages of rice produced by the state. In Benue State, at the request of NAPP a scholar liaised with MANR staff before she went to MSU and then worked with them on the farmer-herder project during the conflicts in the state. After her program at MSU, she trained MANR staff and farmers on the use of Landpks mobile app, which saved farmers the trouble of going to laboratories for soil tests and is now in use in eight states. An advisor said that following his engagements with NAPP, he has undertaken the Agriculture Investment Plan (AIP) consultancy assignment for one of the ministries. He notes that “because of consultations done with ministries for NAPP the MDAs knew me, so I was a natural choice when they wanted a local consultant to undertake the assignment.” APRnet has shared seven of its journals with FMARD. Also, In Kogi State, APRnet made inputs into the state’s rice project and its suggestions and advice were adopted. In Delta State, one collaborator pointed out that before the NAPP came on board, he was included in the committee to draft the Agricultural Policy for Delta State which was initially sponsored by Partnership Initiatives in the Niger Delta and some other groups. The drafting process is still on-going, and he has a good insight into practical aspects of policy. The policy currently under review is well in line with the National Agricultural Policy, so there is a synergy between state agricultural policy and national agricultural policy. However, the collaborator noted that “though there were a few activities within the NAPP that brought the faculty together with the state ministry, and that the ministry should be making use of evidence-based research from the university because they should be the first beneficiary of their research output, but rather, the ministry is not interested in the agricultural research output. This makes it difficult to establish and build a strong lasing synergy between the university faculty and the ministry.” One of the 44 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION collaborators further explained that no one from Delta State University has ever participated in collaborative research under the NAPP; that applies everywhere among the seven targeted states this intervention. There is evidence of periodic engagement of actors to advance policy processes in Kebbi State. For example, a “brown bag” series was created including researchers from KSUST, policy makers (state MANR), and farmers. These meetings were a result of NAPP’s activities; during development of the policy document, the need was seen to capture research inputs from stakeholders, such as farmers, and communicate them to policy makers. Stakeholders in this series met three times—April 2019 (inaugural meeting), July 2019, and November 2019. This series suggests that relevant research was carried out and that the policy makers obtained the required evidence for effective policy making. Furthermore, the MANR in Kebbi and Benue States reported that local researchers were instrumental in developing and finalizing (for Benue) the state agricultural policy document. Also, the Kebbi MANR noted that there had been a marked improvement in rice production in the state, because of awareness raised during the development of the agriculture policy document. The Kebbi MANR further noted that they now know that they no longer have to send out their research needs to the research institute in Zaria; they now have confidence in the capability of KSUST to carry out needed ministry research. 5.12.6 EQ3: Conclusions ▪ The capacity of beneficiaries to conduct policy research was compared across eight stakeholder categories: State government, Federal government, Private sector, Media, NGOs, Scholars, Advisors and others (retirees and unemployed), using a one-way between group ANOVA. The results show a statistically significant difference in the capacity of the groups. This implies that the benefits accruing to different groups are not the same. Judging by the results of the descriptive analysis, improvement in capacity is higher for scholars, advisors and media groups than for state MDAs, FMARD and NGOs. Those with the least improved capacity are found among the private sector and others (Annex 1). ▪ In the same way, improvement in the capacity of beneficiaries to conduct policy research was compared across educational qualifications, using a one-way between group ANOVA. As expected, the results show a statistically significant difference in improved capacity. Capacity improvement was highest among participants with post-graduate degrees compared with those having a bachelor’s degree or lower qualifications (Annex 1). ▪ NAPP activities have strengthened the capacity of local researchers and improved their ability to be involved in policy debates. ▪ Researchers’ participation in NAPP, in addition to increasing their exposure to new techniques, has given them a platform through which to engage with MDAs and thus build ministry confidence in the policy process. For instance, in Niger State, researchers have been able “to engage with MDAs to find out the areas of concern and are working to provide needed support (studies or other) to address the identified issues.” Also, researchers from KSUST were involved in the development of an agricultural policy for Kebbi State, which is instrumental in guiding agricultural activities in the state. In Benue State, local researchers were instrumental in finalizing the state agricultural policy document. Also, one of the NAPP scholars trained and provided rice and maize farmers with a software needed for testing soil fertility. 45 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ▪ The visit to MSU is a rich experience and improved the capacity for an informed policy debate among scholars, advisors, and policy makers. ▪ In Kebbi State, the state agricultural policy document is regarded as key achievement of NAPP as it has improved the capacity of the MANR in their budgeting and planning activities. 5.12.7 EQ3: Recommendations ▪ Future policy programs should continue this approach of involving local researchers in interventions targeted at MDAs in order to build MDAs familiarity with and confidence in local research capability. ▪ Participants in training should be given logistics incentives or a payment to cover part of transport costs incurred to attend program events. Popular methods used are giving fixed rates for mileage covered to attend events. ▪ Agreements with project facilitators should developed such that it will clearly state the expectations from facilitators. This way, there is little room for misinterpretation of the contents of the agreement. Also, any changes to the agreement should be clearly communicated to the facilitators with reasons given for the changes, to ensure partners are carried along at various stages of the project. 5.13 Evaluation Question 4 (EQ4) EQ4: To what Extent Has the Capacity of the Media Been Increased To Undertake informed Reporting on Agricultural Policy Issues? 5.13.1 EQ4: Findings Capacity of the media to undertake informed reporting has been increased to a great extent. Feedback from media respondents indicate that under NAPP media actors have learned about various reporting outlets, including social media, for their stories. One respondent stated that “I learned from the training that the public needs to know a lot about agriculture.” Media actors attending the workshops in Niger State reported that they applied methodologies learned to undertake investigative journalism about food insecurity in their state. Also, knowledge from the training was shared with other journalists and media students. One media actor in Kebbi State pointed out that the training enabled her to produce radio call-in programs, which helped to reveal the challenges farmers face in Kebbi, particularly threats from an anchor borrowers’ scheme. Following the training, this media actor and other colleagues investigated and published reports in various media, which led to the inclusion of previously excluded rice farmers in a loan scheme., as well as a focus on measures to mitigate the effects of flooding on food security in both Kebbi and in the country at large. The respondent also claimed to have gained greater knowledge of scripting of agricultural messages, targeting the applicable audiences. When reporting on agricultural issues, this media actor now considers factors which add depth and credibility to her reporting, to elicit behavior change in relevant stakeholders, or changes in policy. It should be noted that this media actor works in the agricultural sector and it remains to be seen whether this training will have similar effects on the performance of mainstream media actors. The training offered to the media actors under NAPP has strengthened the capacity of local reporters, led to career advancements, and resulted in higher quality reporting of agricultural policy issues. 46 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION APRnet organized conferences for the media and engaged with them in most of the focal states to disseminate NAPP research findings on policy issues to a wider audience. The Benue MANR reported that the media often covered the activities undertaken by the ministry under NAPP. The ministry has an information unit which invites staff to their functions to publicize accomplishments. Communications with the media are vetted by the Director of Statistics before being approved by the Permanent Secretary and shared with the media. Communications are usually via radio and the state’s Voice newspaper. The Delta MANR reported that the training of local reporters boosted their confidence and knowledge on reporting agriculture-related issues and were happy to be involved for the first time. One respondent remarked, “They were happy to have someone listen to their problems and views and suggest better ways through a special training.” 5.13.2 EQ4: Conclusions ▪ NAPP contributed immensely to building the capacity of the media to undertake informed reporting on policy issues. The training was beneficial to the careers of the respondents and improved their capacity to report agricultural policy issues. ▪ The media actors in Niger and Kebbi found the media training, particularly the senior master class training, and workshops they attended to be useful; they believe they led to improved capacities for informed reporting to better enlighten the public about agricultural issues. 5.13.3 EQ4: Recommendations ▪ Future policy programs should build upon NAPPs work in growing the capacity of the communications unit in relevant agriculture agencies, as well as the capacity of mainstream media actors, to ensure improved analysis and reporting of agriculture-related issues. ▪ Considering that agriculture is not a typical area covered by media actors, future agriculture policy programs targeting media actors should allocate more time for the training workshop, to allow for relevant topics to be covered in greater depth and to allow for feedback from participants, rather than scheduling the workshop for two or three days. 5.14 Evaluation Question 5 (EQ5) EQ5: What measures is the activity putting in place to ensure: (i) Sustainability of the interventions and the changing needs of the policy environment at the national/state levels? (ii) Policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs? 5.14.1 EQ5: Findings The IPs noted that key aspects of sustainability put in place by NAPP were collaborative research and the scholars’ program. The reasoning behind this was reportedly that if the goal was to strengthen research networks and have “go-to” people to provide an evidence base for the policy process, then one would seek out people whose capacities are strong and are close to policy conversations. The most convincing evidence for sustainability is the knowledge gained by the MSU scholars, research collaborators, and capacity-building beneficiaries. It is important to stress, however, that whereas policy research was collaborative in nature, the leadership of the research teams is restricted to IFPRI and MSU researchers. This limits sustainability. 47 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Limited sources of funding are identified as a major risk to sustainability. According to an association, “All our activities require funding and we are not sure where will we get it from when NAPP ends.” He pointed out that the membership fee of N10,000 can barely sustain them and that members rarely pay this fee, unless compelled in order to attend events. “Even then membership funds would not sustain the network……the future is shaky.” Some respondents were skeptical about sustainability on account of inadequate funding and lack of political will. According to one respondent: Efforts were made by NAPP to ensure sustainability, but funding is an issue and beneficiaries and institutions do not have these funds… Although, participants expected transport money, they paid their way - transport and accommodation - when they attended NAPP events if events were located within their state, regardless of how far they had to travel. It is only when they were required to travel out of state that they might be sponsored. Regarding continuity of policy dialogue, one association noted that: The policy dialogue could only continue if the government is willing to fund and manage the activity. The level of government involvement will determine its continuation or sustainability. I am not sure dialogues will continue. Everything boils down to funding and when the ministries no longer hear about a project, everything will go cold until another project comes on board. 5.14.2 EQ5: Conclusions ▪ The training-of-trainer model promoted under NAPP is viewed as key to the sustainability of the interventions. With the 13 scholars, scholars’ advisors, APRNet, and NAAE, among others, there is now a critical mass of people who the project can call upon to do the work without MSU or IFPRI being physically present. As noted by a respondent: “We have gained a cohort of people over the years who can do the work required for the policy process independently, so if ADAN [Association of Deans of Agriculture in Nigeria Universities] requires training on R or Stata for instance, there are now people who can do this in Nigeria.” ▪ Although there are several risks to the sustainability of the interventions, the two major risks identified are financial and political. ▪ By and large, it is moderately likely that the outcomes and effects of NAPP will be sustained. 5.14.3 EQ5: Recommendations ▪ There is a need to strengthen promising aspects of NAPP, particularly the capacity￾building/scholars’ program and policy development aspects in future policy projects, so that future projects adequately address gaps identified in NAPP. ▪ Nigerian policy researchers should be included in the conduct of policy research under the activity. This should be achieved through an entrenched and systematic process of calling for research proposals from Nigerian researchers in already identified thematic policy areas. For example, the activity can issue requests for proposals from the agricultural policy research community and select the best proposals for funding under the identified research themes. 48 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ▪ Rather than limiting the activities to IFPRI and MSU, funds should be provided to local institutions to build partnerships and jointly carry out specific research, training, and dissemination activities. This is an important built-in mechanism to foster the sustainability of the activity. ▪ Interventions to help states develop their policies and the scholars’ programs are the key areas which should be sustained. ▪ Careful attention should be given to exit strategies for interventions during implementation, including alternative funding and resources for activities, so sufficient support is provided to ensure interventions self-sustain after implementation. This will ensure any issues with institutionalizing learnings are sorted out while the program is still running. ▪ Leverage resources from relevant USAID and FTF programs, including private sector intervention programs, that are capable of strengthening linkages with smallholder farmers. SECTION 6. CONSTRAINTS AND WEAKNESSES The most remarkable constraint of NAPP was the COVID-19 pandemic, which hit the world dramatically during the last year of the project and resulted in a disruption of some activities already planned out, especially in the last three quarters of the year. Nonetheless, efforts were made to continue with the implementation of the project to the extent permissible by the prevailing public health measures and protocols in many working places. Over the life of the project, considerable achievements were made in areas relating to the evaluation questions. However, it might have been possible to achieve even greater results, but for some inherent weaknesses in the implementation procedure, discussed below. Section 5 above has highlighted the strengths of the project, especially as they relate to the evaluation questions. There were great achievements in the training programs and other benefits derived by various stakeholders including state partners, scholars, advisory professors, women and youth, research collaborators, and the media. Nonetheless, it is expedient to consider some weaker aspects in a final performance evaluation of this nature, in order to determine the future direction the project must follow and the lessons learned to inform future implementation, in Nigeria or elsewhere. 6.1 Issue of Baseline not adequately Addressed The assessment of achievements in this project based on the specified performance indicators would have been easier to measure if baseline information were available to guide the process. In the absence of such data, the project relied on achievements in the first year of operation as baseline. This makes a comparison between baseline and endline impossible. 6.2 Use of Cumbersome Custom Indicators The use of indicators which are not SMART, such as (i) “Index (or scorecard) of quality of agriculture and food security policy processes in Nigeria as measured by stakeholder evaluation to capture level of satisfaction and confidence,” and, (ii) “Index (or scorecard) of quality of the institutional architecture for agriculture and food security policy processes in Nigeria, as measured by stakeholder evaluation survey to capture level of satisfaction and confidence,” is ill-conceived and ill-advised. This use of cumbersome indicators has taken a lot of resources to resolve and the results of the third round of measurement are still being awaited. Nebulous or poorly defined indicators are problematic. 49 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 6.3 Delay in Conduct of Midterm Evaluation In general, it is important for a midterm performance evaluation to be strategically scheduled to allow midline lessons to be incorporated in the other half of the activity, in order that the program can be course-corrected before the end of the Activity. Whereas some recommendations from the midterm performance evaluation, such as the Activity’s shift from continuous research to using research to engage with stakeholders, as well as the IPs ensuring that the Activity agenda was set by stakeholders instead of the IPs themselves, were incorporated into the Activity work plans, other aspects were not. For instance the Activity failed to address (i) the issue of weak demand for research products, by strengthening research infrastructure among local partners and stakeholders; (ii) the disproportionate emphasis on academic￾style research that may be seen as advancing the careers of the IFPRI and MSU researchers, rather than on bringing Nigerian collaborators to the forefront; (iii) the provision of financial support to execute collaborative policy works with the government in the focal states; and, (iv) the introduction of an entrenched and systematic process of calling for research proposals from Nigerian researchers in already identified thematic policy areas to expand the participation of local researchers in the conduct of research under the Activity. The midterm evaluation, covering NAPP’s performance from 2015 through September 2018, did not commence until November 2018 and lasted until March 2019, when the end of the project was considerably less than two years out. Also, the last year of the project was affected by the COVID-19 pandemic, which disrupted operations at the level of project implementation. As it turned out, although the midterm review brought out some recommendations which were implemented to smoothen the implementation process, the effects of such changes would have yielded more results if the implementers had had sufficient time to adjust. 6.4 Partnership Relationships Weaknesses in partnership relationships arose due to the informal nature of the partnerships, or to poor identification and definition of roles and responsibilities. The midterm evaluation highlighted some of these weaknesses. Although the IPs attempted to address the situation, problems persisted as can be seen in the following examples. According to a member of the Association of Deans of Agriculture in Nigeria Universities (ADAN), the project did not share any feedback from the midterm report and there had been only “minimal change” in regard to relationships since 2018. It was pointed out that one adjustment made by the project since the midterm report was the sending of training schedules to the university with the topics clearly stated. Nevertheless, he noted that the training topics were selected independently of any consultation with the university. …No needs assessment or discussions with the university is carried out to determine the selection of training topic. Although the training topics were relevant to the students, the process of arriving at the topics should have been a collaborative one. The respondent further noted that the synergy between NAPP and collaborators was poor and that the collaborators did not have a clear idea of the direction of the project. …There was no documentation on the collaboration plan and it appeared we were only given information on a need-to-know basis. NAPP would just send information on what they want to 50 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION do, e.g., conduct trainings… so communication has been haphazard; it was more of instructions….We see ourselves more as facilitators and not collaborators, we seem to only come in when they need us to reach the MDAs, get enumerators for research, reach out to farmers, or students for trainings. Having focal points, i.e. facilitators, in each targeted state could lead to better organization of activities and better delivery of services. Where some facilitators were engaged, however, lack of proper definition of roles and responsibilities caused disaffection. In spite of the critical need for facilitation, it was not until January 2019 that facilitators were engaged - on eleven-month contracts. Unfortunately, the contracts were terminated in April 2019, even before desired results could be achieved, reportedly on account of misunderstandings about contractual obligations. According to the argument of a former facilitator/collaborator: As a facilitator, I incurred a lot of out-of-pocket expenses, but my contract said I was only entitled to allowances, no accommodation, no transport allowance, as long as it was within the same state, even though we often had to travel far distances within the state. There was also no office space. It was natural therefore to assume this was a consultancy and not a full-time position, contrary to the expectation of NAPP. 6.5 Communication Lapses In the case of partnership with associations, apathy on the part of some facilitators has been experienced in part due to communication lapses between them and NAPP, resulting in a negative impact on the effectiveness of some NAPP activities. In the case of APRNet, the misunderstanding of their role following the departure of the Chief of Party was part of the problem they have had with communication. APRNet recalled the good experiences they had with NAPP, but the last major activity APRNet conducted for NAPP was in August 2019. Their contract for 2020 was not renewed when they submitted their planned activities to NAPP in January. They were told verbally that NAPP was ending in June 2020 and that it could only fund one stakeholder engagement in 2020. However, nothing was communicated in writing, contracts were not renewed, and they heard nothing more. APRNet believes NAPP managers could have done better in communicating and preparing them for the end of their engagement with the project. 6.6 Advisors’ Visits to MSU Feedback from an advisor shows that there should be a clear plan about activities to take place at MSU which should be agreed upon with advisors even before they get to MSU. In case of one advisor, further knowledge and input was needed for his area of research, and things changed when he got to MSU. Eventually, he had to give a presentation on land, which was preferred by MSU, rather than on the issue of entrepreneurship and economic analysis which was actually his priority. It was the contention of the advisor that the change in the area of focus should have been communicated beforehand, instead of asking him to realign his interests upon arrival at MSU. 6.7 Training Logistics While it was reported that the training was adequate, there were reservations about the welfare of people who participated in training events conducted in some states, e.g., Delta and Niger. They said, “some of the participants came from a distance – paying transportation but were not compensated for it.” And in Niger State, respondents noted that; “Although some form of cost-sharing is ideal to increase perceived 51 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION value for events by participants, funding for trainings and other events did not factor in transport costs which would be incurred by participants who had to travel far distances, across cities or LGAs, within states.” SECTION 7. LESSONS LEARNED & RECOMMENDATIONS 7.1 Lessons Learned ▪ One of the changes implementing partners noted which was made by the project after the midterm review was a shift of project focus from continuous research to using the research to engage with stakeholders. In engagements with stakeholders, they found that further work was needed to tailor the research to suit the various audience. As such, they converted the research to non-technical pieces. For example, they put key information of one the scholars’ work on food nutrition in pamphlets and translated this to various languages which were used to train households. Thus, they found that while research is critical to their work and cannot be stopped, it is critical to package it in a form that is useful for policy discussions for various audiences. ▪ Another change that the partners said they made after the midterm was to ensure the agenda is set/research is determined by stakeholders in Nigeria and not by MSU/ IFPRI. Particularly in Year 5, they consulted the stakeholders to understand their research needs. An example of stakeholder-led work undertaken by the project is nutrition research based on discussions with FMARD. This is the same for the ongoing extension work and capacity-building activities undertaken for FMARD. The implementing partners also noted that policy brief training events were based on requests from state ministries. In general, they said they learned that Nigeria needs to set the agenda while the project should support that agenda, even if it requires that the project adapt. In terms of resources, this means the IPs cannot plan based on the project’s agenda but rather on the country’s agenda, which they in turn then support. As noted earlier, however, there were other recommendations in the MTR which the IPs failed to address for reasons best known to them but evidently unrelated either to shortage of time or the COVID-19 pandemic. If such recommendations had been incorporated and implemented there might have been better success stories to tell than it is currently the case. ▪ It is vital to provide suitable and adequate personnel for effective project implementation. USAID wanted representatives or facilitators in each of the seven focal states to strengthen the delivery of project activities. Even though this was a step in the right direction, it came too late in the life of the project. Unfortunately, the facilitators engaged who were recruited had to be disengaged due to circumstances that were not totally unavoidable. ▪ Incentives should be provided for stakeholders that are not direct beneficiaries of the project. This particularly refers to the members of the NAC. The implementing partners held back from asking them to do more than what they originally contributed, because the project was conscious of the fact that NAC members were offering their services free of charge. ▪ There should be more feedback mechanisms. The lack of feedback mechanisms tended to limit the impact of capacity-building programs for stakeholders like the media, farmers, and state partners. 52 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 7.2 Recommendations The findings from the key questions guiding this evaluation show that NAPP has been beneficial in several respects in terms of improving skills and capacity to conduct policy research and disseminating the results widely. The benefits are distributed in different ways and proportions across locations and stakeholders. The future of the project can therefore be guided by the following recommendations. 7.2.1 Way Forward A continuation of the NAPP activity is strongly recommended, with a focus on areas where substantial impact has been made and where benefits can be upscaled. It should be noted, however, that there are a number of design and implementation issues that must be appropriately adjusted as decisions to do a follow-on is considered. Going forward, there should be changes in the types of activities, beneficiary targeting, implementation procedures (issues of baseline, NAC membership), performance indicators, partnership arrangements (and ways of forging partnerships, MOU, etc.), stakeholders’ roles and responsibilities, and feedback mechanisms. In light of the foregoing, the following specific recommendations should be considered. 7.2.2 Redefine Focus of Follow-On Project The policy space has different segments that have to be knowledge-driven. The role of research to supply requisite knowledge for evidence-based policy cannot be overemphasized. However, the required knowledge does not have to emanate solely from academic research from university campuses. An agricultural policy project must target potential beneficiaries and stakeholders that nurture and influence the policy decision-making process. Considering this process and intended outcomes and policy impact, a follow-on project must target specific policy segments and work with the stakeholders to achieve the desired results. The project must dwell more on tangible activities and results that can generate returns within the short to medium term. 7.2.3 Focus Follow-On on Specific Policy Areas The focus of NAPP is rather too broad, directly following the rather broad coverage of agricultural policy. For better impact, the project must be designed to target specific policy areas. The perception that impact of agricultural policy takes a long time to observe is based on the omnibus nature of the project policy coverage. This in turn leads to resources being committed in such a diffuse manner as to be incapable of generating any tangible effect and outcome that can be attributed to the specific interventions during the life of the activity. The project should be re-designed to target some of the following specific activities: 1. Strengthen policy development at the state level (this has not been achieved in the first phase) 2. Strengthen partner institutions (how best to achieve this) – APRNET, NAAE, ADAN 3. Award small grants to strengthen the capacity of smallholder farmers (and other beneficiary categories such as media) 4. Provide linkage between farmers and input sources, markets, and finance 5. Provide linkage between NAPP and FTF agricultural extension activities 6. Strengthen collaborative research among Nigerian scholars and institutions 53 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 7.2.4 Ensure proper Targeting of Beneficiaries By following specific activities outlined above, it should be possible to clearly identify the type of potential beneficiaries to be selected. Appropriate targeting of beneficiaries can also be useful in guiding the articulation of the expected outcomes and construction of performance indicators for monitoring progress towards attainment of such outcomes. More importantly, targeting of beneficiaries should be guided by objectives and should be activity-driven. Targeting of project beneficiaries must be conditioned upon the relationship between the intended beneficiary and the specific activity areas of focus. Hand￾picking beneficiaries outside this framework should be discontinued. 7.2.5 Strengthen Process of Establishing and Maintaining Partnerships There must be a formal arrangement for establishing partnerships to harmonize relationships, build trust and confidence, and lay a good foundation for sustainability. Though individual expertise is important, institutional contractual arrangements should be encouraged. Identify relevant partners that have comparative advantage in the specific activities to be included in a follow-on project and establish a formal relationship with them. Where partners are to render services in support of the project, there must be a defined system of reward for the assigned tasks. 7.2.6 Strengthen Implementation Procedure It is important that a baseline be established in a project of this nature in order to measure project achievements. Going forward, the coverage of activities should be narrowed, and appropriate performance indicators constructed to monitor progress. Both output and outcome indicators should be articulated to track implementation performance. 7.2.7 Ensure efficient Delivery of Project Activities It is important to consider the most efficient methods of carrying out activities to ensure the greatest impact for the investment. Going forward, consideration should be given to activities that can be performed through virtual platforms to reach even wider audience in some instances. Additionally, local experts should be involved in carrying out substantive rather than ancillary or residual services to achieve results in a more efficient manner than outsourcing experts. Progress-tracking procedures should be devoid of counterintuitive and expensive surveys to monitor nebulous indices. Such surveys to obtain average scores to measure such things as; “quality of agriculture and food security policy processes in Nigeria as measured by stakeholder evaluation to capture level of satisfaction and confidence,” are grossly inefficient, as they essentially verify nothing. 7.2.8 Encourage Use of Stable and “Smart” Performance Indicators It is important to avoid frequent changes in performance indicators to avoid confusion and reporting challenges. If there is a follow-on, there should be agreement on performance indicators with implementing partners at the start of the project. This is particularly important for baselines, because changes in indicators would necessitate baselines to be conducted for each new indicator, particularly in the case of outcome indicators. This is a complication that can be avoided. Performance monitoring may reveal changes in project components, necessitating the cessation of some project activities. It is understandable that in such a situation, indicators associated with such activities will no longer be applicable. This is very different from introducing new indicators from time to time during the life of the project without any significant change in project objectives and components—as it is the case under NAPP. 54 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 7.2.9 Sustainability ▪ Rather than limiting the activities to IFPRI and MSU, funds should be provided to the local institutions to build partnerships and jointly carry out specific research, training, and dissemination activities. This is an important built-in mechanism to foster the sustainability of the activity. ▪ Establish linkages with other FTF programs. For instance, the farmers under NAPP might be linked with the USAID-funded agri-business program to enhance their access to modern technology and provide better market access and integration. 1 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ANNEX 1: STATEMENT OF WORK AGRICULTURAL POLICY FINAL PERFORMANCE EVALUATION STATEMENT OF WORK I. BACKGROUND INFORMATION A) Identifying Information B) Development Context 1. Problem or Opportunity Addressed by the Project/Activity being evaluated When the Feed the Future Nigeria Agricultural Policy Activity was awarded in 2012, Nigeria had a population of about 170 million, with a rapid growth rate of about three percent per annum. According to the rebased GDP calculation, agriculture contributed about 22 percent of the National GDP figures. Up to 70 percent of the population derived their livelihood from agriculture. Stimulating growth in agriculture was expected to be an effective way to reduce poverty; 33 percent of the population was estimated to be living below the poverty line, with much higher rates of poverty in Northern Nigeria compared to Southern Nigeria as per World Bank recalculations. Nigeria is a signatory to the Comprehensive Africa Agriculture Development Program (CAADP). While the CAADP agreement was expected to provide a common framework for achieving sustained public expenditure to support agricultural growth and poverty reduction, in Nigeria federal budget allocations for agriculture have decreased. Moreover, even if public sector expenditure for agriculture was increasing, it alone would not be sufficient to address existing development challenges. Better policies and a more conducive business-enabling environment DevelopmentObjective Activity Title Broadened and Inclusive Growth Feed the Future Nigeria Agricultural Policy Activity Award Number Project Dates AID-620-LA-15-00001 July 1, 2015 to June 30, 2020 Type of Contract Project/Activity Funding Cooperative Agreement $12,499,999 Agreement Officer’s Representative (AOR) Implementing Partner (IP) Dr. Damba Kawa Michigan State University (MSU) and International Food Policy Research Institute (IFPRI) 2 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION must accompany public investment in agriculture. In the absence of effective policies, the private sector, from input dealers and smallholders farmers to local processors and multi-national agribusiness firms, will not have adequate incentives to invest in the food system, undermining the impact and rationale for public investments. Evidence-based, well-informed policy making is best done with adequate information on farm, firm and market institutions and structures. Weak food security policy capacity in governments, the private sector, civil society, and academia has resulted in unsustainable program and policy responses. The phrase “frequent policy summersault” is often used to describe the Government of Nigeria process of formulating and changing guidelines and policies for the agricultural sector. A recent attempt by the GON in addressing these phenomena is to go through legislation; policies are framed into law and sent to legislative assembly for passage into law. For example, the Growth Enhancement Support Scheme (GES) and the Staple Crop Processing Zone (SCPZ) program are to be established by law. The understanding is that any policy change must go through legislative process that involves consultation with stakeholders, although it is unclear if a transparent consultation process is actually being undertaken by the GON. While this may address the issue of frequent policy changes, it is unlikely to address the issue of evidence required for developing good policy options. In June 2013, the GON signed on to the New Alliance for Food Security and Nutrition, which brings together private sector commitments to invest in agriculture and agri-business with public sector commitment to create an improved enabling environment for food security investments. Development partners agreed to fund supportive development approaches to expedite these commitments for investment and reform. The policy matrix that was developed for the Nigerian New Alliance Cooperation framework was developed with donor support and includes an ambitious reform agenda, which If fully implemented would herald true reform in Nigeria but only if such reforms are maintained and are fully owned by the GON even through changes in administration. In the future such policy matrices should be developed by Nigerian institutions rather than donor funded organizations. The FTF Nigeria Agricultural Policy Activity is designed to address critical evidence gaps for informed policy debate and formulation at country, regional, and global levels. It will synthesize, and disseminate new knowledge on targeted policy issues for which the current evidence base is insufficient or inadequately understood to permit confident formulation and implementation of effective at country, regional and global levels. It will also foster credible, inclusive transparent and sustainable policy process at country level; strengthen building blocks for national policy systems in their regional contexts, and promote inclusion and dialogue among all stakeholders around critical policy issues. 2. Target Objectives and Stakeholders The Activity focused on three integrated objectives as follows: Objective 1: To strengthen the national capacity for greater evidence based policy processes in agriculture by increasing the capacity of Nigerian analysts to undertake and make widely available relevant evidence-based policy analysis; 3 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Objective 2: To promote and foster informed policy dialogue among all stakeholders in the agricultural sector through an inclusive, transparent, and sustainable process at the country level, building blocks for a well-integrated and developed national policy system; Objective 3: To help federal and state governments improve their capacity to plan and implement effective policy analyses and programs, and demand and absorb policy research in their policy processes. Targeted stakeholders included government; academia, e.g. universities and research institutions; media; private sector; civil society organizations; and Nigerian think tank institutions. C. Intended Results of the Project/Activity being Evaluated At the end of this agreement, USAID expects the Activity to achieve the following results: 1) Increased national capacity to generate and analyze information, and to formulate evidence￾based policy options for consideration by decision makers and stakeholders; and, 2) Improved policy processes driven by strong empirical evidence and active dialogue among relevant stakeholders. The results framework adapted for the Activity is illustrated in the Figure below. The various tiers of the results framework are aligned with the overall objectives of the FTF initiative as well as that of CDCS. The goal, Development Objective (DO) and the IRs are linked with the CDCS of USAID/Nigeria. The two sub-intermediate results (Sub-IR 1.1 and 1.3, represented by boxes with solid black lines) are aligned with the FTF initiative. These Sub-IRs are also linked with CDCS’s Sub-IRs 1.1.1 (increased agricultural productivity), Sub-IR 1.1.2 (improved trade and transportation), Sub-IR 1.1.3 (increased resiliency of vulnerable households and communities), Sub-IR 1.2.1 (improved legal and regulatory environment), Sub-IR 1.2.2 (improved access to finance/credit by farmers), Sub-IR 3.2.1 (strengthened management capacity and transparency of government authorities) and Sub-IR 3.2.2 (strengthened public financial management and transparency) of CDCS. The Nigerian Agricultural Policy Activity results framework also includes four strategic results (SR) across the project components that were designed to contribute to the Sub-IRs. These tiers are arranged to illustrate the causal relationship between planned efforts under the Activity and their expected impact, and to identify the IRs critical to achieving the objectives of the Activity. Specifically, the framework demonstrates how planned activities and deliverables were intended to lead to expected outputs, outcomes, results, and eventual impact. Together, the DOs, IRs, Sub￾IRs, and SRs identified in Figure 1, provide the framework for identification and implementation of the Activity that is designed to achieve the required results. 4 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Figure 1: Results framework for NAFSP Program D. Approach and Implementation The FTF Nigeria Agricultural Policy activity planned to accomplish the SRs, IRs and project objectives through three interventions: 1) Enhanced skills providing training and building institutional capacity; 2) Promoted policy driven collaborative research and analyses; and, 3) Promoted evidence-based policy process and impact through improved dialogue, engagement and outreach strategy. Specific interventions proposed to address these themes are necessary to create the appropriate enabling environment to achieve the intermediate and sub-intermediate results as well as the overall goal of the Activity. Types of interventions include: 5 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Interventions to enhance skills, training, and institutional capacity: The Activity proposed a robust approach to enhance skills, training, and institutional capacity for meeting the demands for policy analysis by FMARD and the national food security and agricultural policy process. The focus for training was on methods and applications for research and policy analysis while the actual content varied, depending on the targeted audience and their particular needs, To this end, the Activity undertook consultations with stakeholders, including government, academia, media, private sector and associations to discuss the content and ensure their appropriateness in addressing the most serious gaps in knowledge, analytical and research approaches, and application tools in Nigeria. The target audience also included promising graduate students from Nigerian universities, and young professional researchers and policy analysts from various governmental and nongovernmental institutions or organizations. At least two primary tracks were envisioned in the training and transfer of tools – a policy research and policy analysis track. The policy research track that involved both direct training of select graduate students in Nigerian Universities (collaboratively between MSU and IFPRI researchers and their respective institution’s research supervisors) and periodic training workshops. For the training workshops, these were primarily focused on research and survey methods, targeted at other researchers in the university system, private sector, and for-profit think tanks. The ultimate goal of this track was to strengthen the capacities of the next generation of policy researchers in Nigeria. The policy analysis track, on the other hand, focused more on policy analysis, both thematic and on analytical methods, and targeted a broader audience that included graduate students, faculty, practitioners in government, CSOs, NGOS, private sector, and think tanks. To promote the visibility of the policy research and analysis being undertaken, the Activity proposed to organize three major conferences during the period of the program (2015-2020) which will bring together experts in selected issues in agriculture and economic development from their various institutions and partners to provide guidance to Nigerian graduate students and young research professionals working in these areas. Policy driven collaborative research and analysis: The policy driven collaborative research and analysis was intended to directly support the knowledge needs of the policy process and sub￾divided into two components, a research and a policy analysis component. The first component involved research teams composed of staff from IFPRI, MSU, and local research and academic institutions. Priorities on the research topics were carried out in consultation with FMARD, development partners, and other key stakeholders. For the initial two years of the program, research topics were identified based on such a consultative process in 2013 and 2014, including but not limited to: (i) the principal determinants of effective and sustainable agricultural transformation in Nigeria; (ii) land governance and investment necessary to achieve sustainable transformation; (iii) the overall crucial nexus between agricultural policy and nutrition and how to achieve it, (iv) how climate change affects agricultural transformation and what can be done about it and (v) a more in-depth and refined understanding of how policy change occurs, as well as of the bottlenecks to achieving better policy implementation and outcomes, for strengthening agricultural and food security policy in Nigeria. 6 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Furthermore, the Activity proposed to strengthen local capacities and dialogue by undertaking policy research and analyses within the purposefully constituted research teams involving local partners who were employed by reputed Nigerian research and academic institutions. These researchers and their network were called upon to inform policy debates with empirical evidence to increase interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders). The involvement of Nigerian graduate students at the collaborating Nigerian Universities were paramount to the process to begin to strengthen the capacity of the future generation of Nigerian researchers. The analytical component on the other hand focused more on demand driven and shorter term policy analyses that originate from requests of FMARD and their development partners. In addition to linking FMARD with various actors in the National research system, the Activity actively engaged and collaborated closely with policy analysts at FMARD and other government institutions charged with informing the policy process when undertaking policy relevant analyses on a revolving demand basis, including pursuing with government and development partners the need for new institutional innovations to establish a strong policy analysis unit within FMARD. Strengthening evidence-based policy process and promoting impact: This was an important intervention of the Activity, since it allowed for policy impact through increased and targeted policy communications, including: directly contributing to the donor-government policy dialogue on agricultural policy, such as through the Joint Task Force on Agriculture; launching a policy brief series for the Activity to serve as an outlet for early results of ongoing research, policy analysis, and/or outcomes from roundtable discussions or seminar dialogues sponsored by the project or jointly with collaborating institutional partners of the project; promoting a “think tank” culture within the agricultural policy process – through organizing various seminars and events targeted at all the actors in the process, including policy makers, local research community, FMARD, development partners, and the general media. This also aimed to strengthen the visibility and credibility of local research networks, such as through co-organizing or contributing to nationally or regionally sponsored workshops and research conferences in collaboration with local institutions and/or existing networks to foster more direct interaction between internationally renowned researchers and the next generation of Nigerian researchers. The Nigerian Agricultural Policy Activity was developed based on the following development challenges: ● Need and support for evidence-based policy and program actions to address the challenges of agricultural transformation, and forge long-term solutions to reducing poverty in rural areas and chronic food insecurity. ● Need for improving policies and increasing public and private investments to achieve food security goal through agricultural and economic growth. ● Need to strengthen policy institutions and processes through coordinated actions that support greater engagement and participation by civil society and private sector participation in policy systems. ● The need to have impact at scale. 7 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION There were a few critical assumptions for the Feed the Future Nigerian Agricultural Policy Activity to achieve its targets and objectives. One was that the policy making environment in Nigeria remained relatively stable and was not subjected to major upheavals that were often associated with shifts in political power resulting from undemocratic forces. Second assumption was that the government, policy makers and stakeholders targeted by the Activity are receptive to new ideas and responsive to evidence presented and committed to implementing and enforcing the outcomes of policy research and analysis. In other words, the Results Framework relied on there being political will and support for the policy change agenda and that the project will receive cooperation and participation of targeted stakeholders in program activities and its outputs. While the above factors pose serious challenges, every effort will be made to achieve the mission of the project, which is to inform the process through rigorous and timely evidence building and help improve local capacity to use this evidence for policy analysis and advocacy. Existing Data The Evaluation Team will have access to the vital documents relevant to conducting this evaluation. These documents will include the technical proposal, original contract, monitoring and evaluation plan, annual work plans, quarterly and annual reports, data quality reports, contract amendments, memorandums of understanding, and any other relevant materials documenting the management, implementation process and results for the Nigeria Agricultural Policy Activity permitted by the Office of Acquisition and Assistance. II. EVALUATION RATIONALE A) Evaluation Purpose The primary purpose of this final performance evaluation is to determine whether the assistance provided by USAID/Nigeria through the Agricultural Policy Activity achieved the stated development objectives, and to understand the lessons learned from this particular Activity in Nigeria, taking into account the findings and recommendations of the mid-term performance evaluation. It is intended to provide an independent examination of the overall progress and accomplishments of the Activity. The evaluation should provide a detailed picture of the major accomplishments and weaknesses of the Activity since its inception, and determine how its successes can be sustained. This final performance evaluation will provide USAID/Nigeria, its implementing partners, and agriculture sector stakeholders with data on outcomes and achievements. The evaluation should also elucidate lessons learned and include specific recommendations to USAID/Nigeria on how the Activity interventions can be sustained and scaled by other FTF activities, partners, private sector firms, or by the Government of Nigeria (GON). B) Audience and Intended Use The primary users of the evaluation findings are the USAID Economic Growth and Environment (EGE) Office, other U.S. Government (USG) officials, non-USG donor organizations, GON, and private sector firms, all of whom can use the findings to improve and build knowledge (Table 1 below). 8 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Table 1: Audience and Intended Uses # Intended Use Target Audience USAI D/ Nigeri a FtF Feedbac k/Other USG Implementi ng Partners GON /Other Dev. Partners Research Communiti es 1 Inform Policy ✓ ✓ ✓ 2 Inform Project design ✓ ✓ ✓ ✓ ✓ 3 Improve project monitoring ✓ ✓ ✓ ✓ ✓ 4 Improve operational policy and planning ✓ ✓ ✓ ✓ 5 Improve resource management ✓ ✓ ✓ ✓ 6 Enhance professional growth ✓ ✓ ✓ ✓ ✓ 7 Add scientific Knowledge ✓ Importantly, explain how they will most likely use the findings from this evaluation. The result of this evaluation will be used by the Mission, FTF implementing partners and other stakeholders on enhancing policy formulation, analysis and implementation in the country USAID will develop a dissemination plan in accordance with the Evaluation Policy as specified in Automated Directive System (ADS) Chapter 201saj. The final report will be published in hard and electronic copies for distribution within USAID/Nigeria and Implementing Partner(s), as well as being posted to the Development Experience Clearinghouse. C) Evaluation Questions The Agricultural Policy activity had three principal objectives: ● To strengthen the national capacity for greater evidence based policy processes in agriculture by increasing the credibility and supply of research conducted by Nigerians. ● To promote and foster informed dialogue among all stakeholders in the agricultural sector through an inclusive, transparent, and sustainable process at the country level, building blocks for a well-integrated and developed national policy system. ● To help federal and state governments improve their capacity to plan and implement effective policy analyses and programs, and demand and absorbs policy research in their policy process. 9 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION In determining the achievement of these objectives, the following key evaluation questions should be answered: 1. Gender/youth: To what extent have women and youth benefited from policy intervention undertaken by the activity? ● Which approaches have been most effective in reaching women and youth? ● How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? ● How have women and youth been part of the dialogue in the agriculture sector in developing national policy system? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? ● How have federal and state governments implemented effective policies 2. To what extent have the direct training of select graduate students in Nigerian Universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? 3. To what extent has the capacity of local research partners been strengthened to inform policy debates with empirical evidence towards increasing interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? 4. To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? 5. What measures have the Activity put in place to ensure sustainability of the interventions? III. Timeframe & Travel A) Timeframe Provide the timeframe for the evaluation: State Date: November 1, 2019 End Date: March 30, 2020 The final evaluation report must be delivered by March 30, 2020 Months Nov 2019 Dec 2019 Jan 2020 Feb 2020 Mar 2020 Activities Contract Award 10 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION Submission of Work Plan Review of Work Plan Preparation of Guidelines for Data Collection Field Visits/ Data Collection Data Analysis Preparation of Report Submission of Draft Report Review Draft Report Review/ Accept Final Report B) Travel Field sites suggestions for data collection and duration of travel 11 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION S/N Destinations Government (G) Academia (A) Media (M) Private Sector (P) CSOs/Associations (C) Duration (days) 1 Cross River G,A,M,P,C 3 2 Delta G,A,M,P,C 3 3 Ebonyi G,A,M,P,C 3 4 Benue G,A,M,P,C 3 5 Niger G,A,M,P,C 3 6 Kebbi G,A,M,P,C 3 7 Kaduna G,A,M,P,C 3 8 FTC G, M 3 Notes: *May include, farmers, private sector (large, MSME, micro), civil society, cooperatives, etc ** Includes State offices (governors, ag, municipalities) and Federal (ministries, quasi government) IV. DELIVERABLES A) Deliverables The following deliverables will be submitted to USAID/Nigeria. The timeline for submission of deliverables will be finalized and agreed upon with the contractor. 1. Evaluation work plan and timeline: The evaluation work plan and timeline will be developed in consultation with USAID/Nigeria. 12 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 2. Detail Report Outline: This will be agreed upon during the team presentation meeting. 3. Evaluation design/methodology, including questionnaire/guidelines for conducting key informant interviews/focus group discussion guides: These documents will be prepared and submitted to USAID/Nigeria for review and approval prior to the initiation of key informant interviews and site visits. 4. Other Evaluations Materials (list of proposed sites, and list of respondents): Some of these materials such as list of sites to be visited will be shared with USAID/Nigeria prior to visiting the field. List of respondents interviewed as key informants especially will also be provided upon return from field visit. 5. Interviewing Notes and completed surveys: All interviews notes and completed survey instruments used for the evaluation will be submitted to USAID/Nigeria. The data used for analysis will also be submitted to USAID/Nigeria. 6. Debriefing(s): The Team Leader will regularly debrief USAID/Nigeria on the progress being made with the evaluation during field work. At the end of field work, a debriefing meeting will occur with USAID/Nigeria (EGE Team) and include the evaluation team’s findings, conclusions and recommendations, before they leave Nigeria. Power-point presentations (one electronic copy) as well as hard copies for the debriefing will summarize findings, conclusions and recommendations and will be distributed during the meeting. 7. Draft Evaluation Report: A synthesized draft report will include, at a minimum, the following: scope and methodology used; important findings (empirical facts collected by evaluators); conclusions (evaluators’ interpretations and judgements based on the findings); recommendations (proposed actions for management based on the conclusions); and lessons learned (implications for future designs and for others to incorporate into similar programs). The evaluation team will provide USAID/Nigeria with a draft report that includes all the components of the final evaluation report on March 30, 2020. 8. Final Evaluation Report: In addition to being compliant with the criteria of ensuring the quality of the evaluation report, the final report will address the comments provided by USAID/Nigeria and other stakeholders on the draft report. The Evaluation Team Leader will revise the draft report and deliver a final revised version to USAID/Nigeria within three weeks of receiving USAID feedback. The final report in both hard and electronic format will include all elements described in ADS 201mah USAID Evaluation Report Requirements and will be submitted to USAID/Nigeria and approval given before submission to the Development Experience Clearinghouse (DEC). 9. Data and all supporting documentation qualitative and/or quantitative data should be submitted as part of ADS 579.3.2.6 All reports are to be submitted in English in both electronic and hard copies. The Team will provide five printed copies of the Final Evaluation Report. The consultants will be responsible for report production. The Final Evaluation Report should not exceed 30 pages in length in its body, not including title page; Table of Contents; List of Acronyms; usage of space for tables, graphs, charts, or pictures; and/ or any material deemed important and included as Annexes. The Final Evaluation Report and PowerPoint addressing the Mission's comments should be submitted in both Word and 13 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION PDF formats. Once the PDF format has been approved by the Mission, the Team will submit the Final Evaluation Report to the Development Experience Clearinghouse for archiving. Reports should be submitted consistent with the Automated Directives System (ADS) 579. V. TEAM COMPOSITION & SUGGESTED LOE The Evaluation Team will consist of four key members, including at least one (1) international consultant (who will lead the team) and local consultants. The team members should be represent a balance of several types of knowledge and expertise related to agriculture development. USAID/Nigeria recommends the following staffing for the evaluation. Team Leader: An international Senior Evaluation Specialist with extensive experience in evaluating agriculture activities in developing countries with the following qualifications:. · ● Graduate degree in Development Economics, Public Policy, Economics, Business Administration, Agriculture, or a related field. ● At least ten years of experience assessing or evaluating USAID-supported agricultural policy and agricultural value chain activities. ● Previous experience serving as an evaluation Team Leader on a USAID-supported agricultural policy activity. ● Previous experience working in Africa. ● Experience facilitating and providing leadership in collaborative and participatory evaluations with multiple stakeholders. ● Excellent writing, communication and presentation skills. ● Ability to produce preliminary and final reports on time. The Team Leader will be responsible for the overall management of the evaluation, including coordinating and packaging the deliverables. The team leader will develop the outline for the draft report, present the report, and after incorporating USAID/Nigeria staff comments, submit a final report to USAID/Nigeria within the prescribed timeline. In addition to the responsibilities outlined above, the Team Leader will: Preparations 1. Finalize and negotiate with USAID/Nigeria the team’s work plan. 2. Establish assignment roles, responsibilities, and tasks for each team member. 3. Ensure that the logistics arrangements in the field are complete. 4. Ensure gender-sensitive data collection methods e.g. ensuring both male and female team members, noting different cultural environment between peoples in Zone of Influence (Delta, Cross River, Benue, Kaduna, Kebbi, Niger, and Ebonyi) Management 1. Take lead on preparing, coordinating team member input, submitting, revising and finalizing the assignment report. 14 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION 2. Manage process of report writing 3. Manage team coordination meetings in the field 4. Coordinate workflow and tasks and ensure that team members are adhering to the schedule. Communication 1. Handle conflict within the team 2. Serve as primary interface with USAID/Nigeria and serve as the spokesperson for the team. 3. Debrief USAID/Nigeria as the evaluation progresses, and organize a final debriefing. 4. Keep the USAID/Nigeria appraised of progress challenges, work changes, team travel plans in the field, and report preparation via phone conversation or email at least once a week. 5. Serve as primary interface with USAID/Nigeria for the submission of draft and final reports and deliverables to USAID/Nigeria 6. Make decisions in conjunction with USAID/Nigeria about the safety and security of the team. Other team members will include: A host-country national or international Senior Evaluation Specialist: The Senior Evaluation Specialist must have at least a Master's Degree or equivalent in Development Economics, Public Policy, Public Administration, Business Administration, Social Sciences, or related fields. S/he must have a minimum of seven (7) years conducting both qualitative and quantitative evaluations/assessments around improving capacity for service delivery and policy formulation in developing countries. S/he must demonstrate good interpersonal and diplomatic skills as well as strong report writing and analytical skills and excellent oral communication skills in English A host country mid-level Agriculture Economist: The Agriculture Economist must possess technical competence in the field of Agriculture, Agricultural Policy, Agribusiness and Agriculture value chains, Economics, or related fields. S/he must have a minimum of five (5) years conducting both qualitative and quantitative evaluations with strong report writing and analytical skills. S/he must demonstrate strong knowledge of the Nigerian agricultural policy environment. A national Evaluation Specialist with at least five (5) years relevant experience and strong logistics and planning skills. S/he must demonstrate experience in organizational capacity building/assessment required. S/he must demonstrate strong experience with data collection procedures, surveys, and analysis of data using statistical analysis tools At least one of the senior technical team members must have experience analyzing data from agriculture policy projects. At least one of the senior team members should have significant experience with gender integration and institutional capacity experience and expertise. 15 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION USAID leaves to DevTech’s discretion other necessary team members/staff for the evaluation (e.g. logistics, scheduling and translation, data analysis). Aside from the above mentioned key personnel, the offeror must decide how the evaluation team should be structured in order to successfully address the evaluation questions. All attempts should be made for the team to be gender balanced and to include local (Nigerian) experts. A statement of potential bias or conflict of interest (or lack thereof) is required from each team member. USAID may propose internal staff from USAID/Nigeria or from Washington to accompany the team in this evaluation as observers. As observers, their role will be to provide guidance and background information, and to reply to the external evaluators’ questions. They will review and comment on the report for accuracy, but evaluators may accept or reject comments. The final report should reflect the opinions of the external evaluators and is the sole responsibility of the selected Evaluation Team. Conflicts of Interest All evaluation team members will provide a signed statement attesting to a lack of conflicts of interest, or describing an existing conflict of interest relative to the activity being evaluated. USAID/Nigeria will provide the conflict of interest forms. Level of Effort Suggested LOE of each team member: Position LOE (Days) LOE (Hrs.) International Team Lead 55 440 Senior Evaluation Specialist 51 408 Agriculture Economist 46 368 Evaluation Specialist 46 368 VI. Intended Participation of Other Parties USAID may propose internal staff from USAID/Nigeria or from headquarters, Implementation Partners, National Counterparts and/or beneficiaries to accompany the team in this evaluation as observers. As observers, their role will be to provide guidance and background information, and to rely on the external evaluators’ questions. They will review and comment on the report for accuracy, but evaluators may accept or reject comments. The final report should reflect the opinions of the external evaluators and is the sole responsibility of the selected evaluation team. 1 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION ANNEX 2 NAPP Final Performance Evaluation Work Plan FEED THE FUTURE AGRICULTURAL POLICY PROGRAM FINAL EVALUATION DRAFT WORKPLAN AUGUST 1 2020 This document was produced at the request of the United States Agency for International Development. It was prepared independently by the Monitoring, Evaluation and Learning program, DevTech Systems, Inc. 2 FEED THE FUTURE NIGERIA AGRICULTURAL POLICY PROJECT (NAPP) FINAL PERFORMANCE EVALUATION FEED THE FUTURE AGRICULTURAL POLICY PROGRAM FINAL PERFORMANCE EVALUATION DRAFT WORKPLAN Contract No: AID-OAA-I-15-00018 Submitted to: Dr. Chidimma Anyanwu, Contracting Officer’s Representative (COR), MEL Activity, USAID/Nigeria Copied to: Dr. Samba Kawa, Agreement Officer’s Representative (AOR), NAPP Program, USAID Office of Economic Growth and Environment (EGE) Mr. Oladele Kolade, M&E Specialist and Mission Environment Officer (MEO), USAID Office of Economic Growth and Environment (EGE) Submitted by: Paul DeLucco Chief of Party, MEL Activity DevTech Systems, Inc. Email: PDelucco@devtechsys.com Cover Photo: NAPP Project Scholars Panel held on August 14-16, 2018, Transcorp Hilton, Abuja. Credit: Oyinkansola Tasie DISCLAIMER This report is made possible by the support of the American People through the United State Agency for International Development (USAID). The contents of this report are the sole responsibility of DevTech and do not necessarily reflect the views of USAID or the United States Government. 3 CONTENTS SECTION 1: EVALUATION OVERVIEW...................................................................................................................6 SECTION 2: BACKGROUND ...............................................................................................................................6 SECTION 3: PROJECT OVERVIEW.............................................................................................................................7 3.1 Results Framework .................................................................................................................................................7 3.2 Agricultural Policy Project’s Strategy ..................................................................................................................10 3.3 Main Activities.......................................................................................................................................................10 3.4 Critical Challenges/Assumptions .........................................................................................................................12 3.5 Performance Indicators ........................................................................................................................................12 SECTION 4: EVALUATION PURPOSE.....................................................................................................................13 SECTION 5: EVALUATION QUESTIONS...............................................................................................................14 SECTION 6: EVALUATION METHODOLOGY ....................................................................................................14 6.1 Document Review ................................................................................................................................................15 6.1.1 Focus of Document Review 15 6.1.2 Main Observation from Document Review: Gaps in REQUIRED Documents 15 6.2 Data Collection Tool s and Protocols ................................................................................................................16 6.3 Sample Size and Selection of Respondents.........................................................................................................16 6.3.1 Sampling Strategy 16 6.3.2 Sample Size 17 6.4 Mapping of Evaluation Questions with Category of Stakeholders....................................................................18 6.5 Proposed Quality Control Mechanism for Data GATHERING, Analysis AND Oversight ..............................19 6.6 Data Analysis............................................................................................................................................................19 EVALUATION QUESTION 20 DATA COLLECTION SOURCE 20 ANALYTICAL APPROACH 20 6.7 Risks.......................................................................................................................................................................21 6.8 Guiding Principles and Values .................................................................................................................................22 SECTION 7: WORK SCHEDULE........................................................................................................................22 7.1 Team Lead Responsibilities..................................................................................................................................22 7.1.1 Preparations 22 7.1.2 Management 22 7.1.3 Communication 22 7.1.4 Direction 23 SECTION 8: ANNEXES......................................................................................Error! Bookmark not defined. ANNEX 1. DATA COLLECTION TOOLS AND PROTOCOLS ...............................................................................1 1.1 KEY INFORMANT INTERVIEWS – USAID/NIGERIA STAFF 1 1.2 KEY INFORMANT INTERVIEWS – ACTIVITY IMPLEMENTERS 3 1.3 KEY INFORMANT INTERVIEWS – NATIONAL ADVISORY COMMITTEE 5 1.4 KEY INFORMANT INTERVIEWS – FEDERAL AND STATE MDAS 7 1.5 KEY INFORMANT INTERVIEWS – RESEARCH COLLABORATORS 9 1.6 KEY INFORMANT INTERVIEWS – PROJECT SCHOLARS 11 1.7 KEY INFORMANT INTERVIEWS – SCHOLARS’ ADVISORS 13 1.8 KEY INFORMANT INTERVIEWS – CSO/ASSOCIATIONS 16 1.9 KEY INFORMANT INTERVIEWS – PRIVATE SECTOR ACTORS 18 1.10 KEY INFORMANT INTERVIEWS – MEDIA 19 4 1.11 QUESTIONNAIRE FOR TRAINING AND CONFERENCE BENEFICIARIES 20 ANNEX 2: KEY STAKEHOLDERS FOR KII.....................................................Error! Bookmark not defined. ANNEX 3: DETAILS OF SELECTED TELEPHONE INTERVIEWEES – TRAINING BENEFICIARIES.......... Error! Bookmark not defined. ANNEX 4: DETAILS OF SELECTED TELEPHONE INTERVIEWEES – CONFERENCE BENEFICIARIES... Error! Bookmark not defined. LIST OF TABLES Table 1: Agricultural Policy Project Performance Indicators................................................................................12 Table 2: Sample Size by Gender and Category of Participants.............................................................................17 Table 3: Evaluation Questions and Target Respondents........................................................................................18 Table 4: Analytical Approach and Data Collection Sources to Answer Each Evaluation Question............20 Table 5: Timeline and LOE for the USAID/Nigeria Agricultural Policy Program Activity Final Evaluation23 LIST OF FIGURES Figure 1: USAID/Nigeria Economic Growth Results Framework (2015 -2019 CDCS)...................................8 5 ACRONYMS ABU Ahmadu Bello University APRnet Association of Deans of Agriculture Universities Agriculture Policy Research Network ARCN Agricultural Research Council of Nigeria ATA Agricultural Transformation Agenda AU African Union CBO Community-Based Organization CDCS Country Development Cooperation Strategy CSEA Centre for the Study of the Economies of Africa CSO Civil Society Organizations DO Development Objectives DQA ET Data Quality Assessment Evaluation Team FMARD Federal Ministry of Agriculture and Rural Development FSP Food Security Policy FTF Feed the Future FTFMS Feed the Future Monitoring System FUNAI Federal University Ndufu-Alike GDP Gross Domestic Product GIS Geographic Information Systems GON Federal Government of Nigeria IFPRI International Food Policy Research Institute IITA International Institute of Tropical Agriculture IR Intermediate Result KII Key Informant Interview MANR Ministry of Agriculture and Natural Resources MOAF Ministry of Agriculture and Forests MSU Michigan State University NBS National Bureau of Statistics NEPAD New Partnership for African Development NGO Non-Governmental Organizations SIR Sub-Intermediate Result SR Strategic Results TOC Theory of Change USAID United States Agency for International Development USG United States Government WEF World Economic Forum 6 SECTION 1: EVALUATION OVERVIEW This workplan is for the final performance evaluation of the Feed the Future (FTF) Nigeria Agricultural Policy Program (NAPP), a five-year capacity-building and policy research and dissemination activity. Michigan State University (MSU) in partnership with the International Food Policy Research Institute (IFPRI) began implementing the NAPP contract on July 1, 2015 with oversight from the United States Agency for International Development (USAID). Although NAPP was originally scheduled to end on June 30, 2020, it was extended by USAID to December 31, 2020. The goal of the Program, hereafter referred to as NAPP, is to, 1) Increase national capacity to generate and analyze information, and to formulate evidence-based policy options for considerations by decision makers and stakeholders, and, 2) Improve policy processes driven by empirical evidence and active dialogue among decision makers and stakeholders. This two-part goal will be achieved through: a) Strengthening the national capacity for greater evidence-based policy processes in agriculture through collaborative research, with due recognition of gender dimensions, for informed agricultural policy debate and formulation at the national and regional levels to improve the performance of the agricultural sector and to achieve policy impact, b) Promoting and fostering informed policy dialogue among all agricultural sector stakeholders to promote a “think tank” culture within the agricultural policy landscape. This transformation will occur through strengthening the interaction between the various stakeholders in the agricultural policy process, and, c) Assisting federal and state governments to improve their capacities to plan and implement effective policy analyses and programs, and demand and absorb policy research in their policy process. The purpose of this final performance evaluation is to determine whether the assistance provided by USAID through the NAPP achieved the stated development objectives, and to understand the lessons learned from this particular Program in Nigeria, taking into account the findings and recommendations of the mid-term performance evaluation. It is intended to provide an independent examination of the overall progress and accomplishments of the Program. SECTION 2: BACKGROUND With an abundant and diversified agricultural base, Nigeria’s economy has a large agricultural component. The agricultural sector generates about 22 percent of the gross domestic product (GDP) and employs about 70 percent of the labor force. Until the early 1970s, Nigeria was self-sufficient in food production with a small surplus for export and agriculture was the main foreign exchange earner. However, post￾1970, the sector stagnated for a number of reasons, chief among them, the discovery, exploitation, and exports of oil as well as a deliberate policy to shift resources from agriculture to industry (Oyejide:1986). Because agriculture employs an overwhelming share of the Nigerian labor force, stagnation of the sector resulted in increased poverty. The poverty rate (headcount measure at $2.00/day consumption) increased from 28 percent in 1980 to 68 percent in 2012. However, since 2015, the performance of the agricultural sector is believed to have improved; now contributing 25 percent to the country’s GDP (National Bureau of Statistics [NBS], 2017; CBN, 2018). Nigeria is relatively developed by sub-Saharan Africa standards, but not as developed as other parts of the developing world, especially the fast-growing countries of South Asia. Nigeria, the largest country by population and size of the economy in Africa, has abundant and rich agricultural resources, human capital, and a diversified and rich natural resource base. Until recently, economic performance had been slow and driven primarily by the oil sector. However, the agricultural sector has been growing rapidly since 2005 with growth in value added in the sector averaging about seven percent annually. Although the agriculture 7 sector employs about 48 percent of the labor force, in 2017, agriculture contributed around 25 percent to the GDP of Nigeria, suggesting that productivity and incomes in the sector were low (NBS: 2018). In spite of the recent perceived improved performance in the agricultural sector, labor productivity is low and slow-growing. Unemployment as well as poverty in rural areas, especially among the youth, tends to be high. According to the National Bureau of Statistics (NBS), poverty incidence (using $1.25/day per capita as the measure) in 2011 was 52 percent in urban areas but 66.1 percent in rural areas. An explanation for the high poverty incidence may partly lie in high unemployment rates combined with low productivity in the agricultural sector. Although sectoral poverty data are not available, rural unemployment rates are higher than urban unemployment rates: 33.5 percent in urban areas, compared to 38.2 percent in rural areas. Recognizing the need to revitalize and transform the agriculture sector to increase rural incomes and grow its economy, the Federal Government of Nigeria (GON) embarked, in 2011, on a visionary strategy for the sector by launching the Agricultural Transformation Agenda (ATA). The goal of the ATA was to increase rural incomes and grow its economy by providing an improved enabling environment for the private sector to lead the agricultural transformation process. More recently, in 2013, Nigeria joined the New Alliance for Food Security and Nutrition led by the G-8 and the Grow Africa initiative led jointly by the African Union (AU), the New Partnership for African Development (NEPAD), and the World Economic Forum (WEF) as a way to share knowledge about its own policy and investment commitments through the ATA and the shared goals of promoting greater private sector investments. NAPP is intended to ensure that all partners meet the growing expectations of the positive role that national and international research can play in supporting the capacity, knowledge and information needs of the Federal Ministry of Agriculture and Rural Development (FMARD). SECTION 3: PROJECT OVERVIEW The goal of NAPP is to a) Increase national capacity to generate and analyze information, and to formulate evidence-based policy options for considerations by decision makers and stakeholders, and b) Improve policy processes driven by empirical evidence and active dialogue among relevant stakeholders. NAPP has three principal objectives: 1. To strengthen national capacity for greater evidence-based policy processes in agriculture by increasing the capacity of Nigerian analysts to undertake and make widely available relevant evidence-based policy analysis; 2. To promote and foster informed policy dialogue among all stakeholders in the agricultural sector through an inclusive, transparent, and sustainable process at the country level, building blocks for a well-integrated and developed national policy system; and 3. To help federal and state governments improve their capacities to plan and implement effective policy analyses and programs, and demand and absorb policy research in their policy process. 3.1 Results Framework To achieve this goal and the principal objectives, the overall results framework (Figure 2) for the FTF Innovation Lab for Food Security Policy (FSP) project and the Country Development Cooperation Strategy (CDCS) provide the model for the results framework for the Project. The FSP results framework is itself consistent with that of FTF, the United States Government’s (USG) global hunger and food security initiative. The results framework adapted for this Activity is illustrated in Figure 3 below. It shows the pathways by which the Activity will achieve its goals of promoting more-inclusive, private sector-led 8 agricultural growth (which is the FTF strategy’s First Level Objective) and reducing poverty (which is the goal of the Country Development Cooperation Strategy, or CDCS). This framework is central to the management, monitoring, and evaluation of this Activity. Within the USAID CDCS, the FTF Agricultural Policy project is located under Development Objective (DO) 1, Broadened and inclusive growth, Intermediate Result (IR) 1.2: Business Environment Improved, Sub-IRs 1.2.1: Legal and Regulatory Environment Improved. Figure 1: USAID/Nigeria Economic Growth Results Framework (2015 -2019 CDCS) 9 Figure 2: FTF Nigeria Agricultural Policy Project Results Framework The NAPP results framework also includes one customized sub-sub IR and five strategic results (SR) across the activity components that will contribute to the sub-IRs. These tiers are arranged to illustrate the causal relationship between planned Interventions under the Activity and their expected impact and to identify the intermediate results that are critical to achieving activity objectives. Specifically, the framework demonstrates how planned interventions and deliverables will lead to expected outputs, outcomes, results, and eventual impact. Together the DOs, IRs, sub-IRs and SRs, identified in Figure 3, provide the framework for identification and implementation of activities designed to achieve the required results. 10 3.2 Agricultural Policy Project’s Strategy The NAPP uses a robust approach to enhance skills, training, and institutional capacities to meet the demands for policy analysis by the FMARD and the national food security and agricultural policy process. In order to achieve this, the project engages in training collaborations with FMARD, academic institutions and other key stakeholders. It strives to strengthen the links between various actors and to foster the development of a network of institutions that can independently interact during the policy process. The policy driven collaborative research and analysis component directly supports the knowledge needs of the policy process and is sub-divided into two sub-components: Research and policy analysis. The research component involves research teams composed of the Agricultural Policy project researchers and Nigerian researchers with priority topics identified in consultation with FMARD, state ministries of agriculture, development partners, and other key stakeholders. These researchers and their networks provide their expertise to inform policy debates with empirical evidence as they increase interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders). The policy analysis component, on the other hand, is more focused on demand driven and shorter-term policy analyses that originate from requests of FMARD, state ministries of agriculture and their development partners. The strengthening evidence‐based policy process and promoting impact component addresses policy impact through targeted communication and increased and targeted capacity building at universities and FMARD. Strategies will include directly contributing to the donor-government policy dialogue on agricultural policy, launching a policy brief series for the project, promote a “think tank” culture within the agricultural policy process through organizing various seminars and events targeted at all the actors in the process, training courses provided by the Agricultural Policy Project Activity team for FMARD and scholars, and provisions for selected Nigerian graduate students to take advanced courses at MSU. 3.3 Main Activities The NAPP planned to accomplish the SRs, IRs and activity objectives through an activity work program organized around three themes – capacity building, research and outreach. Examples of interventions funded by the Activity include: 1. Skill-enhancing training of staff from Nigerian partner institutions on methods and applications for research and policy analysis. The content of this training was developed in consultations with FMARD, Ahmadu Bello University (ABU), the University of Ibadan, and other key stakeholders to ensure their appropriateness in addressing the most serious gaps in knowledge, analytical and research approaches, and application tools in Nigeria. 2. Organizing and instituting a direct training collaboration between researchers at MSU and IFPRI with graduate student researchers and their supervisors in Nigerian Universities. A competitive annual selection of two or three Nigerian Postgraduate Students (one at the Masters Level and one or two at the PhD Level) was used to select graduate students for this program. These students and their research advisors form a research team with MSU and IFPRI researchers on the proposed research topics. Where possible (and to be encouraged), the research project will constitute part of the students’ thesis or dissertation. The Nigerian students attend technical courses (e.g., econometrics, research design and/or modelling techniques) at MSU for one semester for Masters-level students and one year for PhD students. Periodic team meetings were scheduled to discuss team findings and progress and facilitate more learning by junior researchers. 3. Nigerian research supervisors visiting the U.S. to participate in various activities to broaden their outlook and to facilitate the provision of feedback for their research students. The visits also 11 facilitate the interaction of the Nigerian professors with faculty at the department of Agricultural, Food and Resource Economics as well as faculty at the College of Agriculture and Natural Resources to broaden their outlook as well. The Nigerian professors collaborate with senior IFPRI researchers based in Washington, D.C. 4. Based on demand, organizing periodic training courses at Nigerian universities on various research methods and tools to reach a broader set of promising graduate students and young research professionals. Lecturers and students in Nigerian universities have access to all course materials developed. Additionally, as part of this, periodic training, lectures by visiting MSU and IFPRI staff are encouraged at the program’s collaborating institutions. This is a key long-term effort at institutional capacity building to facilitate the training of the next generation of policy researchers and analysts in Nigeria. The training courses, which cover policy analysis methods and tools, target a broad audience including the research community, practitioners in government (such as at FMARD and other ministries related to agriculture), Civil Society Organizations (CSO), Non￾governmental Organizations (NGO), private sector, and think tanks. 5. Organize major conferences to promote the visibility of the policy research and analysis being undertaken, based on project workshops and other activities. MSU and IFPRI have proposed organizing three major conferences during the period (2015-2020) which will bring together experts on selected issues in agriculture and economic development to provide guidance to Nigerian graduate students and young research professionals working in these areas. In addition, the Activity funds select graduate students to attend a national/regional conference and present their work. Such conference participation will broaden the reach of these conferences and increase interaction between actors at various institutions interested in agricultural policy. The interactions are intended to strengthen the links between such actors and foster the development of a network of institutions that can independently interact during the policy process. 6. Establishment of formal ties with two or more Nigerian institutions in order to develop appropriate skills, knowledge and tools needed to strengthen the credibility and relevance of evidence generated by policy analysis and research for informing policy. In addition to two already identified institutions (ABU and University of Ibadan), additional institutions are to be selected in line with FTF focus States and their interest and/or relevance to project research topics and in order to broaden the project’s interaction with institutions across Nigeria’s diverse agro￾ecological and/or geopolitical systems. Specific activities are necessary to create the appropriate enabling environment to achieve the IRs and sub-IRs as well as the overall goal of the project. The NAPP was designed to address the following development challenges and needs: 1. The need for evidence-based policy and program actions to address the challenges of agricultural transformation, and forge long-term solutions to reduce poverty in rural areas and chronic food insecurity; 2. The need for improving agricultural and economic growth-related policies and increasing public and private investments to achieve food security goals; 3. The need to strengthen policy institutions and processes through coordinated actions that support greater engagement and participation by civil society and private sector participation in policy systems; and, 4. The need to have impact at scale. 12 3.4 Critical Challenges/Assumptions There were several critical assumptions for NAPP to achieve its targets and objectives. The first major assumption is that the policy-making environment in Nigeria remains relatively stable and is not subjected to major upheavals often associated with shifts in political power resulting from undemocratic forces. The second is that the government, policy makers and stakeholders targeted by the Activity are receptive to new ideas and responsive to presented evidence and are committed to implementing and enforcing the outcomes of policy research and analysis. In other words, the Results Framework relies on there being political will and support for the policy change agenda and that the project will receive cooperation and participation of targeted stakeholders in program activities and its outputs. While the above factors pose serious challenges as Nigeria is new to broad policy reform as, until recently, much of Nigeria’s policies were concerned with changing the urban landscape and the fortunes of urban dwellers.3 The Agricultural Policy project Activity, thus, will provide timely evidence to improve the local capacity to conduct evidence-based policy analysis and advocacy to overcome these challenges. 3.5 Performance Indicators The NAPP performance indicators for monitoring and reporting purposes are listed in Table 1. These 10 indicators were identified by reviewing the performance indicators developed for Feed the Future (FtF), as used in the FtF Monitoring System (FTFMS), and those developed for the global FSP project. These indicators in this list track the delivery of outputs and outcomes related to project efforts to more broadly strengthen evidence-based policy processes for agriculture and food security in Nigeria. These indicators track the delivery of outputs and outcomes related to the Activity’s efforts to more broadly strengthen evidence-based policy processes for agriculture and food security in Nigeria. Table 1: Agricultural Policy Project Performance Indicators # Indicator Title Indicator ID in FTFMS Unit of Measurement 1 Number of high-quality research reports published. Custom Number 2 Number of participants attending project-organized research and policy events. Custom Number 3 Number of agricultural and nutritional enabling environment policies analyzed, consulted on, drafted, or revised, approved, and implemented with USG assistance (RAA). EG.3.1-12 (Custom) Number 4 Number of individuals participating in USG food security programs. EG.3.2 Number 5 Number of individuals who have received USG supported degree-granting agricultural sector productivity or food security training. EG.3.2-2 Number 6 Percent of USG-assisted organizations with improved performance [IM-level]. CBLD - 9 Percentage 7 Number of agriculture policy communications developed and/or written for stakeholder consumption. Custom Number 3 Ering, Simon; Odey, Judith; Eteng, Out; and Esther Patrick Archibong. (2014). Rural Development Policies in Nigeria: A Critical Appraisal. International Journal of Education and Research. 2:9, Sept. 2014. 13 # Indicator Title Indicator ID in FTFMS Unit of Measurement 8 Number of public private advocacy dialogues focused on policy that supports private sector investment Custom Number 9 Index (or scorecard) of quality of agriculture and food security policy processes in Nigeria, as measured by stakeholder evaluation to capture level of satisfaction and confidence. Custom Average Score 10 Index (or scorecard) of quality of the institutional architecture for agriculture and food security policy processes in Nigeria, as measured by stakeholder evaluation survey to capture level of satisfaction and confidence. Custom Average Score SECTION 4: EVALUATION PURPOSE The primary purpose of this final performance evaluation is to determine whether the assistance provided by USAID/Nigeria through the Agricultural Policy Activity achieved the stated development objectives, and to understand the lessons learned from this particular Activity in Nigeria, taking into account the findings and recommendations of the mid-term performance evaluation. It is intended to provide an independent examination of the overall progress and accomplishments of the Activity. The evaluation should provide a detailed picture of the major accomplishments and weaknesses of the Activity since its inception, and determine how its successes can be sustained. This final performance evaluation will provide USAID/Nigeria, its implementing partners, and agriculture sector stakeholders with data on outcomes and achievements. The evaluation should also elucidate lessons learned and include specific recommendations to USAID/Nigeria on how the Activity interventions can be sustained and scaled by other FTF activities, partners, private sector firms, or by the Government of Nigeria (GON). 14 SECTION 5: EVALUATION QUESTIONS In assessing the achievements of the NAPP objectives, the following key evaluation questions are to be answered. 1. Gender/youth: To what extent have women and youth (15 – 29 years) benefited from policy intervention undertaken by the activity? ● Which approaches have been most effective in reaching women and youth? ● How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? ● How have women and youth been part of the dialogue in the agriculture sector in developing national policy systems? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? 2. To what extent has the direct training of select graduate students in Nigerian universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? ● How have policy issues chosen by the Activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? 3. To what extent has the capacity of local research partners been strengthened to inform policy debates and implementation with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? ● How well are local research partners providing needed support to relevant State Ministries in the formulation and implementation of policy processes? ● How often do policy actors interact to advance policy processes at national and state levels? 4. To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? 5. What measures has the Activity put in place to ensure: ● Sustainability of the interventions and the changing needs of the policy environment at the national/state levels: ● Policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs. SECTION 6: EVALUATION METHODOLOGY The Evaluation Team will use a mixed methods approach, quantitative and qualitative research methods, involving the use of primary and secondary data to complete a thorough and effective final performance evaluation. The procedures for primary data collection are twofold. a. Virtual meetings – Virtual meetings will be set up with all categories of stakeholders during which the required data will be collected through FGDs and KIIs based on pre-arranged schedule of meetings. Categories of respondents scheduled for these meetings are USAID/Nigeria, Implementing Partners (MSU, IFPRI),National Advisory Committee, CSOs/Associations, Project Scholars, Scholars’ Advisors, the media, FMARD and state partners - s- Universities and State Ministries (Ebonyi, Delta, Kebbi, Kaduna, Oyo, Cross River, Niger, Benue). b. Use of structured questionnaires to elicit information from project beneficiaries who participated in training workshops and conferences. This is with a view to eliciting information regarding the extent to which the outcomes of the project have been achieved in line with relevant evaluation questions. The questionnaires will be administered through telephone conversations. DevTech has concluded 15 arrangements to engage experienced enumerators to conduct the telephone interviews following a short period of training. The evaluation team will break into two groups for the purpose of FGD/Virtual meetings. It is expected that DevTech will provide each group with necessary technical and administrative support for setting up and managing the meetings. 6.1 Document Review Various documents and reports have been reviewed at the initial stage (July 20th to 24th, 2020) of the evaluation to enrich the evaluation process technically. The documents reviewed include technical proposal, original project agreement and amendments; USAID/Nigeria strategy document; Feed the Future Project Appraisal Document, activity annual work plans, quarterly and annual progress reports, monitoring and evaluation plan, data quality assessment reports (DQAs), activity technical studies; mid-term evaluation report and other relevant documents such as the specification of the activity deliverables to be provided by USAID/Nigeria and MSU/IFPRI. 6.1.1 Focus of Document Review Essentially the Evaluation Team (ET) endeavored to address the following issues in the review. 1. Categorize the available documents and identify gaps. 2. Review the NAPP implementation plan including the periodic revisions and determine whether the activities therein are consistent with the evaluation questions. 3. Determine whether the project outcomes/impacts/benefits are adequately captured in the quarterly and annual reports and appropriately reflected in the evaluation questions. 4. Indicate whether the evaluation time frame presented in the SOW document is consistent with the contract documents and prevailing circumstances - especially with regard to timing and scheduling of activities. 5. Identify the relevant categories of respondents to be involved in the evaluation for the purpose of data gathering based on the understanding of stakeholders' roles and responsibilities and type of project beneficiaries as indicated in various documents. 6. Determine on the basis of 5 above whether the category of respondents to be covered as indicated in the SOW document is adequate. Indicate other relevant category of respondents if necessary. 6.1.2 Main Observation from Document Review: Gaps in REQUIRED Documents The baseline information, i.e. survey report, etc., was not part of the initial set of documents submitted to the ET but on request one was made available during the week. We now have 2 rounds of stakeholders’ survey – 2016 and 2018 to monitor progress in improving: 1) the quality of the agriculture and food security policy processes in Nigeria and, 2) the quality of the institutional architecture within which those processes proceed. The 3rd round is scheduled for 2020; but there is no indication as to whether this has been finalized. Available information indicates that the process is underway but the completion date is still unknown. It is expected that the survey report will be made available to the evaluation team while they are drafting the evaluation report by mid-September. Although the analysis of these survey reports will be useful in tracking progress, the baseline information is restrictive since no data relate to several other performance indicators stipulated for the assessment of the extent to which Activity objectives have been 16 achieved. The implication is that it will not be possible to compare the performance indicators at the end line with the situation at the beginning of the project. To address this, the content of the available monitoring surveys combined possibly with the results of the 3rd round surveys will be examined to identify the areas relevant for doing justice to the evaluation questions. We shall also come up with lessons in this regard to guide future interventions by USAID and partners engaged to implement this type of project. 6.2 Data Collection Tool s and Protocols The team will conduct interviews and focus group discussions with a variety of stakeholders including USAID staff, Activity beneficiaries, government staff, implementing partner staff, and other key donor partners. The required data will be collected using key informant interviews (KIIs) guides and other tools depending on the category of respondents. There will be 10 main groups of key informants for this evaluation: 1. USAID/Nigeria Staff; 2. Implementing partner (MSU and IFPRI) staff; 3. National Advisory Committee Members; 4. Government partners (Federal includes FMARD etc. while State Partners are drawn from Ebonyi, Oyo, Kebbi, Benue, Delta, Niger, Kaduna and Cross River states); 5. Research collaborators; 6. Project scholars - Universities involved in the Activity in USAID/Nigeria’s focus states; 7. Project scholars’ advisors; 8. Representatives of CSOs/Associations; 9. Representatives of the private sector; 10. Representatives of the media. The tools and protocols to be used as KII guides are provided in Annexes 1.1 to 1.10 respectively. The collection of data from project beneficiaries will be done through telephone interviews, using appropriate CAPI tool such as KoBoCollect or ODK. It is expected that DevTech will assist with telephone support services, data entry and export into Excel files for analysis by the ET. The multiplicity of data sources and triangulation that will be adopted in the evaluation seek to ensure data validity throughout the evaluation process. The ET will apply robust data analysis techniques4 to draw evidence-based conclusions and recommendations. Data collected from key informant interviews will be compared with the results from the Activity documents and reports to validate some key findings. 6.3 Sample Size and Selection of Respondents 6.3.1 Sampling Strategy The database consists of a total of 6,610 Activity participants stratified into 80 key informants, 431 conference participants and 6,099 training workshop participants. Each category of participants have been further stratified by gender with the key informants consisting of 58 males and 22 females while the 4 Atlas.ti will be used to analyse the qualitative information collected. This will complement the analysis of information obtained from secondary sources; especially the various quarterly and annual reports. 17 conference participants comprise 344 males and 87 females. The trainers’ category is the largest; consisting of 4521 males and 1578 females. 6.3.2 Sample Size The size of the sample to be included in the evaluation will be calculated using RaoSoft Software.5 This statistical approach will be used to ensure a representative distribution of the sample frame across all the states covered by the Activity and will allow the study to generalize its results to the population. A confidence level of 95 percent and 5 percent margin of error will be used to select a sample size of 67 key informants, 204 conference participants, and 362 training participants (Table 2), making a total of 633 respondents to be included in evaluation interviews. This represents about 10 percent of the total population of project participants in the database. In selecting the sample of respondents, the Excel-based random sampling procedure will be used. Details of the potential respondents randomly selected from each category are presented in Annex 2, Annex 3, and Annex 4 for the key stakeholders for KIIs, phone interviewees - training participants and phone interviewees - conference participants respectively. Table 2: Sample Size by Gender and Category of Participants Gender Total Percent of Total Sample Size* Category (a) Project Partners/Key informants Male 58 72.5 49 Female 22 27.5 18 Total 80 100.00 67 Category (b) Conference Participants Male 344 79.81 163 Female 87 20.19 41 Total 431 100.00 204 Category (C) Training Participants Male 4521 74.13 268 Female 1578 25.87 94 Total 6099 100.00 362 *Based on the 95% confidence interval and 5% margin of error 5 http://www.raosoft.com/samplesize.html 18 6.3.3 Secondary Data Collection Secondary data will be collected from relevant stakeholders such as MSU, IFPRI and FMARD. Also, documents will be assessed online from MSU’s and IFPRI respective websites where necessary. 6.4 Mapping of Evaluation Questions with Category of Stakeholders The five evaluation questions earlier presented are to be addressed in accordance with relevance of stakeholders to particular aspects of the Agricultural Policy project Activity. This fact is illustrated in Table 3 below. On the basis of correspondence between the evaluation issue and category of stakeholders, research instruments will be prepared to drill the interview exercise down to necessary sub-questions as shown in the collection protocols (Annex 1). Table 3: Evaluation Questions and Target Respondents EVALUATION QUESTIONS RELEVANT STAKEHOLDERS 1. Gender/Youth: To what extent have women and youth (15 – 29 years) benefited from policy intervention undertaken by the activity? ● Which approaches have been most effective in reaching women and youth? ● How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? ● How have women and youth been part of the dialogue in the agriculture sector in developing national policy system? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? USAID MSU IFPRI Advisory Committee members Partner States Project scholars Scholars’ advisors Training participants 2. To what extent have the direct training of select graduate students in Nigerian Universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? ● How have policy issues chosen by the Activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? MSU IFPRI FMARD Partner States Project scholars Scholars’ advisors Training participants and Activity Advisory Committee 3. To what extent has the capacity of local research partners been strengthened to inform policy debates with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? • How well are local research partners providing needed support to relevant State Ministries in the formulation and implementation of policy processes? ● How often do policy actors interact to advance policy processes at national and state levels? Research Collaborators (including Project scholars and Scholars advisors) MSU, IFPRI, FMARD Partner States Activity Advisory Committee 4. To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? Representative of Media MSU, IFPRI, FMARD 19 EVALUATION QUESTIONS RELEVANT STAKEHOLDERS Partner States 5. What measures has the Activity put in place to ensure: ● Sustainability of the interventions and the changing needs of the policy environment in the national/state levels: ● Policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs. USAID MSU/IFPRI FMARD Partner States Other Donors 6.5 Proposed Quality Control Mechanism for Data GATHERING, Analysis AND Oversight Both primary and secondary data will be collected for this evaluation. The secondary data will be obtained from the Activity units, Activity beneficiaries and periodic reports (e.g., quarterly and annual reports). For ease of access, formal letters requesting such data will be obtained from MSU/IFPRI and presented by the Evaluation Team to appropriate authorities concerned with the release. By so doing, there will be no holding back of critical information required for successfully carrying out this evaluation. The primary data will be obtained through KIIs and FGDs. As a quality control mechanism, Evaluation Team members will conduct all KIIs and FGDs. To minimize waste of time during interview, all responses will be recorded via a digital recording device with the respondents’ consent. This will make programming of appointments efficient and fact-checking possible during subsequent stages of the evaluation. The team members conducting the interviews will take handwritten notes on emerging key issues relating specifically to the evaluation questions and sub-questions. This will be necessary for better interpretation of the data. Recording is embedded in Zoom Cloud which will, therefore, be the preferred virtual meeting platform for the identified stakeholders relevant for FGDs. 6.6 Data Analysis The data collected will be analyzed quantitatively and qualitatively6. The qualitative analysis will be guided by the main evaluation questions regarding the ways in which women have benefited from national research capacity strengthening, improvement in policy research capacity of next generation of researchers, improvement in capacity to plan and implement effective policy analyses as well as improvement in policy dialogue and communication. The quantitative analysis will address project implementation performance and extent to which project objectives have been achieved. Some of the data obtained will be subjected to descriptive analysis using suitable descriptive statistics (e.g., mean, percentages, etc.) focusing on performance indicators as variables and the proportion of respondents attesting to the relevance of the activities and the benefits derived. The way the information supplied by respondents varies across groups of stakeholders will also be tabulated and described as a way of assessing the project’s benefits and the different ways they are distributed among participants. To ensure this, questions posed to proposed respondents have been grouped by the relevant evaluation questions (see section 8). The analytical approach in respect of each of the EQs is highlighted in Table 4. 6 Data generated from the KIIs will be coded and analyzed using the ATLAS.ti software tool for qualitative analysis. Themes from the respondents’ answers will be coded and analyzed, and afterwards using STATA, cross-tabulations and bar charts will be generated to show the percentage distribution of responses to quantitative data from women that benefited from national research capacity strengthening, improvement in policy research capacity of next generation of researchers, improvement in capacity to plan and implement effective policy analyses as well as improvement in policy dialogue and communication. 20 Table 4: Analytical Approach and Data Collection Sources to Answer Each Evaluation Question EVALUATION QUESTION DATA COLLECTION SOURCE ANALYTICAL APPROACH 1. Gender/youth: To what extent have women and youth (15 – 29 years) benefited from policy intervention undertaken by the activity? ● Which approaches have been most effective in reaching women and youth? ● How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? ● How have women and youth been part of the dialogue in the agriculture sector in developing national policy system? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? Document review; KIIs with MSU/IFPRI, Government (Federal and state), private sector, media, CSOs, project scholars, scholars’ advisors, training and conference beneficiaries and Activity Advisory Committee members. Both qualitative and quantitative analytical tool will be used. The ways in which women have benefited from national research capacity strengthening will be analyzed qualitatively. There will be quantitative analysis of project implementation performance and extent to which project objectives have been achieved. Some of the data obtained will be subjected to descriptive analysis using suitable descriptive statistics (e.g., mean, percentages, etc.) focusing on performance indicators as variables and the proportion of respondents attesting to the relevance of the activities and the benefits derived from the interventions. The way the information supplied by respondents varies across groups of stakeholders will also be tabulated and described as a way of assessing the project’s benefits and the different ways they are distributed among participants. 2. To what extent have the direct training of select graduate students in Nigerian Universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? ● How have policy issues chosen by the Activity in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? Document review; KIIs with MSU/IFPRI, Government (Federal and state), private sector, media, CSOs, project scholars, scholars’ advisors, training and conference beneficiaries and Activity Advisory Committee members. Qualitative analysis of the improvement in capacity to plan and implement effective policy analyses as well as improvement in policy demand and utilization will be analyzed qualitatively. Quantitative analysis of the benefits derived from the training programs including variations in the responses from different categories of stakeholders 3. To what extent has the capacity of local research partners been strengthened to inform policy debates with empirical evidence or increase the interaction between the supply of information (Nigerian researchers) and the demand for information (FMARD and other stakeholders)? ●How well are local research partners providing needed support to relevant State Ministries in the formulation and implementation of policy processes? Document review; KIIs with MSU/IFPRI, Government (Federal and state), project scholars, scholars’ advisors, and Activity Advisory Committee members. Qualitative analysis of improvement in policy research capacity of next generation of researchers Quantitative analysis of the variations in the responses from different categories of stakeholders 21 EVALUATION QUESTION DATA COLLECTION SOURCE ANALYTICAL APPROACH ●How often do policy actors interact to advance policy processes at national and state levels? 4. To what extent has the capacity of the media been increased to undertake informed reporting on agricultural policy issues? Document review; KIIs with MSU/IFPRI, media Qualitative analysis of improved capacity for policy engagement and communication 5. What measures has the Activity put in place to ensure: ● Sustainability of the interventions and the changing needs of the policy environment in the national/state levels: ● Policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs. Document review; Interviews with Activity staff; KIIs with government, and other stakeholders on sustainability of the Activity results Comparison of sustainability measures/strategy developed by the Activity with what has been implemented. Comparison stakeholders’ sustainability options with reality and with a view to determining how the desired change will be achieved and sustained. 6.7 Risks There are always risks in doing any research and an Evaluation is often more prone to them than controlled research. Some potential risks: The limited timeline will ultimately dictate how many people the ET can contact. Key informants may not be forthcoming and may not provide honest, unbiased feedback. To mitigate against these risks, the Evaluation Team will closely communicate with each other, with MEL Activity team, and with the MSU/IFPRI team. Other remedial measures will include prior telephone contacts with potential respondents, scheduling of appointments, and sending prior notices electronically (through emails) to potential respondents across the states. In view of the need to schedule appointments and to prevent reluctance on the part of some respondents, the timing of the interviews has to be carefully considered especially when respondents outside Nigeria (MSU and IFPRI staff) are involved. The greatest risk at this moment is the COVID-19 pandemic and the associated restrictions to movement imposed by the government across the country. Although the restrictions to movement are gradually being eased, infections rates across the country are still on the rise. Consequently, all beneficiary interviews will be conducted virtually, using the methods stated above. It is important to note that this method will have no impact on the quality of the data collected, as the only difference between virtual data collection and in-person interviews, is meeting physically with respondents. Also, the ET has extensive experience collecting information using this approach. Finally, the pandemic will make the physical meeting of the ET almost impossible, during data collection. This will escalate communication costs among the ET and may prevent their physical presence in Abuja for a considerable length of time during the period of this evaluation. Under these circumstances, organizing in-brief and possibly debrief sessions will require additional technical and administrative support from the MEL Activity to provide the presentation logistics. 22 6.8 Guiding Principles and Values Ethical considerations are of the utmost importance and the evaluation will be conducted with integrity and sensitivity, showing the highest level of respect to all participants. Informed consent will be obtained from all respondents prior to their involvement in the data collection process. Respondents will be referred to by role/job title and location, if necessary, but only where this information would not reveal the identity of the respondent(s). The team shall adhere to the “Do No Harm” principle placing the protection of human subjects as our highest priority. The team will be required to follow DevTech and USAID’s security advice. The team shall ensure the confidentiality of data, respect the privacy of all individuals concerned and make all data collected available to DevTech and USAID in a usable format. SECTION 7: WORK SCHEDULE The evaluation will be implemented according to the division of responsibilities listed below and the step￾by-step work plan outlined in the Table 5. 7.1 Team Lead Responsibilities Team Leader: The Team Leader will be responsible for the overall management of the evaluation, including coordinating and packaging the deliverables. The Team Leader will develop the outline for the draft report, present the report, and after incorporating USAID/Nigeria staff comments, submit the final report to USAID/Nigeria through DevTech Systems Inc. In addition to the responsibilities outlined above, the Team Leader will: 7.1.1 Preparations 1. Finalize and negotiate with USAID/Nigeria the team’s work plan. 2. Establish assignment roles, responsibilities, and tasks for each team member. 3. Ensure that the logistics arrangements in the field are complete. 7.1.2 Management 1. Take the lead on preparing, coordinating team member input, submitting, revising and finalizing the assignment report. 2. Manage the process of report writing. 3. Manage team coordination for the virtual meetings. 4. Coordinate the workflow and tasks and ensure that team members are adhering to the schedule. 7.1.3 Communication 1. Ensure harmony within the team. 2. Serve as primary interface with USAID/Nigeria and serve as the spokesperson for the team. 3. Debrief USAID/Nigeria as the evaluation progresses, and organize a final debriefing. 4. Keep USAID/Nigeria apprised of progress challenges, work changes, schedule of virtual meetings with various categories of stakeholders and report preparation via phone conversation or email at least once a week. 5. Serve as primary interface with USAID/Nigeria for the submission of draft and final reports, and deliverables to USAID/Nigeria. 6. Make decisions in conjunction with USAID/Nigeria about the safety and security of the team. 23 7.1.4 Direction Assume technical direction lead, as required, in order to ensure quality and appropriateness of assignment and report content. Other team members will include: 1. Host-country national Senior Evaluation (Agricultural Policy) Specialist: S/he must demonstrate good interpersonal and diplomatic skills as well as strong report writing and analytical skills and an excellent oral communication skill in English; 2. Host country mid-level Agriculture Economist; and 3. National Evaluation Specialist. Table 5: Timeline and LOE for the USAID/Nigeria Agricultural Policy Program Activity Final Evaluation Item Period of Performance Number of Days Team Leader Senior Evaluation Specialist (Policy) Agricultural Economist Local Evaluation Specialist Recruitment and Formalizing team members engagement June 2020 Desk Review of background documents and initial preparation work towards draft data collection tools, sampling, and workplan July 20 – 24 5 5 5 5 Preparation and submission of draft workplan & draft data collection tools (protocols) July 27 – 29 3 3 3 3 Submission of Workplan and evaluation design /protocols July 31 0 0 0 0 USAID provides feedback on work plan and evaluation design/ protocol August 3-14 0 0 0 0 Arrival (Abuja Nigeria) for ET consultants August 16 0 0 0 0 In brief/Team Preparation Meeting with USAID August 17-18 2 2 2 2 Reviewing evaluation tools to incorporate USAID comments August 19 - 20 2 2 2 2 Virtual Data Collection August 21 – September 15 18 18 18 18 Analysis of results, Report writing and preparation of ppt for USAID debrief September 16- 23 6 6 5 5 Debrief with USAID (preliminary findings) September 24 1 1 1 1 Finalizing evaluation report September 25 – October 5 7 7 7 7 Submission of draft report (to MEL) October 5 0 0 0 0 Local Travel home for Consultants October 6 0 0 0 0 Submission of draft report (to USAID) October 16 0 0 0 0 24 Item Period of Performance Number of Days Team Leader Senior Evaluation Specialist (Policy) Agricultural Economist Local Evaluation Specialist USAID reviews and comments on final draft evaluation report October 19 – 30 0 0 0 0 Submission of report back to MEL with USAID feedback October 30 0 0 0 0 Team addresses USAID comments and finalizes the report November 2 – 9 6 5 4 4 Team submits reviewed report to MEL for additional review and send-off for copy editing November 9 -20 0 0 0 0 Submit 2-3 Page Summary November 20 1 Submission of final report to USAID November 20 TOTAL LOE 51 49 47 47 1 ANNEX 3. DATA COLLECTION TOOLS AND PROTOCOLS This section outlines the specific data collection protocol for each category of stakeholders and the tools that will be used to obtain the information. 1.1 KEY INFORMANT INTERVIEWS – USAID/NIGERIA STAFF DATA COLLECTION PROTOCOLS – MLE 1 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gende r: Male □ Female □ USAID Key Informant Interviews A. PROJECT IMPLEMENTATION PROCEDURE 1. Were the baseline conditions well-articulated and adequately captured by the performance indicators? 2. Why were performance indicators changing so frequently during the life of the project? 2 3. Were you satisfied with the appropriateness (SMART-NESS) of the performance indicators? Specific, Measurable, Achievable, Relevant ant Time-bound. 4. Were you concerned that some of the indicators were not addressed during the life of the project? B. PROJECT TRACKING AND MONITORING OF PROGRESS 5. What project tracking tools did you deploy for the NAPP? 6. What is the dissemination plan for Evaluation Reports? 7. Some stakeholders (ADAN) complained they never saw the MTER. Did you consider it necessary to make the MTER available to project stakeholders? 8. Were quarterly and annual project implementation reports (PIRs) regularly and timely submitted? 9. What lessons were derived from the PIRs and MTER? And how were the lessons used to steer the project in a positive direction? Were changes made to project implementation? C. PROJECT PARTNERSHIP ARRANGEMENTS 10. Were the partnership arrangements properly identified and roles and responsibilities properly negotiated prior to project approval? [At issue here is the termination of appointments of facilitators – appointed midway and disengaged prematurely] 11. Are you satisfied with the spread of NAC among key stakeholders? Specifically, are state partners well represented? D. LEVERAGING PROJECT FINANCING 12. Did you envisage the existence of leveraged resources for effective implementation of NAPP? [Consider stakeholders’ contribution or associated financing by other donors] E. SUSTAINABILITY 13. What challenges have you experienced in sustaining policy-related interventions in Nigeria? 14. What measures were put in place to ensure sustainability of interventions under NAPP? 15. Which interventions do you think should be sustained? Why? 16. What are your expectations if these interventions are sustained? 17. What sort or policy dialogue came up during the implementation of NAPP? Who were the actors and did it play out? 18. The risks to the sustainability of the achievements and outcomes of NAPP can be social, economic, financial, political and administrative. Which of these do you think are of great concern going forward? 19. How will you rate the likelihood of sustainability of NAPP? (i) Likely [4] negligible risk (ii) Moderately likely [3] moderate risk (iii) Moderately unlikely [2] substantial risk 3 (iv) Unlikely [1] severe risk 1.2 KEY INFORMANT INTERVIEWS – ACTIVITY IMPLEMENTERS – MSU/IFPRI DATA COLLECTION PROTOCOLS – MLE 2 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul Delucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gender: Male □ Female □ MSU/IFPRI Key Informant Interviews 1. Why was there no baseline survey in the true sense of the tool? 2. When will the result of the 3rd monitoring survey be ready for dissemination? And possibly for our use? 3. Why was the monitoring survey limited to 2 broad indicators/issues? 4 4. What is the idea behind collaborative research under the project? How were the research topics decided/selected? Who participated? How were the findings utilized? 5. Why do you have Research Collaborators in some states and none in others? 6. Why were facilitators contracted in 2018? What led to the termination of their contracts? 7. Did the project undergo any changes as a result of recommendations from the mid-term evaluation? Or as a result of other review procedures? Explain the process and implications. 8. What is the most important lesson you learn from the MTER and how impactful has it been in the second stage of implementation of the project? 9. Were there measures put in place to ensure sustainability of NAPP interventions? 10. Which interventions do you think should be sustained? Why? 11. What challenges do you foresee in sustaining the policy related interventions under NAPP? 12. What is your expectation if these interventions are sustained? 13. What sort or policy dialogue came up during the implementation of NAPP? Who were the actors and did it play out? 14. The risks to the sustainability of the achievements and outcomes of NAPP can be social, economic, financial, political and administrative. Which of these do you think are of great concern going forward? 15. How will you rate the likelihood of sustainability of NAPP? (i) Likely [4] negligible risk (ii) Moderately likely [3] moderate risk (iii) Moderately unlikely [2] substantial risk (iv) Unlikely [1] severe risk 5 1.3 KEY INFORMANT INTERVIEWS – NATIONAL ADVISORY COMMITTEE DATA COLLECTION PROTOCOLS – MLE 3 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gender : Male □ Female □ National Advisory Committee Key Informant Interviews Gender and Youth 1. To what extent have women and youth (15 – 29 years) benefited from policy intervention undertaken by the Project? - Which approaches have been most effective in reaching women and youth? - Was there a fair representation of youth and women in the program? If Yes, how so? - What do you think would have been the best approach to ensure increased women and youth participation in FTF program? - Why do you think it was important to involve women and youth in the program? 6 - How have women and youth benefited from the program? - What mix of women and youth would you recommend to yield the most impact for future programs in the future? Please explain this. 2. How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? - What research skills have been acquired by women who participated in the Project to strengthen capacity? - What changes (if any) do you feel the Project has made to women’s access to relevant up-to-date research and research programs? - What more do you think is needed to build more policy research capacity for women? 3. What effect do you think the trainings and workshops organized by the Project has had on women and youth? - Which of the project’s activities do you think contributed most to enhancing women and youth understanding of the country’s main agriculture policies? - Have women and youth become more aware of agriculture policy issues as a result of the Project’s activities? How so? - In what ways have women and youth become involved in agriculture policy dialogues in Nigeria as a result of the Project? - Do you think women and youth now possess the skills and knowledge required for agriculture policy dialogue in Nigeria after the trainings and workshops in programs? 4. Are there agriculture policy issues concerning women and youth that were not addressed by the Project? What issues? - Do you recommend that issues be included in future policy related capacity building programs? Why? 5. What do you think should be done to improve youth and women participation in agriculture policy dialoguing in Nigeria? - Are there currently national opportunities in agriculture policy dialogue for women and youth that can be leveraged? What are these? Trainings 6. How do you think the graduate scholar program has strengthened the capacities of the next generation of policy researchers in Nigeria? - Are you satisfied with results from the graduate scholar program? Are you satisfied with the quality of training and workshop delivery at MSU for the graduate scholars? - What is your impression about the understanding and application of policy analysis by graduate scholars as demonstrated through publications, trainings and presentations in conferences? - What course(s) offered and project activities carried out do you think contributed most to enhanced the scholars’ understanding of policy research? - What changes (if any) do you think should be made to the scholars’ program to improve its effectiveness in future projects? - In your opinion how have the skills and knowledge acquired by graduate scholars helped in improving research in Nigerian universities and enhancing capacity for national policy analysis? 7. Have the policy issues chosen by the Project had effect in transforming the agriculture sector? 7 - What effect have the trainings, workshops and conferences had on policy analyses by the participating researchers and graduate scholars? - In your opinion, were they training and workshop designs sufficient to for participants to know how to plan and implement policies? Capacity of Local Researchers 8. In your opinion, how has the capacity of local researchers been strengthened to inform policy debates? - What was the level of participation of local researchers in the project’s training workshops and dissemination activities? - How confident are you in the ability of local research partners to inform policy debates with empirical evidence after their participation in the program? - Are there any courses, activities or workshop designs you would recommend to further strengthen local research capacity to inform policy debates with empirical evidence? Sustainability 9. What measures did the Project put in place to ensure sustainability of the interventions amidst the changing needs of the policy environment at the national/state levels? - Which interventions do you think are likely to be sustained? Why? - Do you anticipate any challenges for the sustainability of policy related interventions? - How do you think these interventions will continue after the Project? 10. What measures did the Project put in place to ensure policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs? - Which courses were designed and executed in the program that facilitated policy dialogue with USAID/Nigeria’s FTF and other donor programs? 11. Do you anticipate a continuous dialogue between policy actors and stakeholders after the project ends? Why or why not? 1.4 KEY INFORMANT INTERVIEWS – FEDERAL AND STATE MDAS DATA COLLECTION PROTOCOLS – MLE 4 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. 8 Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gender: Male □ Female □ MDAs (Federal and State) Key Informant Interviews 1. In general, what was your experience with Nigeria Agricultural Policy Project (NAPP)? Gender and Youth 2. What do you feel about the participation of women in NAPP interventions? - Were there challenges in ensuring that women benefitted from the Project’s interventions? - What strategies were employed to ensure adequate participation of women and youth in the project’s intervention? Which of these strategies did you consider to be the most effective? Why? - How appropriate are these interventions to the career needs of women and youth who participated in the interventions? - Did the project help to strengthen the capacity of women in research? How? - What changes has the Project made in the participation of women in policy processes and agriculture policy dialogue? Trainings 3. What was your experience with the training(s) you participated in under NAPP? - Did the trainings have any effect on your views on/ ability to, conduct (directly or indirectly) policy research and analysis? What effect? 4. What effect have the policy issues chosen (through research, training and conferences) by the Project had on your capacity to plan and implement policy analyses and programs? - What effect have the policy issues had on the agricultural sector (local or National)? 5. What effect have the policy issues chosen (through research, training and conferences) by the Project had on your capacity to demand /or absorb policy research in the policy processes? Capacity of Local Researchers 6. Were you involved in any debates/dialogues to inform policy with empirical evidence, organized by NAPP? If yes, give instances - How has your capacity been strengthened in this regard? 9 - What do you consider to be the most critical constraints to building your capacity for informed policy debates? - How has the project impacted the policy process from the use of collaborative research results? Sustainability 7. Do you anticipate a continuous dialogue between policy actors and stakeholders even after the project ends? Please explain - Do you have formal ties with any institutions that have a mandate for developing policy for the government? 8. What do you suggest Future Ag Policy projects can do to improve the policy process for the Ag sector in Nigeria? 1.5 KEY INFORMANT INTERVIEWS – RESEARCH COLLABORATORS DATA COLLECTION PROTOCOLS – MLE 5 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: 10 State of residence: Gender : Male □ Female □ Age____________ __ Research Collaborators Key Informant Interviews 1. In general, what was your experience with Nigeria Agricultural Policy Project (NAPP)? Gender and Youth 2. What do you feel about the participation of women in NAPP interventions? - Were there challenges in ensuring that women benefitted from the Project’s interventions? - What strategies were employed to ensure adequate participation of women and youth in the project’s intervention? Which of these strategies did you consider to be the most effective? Why? - How appropriate are these interventions to the career needs of women and youth who participated in the interventions? - Did the project help to strengthen the capacity of women in research? How? - What changes has the Project made in the participation of women in policy processes and agriculture policy dialogue? Training 3. What was your experience with the training(s) you participated in under NAPP? - Did the trainings have any effect on your ability to, conduct policy research and analysis? What effect? - Do you have any plans to conduct independent research in the future? What plans? Was this because of your involvement in NAPP? Capacity of Local Researchers 4. Were you involved in any debates/dialogues to inform policy with empirical evidence, organized by NAPP? If yes, give instances - How would you say your capacity has been strengthened to inform policy debates? Have you been involved in similar policy dialogues independent of NAPP? - What do you consider to be the most critical constraints to building your capacity for informed policy debates? - What effect has your partnership with NAPP had on your interaction with MDAs (state and federal)? - How has your interaction with MDAs (federal and state) (or other policy stakeholders) changed in terms of your supply of information to them or their demand for information from you as a policy researcher? Please give examples. 5. Has your participation in NAPP improved your capacity to effectively conduct independent research? - How do you communicate your research results to policy makers? 6. How has the project impacted the policy process from the use of collaborative research results? - Were you involved in the Project collaborative research? if yes, what were they? - What is the impact that the participating in the collaborative research had on your career? 11 Sustainability 7. Do you anticipate a continuous dialogue between policy actors and stakeholders even after the project ends? Please explain - Do you have formal ties with any institutions that have a mandate for developing policy for the government? 8. What do you suggest Future Ag Policy projects can do to improve the policy process for the Ag sector in Nigeria? 1.6 KEY INFORMANT INTERVIEWS – PROJECT SCHOLARS DATA COLLECTION PROTOCOLS – MLE 6 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gender : Male □ Female □ Age____________ __ Project’s Scholars Key Informant Interviews 12 Gender and Youth 1. Did any of the research you undertook under the NAPP target women or youth? - What specific research targeted women or youth? - In what way did the research target women and youth? 2. How many activities were you involved in under the NAPP? - Which of these activities did you find the most useful? - Why/how was it useful to you? - In your opinion which of these activities were most beneficial to women and youth? Why so? 3. Do you feel the activities under the NAPP (such as collaborative research involving graduate students and trainings, policy dialogues, conferences etc.) have helped to strengthen the capacity of female researchers in Nigeria as a whole? - How have the activities/ interventions strengthened national capacity for female researchers? - Can you give examples of any work done by female researchers because of their improved capacities? - How have women benefitted from this capacity strengthening? - Have you participated in any dialogue for developing national agricultural policy, as a result of NAPP? What dialogues were these? - Did women and youth participate in these sessions? To what extent? What was their level of involvement in these dialogue sessions? - Did any of the sessions you attended identify or prioritize women or youth policy issues? Please give examples of what sessions this was done. Trainings 4. What was your experience with the Project’s capacity building activities? - Did you learn any new skill from the trainings? What skills? - Have you been able to apply these skills to your work or education? How? - How have the skills impacted your work or education? - Have you been able to pass on these skills to other colleagues/ students? How? 5. As far as you know what sort of experience have your colleagues who received similar trainings, had? 6. How relevant do you think the policy issues you have researched on, have been in improving the agricultural sector? - How many of these policy issues were presented to a wider body? To what sort of audience were these presentations made? - What was the outcome of the research/ presentation of the research? - As far as you know was the research used either locally or nationally? How? - As far as you know have there been any changes in the sector (in the state or nationally) because of the research? - Has there been a demand for similar research from government authorities/ other relevant stakeholders? Are you aware if any of these research demands have been met and used? 13 Capacity of Local Researchers 7. Do you feel your involvement in the NAPP has had any effect on your capacity to inform policy debates? - Has your work your work informed policy debates in the past (before NAPP)? - How did your engagement/ interactions with NAPP affect your capacity to inform policy debates? - In what way has your work informed policy debates that it did not do, before NAPP? 8. Have you been able to generate additional research (independent of NAPP) to inform relevant policies since your/ as a result of your interaction with NAPP? - What new research have you generated? - What agricultural policy did the research feed into? 9. As far as you know how have the capacities of other local researchers who have participated in any of the NAPP activities been impacted? Sustainability 10. Do you feel you have been appropriately equipped to continue to inform policy issues? - Do you have plans to continue to conduct relevant research to inform agricultural policy after the NAPP has ended? How do you plan to do this? - How will you decide on the policy issues? - How will you disseminate or use the findings of your research? - What systems have been put in place to continue interactions with government and other relevant stakeholders on topical agricultural policy issues? 1.7 KEY INFORMANT INTERVIEWS – SCHOLARS’ ADVISORS DATA COLLECTION PROTOCOLS – MLE 7 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: 14 Contact information: Office address: E-mail address: Telephone number: State of residence: Gender : Male □ Female □ Age____________ __ Scholar’s Advisors Key Informant Interviews Gender and Youth 1. To what extent were you involved in the selection of students for participation in the NAPP interventions (e.g. short-term training abroad, participation in conferences, or participation in trainings in the university)? - Did you have a preference for which of your students participated? Did your preference play any role in deciding which of your students were ultimately selected? What informed your choices? 2. On average how many university students were involved in the collaborative research teams formed by the NAPP project (IFPRI/ MSU)? How many females? How many youths? 3. Did any of the research teams target women or youth in their research? - What specific research targeted women or youth? - How did the research target women and youth? 4. In your opinion, which of the activities under the NAPP (such as trainings or subject matter targeting) were the most effective in reaching women and youth? - How useful were these approaches in reaching the number of women and youths who attended these events? - How useful were these activities in ensuring that a significant number of women and youth were target/end beneficiaries? 5. Do you feel the activities under the NAPP (such as collaborative research involving graduate students and trainings, policy dialogues etc.,) have helped to strengthen the capacity of female researchers in Nigeria as a whole? - How have the activities/ interventions strengthened national capacity for female researchers? - Can you give examples of any work done by female researchers as a result of their improved capacities? - How have women benefitted from this capacity strengthening? 6. How many youth /female students have participated in dialogues for developing national agricultural policy, as a result of NAPP? - What dialogues were these? - Were any of these your students? What was their level of involvement in these dialogue sessions? - Did any of the sessions identify or prioritize women or youth policy issues? Please give examples of what sessions this was done? 15 Trainings 7. What was your experience with the Project’s capacity building activities (exchange visits to MSU and trainings received at local universities)? - Did you learn any new skill from the trainings/ exchange visits? What skills? - Have you been able to apply these skills to your work/ has it affected your style of teaching in any way? How? 8. What impact have the trainings had on your students who received the trainings? 9. As far as you know what sort of experience have your colleagues who received similar trainings, had? 10. How relevant do you think the policy issues your research teams have researched on, have been in improving the agricultural sector? - How many of these policy issues were presented to a wider body? To what sort of audience were these presentations made? - What was the outcome of the research/ presentation of the research? - As far as you know was the research used either locally or nationally? How? - As far as you know were there any changes in the sector (in the state or nationally) because of the research? - Has there been a demand for similar research from government authorities/ other relevant stakeholders? Are you aware if any of these research demands have been met and used? Capacity of Local Researchers 11. Do you feel your involvement in the NAPP has had any effect on your capacity to inform policy debates? - How has your work typically informed policy debates (before NAPP)? - How has your engagement/ interactions with NAPP change your capacity to inform policy debates? - What effect has the collaborative development/ dissemination of policy issues had your profile as a researcher? Have government or other stakeholders contacted you to conduct similar research because of your interaction with NAPP? 12. Have you been able to independently generate additional research to inform relevant policies as a result of your interaction with NAPP? - What new research have you generated? - What agricultural policy did the research feed into? 13. As far as you know how have the capacities of other local researchers who have participated in any of the NAPP activities been impacted? Sustainability 14. Do you feel you have been appropriately equipped to continue to inform policy issues? - Do you have plans to continue to conduct relevant research to inform agricultural policy after the NAPP has ended? How do you plan to do this? - How will you decide on the policy issues? - How will you disseminate or use the findings of your research? - What systems have been put in place to continue interactions with government and other relevant stakeholders on topical agricultural policy issues? 16 1.8 KEY INFORMANT INTERVIEWS – CSO/ASSOCIATIONS DATA COLLECTION PROTOCOLS – MLE 8 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gend er: Male □ Female □ Age____________ __ Civil Society Organizations Key Informant Interviews Gender and Youth 1. To what extent have women and youth (15 – 29 years) benefited from policy intervention undertaken by the Project? - Would you say these benefits were adequate? If no, how can this be improved? 2. Which approaches were most effective in reaching women and youth? 17 - How were women and youth encouraged to participated? Do you think this approach was efficient? If no, in what ways do you think this approach could have been better? - Were there any challenges integrating youth and women in the program? If yes, how can these challenges be managed? 3. How has the national capacity been strengthened for women in research and how have women benefited by this national capacity strengthening? - Do you think women were sufficiently represented in the project? If no, what approach will you suggest for future projects to capture sufficient number of women? - How has the intervention improved the capacity of women in research and policy analysis? 4. How have women and youth been part of the dialogue in the agriculture sector in developing national policy system? Are there examples where women and/or youth policy issues were singled out for priority? If yes, what are they? - Do you believe women and youth have been integrated in dialogue in the national ag. policy system? If no, what do you think are the major barriers resulting in the marginalization of women and youth. - What are your recommendations for facilitating inclusion? Trainings 5. To what extent have the direct training of select graduate students in Nigerian Universities and periodic training workshops organized for stakeholders strengthened the capacities of the next generation of policy researchers in Nigeria? - In your opinion, were benefits in the training of graduate students to the ag. sector? What benefits - Have you noticed any improvement in research and policy analysis as a result of this training? What improvements? 6. How have policy issues chosen by the Project in transforming the agriculture sector improved the capacity to plan and implement effective policy analyses and programs, including capacity to demand and absorb policy research in the policy processes? - How have key stakeholders enhanced their capacities in implementing policy analysis? Sustainability 7. What measures has the Project put in place to ensure sustainability of the interventions in the changing needs of the policy environment at the national/state levels? - Do you anticipate a continuous dialogue between policy actors and stakeholders even after the project ends? Please explain - Are there other programs with an agenda to improve ag. policy development/ analysis? 8. What measures has the Project put in place to ensure policy dialogue with and/or among USAID/Nigeria’s FTF activities and other donor programs. - Was there policy dialogue with other donor programs? In what areas? - Were these policy dialogues with donor programs adequate? - Which courses designed and executed by the project do you believe significantly contributed to enabling policy dialogue between USAID/Nigeria’s FTF and other donor programs? - Were there any trainings and workshops more suited to ensuring continued dialogue USAID/Nigeria’s FTF but were not provided by the project? Please explain 18 1.9 KEY INFORMANT INTERVIEWS – PRIVATE SECTOR ACTORS DATA COLLECTION PROTOCOLS – MLE 9 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gende r: Male □ Female □ Private Sector Key Informant Interviews 7. How has the Agricultural Policy Project helped you? 8. What could they have done better? 9. Has your agriculture and livestock productivity improved? (weather aside) 10. The project works within certain constraints, and the project’s own capabilities. Have there been constraints that have affected you? 11. Nigeria has an interesting set of trade policy rules. Interesting and complicated. For example, a ban on rice imports—in violation of Nigeria’s ECOWAS commitments—yet Nigeria imports about 3.5 million metric tons of rice per year. Did the Project contribute to the improvement of any policy which aided your business? 12. Gender equality is a major goal for USAID. What is the context like in Nigeria? Do you think the project helped to improve women’s opportunities in the agricultural sector? How so? 19 13. Youth employment is a high priority for the African Union (the year 2011 was “The Year of African Youth”). In the U.S., for several decades 1970s, 1980s and 1990s, youths who grew up in farming families left the business, leading to an aging farmer population. In this century, in the U.S., many young people are becoming farmers, with an emphasis on organic production and “quality of life.” What is the situation with Nigeria’s young people (under 30 years of age)? Do you think the project helped to improve the opportunities for youth in the agricultural sector? How so? 1.10 KEY INFORMANT INTERVIEWS – MEDIA DATA COLLECTION PROTOCOLS – MLE 10 NIGERIA This qualitative research questionnaire is one part of the final performance evaluation of the Feed the Future Nigeria Agricultural Policy Project. This project is jointly implemented by the International Food Policy Research Institute (IFPRI) and Michigan State University (MSU) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. You will be asked to verbally respond to a series of questions related to agricultural policy and the Nigerian Agricultural Policy Project. The purpose of this questionnaire is to assess the overall performance of the Project. As a stakeholder, we want to hear your views and opinions concerning this Project. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria. For any questions about the study, contact: Paul DeLucco – Chief of Party Monitoring, The Monitoring, Evaluation and Learning Activity DevTech Systems, Inc. No. 40 Mississippi Street, Off Alvan Ikoku Way Maitama, Abuja, FCT, Nigeria – PDelucco@devTechsys.com. Name : Position : Organization: Contact information: Office address: E-mail address: Telephone number: State of residence: Gender: Male □ Female □ Age____________ __ Media Key Informant/Group Interviews 1. What is your overall impression of NAPP? 2. To what extent has the Activity strengthened your capacity to undertake informed reporting of agricultural policy issues? 20 0 = Not at all 1 = To a small extent 2 = To a moderate extent 3 = To a great extent 4 = To a very great extent 3. What effect did the Project have on your capacity to undertake informed reporting on agricultural policy issues? - Are there other ways in which the Project could have impacted your ability in this regard? - Which trainings and workshops do you think contributed the most to building the capacity of the media in undertaking informed reporting on agricultural policy issues? - Were there any issues concerning policy communication and reporting that were not covered by the trainings and workshops? Please explain. - Are there other approaches in empowering the media that you think would have been more effective that were not covered by the trainings and workshops? Please explain 4. What would you consider the most critical constraints in building your capacity to undertake informed reporting of agricultural policy issues? 5. What other workshops/conferences have you attended between 2015 and 2020 that relate to agricultural policy and development? 6. How have they assisted you in the performance of your job? 7. How do you compare your experience in those events with that of NAPP? 8. What would IFPRI have done better? 9. What are your suggestions about sustainability of the NAPP? 1.11 QUESTIONNAIRE FOR TRAINING AND CONFERENCE BENEFICIARIES FTF Nigeria Agriculture Policy Project – Questionnaire for Training and Conference Beneficiaries Name of enumerator: Date of Interview: Time of the interview SECTION A 1. Name of the respondent 2. What is the sex of the respondent? 1=Female 2=Male 3. What is the name of your organization? 4. What is your position in the organization? 5. Your phone number? 21 6. Age of respondent: 7. Educational Qualification: 1=Post-Secondary (A-level, OND, NCE, Diplomas) 2=Bachelor's Degree 3=Post-graduate degrees 4=Doctoral degrees 5=Other 8. Have you participated directly in the training organized under NAPP? 1-Yes, 2-No 9. What was your experience with the Project's capacity building activities? 1=Excellent 2=Good 3=Neutral 4=Bad 5=Ugly Comment from What was your experience with the Project's capacity building activities? 10. Which of the trainings/workshop did you attend? 1=Software Training 2=Policy Training 3=Communication Training 4=Monitoring and Evaluation Training 5=Agricultural development and nutrition 6=Group dynamics 7=Others (specify): 11. Did you learn new skills? 1=Yes 2=No 12. Are those skills applicable in your work or educational path? 22 1=Yes 2=No Comment from - Are those skills applicable in your work or educational path? 14. What was most useful about the capacity-building training 15. What did you gain from participating at the training/workshop? 1=New technical skills 2=New methods of performing roles and responsibilities 3=others (specify) Comments on - What did you gain from participating at the training/workshop? 16. How relevant was the training/workshop to agricultural policy process? 1=Highly relevant 2=Just relevant 3= Not relevant 4=Highly irrelevant Comments on - How relevant was the training/workshop to agricultural policy process? 17. What was the mode of the delivery of the training/workshop? 1=Participatory 2=Straight lecture Comments on - Were there any hands-on, participatory exercises? Or were they straight lectures? 18. Were the facilities suitable? 1= Yes, 2= No 19. How skilled were the facilitators 1=Well Skilled 2=Just skilled 3=Not well skilled Comments on - How skilled were the facilitators? 20. What would you suggest the Project do differently when it comes to the type of training you attended 21. Are you more involved in policy research activities after the training? 23 1=Yes 2=No 22. How much influence did the training received have on your decision to conduct policy research in the future? 0= no influence at all 1= of little influence 2= highly influential 23. What can you say about the strength of your capacity to conduct policy research as a result of the training you received? 0 – unchanged 1 – improved 2 – much improved 24. What is it that you can do better based on the new skill acquired during the training? 25. Would you be interested in conducting independent policy research in the future? 1-Yes 2-No If yes, what motivates you to do so? 26. Were there hands-on or participatory exercises in the training? 1-Yes 2-No If yes, how challenging was that task? 1 – Very difficult 2– Difficult 3 – Neutral 4 – Easy 5 – Very easy 27. Were the facilities suitable for the training? 1-Yes 2-No 28. What is the likelihood of attending more training under NAPP in the future? 1 – Extremely unlikely 2 – unlikely 3 – Neutral 4 – likely 5 – Extremely likely 24 29. Have the policy issues chosen (through research, training and conferences) by the Activity in transforming the agricultural sector improved your capacity to plan and implement effective policy analyses and programs? If yes, in what ways? 1 ANNEX 4: Results from Descriptive and Inferential Analysis 1. There is no significant difference in the strength of male and female beneficiary’s capacity to conduct policy research as a result of the NAPP training Table 1: Independent Sample t-test of the strength of male and female beneficiary’s capacity to conduct policy research as a result of the NAPP training Sex of Respondents N Mean Std. Deviation F Sig. Df t What can you say about the strength of your capacity to conduct policy research as a result of the training you received? Female 123 2.14 .618 0.065 0.798 458 Male 337 2.16 .593 .396 The result showed no significant difference in the capacity to conduct policy research as a result of training received, female (M = 2.14, SD = 0.618) and male (M = 2.16, SD = 0.593); t (458) = .396, p = 0.798. The finding implied that the capacity to conduct policy research due to the training received by beneficiaries had nothing to do with their gender 2. There is no significant difference in the experience of male and female beneficiaries on the project’s capacity building activities Table 2: Independent Sample t-test of the experience of beneficiaries on the project’s capacity building activities based on their sex Sex of Respondents N Mean Std. Deviation F Sig. df t What was your experience with the Project's capacity building activities? Female 137 4.39 0.533 0.277 0.599 501 .117 Male 366 4.38 0.525 .116 The result showed no significant difference in the experience beneficiaries on the project’s capacity building activities, female (M = 4.39, SD = 0.533) and male (M = 4.38, SD = 0.525); t (501) = .117, p = 0.599. The finding implied that the experience of the participants had nothing to do with their gender and the NAPP activity was not discriminatory NIGERIA AGRICULTURE POLICY PROJECT • FINAL PERFORMANCE EVALUATION 3. There is no significant difference in the strength of beneficiary’s capacity to conduct policy research as a result of the NAPP training based on their age group Table 3: One Way ANOVA comparing differences in capacity to conduct policy research of beneficiaries based on their age group Sum of Squares Df Mean Square F Sig. Between Groups 4.828 4 1.207 3.435 .009 Within Groups 159.902 455 .351 Total 164.730 459 The capacity of beneficiaries to conduct policy research was compared across age groups using a one￾way between group ANOVA. The test showed a significant difference in the improvement capacity based on age groups, at the p<.05 F(_4, 455 = 3.435, p = .009. Table 4: Descriptive analysis showing the capacity to conduct policy research of the beneficiaries by age group Groups N Mean Std. Dev Lower Bound Upper Bound 31-40 154 2.29 .580 .047 2.19 41-50 141 2.13 .635 .053 2.03 51-60 90 2.10 .498 .053 2.00 61-70 21 2.05 .498 .109 1.82 20-30 54 1.98 .687 .093 1.79 Total 460 2.16 .599 .028 2.10 Although the beneficiary’s capacity was improved for policy research due to the NAPP training, the descriptive analysis showed that beneficiaries between the ages of 31 to 50 had a higher improved capacity from the NAPP activities compared to 51 years and above and also the 20 to 30 age group. This also confirms the previous findings that retirees and the unemployed had the least improvement based on occupational categories 4. There is no significant difference in the strength of beneficiary’s capacity to conduct policy research as a result of the NAPP training based on occupational categories Table 5: One Way ANOVA comparing differences in the strength of beneficiaries to conduct policy research based on their occupational categories Sum of Squares Df Mean Square F Sig. Between Groups 6.564 7 .938 2.703 .009 Within Groups 155.428 448 .347 Total 161.991 455 The capacity of beneficiaries to conduct policy research was compared across eight (8) occupational categories, State government, Federal government, Private sector, Media, NGOs, Scholars, Advisors and others using a one-way between group ANOVA. The test showed a significant difference in the capacity of the groups, at the p<.05 F(7, 448 = 2.703, p = .009. This implied that, the capacity of the different categories to conduct policy research due to the training received is significantly different NIGERIA AGRICULTURE POLICY PROJECT • FINAL PERFORMANCE EVALUATION Table 6: Descriptive analysis showing the capacity of the beneficiaries by occupational categories to conduct policy research Groups N Mean Std. Dev Lower Bound Upper Bound Media 49 2.27 .491 2.12 2.41 Advisors 113 2.25 .543 2.15 2.35 Scholars 20 2.20 .523 1.96 2.44 State Government 62 2.19 .649 2.03 2.36 Federal Government 53 2.19 .521 2.05 2.33 NGO 80 2.16 .625 2.02 2.30 Private 63 1.97 .671 1.80 2.14 Others 16 1.75 .683 1.39 2.11 Total 456 2.16 .597 2.11 2.22 However, from the descriptive analysis, Media, Advisors and Scholars had a higher improved capacity than other beneficiaries, the group categorized as others had the least improved capacity due to NAPP activities. This group is made up of retirees and the unemployed. 5. There is no significant difference in the strength of beneficiary’s capacity to conduct policy research as a result of the NAPP training based on their educational qualifications Table 7: One Way ANOVA comparing differences in capacity to conduct policy research of beneficiaries based on their educational qualifications Sum of Squares Df Mean Square F Sig. Between Groups 3.059 3 1.020 2.882 .036 Within Groups 159.548 451 .354 Total 162.607 454 The improvement capacity of beneficiaries to conduct policy research was compared across their educational qualifications using a one-way between group ANOVA. The test showed a significant difference in the improvement capacity based on qualifications, at the p<.05 F(3, 451) = 2.882, p = .036. Table 8: Descriptive analysis showing the capacity of the beneficiaries to conduct policy research by occupational categories Groups N Mean Std. Dev Lower Bound Upper Bound Post-Secondary (A￾level, OND, NCE, Diplomas) 63 2.11 .625 1.95 2.27 Bachelor's Degree 137 2.06 .591 1.96 2.16 Post-graduate degrees 246 2.23 .585 2.16 2.31 Total 455 2.16 .598 2.10 2.21 Although the beneficiary’s capacity was improved for policy research due to the NAPP activities, the descriptive analysis showed that the capacity of those with post-graduate qualifications improved higher than others 1 ANNEX 5: List of Key Informants Contacted in the North during the Evaluation SN Name Interview Day-Date Respondent Category Organisation Designation Phone number Email Redacted for privacy concerns. 1 ANNEX 6: List of Key Informants Contacted in the South during the Evaluation Name Interview Date Respondent Category Organisation Designation Phone Number Email Redacted for privacy concerns. 1 ANNEX 7: List of Training Workshop Beneficiaries Contacted and Interviewed Name of the respondent Organisation Phone No. NIGERIA AGRICULTURE POLICY PROJECT • FINAL PERFORMANCE EVALUATION 1 ANNEX 8: List of Conference Beneficiaries Contacted and Interviewed Redacted for privacy concerns. 1 ANNEX 9. REFERENCES Ering, Simon; Odey, Judith; Eteng, Out; and Esther Patrick Archibong. (2014). Rural Development Policies in Nigeria: A Critical Appraisal. International Journal of Education and Research. 2:9, Sept. 2014. Oyejide, T. Ademola., 1986. "The effects of trade and exchange rate policies on agriculture in Nigeria.:," Research reports 55, International Food Policy Research ... USAID NAPP (2015a). “Year 0 and Year 1 Work Plan”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, October 2015. USAID NAPP (2015b). Year 2 “Quarterly Report: First Quarter – October 1- December 31, 2015”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted January 2016. USAID MEP (2015c): Monitoring and Evaluation Plan (MEP): Revised December 4, 2017. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute. USAID MEP (2015d): Monitoring and Evaluation Plan (MEP): Revised December 21, 2017. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute. USAID MELP (2015e): Monitoring and Evaluation Learning Plan (MELP): Final Revision, July 31, 2019. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute. USAID NAPP (2016a). “Year 2 Work Plan”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, October 2016 – September 2017 USAID NAPP (2016b). Year 2 “Quarterly Report: Second Quarter: January 1- March 31, 2016”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted May 2016. USAID NAPP (2016c). Year 2 “Quarterly Report: Third Quarter: April 1- June 30, 2016”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted August 2016. USAID NAPP (2016d). Year 2 “Quarterly Report: Fourth Quarter: July 1- October 1, 2016”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted October 2016. USAID NAPP (2016e). Year 3 “Quarterly Report: First Quarter – October 1- December 31, 2016”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted February 2017. USAID NAPP (2017a). “Year 3 Work Plan”: October 1, 2017 – September 30, 2018. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted November 2017. USAID NAPP (2017b). Year 3 “Quarterly Report: Second Quarter: January 1- March 31, 2017”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted June 2017. NIGERIA AGRICULTURE POLICY PROJECT • FINAL PERFORMANCE EVALUATION USAID NAPP (2017c). Year 3 “Quarterly Report: Third Quarter: April 1- June 30, 2017”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted August 2017. USAID NAPP (2017d). Year 3 “Quarterly/ Annual Report: Fourth Quarter version 1: July 1- September 1, 2017”/ Annual October 1, 2016 – September 30, 2017. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted October 2017. USAID NAPP (2017e). Year 3 “Quarterly/ Annual Report: Fourth Quarter version 2: July 1- September 1, 2017”/ Annual October 1, 2016 – September 30, 2017 -. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted November 2017. USAID NAPP (2018a). “Year 4 Work Plan”: October 1, 2018 – September 30, 2019. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted September 2018. USAID NAPP (2018b). Year 4 “Quarterly Report: First Quarter – October 1- December 31, 2017”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted January 2018. USAID NAPP (2018c). Year 4 “Quarterly Report: Second Quarter: January 1- March 31, 2018”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted April 2018. USAID NAPP (2018d). Year 4 “Quarterly Report: Third Quarter: April 1- June 30, 2018”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted July 2018. USAID NAPP (2018e). Year 4 “Quarterly/ Annual Report: Fourth Quarter: July 1- September 30, 2018”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted October 2018. USAID NAPP (2018f). Year 4 “Quarterly/ Annual Report: Fourth Quarter: July 1- September 30, 2018”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted October 2018. USAID NAPP (2019a). “Year 5 Work Plan”: October 1, 2019 – June 30, 2020. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted September 2019. USAID NAPP (2019b). Year 5 “Quarterly Report: First Quarter – October 1- December 31, 2019”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted January 2020. USAID NAPP (2019c). Year 4 “Quarterly Report: Second Quarter: January 1- March 31, 2019”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted April 2019. USAID NAPP (2019d). Year 4 “Quarterly Report: Third Quarter: April 1- June 30, 2019”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted July 2019. USAID NAPP (2019e): NAPP Mid-Term Performance Evaluation: July 2019. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute. NIGERIA AGRICULTURE POLICY PROJECT • FINAL PERFORMANCE EVALUATION USAID NAPP (2020): Agricultural Policy Final Performance Evaluation Scope of Work: June 30, 2020. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute. USAID NAPP (2020). Year 5 “Quarterly Report: First Quarter: October 1- December 31, 2019”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted January 31, 2020. USAID NAPP (2020). Year 5 “Quarterly Report: Second Quarter: January 1- March 31, 2020”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted July 31, 2020. USAID NAPP (2020). Year 5 “Quarterly Report: Third Quarter: April 1- June 30, 2020”. Prepared by: Michigan State University in partnership with the International Food Policy Research Institute, submitted April 30, 2020. 1 ANNEX 10: EVALUATION TEAM NAPP Evaluation Team Position Title 1 OLOMOLA-NISER Aderibige Team Leader 2 AKINLUYI Toni Senior Evaluation Specialist 3 AJAO Olajide Abraham Local Agric. Economist 4 BELLO Mohammad Local Evaluation Specialist 5 GISANRIN Olufemi MEL Specialist Economic Growth and Environment TEAM LEADER Mr. Aderibigbe Olomola-Niser, PhD Dr. Olomola is an agricultural policy scholar with over thirty years of experience in research and teaching at the Nigerian Institute of Social and Economic Research (NISER), over fifteen years of which involve policy research coordination and management. Dr. Olomola has served in various leadership positions within NISER such as the head of the Macroeconomic and Strategic Modeling Unit (MASMU), Director of the Research and Consultancy Unit, Director of the Agriculture and Rural Development Department (ARDD), and Director of the Surveillance and Forecasting Department. Dr. Olomola has served as a consultant for institutions such as Federal Government of Nigeria as well as national and international organizations including: Agricultural Resource Council of Nigeria, West Africa Agricultural Productivity Program, the World Bank, the United Nations Development Programme, the United Nations Institute for Economic Development and Planning (UNIDEP) the International Labor Organization, the Food and Agricultural Organization, International Food Policy Research Institute (IFPRI), Overseas Development Institute (ODI), Gesellschaft für Internationale Zusammenarbeit (GIZ), and Brookings Institution in the areas of agriculture, food security and development economics. Dr. Olomola has served as coordinator of courses on monitoring and evaluation and delivered capacity building programs on M&E at NISER, as well as supervised student at the Ph.D. level. Dr. Olomola has served as external examiner and facilitated various courses in and outside Nigeria especially at the United Nations Institute for Development and Economic Planning (Senegal) where he has served as Course Director from 2014 to date in the areas of Development Policy and Fundamentals of Development Planning. He was also a member of a seven-man team of experts drawn from various sub-regions of Africa commissioned by the African Union Commission (AUC) in 2011 to prepare a document in support of setting up the African Research Council. Dr. Olomola holds a PhD in Agricultural Economics from the University of Ibadan. SR. EVALUATION SPECALIST Ms. Toni Lois Akinluyi Monitoring, Evaluation and Learning Specialist; Researcher; Private Sector Development (PSD) Professional with over 14 years of experience working in results measurement, conducting research and carrying out evaluations for various donors, private, and public-sector clients in Nigeria. Her areas of experience include business environment reforms, economic growth, health, education, WEE- girls and women, service delivery and governance, taxation and public sector reforms, among other. Proficient in Econometric and Statistical Software including SPSS, STATA and EVIEWS, Ms. Akinluyi brings a strong NIGERIA AGRICULTURE POLICY PROJECT • FINAL PERFORMANCE EVALUATION understanding of managing teams across a range of research and learning activities required for large programmes, implementing and advising on the collection and use of data for designing and adapting intervention approaches. Ms. Akinluyi has a Masters in International Economics and Public Policy from Cardiff University. AGRICULTURE ECONOMIST Mr. Olajide Ajao, PhD Dr. Ajao is an agriculture economist with over 20 years of teaching and research experience in agriculture economics. Dr. Ajao has published thirty-eight publications on agriculture productivity, farming, gender and agriculture, and efficiency or effectiveness of different agricultural production techniques in Nigeria￾including land use and smallholder fish farms. Dr. Ajao currently serves as the Head of the Department of Agricultural Economics for Ladoke Akintola University of Technology in Ogbomoso-Oyo State. Dr. Ajao holds a PhD in Agriculture Economics from Obafemi Awolowo University in Ile-Ife, Nigeria. EVALUATION SPECIALIST Mr. Muhammad Bello, PhD Dr. Bello is an agriculture and food economist with over 15 years of experience conducting feasibility studies, value chain analyses, and Cost/Benefit analyses of nascent agricultural and animal husbandry programs in Africa in order to determine policy implications and likelihood of success. For the last 15 years, he has served as Lecturer/Researcher in the Department of Agricultural Economics and Extension, Bayero University, Kano, Nigeria. Dr. Bello holds a PhD in Agriculture and Food Economics from the University of Kiel in Germany. ECONOMIC GROWTH AND ENVIRONMENT MEL SPECIALIST Mr. Olufemi Gisanrin Mr. Gisanrin has over 25 years of experience that cuts across private and agri-business development. His first 13 years of experience were multidisciplinary, working with multinational teams in the design and management of an integrated dairy farm involving production of various dairy products from fresh milk from exotic and cross-bred dairy cows, and processing and marketing of dairy products in Nigeria. Since 2006, Mr. Gisanrin has managed M&E contracts for key USAID Agricultural development projects including: MARKETS, Bridge to MARKETS II and MARKETS II. He has both field level and supervisory experience managing project and activity monitoring and evaluation systems. Most recently he has led a technical team of six professional on a USD 64 million agriculture project funded by the United States Agency for International development. Mr. Gisanrin guided regional teams and service providers to ensure effective value chain focused M&E processes and mechanisms were in place for accurate monitoring and reporting on over 1,500,000 beneficiaries spread across 22 states. Mr. Gisanrin is currently a Senior Monitoring, Evaluation and Learning Specialist on the USAID funded Monitoring, Evaluation and Learning (MEL) Activity supporting the Economic Growth and Environment (EGE) program office of USAID Nigeria mission. He has his Master’s degree in Development Studies from the Nigerian Defense Academy in Kaduna Nigeria. NIGERIA AGRICULTURE POLICY PROJECT • FINAL PERFORMANCE EVALUATION ANNEX 11: EVALUATION TEAM CONFLICT OF INTEREST STATEMENTS 1 ANNEX 11: EVALUATION TEAM CONFLICT OF INTEREST STATEMENTS