FINAL PERFORMANCE EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR COLLABORATIVE RESEARCH ON AQUACULTURE & FISHERIES September 18, 2018 FINAL PERFORMANCE EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR COLLABORATIVE RESEARCH ON AQUACULTURE & FISHERIES (AQUAFISH IL) This publication was produced at the request of the United States Agency for International Development Developed under Contract/Task Order Number: AID-OAA-1-15-00024/AID-OAA￾TO- 16-00008. Submitted to: USAID/Feed the Future Contact: Lesley Perlman Email: lperlman@usaid.gov Submitted by: David Rinck, Team Leader Simon Ellis, Technical Expert Contractor: Program Evaluation for Effectiveness and Learning (PEEL) ME&A 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 Email: www.meandahq.com Cover Photo: Fish raised in aquaculture ponds in Bangladesh by Lucy O’Bryan. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT The purpose of the Feed the Future Innovation Lab for Collaborative Research on Aquaculture & Fisheries (AquaFish IL) is to expand research and the pool of researchers pursuing topics relevant to the primary thematic areas of aquaculture and fisheries. The AquaFish IL evaluation provides evidence-based findings and recommendations that USAID can use to inform and improve future investments in aquaculture and fisheries research to better achieve intended outcomes. The evaluation used document review and key informant interviews to answer questions related to the Management Entity’s (ME) effectiveness in project implementation; coordination and communication; generation of quality research and collaborations; response to requests for technical assistance; and generation and development of the research technologies with greatest potential for future development. The evaluation found that, across most areas, the program performed well. AquaFish IL produced quality research through 103 peer-reviewed publications by the time of this evaluation. Another 40 to 50 were in preparation. AquaFish IL supported over 275 long-term degree-seeking students (51 percent women) and provided short-term training for over 2,000 participants (45 percent women). It also produced innovative technologies, including developing Pangasius culture in hyposaline conditions in Bangladesh. However, the ME had limited success in forging strong relationships with USAID Missions. This was partly because the Mission staff had time constraints, which limited their engagement. Interviewees recommended that fish feeds and feeding, production systems and management, mobile phone-based technologies, and air-breathing and indigenous fishes merit future research as they are high priorities for host countries participating in AquaFish IL. CONTENTS EXECUTIVE SUMMARY................................................................................................................................................... I EVALUATION PURPOSE ................................................................................................................................. I PROGRAM BACKGROUND........................................................................................................................... I EVALUATION METHODS AND LIMITATIONS....................................................................................... I MAIN FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ..................................................II 1.0 EVALUATION PURPOSE AND QUESTIONS ...................................................................................................1 1.1 EVALUATION PURPOSE .................................................................................................................1 1.2 AUDIENCE AND INTENDED USES.............................................................................................1 1.3 EVALUATION QUESTIONS............................................................................................................1 2.0 BACKGROUND.........................................................................................................................................................2 2.1 AQUAFISH IL APPROACH..............................................................................................................3 2.2 STAKEHOLDERS AND IMPLEMENTATION..............................................................................4 3.0 METHODOLOGY AND LIMITATIONS..............................................................................................................5 3.1 EVALUATION METHODOLOGY.................................................................................................5 3.2 EVALUATION LIMITATIONS.........................................................................................................7 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS .......................................................................8 4.1 EVALUATION QUESTION 1..........................................................................................................8 4.2 EVALUATION QUESTION 2........................................................................................................14 4.3 EVALUATION QUESTION 3........................................................................................................17 4.4 EVALUATION QUESTION 4........................................................................................................24 4.5 EVALUATION QUESTION 5........................................................................................................27 4.6 EVALUATION QUESTION 6........................................................................................................31 4.7 EVALUATION QUESTION 7........................................................................................................32 ANNEXES .........................................................................................................................................................................39 ANNEX A: EXPRESSION OF INTEREST ..................................................................................................40 ANNEX B: EVALUATION TEAM................................................................................................................47 ANNEX C: EVALUATION SCHEDULE.....................................................................................................49 ANNEX D: LIST OF DOCUMENTS REVIEWED AND REFERENCES..............................................51 ANNEX E: KEY INFORMANT INTERVIEWS ..........................................................................................54 ANNEX F: REQUEST FOR STUDENT PARTICIPATION....................................................................57 ANNEX G: DATA COLLECTION TOOLS..............................................................................................59 ANNEX H: BROCHURE EXAMPLE............................................................................................................73 ANNEX I: STATEMENT OF DIFFERENCE.............................................................................................. 76 ACRONYMS AOR Agreement Officer Representative AquaFish IL Feed the Future Innovation Lab for Collaborative Research on Aquaculture and Fisheries CRSP Collaborative Research Support Program BFS Bureau for Food Security EFMIS Enhanced Fish Market Information Service EQ Evaluation Question ET Evaluation Team FAS Foreign Agricultural Service FTFMS Feed the Future Monitoring System FY Fiscal Year HICD Human and Institutional Capacity Development IFReDI Inland Fisheries Research and Development Institute IL Innovation Lab IMS Institute of Marine Sciences IP Implementing Partner KII Key Informant Interview KNUST Kwame Nkrumah University of Science and Technology M&E Monitoring and Evaluation m2 Square Meter ME Management Entity MT Metric Ton NaFIRRI National Fisheries Resources Research Institute NARC Nepal Agricultural Research Council NARS National Agricultural Research System NGO Non-Governmental Organization NSF National Science Foundation OSU Oregon State University PE Performance Evaluation PEEL Program Evaluation for Effectiveness and Learning POC Point of Contact ppt Parts Per Thousand RFA Request for Applications RFP Request for Proposal SMIS Seafood Market Information System STIP Science, Technology, Innovation, and Partnership UM University of Michigan U.S. United States USAID United States Agency for International Development WAS World Aquaculture Society WIOMSA Western Indian Ocean Marine Sciences Association i EXECUTIVE SUMMARY EVALUATION PURPOSE The Feed the Future Innovation Lab for Collaborative Research on Aquaculture and Fisheries (AquaFish IL) is funded by the United States Agency for International Development (USAID) Bureau for Food Security (BFS) for a five-year period (2013-2018). The purpose of the AquaFish IL evaluation was to provide evidence-based findings and recommendations, which USAID/BFS can use to inform new investments in aquaculture and fisheries research and more effectively achieve intended outcomes in similar future projects. The findings from the evaluation could also inform both future research topic areas1 and research design of other development programs. The primary audience for the evaluation is USAID/BFS. The evaluation was conducted by a team assembled by ME&A under the Program Evaluation for Effectiveness and Learning (PEEL) project. It was implemented between November 2017 and February 2018. The evaluation sought to answer seven evaluation questions (EQs) related to the Management Entity’s (ME) effectiveness in: project management and implementation; coordination and communication; generation of quality research and collaborations; response to requests for technical assistance; and generation and development of the research technologies that have the greatest potential for future development. PROGRAM BACKGROUND AquaFish IL is one of 24 Feed the Future innovation labs (ILs) funded by USAID/BFS. AquaFish IL’s mission is to “enrich livelihoods and promote health by cultivating international multidisciplinary partnerships that advance science, research, education, and outreach in aquaculture and fisheries in environmentally and socially acceptable ways.” AquaFish IL was initiated in 2013 as a successor to the Aquaculture Collaborative Research Support Program that began in 2006. AquaFish IL aims to expand research and the pool of researchers pursuing topics relevant to aquaculture and fisheries research for development in five country groups based in Africa or Asia. Country groups include 1) Kenya and Uganda; 2) Bangladesh; 3) Ghana and Tanzania; 4) Cambodia and Vietnam, and 5) Nepal. AquaFish IL also focuses on four conceptual “targets” 1) research; 2) capacity building; 3) information dissemination; and 4) gender integration—a cross-cutting target. Oregon State University (OSU), the ME for AquaFish IL, oversees the core research program. It works with five other United States (U.S.)-based universities (subcontractors), including Auburn University; North Carolina State University; Purdue University; University of Connecticut, Avery Point; and University of Michigan. In turn, these subcontractors work with five host country lead universities and 27 local partner institutions. They also work with additional U.S. and host country universities, specialized research associations and institutions, and non-governmental organizations (NGOs). The reach of AquaFish IL extends to 37 funded partner institutions and 62 collaborative partners. EVALUATION METHODS AND LIMITATIONS This evaluation used a desk study design with no fieldwork. Data were collected through document review and remote interviews with 43 key informants. The interviews were conducted remotely with purposively selected key informants across all program countries and relevant stakeholder groups associated with AquaFish IL. Stakeholder groups included ME program personnel; U.S. lead partner institutions; host country partners; USAID Mission staff; USAID/BFS staff; current or former long-term degree-seeking 1 The purpose was revised during the third phase of the evaluation so it would not be used to design the next Aquaculture investment but could inform any other investments in this field at any stage. viii students; and AquaFish IL-supported researchers (see Table 3 and Annex E: Key Informant Interviews). Reviewed documents included, but were not limited to, technical proposal, annual reports, and published materials. The evaluation team (ET) used content analysis, which allowed the team to review key informant interview (KII) responses as well as identify and highlight factors contributing to or hindering realization of AquaFish IL’s mission. Data were also analyzed for emerging themes, patterns, and trends, which helped examine outcomes over time and identify patterns of convergence (or divergence) of program outputs and outcomes leading to the stated program mission. The evaluation had several limitations that affected interpretation and generalization of the evaluation findings, but the two primary limitations were: 1. The limited number of USAID Missions (one out of the eight countries implementing AquaFish) were willing to participate in remote interviews and the absence of in-country Feed the Future implementing partners (IPs) to interview. This limitation meant that perspectives from these stakeholder groups are not really represented in the findings and recommendations. Although the ET received information for 15 Mission points of contact (POCs) in the eight countries where AquaFish IL was implemented, only one Mission respondent agreed to be interviewed. Because there were no identified in-country Feed the Future program IPs active in the aquaculture sector in the eight countries, the ET could not interview any. This introduced a bias in that the perspectives of university stakeholder groups (both U.S. and host countries) was overrepresented and other groups (Missions, practitioners) were underrepresented in the evaluation. 2. This assessment was a desk study with no field visits, supplemented with a limited number of remote key informant interviews, which affected the amount of data collected and the answers to key questions that otherwise would have easily been addressed through face-to-face interviews and direct observations. Interviews conducted via telephone or Skype present logistical challenges and afford limited opportunity for relevant probing questions. MAIN FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Findings Conclusions EQ1: To what extent has the ME effectively implemented and managed AquaFish IL’s activities in the five target regions of Africa and Asia? • Most local partner institution interviewees (n=12) and U.S. interviewees (n=13) were positive in their opinion of the ME's project implementation and management. • Most host country interviewees (n=16) stated that the proposal process allowed them to adequately address the key priorities in aquaculture research development for their country or region. Eighty (80) percent of statements expressing opinions on this issue were positive. • Most host country university interviewees (including students) (n=16) agreed that research topics were broad enough to allow them to address key bottlenecks of national/regional priorities. • All U.S. subcontracting lead project institutions and host • The ME effectively implemented and managed the key processes critical to management and implementation of AquaFish IL. However, there were elements that challenged implementation and management, which were external in nature and beyond the ME’s control. These included: - A reduction in funding for the project; - Delays in the disbursement of funding in the multi-tiered contracting structure of AquaFish IL; - Host country institution compliance with U.S. university requirements and regulations relating to human subjects and research animals; iii country institutions (n=11) brought up disruptions due to reduced funding and delays in receipt of funds, as corroborated by 15 negative comments and no positive or neutral comments on this issue. • Informants from all relevant stakeholder entities (n=43) stated that USAID Mission staff were not engaged in AquaFish IL, due to time constraints for the Mission staff. • Seventy (70) percent of key respondent remarks (n=25) in U.S. and local institutions characterized administrative procedures associated with the program as time￾consuming. • All ME personnel interviewed (n=5) expressed considerable concern for the administrative burden placed on subcontractors and stated their interest in working to reduce this to the degree possible. • No new associate awards were made during AquaFish IL (Phase II) between 2013 and 2018. - The burden of administrative procedures, especially compliance with USAID reporting guidelines; and - Project implementation delays due to political factors and environmental disasters. • The ME managed and coordinated effectively among the five subcontracting lead project U.S. universities and host country universities, 27 partner institutions, and USAID. Coordination with the eight USAID Missions was challenged by time constraints on the part of Mission personnel. • There were no new associate awards during the life of AquaFish IL (Phase II). Recommendations • The ME should work with USAID/BFS particularly at the request for proposal (RFP) stage to streamline their multi-tiered contractual process to ensure that funds are dispersed in a timely manner to the host country research implementers. • The ME should reorganize its administrative structure to reduce the several layers of partnerships in the host country that are inherently bottlenecks making management of the program cumbersome, which led to delays that adversely affect research schedules. • The ME should also require, in RFPs from host countries, that lead host country universities innovatively and creatively propose how they will establish effective partnerships with other in-country stakeholders beyond the local research partner institutions. This requires concerted efforts in the RFP process to require host country research institutions to partner with existing Feed the Future program practitioners and policymakers in target countries to set local research priorities and promote the relevance of AquaFish IL products. EQ2: To what extent have effective communication and coordination supported project implementation and outcomes regarding research and dissemination and capacity development? • Ninety (90) percent of the remarks made by 29 key informants from U.S. and host country institutions praised the ME for their communication and coordination abilities. • Host country and local participating institutions personnel and researchers (n=33) described the regional and cross￾regional exchanges, both in Africa and Asia, as the most useful communications and learning opportunities, and the activity they would most like to see expanded. • A number of host country and local participating institutions personnel (n=9) stated that regional exchange meetings were most useful when they occurred in countries with “more developed” aquaculture sectors. • Effective communication and coordination between institutional stakeholders has been key to program implementation and outcomes regarding research and dissemination, and capacity development. • The critical element in project success is the highly cohesive nature of the aquaculture research community. This is derived from the extensive collaboration experience of the stakeholder institutions, intimate knowledge of research objectives, and strong personal bonds between institutional staff. • Communication and coordination by expanding inter-regional exchanges, especially between countries with less developed and those with more developed aquaculture sectors were useful. Recommendations • The ME and USAID/BFS should agree at the start of future aquaculture IL projects to establish a clear mechanism for engaging and communicating with USAID host country Missions. • In future aquaculture ILs, USAID/BFS should increase resources for further expansion of inter-institutional exchanges, especially exchanges that entail opportunities for partners from less developed institutions to gain exposure to appropriate technologies unavailable in their own countries. In particular, this could include viii exchanges that enable personnel to participate in strategic research partnership initiatives. • The ME should establish an in-country aquaculture steering committee that is made of researchers, practitioners, and policymakers to oversee and guide the in-country research directions instead of relying on the lead host country universities, who may be biased towards their specific research topics. EQ3: To what extent did AquaFish IL generate quality research technologies/outputs using appropriate metrics (e.g., peer reviewed publications)? • The total target for peer-reviewed publications throughout the five years of AquaFish IL was 100. AquaFish IL annual reports show 103 publications from March 2013 to September 2017, exceeding the target. • Ninety-four (94) percent of the remarks made by all key informants were positive about the publication record of AquaFish IL and the opportunities it afforded for publication. • The AquaFish IL USAID-approved monitoring and evaluation (M&E) plan had five benchmarks. The only one met was AquaFish IL developing at least one innovative aquaculture technology for dissemination in each region (Asia or Africa). The ET could not evaluate the other four since they lacked data and well-defined performance measurement metrics. • Extension and outreach of research results to end users follow established extension methodologies, with some innovative exceptions in Nepal, Bangladesh, Kenya, Tanzania, and Uganda. • Key informants (n=29) were positive about the public dissemination efforts of AquaFish IL, as evidenced by 99 percent positive statements from interviewees expressing opinions on this issue. • Review of AquaFish IL Fiscal Year (FY) 2017 Annual Report shows 103 publications, directly linked to AquaFish IL’s funding since 2013. However, due to the natural lag time between completion of investigations and publication, peer￾review publications attributable to AquaFish IL’s support should come into print for some years to come after the end of AquaFish IL. • Participating institutions have disseminated research results to host country stakeholders through a variety of non-traditional and traditional extension and outreach methods. • Key informants described a lack of clearly defined performance metrics as having impeded a complete evaluation of the project impact. One benchmark was met. • Research results were made available to stakeholders, including USAID Missions, in the host countries through a variety of methods, including innovative (cell phone-based technologies) and traditional (brochures and booklets). Recommendations • The ME and partners should encourage use of innovative technologies to disseminate research outputs, technical information, and market data, possibly including: online cloud-based tools that support researchers in collaborating with others during the research process ranging from; services that help host country researchers prepare and submit articles to journals such as support for language, editing, and early peer review of articles to services that identify relevant journals in a given field; and new web-based platforms for mobile phones that use technology to link users to relevant content. • The ME should promote and support inter-institutional research and in-country dissemination of findings, and cross-country research and dissemination to other new countries facilitated through cross-regional meetings and exchanges. • The ME and its partners should develop clear M&E plans that include measures for quality of research outputs, relevance, and effectiveness of dissemination to, and uptake by, stakeholders. • USAID/BFS’ future investments in AquaFish IL projects should keep the end users of the research in mind (practitioners and policymakers) by requiring in the RFP that the ME establish a diverse partnership that includes research institutions, practitioners such as the Feed the Future IPs working in aquaculture, and policymakers to facilitate uptake of research findings. EQ4: How well has the Lab identified and addressed the academic and technical capacity needs of host country stakeholders? How could AquaFish IL better serve and provide for the capacity needs of these stakeholders? v • Interviews with ME personnel (n=4) and U.S. partner institutions (n=5) indicated that current policies are in place to ensure that there is a strong focus on host country research development. • According to ME and subcontracting lead project U.S. university informants (n=17), academic and technical capacity in the host countries is strengthened through direct mentoring, longer-term student training, and short￾term training. • In terms of longer-term capacity building, AquaFish IL provides financial and technical support to students from the host countries to do undergraduate and postgraduate studies both in the U.S. and in their respective regions, as evidenced by interviews with some beneficiary students (n=4). • To achieve cost savings, some students underwent so￾called “sandwich” degrees, under which they spent some time in their host country and some time at a subcontracting lead project U.S. university. • The project addressed the academic and technical needs of the host countries through short-term training. From March 2013 through September 2017, 61 trainings have been conducted and 95 percent of the comments from key informants (n=29) were positive about them. • Host country institutional stakeholders agreed that the research agenda for AquaFish addressed national priorities. • The project placed considerable emphasis on addressing the academic and technical capacity needs of host country stakeholders. This was done through: - Direct mentoring of junior host country university and partner institution participants by senior university faculty in areas such as research methodologies, presentation, outreach, and publication techniques. - Longer-term student training through financial and technical support to students from the host countries to do undergraduate and postgraduate studies, both in the U.S. and in their respective region; and - Short-term trainings focusing on farmers and other end users in practical workshop-style settings. Recommendations • AquaFish IL should expand the opportunities for U.S.-based university researchers to conduct onsite mentorship visits in host country institutions that are mutually beneficial to both groups. The mentorship visits should include longer-term assignments (i.e., 6 – 12 months), in order to: 1) establish and strengthen relationships between U.S. universities and host-country institutions; and 2) gain a clearer picture of the context in which the U.S. and host￾country partnerships will operate. • For longer-term capacity development, AquaFish IL should increase financial and technical support for undergraduate and post-graduate degrees in both U.S. and regional universities to ensure sustainability. EQ5: How has the ME performed in establishing productive collaborations? • According to ME personnel (n=5), the subcontracting lead project U.S. universities have a large role in forming and managing productive collaborations with researchers from the host country university and partner institutions. • Comments in interviews with a number of key informants (n=29) were 89 percent positive in expressing opinions about collaborative efforts among researchers. • In contrast, those same informants described the ability of the ME to build collaborations with other USAID Feed the Future ILs and projects in the target countries as having mixed results. In general, informants described very little collaboration between USAID Missions, other Feed the Future IPs working in aquaculture, and AquaFish IL personnel. • The ME has been highly successful in establishing productive collaborations between the host country universities and the partner institutions. • In several cases, collaborations have yielded significant results, including several publications that could otherwise not have been achieved in the absence of inter-institutional collaboration. These collaborations are mainly due to the cohesive nature of the aquaculture research community involved in AquaFish IL. Notably, the five subcontracting lead project U.S. universities selected in 2013 continued from the previous activity initiated in 2006. viii • One subcontracting lead project U.S. university informant mentioned considerable success developing synergies between ILs and the USAID Farmer-to-Farmer project, and with the Foreign Agricultural Service (FAS) as an example of a productive collaboration. • Despite significant efforts, the ME has been unsuccessful in establishing funded collaborations with other Feed the Future activities in the target countries, which could have been facilitated by Missions if better collaboration were in place. • There is the potential for project synergy between ILs such as the AquaFish IL and other U.S. or foreign assistance programs. FAS and the Farmer￾to-Farmer project are two examples. Recommendations • In the future, the ME should expand their research pool of partners instead of relying primarily on previously established relationships among consortium members. They could identify and invite new host country research institutions into the consortium via their publications or conference presentations about aquaculture issues and might benefit from collaboration. • For new procurements, ME proposals should require submissions that demonstrate local research priorities and relevance of research solutions that take into consideration the demand from local research institutions, national policymakers, USAID and other donors, practitioners, and the private sector. • USAID/BFS should consider finding ways to link aquaculture researchers with other relevant Feed the Future ILs and the USAID Missions. This could include regular meetings to exchange ideas and information, or virtual or in-person forums around specific technical topics. EQ6: In what ways has the ME been responsive to requests for technical assistance? • Informants from the partner institutions (n=5) said that they primarily took their requests for technical assistance to the subcontracting lead project U.S. university personnel because of their historical relationships. • Host country institution requests for technical assistance varied but generally focused on experimental design, statistical analysis, and editing of manuscripts. • The ME made regular outreach attempts to USAID/BFS and the eight USAID Missions and other Feed the Future IPs working in aquaculture. • There were no requests for technical assistance from USAID/BFS, and the eight USAID Missions and other Feed the Future IPs after 2013. • Participants from the partner institutions were comfortable with relaying their request of technical assistance through subcontracting U.S. universities. • Host countries had the greatest need for assistance with research design, data analysis and dissemination of their research. • The ME attempted to build demand for technical services among USAID Missions and Feed the Future IPs working in aquaculture. However, they were unable to establish working relationships due, in part, to the fact that there were no aquaculture projects in the countries they were working in at the time. Recommendations • For future aquaculture projects, the ME should conduct an initial assessment of technical assistance needs on aquaculture topics in each target country and develop a plan on how those needs will be met throughout the life of the project. • Future aquaculture research topics should focus upon primary constraints with respect to markets and supply chains; diet ingredients and additives; genetic selection and classic selective breeding programs; survival rates; economics and regulation; and, most importantly, climate change and resilience. EQ7: Which research technologies had the greatest impact or potential for the greatest impact? vii • Host country informants were 90 percent positive about the outcomes and impact of their own projects and recommended future work focusing primarily on their specific research areas. • Feeds and feeding were cited as the most important topics for future research, followed by general production, pond systems, and management. Mobile phone-based technologies as well as air-breathing and indigenous species, both ranked third. • Other informants (n=11) emphasized the need for breeding and genetic work, despite the fact that little work in this area was conducted during AquaFish IL. • Host country informants’ attitudes about impact were influenced by their own areas of research and, therefore, they primarily listed technologies related to their research as the most important. • AquaFish IL-funded researchers agreed on eight research technologies as having the greatest impact or potential for the greatest impact, thus meriting new or continuing research investment. Recommendations • USAID/BFS should focus future aquaculture research funding on the technologies identified by AquaFish IL-funded researchers as having the greatest impact or potential for the greatest impact. These are, in descending order from most important to least important: - Feeds and feeding; - General production, pond systems, and management; - Air-breathing and indigenous species; - Mobile phone-based technologies; - Breeding and genetics; - Hyposaline and coastal aquaculture development; - Aquaponics; and - Post-harvest management. Lessons Learned2 • An active ME with well-established partnerships, reliable leadership, and good communication has greatly enhanced the value of AquaFish and is vital to the success of such projects. • The AquaFish IL research project was mainly designed and implemented by researchers (U.S. and host country universities) with limited involvement of practitioners, policymakers, and beneficiaries to set priorities. As a result, the research products were mainly targeted towards producing peer-reviewed publications and less on their use for programmatic and policy decisions. • To help ensure that results are useful and sustainable, award criteria for any similar Feed the Future ILs should, in part, include demonstrated integration with USAID Mission programming and existing Feed the Future programs in the proposed host countries or regions, and potential for ongoing Mission engagement. Otherwise, Mission staff will not be engaged in the research process given their time constraints. • Scoping visits to establish research needs in countries selected for IL programming should be undertaken in order to identify Mission priorities and host country needs. • Consistent oversight of centrally funded projects that do not have an in-country presence is necessary in order to establish effective linkages between those projects, USAID Missions, and other Feed the Future projects. • If centrally funded research projects such as AquaFish ILs are implemented in countries without related Feed the Future projects or planned projects, linkages with Missions and IPs are obviated. Therefore, if a future project is seeking to establish such linkages, the selection of countries should be based upon whether or not there are active or considered projects therein. • Associate awards are intended to allow Missions or Bureaus within USAID to support work related to the Leader program. However, research projects such as AquaFish may come to depend upon these awards for scaling up results and technologies rather than formulating plans for greater dissemination and sustainability. • Expected results need to be clearly linked by causality and partners in all IL projects need to understand the linkage—in particular, knowing how the research can ultimately benefit stakeholders in developing countries. 2 These Lessons Learned are generalizations based on the ET’s evaluation experience as applied to the specific findings of this project. These lessons highlight both strengths and weaknesses in planning, design, and implementation that may have affected the performance, outcome and/or impact of the project. viii • Multi-country innovative research projects could benefit from more frequent and cross-country assessments than those projects implemented in a single country to promote cross-fertilization learning. Field visits should be documented to assure a comprehensive “train of evidence” on the impact of the project over the years beyond reports and monitoring data. 1 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE The Feed the Future Innovation Lab for Collaborative Research on Aquaculture and Fisheries (AquaFish IL) is a research project funded by the United States Agency for International Development (USAID) Bureau for Food Security (BFS) and implemented by participating United States (U.S.) and host country institutions. AquaFish IL is set up to achieve maximum program impact, particularly for small-scale farmers and fishers in Feed the Future host countries and beyond. AquaFish IL’s objectives are to address the need for world-class research, capacity building, and information dissemination. The Lab is administered through a Management Entity (ME) at Oregon State University (OSU) and implemented in partnership with five subcontracting lead project U.S.-based universities—Auburn University; North Carolina State University; Purdue University; University of Connecticut, Avery Point; and University of Michigan. The five U.S.-based universities work with five host country lead universities and 27 partner institutions across eight countries. The evaluation of AquaFish was conducted to provide evidence-based findings, conclusions, and recommendations, which USAID/BFS can use to both inform new investments in aquaculture and fisheries research and more effectively achieve intended outcomes3 of similar projects. The findings from the evaluation might also inform future research topic areas and research designs of other development projects. 1.2 AUDIENCE AND INTENDED USES The primary audience for the evaluation is the USAID/BFS, USAID Feed the Future projects, and other aquaculture and fisheries implementing partners (IPs). The evaluation team’s (ET) recommendations are expected to assist USAID/BFS to improve the effectiveness and efficiency of implementation and management of similar future innovation labs (ILs). USAID/BFS can use evaluation results to design and establish plans for future research on nutritious and safe foods related to aquaculture and fisheries. 1.3 EVALUATION QUESTIONS The evaluation addressed seven sets of evaluation questions (EQs): 1. To what extent has the ME effectively implemented and managed AquaFish IL’s activities in the five target regions of Africa and Asia? How effectively has the ME managed and coordinated among the five U.S. universities and host country universities, 19 partner institutions,4 USAID and eight USAID Missions and their Feed the Future IPs working in aquaculture, and the long-term degree￾seeking students, host country trainees that have attended short-term training events, and local researchers? In what ways could implementation and management be improved? 2. To what extent have effective communication and coordination supported AquaFish IL’s implementation and outcomes regarding research and dissemination and capacity development? In what ways could communication and coordination be improved between USAID/BFS, the eight USAID Missions, the ME, the five lead U.S. institutions and host country universities, and other stakeholders, as a way to better achieve project goals and objectives, create and carry out a shared research agenda, meet reporting requirements, and expand collaboration? 3 The purpose was revised during the third phase of the evaluation so it would not be used to design the next Aquaculture investment but could inform any other investments in this field at any stage. 4 Since the EQs were devised, the number of partner institutions has increased to 27. 2 3. To what extent did AquaFish IL generate quality research technologies/outputs using appropriate metrics (e.g., peer-reviewed publications)? In what ways have the AquaFish IL ME and the five subcontracting lead project U.S. institutions fostered research results dissemination to host country stakeholders? Moving forward, what opportunities are there to improve research outputs (both increasing quality and generating appropriate outputs for Feed the Future countries)? 4. How well has AquaFish IL identified and addressed the academic and technical capacity needs of host country stakeholders? How could it better serve and provide for the capacity needs of these stakeholders? 5. How has the ME performed in establishing productive collaborations with the host country universities, 19 partner institutions, other Feed the Future ILs, and other relevant USAID programs in the target countries? How could the ME improve in building these collaborations? 6. In what ways has the ME been responsive to requests for technical assistance from the 19 partner institutions, USAID, the eight USAID Missions, and Feed the Future IPs working in aquaculture? In what ways could the ME have been more responsive? 7. Describe and highlight the research technologies that had the greatest impact or potential for the greatest impact. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (e.g., breeding vs. feeds development vs. post-harvest management)? What topics are promising areas for further investment in aquaculture and fisheries? 2.0 BACKGROUND Initiated in 2013, AquaFish IL is a successor to the Aquaculture and Fisheries Collaborative Research Support Program (AquaFish CRSP) that began in 2006. AquaFish IL is one of 24 Feed the Future ILs funded by the USAID/BFS. Its mission is “to enrich livelihoods and promote health by cultivating international multidisciplinary partnerships that advance science, research, education, and outreach in aquaculture and fisheries in environmentally and socially acceptable ways.”5 AquaFish IL adheres to USAID’s requirements by focusing core resources on aquaculture and striving to: • Develop sustainable end user-level aquaculture systems to increase productivity, enhance international trade opportunities, and contribute to responsible aquatic resource management; • Enhance local capacity in aquaculture and aquatic resource management to ensure long-term program impacts at the community and national levels; • Foster broad dissemination of research results and technologies to local stakeholders at all levels, including end users, researchers, and government officials; and • Increase host country productivity and capacity to contribute to national food security, income generation, and market access, and to lead research through scientific inquiry and problem-solving. AquaFish IL focuses on four broad themes: 1. Improved Human Health and Nutrition, Food Quality, and Food Safety; 2. Income Generation for Small-Scale Fish Farmers and Fishers; 3. Environmental Management for Sustainable Aquatic Resources Use; and 4. Enhanced Trade Opportunities for Global Fishery Markets. The 10 areas of inquiry are separated into two major categories: 1. Integrated Production Systems • Production System Design and Best Management Alternatives; 5 http://aquafishcrsp.oregonstate.edu 3 • Sustainable Feed Technology and Nutrient Input Systems; • Climate Change Adaptation: Indigenous Species Development; and • Quality Seed Stock Development. 2. People, Livelihoods, and Ecosystem Interrelationships • Human Nutrition and Human Health Impacts of Aquaculture; • Food Safety, Post-Harvest, and Value-Added Product Development; • Policy Development; • Marketing, Economic Risk Assessment, and Trade; • Watershed and Integrated Coastal Zone Management; and • Mitigating Negative Environmental Impacts. 2.1 AQUAFISH IL APPROACH AquaFish IL aims to enrich livelihoods and promote health by cultivating international multidisciplinary partnerships that advance science, research, education, and outreach in aquaculture and fisheries through environmentally and socially acceptable means. By bringing together resources from the U.S. and host country institutions and partners, AquaFish IL strives to strengthen the capacities of its participating institutions, increase the efficiency of aquaculture sustainably, and disseminate research results to a broad audience. Since 2013, AquaFish IL pursued topics relevant to aquaculture and fisheries research in order to develop thematic areas in eight countries, which are consolidated into five groups: 1) Kenya and Uganda; 2) Bangladesh; 3) Ghana and Tanzania; 4) Cambodia and Vietnam, and 5) Nepal. AquaFish IL focuses on four conceptual targets: 1) research; 2) capacity building; 3) information dissemination; and 4) gender integration (a cross-cutting target). To implement its mission, AquaFish IL further focuses on the following three products: 1) long-term degree-seeking students; 2) short-term training events aimed at host country stakeholders, and 3) relevant research investigations and dissemination of information to diverse audiences. AquaFish IL is led by the Lead Principal Investigator-Director at OSU and is administered through a Management Team and other support services at the ME. The core research program is implemented collaboratively through five subcontracting lead project U.S. universities. These lead project U.S. universities, in turn, contracted with five lead host country universities and several partner institutions. The U.S. and host country partner institutions collaborate on projects equally; however, the contractual mechanism for funding flows from the ME through various pathways to the field. Additional implementation occurs with the ME directly collaborating and contracting with host country and other partner institutions for research, capacity building, and outreach. AquaFish IL’s reach extends to 37 funded partner institutions and to 62 collaborative (unfunded) partners. The five subcontracting lead project U.S. universities and their country groups, titles, and themes are illustrated in Table 1, below. Table 1: U.S. Lead Universities, Regions, Titles of Activities, and Research Themes Lead U.S. University Region Title of Activity Research Themes Auburn University Kenya and Uganda Aquaculture Development in Kenya and Uganda: Advancing Cost-effective Technology, Market Assessment, and End￾user Engagement Income Generation for Small-Scale Fish Farmers and Fishers North Carolina State University Bangladesh Enhancing Aquaculture Production Efficiency, Sustainability, and Adaptive Measures to Climate Change Impacts in Bangladesh Environmental Management for Sustainable Aquatic Resources Use 4 Lead U.S. University Region Title of Activity Research Themes Purdue University Ghana and Tanzania Aquaculture Development and the Impact on Food Supply, Nutrition, and Health in Ghana and Tanzania Improved Human Health and Nutrition, Food Quality, and Food Safety University of Connecticut Avery Point Cambodia and Vietnam Improving Food Security, Household Nutrition, and Trade, Through Sustainable Aquaculture and Aquatic Resource Management in Cambodia and Vietnam Enhanced Trade Opportunities for Global Fishery Markets University of Michigan Nepal Advancing Aquaculture Systems in Nepal for More Social and Environmental Sustainability Environmental Management for Sustainable Aquatic Resources Use 2.2 STAKEHOLDERS AND IMPLEMENTATION To support a broad academic community and a wide array of stakeholders, the five subcontracting lead project U.S. universities work with five host country universities and 27 partner institutions. These partner institutions are comprised of additional U.S. and host country universities and aquaculture-specialized non￾governmental organizations (NGOs), as follows: • Kenya and Uganda (eight partner institutions); • Bangladesh (seven partner institutions); • Ghana and Tanzania (eight partner institutions); • Vietnam and Cambodia (two partner institutions); and • Nepal (two partner institutions). Table 2: Lead U.S. Universities, Lead Host Country University, and Local Partner Institutions Lead U.S. University Lead Host Country University Partner Institutions Auburn University Makerere University (through Spring 2016), National Fisheries Resources Research Institute (NaFIRRI (from Spring 2016) • U.S.: University of Arizona, Alabama A&M University, North Carolina State University • Kenya: University of Eldoret, Kenyatta University, Kenya Ministry of Agriculture • Uganda: Gulu University North Carolina State University Bangladesh Agricultural University • Bangladesh: Khulna University, Patuakhali Science and Technology University, Shushilan (NGO), Hajee Mohammad Danesh Science and Technology University, University of Dhaka • Philippines: Southeast Asian Fisheries Development Center – Aquaculture Division, Central Luzon State University Purdue University Kwame Nkrumah University of Science and Technology (KNUST) • U.S.: Virginia Polytechnic Institute and State University, University of Arkansas at Pine Bluff, University of Hawaii – Hilo • Ghana: Farmer Line, University of Development Studies • Tanzania: Sokoine University of Agriculture, University of Dar es Salaam – Institute of Marine Sciences (IMS), Western Indian Ocean Marine Sciences Association (WIOMSA) University of Connecticut Avery Point Inland Fisheries Research and Development Institute • U.S.: University of Rhode Island • Vietnam: Can Tho University 5 University of Michigan (UM) Agriculture and Forestry University • Nepal Agricultural Research Council (NARC) • Directorate of Fisheries Development AquaFish IL’s ME and partner institutions (U.S. and host country) offer technical support to USAID Missions in eight Feed the Future countries and to other USAID Feed the Future IPs working in aquaculture and fisheries. For this assessment, additional stakeholders include long-term degree-seeking students (BSc, MSc, Ph.D.), host country trainees who have attended short-term training events, and local AquaFish IL-supported researchers (former long-term student beneficiaries who now work as AquaFish researchers). The ET noted that there were many other AquaFish IL partners in industry, national and local governments, NGOs, and cooperatives. 3.0 METHODOLOGY AND LIMITATIONS 3.1 EVALUATION METHODOLOGY This evaluation was a desk study implemented by a two-person team between November 2017 and February 2018. The ET employed a mixed methods approach that included: 1) document reviews of annual reports and other AquaFish IL documents, such as monitoring and evaluation (M&E) plans, peer-reviewed publications, training reports, etc.; and 2) key informant interviews (KIIs) that were conducted remotely with relevant stakeholder groups in the host countries and the U.S. The ET did not conduct fieldwork. Document Review The ET reviewed relevant AquaFish IL documents, including: • AquaFish IL Requests for Proposal (RFPs); • AquaFish M&E plans; • Program implementation plans; • Annual work plans and reports; • AquaFish IL newsletters; • Program descriptions; • Technical reports; and • Relevant host country university and local partner program and research documents. The ET reviewed USAID documentation relevant to AquaFish IL and participating Feed the Future countries, technical data, and reference works related to aquaculture and fisheries. Document review informed the ET’s interview planning and provided context for probing questions. All documents reviewed are referenced in Annex D: List of Documents Reviewed and References. All citations are footnoted. Key Informant Interviews The ET conducted 43 KIIs with a purposive sample of representatives from AquaFish IL-associated stakeholder groups. The relevant stakeholder groups and KII respondents that were identified and included in the interviews were: ME personnel (n=4); personnel from the subcontracting lead project U.S. university (n=5), representatives from the host country lead partner institution (n=5); representatives from the local partner institutions (n=19); current or former long-term degree-seeking students (n=5); USAID Mission staff (n=1); USAID/BFS staff (n=1); and AquaFish IL-supported researchers (n=3)—former long-term degree-seeking students now working as researchers (see Table 3 and Annex E: Key Informant Interviews). 6 The ET worked with USAID/BFS to recruit and select respondents by discussing and agreeing on the list of key informants across all the AquaFish IL program implementation levels. USAID/BFS provided contact information to the ET for 15 potential key informants from across the eight Missions (Bangladesh, Cambodia, Ghana, Kenya, Nepal, Tanzania, Uganda, and Vietnam). The ET requested interviews with all of them. Two requests solicited short response emails, in which these USAID Mission respondents stated that they had insufficient experience with AquaFish IL to provide interviews. Despite several calls and follow-up emails to the Points of Contact (POCs) in the eight Missions, only one respondent completed a KII with the ET. Other identified groups—including past short-term trainees and other USAID Feed the Future IPs working in aquaculture related areas—could not be reached for interviews. As anticipated, locating and interviewing short-term trainees from the past was a challenge. The ET did not have access to this stakeholder group’s current phone numbers or email addresses, which prevented interviews. See Table 3 and Annex E: List of Key Informants Interviewed. The ET used phone or Skype to conduct all KIIs. To identify ideal POCs before interviews, the ET gave each institution a short synopsis of the program and outlined the overall purpose of the evaluation. Then the ET requested a list of staff with knowledge of the AquaFish IL program at each level of the implementation chain. Where possible, to provide gender balance, both men and women from the same stakeholder group were interviewed. The ET also conducted KIIs with five current students and three former long-term degree students (who received support from AquaFish IL and are now employed as researchers). Before contacting students and researchers, the ET informed interviewees of the evaluation purpose, told them that their participation was voluntary, and obtained a release form from the students (see Annex F: Request Form for Student Participation). The ET solicited verbal consent from all KIIs before the interviews. Table 3: Key Informant Interviews by Stakeholder Group AquaFish Stakeholder Groups Number of KIIs ME (OSU) – Lead Institution 4 U.S. Subcontracting Lead Project Universities 5 Host Country Lead Partnering Institutions (Universities) 6 Local Partner Institutions 19 USAID/BFS staff 1 USAID Mission staff 1 Long-term degree-seeking students 5 AquaFish-Supported Researchers (former students) 3 Total 43 To conduct the KIIs, the ET used three separate interview guides for the different stakeholder groups (see Annex G: Data Collection Tools), and customized questions to reflect specific interviewee experiences and account for the agreed-upon interview time (between 60-90 minutes). The ET used the KIIs to understand management processes related to each AquaFish IL activity and to generate recommendations for each of the seven EQs. The ET recorded the KIIs and used Cabbage Tree Solutions (an audio service transcription service) for transcriptions. The ET conducted preliminary data analysis throughout data collection and analyzed feedback on an ongoing basis to determine emerging trends and to generate probing questions for further inquiry. Data Triangulation The ET triangulated the data collected on the same questions and sub-questions iteratively across different groups of KIIs. In cases where the results obtained through one data source were less conclusive than 7 others, the ET used triangulation to strengthen the potential linkages and data accuracy. Triangulation (consensus between multiple interviewees and other data collection methods and sources) helped verify and validate findings that emerged and formed the ET’s conclusions. Content Analysis The ET used content analysis to review KII responses and identify and highlight examples of factors that contributed to or hindered the realization of AquaFish IL’s mission. During and after data collection, the ET aggregated data the obtained from the KIIs around common themes related to the seven EQs. The ET tabulated responses and disaggregated data by target country group and program theme. The ET also used trend analysis to examine outcomes related to the EQs over time and to identify patterns of convergence (or divergence) of program outputs and outcomes leading toward the stated mission. The ET used a systematic, rigorous data analysis approach to triangulate the data derived from the qualitative findings of the document review—this ensured that findings were credible and reliable. By using qualitative data in this way, the ET examined the context for why something did or did not happen and compared stakeholder perceptions of issues such as potential technical relevance or specific training or research, and other issues related to EQ7. Since the number of interviews was small, the ET conducted further content analysis using simple opinion mining (sentiment analysis) methods to count positive and negative statements. The ET reviewed each interview for statements pertinent to EQ answers and extracted certain keywords to determine how many remarks that used those keywords were positive or negative. A close review of the interview data indicated that no statements (for example, “program management was extremely poor” or its alternative, “program management was stellar”) exhibited extreme polarity. Therefore, three categories instead of five were selected. Example statements in each of the categories are: 1. Positive statement: “Collaboration between the partners was good.” 2. Neutral statement: “There was collaboration between the partners.” 3. Negative statement: “The partners didn't really collaborate.” 3.2 EVALUATION LIMITATIONS This evaluation’s methodology has limitations that affect the interpretation and generalization of the evaluation findings and conclusions. These limitations are discussed below. Subjective Measurements The use of qualitative approaches can result in the performance analysis being dependent on the ET’s professional opinions and experience. In turn, this may affect the explanation of findings, conclusions, and recommendations based upon the ET’s subjective interpretation. To mitigate this bias, the ET systematically triangulated findings across stakeholder groups and data sources and, where possible, drew evidence-based conclusions and recommendations based on expert opinion. Response Bias The limited number of USAID Missions willing to participate and the absence of in-country Feed the Future IPs to interview meant that these perspectives are not fully represented in the findings, conclusions, and recommendations. Although the ET received information for 15 Mission POCs in the eight countries where AquaFish IL was implemented, only one Mission respondent agreed to be interviewed. The ET did not interview in-country project implementers active in the aquaculture sector because, in the eight countries in which AquaFish was implemented, there were none. This introduced a bias in that the perspectives of university stakeholder groups (both U.S. and host country) was overrepresented in the evaluations, and other groups (Missions, practitioners) were underrepresented. 8 Desk Study This assessment is a desk study with no field visits and a limited number of remote interviews, which affected both the amount of data collected and the answers to key questions that would have been easily addressed through face-to-face interviews and direct observations. Further, interviews conducted via telephone or Skype present logistical challenges and afford limited opportunity for exchange or timely and relevant probing questions. To address this limitation, the ET worked closely with USAID and the U.S.- based and host country lead partners to identify relevant documentation and develop a reasonably representative set of key contacts across AquaFish IL stakeholder groups. Selection Bias Selection bias is an inherent risk when implementers help facilitate contact with program beneficiaries. The ET worked closely with the ME and subcontracting lead project U.S. universities to identify KII participants in host countries. The ET then worked with those host country lead universities and local partners to organize KIIs with selected long-term degree-seeking students and AquaFish IL-supported researchers. There is a potential risk that stakeholders chose the most active, responsive, or engaged beneficiaries, or that they chose individuals pursuing topics of specific personal interest. As a result, the ET may have only heard from students and researchers whose interests match those of the referring stakeholders. To counter this risk, the ET requested that stakeholders provide—in advance of KIIs—a list of selected students and researchers disaggregated by topic. Where possible, the ET identified individuals from this list, which mitigated the risk of selection bias. Evaluation Team Turnover This final evaluation took longer to start because of the unexpected departure of the first two team leaders. The first team leader joined the evaluation in February 2017 but dropped out in June 2017. The second team leader stayed for a month that spanned August and September of 2017 and then, citing another commitment, left. Neither team lead began collecting data. The Program Evaluation for Effectiveness and Learning (PEEL) recruited the third team leader who successfully saw the evaluation to completion. This turnover contributed to knowledge loss and review fatigue among both AquaFish IL participants and the ME. 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 EVALUATION QUESTION 1 To what extent has the ME effectively implemented and managed AquaFish IL’s activities in the five target regions of Africa and Asia? How effectively has the ME managed and coordinated among the five subcontracting lead project U.S. universities and host country universities, 19 partner institutions,6 USAID, and eight USAID Missions and their other Feed the Future IPs working in aquaculture, and the long-term degree-seeking students, host country trainees that have attended short-term training events, and local researchers? In what ways could implementation and management be improved? FINDINGS Most local partner university interviewees (n=12 out of 19) and subcontracting lead project university interviewees (n=13 out of 15) had positive opinions about the ME’s project implementation and management. Descriptions of the ME’s management frequently included terms such as “fast” and “helpful.” 6 Nineteen (19) partner institutions was the number used in the evaluation protocol approved by USAID but during the evaluation it was found that the actual number is 27 which is corroborated by the ME. 9 “In my opinion, Oregon State University has provided excellent leadership of the AquaFish Innovation Laboratory.” – Subcontracting lead project U.S. university participant Key informants at local partner and lead host country universities were equally complimentary when asked about AquaFish IL implementation and management, although they described having significantly less direct interaction with the ME. “We have [a] very good relationship with [subcontracting lead project U.S. university partner]. So, any problem, I talk to [subcontracting lead project U.S. university partner] and he talks to [ME personnel])” – Lead host country university participant KII participants identified four key management processes related to coordination and implementation of the AquaFish IL, including: 1. The initial development of AquaFish IL along its thematic lines (see Section 2.0 Program Overview); 2. Funding disbursement; 3. Coordination; and 4. Reporting. Program Development AquaFish CRSP ended in Fiscal Year (FY) 2011. For the next 18 months the program “experienced a 70 percent cut from normal funding levels and consequently was forced to reduce staff and project activities while waiting for a decision from Washington regarding the program’s future.” 7 On December 21, 2012, following three external program reviews, USAID invited OSU to submit a proposal for a new five-year AquaFish IL. In order to create a fair and open framework for later engagement with other universities, the ME personnel wrote and submitted the proposal without input from existing partner institutions that, potentially, could be brought in later. On March 31, 2013, the ME received notification of award from USAID for a five-year extension “to initiate new programs that build on past successes.”8 During the transition period, AquaFish IL operated on only 30 percent of normal funding levels. In its current phase, AquaFish IL operates under a new, USAID-approved title: Feed the Future Innovation Lab for Collaborative Research on Aquaculture and Fisheries, or simply AquaFish Innovation Lab.9 Shortly after receiving the AquaFish IL award, OSU issued an RFP designed to build a strong collaborative research program, elicit talent, and expand partnerships. Apart from the RFP mechanism, OSU commissioned research and engaged partner institutions from host countries and the U.S. directly in bridge-building activities. The ME’s RFP process resulted in establishing AquaFish IL’s lead subcontracting partners. Applicants were asked to structure their proposals around the two topical categories (Aquaculture and Fisheries) and 10 respective topic areas based on the judgment of the director. Following submission, the proposals were subjected to “a National Science Foundation (NSF)-style external review process and USAID programmatic and ME team reviews.” Care was taken to avoid conflicts of interest.10 Feed the Future ILs (Nutrition and Legume) personnel participated in the review. “The director selected the final portfolio of projects based on the weight of the external reviews, adherence to the design principles in the RFP, and overall alignment with the approved AquaFish IL mission.”11 AquaFish CRSP had a total of eight projects, from which five were chosen to continue through AquaFish IL. 7 Egna, H., Ichien, S., and Borberg, J. (eds), “AquaFish Innovation Lab - Seventh Annual Report,” OSU, Corvallis, Oregon, October 2013. 8 Ibid. 9 Egna, H., Ichien, S., and Borberg, J. (eds), “AquaFish Innovation Lab - Seventh Annual Report,” OSU, Corvallis, Oregon, October 2013. 10 Egna, H., Borberg, J., and Edwards, P. (eds), “AquaFish Innovation Lab - Implementation Plan 2013-2015,” OSU, January 2014. 11 Egna, H., Borberg, J., and Edwards, P. (eds,) “AquaFish Innovation Lab - Implementation Plan 2013-2015,” OSU, January 2014. 10 At USAID’s request, the ME also solicited proposals for work in Burma, which the ME reviewed and assessed to ensure alignment with AquaFish IL’s research goals. One ME informant stated that, due to political uncertainty in Burma, a full-scale project including “international travel, student support, and in￾country research and training activities” was eschewed. Burma remained an area of interest for USAID in FY 2016, so AquaFish IL’s ME developed a partnership with Yangon University to evaluate and enhance faculty and institutional capacity in aquaculture. This project leveraged another USAID Mission project in Burma. Also, in 2016, AquaFish IL established a partnership with UM through a subcontract to conduct a U.S.-based desk study on aquaculture and fisheries in Burma. Thus, two smaller (not lead) project studies involving Burma were conducted, one by the ME and one by UM. When asked during the KIIs if they felt that the proposal process and guidance—including the designated topical categories and topic areas—allowed them to adequately address the key aquaculture development priorities for their country or region, 63 percent of the host country interviewees (n=12) indicated that they did. Eighty-six (86) percent of the host country interviews, including students, stated that the research topics were broad enough to allow them to address key bottlenecks or national/regional priorities. Only two key informants, both from subcontracting lead project U.S. universities and working in Africa, disagreed with this view. Both stated that the ME-defined process limited their ability to address their institutions’ research priorities. Informants also stated that the process in each country included stakeholders from many different organizations. For example, in Bangladesh, an informant from the subcontracting lead project U.S. university described having to consult with the Development Bank of Bangladesh about development priorities. Likewise, in Nepal, the subcontracting lead project U.S. university consulted with a team that included the Director of Fisheries Development and staff from the NARC. “We’re looking at the priority fishes that benefit the community and there’s a list of those commodities, which are in the national strategy for fisheries research” – Host country lead University participant All informants agreed that the proposal process was effective and not overly burdensome. Subcontracting lead project U.S. university informants explained that they were familiar with host country institution and partner institution personnel and that long-established relationships and knowledge of administrative processes at these institutions facilitated program proposal development. Funding Disbursement The ME implemented AquaFish IL’s research activities largely through a multi-tiered contracting approach. They started by subcontracting five U.S. universities, which subsequently subcontracted to the five host country lead universities. The lead institutions, in turn, subcontracted 27 local partner institutions. Interviewees compared this arrangement illustrated in Figure 1, showing the ME in the center, the subcontracting lead project U.S. universities in the next circle, the host country lead institutions in the next circle and, finally, the 27 local partner institutions in the outer circle. Figure 1 is a simplified model of how the majority of funds flow through AquaFish IL’s structure. Exceptions did exist where the ME directly funded host country and partner institutions. 11 Figure 1: Illustrative AquaFish Multi-Tiered Contracting Structure Informants from all stakeholder groups explained that funds for research, short-term training, and longer￾term student engagement are disbursed in a cascading manner from the ME through the various entities. U.S. subcontracting lead project institution personnel pointed out that a significant role of the ME is ensuring efficient disbursement of funds to the subcontracting lead project U.S. universities. “The arrangement is like a Russian Easter egg with the ME in the center.” – Partner institution participant This was primarily the case among partner institutions, where funds passed through several entities before being received. One ME participant mentioned that the invoice payments to the subcontracting lead project U.S. universities occur on a cost-reimbursable basis and payment is made within 30 days of receipt. This informant also mentioned that no sub-awardees “have experienced a funds shortage or funding delay as a result of processes that are in place to ensure delivery, cost and research compliance.” Coordination In addition to the ME, subcontracting lead project U.S. universities personnel explained that they play a significant role in coordinating host country university and partner institution activities. The subcontracting lead project U.S. universities respond to the Request for Applications (RFA), assemble the team of researchers and associates in the host countries and take care of subcontracting and disbursing funds to these entities. The subcontracting lead project U.S. universities also provide the majority of technical assistance to the host country counterparts at both the lead and partner levels. They also provide non￾federal cost-share funds required for AquaFish IL and are responsible for disbursing those funds. The relationships between subcontracting lead project U.S. universities and host country lead universities and partner institutions was established during (or before) the start of AquaFish IL. An example of the collaboration is the development of African Lungfish farming in Uganda (Text Box 1). Text Box 1: Indigenous Species Development in Uganda Developing African Lungfish (Protopterus aethiopicus) Farming in Uganda, East Africa: The development of African lungfish (Protopterus aethiopicus) farming falls under AquaFish’s thematic area of Indigenous Species Development and Mitigating Negative Environmental Impacts. The African lungfish population is declining due to overfishing, environmental degradation, and conversion of wetlands into agricultural land. The ability of lungfish to survive in low oxygen environments, coupled with high market demand, makes it an appropriate candidate for aquaculture development. Local Partner Institutions (27) Host Country Lead Universities (5) U.S. Lead Institutions (5) Management Entity (OSU) USAID 12 Under AquaFish IL, a research team from Auburn University partnered with the National Fisheries Resources Research Institute (NaFIRRI) in Uganda to develop low-cost, captive breeding and hatchery technologies for African lungfish. Captive breeding experiments began in FY 2014.12 In FY 2015 the breeding program-produced fingerlings had grown to the market size of 200 grams. Feeding trials were also conducted to determine the most effective and economical means of culture.13 Key informants felt the lungfish aquaculture work, especially the captive breeding, was very impactful for reasons that included drought resilience, low oxygen tolerance, cultural importance, declining wild populations, and high market value. “It fits the criteria of what USAID wants. It has attributes like it’s an air-breathing fish, and it can withstand drought areas or drought season because of its burrowing habits. It can aestivate for three months. And there are very many communities which depend on it for livelihoods and nutritional purposes.” – Local partner institution participant “It tastes very good and some cultural beliefs are that is has nutritional or medicinal purposes and helps many people with HIV/AIDS. You find them making a soup because that’s the belief of the fisherfolk that it has some medicinal value. But the thing is that the numbers are declining because of overfishing and this is a big problem. With people destroying wetlands, we needed to come up with technologies to propagate this fish.” – Local partner institution participant The partner institutions subcontracted through the host country universities are typically removed from regular subcontracting conversations with the ME about financial management—except when problems arise. While host country university partners tend to meet with the ME at regional meetings and during country site visits, interviewed personnel from the partner institutions stated that they interacted with the ME through occasional emails (although they mentioned personal interactions at regional meetings). The lead host country institutions interact frequently with the ME through emails and monthly Skype meetings. One external issue raised by key informants from both subcontracting lead project U.S. universities and host country universities was the need for host country universities to comply with U.S. universities’ requirements and regulations relating to human subjects and research animals. This often led to delays initiating certain research investigations of a month or more. Occasionally, work in the host countries was also delayed by natural disasters (specifically earthquakes in Nepal and flooding in Uganda). While the ME attempted to mitigate these challenges, delays nevertheless occurred. ME personnel described their relationship with USAID as a positive one. AquaFish IL has a USAID Agreement Officer Representative (AOR) who supervises the program; however, this position has seen changes in personnel. The current AOR has attended AquaFish IL’s annual meetings, made site visits, and liaised regularly with the ME. In contrast, informants from most institutional stakeholder groups (subcontracting lead project U.S. universities, host country universities) describe relationships with the eight USAID Missions and their Feed the Future IPs working in aquaculture as virtually non-existent. A notable exception was significant ME collaboration with two other Feed the Future ILs related to aquaculture (Nutrition IL and Horticulture IL in Bangladesh) where the director served as the technical advisor on a cross-IL project. The Feed the Future IL community, along with USAID/BFS and Missions, meets annually for a Feed the Future summit in each focus country. AquaFish IL has participated in previous Feed the Future summits in Ghana, Uganda, Malawi, Senegal, and Tanzania. ME personnel encouraged the subcontracting lead project U.S. university personnel to pay courtesy visits to the USAID Missions in countries where they do site 12 Egna, H., K. Goetting, C. Price, J. Borberg, P. Edwards, S. Ichien, and F. Evans, “AquaFish Innovation Lab - Eighth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, October 2014. 13 Egna, H., K. Goetting, J. Borberg, S. Ichien, C. Price, M. Chow, and F. Evans, “AquaFish Innovation Lab - Ninth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, September 2015. 13 visits. Informants in all relevant stakeholder entities except USAID BFS staff (n=42) stated that USAID Mission staff were interested but not engaged in AquaFish IL. These informants indicated that USAID staff non-engagement was mainly due to time constraints. For example, a subcontracting lead project U.S. university personnel described being turned down for visits to Missions in countries in both Africa and Asia because Mission staff was “extremely busy.” Between 2006 and 2011, the USAID Mission in Mali funded associate awards for AquaFish CRSP. The institutional stakeholders stated that these awards showed the support of the local USAID Mission and directly involved Missions in AquaFish CRSP activities. However, no new associate awards were made between 2013 and 2018 (respondents did not know why), except for one associate award from AquaFish CRSP carried over for a year into AquaFish IL. This associate award was included in AquaFish IL’s M&E plan “partnerships strengthened among U.S. and host country universities, NGOs, [National Agricultural Research System] NARS, and USAID Missions through associate awards.” The associate awards and their role in having USAID Missions engaged were clearly identified by all ME interviewees (n=4) as important. Reporting Ten (10) respondents at the ME and their partner institutions described administrative procedures associated with AquaFish IL as “heavy,” noting they required periodic (monthly, quarterly, annual) reports instead of summary reports upon research completion. The reporting that was required for compliance with USAID guidelines was the most difficult aspect of AquaFish IL to manage. When asked how the administrative burden compared with other research funding sources such as NSF grants, informants stated that the burden was considerably heavier. “We have quarterly reports. We have completion reports. We have quarterly conference calls. We also have annual meetings…To add much more to that, it becomes almost a full-time job just to manage and I don’t think anybody on the science side would want to have that.” – Subcontracting lead project U.S. university participant One ME informant mentioned that the numerous requests and periodic data calls from USAID led to ineffectiveness and what they felt was an inefficient use of time, which otherwise could be dedicated to research activities. “I’m talking every day, another request. Every day, you’re dropping everything. ‘What percentage do you put into fisheries or aquaculture?’ ‘Send us three success stories.’ . . . It's these offline, one￾off, need-this-by-tomorrow constantly happening. We shield our subs [subcontractors] from that. That comes at a great cost to OSU and what I can accomplish too.” – ME participant All ME personnel interviewed (n=4) expressed considerable concern for the administrative burden of compliance placed on subcontractors, noting their interest in working to reduce this where possible. For example, ME personnel tried to work with the subcontractors on required deliverables. Likewise, several informants described the ME personnel as being helpful in managing the administrative work. “It has become more administrative in recent time. I think [ME personnel] is very focused on the fact that there is a heavy administrative burden.” – Subcontracting lead project U.S. university participant “As a researcher who has worked either directly for the federal government or on government￾funded grants for over 40 years, I appreciate the efforts of OSU to work with USAID without unnecessarily involving me in dealing with the vagaries of the federal bureaucracy.” – Subcontracting lead project U.S. university participant CONCLUSIONS • The ME has effectively implemented and managed the four key management processes identified by stakeholder informants related to management and implementation of AquaFish IL. The principal elements challenging the implementation and management were exogenous factors beyond the ME’s control, including: 14 o An 18-month delay and reduction in funding between AquaFish CRSP and AquaFish IL; o Delays in funding disbursement between the layers in the multi-tiered contracting structure of the program; o Host country institution compliance with subcontracting lead project U.S. university requirements and regulations relating to human subjects and research animals; o The burden of administrative procedures, especially compliance with USAID reporting guidelines; and o Project implementation delays due to natural disasters. • The ME managed and coordinated effectively among the five subcontracting lead project U.S. universities and host country lead universities, 27 local partner institutions, and USAID. Time constraints on the part of Mission staff challenged coordination with the eight USAID Missions and their Feed the Future IPs working in aquaculture. No new associate awards were made during AquaFish IL (Phase II) between 2013 to 2018. RECOMMENDATIONS • The ME should work with USAID/BFS particularly at the request for proposal (RFP) stage to streamline their multi-tiered contractual process to ensure that funds are dispersed in a timely manner to the host country research implementers. • The ME should reorganize its administrative structure to reduce the several layers of partnerships in the host country that are inherently bottlenecks making management of the program cumbersome, which led to delays that adversely affect research schedules. • The ME should also require, in their RFPs from host countries, that lead host country universities innovatively and creatively propose how they will establish effective partnerships with other in￾country stakeholders beyond the local research partner institutions. This requires concerted efforts in the RFP process to require host country research institutions to partner with existing Feed the Future program practitioners and policymakers in target countries to set local research priorities and promote the relevance of AquaFish IL products. 4.2 EVALUATION QUESTION 2 To what extent have effective communication and coordination supported program implementation and outcomes regarding research and dissemination and capacity development? In what ways could communication and coordination be improved between USAID/BFS, the eight USAID Missions, the ME, the five lead project U.S. institutions and host country universities, and other stakeholders, as a way to better achieve program goals and objectives, create and carry out a shared research agenda, meet reporting requirements, and expand collaboration? FINDINGS Key informants from U.S. and host country institutions (n=29) praised the ME for their communication and coordination abilities. These opinions are corroborated by the fact that 92 percent of the statements expressing opinions were positive. Ninety-six (96) percent of them also praised the overall AquaFish IL communication and coordination strategies. Communication and coordination occurs through conference calls, the AquaNews newsletter, AquaFish IL website, emails, and annual meetings. Combined, all of these allow program participants (from ME personnel to students) to feel informed and connected. Informants also mentioned that the cohesive nature of the aquaculture research community (born from the fact that many have been working together for years) is a key factor in effective communication and coordination. “It almost becomes like a family…” – Subcontracting lead project U.S. university informant 15 Informants stated that the cohesion has led to a deeper knowledge of research objectives and strong personal bonds between institutional personnel. Many key informants from the subcontracting lead project U.S. universities, host country lead universities, and the local partner institutions mentioned having been part of AquaFish CRSP. “I am a member of the aquaculture family. We have been working together for many years.” – Host country lead university participant Text Box 2 presents an example of successful collaboration in Bangladesh that yielded useful results, had positive local impact, and allowed development of a species for integration into production systems. Text Box 2: Collaborative Research on Integrated Production Systems in Bangladesh Developing River Catfish (Pangasius hypophthalmus) Farming in Hyposaline Waters in Bangladesh: The river catfish (Pangasius hypophthalmus) is a popular culture species in Bangladesh where approximately 300,000 metric tons (MT) are produced annually. However, until now, culture of this species has been restricted to freshwater systems of the Mekong Delta, Thailand, and parts of Bangladesh. Pangasius was introduced to Bangladesh from Thailand 1990s and is farmed primarily in the North and Central regions of the country. Farming involves using freshwater intensive, semi-intensive, and extensive methods. In Bangladesh, polyculture with both carp and tilapia is also practiced.14 The Barisal and other coastal regions of Bangladesh are increasingly impacted by climate change, which results in increased storm activity and salt-water intrusion. Nearly 40 percent of farmable water bodies in the greater Barisal district are now hyposaline. These regions have also seen a significant decline in their shrimp farming industry due to disease problems. Half a million fishers live in poverty in the Barisal coastal region. During the AquaFish IL project, a collaborative team from the University of North Carolina, Patuakhali University in Bangladesh, and Bangladesh Agriculture University tested the efficacy of growing Pangasius in saline environments as a means of providing food and income for Bangladeshi coastal residents. Initial results showed that Pangasius survived, grew, and converted feed in 10-12 parts per thousand (ppt) salinities—just as well as in freshwater. In fact, researchers recorded fewer disease incidences. Using locally produced feeds, researchers also attained increased production of Pangasius from two to three fish per square meter (m2) in brackish water. Polyculture of Pangasius with tilapia and koi (an indigenous fish to Bangladesh) in hyposaline waters was also successfully demonstrated during the Lab. Notably, this work was the focus of a USAID Success Story in October 2015. Key informants expressed the idea that this work had great potential for improving the livelihoods of people in the coastal areas of Bangladesh. One host country researcher estimated that, due to this breakthrough, the production of Pangasius could double, impacting a large number of people. “So, our main target is one million people all over the coastal area with Pangasius culture. We can expand this.” – Host country lead university participant “I think we’ve developed a new industry in the sense that we have shown that you can grow Pangasius, which is a freshwater catfish in hyposaline waters. It can grow in 10 to 12 ppt, which is what a lot of these ponds now are.” – Subcontracting lead project U.S university participant Email, phone, and Skype are the primary methods of communication between the ME, subcontracting lead project U.S. universities, and the lead host country personnel. The ME conducted quarterly conference calls with subcontracting lead project U.S. university personnel to discuss relevant issues. During 2017, the ME also initiated quarterly calls with host country university personnel. Relevant information from these meetings is disseminated to the host country lead university and local partners via email. Subcontracting lead project U.S. university personnel and collaborators visit their respective host 14 AquaNews Volume 30/Summer/Fall 2015. 16 countries once or twice a year to provide short-term technical support and monitor progress. Host country informants described these visits as “extremely useful” to both the U.S. and host country institution partners in terms of capacity building and information dissemination. The ME also conducts regular calls with the USAID AOR. ME personnel produce and disseminate a quarterly newsletter, AquaNews, which details current events related to AquaFish and includes a list of recent publications. They also maintain an extensive website that makes available publications and reports produced through the program and provides updates on the Lab activities. Unfortunately, in mid-February of 2017, it became apparent that the website was hacked. While some material was lost, the site was restored and relaunched in April 2017. “Since the launch, the AquaFish website has averaged 197 hits per day, with a total of 12,466 sessions, 7,889 users, and 59,360-page views” – ME participant AquaFish IL sponsors annual meetings, which nearly all program personnel attend, including longer-term degree-seeking students (BSc, MSc, Ph.D.). At these meetings, AquaFish IL researchers deliver presentations on their current research project outcomes and program personnel can interact. When possible, these meetings are held in conjunction with larger aquaculture meetings, such as the World Aquaculture Society (WAS) meetings. The ME also holds regional meetings where personnel from either Asia or Africa come together. The USAID AOR attended some meetings. Key informants at all levels of AquaFish IL involvement see these annual and regional meetings as beneficial. Informants described them as an opportunity for partners to share information, form collaborations, and improve capacity. “The [annual] meetings were the best opportunity to interact with other staff and get to know what they are working on.” – AquaFish supported researcher participant Host country and local participating institutions personnel and researchers (n=33) described these regional and cross-regional meetings and exchanges, both in Africa and Asia, as the most useful communications and learning opportunities. These same informants noted that these were the activities they would most like to see expanded. Notably, host country and local participating institution informants (n=9) stated that regional exchange meetings were most useful when they occurred in countries with “more developed” aquaculture sectors. Examples of countries they would find most useful to visit and learn from were Thailand and Mexico. “The opportunity to visit other countries and see what is being done is really valuable. It allows us to see technologies that are not available in our country.” – Host country lead university participant As mentioned in EQ1 and further developed in EQ5, communication was often minimal or lacking between the USAID Missions, the ME, subcontracting lead project U.S. universities, and local partner institutions. Views on this communication matter varied across host countries and respondents. Some of the KIIs stated that they attempted to establish communications with the Missions with limited success (n=8). Of the 15 USAID Mission contacts USAID/BFS provided to the ET for this evaluation, only four individuals responded and only one KII was conducted. One USAID Mission staff responded by email that their Mission “is not aware of any AquaFish activities in the country.” Conversely, a key informant working in the same country stated: “So, every time, I’ve gone to [host country], I contacted the [US]AID Mission and asked for an appointment to meet with somebody to talk about what we’re doing to see what they think we should be doing. One of those times, they followed through, they actually had a meeting. Whenever we try to get information from them or communication, we really just don’t get anywhere.” – Subcontracting lead project U.S. university participant “The (ME) Director was warmly greeted by Mission personnel in AquaFish host countries in Ghana, Kenya, Nepal, Cambodia, Kenya, Uganda, and Bangladesh. Half of these visits took place 17 in the Mission building, and half were in country but outside the Mission. The courtesy visits helped to maintain good relations for in-country teams, who also engaged the Missions themselves.” – ME participant CONCLUSIONS • Effective communication and coordination between institutional stakeholders has been key to AquaFish implementation and outcomes regarding research and dissemination, and capacity development. Communications and coordination activities supported by AquaFish IL, including conference calls, the newsletter, the AquaFish website, emails, and annual meetings have been extremely helpful. The critical element to the Lab’s success has been the highly cohesive nature of the aquaculture research community—a relationship derived from extensive experience in collaboration with the stakeholder institutions, intimate knowledge of research objectives, and strong personal bonds between institutional personnel. This cohesion provides a strong capacity building platform for newcomers to the arena such as partner institutions, students, and U.S. universities. • Communication and coordination could be improved by expanding inter-regional exchanges, especially between Feed the Future countries with less developed aquaculture sectors and countries with more developed aquaculture sectors. Exchanges offering opportunities for exposure to appropriate technologies unavailable in their own countries are particularly useful to participants. • Communication between AquaFish IL personnel and the USAID Missions in the host countries is minimal to non-existent, depending on the location. RECOMMENDATIONS • At the start of future aquaculture IL projects, the ME and USAID/BFS should agree to establish a clear mechanism for engaging and communicating with USAID host country Missions. • In future aquaculture ILs, USAID/BFS should increase resources for further expansion of inter￾institutional exchanges, especially exchanges that provide opportunities for partners from less developed institutions to gain exposure to appropriate technologies unavailable in their own countries. In particular, this could include exchanges that enable personnel to participate in strategic research partnership initiatives. • The ME should establish an in-country aquaculture steering committee that is made of researchers, practitioners, and policymakers to oversee and guide the in-country research directions instead of relying on the lead host country universities, who may have a bias towards their specific topics. 4.3 EVALUATION QUESTION 3 To what extent did the AquaFish IL generate quality research technologies/outputs using appropriate metrics (e.g., peer-reviewed publications)? In what ways have AquaFish IL ME personnel and the five subcontracting lead project U.S. institutions fostered dissemination of research results to host country stakeholders? What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward? FINDINGS Research production during AquaFish IL has been broken down into two phases: 2013-2015 and 2016- 2018. The ME published an initial Implementation Plan to cover research work for 2013-2015. The plan detailed 33 research investigations that were shared between the five subcontracting lead project U.S. universities and their host country lead partners. Two further addendums, each containing one further 18 investigation, were later added. 15 Subsequently, the ME developed a second Implementation Plan that detailed 26 investigations building upon successful projects to cover research activities for 2016-2018. 16 The ME personnel approved these investigations through an NSF-style review process that required experts in the proposed research field (with no conflict of interest with the applicant) to review the proposals. Besides reviewing the original RFA submissions, ME personnel, USAID, and other USAID Feed the Future program (Legume IL and Nutrition IL) staff reviewed the proposals for programmatic content. Work outlined in the 2013-2015 Implementation Plan had a 20 percent U.S. non-federal match requirement, and work in the 2016-2018 Implementation Plan had a 25 percent U.S. non-federal match requirement. In all but two cases, subcontracting lead project U.S. university and host country lead university key informants felt that, although the research topics aligned well with host country priorities for aquaculture development, there were other important research issues that should have been considered. For example, some of the KII participants felt that there were research issues of local relevance that might have been included if there had been more consultation with local aquaculture IPs. This was described by two participants in the following way: “I think there are a lot of things that could have been done and that would have been more innovative and less restricted.” – Subcontracting lead U.S university participants “There is very limited leeway for the host countries to develop topics that are currently germane.” – Subcontracting lead U.S. university participants Upon award in March 2013, USAID reviewed and approved the AquaFish IL M&E plan. Later, in September of 2013, the ME amended the plan, and USAID reapproved it, indicating that AquaFish IL and USAID/BFS had mutually agreed upon monitoring indicators. The AquaFish IL USAID-approved M&E plan had the following benchmarks listed for research in years 2-5 (2014-2017): a. Innovative aquaculture and fisheries technology or strategy developed and disseminated throughout each region (Asia or Africa); b. AquaFish IL activities remain locally appropriate by receiving regular input through the Regional Centers of Excellence and Development Theme Advisory Panels; c. Established baselines and produced measurable increases in farm productivity, farmer incomes, market access, and export value achieved following adoption of AquaFish IL’s recommendations and technologies; d. Threats to biodiversity resulting from aquaculture activities ameliorated, and biologically significant areas positively impacted, and e. Using baseline information, make positive changes for generally improved household (or targeted group) access to high-quality foods. Review of AquaFish IL’s annual reports showed that the Lab met its first benchmark (a). AquaFish IL developed at least one innovative aquaculture technology that was disseminated in each region (benchmark a, illustrated in Text Boxes 1 and 2). However, benchmarks (b-e) were addressed in the implementation plans and annual reports but, since they lacked baseline data and well-defined performance measurement metrics, were difficult for the ET to evaluate. Therefore, the ET could not determine whether or not benchmarks (b-e) were met. There are several ways to evaluate research quality, including its relevance and the usefulness of its products to end users. However, this evaluation used peer-reviewed publications as the primary metric to determine the number of publications in the scientific community and to evaluate the quality of research outputs by AquaFish IL personnel. 15 Egna, H., Borberg, J., and Edwards, P. (eds), “AquaFish Innovation Lab - Implementation Plan 2013-2015,” AquaFish Innovation Lab, OSU, Corvallis, Oregon, January 2014. 16 Egna, H., Borberg, J., and Hyman, A. (eds), “AquaFish Innovation Lab - Implementation Plan 2016-2018,” AquaFish Innovation Lab, OSU, Corvallis, Oregon, December 2016. 19 “Peer review has been defined as a process of subjecting an author’s scholarly work, research, or ideas to the scrutiny of others who are experts in the same field. It functions to encourage authors to meet the accepted high standards of their discipline and to control the dissemination of research data to ensure that unwarranted claims, unacceptable interpretations, or personal views are not published without prior expert review.”17 In September 2013, the ME proposed and established a target indicator for measuring progress for the five subcontracting lead project U.S. universities and host country lead universities—produce five peer￾reviewed publications per lead project (U.S.-based researcher and host country research partners) every fiscal year. These would entail collaborative efforts among all participants. The total target was 100 peer￾reviewed publications over the five years of the Lab. However, an ME informant stated that this indicator was a challenging metric to measure. A review of the five AquaFish IL reports showed a total of 103 peer￾review publications between 2013 and 2015—all attributed to the implementation plans, which exceeded the target. Due to the natural lag time between completing investigations and publication approval, peer￾review publications attributable to AquaFish IL will continue to come into print for several years after the end of the project. This suggests that the target will be surpassed by a wide margin. 18 19 20 21 22 KIIs echoed this sentiment, with one subcontracting lead project U.S. university personnel saying: “We lagged a little bit early on, and of course there’s that lag anyway because you have to do the research and then you have to write the thing, and then finally get it published. So, there’s that lag anyway.” – Subcontracting lead project U.S. university participant Key informants (n=41) praised the publication record of AquaFish IL and the opportunities it afforded for publication, as evidenced by 94 percent positive statements from interviewees. Dissemination of research results from AquaFish IL was expected to occur at multiple levels to host countries, USAID Missions, other relevant Feed the Future programs, scientific/academic communities, government agencies and, ultimately, host country aquaculture farmers. For stakeholder groups with Internet-connected devices, the AquaFish IL website and AquaNews newsletter also serve as sources of information dissemination. 23 24 Users can access and download reports and publications from the AquaFish IL website and obtain research result synopses and other information from the newsletter. Key informants (n=29) were positive about the public dissemination efforts of AquaFish IL. This was evidenced by the 99 percent positive statements from interviewees from these KIIs, all of whom expressed opinions on this issue. However, no AquaFish IL ME or other personnel made direct mention of disseminating research findings to USAID Missions or other aquaculture-related host country Feed the Future program implementers. AquaFish IL’s ME management staff stated the following regarding dissemination of results: “The ME frequently disseminated research results to USAID, including the Missions . . . [T]he ME wrote two research ‘dissemination’ articles for the USAID Feed the Future newsletter, and four for USAID’s knowledge management group, AgriLinks. Other examples include the fact that [the] ME presented or produced over ten technical talks and posters for USAID events.” 17 Kelly, J., Sadeghieh, T., & Adeli, K., “Peer Review in Scientific Publications: Benefits, Critiques, & A Survival Guide,” EJIFCC, 25(3), 2014. 18 Egna, H., Ichien, S., and Borberg, J. (eds), “AquaFish Innovation Lab - Seventh Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, October 2013. 19 Egna, H., K. Goetting, C. Price, J. Borberg, P. Edwards, S. Ichien, and F. Evans, “AquaFish Innovation Lab - Eighth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, October 2014. 20 Egna, H., K. Goetting, J. Borberg, S. Ichien, C. Price, M. Chow, and F. Evans, “AquaFish Innovation Lab - Ninth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, September 2015. 21 Egna, H., K. Goetting, J. Borberg, S. Ichien, B. Goodwin, S. Bodman, F. Evans, and L. Carroll, “AquaFish Innovation Lab - Tenth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2016, January 2017. 22 Egna, H., K. Goetting, J. Borberg, S. Ichien, A.A. Hyman, L. Carroll, and F. Evans, “AquaFish Innovation Lab - Eleventh Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2017, November 2017. 23 https://aquafishcrsp.oregonstate.edu. 24 https://aquafishcrsp.oregonstate.edu/aquanews. 20 For the scientific and academic community in the host countries, research results are disseminated both through peer-reviewed publications and via oral presentations and posters at meetings and symposia. Results are shared at both regional and annual meetings of AquaFish IL personnel. AquaFish IL annual meetings are held in conjunction with large aquaculture conferences such as the WAS Annual Meeting. In one case, AquaFish IL hosted a dedicated technical section to showcase its research findings.25 Host country researchers and students involved with AquaFish IL prepare and deliver presentations at the annual AquaFish IL meetings. However, some informants explained that language challenges can be a limiting factor for AquaFish IL researchers from various regions to present their findings. One informant explained that a significant difference between host country stakeholders in the African and Asian countries is English fluency—African host country stakeholders can often speak and read English, whereas many stakeholders in Asian host countries cannot. “Uganda is an English-speaking country, so even for students, you just have to edit it here and there.” – AquaFish-supported researcher participant Researchers were permitted to request assistance from other AquaFish IL researchers. For example, one subcontracting lead project U.S. university researcher who focused on Asia visited Africa to provide training and technical assistance to the local partner institutions. A local partner participant described the experience: “In the AquaFish scheme, we are allowed to borrow from other projects. So, for example, we had a weakness in breeding, so we had to get expertise from another project, so we’re working together with him in developing that breeding technology.” – Local partner institution participant Documentation shows that AquaFish IL placed a strong emphasis on short-term training (1-7 days) for disseminating research findings. Extension and outreach of research results to end users, such as farmers, appear to follow established extension methodologies, although some innovative exceptions have included outreach through schools. For example, one program in Nepal created a series of single fish ponds in high schools to disseminate information to students and their parents (see Text Box 3). Text Box 3: Innovative and Adaptive Projects Developing School Fish Ponds and Teaching Curriculum to Enhance Household Nutrition and Food Security in Nepal: In Nepal, rural poverty is a key factor affecting the health of women and children. Forty-one (41) percent of Nepali children under the age of 5 are malnourished. Malnutrition affects 8 percent of the total population. It is recommended that 33 percent of protein come from animal sources. However, in Nepal, only about 10 percent of protein consumed comes from animal sources. Pond aquaculture provides an opportunity for rural Nepalese to have a readily available animal protein source throughout the year. Not only does fish contain high-quality protein, but it also includes essential vitamins, minerals, and fatty acids.26 25 AquaNews 32/3, Fall 2017. 26 https://reliefweb.int/report/nepal/fish-fill-ponds-plates-and-pocketbooks-nepal. 21 AquaFish IL researchers from UM, OSU, and the Agriculture and Forestry University of Nepal collaborated to work with four local schools in the Chitwan and Nawalparasi to establish fish ponds.27 Ponds ranging from 76-264 m2 were constructed and stocked with tilapia and a variety of carp species and a curriculum was developed to teach students about aquaculture. In each school community, two women’s groups were formed and given training on aquaculture and how it relates to family nutrition. Two more schools were added to the program in FY 2017.28 Both students’ and women’s groups garnered increased knowledge of aquaculture due to the program. After one year, pond ownership among student households increased by 4 percent, and fish consumption rose by 47 percent. The program was the subject of a USAID Success Story, which explained how these activities showed significantly increased awareness of the value of fish consumption for rural households in Nepal. Informants also saw value and innovation in the program. “(The project) involved students from grades eight, nine, and ten, and we had aquaculture training for them every week. We built one pond and we show them how to culture fish. This is basically awareness for the aquaculture training for the young students. The parents of the students basically relate [to this experience based on their own experience with aquaculture]. They now know the importance of fish, the importance of nutrition. That’s the program we did, and it had a good success.” – Host country lead university participant In Bangladesh, AquaFish IL researchers contracted a community-based NGO to implement 45 demonstration sites in three coastal provinces. Informants explained that these types of projects served as experiential learning nodes for farmer groups. “We have implemented 45 demonstrations in three districts of the coastal belt… Demonstration is on the crab and tilapia polyculture and the installation of the demonstration plot, how they can practice in the field.” – Local partner institution participant In Ghana, Uganda, and Kenya, AquaFish IL researched use of cell phone-based technologies for providing information to aquaculture farmers, educators, and buyers (see Text Box 4). Cell phone platforms were developed to provide technical information and fish prices. Text Box 4: Coordinating Around Marketing, Economic Risk Assessment, and Trade Development of Mobile Phone-Based Marketing and Information Networks in Ghana, Kenya, and Uganda: The Feed the Future Initiative began in 2009. Between 2010 and 2015, the number of people in Feed the Future countries using mobile phones increased by 75 percent, with an exponentially larger increase in people using smartphones. The boom in mobile and smartphone usage increased the potential for improving outreach and information sharing via these platforms. Mobile phones are becoming more available across Africa, with ownership ranging from 65 to 83 percent between 2013 and 2015 in Uganda, Ghana, and Kenya.29 In response, “USAID, through its Bureau for Food Security and the U.S. Global Development Lab, has been facilitating broader adoption of digital technologies across Feed the Future.”30 27 Egna, H., K. Goetting, C. Price, J. Borberg, P. Edwards, S. Ichien, and F. Evans, “AquaFish Innovation Lab - Eighth Annual Report,” AquaFish Innovation Lab, OSU, Corvallis, Oregon, October 2014. 28 Egna, H., K. Goetting, J. Borberg, S. Ichien, A.A. Hyman, L. Carroll, and F. Evans, “AquaFish Innovation Lab - Eleventh Annual Report,” AquaFish Innovation Lab, OSU, Corvallis, Oregon. 2017, November 2017. 29 AquaNews 32/4, Summer 2017. 30 Burns, Payne, Agrilink Blog 2016. 22 AquaFish IL researchers from Purdue University, Auburn University, OSU, NaFFIRI and Makerere University in Uganda, KNUST in Ghana, and Eldoret University in Kenya leveraged these technology trends by coming together and collaborating to develop cell phone-based applications for providing aquaculture market information to farmers. In Kenya, researchers evaluated the possibility of building aquaculture market information into an existing system for capture-based fisheries called Enhanced Fish Market Information System (EFMIS). Concurrently, researchers in Ghana developed the Seafood Market Information System (SMIS), which allows buyers and farmers to access and enter data regarding fish prices and volumes.31 In Uganda, where cell phone use is lower than in Kenya or Ghana, preliminary research was conducted on needs for a cell phone-based program, including marketing and technical information. Eight modules were developed around pre-identified needs such as pond management, water quality, and market prices. 32 Key informants spoke highly of the potential impact of cell phone-based information and marketing technologies as aquaculture develops and smartphone usage increases. “We think the mobile phone app is one way to get everybody ready access to technical information, to market, the prices, to the availability of input.” – Subcontracting lead project U.S. university informant “I think it’s a great technological advance that the people who can put these phone apps together are really bright folks and I just see a real future in these technologies. They are really valuable.” – Subcontracting lead project U.S. university informant Many host country lead researchers and AquaFish IL local partner institutions hold farmer training sessions or field days at their research facilities. Government field or extension officers are invited to attend these events with the expectation that they will learn and continue to disseminate research results to other farmers in their respective areas. “We used to have a workshop including all government extension officers, research institutions, and ourselves and some farmers too.” – Host country lead university participant “We’ve had several events, farmer field days, and then there’s also a regional conference that we’ve held in Uganda that has brought people from Kenya. Extension officers from Uganda, Congo, Rwanda, and Burundi have been able to attend.” – Subcontracting lead project U.S. university participant In one instance, an NGO in Bangladesh was contracted to conduct outreach and extension. “Our mandate is to develop technology and link up with the extension wing, the government wing, the department of extension. So, whatever we develop, we link with the department of extension, and they’re the ones who have direct linkages with the farmers. So, actually we develop the information that we generate, and it is passed on to this department and through the farmers’ symposia and then we have these agricultural shows. There’s one, which is held every year.” –AquaFish IL researcher participant Some informants, especially in Asia, mentioned brochures or flyers for use during short-term training. “We produce a leaflet as well like an extension fact sheet and then invite all of the government extension agents and university folks as well as many of the stakeholders, and then conduct workshops.” – Subcontracting lead project U.S. university participant An example of one of these brochures is in Appendix H. The brochure was also printed in Nepalese to make it more accessible to the local population. 31 http://aquafishcrsp.oregonstate.edu/news-events/mobile-phones-aquaculture. 32 Egna, H., Ichien, S., and Borberg, J. (eds), “AquaFish Innovation Lab - Eleventh Annual Report,” OSU, Corvallis, Oregon, November 2017. 23 Informants described how on-farm research and using existing farmer facilities for demonstrating new methods are successful ways of experientially disseminating results to end users. “Our farmers are not educated farmers. All are illiterate farmers, no education. So, better, when we go for the demonstration, we take some pond from the farmers and then culture in the pond. Then farmers can see everything” – Host country lead university participant CONCLUSIONS • The AquaFish IL M&E plan, implementation plans, and annual reports provided evidence that M&E benchmarks were established in each region. However, key informants described a lack of clearly defined performance metrics as having impeded a complete evaluation of the program impact. One benchmark (a) was met. • AquaFish IL set a target of generating five peer-reviewed publications per year or a total of 100 publications based on a total of five lead projects participating in the program. Review of AquaFish IL annual reports shows these institutions produced 103 publications directly linked to AquaFish IL research over the course of the life of the program. Due to the natural lag time between completion of investigations and publication approval, peer-reviewed publications attributable to AquaFish IL’s support should come into print for several years after the end of the program. • Participating institutions fostered dissemination of research results to host country stakeholders through a variety of innovative and traditional methods, including: - Creation of demonstration ponds and corollary training curricula (Bangladesh and Nepal); - Development of cell phone-based technologies for providing information to aquaculture farmers, educators, and buyers (Ghana, Kenya, and Uganda); - Farmer training sessions or field days at research facilities and on farms; - Development of brochures and booklets; and - Demonstration ponds and farms. • Very little direct dissemination of research results to key audiences including USAID Missions and other Feed the Future’s aquaculture related projects took place. RECOMMENDATIONS • The ME and partners should encourage use of innovative technologies to disseminate research outputs, technical information, and market data, possibly including: online cloud-based tools that support researchers in collaborating with others during the research process ranging from; services that help host country researchers prepare and submit articles to journals such as support for language, editing, and early peer review of articles to services that identify relevant journals in a given field; and new web-based platforms for mobile phones that use technology to link users to relevant content. • The ME should promote and support inter-institutional research and in-country dissemination of findings, and cross-country research and dissemination to other new countries facilitated through cross-regional meetings and exchanges. • The ME and its partners should develop clear M&E plans that include measures for quality of research outputs, relevance, and effectiveness of dissemination to, and uptake by, stakeholders. • USAID/BFS’ future investments in AquaFish IL projects should keep the end users of the research in mind (practitioners and policymakers) by requiring in the RFP that the ME establish a diverse partnership that includes research institutions, practitioners such as the Feed the Future IPs working in aquaculture, and policymakers to facilitate uptake of research findings. 24 4.4 EVALUATION QUESTION 4 How well has AquaFish IL identified and addressed the academic and technical capacity needs of host country stakeholders? How could the Lab better serve and provide for the capacity needs of these stakeholders? FINDINGS A review of the AquaFish IL M&E plan showed that USAID approved the following benchmarks related to capacity building for years 2-5 (2014-2017): a. Partnerships were strengthened among the U.S. and host country universities, NGOs, NARS, and USAID Missions through associate awards; b. At least 100 degree-seeking (BSc, MSc, Ph.D.) men and women were enrolled through formal long-term training opportunities in U.S., host country, and regional universities (Africa and Asia); c. Equal numbers of women and men were trained through short- and long-term training opportunities; d. Numerous train-the-trainer workshops were convened to provide host countries with highly skilled extension specialists; and e. At least 30 workshops were held during the five-year life of the award and encouraged equal participation from women and men. These benchmarks provided guidance on what AquaFish IL was expected to achieve, although they lacked well-defined measurements. While the ET could not access numbers for all the set benchmarks, a review of the reports indicated that benchmark (a) was achieved. Through interviews with ME and subcontracting lead project U.S. university informants (n=17), the ET found that, historically, AquaFish IL focused strongly on academic and technical capacity building. Also, document review shows that many host country senior faculty and government fisheries agency staff earned their higher degrees with financial and technical assistance from AquaFish CRSP. Current policies are in place to ensure continued focus on host country capacity development, as outlined in the AquaFish IL 2016-2018 Implementation Plan.33 As described in the documentation, 50 percent of funds must be expended in the host country or region. AquaFish IL funding cannot be used to support U.S. expatriate personnel or consultants. Instead, host country institutional networks and capabilities should be promoted. The AquaFish IL mission emphasizes collaborative projects that involve undergraduate students, graduate students, and post-doctoral researchers. No KIIs at any level of the AquaFish IL structure expressed negative views about the impact of the program in meeting academic and technical capacity needs of the host country stakeholders. The ME and subcontracting lead project U.S. university informants described addressing academic and technical capacity in the host countries through: 1) direct mentoring; 2) longer-term undergraduate and postgraduate degree level training (2-4 years); and 3) short-term training. One subcontracting lead U.S. university informant stated that mentoring takes place between U.S.-based personnel, host country lead university, and local partner institutions. The informant described how U.S.-based personnel expose host country personnel to research methodologies, presentation and outreach skills, and publication techniques. A number of subcontracting lead project U.S. and host country university informants stated that the mentoring model has been highly beneficial, especially concerning publications. That was the case for many Asian host country researchers for whom English is a second language learned later in life. Because most African host country personnel learned English in school, none of them cited English writing skills as a challenge in preparing manuscripts. However, multiple Asia-based host country researchers cited English skills as being an obstacle in preparing manuscripts. 33 Egna, H., Borberg, J., and Hyman, A. (eds), “AquaFish Innovation Lab - Implementation Plan 2016-2018,” AquaFish Innovation Lab, OSU, Corvallis, Oregon, December 2016. 25 “This is also a big requirement for the project every year, every ending of the project [end of project year], we have to produce four, five, six peer-reviewed journal applications. We get supported and all of the manuscripts are in English, so we have to proofread that and [have them] proofread by the American or English native speaker.” – Host country lead university participant Informants at host country institutions described site visits by U.S.-based personnel as mutually beneficial, although available financial resources limited opportunities. One subcontracting lead project U.S. university informant described significant cultural and methodological differences in how research personnel in the U.S. and the host countries conduct research, noting that these differences are not evident until visiting the host country research site. This participant also mentioned that site visits provide an opportunity to share ideas, conduct training, and provide hands-on mentoring between staff and students. Informants cited regional and annual meetings as key opportunities to provide mentoring, and as venues for U.S. and host country personnel and students to interact, establish networks, and learn from one another. For longer-term capacity building, the AquaFish IL provides financial and technical support to students from the host countries to do three to five year undergraduate and postgraduate studies both in the U.S. and in their own regions. Interviews with four students illuminated this aspect of AquaFish IL. The AquaFish IL M&E plan benchmark of engaging 100 longer-term degree-seeking students was met (Table 4), as was the goal of 50 percent participation by women. Table 4: Long-Term Trainees Enrolled Disaggregated by Degree and Sex FY BSc MSc Ph.D. Postdoctoral Male (%) Female (%) Total 2013 28 17 5 1 49 51 51 2014 46 45 16 1 56 44 108 2015 73 67 11 2 53 47 153 2016 91 53 13 1 50 50 158 2017 90 70 13 2 49 51 175 Source: “AquaFish Innovation Lab Annual Reports,” 2013-2017 AquaFish IL implementation plans state that this long-term training is designed to assist host countries to become self-sufficient in aquaculture research and to staff government agencies and develop the private sector. “He got his higher degree here at [subcontracting lead project U.S. university] and went back and did some work with fish and used that data for his dissertation and has gone back. He’s got the biggest contract in [host country]. He does a lot of things at [host country institution]. He’s also an important person for us to support because after the AquaFish is gone, he’s going to be there, and he is well positioned to lead aquaculture development.” – Subcontracting lead project U.S. university participant One subcontracting lead project U.S. university informant estimated that the cost of training one student in the U.S. is 10 times that of training a student in East Africa. This figure is based on the respondents’ knowledge of training costs. Documentation shows that the number of students the program funds for training in the U.S. is lower than in the host countries. Informants also described pressure by U.S. institutions for researchers to bring in funds for graduate students. Some students underwent “sandwich” degrees—spending some time in their host country and some at a lead project U.S. university. Informants acknowledged that training some students in the U.S. can bring significant benefits to the host countries when the student returns with knowledge of advanced techniques, better training and classroom knowledge, and improved English. None of the student key informants mentioned problems returning to their host country to work or study. 26 “The embedding gets lost when you’re away from your home institution for so long. We end up having these sandwich degrees, and they (students) slide back and forth. If you go overseas, you know that it’s still like a feather in your cap to have that international Ph.D., but I’m thinking that as these universities that we’ve been supporting get stronger and stronger, what you might start seeing is the feather in your cap is to go to [host country university].” – ME participant AquaFish IL also addressed the academic and technical needs of the host countries through short-term training. To date, 89 training sessions have been conducted during AquaFish IL, reaching 2,079 people (43.7 percent are women) (Figure 2). Ninety-five (95) percent of the opinions regarding the quality and usefulness of the training from key informants (n=29) were positive. Figure 2: Number of Short-Term Trainees by Host Country, Year, and Sex (N=2,079) Source: “AquaFish Annual Reports,” 2013-2017 The AquaFish IL M&E plan benchmarked 30 short-term training sessions during 2014-2017. So far, 89 have been implemented. This indicates that the program has met and surpassed this indicator target. Documentation revealed that training sessions varied in nature; they tended to focus on raising the technical capacity of farmers, extension agents, and other end users of research findings; and took place in practical workshop-style settings. Thirty-six (36) short-term training sessions, held up by research delays, will be completed under the 2016-2018 AquaFish IL Implementation Plan.34 AquaFish IL published a gender integration strategy. Gender integration is a cross-cutting objective with a benchmark of attaining 50 percent inclusion of women in formal and informal education, and a goal of retaining women scientists in research and administrative positions within the AquaFish IL network.35 During the four full years of AquaFish IL implementation (FY 2014-2017), an average of 48 percent of enrollees in long-term degree-seeking training were women and an average of 44 percent of short-term trainees were women. In both instances, AquaFish IL annual reports show an upward trend in women’s engagement in the program. Although there was no clear trend for women being retained in AquaFish IL institutions or other governmental or non-governmental institutions, two women MA students in Asia, both prior AquaFish IL-funded participants, were interviewed. Both have jobs as government-employed aquaculture agents. 34 Egna, H., Borberg, J., & Hyman, A. (eds), “AquaFish Innovation Lab - Implementation Plan 2016-2018,” AquaFish Innovation Lab, OSU, Corvallis, Oregon, December 2016. 35 http://aquafishcrsp.oregonstate.edu/gender-integration. 27 Informants cited instances of AquaFish IL-supported improvements in infrastructural capacity at host country universities and partner institutions. For example, in Nepal, AquaFish IL supported pond construction. The Agriculture and Forestry University of Nepal used indirect costs paid to them by AquaFish IL to construct a building dedicated to aquaculture education and research. “When we get the fund[ing] from AquaFish, that [funding] had a very good overhead cost, that cost we can make a building. So, they approved it from that and that’s a big thing that we have a new building.” – Host country lead university participant CONCLUSIONS • AquaFish IL met M&E benchmarks measuring technical and capacity building needs of the participating host countries. No KIIs, at any level of the organization or outside, expressed views indicating that AquaFish IL did not meet host country needs in this regard. • AquaFish IL has placed considerable emphasis on identifying and addressing the academic and technical capacity needs of host country stakeholders, especially women. The program mission emphasizes collaborative projects and encourages projects that involve undergraduate students, graduate students, and post-doctoral researchers. In practice, AquaFish IL has addressed the academic and technical capacity needs of host country stakeholders through: - Direct mentoring of host country partner institutions by other U.S. and host country participants in areas like research methodologies, presentation/outreach, and publication techniques. Site visits and meetings provide additional mentoring opportunities; - Longer-term student training through financial and technical support to students from the host countries to do undergraduate and postgraduate studies both in the U.S. and in their respective region; - Short-term training that focuses on farmers and other end users in practical workshop-style settings; and - In some cases, AquaFish IL also provided financial support for improvements in institutional strengthening. • “Sandwich degrees” are helpful for reducing costs and have the added benefit of keeping the students engaged with their host country institution. RECOMMENDATIONS • AquaFish IL should expand the opportunities for U.S.-based university researchers to conduct onsite mentorship visits in host country institutions that are mutually beneficial to both groups. The mentorship visits should include longer-term assignments (i.e., 6 – 12 months), in order to: 1) establish and strengthen relationships between U.S. universities and host-country institutions; and 2) gain a clearer picture of the context in which the U.S. and host-country partnerships will operate. • For longer-term capacity development, AquaFish IL should increase financial and technical support for undergraduate and post-graduate degrees in both U.S. and regional universities to ensure sustainability. 4.5 EVALUATION QUESTION 5 How has the ME performed in establishing productive collaborations with the host country universities, 19 Partner Institutions,36 and other Feed the Future ILs and other relevant USAID programs in the target countries? How could the ME improve in building these collaborations? 36 Nineteen (19) partner institutions was the number used in the evaluation protocol approved by USAID but during the evaluation it was found that the actual number is 27 which is corroborated by the ME. 28 FINDINGS As noted in EQ1 findings, the ME implemented AquaFish IL activities through a multi-tiered contracting approach, which started by engaging five U.S. lead universities that in turn subcontracted five host country lead universities. For their part, the five host country lead universities, subcontracted local partner institutions to collaboratively implement the research. According to ME informants, the subcontracting lead project U.S. universities play a significant role in forming and managing collaborations with researchers from the host country lead universities and their local partner institutions. “The ME directly contracts all levels in the program, including to local partner institutions in host countries as well as to U.S. and country lead universities, yet the majority of funds to host countries are distributed through the five host country lead subcontracts to lead project U.S. universities. The host country institutions, in turn, subcontracted 27 local partner institutions.” – ME participant One ME informant mentioned that there were 27 local partner institutions engaged during AquaFish IL. “The lion’s share of AFIL [AquaFish IL] resources went to 19 local partner institutions (out of the 27) you interviewed, whereas the remaining others [n=8) were shorter term or smaller.” – ME participant37 All partners from the subcontracting lead project U.S. universities previously engaged in AquaFish CRSP remained engaged in AquaFish IL. The participants described strong networks within their respective host countries and the regions where they worked. In many cases, these relationships were formed before 2006. Subcontracting lead project U.S. universities for AquaFish IL were selected from those active under AquaFish CRSP (see Section 4.1. Evaluation Question 1 – Findings). The ME personnel interact regularly with personnel from the host country lead university and those in the 27 local partner institutions. During implementation of AquaFish IL, the ME initiated quarterly conference calls with host country university staff. The ME team members reported regular travel to the host countries for site visits, and regular attendance at regional and annual meetings. During these meetings and country visits, ME members had opportunities to strengthen networks and relationships that can be expanded into collaborative work between institutional stakeholders. These collaborations are illustrated by the success story in Text Box 5. Text Box 5: A Collaborative Inter-Institutional Policy Success Story Lifting the ban on snakehead (Channa striata) farming in Cambodia: Snakehead (Channa striata) farming provides a living for thousands of people in the Mekong Delta area. Production of this fish rose from 14,478 MT in 2006 to 77,682 MT in 2016.38 In less developed Cambodia, snakehead farming was dependent mainly on collecting wild fingerlings, which were fed using small, low-value, wild-caught fish— a valuable human food source. As a result, concerns about depletion of snakehead stocks and the loss of human nutrition from feeding them the small, low-value fish, led the Cambodian government to ban snakehead farming in 2004. This ban caused a loss of income for farmers, increased fishing pressure, and created a black market in snakehead fingerlings.39 During AquaFish IL, a collaborative team of researchers from the University of Connecticut at Avery Point, the University of Rhode Island, Can Tho University in Vietnam, and the Inland Fisheries Research and Development Institute (IFReDI) in Cambodia, launched an effort to identify hatchery rearing methods for Cambodian snakehead to relieve pressure on wild capture of fingerlings. They sought to develop formulated feeds that would mitigate the need for catching the small, low-value fish used for feed. In 2014, work was conducted on developing formulated diets for snakehead that contained up to 40 percent 37See footnote above. 38 AquaNews 32/3, Fall 2017. 39 AquaNews 32/1, Spring 2017. 29 soybean meal, a low-cost alternative to fish meal that would not impact growth. In 2015, successful spawning of snakehead in Cambodia yielded fingerlings that were weaned onto formulated feeds for the first time. In 2016, as a direct result of this research, the Cambodian government responded by lifting the ban on snakehead farming. “In Cambodia, it was really a major successful gain for the government to lift the ban because now tens of thousands of households can now legally engage and re-engage in producing snakehead, which is both used for domestic consumption as well as export.” – Subcontracting lead project U.S. university participant This effort benefited from a high degree of collaboration between institutions. Cambodia and Vietnam shared responsibility for tackling the feed issues because, following the ban, fishermen [in Cambodia] continued to illegally harvest and export small-sized fish for snakehead feeds in neighboring countries. “We have a strength in fishery management and they have a strength in aquaculture, so we exchange a lot of technology, research technology, and knowledge between the two universities and we are working very closely. Every year, we have exchange meetings and exchange knowledge and in result they travel to IFReDI…This is a truly trans-boundary project.” – Local partner institution participant “I have been able to work with my colleagues at UConn [University of Connecticut] and in Vietnam and Cambodia purely on the scientific aspects of the project in a way that has had profound impacts on the aquaculture of snakehead fish in both countries. It has been the most impactful thing I have done in my career (in terms of people affected and monetary value).” – Subcontracting lead project U.S. university participant Eighty-nine (89) percent of the host country and local participating institutions’ interviewees (n=29) had positive perceptions towards the collaborative efforts between researchers. They described the aquaculture research community involved in AquaFish IL as “cohesive.” Of particular note, participants described deep awareness of institutional research interests and priorities and enduring relationships built over long periods of time as the keys to developing productive collaborations. “Since I’ve been doing this for 30 years, I do have a very good sense of good collaborators.” – ME participant Inter-institutional collaboration was seen positively, but numerous informants described the ME’s ability to build collaborations with other Feed the Future ILs and USAID programs related to aquaculture in the target countries with mixed results. These informants (n=8) did not provide specific examples but noted that many subcontracting lead project U.S. university and lead host country personnel who attempted to collaborate with these entities has similarly mixed results. One USAID Mission key informant noted having regular contact with in-country AquaFish IL participants, including discussions three to four times a year and attendance at AquaFish IL-sponsored meetings. The same key informant stated that during AquaFish CRSP there were biannual meetings in-country where all the ILs would meet. This informant attributed the lack of collaboration between the relevant ILs and AquaFish IL to the absence of these meetings. “One thing that did work out very well … before they became Innovation Labs when they were still called the Collaborative Research Support Program [was that] we usually used to have biannual meetings where we would have all the different CRSPs, called now the Innovation Labs, coming and trying to tell us what they are doing and where they would bring in some resources and collaboration. This time around it didn’t work out, so I think there was some element of little collaboration between the respective Innovation Labs like it was happening with the CRSPs.” – USAID Mission participant 30 The ME informants reported routinely attending Feed the Future IL meetings in the U.S. and in Feed the Future countries. This resulted in an effort to develop collaborations between the AquaFish IL, the Horticulture IL, the Nutrition IL, and the USAID Mission in Bangladesh. However, this experience was unique—no other KIIs reported similar collaborations. Informants described very little collaboration between any of the eight USAID Missions, Feed the Future IPs working in aquaculture, and AquaFish IL IPs (ME, lead universities in U.S. and host country, and local partner institutions). Informants stated that despite numerous outreach efforts, the ME and subcontracting lead project U.S. universities and host countries personnel perceived a lack of interest on the part of the USAID Missions. Multiple informants stated that USAID Mission staff were busy and unable or unwilling to engage. “I visited the Mission three times to give them updates on our research. They’re interested, but they seem to have their own agenda.” – Subcontracting lead project U.S. university participant “They’re overcommitted, understaffed, underfunded.” – Subcontracting lead project U.S. university participant “I would like to say that, to the extent possible, we ask our U.S. people and our host country people when they are in the capital to make courtesy visits. The last few courtesy visits I asked for were declined because Mission personnel are very busy, and we’re a centrally-funded project, so sometimes they are less interested. It’s usually very much courtesy-driven. When an associate award is involved, then the Mission interaction is, of course, much more involved.” – ME participant A USAID Mission staff who responded with a short email to a request for an interview stated that: “Although the project (AquaFish) has been operating here for years and the Mission has made multiple and regular efforts to connect to all of the Feed the Future Innovation Lab projects, this one is the only one that has never reached out to USAID/[Host Country]. The only time that I have ever had an interaction with the project was at the Center of Excellence conference poster session. Unfortunately, despite multiple attempts to connect with the project, no one from the project ever responded. My feedback for the project would be to not work in isolation where the Mission could have supported the project, communicated successes, and advocated for policy change that would have helped their research out. Unfortunately, this did not happen and was a missed opportunity. We have over 12 different research projects in the country and we interact with all of them regularly and support them.” – USAID Mission participant A lead project U.S. university partner had the following comments regarding work in the same host country: “In [host country], we’ve provided regular updates to them [USAID Mission] either through me going to them or one of our team members informing them. I don’t think we got a lot of feedback or anything from them. It was just a matter of that this was something that was going on within their country. It wasn’t something that the Mission particularly was funding. It was this thing coming from Washington and so basically, you have to go through a process of letting them know what’s going on. That was pretty much the relationship that we had with them.”– Subcontracting lead project U.S. university participant One subcontracting lead project U.S. university informant mentioned having considerable success developing synergies between ILs and the USAID Farmer-to-Farmer program, as well as with the Foreign Agricultural Service (FAS). This informant recommended attempting to develop these kinds of synergies for the AquaFish IL. 31 CONCLUSIONS • The ME has successfully established productive collaborations between host country universities and the 27 local partner institutions. These collaborations were mainly due to the long-standing cohesive nature of the aquaculture research community and yielded significant results—results that could not have been achieved without inter-institutional collaboration. The five subcontracting lead project U.S. universities for the AquaFish IL project were selected from the AquaFish CRSP project. • Despite significant efforts, the ME has been unsuccessful in establishing funded collaborations with other Feed the Future activities in the target countries, which could have been facilitated by Missions if better collaboration were in place. • There is the potential for project synergy between ILs such as the AquaFish IL and other U.S. or foreign assistance programs. FAS and the Farmer-to-Farmer project are two examples. RECOMMENDATIONS • In the future, the ME should expand their research pool of partners instead of relying primarily on previously established relationships among consortium members. They could identify and invite new host country research institutions into the consortium via their publications or conference presentations about aquaculture issues and might benefit from collaboration. • For new procurements, ME proposals should require submissions that demonstrate local research priorities and relevance of research solutions that take into consideration the demand from local research institutions, national policymakers, USAID and other donors, practitioners, and the private sector. • USAID/BFS should consider finding ways to link aquaculture researchers with other relevant Feed the Future ILs and the USAID Missions. This could include regular meetings to exchange ideas and information, or virtual or in-person forums around specific technical topics. 4.6 EVALUATION QUESTION 6 In what ways has the ME been responsive to requests for technical assistance from the 19 Partner Institutions,40 USAID, and the eight USAID Missions and Feed the Future IPs? In what ways could the ME have been more responsive? FINDINGS As mentioned in EQ2, informants, including those interviewed from the partner institutions, unanimously praised the ME for their communication and coordination leadership. Staff at these institutions did not note making requests for technical assistance to the ME. They indicated that activities were carried out within the framework of the AquaFish IL implementation plans and that their input was relayed through the subcontracting lead project U.S. universities during the design phase. The partner institutions mostly described this process as adequate for addressing their institutional needs for technical assistance. In limited cases where mid-project assistance was required, informants from the partner institutions (n=5) said they took their technical assistance requests to the subcontracting lead project U.S. university personnel and described the response as “prompt” and “quick.” They reiterated that their relationship with these institutions was based on extensive historical experience. Host country institution requests for technical assistance varied but generally focused on experimental design, statistical analysis, and manuscript editing. “I think [subcontracting lead project U.S. university partner] is a really, really good person, very good, cooperative person. In Oregon State University… this is a little far from [ME staff] because 40 Nineteen (19) partner institutions was the number used in the evaluation protocol approved by USAID but during the evaluation it was found that the actual number is 27 which is corroborated by the ME. 32 our first link is [subcontracting lead project U.S. university partner].” – Host country lead university participant USAID/BFS staff and ME personnel did not indicate whether, during the AquaFish IL, requests for technical assistance came from USAID, the eight USAID Missions, or other related Feed the Future IPs. One key informant from a USAID Mission stated that there had never been a need to ask for technical assistance because they (USAID) were just overseeing what was happening in AquaFish IL. One ME participant reported attending Feed the Future IL meetings and conferences and said they routinely communicated with the AOR about AquaFish IL operations. ME personnel also said they provided information to USAID staff through the Feed the Future Monitoring System (FTFMS). “There were two informal inquiries for technical assistance and considerable follow-up work done by the Director for the Bangladesh and Cambodian Missions jointly with three other ILs (Sustainable Intensification, Horticulture, and Nutrition). As well, the ME was in communication with two non-HC Missions (Armenia and Liberia) over possible new projects yet the interest from Liberia occurred during the Ebola crisis and the other inquiry was for work outside AFIL’s core mission. Despite no formal requests for technical assistance buy-ins, the Director served as the Technical Advisor for Aquaculture on a joint Feed the Future IL project funded by the Bangladesh Mission” – ME participant CONCLUSIONS • Participants from the partner institutions were comfortable relaying requests for technical assistance through subcontracting U.S. universities. • Host countries had the greatest need for assistance with research design, data analysis and dissemination of their research. • According to KIs representing the ME, the ME attempted to reach out to USAID/BFS, the eight USAID Missions, and relevant Feed the Future ILs, but few members of those stakeholder groups requested technical assistance from the ME during AquaFish IL’s implementation. However, all of the requests made were promptly met. The ME was responsive to requests for technical assistance from the local partner institutions. RECOMMENDATIONS • For future aquaculture projects, the ME should conduct an initial assessment of technical assistance needs on aquaculture topics in each target country and develop a plan on how those needs will be met throughout the life of the project. • Future aquaculture research topics should focus upon primary constraints with respect to markets and supply chains; diet ingredients and additives; genetic selection and classic selective breeding programs; survival rates; economics and regulation; and, most importantly, climate change and resilience. 4.7 EVALUATION QUESTION 7 Describe and highlight which research technologies had the greatest impact or potential for the greatest impact. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (e.g., breeding vs. feeds development vs. post-harvest management)? What topics are promising areas for further investment in aquaculture and fisheries? All host country and U.S.-based respondents engaged in aquaculture research were asked to respond to EQ7 through open-ended identification of topics they see as having the greatest (or potentially greatest) impact. These could be projects that impact the largest number of people, ones that have potential for cost savings or efficiency, or ones that involve innovative technologies. Informants were also asked about technologies or projects that have the most promise and, therefore, might be worth further investment 33 in aquaculture and fisheries. They were not asked to rank their choices in terms of impact level. Table 5 shows that responses varied greatly depending on respondent location, experience, and knowledge. Note that the order of comparative importance reflects the citation frequency of a topic and that column 5 demonstrates how closely AquaFish IL research projects reflected this order of importance. Table 5: Suggested Areas for Future Research on Aquaculture Future Research Topic Host Country Respondents (n=10) U.S.-Based Respondents (n=11) Total Research Projects Addressing Future Topics During the AquaFish Feeds and feeding methods 3 6 9 18 General production, pond systems, and management 3 5 8 16 Mobile phone-based technologies 2 4 6 4 Air breathing and indigenous species 2 4 6 11 Selective breeding and genetics 1 4 5 2 Hyposaline and coastal aquaculture development 2 3 5 4 Aquaponics 2 1 3 1 Post-harvest management 1 1 2 9 Table 5 shows the number of research investigations addressing future research topics as recommended by KIIs. There were 60 research investigations in the AquaFish IL implementation plans. A total of 65 cross-references to future research topics in Table 5 show that some projects addressed more than one subject area. The number of times a topic was recommended reflected the current specializations of the respondent researchers, including: 1) feeds and feeding (nine recommendations and 18 projects); 2) general production systems (nine recommendations and 16 projects); and 3) air-breathing and indigenous fishes (six recommendations and 11 projects). Four areas to consider in future activities related to aquaculture and fisheries include: 1) mobile phone-based technologies; 2) aquaponics; 3) hyposaline and coastal aquaculture; and 4) selective breeding and genetics. All of these areas had more recommendations from researchers than projects implemented. Despite there being nine implemented projects addressing post-harvest management, only two researchers recommended future work in this area. Feeds and Feeding Methods: The topic cited as most important for future research was feeds and feeding. Informants recommended the following sub-topics related to this area as important: • Alternative feed ingredients; • Improving the food value of fish through feed variations; • Feed mill development; • Reducing fish meal in diets; and • Low-cost feed development. “I think the fish nutrition and feed development area is still going to be a very fruitful area of research for a while.” – Subcontracting lead project U.S. university informant AquaFish IL project implementation established several innovative approaches to developing new feeds and feeding methods. The needs of the individual countries and regions provided a basis for each approach. In Africa, the focus was primarily on tilapia farming. Experiments were conducted using local ingredients 34 and on-farm feed production in Kenya.41 Work on producing house fly maggots and earthworms as protein sources for tilapia was conducted in Tanzania and Uganda. In Cambodia and Vietnam, the focus was on reducing fishmeal and establishing vitamin C dietary levels for snakehead.42 43 In Nepal the project conducted experiments on growing periphyton as a means of enhancing yields in carp and small indigenous fish polyculture.44 Projects in Bangladesh concentrated on reducing feed inputs through pulse feeding and nutritional conditioning during larval development. 45 General Production, Pond Systems, and Management: General production, pond systems, and management was the next most-cited area. Informants specified the following topics as important: • Improving grow-out technologies for tilapia and small indigenous fishes (Nepal); • Sustainable snakehead aquaculture development (Cambodia); • Seed production of tilapia, lungfishes, small indigenous fishes, and snakehead; • Vaccines and immuno-stimulation; • The ecosystemic approach to aquaculture and carrying capacity of environments; • Cage culture systems; • Semi-intensive polyculture pond systems; and • The mono-sex culture of Tilapia (Nepal). “Aquaculture is critically important with the dams on the Mekong. With climate change, there’s going to be significant impacts upon the wild capture fisheries in the Mekong river system and aquaculture in Cambodia.” – Subcontracting lead project U.S. university informant “The value of the three-dimensional space versus a two-dimensional surface area for land animals is trying to use the whole space.” – Subcontracting lead project U.S. university informant AquaFish IL stewarded innovative technologies, including the culture of Pangasius catfish in hyposaline waters and a variety of polyculture systems developments in Bangladesh. In Kenya, the project took a multi-tropics approach to aquaculture by developing simple aquaponics systems. Some research investigations initiated during AquaFish IL support longer-term development and only have incremental impacts. However, many had an immediate impact during AquaFish IL. Two that stand out are the lifting of the snakehead ban in Cambodia (Text Box 5) and the rearing of Pangasius in hyposaline waters (Text Box 2). One host country researcher estimated that as many as one million people might benefit from being engaged in hyposaline Pangasius development. Mobile Phone-Based Technologies: Mobile phone-based technologies ranked third in order of importance. Informants specified the following topics related to mobile-phone technology as important: • Applications that can deliver information on water quality, nutrition, and marketing; and • Cell phones as a means of delivering training materials and market information. “We think the mobile phone app is one way to get everybody ready access to technical information, to market, the prices, to availability of input.” – Subcontracting lead project U.S. university informant 41 Ngugi, C.C., E. Oyoo-okoth, J.O. Manyala, K. Fitzsimmons, and A. Kimotho, “Characterization of the Nutritional Quality of Amaranth Leaf Protein Concentrates and Suitability of Fish Meal Replacement in Nile Tilapia Feeds,” Aquaculture Reports 5, 2017. 42 Hien, T.T.T., B.M. Tam, T.L.C. Tu, and D.A. Bengtson, “Weaning Methods Using Formulated Feeds for Snakehead (Channa striata and Channa micropeltes) Larvae,” Aquaculture Research 48(9), 2017. 43 Hien, T.T.T., T.M. Phu, T.L.C. Tu, N.V. Tien, P.M. Duc, and D.A. Bengtson, “Effects of Replacing Fishmeal with Soy Protein Concentrate on Growth, Feed Efficiency and Digestibility in Diets for Snakehead, Channa striata,” Aquaculture Research, Doi: 10.1111/are.13147, 2016. 44 Goetting, K., M. Chow, S. Bodman, L. Carroll, and H. Egna, (eds), “AquaFish Innovation Lab Technical Reports: Investigations 2013–2015,” AquaFish Innovation Lab, Oregon State University. Corvallis, Oregon, 2016. 45 Ibid. 35 Key accomplishments in this area included developing three distinct mobile phone-based applications for conveying real-time market and technical information to interested parties and stakeholders.46 Air Breathing and Indigenous Species: Also ranking third was the topic of air-breathing and indigenous species. For this topic, informants cited the following as important: • Lungfish (Uganda, Kenya, and Tanzania); • Koi – Climbing Perch (Bangladesh); • Sahar (Nepal); • Mola (Bangladesh); and • Punti and dedhuwa (Nepal). “I would say there is just a lot of payoff in the new species I think because they’re already consumed. They’re already a popular, a reasonably popular food…” – Subcontracting lead project U.S. university staff Accomplishments with the lungfish included captive breeding and feeding trials using formulated diets.47 In Bangladesh, project participants conducted research in polyculture systems on koi, an indigenous air breather, and mola, a small indigenous fish.48 Research in Nepal focused on two small indigenous fishes, the punti and dedhuwa, which were added into traditional carp polyculture systems. Nepal also focused on breeding the sahar, an economically important air-breathing fish.49 Selective Breeding and Genetics: Both breeding and genetics ranked fourth in order of importance to informants, who specified a need to concentrate on the following related topics: • Domestication of snakehead (Cambodia); • Development of better growing tilapia strains; • Development of aquacultured fish and invertebrate strains of fish better able to utilize low protein diets; and • More disease-resistant aquaculture fish and invertebrate strains. “We have to find creative ways to help---especially small-scale and medium-scale farms— understand how important the genetic quality, broodstock management, and those related topics are to a successful aquaculture industry.” – Subcontracting lead project U.S. university participant In direct contrast to the RFPs for the AquaFish IL, which listed “Quality Seed Stock” as a priority under “Integrated Production Systems,” the AquaFish IL project conducted little work in breeding and genetics. However, many informants emphasized the need for this kind of work. In Vietnam and Cambodia, the project conducted work on improving breeding stock and genetic diversity of snakehead, and In Uganda, conducted a genetics study on wild collected African lungfish.50 Hyposaline and Coastal Aquaculture Development: Hyposaline and coastal aquaculture development were jointly fourth in order of importance. Key informants specified the following topics in this research area as important: • Further development of Pangasius culture (coastal Bangladesh); • Alternative for shrimp farming where disease has been a problem; 46 Matuha, M., J. Molnar, C.E. Boyd, and J.S. Terhune, “The Role of Mobile Phones in Facilitating Aquaculture Development in Uganda,” World Aquaculture 47(1): 39-46, 2016. 47 Goetting, K., M. Chow, S. Bodman, L. Carroll, and H. Egna (eds), “AquaFish Innovation Lab Technical Reports: Investigations 2013–2015,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2016. 48 Ibid. 49 Jha, D.K., J.D. Bista, N.P. Pandit, M.K. Shrestha, and J.S. Diana, “Successful Breeding of Sahar Tor putitora in Sub-Tropical Nepal,” World Aquaculture 48(2), 2017. 50 Goetting, K., M. Chow, S. Bodman, L. Carroll, and H. Egna (eds), “AquaFish Innovation Lab Technical Reports: Investigations 2013–2015,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2016. 36 • Alternative aquaculture in areas affected by sea level rise and saltwater intrusion; and • Work with bivalves and seaweeds. “Primarily that [coastal areas] is where the world’s population is.” – Subcontracting lead project U.S. university informant One of the projects with the potential to positively impact the highest number of people is that of Pangasius culture in hyposaline waters (Text Box 2). This work was stimulated by the rise in sea levels, and the resulting saltwater intrusion into coastal areas—a problem likely to get worse as climate change increases.51 Aquaponics: Aquaponics ranked fifth in order of importance. Informants specified the following topics as reasons for its high rank: • Ability to produce vegetables and fish at the same time; and • Ideal for urban and semi-urban environments. “We’ve been very pleased with the diet development that we’ve been doing and the aquaponics there in Kenya.” – Subcontracting lead project U.S. university informant Post-Harvest Management: Post-harvest management ranked sixth in order of importance. Informants specified the following topics related to post-harvest management as important: • Value chain analysis; • Waste reduction; and • Higher value markets and value addition. “A little bit of focus in that area [post-harvest management] will really help move the industry forward.” – Subcontracting lead project U.S. university informant During AquaFish IL implementation, value chain analysis was an area of intensive study in both Asia and Africa. A total of five studies were conducted on value chain analysis in Tanzania, Uganda, Nepal, Bangladesh, Vietnam, and Cambodia.52 53 54 Work on value-adding processing with snakehead was done in Vietnam and Cambodia.55 CONCLUSIONS • Host country informant attitudes about impact were influenced by their areas of research and, therefore, they primarily listed technologies related to their research as the most important. AquaFish IL funded researchers identified eight research technologies that would have, or have the potential to have, the greatest impact. These merit new or continuing research investment. • Three areas for future research recommended by most researchers were feeds and feeding methods, general production systems, and air-breathing and indigenous fishes. • Four future cross-cutting research areas including mobile phone-based technologies, aquaponics, hyposaline and coastal aquaculture, and selective breeding and genetics had more recommendations from researchers than they had projects implemented. This indicates that these 51 Church, J.A., P.U. Clark, A. Cazenave, J.M. Gregory, S. Jevrejeva, A. Levermann, M.A. Merrifield, G.A. Milne, R.S. Nerem, P.D. Nunn, A.J. Payne, W.T. Pfeffer, D. Stammer, and A.S. Unnikrishnan, “Sea Level Change” in Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change,” Stocker, T.F., D. Qin, G.K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley (eds), Cambridge University Press, Cambridge, United Kingdom, and New York, NY, 2013. 52 Egna, H., Borberg, J., and Edwards, P. (eds), “AquaFish Innovation Lab - Implementation Plan 2013-2015,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2013. 53 Egna H., Borberg, J., and Hyman, A.A. (eds), “AquaFish Innovation Lab - Implementation Plan 2016-2018,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2016. 54 Navy, H., T.H. Minh, and R.S. Pomeroy, “Impacts of Climate Change on Snakehead Fish Value Chains in the Lower Mekong Basin of Cambodia and Vietnam,” Aquaculture Economics and Management 21(2), 2017. 55 Goetting, K., M. Chow, S. Bodman, L. Carroll, and H. Egna (eds), “AquaFish Innovation Lab Technical Reports: Investigations 2013–2015,” AquaFish Innovation Lab, OSU, Corvallis, Oregon, 2016. 37 are research areas of increasing interest or importance to researchers—irrespective of their area of expertise. RECOMMENDATIONS • USAID/BFS should focus future aquaculture research funding on the technologies identified by AquaFish IL-funded researchers as having the greatest impact or potential for the greatest impact. These are, in descending order from most important to least important: o Feeds and feeding; o General production, pond systems, and management; o Air-breathing and indigenous species; o Mobile phone-based technologies; o Breeding and genetics; o Hyposaline and coastal aquaculture development; o Aquaponics; and o Post-harvest management. LESSONS LEARNED The following Lessons Learned are generalizations based on the ET’s evaluation experience as applied to the specific findings of this project. These lessons highlight both strengths and weaknesses in planning, design, and implementation of the AquaFish IL project that may have affected the performance, outcome, and/or impact of a project: • An active ME with well-established partnerships, reliable leadership, and good communication has greatly enhanced the value of AquaFish and is vital to the success of such projects. • The AquaFish IL research project was mainly designed and implemented by researchers (U.S. and host country universities) with limited involvement of practitioners, policymakers, and beneficiaries to set priorities. As a result, the research products were mainly targeted towards producing peer-reviewed publications and less on their use for programmatic and policy decisions. • To help ensure that results are useful and sustainable, award criteria for any similar Feed the Future ILs should, in part, include demonstrated integration with USAID Mission programming and existing Feed the Future programs in the proposed host countries or regions, and potential for ongoing Mission engagement. Otherwise, Mission staff will not be engaged in the research process given their time constraints. • Scoping visits to establish research needs in countries selected for IL programming should be undertaken in order to identify Mission priorities and host country needs. • Consistent oversight of centrally funded projects that do not have an in-country presence is necessary in order to establish effective linkages between those projects, USAID Missions, and other Feed the Future projects. • If centrally funded research projects such as AquaFish ILs are implemented in countries without related Feed the Future projects or planned projects, linkages with Missions and IPs are obviated. Therefore, if a future project is seeking to establish such linkages, the selection of countries should be based upon whether or not there are active or considered projects therein. • Associate awards are intended to allow Missions or Bureaus within USAID to support work related to the Leader program. However, research projects such as AquaFish may come to depend upon these awards for scaling up results and technologies rather than formulating plans for greater dissemination and sustainability. • Expected results need to be clearly linked by causality and the linkage needs to be well understood by all partners in IL projects—in particular, knowing how the research can ultimately benefit stakeholders in developing countries. • Multi-country innovative research projects could benefit from more frequent and cross-country assessments than those projects implemented in a single country to promote cross-fertilization 38 learning. Field visits should be documented to assure a comprehensive “train of evidence” on the impact of the project over the years beyond reports and monitoring data. 39 ANNEXES 40 ANNEX A: EXPRESSION OF INTEREST 41 EXTERNAL EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR COLLABORATIVE RESEARCH ON AQUACULTURE & FISHERIES: EXPRESSION OF INTEREST I. BACKGROUND INFORMATION A) Identifying Information 1. Project/Activity Title: Feed the Future Innovation Lab for Collaborative Research on Aquaculture & Fisheries (AquaFish Innovation Lab) 2. Award Number: CA/LWA no. EPP-A-00-06-00012-00 3. Award Dates: 3/31/2013 to 3/29/2018 (based on original award of 9/30/2006 as modified on 3/29/2013) 4. Project/Activity Funding: $19,700,000.00 (Total $34,420,000.00) 5. Implementing Organization: Oregon State University 6. Project/Activity AOR: Shivaun Leonard B) Development Context 1. Opportunity Addressed by the Project/Activity Being Evaluated General Background The overarching goal of the Feed the Future (FTF) Initiative is to sustainably reduce global poverty and hunger. Limited land availability and variable climatic conditions exasperate the problem and increase the need for sustainable yet swift solutions. In order to address the challenge, scientific and technological innovations need to be developed that increase agricultural productivity with minimal environmental impact. In support of the broader FTF goal, the FTF Global Food Security Research portfolio aims to advance the productivity frontier, to transform key production systems, and to enhance nutrition and food safety through agriculture. The Feed the Future Innovation Labs (formerly called CRSPs) were created under Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage U.S. land grant and other eligible universities to address the needs of developing nations while also contributing to U.S. food security and agricultural development. In 2000, Title XII was reauthorized, enabling these U.S. university research efforts to continue “to achieve the mutual goals among nations of ensuring food security, human health, agricultural growth, trade expansion, and the wise and sustainable use of natural resources.” The FTF Innovation Labs draw on the expertise of top U.S. universities and developing country institutions and are an integral part of the Feed the Future Food Security Innovation Center. The Innovation Labs serve to implement the Feed the Future Global Hunger and Food Security Research Strategy and were established in response to two key recommendations from a 2012 Board for International Food and Agricultural Development (BIFAD) commissioned CRSP review56: • To develop an overarching and coordinated strategy for engaging U.S. universities in agriculture and food security research and human and institutional capacity development that includes the CRSPs as a central component; and • To leverage the impact of CRSP investments by strengthening links across universities, U.S. government, global programs, foundations, and other donors. 56 http://transition.usaid.gov/our_work/agriculture/bifad/BIFADREVIEW_CRSP_August2012.pdf 42 The launch of the Food Security Innovation Center in 2012 enabled USAID to manage its research, policy, and capacity-strengthening portfolio through the following seven thematic areas rather than by institutional home: • Program for Research on Climate Resilient Cereals • Program for Research on Legume Productivity • Program for Advanced Approaches to Combat Pests and Diseases • Program for Research on Nutritious and Safe Foods • Program for Markets and Policy Research and Support • Program for Sustainable Intensification • Program for Human and Institutional Capacity Development The Program for Research on Nutritious and Safe Foods, which includes AquaFish and other Innovation Labs, aims to improve the production and safe processing of nutritious agricultural products and to increase our understanding of the role of these products in improving the quality of household diets. The program focuses on improving nutrition in the first 1000 days of life—a critical period in a child’s cognitive and physical growth and development. Description of the AquaFish Innovation Lab The Feed the Future Innovation Lab for Collaborative Research on Aquaculture & Fisheries (AquaFish Innovation Lab, or simply AquaFish) began in 2006, when it was known as AquaFish CRSP. One success of the many years of AquaFish research activities is the development of an expansive collaborative network of international researchers and stakeholders dedicated to the sustainable development of aquaculture and fisheries innovations. Objective The overall AquaFish mission is to enrich livelihoods and promote health by cultivating international multidisciplinary partnerships that advance science, research, education, and outreach in aquaculture and fisheries (with an emphasis on aquaculture per USAID’s RFA requirement for AquaFish to focus core resources on aquaculture). AquaFish is managed to achieve maximum program impacts, particularly for small-scale farmers and fishers, in Host Countries and more broadly. Program objectives address the need for world-class research, capacity building, and information dissemination. Specifically, AquaFish strives to: • Develop sustainable end user level aquaculture systems to increase productivity, enhance international trade opportunities, and contribute to responsible aquatic resource management; • Enhance local capacity in aquaculture and aquatic resource management to ensure long-term program impacts at community and national levels; • Foster wide dissemination of research results and technologies to local stakeholders at all levels, including end users, researchers, and government officials; and • Increase Host Country capacity and productivity to contribute to national food security, income generation, and market access, and to lead research through scientific inquiry and problem solving. Research & Administration AquaFish projects are integrated across four interrelated global themes: a. Improved Human Health and Nutrition, Food Quality, and Food Safety b. Income Generation for Small-Scale Fish Farmers and Fishers c. Environmental Management for Sustainable Aquatic Resources Use d. Enhanced Trade Opportunities for Global Fishery Markets Each project focuses on one primary AquaFish theme yet incorporates all four themes to achieve a systems approach. The global themes of AquaFish are cross-cutting and address gender, capacity building, institutional strengthening, climate change, sustainability, and outreach, as well as USAID policies. 43 All projects are further organized around 10 areas of inquiry called topic areas, developed to characterize the research. The 10 AquaFish topic areas (listed below) integrate across two categories: Integrated Production Systems and People, Livelihoods, and Ecosystem Interrelationships. Integrated Production Systems • Production System Design & Best Management Alternatives • Sustainable Feed Technology and Nutrient Input Systems • Climate Change Adaptation: Indigenous Species Development • Quality Seed Stock Development People, Livelihoods, and Ecosystem Interrelationships • Human Nutrition and Human Health Impacts of Aquaculture • Food Safety, Post-Harvest, and Value-Added Product Development • Policy Development • Marketing, Economic Risk Assessment, and Trade • Watershed & Integrated Coastal Zone Management • Mitigating Negative Environmental Impacts AquaFish is administered through its Management Entity (ME) at Oregon State University under the direction of its director and leadership team. In addition, AquaFish maintains open dialogue with its advisory committees to stay informed of emerging developments and technical issues. 2. Target Areas and Groups Key U.S. Partners AquaFish partners with five U.S. universities to lead projects in Africa and Asia: Auburn University; North Carolina State University; Purdue University; University of Connecticut Avery Point; and The University of Michigan. Geographic Reach AquaFish overseas activities since 2013 occurred in Bangladesh, Cambodia, Ghana, Kenya, Nepal, Philippines, Tanzania, Uganda, and Vietnam. C) Documents The evaluation team will review a wide variety of documents provided by BFS, the ME, and by the sub￾award partners. The primary documents and types of documents that will be provided are listed below. • Technical Application proposal • AquaFish Annual Reports • Implementation Plans and Annual Work Plans • Program publications including success stories, research publications, and Human and Institutional Capacity Development (HICD) outputs • Other AquaFish website materials and information (aquafish.oregonstate.edu) • External review report for Phase I of the program (AquaFish CRSP, 2006-2012) • Feed the Future Research Strategy (feedthefuture.gov) II. EVALUATION RATIONALE A) Evaluation Purpose The purpose of this external performance evaluation of the AquaFish IL is to provide robust evidenced￾based findings, conclusions, and recommendations that BFS can use to inform the development of new investments in aquaculture and fisheries research for development, improve ongoing activity effectiveness, and better achieve intended outcomes. To accomplish this, the evaluation will assess implementation and 44 progress toward objectives amongst the different AquaFish components, including research and dissemination, technology development, and capacity building accomplishments. Furthermore, since the AquaFish Innovation Lab will be completing its second five-year phase in March 2018, the External Evaluation Team (EET) is expected to make recommendations to USAID to help improve effectiveness and efficiency in implementation and management and discuss potential research changes during consideration of an RFA for a new Innovation Lab. B) Audience and Intended Uses These results will be used by USAID/BFS Program for Research on Nutritious and Safe Foods to establish a future RFA, or RFAs, to assess needs and address outstanding research questions connected to aquaculture and fishery productivity for smallholders in developing countries and associated value chains. The evaluation results may also be applicable to others who are involved in designing research for development programs. C) Evaluation Questions Scope of Work This performance evaluation will provide USAID and the Management Entity (ME) with constructive feedback on the program management, research program, training program, and institutional capacity collaboration of the AquaFish Innovation Labs. Specifically, the evaluation team will objectively evaluate the following questions using an evidenced-based and data-driven approach. Program Management 1. To what extent has the Management Entity (ME) effectively implemented and managed the AquaFish Innovation Lab’s respective research and training activities in Africa and Asia? How effectively has the ME managed and coordinated amongst the partners, Missions, host country government partners, local researchers, and other stakeholders? In what ways could implementation and management be improved? 2. To what extent has effective communication and coordination supported project implementation and outcomes regarding research and dissemination and capacity development? In what ways could communication and coordination be improved between USAID/BFS, USAID Missions, the ME, partners, and other stakeholders, as a way to better achieve project goals and objectives, create and carry out a shared research agenda, meet reporting requirements, and expand collaboration? 3. How responsive has the ME been to requests for technical assistance from USAID (headquarters and Missions), host country institutions, and other technical assistance groups? Research and Training Program 1. To what extent did AquaFish generate quality research technologies/outputs using appropriate metrics (e.g., peer reviewed publications)? In what ways has AquaFish fostered dissemination of research results to local stakeholders at all levels? Are the outputs relevant for a demand-driven research-for-development project (e.g., did they generate new feed formulations of relevance to FTF countries)? What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for FTF countries) moving forward? Describe and highlight which research technologies have had the greatest impact or potential for the greatest impact. 45 2. How well has the program identified and addressed academic and technical capacity needs of host country stakeholders? How could the project better serve and provide for the capacity needs of these stakeholders? Program Future 1. In what ways could the AquaFish Innovation Labs’ management, research (i.e., design, implementation, stakeholder involvement, dissemination), capacity development (i.e., recruitment and selection, content, location) programs, and/or institutional capacity collaboration be improved to better achieve objectives? 2. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment? What topics are areas are promising for further investment in aquaculture and fisheries? III. TIMEFRAME & TRAVEL A) Timeframe The evaluation will begin on August 10, 2017 and conclude December 2017. B) Travel This will be a desk study with potentially a survey and key informant interviews. IV. DELIVERABLES & DESIGN A) Deliverables • Draft Evaluation Plan September 1, 2017 (a draft is available) • Final Evaluation Plan September 15, 2017 • Evaluation work complete by November 20, 2017 • Preliminary findings presentation November 30, 2017 • Draft Evaluation Report December 15, 2017 • Final Evaluation Report December 31, 2017 V. TEAM COMPOSITION The technical qualifications of evaluation team should include primary technical areas such as aquaculture, along with specific experience carrying out rigorous performance evaluations. The pool of potential team members with international aquaculture development experience and relevant expertise outlined is small. Care must be taken to avoid conflicts of interest, drawing from both U.S. and non-U.S. candidates. Each member is requested to submit a CV that demonstrates relevant experience in technical, evaluation, and management skills. Team Evaluation Planning Lead x 1: a senior-level evaluator with a minimum of 15 years’ experience in evaluation methodology (including mixed methods evaluation), quantitative and qualitative data collection and analysis, and experience carrying out evaluations of international development projects, preferably with a background in aquaculture research for development and technology dissemination. The preferred candidate will be familiar with USAID (or other donor) funded programs. The candidate will also have: a) the capacity to conduct independent program evaluation; b) a thorough understanding of research methodology; c) the ability to analyze issues and formulate concrete recommendations orally and in writing; d) be available to travel and meet the timelines for completion of the evaluation; and e) not have any conflicts of interest. 46 Technical member x 1: One mid- or one senior-level technical expert with a minimum of 10 years of experience managing and/or evaluating multifaceted international development research and/or university￾based programs. The preferred candidates will be familiar with USAID (or other donor) funded programs. Backgrounds in aquaculture research in the development context, with technical expertise in a field relevant to low- to semi-intensive aquaculture systems in Sub-Saharan Africa and/or Asia is required. The candidates will also have: a) a demonstrated capacity to conduct independent research program evaluation; b) an understanding of USAID’s foreign assistance goals, and its particular objectives related to collaborative research, aquaculture development, and food security; c) the ability to analyze issues and formulate concrete recommendations orally and in writing; and d) not have any conflicts of interest. Specific technical expertise needed by the team as a whole: Team members must have the expertise necessary to evaluate the many sub-award projects, to assess the different technical and programmatic components of the project, including research, management and implementation, capacity development, and partnerships and institutional collaboration, address the evaluation questions rigorously, and collect, triangulate, and analyze primary and secondary data in a robust manner to generated unbiased and evidenced-based findings. Disciplines of all members: The following technical skills should be present among the team as a whole: organizational development, quantitative and qualitative evaluation techniques, aquaculture production systems, aquatic ecology, fish physiology, limnology, economics, aquaculture sociology, human nutrition/food safety, aquatic resources management, fish nutrition, and fish reproduction. VI. SUGGESTED LOE Task/Deliverable Evaluation Team Leader Technical Team Member Conference Call/Desk Review 2 2 Evaluation Plan & Revisions 4 2 Data Collection & Travel 18 15 Presentation of Findings 2 2 Draft Report 5 5 Revisions of Final Report 4 2 Total 35 28 47 ANNEX B: EVALUATION TEAM 48 Team Leader: David Rinck has more than 20 years of experience and specialization in project evaluation, food security, agriculture and agro-enterprise development, market systems, and economic policy analysis. He has successfully designed, implemented, and managed evaluations of agricultural development projects as both Team Leader and as a specialized team member. In his current role as the Managing Director of Absolute Options LLC (AO), Mr. Rinck has provided technical direction to over a dozen evaluations and agricultural value chain/market systems assessments, including the: performance evaluation (PE) of the USAID Solutions for African Food Enterprises (SAFE) Program in Kenya, Malawi, and Zambia; Mid-Term Evaluation of the Inter-American Development Bank Creating Alliances in Cacao for Improved Access and Organization (CACAO) Project in Haiti; Mid-Term Evaluation of Phase 2 of the United States Department of Agriculture (USDA) Cashew Value Chain Enhancement Project in Senegal and Gambia; Team Leader of the Final Evaluation of the USDA FEEDing Pakistan Project (Soy foods supply development and training for fish farming); and others. Before establishing AO, Mr. Rinck served as the Chief of Party on the U.S. Department of State Results Oriented Commercial-Organization Capacity Development Project in Tunisia. In this role, he managed a team of local and international staff to improve policy advocacy capacity and promote agricultural trade and investment. Previously, Mr. Rinck also spent five years at the USAID East Africa Regional Mission in Nairobi, Kenya. In this role, he developed and managed new initiatives, including the Alliance for Commodity Trade in Eastern and Southern Africa, a specialized agency for agricultural market systems development under the Common Market for Eastern and Southern Africa regional trade zone, and the Market Linkages Initiative to improve systems for staple crops in Central and Southern Africa. Between 2001 and 2006, he served as Regional Technical Adviser for Agro-Enterprise Development in Eastern and Southern Africa for CRS, where he represented the agency in an Agro-Enterprise Learning Alliance with the Centro International de Agricultura Tropical to develop smallholder market systems. He was also the Market Systems Recovery Expert at the Business Development (BDS) Seminars implemented by the International Labour Organization (ILO) between 2003 and 2008. His publications include “Implementing Sustainable Private Sector Development: Relief to Development in Crisis-Affected Economies,” ILO BDS Reader, September 2006 and “Market Development in Crisis-Affected Environments: Emerging Lessons for Achieving Pro-Poor Economic Reconstruction,” Small Enterprise Education and Promotion Network, 2007. Mr. Rinck holds a Master’s degree in Social Sciences from the University of Chicago, and a Bachelor’s degree in Agricultural Economics from the University of California at Davis. He has also studied economics, languages, and sociology at many locations worldwide, including Beijing University, as well as in the Middle East at Damascus, Beirut, and Sana’a. He speaks English, French, Portuguese, and Spanish. Technical Expert: Simon Ellis has 30 years of experience working in the fields of aquaculture, sustainable development, and conservation. He concurrently holds three long-term positions as Director of the Marine and Environmental Research Institute of Pohnpei (MERIP), an NGO focusing on sustainable aquaculture development; affiliate faculty at the University of Hawaii Hilo – Pacific Aquaculture and Coastal Resources Center; and Micronesia representative for Seacology, a Berkeley California-based conservation NGO. Mr. Ellis has extensive experience in freshwater and marine aquaculture that covers pond and cage culture of fishes, bivalves, and other shellfish farming. Presently, Mr. Ellis is project manager on two USAID climate change-related projects focusing on aquaculture through the Pacific American Climate Fund. These projects focus on cage farming of fish, pearl farming, and culture of invertebrates for the marine ornamental trade. In addition to these long-term projects, Mr. Ellis has completed a number of short-term consultancy projects for the European Union, Secretariat of the Pacific Community, and other regional organizations. Mr. Ellis has lived in Micronesia for 20 years and has extensive experience in the Pacific region as well as Africa. Prior to living in Micronesia, he lived in the Bahamas for five years. Mr. Ellis holds a Master’s degree from Louisiana State University in Fisheries and Aquaculture and a Bachelor’s degree in Marine Biology from Heriot-Watt University in Edinburgh, Scotland. He spent his formative years in Nigeria and Ghana. 49 ANNEX C: EVALUATION SCHEDULE 50 Activity Date/a Phase 1: Planning and Design ET revises existing Draft Evaluation Protocol October 16-30, 2017 Submit revised Draft Evaluation Protocol (including Data Collection Tools) to USAID November 1, 2017 Receive comments on Draft Evaluation Protocol from USAID November 5, 2017 Integrate USAID review and submit to PEEL November 6, 2017 Submit final Evaluation Protocol to USAID (including Data Collection Tools) November 7, 2017 USAID approves Evaluation Protocol (including Data Collection Tools) November 9, 2017 Phase 2: Data Collection Phase Document review November 9-10, 2017 Remote key informant interviews November 13-December 22, 2017 Phase 3: Analysis and Reporting Data analysis December 22, 2017-January 21, 2018 Submit first draft Evaluation Report to PEEL (50-pages maximum excluding annexes) February 18, 2018 Submit first draft Evaluation Report to USAID (50-pages maximum excluding annexes) February 23, 2018 Receive comments from all reviewers March 23, 2018 Submit second draft Evaluation Report to PEEL (50-pages maximum excluding annexes) April 19, 2018 Submit second draft Evaluation Report to USAID (50-pages maximum excluding annexes) April 30, 2018 Receive comments from all reviewers July 2, 2018 Submit final Evaluation Report to PEEL (50-pages maximum excluding annexes) July 11, 2018 USAID approves final Evaluation Report (50-pages maximum excluding annexes) July 18, 2018 Phase 4: Dissemination 508 compliance of final report July 25, 2018 Upload final report to Development Experience Clearinghouse (DEC) July 26, 2018 Share data with USAID for Development Data Library (DDL) N/A a. Final dates dependent on approvals. 51 ANNEX D: LIST OF DOCUMENTS REVIEWED AND REFERENCES 52 AquaFish Innovation Lab “Request for Proposals 2013-2015: Global Research, Capacity Building, and Institutional Development in Aquaculture and Aquatic Resources Management” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon. 2013. AquaFish Innovation Lab “Appendix: Monitoring and Evaluation Plan” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon. 2013. AquaFish Innovation Lab “Request for Continuation Proposals 2013-2015: Global Research, Capacity Building, and Institutional Development in Aquaculture and Aquatic Resources Management” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon. 2015. AquaFish Innovation Lab “AquaFish Monitoring and Evaluation Plan, DRAFT Report 2013-2018” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon. 2018. AquaNews 32/1, Spring 2017, Fall 2017, Summer 2017. Burns, Payne, Agrilink Blog 2016. Church, J.A., P.U. Clark, A. Cazenave, J.M. Gregory, S. Jevrejeva, A. Levermann, M.A. Merrifield, G.A. Milne, R.S. Nerem, P.D. Nunn, A.J. Payne, W.T. Pfeffer, D. Stammer, and A.S. Unnikrishnan, “Sea Level Change” in Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, Stocker, T.F., D. Qin, G.K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley (eds), Cambridge University Press, Cambridge, United Kingdom and New York, NY, 2013. Egna, H., Borberg, J., and Edwards, P. (eds), “AquaFish Innovation Lab - Implementation Plan 2013-2015,” Oregon State University, January 2014. Egna, H., Borberg, J., and Hyman, A. (eds), “AquaFish Innovation Lab - Implementation Plan 2016-2018,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, December 2016. Egna, H., Ichien, S., and Borberg, J. (eds), “AquaFish Innovation Lab - Seventh Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, October 2013. Goetting, K., M. Chow, S. Bodman, L. Carroll, and H. Egna, (eds), “AquaFish Innovation Lab Technical Reports: Investigations 2013–2015,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2016. Egna, H., K. Goetting, C. Price, J. Borberg, P. Edwards, S. Ichien, and F. Evans, “AquaFish Innovation Lab - Eighth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, October 2014. Egna, H., K. Goetting, J. Borberg, S. Ichien, C. Price, M. Chow, and F. Evans, “AquaFish Innovation Lab - Ninth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, September 2015. Egna, H., K. Goetting, J. Borberg, S. Ichien, B. Goodwin, S. Bodman, F. Evans, and L. Carroll, “AquaFish Innovation Lab - Tenth Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2016, January 2017. Egna, H., K. Goetting, J. Borberg, S. Ichien, A.A. Hyman, L. Carroll, and F. Evans, “AquaFish Innovation Lab - Eleventh Annual Report,” AquaFish Innovation Lab, Oregon State University, Corvallis, Oregon, 2017. Hien, T.T.T., B.M. Tam, T.L.C. Tu, and D.A. Bengtson, “Weaning Methods Using Formulated Feeds for Snakehead (Channa striata and Channa micropeltes) Larvae,” Aquaculture Research 48(9), 2017. Hien, T.T.T., T.M. Phu, T.L.C. Tu, N.V. Tien, P.M. Duc, and D.A. Bengtson, “Effects of Replacing Fishmeal with Soy Protein Concentrate on Growth, Feed Efficiency and Digestibility in Diets for Snakehead, Channa striata,” Aquaculture Research, Doi: 10.1111/are.13147, 2016. Jha, D.K., J.D. Bista, N.P. Pandit, M.K. Shrestha, and J.S. Diana, “Successful Breeding of Sahar Tor putitora in 53 Sub-Tropical Nepal,” World Aquaculture 48(2), 2017. Matuha, M., J. Molnar, C.E. Boyd, and J.S. Terhune, “The Role of Mobile Phones in Facilitating Aquaculture Development in Uganda,” World Aquaculture 47(1): 39-46, 2016. Navy, H., T.H. Minh, and R.S. Pomeroy, “Impacts of Climate Change on Snakehead Fish Value Chains in the Lower Mekong Basin of Cambodia and Vietnam,” Aquaculture Economics and Management 21(2), 2017. Ngugi, C.C., E. Oyoo-okoth, J.O. Manyala, K. Fitzsimmons, and A. Kimotho, “Characterization of the Nutritional Quality of Amaranth Leaf Protein Concentrates and Suitability of Fish Meal Replacement in Nile Tilapia Feeds,” Aquaculture Reports 5, 2017. “Review of Science, Technology, Innovation, and Partnership (STIP) For Development and Implications for the Future of USAID,” National Academies of Sciences, Engineering, and Medicine, National Academies Press, Washington, DC, 2017. Shrestha, M.K. and J. Pant (eds), “Small-scale Aquaculture for Rural Livelihoods: Proceedings of the National Symposium on Small-scale Aquaculture for Increasing Resilience of Rural Livelihoods in Nepal,” Institute of Agriculture and Animal Science, Tribhuvan University, Rampur, Chitwan, Nepal, and The WorldFish Center, Penang, Malaysia, 2012. Tidwell, James H. “Aquaculture Production Systems, 1st Edition,” World Aquaculture Society, June 26, 2012. United States Agency for International Development (USAID) “Request for Applications Number M-OAA￾EGAT-06-1395 Aquaculture & Fisheries Collaborative Research Support Program (CRSP)” USAID. 2006 United States Agency for International Development (USAID) “2012-PHASE 2-EXTENSION-RFA” USAID. 2012 54 ANNEX E: KEY INFORMANT INTERVIEWS 55 Date Title Institution 13-Nov-17 Assistant Director Oregon State University 13-Nov-17 Associate Director for Research Oregon State University 13-Nov-17 Contracts and Compliance Officer Oregon State University 27-Nov-17 Associate Professor Purdue University 28-Nov-17 Professor Auburn University 28-Nov-17 Professor, Director Michigan Sea Grant University of Michigan 28-Nov-17 Professor University of Connecticut, Avery Point 29-Nov-17 Professor University of Arizona 29-Nov-17 Associate Professor University of Hawaii, Hilo 1-Dec-17 Professor North Carolina State University 1-Dec-17 Professor Bangladesh Agricultural University, Bangladesh 1-Dec-17 Professor Agriculture and Forestry University, Nepal 2-Dec-17 Professor and Program Coordinator Agriculture and Forestry University, Nepal 3-Dec-17 Senior Lecturer University of Dar es Salaam, Institute of Marine Science, Tanzania 3-Dec-17 Professor Patuakhali Science and Technology University, Bangladesh 4-Dec-17 Professor, Agriculture Resource Management Kenyatta University, Kenya 4-Dec-17 Professor, Director University of Arkansas, Pine Bluff 4-Dec-17 Associate Professor, Agriculture Economics and Agribusiness Makerere University, Uganda 5-Dec-17 College of Agricultural Sciences Unit Leader, Associate Professor, Director AquaFish IL Oregon State University 6-Dec-17 Associate Professor, Fisheries University of Eldoret, Kenya 7-Dec-17 Associate Professor, Vice Rector Can Tho University, Vietnam 7-Dec-17 Associate Professor, Coastal Aquaculture Can Tho University, Vietnam 7-Dec-17 Professor Animal Science Sokoine University of Agriculture, Tanzania 8-Dec-17 Provost of College of Agriculture and Natural Resources Kwame Nkrumah University, Ghana 8-Dec-17 Associate Professor Fish and Wildlife Conservation Virginia Tech 8-Dec-17 Research Assistant (graduated student) Agriculture and Forestry University, Nepal 10-Dec-17 Assistant Director, Programs Shushilan NGO, Bangladesh 10-Dec-17 Senior Lecturer, Aquaculture and Fisheries Sokoine University of Agriculture, Tanzania 10-Dec-17 Senior Research Officer National Fisheries Resources Research Institute, Uganda 11-Dec-17 Agreement Officer’s Representative USAID/BFS 11-Dec-17 Professor Emeritus University of Rhode Island 56 Date Title Institution 11-Dec-17 Director Inland Fisheries Research and Development Institute, Cambodia 12-Dec-17 Professor, Research Economics Alabama A&M University 13-Dec-17 Ph.D. Student Agriculture and Forestry University, Nepal 14-Dec-17 Associate Director, Administration Officer Oregon State University 14-Dec-17 Graduated Student (now Fisheries Extension Officer) Can Tho University, Vietnam 14-Dec-17 Graduated Student (now Aquaculture Health Agent) Can Tho University, Vietnam 14-Dec-17 Student Bangladesh Agricultural University, Bangladesh 14-Dec-17 Fisheries Consultant Fisheries Consultant, Kenya 18-Dec-17 Student Sokoine University of Agriculture, Tanzania 18-Dec-17 Student Sokoine University of Agriculture, Tanzania 18-Dec-17 Former Student National Fisheries Resources Research Institute 12-Feb-18 Program Management Specialist/Agronomist USAID Mission Uganda 57 ANNEX F: REQUEST FOR STUDENT PARTICIPATION 58 External Evaluation of the USAID’s AquaFish Program Request for Student Participation My name is Simon Ellis and I am an external evaluator for a USAID funded AquaFish Innovation Lab program, that among other activities has supported students globally to undertake studies related to Aquaculture. I received your contact information from your [INSERT ADVISER NAME INSTITUTION] to ask about your interest in participating in an evaluation to understand the experiences of students like you who are supported by the USAID-funded AquaFish program. The AquaFish Innovation Lab program is currently undergoing an external evaluation, sponsored by USAID, to better understand its effectiveness and efficiency, and areas for improvement. The evaluation is being conducted to learn more about how students experience and perceive the selection process and ways that they can be supported to be successful in the program. Ultimately this information may help to improve AquaFish processes and support and inform how students like you can be better supported from the recruitment to completion of their studies. If you are interested, we would like to schedule a Skype or phone interview within the next two weeks to ask you some questions. Your adviser has said it is okay, but it is up to you to decide if you want to participate. The interview will take approximately 30 minutes. Your participation is confidential and no information about you or your school will be identified in any published report. Responses and results are private, and your adviser will not have access to the answers you give. Are you interested in participating in the interview? 1. Are you willing to participate in such an interview via Skype or phone? If yes, kindly provide a time you would be available by responding via email to with a yes, date/time, and your preferred method (Skype or phone). We will also send you a follow up text to confirm that the date/time you’ve selected are still good. 2. If no, please respond with a message “no” via email. We sincerely hope that you will agree to participate in this important interview and your feedback will be greatly benefit and improve the AquaFish and other similar projects supporting students like yourself. Thank you. Sincerely, Simon Ellis and David Rinck (contacts) 59 ANNEX G: DATA COLLECTION TOOLS 60 1. KII Guide – ME (OSU), U.S. Lead Universities, Host Country University, and Local Partner Staff Interview date: Interviewer: Name(s): Institution: Title(s): Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called ME&A. Our team is conducting an evaluation the AquaFish Innovation Lab, which was funded by USAID. We would like to conduct a brief interview today to learn about your perspectives on project performance. This information will be used in a report for USAID that will be publicly available. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. The objective of this research is to improve the performance of projects like AquaFish. The information may be used by other organizations as well. There is no direct benefit to USAID for your participation in this study. The purpose is only to help us improve the services of projects like this one. If you chose, your responses can remain anonymous. The interview is expected to take about two hours. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation, and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate. ◻ No, I will not participate. 61 1. To what extent has the ME effectively implemented and managed the AquaFish Innovation Lab’s activities in the five target regions of Africa and Asia? How effectively has the ME managed and coordinated among the five Lead U.S. Universities and Host Country Universities, 19 Partner Institutions, USAID and the eight USAID Missions and their Feed the Future implementing partners, and the long-term degree-seeking students, host country trainees that have attended short-term training events, and local researchers? In what ways could implementation and management be improved? 1.1. What implementation and management processes were used related to coordination with the five Lead U.S. Universities, Host Country Universities, and Partner Institutions? 1.2. What have been the primary implementation and management challenges related to coordination with the five Lead U.S. Universities, Host Country Universities, and Partner Institutions (rank these challenges)? 1.3. In what ways could implementation and management processes related to coordination with the five Lead U.S. Universities, Host Country Universities, and Partner Institutions be improved (rank the ways these could be improved)? 1.4. What implementation and management processes were used related to coordination with USAID, the eight USAID Missions, and Feed the Future implementing partners? 1.5. What have been the primary implementation and management challenges related to coordination with USAID, the eight USAID Missions, and Feed the Future implementing partners (rank these challenges)? 1.6. In what ways could implementation and management processes related to coordination with USAID, the eight USAID Missions, and Feed the Future implementing partners be improved (rank the ways these could be improved)? 1.7. What implementation and management processes were used related to coordination with the long-term degree-seeking students, host country trainees that have attended short-term training events, and local researchers? 1.8. What have been the primary implementation and management challenges related to coordination with the long-term degree-seeking students, host country trainees that have attended short-term training events, and local researchers (rank these challenges)? 1.9. In what ways could implementation and management processes related to coordination with the long-term degree-seeking students, host country trainees that have attended short-term training events, and local researchers be improved (rank the ways these could be improved)? 2. To what extent have effective communication and coordination supported project implementation and outcomes regarding research and dissemination and capacity development? In what ways could communication and coordination be improved between USAID/BFS, the eight USAID Missions, the ME, the five Lead U.S. Institutions and Host Country Universities, and other stakeholders as a way to better achieve project goals and objectives, create and carry out a shared research agenda, meet reporting requirements, and expand collaboration? 62 2.1. What communication and coordination processes were used related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with the five Lead U.S. Universities, Host Country Universities, and Partner Institutions? 2.2. What have been the primary communication and coordination challenges related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with the five Lead U.S. Universities, Host Country Universities, and Partner Institutions (rank these challenges)? 2.3. In what ways could communication and coordination processes related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with the five Lead U.S. Universities, Host Country Universities, and Partner Institutions be improved (rank the ways these could be improved)? 2.4. What communication and coordination processes were used related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with USAID/BFS, the eight USAID Missions, and other stakeholders? 2.5. What have been the primary communication and coordination challenges related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with USAID/BFS, the eight USAID Missions, and other stakeholders (rank these challenges)? 2.6. In what ways could communication and coordination processes related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with the USAID/BFS, the eight USAID Missions, and other stakeholders be improved (rank the ways these could be improved)? 3. To what extent did AquaFish generate quality research technologies/outputs using appropriate metrics (e.g., peer reviewed publications)? In what ways have the AquaFish ME and the five Lead U.S. Institutions fostered dissemination of research results to host country stakeholders? What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward? 3.1. How did AquaFish generate quality research technologies/outputs using appropriate metrics, such as peer-reviewed publications? 3.2. What have been the primary challenges to generating quality research technologies/outputs using appropriate metrics (rank these challenges)? 3.3. In what ways could AquaFish improve the generation of quality research technologies (rank the ways this could be improved)? 3.4. What processes have been used by the ME and the five Lead U.S. Institutions to foster dissemination of research results to host country stakeholders? 63 3.5. What have been the primary challenges faced by the ME and the five Lead U.S. Institutions to fostering dissemination of research results to host country stakeholders (rank these challenges)? 3.6. In what ways could the ME and the five Lead U.S. Institutions improve the dissemination of research results to host country stakeholders (rank the ways this could be improved)? 3.7. What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward (rank these opportunities)? 4. How well has the program identified and addressed the academic and technical capacity needs of host country stakeholders? How could the program better serve and provide for the capacity needs of these stakeholders? 4.1. What processes have been used to identify and address the academic and technical capacity needs of host country stakeholders? 4.2. What have been the primary challenges to identifying and addressing the academic and technical capacity needs of host country stakeholders (rank these challenges)? 4.3. In what ways could the academic and technical capacity needs of host country stakeholders be better identified and addressed (rank the ways this could be improved)? 5. How has the ME performed in establishing productive collaborations with the Host Country Universities, 19 Partner Institutions, and other Feed the Future Innovation Labs and other relevant USAID programs in the target countries? How could the ME improve in building these collaborations? 5.1. What processes have been used to establish productive collaborations with the Host Country Universities and Partner Institutions? 5.2. What have been the primary challenges to establishing productive collaborations with the Host Country Universities and Partner Institutions (rank these challenges)? 5.3. In what ways could the program improve collaborations with the Host Country Universities and Partner Institutions (rank the ways these could be improved)? 5.4. What processes have been used to establish productive collaborations with other Feed the Future Innovation Labs and other relevant USAID programs? 5.5. What have been the primary challenges to establishing productive collaborations with other Feed the Future Innovation Labs and other relevant USAID programs (rank these challenges)? 5.6. In what ways could the program improve collaborations with other Feed the Future Innovation Labs and other relevant USAID programs (rank the ways these could be improved)? 6. In what ways has the ME been responsive to requests for technical assistance from the 19 Partner Institutions, USAID, and the eight USAID Missions and Feed the Future implementing partners? In what ways could the ME have been more responsive? 64 6.1. What processes have been used respond to requests for technical assistance from the Partner Institutions? 6.2. What have been the primary challenges to responding to requests for technical assistance from the Partner Institutions (rank these challenges)? 6.3. In what ways could the program improve its response to requests for technical assistance from the Partner Institutions (rank the ways this could be improved)? 6.4. What processes have been used respond to requests for technical assistance from USAID, and the eight USAID Missions and Feed the Future implementing partners? 6.5. What have been the primary challenges to responding to requests for technical assistance from USAID, and the eight USAID Missions and Feed the Future implementing partners (rank these challenges)? 6.6. In what ways could the program improve its response to requests for technical assistance from USAID, and the eight USAID Missions and Feed the Future implementing partners (rank the ways this could be improved)? 7. Describe and highlight which research technologies had the greatest impact or potential for the greatest impact. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (e.g., breeding vs. feeds development vs. post-harvest management)? What topics are promising areas for further investment in aquaculture and fisheries? 7.1. What research technologies had the greatest impact or potential for the greatest impact (rank these topics)? 7.2. What evidence exists that these research topics had the greatest impact or potential for the greatest impact? 7.3. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (rank these topics)? 7.4. What evidence exists that these specific technical areas within aquaculture and fisheries merit new or continuing research investment? 7.5. What topics are promising areas for further investment in aquaculture and fisheries (rank these topics)? 7.6. What evidence exists that these specific topics are promising areas for further investment in aquaculture and fisheries? 65 2. KII Guide – USAID Staff, USAID Mission Staff, and Feed the Future Implementing Partner Staff Interview date: Interviewer: Name(s): Institution: Title(s): Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called ME&A. Our team is conducting an evaluation the AquaFish Innovation Lab, which was funded by USAID. We would like to conduct a brief interview today to learn about your perspectives on project performance. This information will be used in a report for USAID that will be publicly available. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. The objective of this research is to improve the performance of projects like AquaFish. The information may be used by other organizations as well. There is no direct benefit to USAID for your participation in this study. The purpose is only to help us improve the services of projects like this one. If you chose, your responses can remain anonymous. The interview is expected to take about one hour. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation, and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate. ◻ No, I will not participate. 66 1. To what extent has the ME effectively implemented and managed the AquaFish Innovation Lab’s activities in the five target regions of Africa and Asia? How effectively has the ME managed and coordinated among the five Lead U.S. Universities and Host Country Universities, 19 Partner Institutions, USAID and the eight USAID Missions and their Feed the Future implementing partners, and the long-term degree-seeking students, host country trainees that have attended short-term training events, and local researchers? In what ways could implementation and management be improved? 1.1. What implementation and management processes did AquaFish use related to coordination with USAID, the USAID Missions, and the Feed the Future implementing partners? 1.2. How did the implementation and management processes used by AquaFish related to coordination with USAID, the eight USAID Missions, and the Feed the Future implementing partners facilitate your work? 1.3. What have been the primary implementation and management challenges with AquaFish related to coordination with USAID, the eight USAID Missions, and the Feed the Future implementing partners (rank these challenges)? 1.4. How did the implementation and management challenges related to coordination with USAID, the eight USAID Missions, and the Feed the Future implementing partners affect your work? 1.5. In what ways could implementation and management processes related to coordination with USAID, the eight USAID Missions, and the Feed the Future implementing partners be improved (rank the ways these could be improved)? 2. To what extent have effective communication and coordination supported project implementation and outcomes regarding research and dissemination and capacity development? In what ways could communication and coordination be improved between USAID/BFS, the eight USAID Missions, the ME, the five Lead U.S. Institutions and Host Country Universities, and other stakeholders as a way to better achieve project goals and objectives, create and carry out a shared research agenda, meet reporting requirements, and expand collaboration? 2.1. What communication and coordination processes were used by AquaFish related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with USAID/BFS, the eight USAID Missions, and other stakeholders? 2.2. How did the communication and coordination processes used by AquaFish related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with USAID/BFS, the eight USAID Missions, and other stakeholders facilitate your work? 2.3. What have been the primary communication and coordination challenges with AquaFish related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with USAID/BFS, the eight USAID Missions, and other stakeholders (rank these challenges)? 67 2.4. How did the communication and coordination challenges with AquaFish related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with USAID/BFS, the eight USAID Missions, and other stakeholders affect your work? 2.5. In what ways could communication and coordination processes related to creating and disseminating a shared research agenda, meeting reporting requirements, and expanding collaboration with the USAID/BFS, the eight USAID Missions, and other stakeholders be improved (rank the ways these could be improved)? 3. To what extent did AquaFish generate quality research technologies/outputs using appropriate metrics (e.g., peer reviewed publications)? In what ways have the AquaFish ME and the five Lead U.S. Institutions fostered dissemination of research results to host country stakeholders? What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward? 3.1. How did AquaFish generate quality research technologies/outputs using appropriate metrics, such as peer-reviewed publications? 3.2. What have been the primary challenges AquaFish has faced in generating quality research technologies/outputs using appropriate metrics (rank these challenges)? 3.3. In what ways could AquaFish improve the generation of quality research technologies (rank the ways this could be improved)? 3.4. What processes have been used by the ME and the five Lead U.S. Institutions to foster dissemination of research results to host country stakeholders? 3.5. What have been the primary challenges faced by the ME and the five Lead U.S. Institutions to fostering dissemination of research results to host country stakeholders (rank these challenges)? 3.6. In what ways could the ME and the five Lead U.S. Institutions improve the dissemination of research results to host country stakeholders (rank the ways this could be improved)? 3.7. What opportunities are there for AquaFish to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward (rank these opportunities)? 4. How well has the program identified and addressed the academic and technical capacity needs of host country stakeholders? How could the program better serve and provide for the capacity needs of these stakeholders? 4.1. What processes has AquaFish used to identify and address the academic and technical capacity needs of host country stakeholders? 4.2. What have been the primary challenges AquaFish has faced in identifying and addressing the academic and technical capacity needs of host country stakeholders (rank these challenges)? 68 4.3. In what ways could AquaFish better identify and address the academic and technical capacity needs of host country stakeholders (rank the ways this could be improved)? 5. How has the ME performed in establishing productive collaborations with the Host Country Universities, 19 Partner Institutions, and other Feed the Future Innovation Labs and other relevant USAID programs in the target countries? How could the ME improve in building these collaborations? 5.1. What processes has AquaFish used to establish productive collaborations with other Feed the Future Innovation Labs and other relevant USAID programs? 5.2. How did the productive collaborations between AquaFish and other Feed the Future Innovation Labs and other relevant USAID programs facilitate your work? 5.3. What have been the primary challenges AquaFish has faced in establishing productive collaborations with other Feed the Future Innovation Labs and other relevant USAID programs (rank these challenges)? 5.4. How did the challenges AquaFish has faced in establishing productive collaborations with other Feed the Future Innovation Labs and other relevant USAID programs affect your work? 5.5. In what ways could AquaFish improve collaborations with other Feed the Future Innovation Labs and other relevant USAID programs (rank the ways these could be improved)? 6. In what ways has the ME been responsive to requests for technical assistance from the 19 Partner Institutions, USAID, and the eight USAID Missions and Feed the Future implementing partners? In what ways could the ME have been more responsive? 6.1. What processes has AquaFish used to respond to requests for technical assistance from USAID, and the eight USAID Missions and Feed the Future implementing partners? 6.2. How did AquaFish technical assistance facilitate your work? 6.3. What have been the primary challenges faced by AquaFish in responding to requests for technical assistance from USAID, and the eight USAID Missions and Feed the Future implementing partners (rank these challenges)? 6.4. How have challenges in delivering technical assistance affected your work? 6.5. In what ways could AquaFish improve its response to requests for technical assistance from USAID, and the eight USAID Missions and Feed the Future implementing partners (rank the ways this could be improved)? 7. Describe and highlight which research technologies had the greatest impact or potential for the greatest impact. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (e.g., breeding vs. feeds development vs. post-harvest management)? What topics are promising areas for further investment in aquaculture and fisheries? 69 7.1. What research technologies had the greatest impact or potential for the greatest impact (rank these topics)? 7.2. What evidence exists that these research topics had the greatest impact or potential for the greatest impact? 7.3. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (rank these topics)? 7.4. What evidence exists that these specific technical areas within aquaculture and fisheries merit new or continuing research investment? 7.5. What topics are promising areas for further investment in aquaculture and fisheries (rank these topics)? 7.6. What evidence exists that these specific topics are promising areas for further investment in aquaculture and fisheries? 70 3. KII Guide – Long-Term Degree Seeking Students, Trainees, and AquaFish-Supported Researchers Interview date: Interviewer: Name(s): Institution: Title(s): Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called ME&A. Our team is conducting an evaluation the AquaFish Innovation Lab, which was funded by USAID. We would like to conduct a brief interview today to learn about your perspectives on project performance. This information will be used in a report for USAID that will be publicly available. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. The objective of this research is to improve the performance of projects like AquaFish. The information may be used by other organizations as well. There is no direct benefit to USAID for your participation in this study. The purpose is only to help us improve the services of projects like this one. If you chose, your responses can remain anonymous. The interview is expected to take about two hours. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation, and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate. ◻ No, I will not participate. 71 1. To what extent has the ME effectively implemented and managed the AquaFish Innovation Lab’s activities in the five target regions of Africa and Asia? How effectively has the ME managed and coordinated among the five Lead U.S. Universities and Host Country Universities, 19 Partner Institutions, USAID and the eight USAID Missions and their Feed the Future implementing partners and the long-term degree-seeking students, host country trainees that have attended short-term training events, and local researchers? In what ways could implementation and management be improved? 1.1. What implementation and management processes did AquaFish use related to coordination between the Lead U.S. University, the Host Country University, and your Partner Institution? 1.2. How did the implementation and management processes used by AquaFish related to coordination between the Lead U.S. University, the Host Country University, and your Partner Institution facilitate your work? 1.3. What have been the primary implementation and management challenges with AquaFish related to coordination between the Lead U.S. University, the Host Country University, and your Partner Institution (rank these challenges)? 1.4. How did the implementation and management challenges related to coordination between the Lead U.S. University, the Host Country University, and your Partner Institution affect your work? 1.5. In what ways could implementation and management processes related to coordination between the Lead U.S. University, the Host Country University, and your Partner Institution be improved (rank the ways these could be improved)? 2. To what extent did AquaFish generate quality research technologies/outputs using appropriate metrics (e.g., peer reviewed publications)? In what ways have the AquaFish ME and the five Lead U.S. Institutions fostered dissemination of research results to host country stakeholders? What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward? 2.1 How did AquaFish generate quality research technologies/outputs using appropriate metrics, such as peer-reviewed publications? 2.2 What have been the primary challenges AquaFish has faced in generating quality research technologies/outputs using appropriate metrics (rank these challenges)? 2.3 In what ways could AquaFish improve the generation of quality research technologies (rank the ways this could be improved)? 2.3 What processes have been used by the ME and the five Lead U.S. Institutions to foster dissemination of research results to host country stakeholders? 2.4 What have been the primary challenges faced by the ME and the five Lead U.S. Institutions to fostering dissemination of research results to host country stakeholders (rank these challenges)? 72 2.5 In what ways could the ME and the five Lead U.S. Institutions improve the dissemination of research results to host country stakeholders (rank the ways this could be improved)? 2.6 What opportunities are there for AquaFish to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward (rank these opportunities)? 3. Describe and highlight which research technologies had the greatest impact or potential for the greatest impact. What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (e.g., breeding vs. feeds development vs. post-harvest management)? What topics are promising areas for further investment in aquaculture and fisheries? 3.1 What research technologies had the greatest impact or potential for the greatest impact (rank these topics)? 3.2 What evidence exists that these research topics had the greatest impact or potential for the greatest impact? 3.3 What specific technical areas, if any, within aquaculture and fisheries merit new or continuing research investment (rank these topics)? 3.4 What evidence exists that these specific technical areas within aquaculture and fisheries merit new or continuing research investment? 3.5 What topics are promising areas for further investment in aquaculture and fisheries (rank these topics)? 3.6 What evidence exists that these specific topics are promising areas for further investment in aquaculture and fisheries? 73 ANNEX H: BROCHURE EXAMPLE 74 75 76 ANNEX I: STATEMENT OF DIFFERENCE AquaFish Innovation Lab Oregon State University Corvallis, Oregon 97331-1643 USA P 541-737-6426 F 541-737-6408 aquafish.oregonstate.edu September 20, 2018 Dear Lesley and Cibeles, Below is a Statement of Differences submitted in anticipation of ME&A's final performance evaluation of the AquaFish Innovation Lab. The Lab ends on 28 Sep 2018, which is in 7 business days and during which time key personnel are unavailable for additional document review should the final report eventually be sent to OSU. Thus, it was not feasible to conduct a review of the final report of this evaluation. This is noted in the Statement of Differences. We note that the Evaluation Team has attended to some, but not all, of our previous comments in this latest version, and therefore the AquaFish Innovation Lab can not fully endorse this report. I look forward to seeing this Statement of Differences appended to the final ME&A report when it is uploaded to DEC, per information received from Cibeles Garcia Burt. Sincerely, Hillary Egna Hillary S. Egna, PhD Director and Lead PI AquaFish Innovation Lab GENERAL COMMENTS: Overall, we acknowledge the two members of the Evaluation Team (ET) for their quick work on this report during the short interview and data collection time period from Nov –Dec 2017 (1 month). We appreciate that they conducted their review independently and respected the time of the AquaFish IL's many partners. This Statement of Differences, requested by USAID, captures notable gaps, weaknesses, and concerns regarding the evaluation and the evaluation process. The final ME&A report was not received in a timely manner prior to the AquaFish Innovation Lab closedown on 28 Sept 2018. Thus, this Statement of Differences is based on previous versions of the report. The Final Report was not examined. Overarching concerns are summarized below, with specific comments highlighted in the next section. • This ME&A Review of AquaFish IL was their third in a series of review attempts since 2017, with two false starts in February and again in August 2017. The ME&A draft reports failed to disclose these various attempts, and in so doing overlooked a major limiting factor of the review. The evaluation process experienced three team turnovers, went through three team leads and numerous membership changes from the start of the process in February 2017, and had several changes of review coordinators within USAID. This turnover contributed to loss of knowledge and materials, as well as review fatigue among AquaFish IL participants and the Management Entity (ME). Amongst the most positive comments regarding the evaluation process are ironically the same comments that contribute to the evaluation’s overall weakness: it was done quickly and demanded little time of any single person in the AquaFish IL. Loss of information and materials through the three review attempts resulted in another major shortcoming: ME&A reviewers did not access critical documents prior to their Key Informant Interviews, and quite possibly prior to analysis. Critical documents had apparently not been provided by USAID to this third review team, and were subsequently requested from the ME at the end of January 2018, after most of the First Draft Report had already been written. Key documents sent were the USAID RFA for the Innovation Lab; the USAID-approved M&E Plan and FTFMS Indicators; and the internal RFP(s) for soliciting research sub-projects. • ME&A’s apparent disregard for personally identifiable information (PII) was evident throughout the evaluation drafts. Two examples are: 1) the Review Team pushed to interview students who are protected in the USA under FERPA; after being alerted twice by the Director to this compliance issue, the Review Team eventually sought clearance from USAID and developed a consent form for students to sign; 2) quotes in the report can easily be attributed to specific informants, and can be damaging. • The language of the draft reports revealed implicit bias related to gender and host country expertise. Examples include: 1) failure to attribute or use the words “leadership” when describing the Director (a woman), instead using “administer/ed” even when informants said “leadership”; 2) confusing gender and sex; 3) using words like “claim” and “assert” when referring to comments made by the ME (most are women) and words like “state” when referring to male PIs. Note that ME comments on the first draft such as: “claimed” gives the impression that these women are not to be believed, resulted in “claimed” being changed in the second draft report; and 4) failure to attribute mentorship qualities to HC scientists and professionals, instead stating that mentorship only occurs from US to HC scientists. • Concern was expressed by the ME regarding conflict of interest (COI) pertaining to one of the two reviewers who worked part-time on an AquaFish IL Project and was listed as a co-author on an AquaFish IL (CRSP at the time) investigation. It is unclear how COI was identified and evaluated by ME&A or by USAID, however, this level of conflict would be considered serious in academia. SPECIFIC COMMENTS: [Note reference to the Matrix are based on the draft report received June 2018. We did not receive the Final Report in time to review it prior to the end of the AquaFish IL] 1. Recognition of approved naming conventions throughout drafts report a. (Matrix line #2; “DONE”) The correct full name for AquaFish Innovation Lab (as designated by USAID/BFS) is: Feed the Future Innovation Lab for Collaborative Research on Aquaculture & Fisheries. ‘AquaFish Innovation Lab’ (or other appropriate abbreviations approved by USAID for purposes of this Evaluation Report) is the correct shortened name. AquaFish -- without the Innovation Lab ending -- is the name of a group that preceded USAID’s Innovation Lab and its use causes unnecessary confusion, especially when AFIL, Lab, or another abbreviated name can be used instead for the express purposes of this report and defined in the report’s first pages. This report is, after all, examining the performance of a mechanism under BFS, and needs to be clear. 2. Recognition of cost-share contributions, matching, and leveraging a. (Matrix line #5; “CHANGED THE TEXT”) The AquaFish Innovation Lab operates with a substantial amount of real money contributed by participating institutions outside USAID (non-federal Cost Share from participating US institutions and matching funds from other partners). The total cost of the program includes these non-USAID contributions, and was not captured in earlier versions of this Report. The wording used in our approved USAID reports is: “Program activities are funded in part by the United States Agency for International Development (USAID) under CA/LWA No. EPP-A-00-06-00012-00 and by participating US and Host Country institutions.” 3. Inappropriate use of this Evaluation for Informing Future Investments: Misrepresentation about Evaluation Purpose a. (Matrix line #7; “CHANGED THE TEXT”) The purpose of “informing future investments” was originally specifically excluded from this Evaluation, per conversations with USAID. This stated non-purpose has become 1/7th of the report and is problematic, as the evaluation design did not support robust gathering of information for setting strategies for a new $35M program or for investments in the sector. This superficial mechanism of strategy setting for publicly funded science is flawed and suffers from self-interested input bias, as well as a lack of vetted expert advice. Furthermore, AFIL participants were earlier advised that this evaluation would NOT be informing strategy for future aquaculture investments. Had participants realized that this evaluation would be informing a future agenda, AFIL participants surely would have focused more on this question and less on other questions. 4. Failure to recognize roles and responsibilities a. (Matrix line #8; “THANKS, CHANGED”) To improve accuracy, the MT suggested changing the word “administered” to "led" or "overseen". The word "administered," used throughout earlier drafts, gives a half-right half-wrong impression. The Director created and developed the overall vision for the AquaFish Innovation Lab and before that the Aquaculture & Fisheries CRSP. This report language may be reflecting attitudes towards a woman-led program, as much implicit bias has been attributable to this particular field. Hence words like leading and overseeing more satisfactorily describe the Director’s role in sharing her broad vision for the program, and reaching out and enrolling many actors and stakeholders, some of which, though not all, are the 5 major research projects. The ME also administers the program and its many moving parts, which is reflected in the revised text. However, the primary role of the Director is "leadership" of the program, as it is not a pass-through. b. (Matrix line #11; “CHANGED TO ‘PERSONNEL’”) The MT also suggested changing “staff” to “personnel,” “informants,” or “participants” when talking about AquaFish IL partners, ME, etc. “Staff” implies an employee-employer reporting relationship and is thus inaccurately used throughout the report. 5. Concerns with data collection and analysis a. (Matrix line #9; “THANKS, CORRECTED”) The ETs data collection and analysis had weaknesses and errors throughout the various iterations of this report. KII numbers did not add up correctly or match the table in the appendix, the designations in the KII table were incorrect, and names were misspelled. These errors affected all analyses using the KII Table and designations. KII were held with more than 4 current or former students; former students are important to identify as a group because the Report gives the impression that former “trainees” were not interviewed. This raised concerns regarding data collection and analysis. Time constraints for the team may have prevented the necessary QA/QC. In ME responses to previous draft, these errors were called out. b. New analyses and statements were riddled throughout the Second Draft. It is unclear why the new analyses and numbers were not presented in the First Draft. ME&A presented a new set of analyses that were not included in their First Draft Report. Again, additional (harsher) statements from ME&A may constitute retribution. 6. Limitations of the Study a. The Report neglects to disclose that the evaluation by ME&A had additional limitations associated with two previous reviews in the same calendar year that had false starts (Feb 2017 and Aug 2017) ahead of this Review (Nov 2017). The Evaluation process thus experienced several team turnovers, went through three team leads, and numerous membership changes from the start of the process in February 2017. This turnover contributed to loss of knowledge, and to review fatigue among AquaFish IL participants and the ME. Note: A similar situation occurred in an ME&A evaluation of another Innovation Lab, and statements to that effect were included in the respective Evaluation Report. b. (Matrix line #12; “INSERTED FOOTNOTE”) ME&A inserted a footnote to explain the discrepancy between the actual number of AFIL’s partnering institutions (27) and the incorrect number (19) of PIs which still appears throughout the report. It makes sense for ME&A, or USAID, to qualify the scope of the Review within the first few pages of the Report. Qualifying statements might include that the review focused on Phase II, from Mar 2013 through Sep 2017; focused on 19PIs instead of the 27 actual PIs because of the ToR from USAID; shortens the name AquaFish Innovation Lab to AFIL for the purposes of the report, etc. As was pointed out to ME&A in comments made on the First Draft: 19PIs was evidently the number of Partner Institutions used in the ToR that was approved by USAID. ME&A’s choice was to leave it as was approved in the EQs for the purposes of this limited Evaluation. c. (Matrix line #32; “WORDING SLIGHTLY CHANGED”) AFIL participants noticed the EQs (evaluation questions) were constructed as a way to ferret out problems, the opposite of what the ET reports concerning positive bias during the KIIs. The negative slanted questions dominated the survey instrument (~75% negative leaning questions) and can cause response bias. It says a lot that the Review is largely favorable and documents AFIL’s successful performance. 7. Failure to accurately capture or understand program development and history a. (Matrix line #36; “REVISED”) One area of concern throughout the various drafts of this Report was the inaccurate sequence of chronological events. For example, the RFP that Dr. Hillary Egna (Director and Lead Principal Investigator) wrote for future subcontracting partners was issued after she wrote the proposal to USAID and after OSU was awarded funding from USAID for the new AFIL. Future partners’ proposals were not part of her proposal for the new AFIL, and indeed were not even written by future applicants until well after OSU received its funding for the new AFIL. It is also worth noting that program development did not occur only via OSUs RFP mechanisms for future partners. The ME directly contracted with PIs on smaller research and outreach activities as well (hence the large number of PIs mentioned earlier). Perhaps the ET is confused because the structure of ILs has changed considerably at USAID since the start of Phase II. b. (Matrix line #41; “THE DATA INCLUDED REVIEW OF DOCUMENTS AND TELEPHONE INTERVIEWS”) While some PIs may have felt that the research topics limited their ability to address perceived development priorities, this is not a pass-through award for the ME to projects for their “individually” disconnected wants. Together all parts are building towards a grander vision outlined in the AFIL award. Thus, the gestalt is important, while at the same time addressing local institutional needs. Although some new investigations have been approved as “peripheral and anticipatory,” in general the larger projects' proposals must show close alignment and ability to deliver quality collaborative work. In Year 3 of Phase II, after expected completion of IP2013-15 some lines of work (investigations) were not successful or were critically delayed and were unable to go forward. Those investigations (of 33 total) included: one at KNUST Ghana; three in Kenya; two in Zanzibar. These investigations were given no/low-cost extensions and were still included as part of the network, supported to attend meetings, allowed to complete delayed work previously funded, etc. Non￾continuation possibly introduced disappointment from these Africa projects, as manifested in the KIIs. This process was transparent and stated in the CA/LWA. 8. Omissions: Inadequate sampling and interview time for covering the diversity of the program. That something is not mentioned does not mean it did not occur. a. (Matrix line #16; “REMOVED THIS SENTENCE”) It is worth noting that ME&A failed to acknowledge or make a brief reference to the previous Phase I HCPI Exchange Project. This important project, implemented in AquaFish CRSP Phase I, was carried forward from previous Exchange Programs. Information was available in previous Annual Reports under this CA/LWA. It set the stage for much of the South-South collaborations and regional work in Phase II. b. (Matrix line #63; “FINDINGS ARE BASED UPON THE INTERVIEW DATA.”) In early drafts, the ET stated that "virtually no dissemination of research results to USAID” occurred which is factually incorrect. The ET insufficiently queried KIIs on this point in particular -- perhaps due to the time constraints of the Review. There has been significant dissemination to USAID in the period covered by this review. Supporting data: • In Phase II to date, the ME wrote two research "dissemination" articles for the USAID FtF Newsletter and four for USAIDs knowledge management group, AgriLinks. • The Director and team presented/produced over 15 technical talks and posters for USAID events. • The ME, not counting the 37 PIs, alone had over 4000 email interactions with USAID, 20% of which included or involved the Missions. Conservatively, 40% of the interactions involved exchange of information regarding AquaFish IL research results. c. Throughout drafts of the Report, the ET imprecisely describes Mission interactions with the MT. “Virtually no”, “very little” and “too busy” (words used by ME&A in reference to Mission interactions) lacks context to be fair to either the Missions or to BFS (now BRFS) and AFIL participants. A differently designed assessment tool would be needed to review Mission interaction -– indeed, the Evaluation Team themselves complained that they were unable to draw Mission personnel for interviews (they got only one from 15 asked) and perhaps reflected their own biases. Further, the Team failed to adequately query the ME on dissemination to Missions or more broadly. The Report seems to lay everything at the doorstep of the Missions with about half the Team’s recommendations requiring more work on the part of the Missions. d. (Matrix line #25; “THIS REFERENCE WAS DELETED”) Early drafts of the report failed to mention that not ALL partners remained engaged from Phase I to Phase II. From Phase I to II there was a 40% change of leads, from 8 down to 5. The 5 remaining lead projects were drawn from the most successful and relevant leads from Phase I, given that the terms in Phase II had changed considerably with USAIDs new RFA. e. (Matrix line #47; “BASED UPON INFORMATION COLLECTED”) ME&A neglected to mention that principal AFIL participants in the US and HC interact frequently with the ME. The ME estimates the email interaction between the ME and PIs to be over 1000 emails each month. The ET did not ask the ME this question specifically. In the ME first review of the First Draft Report, we suggested modifications to reflect this point. All participants are important to the network and are invited to annual and regional meetings (eg, for Asia regional meetings, African participants are invited, and vice versa). f. (Matrix line #46; “NOT ADDRESSED”) In order to serve as an “example of collaboration” as stated in the Report in Section 4.0, ME&A seems to have missed the point and did not include the following language (from the review of the First Draft Report): This investigation illustrates a successful collaborative effort involving a number of actors throughout the AquaFish IL network. NaFIRRI’s lead researcher on this investigation was supported in a PhD degree program at AU under AquaFish Phase I (graduating in Dec 2013), and before that as a Master’s degree student at AU under another USAID award to OSU as ME. This returning student was mentored into an enhanced role at NaFIRRI and later became HC project principal investigator with overarching responsibility for coordinating activities in Uganda and Kenya. His research investigation was made possible through a project the Director started on air￾breathing fish in 2006, involving researchers in Asia, LAC, USA, and Africa. In 2016, when a successful preliminary result on the Uganda lungfish investigation required replication, a collaboration among the Director (OSU/ME), researchers from a lead project in Asia under NSCU, and NaFIRRI came into being. Collaboratively, three PIs (2 lead US projects, 1 lead HC PI) and the ME are working to solve the complex problem of lungfish reproduction and establish a breeding program for this indigenous species. g. (Matrix line #72; “NOT A FINDING BASED UPON DATA COLLECTED”) It is worth highlighting that in Phase II (since March 2013) AFIL has indeed been successful in establishing many productive collaborations with other ILs, just not funded collaborations. Some recent examples from the past year alone, besides the Bangladesh Mission joint project with Horticulture and Nutrition ILs: the ME is also working with the SIIL in Cambodia under CESAIN which falls under the Mission; ME works with the IL Council and contributes in full. h. (Matrix line #73; “NOT A FINDING BASED UPON DATA COLLECTED”) The Reviewers neglected to ask the ME about providing technical assistance to US and HC PIs. Indeed, the ME received many requests from PIs for technical assistance. Most recently, the Director has been working on 4 co-authored papers with PIs, and is supplying data on historical parental stocks for GIFT fish genetics for a Ghanaian graduate student. i. (Matrix line #61; “THANKS. REVISED ACCORDINGLY”) OSU was omitted as a contributing research partner in early drafts of the report. In the case of cell phone applications in Africa, the Director has been involved in collaborative research efforts with this suite of projects, and worked to link them along with generally conceiving this project idea in the proposals she vetted to USAID in previous awards, as well as in the AquaFish IL program. Other support from the ME team has been provided as well. l. (Matrix line #65; “THIS WAS NOT A FINDING OF THE ET”) Based on KIIs, the ET identified three ways in which academic and technical capacity building was achieved. Additional language should be added to identify institutional strengthening and capacity building that occurred through degree program development (e.g., KNUST), internships and service learning (all), research administration process support for compliance (e.g., BAU), and engagement in professional conferences (all). 9. Insufficient data collection: Coming to conclusions and then adding data afterwards. Data collection on M&E occurred after all interviews were completed, and the first draft was likely written. ME&A was unable to rectify this in either draft version of their report. a. (Matrix line #29; “LANGUAGE REVISED ON THIS SUBJECT”) Early drafts of this Report operated on the assumption that AFIL did not possess a functional M&E Plan. AFIL does indeed have an M&E Plan (and Framework) which was approved by USAID when the AFIL proposal was reviewed in March 2013, and amended 6 months later in Sep 2013, and again approved by USAID. A “Results Framework” per se was not requested in the USAID RFA, as it was considered an outmoded vehicle for M&E at the time of application in 2013. M&E is reported on annually according to the format requested by USAID for the Annual Reports and FTFMS indicator reporting. Within the USAID approved M&E plan, there are specific indicators, benchmarks, and milestones grouped into 4 categories—research, information dissemination, capacity building, and gender integration. Further, the M&E plan includes specific metrics in FTFMS. Additionally, there is specificity in many of the Plan’s benchmarks and milestones (e.g., 100 degrees, 30 workshops), which were designed to serve as guideposts for progress. Alas, the ET did not have a copy of the AFIL approved M&E plan during their review period. At the end of January 2018, the Director received a last minute request from the ET for the AFIL M&E plan and other important foundational documents (such as the RFA) so it is doubtful that the ET had adequate time to understand the approved M&E plan. The ET received these documents after all their interviews had been completed, and quite possibly after most of their report had already been drafted. b. (Matrix line #44; “ANALYSIS APPROPRIATE TO THE DATA”) The ET uses a nested “Russian Easter Egg” model to describe the AFIL subcontracting structure. However, there are significant exceptions and inaccuracies with this nested model. The complexity of the relationships and reporting structure and the reasons behind them are not captured with this simplistic model. Further, simplified (centralized) arrangements tend to result in responsibility and authority getting disconnected. The complex (“Russian Easter Egg”) relationships require more transactions. There is a careful balance, and greater empowerment in the field, in a distributed decentralized model that the ME oversees. OSU (ME and the Lead US Institution) “directly” contracts with all levels mentioned in this diagram. Examples of arrangements that do not operate as an “Easter Egg” (noting that these are non-exhaustive illustrations of the contractual arrangements described): 1. OSU/ME contracts to AU. AU subsequently contracts directly to all PIs/institutions (US & HC) under its umbrella. 2. PU contracts to VT (US), which contracts to KNUST in Ghana for its specific investigations. PU also contracts directly to KNUST. 3. OSU directly contracts to PIs in KNUST (Ghana), SUA (Tanzania), Nong Lam University (Vietnam) and does not pass through US lead projects such as Purdue or Auburn. C. (Matrix line #68; “BASED UPON EVIDENCE GATHERED”) ME&A conclude that “’Sandwich degrees’ are helpful for reducing costs and have the added benefit of keeping the students engaged with their host country institution.” While this suggestion can be helpful to US universities for expanding opportunities, the basis for this recommendation is not supported by robust data or evidence presented in this report. As well, it is doubtfully widely shared among AFIL participants many of whom, through the AquaFish CRSP and AFIL networks, have created strong regional universities with aquaculture degrees. Some of the very best universities with aquaculture programs are in Asia (e.g. Can Tho University and Shanghai Ocean University) and were not only part of AFIL, but are now offering scholarships to AFIL students who wish to pursue advanced degrees after AFIL ends. 10. Possibility of Retribution a. The Second Draft Report added different and new information since the previous (first) version and a decidedly more negative slant. Iterative reporting resulted in a very short turn around time for AFIL review and a full response was therefore infeasible. Due to the 5-day turn around time mandated by USAID for the review of the Final Report (which has not been received)– during which time MT personnel were on travel and not available - AFIL can not endorse this Final Report. Through the various iterations of this Report, whole sentences and new ideas were added that were not in the previous versions, including: Theory of Change (which shows that Evaluators are reviewing out of context as Change Theory was not widely used in the development arena when the 2013 AFIL proposal was written); funding disbursement criticism seems to have been invented anew and taken out of context; and so on. ME&A may have been sensitive to earlier rounds of AFIL comments, hence the added edginess on later drafts. b. (Matrix lines #42 & #43; “ALL INFORMATION WAS TAKEN FROM INTERVIEWS AND DOCUMENT REVIEW”) The revelation that all interviewees described disruptions due to delays in receiving funds is a comment that appeared in later drafts of this report. Likely comments were taken out of context, and perhaps this is an example of retribution. In the case of funding: • The ME received and acted on complaints that Auburn’s subcontracting PIs were frustrated by Auburn’s slow repayment mechanism, however, Auburn has always had a positive balance, and no shortage of funds from the ME. Likely, AU was ensuring accountability and deliverables being met by their subcontracting partners. • Contracting actions for new work may have been slow on occasion to ensure research compliance requirements were met (e.g. IRB and Animal Use) and to ensure performance benchmarks were met from previous funding cycles and associated deliverables received by the ME. • Although funding from USAID to the ME was delayed during Phase II, these delays did not directly impact funding allocations from OSU to the lead project U.S. universities. 11. Carelessness about PII (personal identifiable information) and about language choices a. (Matrix line #48; “QUOTE REFLECTS VIEW OF SPEAKERS”) Not all quotes from KIIs are informative or appropriate to include in the Evaluation Report. Previous versions of the ME&A would not drop quotes, even though they may do others harm and didn’t add anything more than was stated in the preceding and following sentences. Other examples of PII are found throughout the various versions of the report. Opinions of two US PIs of Africa lead projects were cited, when there are only two persons fitting this description. A quote identifying that the informant has over 30 years experience, yet there is only one person fitting this description. b. (Matrix line #66; “THANKS. PARAGRAPH REVISED”) Previous drafts of this report included phrasing that was paternalistic, emphasizing only that there were opportunities for HC partners to learn from US-based personnel. AFIL mentoring occurs from HC to HC and from HC to US as well – many of our HC participants are as or more experienced than their US counterparts. Perhaps this informant was thinking of HC junior faculty and students primarily? The same could be said for US junior faculty and students too. c. The Review Team pushed to interview students who are protected in the USA under FERPA; after the Director alerted them to this compliance issue the Review Team eventually sought clearance from USAID and developed a consent form for students to sign. d. (Matrix line #69; “A DIRECT QUOTE”) A USAID Mission staff member was quoted as saying that CRSP meetings used to be held biannually. It is unclear which meetings this informant (likely from the Uganda Mission) is referring to and may be mistaken. The ME&A can choose to include direct quotes, regardless of who said it, however, it is incumbent on the ET to fact-check for accuracy. Please note that the FTF-IL meetings were held annually and moved each year to a different country. f. (Matrix line #71; “NOT A FINDING BASED UPON DATA COLLECTED”) The ET concludes: “These collaborations were mainly due to the cohesive nature of the aquaculture research community, built up over many years prior to the AquaFish IL.” The statement needs clarification indicating that the cohesive nature exists specifically within the AquaFish IL (and previous CRSP) research community by design and not necessarily any broader research community. 12. Other Omissions a. Clarification is needed that this is an evaluation of AquaFish Innovation Lab Phase II from March 2013 through Sep 2017. b. Much of the data supplied following the FY17 Annual report were not used by ME&A to update numbers of students, numbers of peer-reviewed publications, etc. in this Report.