PERFORMANCE EVALUATION REPORT FOR FEED THE FUTURE INNOVATION LAB FOR GENOMICS TO IMPROVE POULTRY (GIP-IL) April 3, 2018 PERFORMANCE EVALUATION REPORT FOR FEED THE FUTURE INNOVATION LAB FOR GENOMICS TO IMPROVE POULTRY (GIP-IL) This publication was produced at the request of the United States Agency for International Development under Contract/Task Order Number: AID-OAA-1-15-00024/AID-OAA-T0-16- 000008. It was prepared independently by: Luis Bernal, MPP. Team Leader Chris Ashwell, PhD. Poultry Genomics Specialist Paul Miller, PhD. Poultry Production Specialist Contractor: Program Evaluation for Effectiveness and Learning (PEEL) Mendez England & Associates 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 www.mendezengland.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT The purpose of the performance evaluation of the Genomics to Improve Poultry - Innovation Lab (GIP￾IL) is to provide evidence that USAID's Bureau for Food Security can use to inform expanded investments in animal genomics research, improve effectiveness of existing programs, and better achieve outcomes. GIP-IL’s aim is improving poultry production among African small farmers through advanced genomic approaches that enhance chicken’s innate resistance to Newcastle Disease Virus (NDV) and heat stress. The GIP-IL was awarded to the University of California at Davis in 2013 with an end date of September 2018. This evaluation covered the period April 2013 - April 2017. The Evaluation Team (ET) used mixed methods to assess the project's progress, its success building host country capacity, and its effectiveness in fostering collaborations. The ET also evaluated project management and prospects for the project's future. Quantitative and qualitative methods included document reviews, key informant interviews, focus group discussions, site visits, and an online survey. The evaluation concluded that the project has had partial success achieving its research objectives and has produced high-quality research. It has successfully built capacity by training researchers and students; and has had success achieving sustainable collaborations with academics and, to a lesser extent, with government, civil society, and private sector partners. The project has also successfully engaged women. Technical and research aspects are managed well, but logistical and other constraints remain. Therefore, the project would benefit from additional funding for an extension to complete research, validate, and distribute poultry among small farmers. CONTENTS EXECUTIVE SUMMARY....................................................................................................................................................i Evaluation Purpose ............................................................................................................................................................i Project Background...........................................................................................................................................................i Evaluation Design, Methods and Limitations................................................................................................................i Evaluation Questions........................................................................................................................................................i Evaluation Limitations.......................................................................................................................................................i Main Findings, Conclusions, and Recommendations................................................................................................. ii 1.0 EVALUATION PURPOSE AND QUESTIONS.......................................................................................... 1 1.1 EVALUATION PURPOSE ................................................................................................................................1 1.2 EVALUATION QUESTIONS...........................................................................................................................1 2.0 PROJECT BACKGROUND............................................................................................................................ 2 2.1 THE DEVELOPMENT PROBLEM...................................................................................................................2 2.1.1 The Feed the Future - GIP-IL ..........................................................................................................3 2.2 THEORY OF CHANGE ...................................................................................................................................3 2.3 PROJECT ACTIVITIES ......................................................................................................................................3 2.3.1 Project Timeline .................................................................................................................................4 3.0 EVALUATION METHODS AND LIMITATIONS..................................................................................... 5 3.1 EVALUATION DESIGN ...................................................................................................................................5 3.1.1 Evaluation Methods............................................................................................................................6 3.2 EVALUATION LIMITATIONS ........................................................................................................................7 4.0 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS............................................................... 7 4.1 EQ 1: RESEARCH ..............................................................................................................................................7 4.1.1 EQ. 1.1: To what extent did the GIP-IL make progress towards developing poultry with enhance NDV and heat resistance?........................................................................7 4.1.2 EQ 1.2: What other research and work remains to fully complete the research objective? .......................................................................................................................................... 11 4.1.3 EQ 1.3: What work remains to get a usable research output (poultry with enhanced resistance to Newcastle Disease Virus) to meet the goal of the GIP-IL?........................... 12 4.1.4 EQ 1.4: To what extent did the GIP-IL produce quality research outputs using appropriate metrics?....................................................................................................................... 12 4.1.5 EQ 1.5: What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for FTF countries) moving forward?.............................................................................................................................. 14 4.2 EQ 2: CAPACITY BUILDING...................................................................................................................... 17 4.2.1 EQ 2.1: Within the context of the GIP-IL research program, how well has UCD identified and addressed the academic and technical capacity needs of the host country stakeholders? .................................................................................................................... 17 4.2.2 EQ 2.2: How could the project better serve and provide for the capacity needs of those stakeholders?.................................................................................................................... 22 4.3 EQ 3: INSTITUTIONAL COLLABORATIONS........................................................................................ 24 4.3.1 EQ 3.1: How has UCD performed in establishing productive collaborations with host country governmental and academic institutions, local NGOs, and other relevant USAID programs in target countries?......................................................................... 24 4.3.2 EQ 3.2: How could the GIP-IL improve its institutional collaborations? ............................ 26 4.4 EQ 4: MANAGEMENT .................................................................................................................................. 27 4.4.1 EQ 4.1: To what extent has the GIP-IL management team effectively implemented and managed the GIP-IL’s respective research and training activities in Ghana and Tanzania?........................................................................................................................................... 27 4.4.2 EQ 3.2: In what ways could implementation and management be improved?................... 29 4.4.3 EQ 4.3: How effectively has the GIP-IL management team managed and coordinated amongst the partners, Missions, host country governments, and local researchers?....... 30 4.4.4 EQ 4.4: In what ways could these partnerships be improved, or new partners engaged? ............................................................................................................................................ 30 4.5 EQ 5: PROGRAM FUTURE .......................................................................................................................... 33 4.5.1 EQ 5.1: What specific technical areas, if any, within the GIP-IL’s research program merit new or continuing research investment?........................................................................ 33 4.5.2 EQ 5.2: What topics are promising for further investment in animal genomics for improved livestock? ........................................................................................................................ 33 4.5.3 EQ 5.3: In what ways could the GIP-IL management, research, capacity development, and/or institutional collaborations be improved to better achieve its objectives?............ 34 4.5.4 EQ 5.4: Is there a need for any social or economic research in the next phase?............. 34 4.6 CROSS-CUTTING ISSUES............................................................................................................................ 36 4.6.1 EQ 6.1: Gender................................................................................................................................ 36 4.6.2 EQ 6.2: Nutrition ............................................................................................................................ 37 4.6.3 EQ 6.3: Environment and Biosecurity......................................................................................... 38 ANNEXES .........................................................................................................................................................................40 ANNEX 1. SCOPE OF WORK AND EXPRESSION OF INTEREST................................................................. 41 ANNEX 2. EVALUATION METHODOLOGY IN DETAIL................................................................................. 50 ANNEX 3. NUMBER INTERVIEWED, SURVEYED, OR IN FOCUS GROUPS.............................................. 58 ANNEX 4. DATA COLLECTION INSTRUMENTS.............................................................................................. 60 ANNEX 5. GIP-IL PUBLICATIONS .......................................................................................................................... 69 ANNEX 6. PROJECT ACTIVITY PROGRESS......................................................................................................... 73 ANNEX 7. EVALUATION TIMELINE ...................................................................................................................... 78 ANNEX 8. Evaluation Team COMPOSITION, ROLES, and CONFLICTs OF INTEREST ........................... 80 ANNEX 9. STATEMENT OF DIFFERENCE............................................................................................................ 85 ACRONYMS AI Avian Influenza BFS USAID Bureau for Food Security BSL-3 Biosafety Level 3 CBDP Chicken Breeding and Distribution Plan ELISA Enzyme-linked Immunosorbent Assay EOI Expression of Interest ET Evaluation Team FAO Food and Agriculture Organization of the United Nations FGD Focus Group Discussion FTA Flinders Technology Associates FTF Feed the Future GIP-IL Genomics to Improve Poultry – Innovation Lab GWAS Genome Wide Association Study IDB Infectious Bursal Disease IEE Initial Environmental Examination ISU Iowa State University KII Key Informant Interview KPI Key Performance Indicators ME Management Entity M&E Monitoring and Evaluation ND New Castle Disease NDV New Castle Disease Virus NGO Non-Government Organization OIE World Organization for Animal Health PCR Polymerase Chain Reaction PEEL Feed the Future Global Program Evaluation for Effectiveness and Learning PI Principal Investigator qPCR Quantitative Polymerase Chain Reaction qRTPCR Quantitative Reverse Transcriptase Polymerase Chain Reaction RNA Ribonucleic Acid UCD University of California, Davis UoG University of Ghana USG U.S. Government SOW Scope of Work SEPRL USDA Southeast Poultry Research Laboratory SNP Single Nucleotide Polymorphism SUA Sokoine University of Agriculture SWOT Strengths, Weaknesses, Opportunities, Threats Analysis USDA United States Department of Agriculture USAID United States Agency for International Development i EXECUTIVE SUMMARY EVALUATION PURPOSE The external performance evaluation of the Genomics to Improve Poultry - Innovation Lab (GIP-IL) project was conducted to assess the project’s efficiency and effectiveness. The evaluation provides actionable recommendations to the U.S. Agency for International Development (USAID) Bureau for Food Security (BFS), USAID Missions, and implementing partners (IPs) and universities, to inform programming decisions for their Advance Approaches to Combat Pests and Disease Program. The results of the evaluation may also be useful for other stakeholders involved in designing research for similar development programs. The USAID/BFS is also interested in recommended changes to consider for the GIP-IL and similar future projects. PROJECT BACKGROUND USAID/Feed the Future awarded the GIP-IL to the University of California at Davis (UCD) as a management entity (ME) in 2013. GIP-IL is a five-year project, which ends in September 2018. The UCD and Iowa State University (ISU) coordinate research activities with University of Ghana (UoG) (Accra) and Sokoine University of Agriculture (SUA) (Morogoro - Tanzania). As per the Cooperative Agreement between USAID and UCD, "The goal of this research program is to improve the production of chicken by households and smallholder farmers, and thereby improve food security, nutrition and livelihoods in Africa,” which directly links to the Feed the Future Program key goals. The GIP-IL research program intends to apply advanced genetics and genomics approaches to sustainably enhance innate resistance to Newcastle Disease Virus (NDV) and heat stress in chickens, thereby improving production by small farmers and households in Africa. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation used a mixed-methods design, combining a comprehensive analysis of existing quantitative data with qualitative techniques to capture information from a wide range of stakeholders. This approach allowed for complementary data triangulation so that linkages between activity inputs, outputs, and outcomes could be determined and understood. The evaluation covered the project’s life cycle from design in 2013 through April 30, 2017. Data collection was conducted in Tanzania and in Ghana between August 23 and September 30, 2017. To answer the evaluation questions (EQs), the ET collected data using stakeholder mapping; a Key Performance Indicator (KPI) assessment; review of scientific publications; Key Informant Interviews (KIIs); Focus Group Discussions (FGDs); an online survey; and site visits to (i) Davis, California, (ii) Accra, Ghana, and (iii) Morogoro, Tanzania. EVALUATION QUESTIONS The EQs, as defined by USAID/BFS, are presented in Section 1.3. They are grouped into six categories; 1) research and training objectives; 2) capacity building; 3) institutional collaborations; 4) program management; 5) program future; 6) and cross-cutting issues. This evaluation also assessed project performance against USAID Evaluation Policy’s accepted evaluation criteria of relevance, effectiveness, efficiency, sustainability, and cross-cutting issues. EVALUATION LIMITATIONS 1. Project team members share research and management responsibilities, which meant it was not possible to report responses by role. This structure limited the ET’s ability to disaggregate responses from the survey and interviews by role. 2. The project has a centralized reporting information system, so KII and FGD responses tended to be repetitive. Particularly in FGDs, most discussion questions elicited a detailed response ii from one stakeholder, which was followed by agreement from the rest of the group. This limited the diversity and richness of perspectives from individuals. MAIN FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Findings Conclusions EQ1: RESEARCH AND TRAINING To What Extent did the GIP-Il Make Progress Towards Developing Poultry with Enhanced NDV and Heat Resistance? • GIP-IL has made significant progress in three out of six research objectives (1, 2 and 6) aimed at developing poultry with enhanced NDV and heat resistance. • With the introduction of the velogenic approach to the research program, GIP-IL experienced a significant expansion of program activities, a change in resource allocation, and additional capacity building requirements that caused a substantial delay in progress. • Many infrastructure, logistical, human capacity, and institutional constraints hindered project progress. • GIP-IL identified many aspects of the genetics behind NDV resistance and provided high-quality training to research team members and students. • Stakeholders are concerned that sustainable breeding of improved chickens will require long-term support for poultry producers. • The project has partially achieved its objectives but underestimated the institutional, logistic, technological, financial, and human capacity constraints faced by African countries. • The decision to include the velogenic approach may help facilitate achievement of the project’s overall goal. • The above decision and other challenges delayed progress toward developing poultry with enhanced NDV and heat resistance; therefore, it is unlikely that the project will meet those objectives by its end date. • There is a consensus among stakeholders that UCD and ISU have brought state-of-the-art research technologies and methodologies to the GIP-IL and its African partners. What Other Research and Work Remains to Fully Complete the Research Objective? • Some key research activities have not started. They include: 1) development of an economical low-density Single Nucleotide Polymorphism (SNP) panel for genetic selection of disease resistance; and 2) validation of disease resistance in African indigenous chicken population using such a low-density SNP panel. • Stakeholders consider the research conducted by GIP-IL to be of high quality but, in order to be of real value, it must be validated and applied. • Objectives 3, 4, and 5 remain to be started and continued work is needed on Objectives 2 and 6. • The knowledge GIP-IL has generated through research has contributed to the understanding of how birds with varying resistance to NDV respond to infection but additional work on validation and distribution is needed. What Work Remains to Get a Usable Research Output (Poultry with Enhanced Resistance to NDV) to Meet the Goal of the GIP-Il? • Activities addressing Objective 5 and 6, which involve developing validated breeding stocks, have yet to be completed and, according to stakeholders, will require time and investment beyond the current project timeline. • Stakeholders support a participatory model that takes into account specific contexts for developing the breeding and distribution plan (BDP). • African chicken populations are diverse and show a low degree of inbreeding. Other donors [e.g., the EU Innovative Management of Animal Genetic Resources (IMAGE) Project] are taking initiatives to develop a low density multi-species genotyping platform. • Diversity in chicken production differs greatly between Tanzania and Ghana. • GIP-IL team members have identified several extensions of the project that would contribute to reaching its goals. These include: studying other chicken ecotypes, further investments in research facilities, and providing long-term breeding support for sustained results. • The work that remains under Objectives 5 and 6 means that usable research outputs may not be generated by the project by its end date. • Continued value chain research in Tanzania and Ghana is needed to ensure BDPs are effective in both countries. • Consultation with other donors could be a good approach to incorporating additional traits, such as growth rate, which could be included in selection using SNP markers in order to address the diversity issue. iii Findings Conclusions To What Extent did the GIP-Il Produce Quality Research Outputs Using Appropriate Metrics? • Research metrics are collected and recorded. However, not all management and research team members are aware of the research metrics collected by the project. • GIP-IL has produced five peer-reviewed publications and has two works in press and two articles undergoing peer reviews. An additional 29 publications are planned or are in progress. • Research metrics are recorded, but they did not seem to be used on a regular basis to guide project management and research decisions. • The project has produced some research that peers have agreed is of sufficient quality for publication and more publications are in the pipeline. Recommendations for EQ1: • GIP-IL project implementers should conduct a detailed assessment of existing capacity and built-in some flexibility in order to develop a comprehensive and practical implementation schedule. • Project implementers should use the management and research metrics developed at the start of the project to measure all aspects of the project on an ongoing basis. • Project implementers should, if allocated more time and investment, prioritize the research objectives that remain to be met and develop a clear implementation plan. • The GIP-IL program should stay focused on completing the discovery phase of work, and carefully design and use the low￾density SNP panel. • To attain significant, long-term, improved NDV and heat stress resistance chickens, future GIP-IL efforts must focus on sustainably developing the validated and improved breeding stocks. • Project implementers need to conduct several more years of research and produce multiple generations to show sustainable effects. • All relevant stakeholders should continue to examine the advantages and disadvantages of natural exposure and controlled exposure for completing Objective 2. • Poultry value chain research should include increased participation of farmers and other community stakeholders, including women, to ensure sustainability. • Management and research team members should review the existing information sharing guidelines to ensure all participants are up-to-date on the research metrics collected and recorded. • Additional time should be allowed in order to complete validation and distribution of chickens for breeding. In order to meet GIP-IL’s goal of developing chickens with enhanced NDV and heat resistance, funding will need to be extended. EQ 2: CAPACITY BUILDING Within the Context of the GIP-Il Research Program, How Well Has UCD Identified and Addressed the Academic and Technical Capacity Needs of the Host Country Stakeholders? • GIP-IL has transferred techniques [such as tear collection, qPCR (Quantitative Polymerase Chain Reaction), specific Enzyme-linked Immunosorbent Assay (ELISA) applications, etc.] previously unknown by UoG and SUA. • Trainees appreciate the “hands-on” approach to training that UCD has provided to UoG and SUA, despite many logistical obstacles. Researchers felt the project made improvements in strengthening experimental skills. • The research teams in Africa have become more aware of their lack of knowledge and experience on biostatistics and data analysis. • Training activities have been designed by management to target women beneficiaries and women’s participation. • GIP-IL has been responsive to the technical and longer-term capacity needs of UoG and SUA. • Training of staff and students, whether short-term or postgraduate-level, is by far the strongest capacity building activity the GIP-IL has brought to the African universities. • Presenting, participating in fora, and writing pieces in trade reports, are notable GIP-IL activities that have provided opportunities to share research results. iv Findings Conclusions How could the Project Better Serve and Provide for the Capacity Needs of those Stakeholders? • Consulted stakeholders believe that greater investment in building the local capacity of young researchers in Africa will help improve overall quality. Identified constraints are: the timely release of funds; the need for stable power and better Internet access at facilities; bio-security and ill-equipped facilities; and overall timeframe of project activities. • Infrastructure improvements may not be as significant as training for capacity building, but they have driven SUA and UoG institutional commitment. Additional planning and resources are necessary to ensure that facilities function as envisioned. • The sustainability of longer-term collaborative research and outreach relationships among partners is highly dependent on securing external funding. Recommendations for EQ2: • The expertise of the U.S. research teams should be better utilized to improve African research capabilities and good practices in research. • A plan should be developed and discussed with local institutional leaders, relevant ministries, and host missions to ensure the timely release of funds, stable power, and better Internet access at facilities, to aid insecure and ill-equipped research facilities and ease the overall timeframe of project activities. • UoG and SUA should expand GIP-IL’s training efforts by sharing the project and its progress with more students in more of their schools. This should be done by developing a knowledge management strategy and it should include existing outreach activities such as attendance at conferences and workshops. The use of social media platforms should also be explored as part of the knowledge management strategy. • A special emphasis should be placed on training students in advanced techniques such as tear collection, use of Flinders Technology Associates (FTA) cards, qPCR, specific ELISA applications, etc. This would ensure that students have access to the most up-to-date tools and protocols. • Investments in capacity-building especially on biostatistics and data analysis should be continued to help the GIP-IL management team to plan and intensify on the job training so that African staff and students can make sense of the results of the project lab works. • GIP-IL should take advantage of its training experience by creating a research and project manual that would last beyond the program and could be used as a training-the-trainer tool. • GIP-IL should produce a research logbook to keep a record of research successes, mistakes, achievements, experiences with various techniques, etc. Such a record would allow current and future participants in similar programs to learn from detailed research accounts and experiences. • GIP-IL should increase the number of publications considered scientific. The project should have a professional editor. • Despite funding concerns, the four universities should try to establish longer-term collaborations beyond and above the GIP-IL. These efforts must involve top-level university officers. EQ 3 : INSTITUTIONAL COLLABORATIONS How has UCD Performed in Establishing Productive Collaborations with Host Country Governmental and Academic Institutions, Local Non-Governmental Organizations (NGOs), and Other Relevant USAID Programs in Target Countries? • UCD has established partnerships and collaborations with a wide range of partners including: public entities, academic institutions, trade associations, private companies, and other related projects. • GIP-IL reports 11 major collaborations at different stages of progress with African and U.S.-based institutions. • Collaborations with Ministries of Agriculture have been established in both Ghana and Tanzania. • The extent to which the project has reached out to local NGO partners is not clear since this information is not systematically captured. • UCD has performed reasonably well in establishing collaborations that are relevant to the project and helpful to its activities. • Collaborations exist not only in formal written partnership agreements, but also through a constant exploration of opportunities for synergy with key individuals and institutions. • The project does not appear to have documented any discussions of establishing NGO collaborations. v Findings Conclusions How could UCD Improve its Institutional Collaborations? • USAID/Tanzania has shown little interest in GIP-IL while USAID/Ghana is actively engaged in GIP-IL’s activities. • Some potential collaborations are not fully developed as in the case for institutions identified in the Cooperative Agreement (Heifer International, Helen Keller International, Save the Children, and the Aga Khan Foundation). • Researchers have suggested that the project explore active research and outreach opportunities with other relevant projects (e.g., the African Chicken Genetic Gains). • Institutional collaborations with institutions identified in the Cooperative Agreement as potential partners is not developed fully and new collaborations are not being actively pursued. Recommendations for EQ3: • GIP-IL should begin planning to strengthen existing partnerships and establish new institutional collaborations that will be necessary to implement and enhance activities under the project’s Objectives 5 and 6. These new partnerships should also consider the project’s overall sustainability. EQ 4: MANAGEMENT To what extent has UCD Effectively Implemented and Managed the GIP-Il’s Respective Research and Training Activities in Ghana and Tanzania • Key informants in Ghana and Tanzania felt positive about the Research Advisory Committee (RAC) member technical contributions. They felt RAC’s engagement with the management team brought in more openness and accountability for technical issues. • The management of contractual procedures at UCD is complicated, especially during project startup, and there is a consensus among stakeholders that the project will not be able to meet all six project objectives within the existing timeframe. • Management is taking steps to address infrastructure and communications problems and other related delays that have had implications for conducting research activities. • The project has built capacity at SUA and UoG in training in advanced genetics sampling and laboratory procedures. The project’s training and capacity building program has increased capacity and generated data and results in Objectives 1 and 2. • The most significant challenges to research and training activities in Ghana and Tanzania are infrastructure and communications problems. In What Ways could Implementation and Management be Improved? • Low management and administrative staffing levels have meant that researchers are required to perform project management and administrative duties. • Researchers want to improve the quality of scientific research by: 1) hiring a biologist with NDV expertise to complement the NDV genomics-only expert into the research team; 2) providing greater opportunities for RAC members to contribute in research; and 3) providing more training and resources for data analysis and better facilities. • RAC members believe that the project’s research strategy, especially as it relates to conducting velogenic NDV challenge studies, is in need of revision. • The lack of dedicated administrative staff in Africa has necessitated that researchers take time away from research to accomplish logistical tasks. • While some actions toward improvement can be taken by project implementers, some factors hindering project management and implementation are beyond the control of the team. These include administrative requirements within universities and limitations in infrastructure and capacity—human, technical, and institutional—at partner institutions in Africa. How Effectively has the Management Team Managed and Coordinated Amongst the Partners, Missions, Host Country Governments, and Local Researchers? • GIP-IL facilitated coordination among its partners and researchers through industry poultry events and annual conferences in 2014, 2015, 2016 and 2017. At these events, participants were given the opportunity to share and discuss current research interests. • Annual conferences have provided an opportunity for the in-person exchange of research results and learning among the project’s stakeholders, researchers, students, and partners. vi Findings Conclusions In what Ways could These Partnerships be Improved, or New Partners Engaged? • UCD and ISU have worked to establish relationships and networks between the African universities. • Private sector and NGO partnerships have been fewer and more difficult to achieve. • Project academic partnerships are the strongest but partnerships outside of academia are less robust. • The project’s management team has prioritized the establishment of partnerships and networks to further the objectives of the project but has not been able to reach all categories of stakeholders identified. • Additional efforts to reach out to the private sector and NGOs are the primary ways in which partnerships can be improved. • Partnerships could be improved by involving other research institutions and ministry officials as well as other donors. Recommendations for EQ4: • UCD should reach out more to the private sector and NGO communities and involve other animal labs and research institutions, ministry partners, local farmers, and international partners in its training activities. • The project’s research strategy should address the conduct of velogenic NDV challenge studies more directly. • GIP-IL management must improve its ability to plan and strategize so it thinks along a product development pathway rather than just developing research outputs. • Based on its own experience, the GIP-IL team should develop a proposal to UCD, ISU, UoG, and SUA management leadership recommending the adoption of administrative processes that are more agile than typical procurement rules, and more attuned to the realities of working in international settings. These recommendations might be useful in managing a second project phase. • Working closely with relevant government officials and stakeholders, GIP-IL must find ways to solve the most critical supply, logistical, operational, and resource limitations of the project, namely inadequate power, Internet access, video conferencing, ad-hoc research facilities, etc. • The project should provide the African principal investigators (PIs) with project coordinators so the PIs do not spend too much time on management, operational, and logistical problems or on running errands that come from scarce financial resources, leaving them less time to focus on research. • Training should be planned and organized in advance and tailored to specific research needs, as opposed to being implemented on an ad hoc basis or in response to emerging circumstances. • Increased GIP-IL interaction with the RAC is highly beneficial for the project and is currently too scarce for RAC members to provide more effective project strategizing and monitoring. • The four universities led by UCD should have USAID clarify the programmatic consequences of Tanzania no longer being a Feed the Future target country. • The GIP-IL management team should develop a sustainability plan for the project beyond its current life cycle. Such a plan must go beyond funding considerations and include: support from the leaders of the four universities; future availability of personnel trained in genomics; data analysis and biostatistics; and a product (validated chickens), which is accepted by the small poultry producers and consumers. These factors should be considered in concert with the current, systematic sustainability plan. EQ 5: PROGRAM FUTURE What Specific Technical Areas, if any, within the GIP-Il’s Research Program Merit New or Continuing Research Investment? What Topics are Promising for Further Investment in Animal Genomics for Improved Livestock? • Stakeholders agreed that the project must stay on track to meet project objectives. They provided specific suggestions for new research investments that would better meet project outcomes. These include: investments that deepen and prioritize the study of genomics and heat stress; additional SNP markers; the addition of chicken ecotypes; sustainable breeding programs; research facility improvements; NDV vaccinations; increased capacity development; and contributions to overarching food security and climate change goals. • In terms of research on animal genomics, stakeholders are optimistic about the ultimate results of the program. However, they are aware that the future depends on university support of the program and, more importantly, on USAID’s continuous engagement and funding. vii Findings Conclusions In What Ways Could The GIP-Il Management, Research, Capacity Development, And/or Institutional Collaborations be Improved to Better Achieve its Objectives? Is There a Need for Any Social or Economic Research in the Next Phase? • To address the development of the Chicken Breeding and Distribution Plan (CBDP), management team members recommended: validating the CBDP models identified in the Wilson study; conducting a market-based study on economic feasibility; conducting a value chain analysis; considering gender integration more; and conducting analysis of public policy frameworks. • The GIP-IL management understands a CBDP requires detailed knowledge of the social and economic realities where chickens will be distributed. Recommendations for EQ5: • Given that USAID is considering a second phase of the project, UCD, ISU, UoG, SUA, and USAID should strategize about the program’s future within the policy framework of the U.S. Global Food Security Act of 2016 and its Global Food Security Strategy. • The USAID Mission in Ghana has engaged with GIP-IL, and the country is one of the selected 12 Feed the Future target countries. This suggests that GIP-IL should reinforce its ties with the Mission to make sure that poultry genomics fits within the country’s Feed the Future priorities. • UCD, ISU, UoG, SUA, and USAID should work together on a GIP-IL’s sustainability plan that includes budget considerations, capacity building, institutional strengthening, and research capabilities, as well as human, logistical, and technological needs for the future. • UCD, ISU, UoG, and SUA should start exploring options for long-term collaboration—even in the absence of USAID funding. CROSS-CUTTING ISSUES Gender, Nutrition, Environment / Biosecurity GIP-IL shows a remarkable level of responsiveness to USAID gender policies, as it: • Embraces four key gender-related considerations in formulating agricultural projects: 1) access to and control of livestock; 2) roles, responsibilities, and decision-making; 3) who maintains the local knowledge; and 4) who has access to agricultural or livestock services. • Recognizes the realities of scarce women’s participation in genomics research programs in Africa. • Includes indicators in reports that record: the gender of all field and laboratory staff; workshop and training participants; graduate students and postdocs; advisory committee members; and the stakeholders consulted to ensure women’s participation and overall gender balance. • In an African context where not many women achieve university-level education, let alone PhD level, the role of women within GIP-IL’s implementation is significant in terms of decision￾making, commitment, and leadership. This is a manifestation of the diverse composition of the project’s research and management team. • Nutrition is not a lab-based activity in the project. Nevertheless, nutrition has played a role in other activities such as the: selection of African ecotypes; considerations of the study; and the GIP-IL follow-up survey on the improved chicken distribution model. • Nutrition is a project design consideration because it is an essential part of the development problem the GIP-IL is meant to solve. Therefore, the project’s objective of improving poultry production through genomics will contribute to nutrition and the overall health of children and families in Ghana and Tanzania. • GIP labs and avian facilities are compliant of relevant World Organization for Animal Health (OIE) biosafety and biosecurity measures, as well as Center for Disease Control (CDC) biosafety measures for Microbiological and Biomedical Laboratories. • Although the environmental impact is not a major concern on GIP-IL’s activities, biosecurity is, and the project has experienced some setbacks due to limitations in implementing strict biosecurity measures. viii Findings Conclusions Recommendations for Cross Cutting Issues: • Moving forward, the main gender-related recommendation is that GIP-IL continue outreach to women in its activities, especially when the time comes for the chicken breeding and distribution program in Africa. • Nutrition must remain a key element of the GIP-IL as Feed the Future target countries in 2017 include Ghana and may potentially include other African countries such as Ethiopia, Kenya, Mali, Niger, Nigeria, Senegal, and Uganda. • GIP-IL should continue to prioritize the highest level of biosecurity at labs and avian facilities. This includes: isolating the chicken houses from noisy areas (i.e. student housing and roads); using a guestbook; using a pass-over; using a pass-through; requiring hairnets; requiring handwashing; enforcing footwear change; and requiring the use of provided clothing. 1 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE The Genomics to Improve Poultry Innovation Lab (GIP-IL) is a five-year (2013-2018) project funded by the U.S. Agency for International Development (USAID) Bureau for Food Security (BFS) as part of the Feed the Future Innovation Lab portfolio of programs. The goal of the GIP-IL is to improve production of poultry in developing countries by applying advanced genetics and genomics to sustainably enhance innate resistance to Newcastle Disease Virus (NDV) and heat stress. The GIP-IL project was awarded to the University of California at Davis (UCD) and three partner universities—Iowa State University (ISU), University of Ghana (UoG), and Sokoine University of Agriculture (SUA) in Tanzania. The purpose of this external performance evaluation of the GIP-IL project is to assess the project’s implementation and progress toward achieving its objectives. The evaluation provides an evidence based findings to the USAID/BFS, USAID Missions, and implementing partners (IPs) and universities. The findings will inform development of new investments in animal genomics research and contribute to both improved ongoing GIP-IL’s activity effectiveness and intended outcomes. Evaluation recommendations will help USAID/BFS enhance effectiveness and efficiency of current GIP-IL implementation and management, and consider changes for potential second phase of GIP-IL or similar projects. The evaluation was conducted by a team of three experts in external evaluation and poultry. The team included: Luis Bernal, Team Leader; Dr. Chris Ashwell, Poultry Science Specialist; and Dr. Paul Miller, Senior Poultry Production Expert. The evaluation was conducted between June and November of 2017. The target audience for the evaluation included USAID/BFS, USAID Missions, and implementing universities. The evaluation results may be applicable to other entities involved in designing relevant research for development programs. 1.2 EVALUATION QUESTIONS As per the USAID/BFS’s Expression of Interest (EOI) (Annex 1), the evaluation questions (EQs) related to six areas: management, implementation, research program, capacity development components, program future, and cross-cutting issues. Research and Training Program • To what extent did the Poultry Genomics IL make progress towards developing poultry with enhanced NDV and heat resistance? • What other research and work remains to fully complete each research objective? • What work remains to get a usable research output [poultry with enhanced resistance to Newcastle Disease (ND)] to meet the goal of the Poultry Genomics IL? • To what extent did the Poultry Genomics IL produce quality research outputs by using appropriate metrics (e.g., peer-reviewed publications)? • What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for FTF countries) moving forward? Capacity Building • Within the context of the Poultry Genomics IL’s research program, how well has UC Davis identified and addressed academic and technical capacity needs of host country stakeholders? 2 • How could the project better serve and provide for the capacity needs of these stakeholders? Institutional Collaborations • How has the Management Entity (ME) performed in establishing productive collaborations with host country governmental and academic institutions, local NGOs, and other relevant USAID programs in the target countries? • How could the ME improve its institutional collaborations? Program Management • To what extent has the UC Davis effectively implemented and managed the Poultry Genomics Innovation Lab’s respective research and training activities in Ghana and Tanzania? • In what ways could implementation and management be improved? • How effectively has UC Davis managed and coordinated amongst the partners, Missions, host country government partners, and local researchers? • In what ways could these partners be improved, or new partners engaged? Program Future • What specific technical areas, if any, within the Poultry Genomics IL’s research program merit new or continuing research investment (e.g. continued genetic characterization of different African ecotypes for enhanced NDV resistance, testing poultry breeding programs in country to disseminate/scale improved poultry to smallholder farmers)? • In what ways could the Poultry Genomics Innovation Lab’s management, research (i.e. design, implementation, stakeholder involvement), capacity development programs, and/or institutional capacity collaboration be improved to better achieve objectives? • What topics are promising for further investment in animal genomics for improved livestock and poultry production in developing countries? • Is there a need for any social or economic research in the next phase? Cross-cutting issues • Gender • Nutrition • Environment 2.0 PROJECT BACKGROUND 2.1 THE DEVELOPMENT PROBLEM The USAID/BFS - UCD Cooperative Agreement describes, in the following terms, the social and economic problems the project was developed to solve: Across rural Africa, women and children raise poultry for household consumption and to sell the eggs and birds. Micronutrient malnutrition is common among the rural poor, causing stunting and reduced cognitive development. Eggs are considered a perfect source of critical micronutrients and protein for pregnant and lactating women, and for children in the first 1,000 days of life. The income from poultry is often one of the few significant sources of income for women and is particularly important for household emergencies, to purchase staple foods, or to pay for an emergency health clinic visit. Selling eggs and birds is often one of the few sources of money for people living under stressed environmental conditions. Poultry is socio-culturally important with few associated religious taboos. Poultry is small, 3 rapid in food production, requires small quantities of feed, is environmentally friendly, and small investment can improve production. With mortality rates as high as 80 percent among chicken flocks in rural communities, ND is the number one hurdle to raising poultry in Africa. Current prevention strategies focus on identifying chicken genes (or genetic markers) associated with NDV and their resistance to heat stress, allowing breeding of innately resistant chicken strains for the use of household producers and small farmers. 2.1.1 The Feed the Future - GIP-IL Considering the socio-economic problem described above, the goal of the GIP-IL is to: “improve the production of chicken by households and small farmers, and thereby improve food security, nutrition and livelihoods in Africa, which are key goals of the Feed the Future Program. The GIP-IL research program is intended to apply advanced genetics and genomics approaches to sustainably enhance innate resistance to NDV and heat stress in chickens to improve their production by small farmers and households in Africa.” 1 Application of GIP-IL results will be consistent with Feed the Future goals, which focus on increasing the number of farmers and others who apply new technologies of genetic marker evaluation and scientific selection of superior breeding stock; increasing the value of farm-level sales due to reduced illness and mortality of chickens; and increasing diversity in the diets of women and children by inclusion of more eggs and poultry. 2.2 THEORY OF CHANGE The GIP-IL is designed to contribute to the achievement of the overarching Feed the Future goal, which is to “sustainably reduce global poverty and hunger.” GIP-IL works through its Global Hunger and Food Security Research Strategy that consists of “engaging U.S. universities in agriculture and food security research, and human and institutional capacity development.” Specifically, GIP-IL’s Theory of Change can be described as follows: If Research-based knowledge of genes and/or genetic markers associated with NDV and heat stress resistance, both in the U.S. and Africa, increases; and If An economical, low-density Single Nucleotide Polymorphism (SNP) panel for genetic selection of resistance to NDV and to heat stress is developed Then A genetic enhancement of resistance to NDV and heat stress in African indigenous chicken ecotypes could be validated; and Then A validated, improved breeding stock with enhanced NDV and heat stress resistance could be developed in Africa; and Therefore mortality of African indigenous chicken ecotypes due to NDV and/or heat stress would decrease; Circulation of chicken with enhanced resistance to NDV and heat stress for family poultry production would become viable; and improved poultry production would enhance the income, nutrition, and livelihoods of poor families throughout rural Africa. 2.3 PROJECT ACTIVITIES The USAID-UCD Cooperative Agreement established the following project objectives: Objective 1. Identify genes, genetic markers, and genetic pathways associated with NDV resistance and heat resistance in three unique chicken populations in the U.S. using advanced genetic and genomic technologies. 1 USAID-UCD Cooperative Agreement No. AID-OAA-A-12-00080, p.14 4 Objective 2. Identify genes, genetic markers associated with NDV resistance in African indigenous chicken ecotypes using whole genome SNP panel. Objective 3. Develop an economical low-density SNP panel for genetic selection of disease resistance to NDV and heat stress by integrating analysis of results from Objectives 1and 2. Objective 4. Validate disease resistance to NDV infection in African indigenous chicken population using economic low-density SNP panel selection. Objective 5. Develop validated improved breeding stock to NDV resistance in Africa. Objective 6. Develop a chicken breeding and distribution plan (CBDP). Institutional Collaborations. Engage key stakeholders — governments, USAID Missions, poultry breeders, small farmer organizations, poultry suppliers, and women farmers groups. Outreach and Publications. Prepare outreach materials and updates, produce journal articles on various aspects and results of the project, and produce reports on final results. Cross-cutting. Address issues such as gender, capacity building, institutional strengthening, and outreach, as well as USAID policies across these issues. 2.3.1 Project Timeline The project timeline was originally planned as follows: UC Davis and ISU Research Activities 5 Project Management Structure The Program Director, on behalf of UCD, leads the management of the GIP-IL. As a donor, USAID has an important management leading role as well. Along with the Program Director, the main management collective bodies are the Management Team and the Research Advisory Committee (RAC). The Management Team is made up of the Research Coordinators at the four partner universities whom, along with a Program Manager, head the management and technical areas of the GIP-IL. The RAC provides technical advice to the program. It consists of seven volunteers, unaffiliated with UCD, who are internationally recognized experts in poultry genetics, genomics, quantitative genetics, poultry vaccination, biodiversity, poultry breeding, production, and disease prevention in African environments. 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION DESIGN The Evaluation Team (ET) adopted a mixed-methods design to address the EQs comprehensively. This approach allowed for collection and triangulation of both qualitative and quantitative data from different sources to determine linkages between project inputs, processes, outputs, and outcomes (see details in Annex 2). Site Selection For conducting research, GIP-IL uses laboratories and poultry breeding facilities at UCD, ISU, UoG, and SUA. In consultation with USAID/BFS, the ET purposively selected three universities—UCD (in the USA), UoG (in Ghana), and SUA (in Tanzania)—for site visits and collected primary data between August 23 and September 30, 2017. The ET visited each of the three universities to observe and assess their research and poultry breeding facilities, note similarities and differences in research protocols, observe researchers in their work environment, and obtain an overall perspective of research activities, which documents or phone interviews could not provide. Stakeholder Mapping and Sample Selection The first step in preparing for data collection was identifying GIP-IL’s stakeholders and determining their relationship with the project. To select participants for key informant interviews (KIIs), focus group discussions (FGDs), and the online survey, the ET identified three groups: 1) current and former people and institutions directly involved in GIP-IL’s day-to-day implementation activities; 2) people and institutions indirectly involved in the project; and 3) potential stakeholders—those individuals and institutions with interest in the project, but no current involvement. The above selection considered the fact that not all project activities have been implemented so some potential stakeholders (such as small farmers) had not been reached when this evaluation took place. Overall, the selection criteria led to the following stakeholder categories: • GIP-IL Research Team • GIP-IL Management Team • GIP-IL Research Advisory Committee • Project Trainees / Students • Project Partners 6 3.1.1 Evaluation Methods To collect data, the ET used various methods, including document review, review and assessment of existing key performance indicators, scientific publications assessment, KIIs, FGDs, site visit observations, and an online survey. Each of these methods is discussed in more detail below. Document Review. The ET reviewed a variety of relevant documents provided by USAID/BFS, UCD, ISU, UoG, and SUA. Documents included, but were not limited to, the UCD Technical Application proposal, USAID-UCD Cooperative Agreement, GIP-IL Annual Work plans and monitoring and evaluation (M&E) plans, GIP-IL semi-annual and annual reports, program publications, meeting minutes, ISU Animal Industry Report, Family Poultry Communications, Livestock Research for Rural Development, GIP-IL website materials and information (gip.ucdavis.edu), and the Feed the Future Research Strategy (feedthefuture.gov). Key Performance Indicators Assessment. Sources of performance data included the GIP-IL semi￾annual and annual implementation reports (Year 1, Year 2, and Year 3), the Feed the Future Monitoring Information System (FFMIS), and implementation data produced by the UCD project management team on an ad-hoc basis. Scientific Publications Assessment. Scientific publications are an expected output of the GIP-IL scientific research program. The ET followed the USAID Scientific Research Policy when assessing each publication. The policy is based on recommendations provided by the International Committee of Medical Journal Editors (ICMJE) for scientific publishing. One indicator of GIP-IL's research contribution is the number of publications in scientific journals; the ET also used this indicator to assess performance. Key Informant Interviews. The ET conducted 38 in-depth KIIs in the U.S., Ghana, and Tanzania. A few of the interviews were conducted by phone or using Zoom.us, a video conference service, but most were conducted in person. The KIIs were grouped into five categories determined through the stakeholder mapping exercise described above. The categories were; Management Team, Research Team, Research Advisory Committee, Students, and Institutional Collaborators. The ET customized the questionnaires according to each category of stakeholders. Focus Group Discussions. The ET convened three FGDs at UCD, UoG, and SUA. One member of the ET conducted each FGD while the other two team members assisted. At each location, all available GIP-IL stakeholders (management and research teams, trainees, and students) were invited to participate in the group. Each FGD aimed to discuss the project’s future in an open, informal, and deliberative setting. The group discussion created a synergy instrumental to exploring ideas, views, and experiences that might not have been brought up during individual interviews. Online Survey. Using Google Forms, an online survey was sent to all the GIP-IL stakeholders (N=30) identified through the stakeholder mapping (Table 1). Each selected stakeholder received the survey after participating in a KII, an FGD, or both (Annex 4). The survey’s purpose was to support a GIP-IL Strengths, Weaknesses, Opportunities, Threats (SWOT) analysis, and to collect data to answer USAID’s questions related to project management improvements and future activities. The SWOT analysis complemented the answers obtained through the KIIs on program future. Some survey questions were multiple choice, some called for scoring, and others were open-ended questions. The survey allowed for data triangulation with KII and FGD responses. The table below summarizes data collection methods by site. 7 Table 1. Number of Respondents by Data Collection Method by Country Data Collection Method USA Ghana Tanzania Remote TOTAL Key Informant Type GIP-IL Management Team 5 1 6 GIP-IL Research Team & Partners 3 5 2 10 Research Advisory Committee 1 1 7 9 USAID and USDA 1 1 2 Graduate Students 1 3 4 3 11 TOTAL 8 6 10 14 38 Online Survey 30 30 3.2 EVALUATION LIMITATIONS The major limitations of this evaluation include: • Project team members share research and management responsibilities, which meant it was not possible to report responses by role. This structure limited the ET’s ability to disaggregate responses from the survey and interviews by role. • The project has a centralized reporting information system, so KII and FGD responses tended to be repetitive. Particularly in FGDs, most discussion questions elicited a detailed response from one stakeholder, which was followed by agreement from the rest of the group. This limited the diversity and richness of perspectives from individuals. 4.0 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 4.1 EQ 1: RESEARCH FINDINGS 4.1.1 EQ. 1.1: To what extent did the GIP-IL make progress towards developing poultry with enhance NDV and heat resistance? The ET found that the GIP-IL project has made significant progress in three of six research objectives aimed at developing poultry with enhanced NDV and heat resistance. As Table 2 shows, Objectives 1, 2, and 6 have either been completed or partially completed. According to the performance indicator report (April 30, 2017), GIP-IL surpassed its targets set for Objective 1 by completing genotyping and sequencing SNP and RNA-seq; identifying associations of SNPs for 55 NDV and 57 heat resistance; and identifying 460 genes, 16 gene families, and 35 gene networks. GIP-IL is on the way to achieving the second objective—identifying genes and genetic markers associated with NDV resistance in African indigenous chicken ecotypes using a whole genome SNP panel. The U.S. research facilities have received over 3,000 samples of chicken and have completed genotyping for 2,999 birds. At the time of the evaluation, the project was developing a chicken breeding and distribution plan (CBDP) and had shared initial results at a planning workshop. 8 Findings from the online survey of 30 stakeholders corroborate the idea that GIP-IL has made great progress, even though they may not meet all their objectives by end of project. Asked about GIP-IL’s progress, 93 percent of stakeholders (28 of 30) agreed or strongly agreed that GIP-IL has made good progress towards the goal of developing ecotypes of African chickens with greater innate resistance to NDV and heat stress (see Table 3). However, 48 percent of stakeholders, or 14 of 30, agreed or strongly agreed that the project will achieve this goal before the end of 2018, the current funding year. Results from the KIIs and FGDs show a similar pattern of responses regarding GIP-IL’s progress. Fourteen (14) respondents, including eight management team members, a RAC member, and five research team members felt that, despite some setbacks, the project was making good progress. Specifically, these stakeholders said that they believe that even if not every objective is completed by the end of the project, GIP-IL is contributing to the ultimate goal of developing NDV and heat stress resistant African chickens. Of the few respondents in KIIs and FGDs who had less favorable views of progress, one member of the management team estimated that the project had only completed 30 percent of objectives. Two members of the research team expressed concerns that validation and distribution would require additional time, beyond the planned five years. These stakeholders assessed project performance based only on whether it will complete all six project objectives within the original timeline. Conversely, the majority of respondents recognized that the logistical delays should have been expected in genomics research in African countries, and that the addition of velogenic challenges to the scope of work will improve GIP-IL’s contribution to NDV resistant chickens. The project’s performance indicators and results (both targets and actuals) as of April 30, 2017, are presented in Table 2. Table 2. GIP-IL Performance Indicators Research Objective Performance Indicator Target September 30,2013 Actual April 30, 2017 1. Identify genes, genetic markers, and genetic pathways associated with NDV resistance and heat resistance in 3 unique chicken populations in the U.S. using whole genome SNP panel and (RNA Ribonucleic Acid) - Seq technique. Animal trials completed 6 10 Lab assays completed ELISA2: 1300 ELISA: 1300 qPCR: 2400 qPCR: 2400 iSTAT: 33,280 iSTAT: 33,280 Data collection completed All above assays plus body weight of 4400 All above assays plus body weight of 4400 Data analysis completed Quality control, processed and statistically analyzed Quality control, processed and statistically analyzed Genotyping and sequencing results in hand 600K SNP: 1100, 660 million bytes 600K SNP: 1100, 660 million bytes RNA-seq: 432 samples, 1,300 billion bytes data RNA-seq: 432 samples, 1,300 billion bytes data Associations of SNPs identified for NDV and heat resistance NDV:60 Heat Stress: 40 NDV:55 Heat Stress: 57 Genes, gene families, and gene networks identified Genes: 200-400 Gene families: 5-10 Gene network: 20-30 Genes: 460 Gene families: 16 Gene network: 35 2 ELISA: Enzyme-linked Immunosorbent Assay 9 Research Objective Performance Indicator Target September 30,2013 Actual April 30, 2017 Shared database operational 1 1 2. Identify genes, genetic markers associated with NDV resistance in African indigenous chicken ecotypes using whole genome SNP panel Breeding and challenging facilities 4 6 Breeding flock established 1600 1654 Animal trials conducted 3000 birds 4727 birds Lab assays completed ELISA: 3200 ELISA: 4727 qPCR3: 6000 qPCR: 4500 Data collection completed All above essays plus body weight of 9000 All above essays plus body weight of 15000 Data analysis completed Quality control, processed and statistically analyzed Quality control, processed and statistically analyzed Samples received in the U.S. 3000 3458 Genotyping results in hand 600K SNP: 3000, 1.8 billion bytes 600K SNP: 2999, 1.8 billion bytes Associations of SNPs identified for NDV 50-100 Still under analysis 3. Develop an economical low-density SNP panel for genetic selection of disease resistance to NDV and to heat stress by integrating analysis of the results from Objectives I and 2. 4. Validate disease resistance to NDV infection in African indigenous chicken population using economic low-density SNP panel selection 5. Develop validated improved breeding stock to NDV resistance in Africa. 6. Develop a chicken breeding and distribution plan (CBDP) Compile list of poultry breeding /distribution experts 1 list completed 1 list completed Plan guidance information organized and summarized 1 workshop 1 workshop Workshop results organized and summarized Plan completed Completed plan In progress 3 qPCR: Quantitative Polymerase Chain Reaction 10 Table 3. Survey Responses on Overall Progress “GIP-IL has made great progress towards developing poultry with enhanced resistance to NDV and heat stress.” “By the end of its current funding end in 2018, the GIP-IL will have met its goal of developing ecotypes of African chickens with greater innate resistance to NDV and heat stress.” Strongly Agree 12 (40%) 7 (24.1%) Agree 16 (53.3%) 7 (24.1%) Neutral 2 (6.7%) 10 (34.5%) Disagree 0 3 (10.3%) Strongly Disagree 0 1 (3.4%) Don’t Know/Decline 0 1 (3.4%) Total Responses 30 29 When this evaluation was being carried out, activities needed to meet Objectives 3, 4, and 5 had not begun. This is highlighted in Table 2 and stems from the fact that the project focused on completing Objectives 1 and 2 and related data analysis with African ecotypes before moving on to Objectives 3, 4, and 5. The following factors have also contributed to this delay: 1. Due to complicated contractual procedures at UCD regarding putting subcontracts in place, the initial collaboration between UCD and the GIP-IL African partners took longer than expected. 2. RAC recommended a substantial increase in the scope of work so that it could include a velogenic NDV challenge of the chicken lines studied in the U.S., and African ecotypes studied in Objective 2. This scientific decision impacted the project implementation timeline, but its inclusion is expected to contribute to developing poultry with enhanced NDV resistance. Specifically, the RAC’s recommendation to include a velogenic NDV challenge in the project’s scope of work led to the following implementation delays: • In view of the constraints related to the use of velogenic NDV in the U.S., birds from the two highly inbred lines were transferred to the U.S. Department of Agriculture (USDA) Southeast Poultry Research Laboratory-SEPRL where its BSL-3 facilities provided the opportunity to evaluate resistance to a field-isolated velogenic NDV. Equivalent experiments were not possible with the African ecotypes because there was no appropriate BSL-3 facility. Therefore, researchers conducted a natural exposure challenge in which the birds previously characterized for their response to LaSota NDV in Objective 2 were moved to remote locations and exposed to velogenic NDV-infected birds in the field. While this approach was less controlled than the lab-based velogenic NDV treatments in the U.S., it provides valuable phenotypic data comparable to how birds would be exposed to the virus in the field. • The RAC recommendation led SUA and UoG to conduct five additional natural velogenic NDV exposure field trials. This recommendation added six additional months per trial to the project timeline. • Power outages resulted in equipment failures and disease outbreaks at the challenge facilities in Africa, which in turn caused delays in completing the animal trials. • The UoG and SUA research teams lacked knowledge in advanced molecular assays and required both initial and follow-up training sessions. These extra sessions delayed laboratory analyses. • Breeder flocks collected from the field hatched unpredictably, and stress from captivity caused limited production and unforeseen losses. • Infectious bursal disease virus (endemic virus) outbreak in challenge flocks highlighted how challenging biosecurity conditions are to the research program in Africa. 11 4.1.2 EQ 1.2: What other research and work remains to fully complete the research objective? The ET found that research activities under Objectives 3, 4, and 5 were lagging and may not be completed by the end of the current phase of the project. Likewise, although Objective 6 has completed some relevant activities conducive to building a CBDP, the plan cannot be fulfilled until Objectives 3, 4, and 5 are completed. The researchers are already integrating analysis of the results from Objectives I and 2 and using the economic low-density SNP panel selection to validate the resistance to NDV infection in African indigenous chickens. The GIP-IL research team indicated that the remaining research activities and timelines include: • Conducting the Africa challenge experiments: − Velogenic NDV natural challenges (estimated completion, September 2017) • Executing molecular assays in Africa challenge experiments − Virus titers (estimated completion, September 2017) • Conducting statistical and bioinformatical analyses; • Identifying additional SNPs associated with NDV in Africa challenge experiments − Identifying extreme groups (resistant vs. susceptible regarding to virus titer and antibody response and pool-sequencing identified extreme groups, and medium group) (estimated completion, September 2017); − Identifying SNPs associated with NDV (estimated completion, October 2017); • Developing an economical diagnostic platform for genetic selection by April 30, 2018; • Validating genetic enhancement of resistance to NDV in African ecotypes (estimated completion, September 2018); • Deciding on an approach—either a high dose of LaSota NDV (or local NDV strain) or a velogenic natural exposure (based on the results from velogenic natural exposure data); • Conducting a genetic selection of NDV resistance enhanced local ecotypes; and • Validating disease resistance to NDV infection in the African indigenous chicken population using economic low-density SNP panel selection. The ET also observed that there is some disagreement among researchers and RAC members over the best method for conducting challenge trials under Objective 3. Two RAC members have suggested a shadow model, “infecting some birds and letting them naturally infect other birds…” to observe virus resistance. However, one of the RAC members cautions that using market birds in natural challenge trials introduces too many confounds that are beyond the researchers’ control. Instead, researchers “need to use a local isolate for infecting clean birds” by controlled exposure. All participants in FGDs and key informants—one management team member, four RAC members, and three research team members—felt that the GIP-IL is still in its learning or “discovery phase,” and will need to complete data analysis on African chickens before moving on to the next stage (next objectives). During the KII, one research team member said, “it seems like developing disease resistant chickens will take more than five years. Validation and distribution will go beyond that.” However, other respondents were more optimistic and believed the project has produced very encouraging results, even though obstacles and additional velogenic trials caused the project to fall behind schedule. According to stakeholders, the project also needs to adjust the plan to distribute improved chicken stocks to local individuals and organizations by country. Specifically, one RAC member pointed out that the current distribution systems are inadequate in low-income countries. Although some groundwork for distribution is already in progress, there is a need for the project to consult in-country poultry producers, breeding associations, and small farmers to validate the breeding stocks of chicken and 12 generate lessons learned for better implementation. When interviewed, three students explained that Tanzania and Ghana might require different distribution plans that address the specific context of each respective country. According to the students in Tanzania, community-level assessment and participatory research approaches are needed in Tanzania to help make the research outputs usable. This is so that region-specific data on effective distribution systems can be developed—plans that take into account widespread and small-scale hatcheries. A better understanding of key stakeholder roles and responsibilities in improving distribution processes requires additional research on value-chain activities. Two students interviewed in Ghana said that even though Ghana has a commercial poultry value chain in place, a more detailed value chain research is needed, especially to include gender in poultry value chain analysis. One RAC member explained that poultry production is “a family thing, even if it’s a company. Often, it’s the women who manage the poultry.” 4.1.3 EQ 1.3: What work remains to get a usable research output (poultry with enhanced resistance to Newcastle Disease Virus) to meet the goal of the GIP-IL? GIP-IL’s overarching goal is improving household and small farmer chicken production by providing them with birds innately resistant to NDV and heat stress. To provide enhanced NDV-resistant chickens, GIP￾IL still needs to complete the remaining research activities mentioned in Section 4.1.2. The activities include the need to implement and achieve Objective 5, which involves developing validated breeding stocks of NDV-and-heat-stress-resistant chickens in Africa. Three research team members, one RAC member, and one student suggested that “Given the extent of work necessary to address all objectives, USAID will need to either extend the project or plan for a second phase.” 4.1.4 EQ 1.4: To what extent did the GIP-IL produce quality research outputs using appropriate metrics? Based on GIP-IL’s progress report by the end of Year 3, the project has produced high-quality research products published in peer-reviewed journals and other professional forums. As of September 2017, GIP-IL’s capacity building support has produced 38 publications in various stages of development. These are listed in Annex 5 and referenced in the section below. The project has also produced promotional, institutional, or outreach publications that have provided information about project activities and results. To determine the scientific rigor of the publications, the ET adopted guidelines established in USAID’s Scientific Research Policy as a benchmark of quality. 4 The policy considers issues like research relevancy and the use of the peer-review process and requires publications to follow a standard format and style, and to ensure that methods are derived from an experimental design. Publications must also verify that results address statistical significance, and that authorship demonstrates associate research. Given these specifications, at the time of the evaluation, the project had published high-quality research, including: • Three manuscripts published in peer-reviewed journals; • Two manuscripts submitted to peer-reviews journals on completed work; • A book chapter has been published; • Several commercial poultry industry reports have been completed; • Dozens of conference proceeding abstracts have been published; • Several additional manuscripts are being prepared. In fact, 39 publications are planned or being conceptualized; and • An additional 40 presentation/posters that are considered promotional, institutional, or outreach publications were developed, but these are not scientific publications. Online survey results indicate that 100 percent of the respondents (N=30) felt the project is conducting high-quality research (23 strongly agree, and seven agree). This finding is in line with responses from KIIs and FGDs—all the participants agreed that the project has produced quality research. These favorable 4 USAID Scientific Research Policy, 2014 13 opinions are attributed to the solid research background, experience, technical, and scientific capacities of the two U.S. universities involved. These universities are known to have access to state-of-the-art technologies like SNPs, GWAS, RNA-sequencing (a novel approach for NDV), and qPCR, and they conduct relevant, cutting-edge studies that make them the leaders in the field. Through document review and KIIs, the ET determined that GIP-IL’s research has made significant contribution to the understanding of how birds with varying resistance to NDV respond to infection. This knowledge might be useful to others working in the areas of tropical animal breeding, chicken genomics, NDV immunity, and vaccinology. All RAC and research team members agree that using high￾density SNP-typing and the 600K panel to identify genomic loci associated with NDV response and heat stress is an appropriate, state-of-the-art approach. Evaluating the gene expression response to NDV and heat stress using RNA-sequencing and pathway analysis is also a valuable and innovative approach. Integrating these two sets of molecular data to identify the most relevant genes/markers into the future low-density panel is novel as well. It will provide valuable analytical approaches for similar future projects. One RAC member said that the project made the “right choices of the technologies used.” However, two RAC members noted that the project is “too focused on genomics and has disregarded biological components of NDV.” These two RAC members went on to caution that vaccination will always be needed to control NDV, and said they believe validation trials should include a sub-sample of chickens vaccinated against NDV. GIP-IL’s management has identified research metrics to assess ongoing performance and quality of research by different partners. GIP-IL uses these metrics on a regular basis to assess the program’s progress and support program decision-making. However, the Cooperative Agreement only requires the project to present results from performance indicators in annual and semi-annual reports. The ET requested detailed research metrics and the management team provided them on an ad hoc basis (see Annex 6). The metrics include the number of completed lab assays, the number of data collection and analysis completed, and overall activity completion. The GIP-IL collects many of its research metrics using the USAID-issued spreadsheet, “Guidance on counting technologies for USAID crop and animal breeding and selection projects.” The spreadsheet format calls for indicators such as novel genes with known major effect(s) on specific traits; transgene or genetic elements for improved traits; tissue specific gene promoters identified and validated; molecular genetic markers linked to genes that control specific traits; transformation-ready gene constructs; identification and validation of Quantitative Trait Loci (QTL) for major effects; panels of genes or markers used in association mapping studies; population of lines or breeds used in phenotyping and large crossing blocks; and QTLs, marker loci or traits incorporated in a background genotype, among others. Each partner documents the GIP-IL research metrics and specifies the relevant chicken breed (i.e., commercial layer line, inbred line) and sample sizes. Annual and semi-annual reports provide evidence of the research metrics that research teams at each university collected and recorded. Table 4 presents examples of research metric results. In KIIs, research team members affirmed that the project has successfully collected and recorded phenotypic and physiological indicator metrics, and that completed analyses are collected and recorded. Although some KIIs expressed concern over the quality of the data, they confirmed that the project is documenting these research metrics. 14 Table 4. Examples of GIP-IL Research Metrics and Results Example Output Metrics Example Result Metrics Type of samples collected (i.e. tissue, tears) Physiological parameters associated with heat stress Analysis conducted (i.e. genome-wide association) Identification of genes and signal pathways associated with heat stress Methodology (i.e. natural exposure challenge vs. controlled exposure) NDV response parameters (i.e. viral load, antibody level) Number of trials complete or in progress Preliminary experimental results (i.e. safe period for challenge trials) Projected time to complete analysis and obtain results Identification of genes and signal pathways associated with NDV infection Status of breeding of ecotypes Results of NDV challenges 4.1.5 EQ 1.5: What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for FTF countries) moving forward? The ET gathered ideas, perspectives, and opinions on ways to improve research outputs from the project’s research team and RAC members. Some findings relate specifically to research, while others refer to the preconditions necessary for conducting quality research—including training and capacity building activities. Responses from all five FGDs with the research and management teams agree that the strongest, most meaningful opportunity for the GIP-IL project is to keep focused on achieving its ultimate goal: “applying advanced genetics and genomics to sustainably enhance resistance to NDV and heat stress in chickens to improve production by small farmers and households in Africa.” Members of the research team and the RAC members recommended that developing an economical, low-density SNP panel should include more than just the project-identified SNP markers. In their opinion, although the project has carefully selected SNPs associated with NDV resistance and heat stress, these were discovered in inbred lines and commercial layer lines, which are completely different from local African chicken populations. In their view, preliminary results obtained in African chicken populations that underline the polygenic nature of NDV resistance mechanisms must be considered, as should additional genome regions that might be relevant for selection. GIP-IL researchers stated that including additional SNP markers would encourage investigating other chicken traits such as growth rate. Likewise, given that African chicken populations are highly diverse and show a low degree of inbreeding, including SNP markers not linked to genomic regions associated with traits currently under selection could help monitor and maintain genetic diversity. Therefore, the SNP array should be used widely, not just for improving NDV and heat stress resistance. All RAC and management team members mentioned other research opportunities, including conducting practical experiments with African ecotypes in countries other than Ghana and Tanzania. According to them, these experiments need to include chicken breeds besides the Kuchi, Ching'wekwe, and Morogoro species. A velogenic NDV trial that allows researchers to use known doses of the virus to challenge the birds is also critical; however, this requires improved BSL3 research facilities in African countries, which the RAC and management team members identified as a valuable investment for Feed the Future countries. Further research opportunity lies in determining the differences between various African ecotypes in genes or regions of the genome identified as having enhanced resistance to NDV. Also worthy of research is using natural challenges to NDV during trials—such as inoculation of susceptible birds co-located with trial birds. 15 KIIs identified opportunities to improve project outcome around chicken breeding and distribution. They noted the importance of a long-term plan for selecting and maintaining the improved animals, and a production and provision plan for them that could follow several generations. The RAC members suggested that UoG and SUA be more strongly involved in the specific breeding and distribution plans for Ghana and Tanzania. They suggested that the project conduct participatory research with village chicken producers and stakeholders along the poultry value chain to determine the best ways to implement a CBDP. They also recommended exploring partnerships with the private sector and other ongoing research projects that have similar disease prevention goals. Some students mentioned the improvement of infrastructure as important for the improvement of research outputs. “Better and dedicated lab would be better, [sample] storage and analysis are in different locations…time could be better spent in having things in one location.” “Have the sample (i.e., blood) storage in the same facility as the ELISA and other equipment.” - “Have ELISA and other equipment dedicated to this project rather than sharing would be better.” Other opportunities mentioned included the improvement of resources and training and are addressed in other sections of this report. CONCLUSIONS EQ1.1: The project has partially achieved its objectives to develop poultry with enhanced NDV and heat resistance by completing Objectives 1, 2, and initiating work on Objective 6. These objectives focused on completing genotyping and sequencing, identifying genes and genetic markers, and developing a poultry CBDP specific to project countries. While the project achieved initial results, it has experienced unanticipated implementation delays due to contracting procedures and an increased scope of work. The decision to include the velogenic approach meant revising the project’s scope, but it will help facilitate achievement of the project’s overall goal of developing poultry with enhanced NDV resistance. The velogenic approach requires different scientific, technological, biosecurity, human capacity, time, and institutional capacity needs. During the planning phase, the GIP-IL underestimated the institutional, logistical, technological, financial, and human capacity constraints African countries currently face in animal genomics. Given the project’s timeline and revised scope, achievement of all six objectives before the end of the current five-year project is unlikely. The project management also underestimated the time needed to implement an effective distribution plan. EQ1.2: The project has contributed high-quality research in relevant technical topics that address validation of resistance to NDV infection in African indigenous chickens. It has also facilitated peer￾reviewed publications and dissemination of research results in both the U.S. and Africa. This research will be useful to individuals and institutions working in similar research areas. Remaining research activities include, but are not limited to, finalizing challenge trials, executing molecular assays, conducting additional statistical analyses, identifying additional SNPs, conducting genetic selection of NDV resistance, and validating disease resistance. There is some disagreement among research team members about whether natural or controlled exposure should be used to complete challenge trials in Objective 2. EQ1.3: The validation, breeding, and distribution stages in Objectives 5 and 6 require a significant amount of time and investment beyond the current project timeline. Chicken production processes and systems differ between Tanzania and Ghana. The poultry value chain in Ghana is more commercial then in Tanzania, where it is decentralized among many small producers. Because of these differences, continued value chain research is required in each country so appropriate distribution systems that reach small-scale hatcheries and farmers can be developed. Gender-specific research in value chain analysis has not been conducted. Once it is, it will contribute to developing gender-focused distribution plans. 16 EQ1.4: Project results documentation includes KPIs, management, research metrics, and documenting the number of publications in scientific peer-reviewed journals. The project’s dissemination activities include scientific publications, and promotional and outreach products that offer more information about project activities. Document review and KIIs reveal that extensive research metrics (both output and outcome level) capturing key project activities are documented, but the ET only received this information on an ad hoc basis. A significant number of research team members were not aware of all research metrics collected, but they did confirm that documentation takes place. EQ1.5: Management and research stakeholders are confident that continued work will enable GIP-IL to meet its overall goal of developing chicken breeds with enhanced NDV and heat resistance. GIP-IL’s team members identified several project improvements that would contribute to reaching its goals. These include extending the research and technical activities to other African countries and their specific poultry ecotypes, investing in BSL3 facilities and other permanent research institutes in Africa, and providing longer-term breeding support for sustained results. Some researchers suggested exploring partnerships with private sector groups and other research projects with similar goals. RECOMMENDATIONS EQ1.1: • The project management team should built-in flexibility when conducting the assessment of the existing gaps in terms of human, institutional, technical, and logistical capacities and needs to address the project’s remaining research and capacity building objectives. Participatory planning processes and analyses with project stakeholders, partners, and beneficiaries would allow development of a more realistic and comprehensive activity implementation schedule. • More time should be allowed for poultry genetic breeding. The differences in the timeline for successful genetic breeding in poultry systems compared to crop systems should be considered when developing poultry research timelines. While crop system breeding typically takes a shorter timeframe to complete, the poultry breeding system takes longer and as such USAID funding for GIP-IL should have allowed more time (than five years). Crop system breeding can be successfully completed in two to four years; however, genetic breeding in poultry systems takes between five and eight years to complete (from discovery phase to distribution and implementation). These differences are due to the number of offspring required, and the ability to generate a population with genetic diversity needed to detect gene controlling traits, reproductive efficiency, and disease traits. EQ1.2: • USAID/BFS should provide additional resources before the end of the current phase to bolster continued research activities. The GIP-IL research program should focus on completing the initial phase of work without worrying about support. This will allow for careful design and use of the Low-Density SNP panel, as described in Objectives 3 and 4 (see Table 2). To attain significant, long-term improvement of NDV and heat stress resistance, future GIP-IL efforts must focus on the sustainable development of the validated and improved breeding stocks. This will likely take an additional three to five years to establish and will require at least two or three generations of chicken stocks, and maybe more, to show sustainable effects. • Additional traits recorded by the research team, such as growth rate, could be included in selection criteria of SNP markers. African chicken populations are highly diverse and show a low degree of inbreeding. Including SNP markers not linked to genomic regions associated with traits under selection will help to monitor and maintain genetic diversity in these populations. The SNP array should have other criteria beyond selection for improved 17 NDV resistance. To this end, the European Union-supported project IMAGE should be consulted for information about its initiatives to develop a low density, multi-species genotyping platform. • GIP-IL should continue capacity building efforts, including training and research facility enhancement. • All relevant stakeholders should continue to examine the advantages and disadvantages of natural exposure and controlled exposure for completing Objective 2. Appropriate controls for potential confounds should be implemented. EQ1.3: • Additional participatory research approaches are necessary. Poultry value chain research should encourage increased participation by farmers, women, and other community stakeholders, which will help researchers better understand value chain activities in specific country contexts and research relevance. To ensure sustainability and prevent dilution of the stock of chickens with enhanced NDV and heat resistance, GIP-IL should develop a plan to enhance long-term support for breeders. Gender considerations should be addressed in value chain analyses. EQ1.4: • USAID/BFS will need to consider providing additional funding and extending the project timeline. In order to meet GIP-IL’s overall goal of developing chickens with enhanced NDV and heat resistance, additional funding and time resources should be provided. Resources should be dedicated to developing an overall project performance monitoring plan that includes preference indicators and targets to monitor and report progress. Management and research team members should review information sharing protocol to ensure all participants are up-to￾date on the research metrics collected and recorded. EQ1.5: • GIP-IL should explore opportunities for partnerships with private sector groups and other research projects with similar goals to improve poultry production. 4.2 EQ 2: CAPACITY BUILDING FINDINGS 4.2.1 EQ 2.1: Within the context of the GIP-IL research program, how well has UCD identified and addressed the academic and technical capacity needs of the host country stakeholders? According to respondents in KIIs and FDGs, building researcher and student capacity has been an important project contribution. The GIP-IL project has established collaborative linkages between U.S.- based universities and the universities in Ghana and Tanzania, and these linkages address capacity needs of country-level researchers. Through this collaboration, research teams are working and learning together and, if funding is available, there is potential for post-GIP-IL collaboration. In interviews with research stakeholders affiliated with both U.S. and country-level universities, the ET found that the GIP￾IL has been responsive to the technical and longer-term capacity needs of UoG and SUA. Trainings in both countries were conducted by graduate students from ISU and UCD, and included biosecurity and lab assays, sample collection, and data analysis. There was also training for veterinary students given jointly by university staff and U.S. project staff on biosecurity, laboratory safety, poultry disease diagnosis, and biosecurity training. Important, tangible outcomes, like enhanced knowledge for lab technicians grew from the trainings. At UoG, scientists were sent to UCD and ISU for animal trials and 18 lab assays. At SUA, a local USAID Borlaug Leadership Enhancement in Agriculture Program fellow was sent to UCD and ISU. Specifically, the GIP-IL project has built academic and technical capacities through the following approaches: • Co-authorship of scientific publications; • Industry shows and conference presentations; • Training new, local researchers in relevant fields such as genetics and virology; • Transfer of knowledge and research methodologies between partners; and • Other capacity building activities dealing with technology and facilities. Table 5, below, illustrates the breakdown of men and women staff members among RAC and training participants, according to GIP-IL records. Table 5. Training Participation by Sex GIP-IL Training Participation by Sex Women Men Total Target % of Women Actual % of Women Staff 3 11 14 50% 21.4% RAC 3 7 10 50% 30% Trainees 58 171 229 33% 25% Graduate Students 5 5 10 33% 50% CDBP Participants NA NA NA NA NA Total 21 39 60 Researchers in KIIs and FGDs felt that the project made substantial improvements in strengthening experimental skills. During KIIs, researchers were asked about how GIP-IL identified and addressed academic and technical capacities of African universities’ veterinary and animal science departments and other host country stakeholders. Illustrative comments from three researchers include the following: “…help[ed] develop scientific capacities and methods, as well, as analyzing the data, [which] are substantial improvements…” “…greatest change is the improved research facilities, training of staff, and training of students to be independent researchers…” "Personnel improvements have been great, they [students] are more comfortable in conducting experiments and developing projects in the scientific method…” GIP-IL’s capacity building in each area includes: Infrastructure Research Capacity Building. The project renovated several breeding and animal trial facilities at both SUA and UoG, which had been abandoned. Renovations upgraded the facilities to make them safer and more conducive to implementing project research activities, and included adapting physical spaces, protecting them against rodent invasion, installing running water and electricity, and implementing biosecurity measures. When asked how the program could better serve and provide capacity needs for host country stakeholders, FGD participants suggested improving research facilities’ infrastructure. One FGD participant said, "…more facility construction (hatchery, petri facility, larger challenge building, Center for Applied Animal Genomics, Power source, Computing power, Data storage, and Back up)." 19 In addition, the FGD participants suggested improving education and training (this was the most selected category); hiring someone to work as the project management/coordinator in the financial area; and improving data analysis training. However, GIP-IL’s renovated lab facilities were to be shared with other projects, and this created logistic and biosecurity problems. As a student stated when questioned about research support, “the lab was used by many people, there was lots of contamination…they had to work hard to get the contamination under control.” During the course of implementation, the project faced other infrastructural challenges as well—mostly related to chronic power failures that disrupt lab activities and communications. Joint research protocol and tool development. UCD, in collaboration with its partner universities, wrote research protocols for poultry husbandry and biosecurity, poultry and challenge trials, sample collection and processing, and laboratory and data analyses. To optimize protocol application on-site, and to provide team members with the experience of running trials and conducting laboratory analyses, the UCD and ISU researchers provided training to African partners. Provision of short-term training. Short-term training sessions included hands-on laboratory teaching practicums given by UCD and ISU staff (see Tables 6). Training sessions increased human capacity in the U.S., Ghana and Tanzania, and generated data and results for Objectives 1 and 2. In the U.S., these training activities provided students with advanced genomic sciences and animal study skills, and international development experience. African researchers had the chance to continue their project work and the capacity to confidently embark on future research projects without external support. Without adequate training opportunities in Africa, proposed experiments and data collection would not have occurred. Table 6. GIP-IL Short-term Training Workshops on Technical Topics Name Subject Date Place Attendee s # Women # (Percent) Animal trials Bird collection, program procedures, bird handling, and tear collection January 25-26 2014 Tanzania 3 0 (0%) Animal trials Bird collection, program procedures, bird handling, and tear collection January 29-30 2014 Ghana 9 3 (33%) Animal trial and Lab Techniques Tear collection, blood collection, necropsy techniques, ELISA, RNA Isolation, qPCR, sample collection protocol February 2014 USA 7 1 (14%) Animal trial and Lab Techniques Tear collection, blood collection, necropsy techniques, ELISA, RNA Isolation, qPCR, sample collection protocol February 2014 USA 3 2 (67%) Sample collection and molecular techniques Tear collection, RNA isolation, qPCR January 11-16, 2015 Ghana 10 2 (20%) Sample collection and laboratory techniques Sample collection and Lab techniques February 2015 Tanzania 5 3 (60%) Biosecurity Biosecurity Principles and Procedures February 27, 2015 Ghana 23 8 (35%) 20 Name Subject Date Place Attendee s # Women # (Percent) Breeding Innovations in Poultry Breeding February 2015 US 3 0 (0%) Biosafety Biosafety level 2 procedures, laboratory techniques (RNA isolation, qPCR, ELISA, and advanced genomic technologies) March 2015 US 3 0 0%) Biosecurity Biosecurity Principles and Procedures March 13-14 2015 Tanzania 23 8 (35%) Laboratory safety training Safety in the Laboratory March 27-28 2015 Tanzania 5 3 (60%) Laboratory safety training Safety in the Laboratory April 10-11, 2015 Ghana 9 4 (44%) Molecular techniques ELISA, Viral RNA and qPCR May 16 - 21, 2015 Ghana 7 2 (29%) Annual meeting training Poultry biosecurity and laboratory techniques September 2015 Tanzania 9 1 11% Molecular techniques ELISA, Viral RNA and qPCR December 14- 19, 2015 Ghana 3 2 (67%) Molecular techniques ELISA, Viral RNA and qPCR December 7- 11, 2015 Tanzania 4 1 25% Poultry Disease and Diagnostics Necropsy and Clinical signs of Newcastle disease February 4-5 2016 & April 9,10 2016 Tanzania 3 0 (0%) Biosecurity Training to Vet Students Biosecurity in Poultry Facilities May 1, 2016 Tanzania 59 7 12%) PCR for velogenic NDV strains Diagnosis of velogenic NDV strains by RT-qPCR March 14-15 2016 Tanzania 7 3 (43%) Poultry Genetics Commercial Poultry Genetics March 2016 US 3 2 (67%) GenSel Analysis GenSel Analysis March 2016 US 1 0 (0%) Annual meeting training Poultry biosecurity and laboratory techniques October 2016 Ghana 10 2 (20%) Molecular techniques Refine lab research protocols and increase workflow December 16, 2016 Virtual with all teams 5 1 (20%) Molecular techniques Refine lab research protocols and increase workflow January 19, 2017 Virtual with all teams 5 1 (20%) Molecular techniques Refine lab research protocols and increase workflow February 16,2017 Virtual with all teams 5 1 (20%) Molecular techniques Refine lab research protocols and increase workflow March 10,2017 Virtual with all teams 5 1 (20%) Post-Graduate Training. The GIP-IL also addressed the longer-term academic capacity of researchers by providing financial and academic support for post-graduate school students. During the evaluation, 10 students were receiving post-graduate degrees from the four universities (the total number of post-graduate students for the entire length of the project to date was not available). 21 Interviews with both instructors and students provided feedback on the project’s training program support, which is summarized below: • GIP-IL has provided training of substantial quantity, quality, and relevance to UoG and SUA; • Training courses that focused on conducting research, laboratory analysis, outreach, assays, and data collection offer students the most in terms of capacity building activities; • GIP-IL has introduced new technologies like tear collection, qPCR, and specific ELISA applications to staff and students in both UoG and SUA; • Logistical obstacles aside, trainees appreciated the hands on/practical training approach UCD and ISU brought to the African universities (although some suggest that this can be improved); and • Joint development of lab protocols is a useful and effective training method that encourages cross-learning (African and U.S. students). Training activities purposely sought to include more women, although participation greatly varied among events (see Table 7). Out of the 10 PhD students GIP-IL supports, six are female. Table 7 summarizes post-graduate students’ training experiences, and describes specific topics addressing technical, communications, and leadership capacity. Table 7. Summary of Post-Graduate Student Training Activities Student Sex University Degree Major Home Country Training Topics Provided by GIP-IL F UCD PhD Microbiology USA Running animal trials including blood, tear and tissue collections and data analysis, communication skills, and leadership development M UCD PhD Genetics and genomics USA Running animal trials including blood, tear and tissue collections and data analysis, molecular assays, statistical analysis, genomics, bioinformatics analysis, international capacity development, communication skills, and leadership development F ISU PhD Genetics USA Running animal trials including blood, tear and tissue collections and data analysis, molecular assays, statistical analysis, genomics, bioinformatics analysis, international capacity development, communication skills, and leadership development F ISU PhD Genetics USA Running animal trials including blood, tear and tissue collections and data analysis, molecular assays, statistical analysis, genomics, bioinformatics analysis, international capacity development, communication skills, and leadership development M SUA PhD Animal Science Tanzania Running animal trials including blood, tear and tissue collections and data analysis, molecular assays, statistical analysis, genomics, bioinformatics analysis, international capacity development, communication skills, and leadership development 22 Student Sex University Degree Major Home Country Training Topics Provided by GIP-IL F UOG PhD Animal Breeding and Genetics Ghana Running animal trials including blood, tear and tissue collections and data analysis, molecular assays, statistical analysis, genomics, bioinformatics analysis, communication skills, and leadership development M UOG MPhil Animal Breeding and Genetics Ghana Running animal trials including blood, tear and tissue collections and data analysis, molecular assays, statistical analysis, and communication skills F UOG Research Associate training Ghana Statistical analysis, genomics, bioinformatics analysis, communication skills, and leadership development M University Ghana MPhil Microbiology and Immunology Ghana Running animal trials including blood, tear and tissue collections and data analysis, molecular assays, statistical analysis, and communication skills F UCD MS International Development USA International agricultural development, survey administration, communication skills, and leadership development Publications and Professional Conference Presentations. By collaborating on publishing research, the experience is shared among U.S. and local researchers, and this helps disseminate results to industry groups and stakeholders interested in novel approaches. GIP-IL has made efforts to ensure that its work is widely distributed, particularly at relevant genomics conferences, international symposiums, technical roundtables, association’s congresses, and other meetings. Participation in these events, and inclusion of project research results in animal industry reports, expanded the project’s dissemination and outreach activities. The extent and effectiveness of outreach strategies on the end￾user are unclear, given the absence of monitoring data. Poultry Shows and Annual Conferences. One way that UCD has built capacity is through providing opportunities to interact and share research results at various workshops and conferences. These include attendance at animal industry events such as poultry shows and the project’s annual conferences in 2014, 2015, 2016 and 2017. About the annual conference, one student attendee said it was important because they “…met the other participants at the annual meeting, and that interaction was very positive, there was a clear multidisciplinary approach and a great collaboration.” 4.2.2 EQ 2.2: How could the project better serve and provide for the capacity needs of those stakeholders? Thirty (30) percent of survey respondents (9 out of 30) identified strengthening the training opportunities for African researchers as a critical factor to improving overall research capacity. Interviews with students revealed that the training programs should include: more opportunities for African students to get additional training at U.S. research sites; onsite guidance from U.S. researchers for African collaborators; and a standard data pipeline analysis to analyze information across universities. GIP-IL should also include courses for undergraduate and postgraduate curricula, especially ones that focus on providing practical experiences. This will facilitate introducing GIP-IL technologies to more African students. Trainees should be involved in specific research activities and in a more comprehensive way that allows them to get the big picture of the overall GIP-IL’s research and project management experience. The expertise of the U.S. research teams should be better utilized to improve African 23 research capabilities and good practices in research. Creating and implementing more continuing education programs would achieve this. For example, the student respondents recommended more focus on practical skills training such as data analysis and suggested peer learning approaches. The following illustrative, direct statements are from three students, who offered these opinions in response to a question about opportunities to improve their involvement and enhance their capacity: “More training is great. More time to practice…the students don’t know how to do data analysis.” “…so, it has been good. But I wish we had more time for hands-on work on analysis.” “More student to student communication would have been great to have…allow for peer learning.” Participants in the FGDs with the management and research teams also agreed that investing and building the local capacity of young researchers in Africa will sustainably improve research quality. According to one participant "…teach them the good points about the new chicken, teach how to carry out research like at UCD; increase human capacity so that work can be continued…" However, many constraints limit the project’s ability to provide the capacity needed to local stakeholders. The respondents in KIIs and FGDs offered several suggestions about how to address these constraints (discussed earlier). These responses are summarized below: • Enhance funding and timely release of funds; • A need for stable power, and better internet service to facilitate communications, project coordination, training activities, animal and lab studies, and data analysis; • Have a well-equipped genomic analysis facility and bio-secure lab available for sample and data analysis, and disease research; • Have facilities for challenging birds with velogenic (local) viral strains and permission to carry out such a challenge; • Have more technical staff and research students on the team; and • Have an extension of the current project activities so that Objectives 5 and 6 can be completed. A majority of GIP-IL management and research teams seemed optimistic that longer-term collaborative research and outreach relationships among UCD, ISU, SUA, and UoG have potential to continue beyond external support. They are aware that if additional funding sources cannot be identified, lack of funds for research and outreach will continue to be a severe constraint to the animal industry. CONCLUSIONS Training staff and students, whether at a short-term or a postgraduate-level, is by far the most important capacity building activity the GIP-IL has provided to the African universities. However, although training has been a substantial capacity building activity, there are specific areas, such as training students on genomics analysis and peer learning, that can be strengthened to have more significant impact. Infrastructure improvements have strengthened SUA’s and UoG’s institutional commitment to the program, but additional planning and resources are necessary to ensure that facilities function as envisioned. Presentations, participation in forums, and contributions to trade report publications are notable GIP-IL activities that provide opportunities to share research results. Documenting research results in scientific journals should be emphasized more heavily. Finally, sustaining the long-term collaborative research and outreach relationships among UCD, ISU, UoG, and SUA needs to be addressed, as it is highly dependent on securing external funding. RECOMMENDATIONS In addition to the discussion under Section 4.2.2, further recommendations on academic and technical capacity building include: 24 • UoG and SUA should expand GIP-IL’s training efforts by increasing awareness of the project’s activities and research results amongst existing students and university partners. This should be done by developing a knowledge management strategy that identifies effective approaches, formats, and tools to disseminate and share project results. It should include existing activities such as attendance at conferences and workshops, but also explore opportunities using social media platforms. The strategy needs to be resourced to ensure available funding for priority dissemination activities. • More emphasis should be placed on the job training of students in advanced techniques such as tear collection, FTA cards for samples, qPCR, specific ELISA applications, and related research practices. This would ensure that students have access to the most up-to-date tools and protocols. UCD and ISU should conduct on the job training on biostatistics and data analysis so that African staff and students can make sense of the results of the project lab works. GIP-IL should document its training, research, management experiences, and best practices in a manual for use as a training-the-trainer tool that universities can use for training on-the-ground beneficiaries. This should be a living document with continual updates that reference new tools and protocols. The manual should have a virtual online learning component that allows students to document research successes, mistakes, achievements, and experiences with various techniques. The online site can link to each university website, which would let current and future participants in similar programs learn from detailed accounts and experiences in research and genomics. • GIP-IL should include an editor as part of its staff to help support scientists across the project, including African universities, to publish and encourage publishing more rigorous scientific reports. Despite funding concerns, the partner universities should make a systematic effort to sow institutional conditions for long-term collaboration in other scientific projects and activities beyond and above the GIP-IL. These efforts must involve top-level university officers. 4.3 EQ 3: INSTITUTIONAL COLLABORATIONS FINDINGS 4.3.1 EQ 3.1: How has UCD performed in establishing productive collaborations with host country governmental and academic institutions, local NGOs, and other relevant USAID programs in target countries? The GIP-IL’s ME, UCD, has established partnerships and collaborations with a wide range of partners in the U.S., Europe, and Africa, including public entities, academic institutions (in the U.S. and in host countries), trade associations, private companies, and other related projects in Africa. As a result, these collaborations created opportunities to enhance activities and better achieve GIP-IL’s project objectives. Some of the productive collaborations UCD established during GIP-IL’s implementation include working with the following institutions: Collaborations with Africa based Institutions • Established collaboration with the Accra Veterinary Laboratory (AVL) at the Ministry of Food and Agriculture in Ghana. AVL assists with lentogenic NDV replication and titration and validating results of PCR for velogenic NDV strain in natural exposure field trials. • Networked with investigators working on the USDA Poultry AMPLIFIES and Ghana Poultry Project (GPP) in Ghana to explore synergies between the projects and shared platforms for poultry breeding and distribution. AMPLIFIES is building downstream value chain capacity for Ghana’s agriculture industry, specifically strengthening market linkages for locally produced maize and soy commodities utilized in feed and poultry production. The GPP is increasing agricultural productivity in the poultry value chain through capacity building, 25 improving input markets, and increasing trade of poultry products by improving product quality, market linkages, and production efficiency. • Collaborated with Tanzania Poultry Breeders Association to identify and partner with breeders interested in maintaining the genome of local indigenous chickens and breeding local ecotypes with enhanced NDV resistance. • Initiated collaboration with the Africa Chicken Genetic Gains Project (ACGG) through co￾mentorship of a Borlaug Fellow. The ACGG project is interested in working with local chicken ecotypes under assessment by the GIP-IL program. • Established collaboration with the ACIAR Nkuku4U project and received external funding to enhance capacities for developing an integrated NDV prevention program in Tanzania, and to validate an SNP for genetic selection of enhanced NDV resistance and improved NDV vaccine efficacy in indigenous African chickens. • Established collaboration with the Nelson Mandela Institute in Tanzania. Students and faculty from the Institute have expressed interest in collaborating on NDV resistance in local ecotypes by utilizing molecular assay protocols implemented by the GIP-IL program. • In cooperation with Hy-Line International scientists, GIP-IL is working to identify genes and genetic markers associated with resistance to NDV and heat stress in the Hy-Line Brown line. This is their commercial layer line, which is widely distributed throughout North and West Africa. Hy-Line International, a global leader in poultry layer breeding and genetics, provided more than 1,200 pedigreed Hy-Line Brown layers and genotype, 13 genes associated with immune response and heat stress, including 223 SNPs in 1,189 Hy-Line Brown birds with phenotypes of virus titer, antibody response and body weight. Collaborations with U.S.-based Partners • In the U.S., UCD has collaborated with investigators at Pennsylvania State University on NDV resistance in Fayoumi and Leghorn lines. • Explored collaboration with the Livestock Systems Innovation Lab. Collaborations with Europe-based Partners • UCD and the University of Sydney (USdy) - entered into a partnership agreement in early 2017. Under the terms of the agreement, both institutions contribute up to $20,000 Australian Dollars (approximately $15,000 USD) annually for two years, to support cooperative research projects and activities under the UCD Seed Feed. GIP-IL was one of the recipients of funds for the joint project called: “Tackling Food Insecurity in Tanzania: Harnessing Genomics and Vaccinology to Control Newcastle Disease in African Chickens.” • GIP-IL is exploring working with the University of Cambridge in typing village chickens from Africa for MHC (Major Histocompatibility Complex) alleles, using a next-generation sequencing (NGS) method, with the eventual goal of determining which MHC haplotypes confer resistance to NDV and response to vaccines in African ecotype chickens. Findings from the online survey show that 10 out of 30 respondents agreed, and 20 out of 30 respondents strongly agreed, that not only has GIP-IL established productive institutional collaborations with universities, non-governmental organizations (NGOs), and the governments of Ghana and Tanzania, but also that the collaborations have enhanced project performance. These partnerships are productive and mutually beneficial, but the extent to which the project has worked with local NGOs and other partners remains unclear. One project partner KII participant said that at present “there is a disconnect between the research being conducted and the understanding of the farmer.” During KII interviews, institutional partners provided additional information regarding UCD’s partnership collaborations. They noted that: 26 • The right partners have been targeted for a mutually beneficial purpose. UCD has been a leader in establishing relationships and setting up unprecedented networks between UoG and SUA. • According to the GIP-IL project team, the USAID Mission in Tanzania is less keen on the GIP-IL project activities. According to the research team, USAID/Tanzania staff have conveyed that there are not enough staff or resources available for livestock projects. This contrasts with the USAID Ghana Mission, which is both interested and actively engaged in GIP-IL’s activities. • In both Ghana and Tanzania, GIP-IL has collaborated with the Ministries of Agriculture in joint activities such as conducting lab assays, training, and identifying local ecotypes in the countries’ provinces. These collaborations are relevant to the project’s intervention areas and helpful to its activities and success. • GIP-IL has pursued institutional collaborations mostly oriented to project-to project opportunities, instead of project-to-institution opportunities. Targeted projects include some supported by USAID (these include PREDICT, RESPOND, IDENTIFY, and PREVENT projects). • Collaborations are not just formal written partnership agreements but are developed by GIP-IL, which is constantly seeking synergy opportunities with key individuals and institutions. Three individuals from institutional partners interviewed during the KIIs said that they identified the following benefits, as a result of the above collaborations: “Was a good opportunity to test the project’s hypotheses…” “This was a more controlled experiment, so the data was good to have.” “Increased opportunities for SUA to collaborate with others.” One respondent from the management team also recognized benefits that institutional collaborations have had in strengthening GIP-IL project results: “GIP-IL benefits from having a network of collaborators that will enhance chances to actually reaching the farmers at the end of the day and impacting them…” CONCLUSIONS • Collaborations are relevant to the project’s subject, helpful to its activities, and synchronize with the pace of the GIP-IL research program progress. • The workload needed to pursue and keep collaborations is in proportion to the human and operational resources available to GIP-IL. • Collaborations are not only formal written partnership agreements but work together in constant exploration for opportunities for synergy with key individuals and institutions. RECOMMENDATIONS • GIP-IL should begin planning to strengthen existing partnerships and establish new institutional collaborations that will be necessary to implement and enhance activities under the project’s Objectives 5 and 6. These new partnerships should also consider the project’s overall sustainability. • GIP-IL must review the status and relevance of its collaborations with those institutions identified in the Cooperative Agreement, namely Heifer International, Helen Keller International, Save the Children, the Aga Khan Foundation. 4.3.2 EQ 3.2: How could the GIP-IL improve its institutional collaborations? Some potential collaborations are not entirely developed since the project is still in its initial phase. There were partnerships with NGOs, for example, that have not been fully explored. The role of donors such as the Gates Foundation, regarding project work in gender and youth is unclear. 27 Research stakeholders have suggested that GIP-IL should focus on establishing more active research and outreach initiatives for collaboration opportunities with projects such as the African Chicken Genetic Gains -ACCG project. This is an Africa-wide collaboration led by the International Livestock Research Institute (ILRI), which tests and makes available high-producing, farmer-preferred genotypes to increase smallholder chicken productivity in Africa. Similarly, the GIP-IL project aims to improve chicken genetics and the delivery of adapted chickens to support poverty reduction, productivity growth, increased household animal protein intake, and the empowerment of women farmers in rural communities. Likewise, GIP-IL research stakeholders suggest pursuing collaboration with other USAID projects working in similar program areas and sectors. Overall, institutional collaborations have increased at the pace of the GIP-IL research program. While core research activities still need to be initiated and completed, the project should begin to strategize about establishing institutional collaborations necessary for implementing Objectives 5 and 6 and address the project’s future sustainability. When asked about how to improve institutional collaborations, three individuals interviewed during the KIIs had the following statements: “We were approached because we have the facilities to conduct challenge experiments…also, we have access to the breeds used in Africa.” “… [We have] access to the strains of the virus that are circulating in Africa.” “The project has provided a forum for closer ties to poultry breeders.” 4.4 EQ 4: MANAGEMENT In assessing this component, management and implementation refer to the entire continuum of the management system that involves all teams, including UCD, ISU, UoG, SUA, and USAID/BFS, and therefore not just UCD as the ME. FINDINGS 4.4.1 EQ 4.1: To what extent has the GIP-IL management team effectively implemented and managed the GIP-IL’s respective research and training activities in Ghana and Tanzania? The ET identified common themes across KII and FGD responses on the management team’s efficiency in implementing the GIP-IL program and its coordination efforts among partners. Findings for this EQ are organized by these themes, which include training and capacity building, scientific research, contract management, personnel management/coordination, infrastructure, partner coordination, and monitoring and evaluation (M&E). Training and Capacity Building Training took various forms and targeted multiple stakeholders—university researchers, scientists, and students. KII participants felt that capacity building efforts to expand GIP-IL’s program work and training conducted for the Ghanaian and Tanzanian partners were well received and appreciated. This is illustrated by an Africa partner participant, who said, “The training has been huge, this has not occurred at any other African universities, it would not have happened without the GIP-IL project.” Most KII respondents commented that they were satisfied with UCD selecting staff from both African universities who had the expertise to work on the program. They alluded to learning from one another, and one African partner participant said, “UCD picked people who had relevant expertise and expanded the expertise and capacities of both locations." 28 The management team consists of a Program Director, Principal Investigator (PI), Program Manager and one representative from each of five 5research partners (see organizational chart in Annex 7). The team meets weekly by conference call to discuss research program decisions, progress, and results. Specifically, the Program Director and the Program Manager are responsible for activity coordination, planning, work-plan development, program reporting, overall program M&E, coordination with USAID and other stakeholders, and representation of the program. The management team interacts with the RAC on key research issues. For the most part, respondents were positive about the contributions the RAC makes to the project's performance. They also thought the RAC brought in more openness and accountability, and provided good technical input to the project, including introducing the natural challenge. One management team member put it this way, “The RAC has been good. Valuable group of experts worldwide. Very critical…. I have learned a lot from them." Contract Management and Sustainability. The ET learned that contractual procedures at UCD are complicated. Putting subcontracts in place for the initial collaboration between UCD and the African GIP-IL partners took longer than expected and caused implementation delays. Also, there was a consensus among all stakeholders that the GIP-IL program would not be able to complete the six research objectives laid out in their scope of work within the 5-year project time frame. However, there were no indications of project plan activities beyond the current phase. For instance, CBDP development is rightly scheduled for the fifth year. However, the plan is logically and programmatically dependent on the validation of disease resistance to NDV in indigenous chicken (Objective 4). In terms of sustainability, the GIP-IL program has initiated several collaborations with other stakeholders and, according to the management team, continually seeks additional sources of funding to sustain the program beyond its period of performance. Nonetheless, concrete plans for the program’s future or a “second phase” remain vague. Infrastructure Support. The two African university partners commented that they saw significant improvements in infrastructure with the implementation of the program, which has positively contributed to research and training work. One African partner said there had been “considerable improvement, four renovated buildings, others rebuilt, and a hatchery.” However, some important infrastructure-related challenges still impact the efficiency of the management team. These include connectivity issues, unreliable electricity, power outages, and the resulting difficulty with communication technology such as Skype. In the words of another African partner, "In Africa, the biggest constraint is the variability of internet access and power outages.” The infrastructure challenges, and suggestions for their improvement are discussed in Section 4.4.2, below, and in the recommendation section. Open Data Management Plan. The project has developed and implemented an Open Data Management Plan, which addresses all policies and procedures regarding program-generated datasets. The plan then records the contact person and institution (UCD, ISU, UOG, or SUA) for each dataset. It also addresses data privacy and use restrictions; data processing steps, including quality assessment and quality control processes; final data deliverable; timeline; data repository (USAID DDL or NCBI); the responsible party (the contact person at UCD or ISU); and the target submission date. Research Management and Coordination. Lack of project administrative coordinators at SUA and UoG compel the PIs to spend time on administrative tasks, running errands, and fixing logistic problems, which hinder the progress of the research program. To enhance research coordination, the management team took the following steps: • The research team collaborated with USDA-ARS Southeast Regional Poultry Research Lab (SEPRL) and received supplemental funding (pending) from USAID/BFS to expand access to needed research facilities. 5 UC Davis, Iowa State University, University of Delaware, University of Ghana, and Sokoine University of Agriculture, Tanzania, [Third Annual Report for Feed the Future Innovation Lab for Genomics to Improve Poultry]. 29 • Backup power systems were installed to avoid future losses of active NDV stock that would impose further delays. The project conducts ongoing maintenance of power systems. • A weekly conference call occurs between U.S. teams and Africa teams to coordinate activities and address potential issues in a timely and efficient manner. KIIs and FGDs identified the following as factors that negatively impacted project management efficiency: 1. The slow and complicated bureaucratic procedures within UCD that were required before collaborating with ISU, UoG and SUA. 2. According to African partners, the project funds are insufficient, and funds are released too slowly. 3. Insufficient critical assets such as research and breeding facilities, bio-security systems, working internet access, video conferencing and computing systems, and insufficient scientific experience and training in areas like advanced molecular assays. 4. Lack of project administrative coordinators at SUA and UoG compel the PIs to spend time on administrative tasks, running errands, and fixing logistic problems, which hinder the progress of the research program. 5. According to African partners, the ability to immediately deploy improvements or corrections in training, supplies, and project management is insufficient (see 4.4.2). 4.4.2 EQ 3.2: In what ways could implementation and management be improved? Given that this evaluation sub-question naturally alludes to recommendations to improve implementation and management, part of it is answered in the recommendations section below. The findings result from KIIs with both U.S. and African partners and from FGDs. Personnel Management/Coordination: When asked about factors affecting management, KII respondents elaborated on the lack of project management, and administrative and financial personnel in Africa. For instance, the absence of a project coordinator in Ghana (UoG) meant that scientists had to take on additional logistical tasks, which took time away from their research. In the words of one African partner, “One [good] thing would be to have a research assistant from the beginning…we needed a manager for the SUA, an administrator for the financial aspects of the project…” Another African partner noted that they “wish we had a project coordinator. I have to run too many errands.” Infrastructure: KII respondents mentioned power infrastructure problems as a factor that caused consequent delays to the research teams. Respondents in KIIs discussed putting backup equipment in place in case primary equipment fails. The ongoing issue of poor infrastructure and the effect it has on communication technology such as Skype was raised several times as an issue to be addressed. Scientific Research: To strengthen technical capacity, the African researchers recommended the GIP-IL management staff consider including an internationally-known expert on NDV in the project. Likewise, budgetary and logistic arrangements should be made so that RAC members have greater involvement in research activities beyond bi-annual video conferences. FGD participants said that with additional funding they would like more training opportunities; more resources for data analysis; and better facilities for scientific testing and research of poultry production. Also, they suggested that the program could improve implementation and better reach its objectives by building supply chains, dealing with electricity shortage, procuring supplies more efficiently, building time and resources for logistics and administration, and establishing relationships with additional partners. This is summed up in the words of one FDG respondent who said, “Build additional time for logistics and administration; continuously consult partner in contingency management; build supply chains, locate sources of supplies (for instance chickens to be used as breeders)." Establish a Forward-looking Research Strategy. According to RAC members, the project does not have a clear, forward-looking strategy for achieving program objectives, and is focused on a linear 30 production of scientific outputs. The project is conducting velogenic NDV challenge studies in Year 3, which were not included in the original proposal because appropriate facilities were lacking. These studies are a necessary change to the research strategy but caused a delay in completing project activities. There is a concern that similar setbacks may occur in the future and should be built into the research strategy moving forward. 4.4.3 EQ 4.3: How effectively has the GIP-IL management team managed and coordinated amongst the partners, Missions, host country governments, and local researchers? The GIP-IL has facilitated coordination among stakeholders and partners through industry poultry shows and annual conferences in 2014, 2015, 2016, and 2017. During these activities, partners, researchers, students, and host country governments had the opportunity to share and discuss research, promote learning, and build capacity. These activities enabled coordination among local researchers and, to an extent, with host country government—technical staff—partners. Participating African partners in FGDs had the following to say: “The grad students always participated in the workshops and communicating with farmers with the poultry show and Ag show.” “This project has expanded to include ISU and additional partners with others at UOG, created a new network to gain from various talents and showcasing the student networks." In FGDs, African partners noted that RAC’s participation at the annual conference was unnecessary. As one FGD participant noted, “Not everyone who is there may need to be there, pros and cons each way, including people from each of the partners …, the RAC may also not need to be there and may be useful to scale back.” In addition to promoting collaboration and coordination through events like the poultry show and the annual conferences—despite local governments perhaps not being ready for such collaboration—the GIP-IL management team has worked to establish relationships and set up networks between the two African university partners for cross-fertilization. The SUA staff explained that they have been able to share resources and information, which contributed to a more successful project. One SUA staff member said, “Before this project relationship involved few people, it has expanded our network of collaboration between the universities.” Another mentioned that “UCD has been the leader in connecting with UOG and SUA.” Through KIIs with the African partners, it seems apparent that USAID Mission staff has been more engaged in Ghana than Tanzania. As the management team pointed out, "So, it is different between the two countries, the Tanzania Mission from the onset was not interested in livestock projects, they are short staffed and did not have the time to invest. Ghana is different and is very interested in getting updates on the project and are keenly interested in being updated all along the way.” 4.4.4 EQ 4.4: In what ways could these partnerships be improved, or new partners engaged? As discussed in Section 4.4.3, the GIP IL management team has facilitated a positive working relationship between SUA and UoG, resulting in enhanced learning and collaboration between the two universities. When asked in KIIs if there are ways these partnerships could be improved, or if new partnerships could be formed, the management team and African partners recommended that UCD reach out to the private sector and NGOs more. Respondents also mentioned that organizing more joint activities for the partners would be beneficial. One African partner noted a “need to design more activities that all of the partners can do together.” 31 All institutional partner respondents said that the number of partners currently engaged in the GIP-IL research is enough, and that building on more partnerships might be too much to manage and coordinate. One responded was very clear on this, saying “Many partners get things too complicated.” The FGDs elicited similar suggestions about how to increase the involvement of various types of stakeholders. Ideas included involving vet lab and animal research institute organizations to gather chickens in several African countries, involving ministry partners and local farmers (and, in Ghana, the Ghana National Association of Poultry Farmer) and involving international partners in training. This was clearly stated by one African partner, “Involve international partners in North America and Europe for finance and training, UoG computing systems, and data services to other projects.” CONCLUSIONS The conclusions are based on the findings of the four sub-questions and are organized by major topics, as appropriate. Training and Capacity Building • The program has successfully built capacity at SUA and UoG in multiple ways, including training in advanced genetics sampling and laboratory procedures, general faculty and staff training; and training graduate students in research methods. • Training increased human capacity in the U.S., Ghana, and Tanzania, and generated data and results for Objectives 1 and 2. In the U.S., these training activities provided students with advanced genomic science skills, animal study skills, and international development experience. Personnel Management/Coordination • There is a lack of dedicated administrative and financial staff at UoG and SUA, which has caused extra work for research staff. • Among the factors hindering project management’s effectiveness, some are beyond the control of the GIP-IL management team. These factors include university administrative requirements and the limitations due to infrastructure and capacity (human, technical, and institutional) at the partner institutions in Africa. Contract Management • At the time of this evaluation, GIP-IL has not attained expected results, and a third of the expected results will not be completed by the end of the project. That is, the program will not be able to complete the six research objectives as laid out in their scope of work within the five￾year project time frame. • GIP-IL’s management has not adequately addressed or planned for project sustainability. Infrastructure • The current infrastructure, including poor electricity, has caused technology and communication challenges that impede efficient communication among and between the U.S. and African partners. Monitoring and Evaluation • There seems to be a discrepancy in M&E knowledge and capacity among the management team. The project uses research metrics to monitor progress but there is no specific M&E plan and system in place to track implementation progress. Partner Coordination • The difference between Ghana's involvement and Tanzania's lesser involvement in project activities could be because Tanzania will no longer be a Feed the Future-focused country. 32 RECOMMENDATIONS Training and Capacity Building • GIP-IL should continue providing ongoing training and capacity building to research staff, students, and faculty in the U.S., Tanzania, and Ghana. • GIP-IL could enhance its training and capacity building programs further by tightly tying training to research long-term goals and planning, and by organizing them in advance. • The project has put systems in place to document the number of women in training activities, but these numbers vary by event. The project should develop a specific gender strategy and implementation plan that goes beyond disaggregating training data by sex. Personnel Management/Coordination • GIP-IL’s management entails a highly collaborative approach and proactive communication, coordination, and problem-solving. However, management must improve its strategic planning abilities so that research results are better used to inform policy and plans at national and local levels. • The GIP-IL team should closely coordinate with UCD, ISU, UoG, and SUA management teams to develop administrative processes that are agile and go beyond typical procurement rules. New methods should account for the realities of projects in international settings, and recommendations might be useful for the project’s continuous management processes. • The project should provide the principal African research investigators with project coordinators, so they do not spend too much time addressing management, operational, and logistic problems, or running errands that come from scarce financial resources, all of which leave them less time to focus on research. Contract Management and Sustainability • The four universities led by UCD should seek clarification from USAID/BFS regarding the programmatic consequences of Tanzania no longer being a Feed the Future target country 6. • The GIP-IL management team should develop a sustainability plan for the project. The ET suggests that sustainability can be achieved not only through USAID funds for the “second phase,” but also through, among other things, support from the leaders of the four universities, future availability of personnel adequately trained in genomics, data analysis and biostatistics, and a product (validated chickens) that is accepted by the small poultry producers and consumers. Infrastructure • GIP-IL must find ways to solve the most critical supply, logistic, operational, and resource limitations—inadequate power, limited Internet access, lack of video conferencing, and paltry project-dedicated research facilities. 6 https://www.feedthefuture.gov/countries, retrieved on 10.20.2017 33 4.5 EQ 5: PROGRAM FUTURE FINDINGS 4.5.1 EQ 5.1: What specific technical areas, if any, within the GIP-IL’s research program merit new or continuing research investment? 4.5.2 EQ 5.2: What topics are promising for further investment in animal genomics for improved livestock? These two EQs are similar and interconnected and will be answered as one question. Stakeholders indicated in survey responses and KIIs that GIP-IL research teams must stay on track to completing its original objectives. However, several ideas for new research investments that may help better meet current program objectives were frequently suggested during data collection: • Deepen and prioritize the study of genomics and heat stress, and different combinations of NDV + stress factor (cold/drought/nutrition). • Undertake research that includes more than just the SNP markers identified in the project, considering that they were discovered in inbred lines and commercial layer lines, which are distinct from local African populations. • At least three management team members and five RAC members suggested adding chicken ecotypes, especially isolated island ecotypes. • Five RAC members emphasized the need for a sustainable breeding program in African universities with local ecotypes. This would allow for follow-up for several generations of the improved chickens. • Four management team members prioritized investment in research facilities, including a permanent poultry research institute in Africa. • All stakeholders supported sustained and increased investment in human capacity development—including training students and technology transfer. Additionally, GIP-IL offered extensions for continued research investment that would contribute to overarching food security goals: • Two management team and two RAC members recommended using genomics to improve productivity, either measured by growth rate or egg production. • Four management team members suggested extending genomics to other poultry species, such as guinea fowl and duck. • Ten management team members suggested extending genomics to other diseases and pathogens related to chicken mortality and food safety. These include bronchitis, Avian influenza, fowl pox, Marek’s disease, salmonella, listeria, and other diseases currently masked by NDV. • Five management team and two RAC members point to chicken’s nutrition as an important research topic for improving disease resistance and production. • Three RAC members recognized the relevance of climate change to food security issues. They recommended that future research take into account adaptations to climate change. If continued investment in poultry genomics is secured, research stakeholders recommend that instead of investing in another independent five-year project like GIP-IL, a long-term commitment should be made to completing the current program activities requiring more time to achieve initial research objectives before translation to the field. Although it is not within the scope of this kind of program, application of this research could mean establishing a poultry research center in Africa staffed with professionals capable of undertaking local research initiatives on their own which would make a more permanent contribution to research capacity in poultry genomics research. As one management team member shared, “[I would] like to have a top-notch poultry research institute… that can do all kinds of poultry 34 research…Priority research areas would be to improve survival against NDV and the diseases that are masked— IBV, Gomboro, and fowl pox.” The SUA research team participated in Tanzania Breeders Association meetings to reach breeders able to maintain indigenous poultry breeds after the program ends. The UoG team communicated with USAID’s Ghana Mission, the USDA AMPLIFIES Ghana project, and the Ghana Poultry Project to discuss opportunities for facilitating poultry breeding and distribution. There is a general concern that poultry genomics in Africa will not be able to advance further without addressing the lack of adequate research infrastructure. Properly equipped and funded genomics research labs in Ghana and Tanzania are necessary and should include sufficient research facilities, hardware, and supplies. Securing basic assets like reliable power, internet access, and data processing capabilities are equally important. Infrastructure limitations and the uncertain financial future beyond GIP-IL are major obstacles to expanded poultry genomics research initiatives. Some stakeholders not directly involved in project design or implementation doubt the overall effectiveness of the genomics approach to NDV prevention. Three respondents shared this position, and one stated, “I wonder if instead of spending all this money in genomics, perhaps a better traditional selection could be as effective. Commercial companies do invest in genomics approaches, but even so the cost/benefit relation of traditional selection might be higher.” According to annual reports, GIP-IL has initiated work with local individuals and organizations to plan distribution of sustained technology and improved genetic stocks. The project has also planned future activities. To educate local poultry breeders and smallholder farmers about the increased value of chickens with innate NDV and heat stress resistance, additional project resources are required. These activities need to be fully implemented for research to be relevant to local farmers and for successful, sustainable distribution of improved chicken breeds. 4.5.3 EQ 5.3: In what ways could the GIP-IL management, research, capacity development, and/or institutional collaborations be improved to better achieve its objectives? The recommendations this EQ calls for will be provided in the recommendations section below. 4.5.4 EQ 5.4: Is there a need for any social or economic research in the next phase? GIP-IL management understands that a CBDP requires detailed knowledge of the social and economic realities in areas where improved chicken would be distributed for regular production. Based on this understanding, the following initiatives were proposed during FGDs with management team members: • Validation with smallholder family poultry producers in various regions of Ghana and Tanzania of the CBDP models Wilson’s study (2016) identified. This will ensure that the innovation is usable and accessible to farmers, and they are interested in the improved ecotypes; • A market-based study of the economic feasibility of distributing specific chicken ecotypes based on the CDBP models that Wilson identified; • A value chain analysis for various non-commercial chicken ecotypes and their delivery to producers and, eventually, to consumers—within the context of the national and regional livestock sector; • Gender considerations in value chain analyses that include capacity development for women, business, social justice, poverty alleviation, and food security; and • Analysis of public policy frameworks for indigenous chicken production in Ghana and Tanzania. 35 CONCLUSIONS Regarding research on animal genomics, stakeholders are optimistic about the project’s ultimate results and their applications for other factors in the poultry value chain. However, they are aware that the future depends on the university support of the program and, more importantly, on USAID’s continuous engagement and funding. Several of the recommended topics for new and continued research depend on sustained investment. Among a small number of external stakeholders, however, there is a lingering concern about the relevance and cost/benefit justification of a genomics approach to fighting NDV. RECOMMENDATIONS Project partners - UCD, ISU, UoG, SUA and USAID/BFS - should address the program’s future within the policy framework of the U.S. Global Food Security Act of 2016, and its Global Food Security Strategy. The USAID Mission in Ghana has been engaged with GIP-IL, and the country is one of the selected 12 Feed the Future target countries. This suggests that GIP-IL must reinforce its ties with the Mission to make sure that poultry genomics fits within the country’s Feed the Future priorities. Moreover, UCD, ISU, UoG, SUA, and USAID should work together on a GIP-IL’s sustainability plan that includes budget considerations, capacity building, institutional strengthening, research capabilities, and the future human, logistic, and technological needs. Likewise, the ET recommends that UCD, ISU, UoG, and SUA start exploring options for long-term collaboration in the absence of USAID funding. The GIP-IL should: • Provide its management team with some program management training. • Improve the ability to quickly stock lab supplies. • Provide a program administrative coordinator in both African universities. • Have a long-term program leader embedded in the African universities. • Allow for flexibility and time when initiating research activities in Africa and other regions with limited institutional research capacities and other logistical difficulties. • Have a formally written strategy for data management and analysis that can be shared and used to train the research team and students. • Reflect and improve the definition and use of metrics. This is important both for project management and research. • Explore options for longer-term funding for the African universities and implement a timely release of funds. • Have a dedicated laboratory space for the GIP-IL team in the African universities to reduce time-consuming and unnecessary movement during sample laboratory analysis. • Include more practical experiments in selecting improved ecotypes in Africa. • Validate results with velogenic NDV challenge and include additional ecotypes to assess greater genetic diversity. • Develop SNPs relevant to a broader scope of African chicken eco-types. • Include chicken breeds besides Kuchi, Ching'wekwe, and Morogoro available in Tanzania since all local ecotypes are similarly attacked by NDV. • Identify differences between the various African ecotypes in genes or regions of the genome identified as having enhanced resistance to NDV. • Use natural challenge to NDV during trials. For instance, inoculate susceptible birds and then co-locate those with trial birds. • Seek a more systematic and lasting transfer of cutting-edge technologies from the U.S. to Tanzania, both in poultry genomics and in bioinformatics and data analysis. 36 4.6 CROSS-CUTTING ISSUES FINDINGS 4.6.1 EQ 6.1: Gender Gender integration in Feed the Future program is an essential objective for all USAID supported programs. It should be integrated from the design stage and threaded through program implementation, M&E activities, and reporting activities. The ET reviewed and analyzed the GIP-IL documents, which show that gender has been integrated into program design, implementation, and reporting. Design: From the project design phase, GIP-IL has actively pursued women as target beneficiaries. During the design phase, GIP-IL analyzed the role women play in poultry farming in Africa and found that women play the role of caretakers for local breeds of poultry. GIP-IL incorporated these findings into training, workshops, and throughout the life of the project. One KII commented that “by focusing on local breeds [of chickens] from the beginning that opens the door for women….” Another noted that “there was an aim to be as gender sensitive as possible… but the deliverables will target mostly women [as] they are the day to day keepers of the chicken.” Another KII supporting GIP-IL’s gender integration strategy at the design stage of project commented: [GIP-IL] encouraged woman participation in the workshops. It was purposeful that women were included in the invite list, for student training, they screen the students to be sure that they are qualified to join the project, [and with a] larger pool of students, we can have better selection of women, [displaying] the active action to include women as student participants in the project. Implementation: The GIP-IL has successfully improved women’s participation in genomics research programs in Africa. During implementation, women’s roles in decision making, leadership, and higher education were also enhanced. A student respondent during KIIs stated that although “student involvement was evaluated regarding women, [and] everyone had a positive experience… it could be better in Ghana and Tanzania, because women are not encouraged to participate in STEM fields.” In interviews, the research team shared this sentiment, stating that women were more active in smaller group discussions where they felt comfortable giving their feedback, inputs, and opinions. However, management team noted that although they recruited women as staff, technicians, and data analysts, it was overall “difficult to find women that were qualified.” Although the respondents did not feel that the qualification component played a role in women’s access to deliverables, it accounts for the lack of African women leadership in GIP-IL. Reporting: The GIP-IL project was committed to reporting on gender disaggregation data across all program activities and indicators. This is illustrated in Table 6, and includes laboratory staff, workshop and training participants, graduate students, post-docs, advisory committee members, and stakeholders who were consulted to ensure participation of women and overall gender balance. This is adopted through the following gender-related performance measures: • The research program had to ensure that at least half of the program field staff and graduate students were women; • At least a third of participants of UCD training activities must be women; • UCD will emphasize outreach to women breeders, farmers, poultry suppliers, and marketers; and • At least a third of the participants in the Breeding Implementation and Distribution Workshop should be women. These indicators successfully promoted gender involvement throughout the project. They held the staff accountable for keeping the inclusion of women participants a high priority. Key informants on the 37 research team stated that women were involved in workshops and training in Morogoro in 2015, although many of them reported that women were mostly involved as trainees, not trainers or researchers. A researcher from the management team stated that this was “one of the easier project[s] to work with due to the recognition of women's roles in the smallholder farms." A number of KII respondents provided suggestions on improving the gender balance and involving women in all components of the project. One respondent suggested, “they [project management] need to work on how to distribute the products, [in order] to include women and keep them in the process of distribution.” Another thought that "keeping an eye on the final product so that it would not exclude women at the end" was important, and a third respondent suggested a “need to include, or look at, when they (management) have workshops that they have them at times when they are available to women.” Overall, interview respondents felt that implementing small changes and making even simple efforts could encourage more women to take an active leadership role in poultry genomics and research. Key informants agreed that women need to be involved in the entire process of production and that training should be offered around women’s schedules so that more women can fully benefit from the project. The quantitative survey results, however, find a contrast with perceptions. As one female survey respondent pointed out: “Promoting gender equality is a major strength of this project. Not only will improving poultry genetics improve the quality of life for many families (women and children in particular), but the research leading up to it has also been beneficial for women. I think it was very beneficial for U.S. based female students to visit Africa and train other graduate students there.” Female participation at UoG and SUA training on biosecurity and laboratory techniques range from 20 - 60 percent. Thirty-one (31) team members, including 7 women, participated in biosecurity training at UoG in 2015. Twenty-three (23) team members, including 8 women, participated in biosecurity training at SUA in 2015. At SUA, five team members, including three women, were trained in laboratory safety and other laboratory techniques. These training experiences will add to the long-term human capacity of female researchers in the field. One key informant from the management team expressed the need “to add a special focus to developing programming for women because of their roles in chicken farming.” 4.6.2 EQ 6.2: Nutrition Because they are not a direct subject of project activities, nutrition results cannot be evaluated. The one research objective of the project that is directly tied to chicken improvements (in terms of availability and health) has not been achieved. Nutrition is a project design consideration because it is essential to the development problem GIP-IL and other Feed the Future projects address. GIP-IL aims to improve nutrition and overall health of children and families in Ghana and Tanzania through genomics. Strengthening chicken production at household level will increase food security, nutritional status, and overall livelihoods. The project recognizes that across rural Africa, women and children raise poultry for both household consumption and to sell eggs and birds. Sales provide increased income for the women and their households. The Cooperative Agreement also notes that micronutrient malnutrition, which causes stunting and reduced cognitive development, is common among the rural poor. Eggs are a good source of critical micronutrients and protein for pregnant and lactating women, and for children in their first 1,000 days of life. Focusing directly on the nutritional value of eggs and chickens—and the clear, beneficial impact their consumption would have on communities—could provide the project a precise, straightforward link to nutrition. Although nutrition was not a direct project intervention, results from KIIs found that it was a relevant topic. Eight KII respondents felt that the project activity design and implementation did address nutrition to a certain extent. A respondent from the management team pointed to project activities focused on 38 improving production of live birds for better access to eggs and meat so that women and children have more of each to sell and consume. Findings from respondents suggest that future project components that focus on nutrition would bring the project from theoretical to practical. For example, student respondents indicate that the nutritional value of chickens and eggs need to be evaluated to create a usable research output. One student said, “there is a disconnect between the research work and the farmer in the field, co-ops [and] broader reach presentations are vital… including the other dimensions of nutrition, and other factors related to disease resistance.” Likewise, many researchers agreed that it is critical to include nutrition in any further investments in animal genomics for improved livestock and poultry production in developing countries. 4.6.3 EQ 6.3: Environment and Biosecurity In terms of environmental consideration, the project aimed to mitigate any potential environmental impact by completing the required Initial Environmental Examination (IEE) and by developing the Environmental Mitigation and Monitoring Plan. USAID approved the IEE before the project began, and management personnel in the field found the plan helpful because it offered guidelines for addressing environmental expectations. The KII respondents from the management team said that they included environmental considerations and protocols in project from design through implementation. Specifically, one key informant said “[there were] impacts from biosecurity, [which] minimize the impact on local poultry production… the plan is reviewed and approved by USAID, [and] acknowledges the risks to the environment, mitigation and monitoring methods are in place.” However, a second pointed out that “... biosecurity of the natural challenge was difficult because the challenge took place 50km away in an isolated location… [and participants] took all disposable plastics and birds were burned.” But a third suggested that these issues were considered and planned for, “We have a biosecurity protocol. We use the litter [and] manure in other parts of the farm.” The Program Director, Program Manager, and members of the U.S. research team inspected the laboratories and poultry facilities at UoG and SUA to ensure they are maintained according to, and comply with, designated biosafety levels and host institution biosafety guidelines. In February and March of 2015, the GIP-IL trained 31 people in biosecurity at UoG, and 23 people at SUA. Biosecurity plans were drafted after the workshop and were adopted at the research facilities. Each new program staff member received biosecurity training, and all incoming staff members will receive the training. In site visits to laboratories and poultry facilities, the Program Director and Program Manager also ensured that relevant World Organization for Animal Health - OIE biosafety and biosecurity measures are being implemented, and that they are following the good laboratory practices outlined in Section IV of the Center for Disease and Control Prevention (CDC) Biosafety in Microbiological and Biomedical Laboratories. The Program Director and Program Manager made site visits to Ghana on October 28, 2016, and in Tanzania on March 17th, 2016. They found that laboratories and poultry facilities at UoG and SUA were practicing good laboratory practices, were in compliance with institutional guidelines, and were implementing relevant OIE biosafety and biosecurity measures. GIP-IL did experience some setbacks due to insufficient biosecurity in African facilities. Management personnel stated that biosecurity issues caused a significant “change-of-course” moment for the project. One management member expressed that “environmental policies require USAID projects to use the same biosecurity requirements that exist in the US, [and] they were unable to complete this due to high cost.” The ET visited some facilities and concluded that additional steps should be taken to ensure that all animal labs should better adhere to biosecurity requirements that including isolating chicken houses from the campus and roads; using a guestbook to register visitors; using a pass-over; using a pass- 39 through; requiring hairnets; providing instruction on proper handling of lab equipment; requiring handwashing; enforcing change of footwear; and requiring the use of provided shirt and pants. RECOMMENDATIONS • In decision-making, commitment, and leadership, women’s role in GIP-IL is significant and reflects the diverse composition of the project’s research and management team. Likewise, the GIP-IL demonstrated continuous compliance with USAID gender policies in data management, monitoring, and gender participation reporting. Moving forward, the main gender-related recommendation is for the GIP-IL to continue outreach to women in Africa, especially through production and implementation of the chicken breeding and distribution programs. • Although GIP-IL does not directly support nutrition as a project activity, if integrated fully, nutrition is a factor that will yield significant and relevant results. • The ET recommends enhanced biosecurity procedures in avian facilities and suggests adding, as an initial project cost, expenses related to implementing these procedures. 40 ANNEXES 41 ANNEX 1. SCOPE OF WORK AND EXPRESSION OF INTEREST 42 EXTERNAL EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR GENOMICS TO IMPROVE POULTRY: SCOPE OF WORK A.l BACKGROUND The Feed the Future Innovation Lab for Genomics to Improve Poultry (The Poultry Genomics Innovation Lab) was awarded to the University of California at Davis in 2013. One success over the past three years of the Poultry Genomics IL research activities is the development of a collaborative network of international researchers and stakeholders dedicated to research and development of improved poultry, with particular emphasis on enhanced resistance to Newcastle Disease virus (NDV) and heat stress. The overall goal of the Poultry Genomics IL is to apply advanced genetics and genomic approaches to sustainably enhance innate resistance to Newcastle disease and heat stress to improve production of poultry in developing countries. The outputs of this program will be new essential and fundamental knowledge of the genomics of resistance to NDV, and the application of this knowledge to develop ecotypes of African chickens with greater innate resistance to NDV. In the process of achieving these outputs, program outcomes will include the development of physical and human infrastructure in Africa, including training of African students and scientists, both on-site in Africa and in the US, in the essential skills that will enable the African partners to sustain and disseminate the results of this program. Research & Administration The Poultry Genomics IL’s research program is divided into six main objectives: Objective 1: Identify genes and/or genetic markers associated with NDV resistance and heat stress in three unique chicken populations in the US using advanced genetic genomic technologies. Objective 2: Identify genes and/or genetic markers associated with NDV resistance in African indigenous chicken ecotypes using whole genome high-density SNP panel. Objective 3: Develop an economical, low-density SNP panel for genetic selection of disease resistance to NDV and to heat stress by integrating the discoveries from objectives 1 and 2. Objective 4: Validate genetic enhancement of resistance to NDV infection in African indigenous chicken ecotypes using economic low-density SNP panel selection. Objective 5: Develop validated, improved breeding stock with enhanced NDV resistance in Africa. Objective 6: Develop a chicken breeding and distribution plan (CBDP). Objective 1 took place at the University of California at Davis (UC Davis) and Iowa State University (ISU). Objective 2 took place at the University of Ghana (UoG) and Sokoine University of Agriculture (SUA) with technical assistance provided by UC Davis and ISU. Objective 3 is currently under development by UC Davis and ISU with technical input from SUA and UoG. Objective 4, 5, and 6 and taking place at UoG and SUA with technical assistance from UC Davis and ISU. Across the Poultry Genomics IL program and six objectives, cross-cutting issues are addressed such as gender, capacity building, institutional strengthening, and outreach, as well as USAID policies. The Poultry Genomics Innovation Lab is administered through its Management Entity (ME) at the University of California at Davis under the direction of its Program Director and leadership team. In addition, the Poultry Genomics Innovation Lab maintains open dialogue with its external advisory board to discuss emerging developments and technical issues. 43 A.2 PURPOSE OF THE ASSESSMENT The purpose of this final performance evaluation of the Poultry Genomics IL is to provide robust evidenced-based findings, conclusions and recommendations that BFS can use to inform the development of new investments in animal genomics research for development, improve ongoing activity effectiveness, and better achieve intended outcomes. To accomplish this, the evaluation will assess implementation and progress toward the six research objectives of the Poultry Genomics IL, including progress in technology development and capacity building accomplishments. Furthermore, since the Poultry Genomics IL will be completing its first five-year phase in September 2018, the Evaluation Team is expected to make recommendations to USAID to help improve effectiveness and efficiency in implementation and management and discuss potential changes during consideration of a second (four to five year) phase of the Poultry Genomics IL. A.3 OVERALL METHODOLOGY The team will design and execute an assessment, including secondary data review, key informant interviews, focus group discussions and discourse analyses. During contract implementation, the USAID Contracting Officer Representative (COR) will provide an initial list of USAIDs’ implementing partners and their contact information. A.4 DUTIES AND RESPONSIBILITIES • This assignment will include meetings and consultations with the USAID Mission staff, counterparts, including implementing partners and other relevant actors from civil society, international community, and academia. • Identify Key Informants Interviews (KII) through skype or telephone calls. Prepare draft questions, devise a schedule and carry out the interviews. Interview USAID Missions, counterparts, donors, religious leaders, community elders, youth, government officials and other relevant actors with insight. • The evaluation team will finalize the details of the design in the Evaluation Plan. A mixed methods approach rooted in robust qualitative and quantitative data collection and analysis is recommended for this performance evaluation. Due to the geographically dispersed nature of the intervention, a creative approach involving case studies can be used. The evaluation shall generate rigorous findings and relevant actionable results for all stakeholders. • If requested, de-brief USAID Mission staffs. • Submit draft and final reports and revise final reports. A.5 DELIVERABLES Scheduled Tasks and Deliverables: Scheduled tasks and deliverables for the assessment are as follows in chronological order: A. A Performance Evaluation Concept Paper: No more than five pages in length using a template provided or approved by BFS. The Concept Paper shall be submitted within 30 days after the work assignment Kick-Off Meeting. B. Performance Evaluation Protocol: Outlines evaluation research questions to be answered, indicators/variables to be measured, sampling methods, data collection and analysis methods, data quality assessment methods, the process or format for recording and reporting results, and budget. The template used for the Protocol shall be provided or approved by BFS. The completed Protocol shall be provided within 60 days of the work assignment 44 Kick-Off Meeting. C. International Review Board (IRB): Documented Official Approval from all relevant institutional review boards and from host country institutions if primary data are to be collected. This Documented Official Approval shall be provided no later than 90 working days after the work assignment Kick-Off Meeting. D. Fieldwork Completion: The team will travel to the identified countries and conduct the evaluation. E. Draft Assessment Report Complete: Reports shall be no more than 50 pages in length and shall follow a template provided or approved by BFS. Data collection instruments and supervisor/enumerator manuals (if applicable) shall be provided as annexes in the Reports. All Reports must meet USAID Criteria to Ensure the Quality of the Evaluation Report (see Attachment J.2) and follow USAID’s How to Note on preparing evaluation reports. F. Final Evaluation Report Complete: The Final Report will be submitted to the (COR) in electronic format within five (5) working days following receipt of comments from the draft report, if any, from the COR. The Final Report shall include an Executive Summary and will not exceed fifty (50) pages (excluding appendices). The Final Report will be prepared in English and shall follow USAID branding procedures. The final report will include but not be limited to the incorporation of COR comments, debriefing comments from USAID, and any other Food Security Service Center reports and information collected throughout the assessment. The final report will take into consideration comments and clarifications received from debriefings. All due dates will be identified during the work assignment kick-off meeting for the evaluation. All quantitative data collected by the team will be provided in an electronic file in easily readable format (Microsoft Office) to the COR. The data will be organized and fully documented for use by those not thoroughly familiar with the assessment. USAID will retain ownership of the survey and all datasets developed. A.6 LEVEL OF EFFORT AND TIMELINE The evaluation should start in April, while the total period of performance of Task Order should not exceed 32 days. Tasks Time (Work Days) Desk Review/ Concept Note 3 days Evaluation Protocol & Tool Development 6 days Data Collection 13 days Data Analysis 2 days Draft Report 5 days Revision of Final Report 3 days Total LOE 32 days A.7 TEAM COMPOSITION The team shall be composed of three international subject matter experts, with Mr. Luis Bernal as the Evaluation Team Leader. The following team members are Dr. Christopher Ashwell, Technical Team Member; and Dr. Paul Miller, Technical Team Member. A.8 EXPRESSION OF INTEREST 45 External Evaluation of the Feed the Future Innovation Lab for Genomics to Improve Poultry: Expression of Interest I. BACKGROUND INFORMATION A) Identifying Information 1. Project/Activity Title: Feed the Future Innovation Lab for Genomics to Improve Poultry 2. Award Number: AID-OAA-A-13-00080 3. Award Dates: 9/27/13 – 9/26/18 4. Project/Activity Funding: $5,999,989.75 5. Implementing Organization: University of California at Davis 6. Project/Activity COR/AOR: Lindsay Parish B) Development Context 1. Opportunity Addressed by the Project/Activity Being Evaluated General Background The overarching goal of the Feed the Future (FTF) Initiative is to sustainably reduce global poverty and hunger. Limited land availability and variable climatic conditions exacerbate the problem and increase the need for sustainable yet swift solutions. In order to address the challenge, scientific and technological innovations need to be developed that increase agricultural productivity with minimal environmental impact. In support of the broader FTF goal, the FTF Global Food Security Research portfolio aims to advance the productivity frontier, to transform key production systems, and to enhance nutrition and food safety through agriculture. The Feed the Future Innovation Labs (formerly called CRSPs) were created under Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage US land grant and other eligible universities to address the needs of developing nations while also contributing to US food security and agricultural development. In 2000, Title XII was reauthorized, enabling these US university research efforts to continue “to achieve the mutual goals among nations of ensuring food security, human health, agricultural growth, trade expansion, and the wise and sustainable use of natural resources.” The FTF Innovation Labs draw on the expertise of top US universities and developing country institutions and are an integral part of the Feed the Future Food Security Innovation Center. The Innovation Labs serve to implement the Feed the Future Global Hunger and Food Security Research Strategy and were established in response to two key recommendations from a 2012 Board for International Food and Agricultural Development (BIFAD) commissioned CRSP review7: • To develop an overarching and coordinated strategy for engaging U.S. universities in agriculture and food security research and human and institutional capacity development that includes the CRSPs as a central component; and 7 http://transition.usaid.gov/our_work/agriculture/bifad/BIFADREVIEW_CRSP_August2012.pdf 46 • To leverage the impact of CRSP investments by strengthening links across universities, U.S. government, global programs, foundations, and other donors. The launch of the Food Security Innovation Center in 2012 enabled USAID to manage its research, policy, and capacity-strengthening portfolio through the following seven thematic areas rather than by institutional home: • Program for Research on Climate Resilient Cereals • Program for Research on Legume Productivity • Program for Advanced Approaches to Combat Pests and Diseases • Program for Research on Nutritious and Safe Foods • Program for Markets and Policy Research and Support • Program for Sustainable Intensification • Program for Human and Institutional Capacity Development The Program for Advance Approaches to Combat Pests and Disease, which includes the Poultry Genomics and other Innovation Labs, aims to use advanced molecular tools to create new animal vaccines and develop crops and animals that are resistant to major pests and diseases. Description of the Poultry Genomics Innovation Lab The Feed the Future Innovation Lab for Genomics to Improve Poultry (The Poultry Genomics Innovation Lab) was awarded to the University of California at Davis in 2013. One success over the past three years of the Poultry Genomics IL research activities is the development of a collaborative network of international researchers and stakeholders dedicated to research and development of improved poultry, with particular emphasis on enhanced resistance to Newcastle Disease virus (NDV) and heat stress. Objective The overall goal of the Poultry Genomics IL is to apply advanced genetics and genomic approaches to sustainably enhance innate resistance to Newcastle disease and heat stress to improve production of poultry in developing countries. The outputs of this program will be new essential and fundamental knowledge of the genomics of resistance to NDV, and the application of this knowledge to develop ecotypes of African chickens with greater innate resistance to NDV. In the process of achieving these outputs, program outcomes will include the development of physical and human infrastructure in Africa, including training of African students and scientists, both on-site in Africa and in the US, in the essential skills that will enable the African partners to sustain and disseminate the results of this program. Research & Administration The Poultry Genomics IL’s research program is divided into six main objectives: Objective 1: Identify genes and/or genetic markers associated with NDV resistance and heat stress in three unique chicken populations in the US using advanced genetic genomic technologies. Objective 2: Identify genes and/or genetic markers associated with NDV resistance in African indigenous chicken ecotypes using whole genome high-density SNP panel. Objective 3: Develop an economical, low-density SNP panel for genetic selection of disease resistance to NDV and to heat stress by integrating the discoveries from objectives 1 and 2. 47 Objective 4: Validate genetic enhancement of resistance to NDV infection in African indigenous chicken ecotypes using economic low-density SNP panel selection. Objective 5: Develop validated, improved breeding stock with enhanced NDV resistance in Africa. Objective 6: Develop a chicken breeding and distribution plan (CBDP). Objective 1 took place at the University of California at Davis (UC Davis) and Iowa State University (ISU). Objective 2 took place at the University of Ghana (UoG) and Sokoine University of Agriculture (SUA) with technical assistance provided by UC Davis and ISU. Objective 3 is currently under development by UC Davis and ISU with technical input from SUA and UoG. Objective 4, 5, and 6 and taking place at UoG and SUA with technical assistance from UC Davis and ISU. Across the Poultry Genomics IL program and six objectives, cross-cutting issues are addressed such as gender, capacity building, institutional strengthening, and outreach, as well as USAID policies. The Poultry Genomics Innovation Lab is administered through its Management Entity (ME) at the University of California at Davis under the direction of its Program Director and leadership team. In addition, the Poultry Genomics Innovation Lab maintains open dialogue with its external advisory board to discuss emerging developments and technical issues. 2. Target Areas and Groups Key U.S. Partners The University of California at Davis partners with three universities to lead projects in Africa: Iowa State University, University of Ghana, and Sokoine University of Agriculture (Tanzania). Geographic Reach The Poultry Genomics Innovation Lab overseas activities have been carried out in Ghana and Tanzania since 2013. C) Documents The evaluation team will review a wide variety of documents provided by BFS, the ME, and by the sub￾award partners. The primary documents and types of documents that will be provided are listed below. ● Technical Application proposal ● Poultry Genomics Innovation Lab Annual Reports ● Implementation Plans and Annual Work Plans ● Program publications including success stories, research publications, and HICD outputs ● Other Poultry Genomics Innovation Lab website materials and information (http://gip.ucdavis.edu/) ● Feed the Future Research Strategy (feedthefuture.gov) II. EVALUATION RATIONALE A) Evaluation Purpose The purpose of this final performance evaluation of the Poultry Genomics IL is to provide robust evidenced-based findings, conclusions and recommendations that BFS can use to inform the development of new investments in animal genomics research for development, improve ongoing activity effectiveness, and better achieve intended outcomes. To accomplish this, the evaluation will assess implementation and progress toward the six research objectives of the Poultry Genomics IL, including progress in technology development and capacity building accomplishments. 48 Furthermore, since the Poultry Genomics IL will be completing its first five-year phase in September 2018, the Evaluation Team is expected to make recommendations to USAID to help improve effectiveness and efficiency in implementation and management and discuss potential changes during consideration of a second (four to five year) phase of the Poultry Genomics IL. B) Audience and Intended Uses These results will be used by USAID/BFS Program for Advance Approaches to Combat Pests and Disease, Missions, and implementing partners and universities. The evaluation results may also be applicable to others who are involved in designing research for development programs. C) Evaluation Questions This performance evaluation will provide USAID and the Management Entity (ME) with constructive feedback on the program management, research program, training program and institutional capacity collaboration of the Poultry Genomics Innovation Lab. The evaluation will cover the entire project, including the management, implementation, research program and capacity development components using an evidenced-based and data-driven approach. Program Management 1. To what extent has the Management Entity effectively implemented and managed the Poultry Genomics Innovation Lab’s respective research and training activities in Ghana and Tanzania? In what ways could implementation and management be improved? 2. How effectively has the ME managed and coordinated amongst the partners, Missions, host country government partners, and local researchers? In what ways could these partners be improved, or new partners engaged? Research and Training Program 1. To what extent did the Poultry Genomics IL make progress towards developing poultry with enhanced NDV resistance? What other research and work remains to fully complete each research objective? What work remains to get a usable research output (poultry with enhanced resistance to Newcastle Disease virus) to meet the goal of the Poultry Genomics IL? 2. To what extent did the Poultry Genomics IL produce quality research outputs using appropriate metrics (e.g., peer reviewed publications)? What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for FTF countries) moving forward? 3. Within the context of the Poultry Genomics IL’s research program, how well has the ME identified and addressed academic and technical capacity needs of host country stakeholders? How could the project better serve and provide for the capacity needs of these stakeholders? Institutional Capacity Collaboration 1. How has the ME performed in establishing productive collaborations with host country governmental and academic institutions, local NGOs, and other relevant USAID programs in the target countries? How could the ME improve its institutional collaborations? Program Future 3. In what ways could the Poultry Genomics Innovation Lab’s management, research (i.e., design, implementation, stakeholder involvement), capacity development programs, and/or institutional capacity collaboration be improved to better achieve objectives? 4. What specific technical areas, if any, within the Poultry Genomics IL’s research program merit new or continuing research investment (e.g., continued genetic characterization of different 49 African ecotypes for enhanced NDV resistance, testing poultry breeding programs in country to disseminate/scale improved poultry to smallholder farmers)? What topics are areas are promising for further investment in animal genomics for improved livestock and poultry production in developing countries? III. TIMEFRAME & TRAVEL A) Timeframe The evaluation will begin o/a March 15, 2017 and conclude o/a August 31, 2017. B) Travel The Evaluation team will need to travel internationally to carry out the evaluation and complete the scope of work. Two two-day site visits to Poultry Genomics IL in Ghana and Tanzania are recommended for data collection and observation. IV. DELIVERABLES A) Please see Section F. 7 of the PEEL Task Order contract V. TEAM COMPOSITION The evaluation team will be composed of 3 members: one evaluation team leader and two technical team members. The technical qualifications of Evaluation Team should include primary technical areas such as animal genomics, along with specific experience carrying out rigorous performance evaluations. The pool of potential team members with international animal genomics development experience and relevant expertise outlined is small. The candidate(s) will also have: a) the capacity to conduct independent program evaluation; b) a thorough understanding of research methodology; c) the ability to analyze issues and formulate concrete recommendations orally and in writing; d) be available to travel and meet the timelines for completion of the evaluation; and e) not have any conflicts of interest. Each member is requested to submit a CV that demonstrates relevant experience in technical, evaluation and management skills. Disciplines of all members: The following technical skills should be present among the team as a whole: organizational development, evaluation methodology, quantitative and qualitative evaluation techniques, livestock/poultry production systems, animal genomics, and poultry diseases. Team members must have the expertise necessary to evaluate the many sub-award projects, to assess the different technical and programmatic components of the project, including research, management and implementation, capacity development, and partnerships and institutional collaboration, address the evaluation questions rigorously, and collect, triangulate and analyze primary and secondary data in a robust manner to generated unbiased and evidenced-based findings. Evaluation Team Lead (LOE: 32 days) One senior-level evaluator with a minimum of 15 years of experience in evaluation methodology (including mixed methods evaluation), quantitative and qualitative data collection and analysis and experience carrying out evaluations of international development projects, preferably with a background in genetics and livestock research for development and technology dissemination. The preferred candidate will be familiar with USAID (or other donor) funded programs. Technical members x 2 (LOE: 27 days each): One mid- and one senior-level technical experts with a minimum of ten years of experience managing and/or evaluating multifaceted international development research and/or university-based programs. The preferred candidates will be familiar with USAID (or other donor) funded programs. Backgrounds in animal genomics research in the development context, with technical expertise in a field relevant to livestock/poultry production systems in Sub-Saharan Africa is required. 50 ANNEX 2. EVALUATION METHODOLOGY IN DETAIL 51 (TAKEN FROM THE EVALUATION PROTOCOL) An initial step in data collection is to identify the GIP-IL stakeholders, their relationship with the project. To select key informants for interviews, focus group discussions (FGDs), site visits, or the online survey, this evaluation will use the following criteria: i) First priority is given to beneficiaries (direct and indirect) of the GIP-IL; ii) Next priority is given to other stakeholders with interest in project results; and iii) Random selection of additional stakeholders where time and budget permit. Constraints of time, cost, and distance are also taken into consideration. Data collection of qualitative information will be based on the following tools ● Document review ● Key Performance Indicators (KPI) assessment ● Scientific Publications Assessment ● Key Informant Interviews (KIIs) ● Focus Group Discussion (FGD) ● Online survey ● Site Visits. The corresponding data collection tools are designed in a way that allows for triangulation. Document Review The evaluation team will continue reviewing a variety of documents provided by BFS, the ME, and by the sub-award partners, including but not limited to: UCD Technical Application proposal, USAID-UCD Cooperative agreement, GIP-IL Annual Work plans and other plans, GIP-IL Semi-Annual and Annual Reports, Program publications including success stories, research publications, and Human and Institutional Capacity Development (HICD) outputs, Food and Agriculture Organization reports on the incidence of NDV and other challenges in Africa, minutes of unofficial meetings and conversations, students’ curricula, transcripts, references, data from Kyeema Foundation and other international aid organizations, ISU Animal Industry Report, Poultry Welfare (book in press), Family Poultry Communications, Livestock Research for Rural Development, African Journal of Poultry Farming, Other Poultry Genomics publications, Innovation Lab website materials and information (http://gip.ucdavis.edu/), Feed the Future Research Strategy (feedthefuture.gov). Key Performance Indicators Assessment A key step in analyzing GIP-IL performance is an assessment of completion of its activities. To undertake this analysis, the following table will be used for each component and all project activities. GIP – IL Key Performance Indicators Assessment Activity Time Scheduled Time completed % of Completion Comments 1. 2. Scientific Publications Assessment As scientific publications are an expected output of a scientific research program, the evaluation team will assess them using the table below which, following USAID Scientific Research Policy 8, is inspired on the recommendations for scientific publishing of the International Committee of Medical Journal Editors (ICMJE). 8 USAID Scientific Research Policy, 2014 52 GIP – IL Scientific Publications Assessment Name Relevance for GIP-IL Peer Reviewed Format & Style Experiment Design Statistical Significance Built on Associate Research Key Informant Interviews In-depth KIIs will be conducted by the evaluation team in the U.S., Ghana, and Tanzania. Most of the interviews will be face-to-face, with some via phone or Skype. The key informant interviewees have been grouped in five categories, although there are some overlaps among them. The categories include: ● Management Team ● Research Team ● Research Advisory Committee ● Students ● Institutional collaborators Focus Group Discussions The evaluation will convene four FGDs; the first three will include all involved in the project at UCD, UOG, and SUA respectively; the fourth, in Tanzania, will be with the breeders that GIP-IL has been in contact with to assess their interest in project activities, current and future. Online Survey Using SurveyMonkey.com, an online survey targeting GIP-IL stakeholders will be conducted to support a SWOT Analysis of the GIP-IL, which will be focused on answering USAID questions on project improvement and future. The SWOT analysis will complement, at higher level, the answers obtained on program future through the KIIs. Likewise, the survey will allow for triangulation. Site Visits As GIP-IL involves using laboratories and poultry breeding facilities in the U.S., as well as in Ghana and Tanzania, and improving them in the African countries, the evaluation team will use site visits to the three universities to observe them, how they are used, look for similarities and differences research wise, watch researchers in their work environment, and get an overall perspective that documents, or phone interviews cannot provide alone. The rationale to visit UCD is the same to visiting UOG and SUA, and therefore the evaluation protocol proposes having KIIs, Focus Groups, and Site Visits, which also will allow for triangulation, and the quest for evidences as required by USAID Evaluation Policy. Altogether, the evaluation team will collect data to answer the evaluation questions using: - A Stakeholder mapping - KPI assessment - A Scientific Publications assessment - 46 KIIs - 3 FGDs - 1 online survey - 3 Site visits (UCD, UOG, SUA) 53 Quality Control To ensure quality control of data collection, KII and FGD responses will be written using a structured format aligned with the evaluation questions. In addition, evaluators will review the data collected after each interview or FGD, crosscheck their notes, and make any necessary clarifications. To ensure that focus group members communicate openly, they will be selected to form homogeneous groups, minimizing differences in factors such as position and influence. Data Processing The informatics tools this evaluation will use are a data base in Google Docs, and a statistics application. In this evaluation, Google Forms, as it is integrated with Google docs, will serve a double purpose: The first one is allowing for entering data collection during field work. The data from interviews, FGDs, site visits and even document reviews, will be collected through Google Forms so that a common repository of field work information will be made by the evaluators and will be available at their disposal later for data analysis. The second use is online surveys which can be easily deployed, captured, managed, and processed. After entry, the data will be transferred to a worksheet format from where it will be cleaned, and all logical data checks performed. Google Forms creates a database that is easy to use and makes real time monitoring, quality checks and improvement possible. Google Docs and Google Forms are free online Office applications. No special software is required; all that is needed to create documents or surveys is an Internet connection and a Google account. Analysis will be done by migrating Google Forms information into Excel. Data Analysis The evaluation team will use both quantitative and qualitative methods for data analysis. For the purposes of this Evaluation Protocol, the evaluation team has broken out the evaluation questions to generate approximately 100 sub-questions allowing for definition of indicators. Analysis will be primarily qualitative, focused on comparing actual performance against project proposal and work plans, and determining possible causes for the difference, if any; as well as plausible causal links between inputs, activities and outcomes, to answer the evaluation questions. Progress of project performance indicators will be analyzed to assess effectiveness and efficiency. Common themes and shared perspectives among stakeholders will be identified across country contexts to generate analytical categories. The evaluation team will use standard questions to generate comparable qualitative data across interviews and throughout the analysis of documents. Open-ended and fixed choice questions will be used in the on-line survey. KII responses will be triangulated with data derived from the document review, online survey and site visits. Online questions will be more general and comprehensive to complement specificities of the KIIs. The evaluation team will focus throughout on gaining a complete understanding of factors linked to the changes that resulted in major modifications of project outputs and outcomes, which along with evaluation team expertise, both in the subject matter and evaluation, will shape their recommendations for project improvement and for the future. The evaluation team will disaggregate data by project component, country, and sex. Evaluation Question Matrix The following matrix refers only to the questions mentioned in Section 2.c. A matrix including the evaluation sub-questions, will be developed for the evaluation protocol. 54 GIP-IL Performance Evaluation Matrix Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) I. Program Management 1. To what extent has the Management Entity effectively implemented and managed the Poultry Genomics Innovation Lab’s respective research and training activities in Ghana and Tanzania? In what ways could implementation and management be improved? • Work plans • Implementation reports • Document review • Project publications • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis 2. How effectively has the ME managed and coordinated amongst the partners, Missions, host country government partners, and local researchers? In what ways could these partners be improved, or new partners engaged? • Work plans • Implementation reports • Document review • Project publications • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis II. Research and Training Program 3. To what extent did the Poultry Genomics IL make progress towards developing poultry with enhanced NDV resistance? What other research and work remains to fully complete each research objective? What work remains to get a usable research output (poultry with enhanced resistance to Newcastle Disease virus) to meet the goal of the Poultry Genomics IL? • Work plans • Implementation reports • Document review • Project publications • Project statistics • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis 4. To what extent did the Poultry Genomics IL produce quality research outputs, using appropriate metrics (e.g., peer reviewed publications)? What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for Feed the Future countries) moving forward? • Work plans • Implementation reports • Document review • Project publications • Project statistics • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis 5. Within the context of the Poultry Genomics IL’s research program, how well has the ME identified and addressed academic and technical capacity needs of host country stakeholders? How could the project better serve and provide for the capacity needs of these stakeholders? • Work plans • Implementation reports • Document review • Project publications • Project statistics • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis 55 GIP-IL Performance Evaluation Matrix Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) III. Institutional Capacity Collaboration 6. How has the ME performed in establishing productive collaborations with host country governmental and academic institutions, local NGOs, and other relevant USAID programs in the target countries? How could the ME improve its institutional collaborations? • Work plans • Implementation reports • Document review • Project publications • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis IV. Program Future 7. In what ways could the Poultry Genomics Innovation Lab’s management, research (i.e., design, implementation, stakeholder involvement), capacity development programs, and/or institutional capacity collaboration be improved to better achieve objectives? • Work plans • Implementation reports • Document review • Project publications • Academic and Industry publications • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis 8. What specific technical areas, if any, within the Poultry Genomics IL’s research program merit new or continuing research investment (e.g., continued genetic characterization of different African ecotypes for enhanced NDV resistance, testing poultry breeding programs in country to disseminate/scale improved poultry to smallholder farmers)? • Work plans • Implementation reports • Document review • Project publications • Academic and Industry publications • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis 9. What topics and areas are promising for further investment in animal genomics for improved livestock and poultry production in developing countries? • Work plans • Implementation reports • Document review • Project publications • Academic and Industry publications • KII • Online surveys • Focus group discussions • Site visits • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis V. Cross-cutting Issues (Specific questions to be worded in the Evaluation Protocol) 10. Gender • Work plans • Implementation reports • Document review • Project publications • Project statistics • • Data comparison • Pattern discovery • Performance indicator progress analysis • Triangulation • Causality analysis 11. Nutrition 12. Environment 13. Project Management 14. Capacity Building 56 Outcome Measures The evaluation assessed outcome measures by reviewing the project log frame and checked against the Key Performance Indicators-KPI with the purpose of answering the evaluation questions. The result will be to highlight the differences GIP-IL has made. Team Members, Roles and Responsibilities Team Members, Roles, and Responsibilities Team Member Qualification Contribution to the Evaluation Roles & Responsibilities Luis Bernal, MPP Economic development, project management, and monitoring and evaluation Specialist Evaluation Team Leader Evaluation design, schedule and logistics management, final production of evaluation deliverables. Chris Ashwell, PhD Poultry science specialist, focused on animal genomics, working on research and teaching at University level Research Program Evaluation Assessing objectives 1,2,3, and 4 (research components of the GIP￾IL), Paul Miller, PhD senior poultry production expert, specialized in bio￾security, value chain, market development, management, and poultry nutrition Poultry Distribution Evaluation Focused in assessing objectives 5 and 6 (improved chicken breeding and distribution). The three members Work collaboratively in evaluation design, field work, data collection, data analysis, and evaluation reporting. Logistics and Supplies The evaluation will undertake data collection at three sites: ● Davis – California (between August 29 and September 8, 2017). Number of day to be determined) ● Accra – Ghana (September 18 – September 23) ● Morogoro – Tanzania (September 25– September 30) Meetings in California, including video-meetings with ISU while in California, will be scheduled by the evaluation team with the help of the GIP-IL Manager. Meetings in Ghana and Tanzania will be scheduled with the support of the GIP-IL managers in each country. Each field trip will include an introductory meeting with the Management Team, including initial discussion to adjust details, followed by a KII with the upper manager in each case. During the fieldwork, the Evaluation Team will conduct a series of KIIs and FGDs. For transportation, in California the evaluation team will use taxis, and in Africa, will follow the recommendations of each country manager. PEEL will help the evaluation team with their visas, medical insurance, and overall international travel to Ghana and Tanzania. The evaluators will take care of getting the required vaccines. Data Management & Security In accordance with USAID’s ADS Chapter 508 (Privacy Program), the handwritten notes of key informant interviews will be secured in a locked safe or suitcase to secure respondents’ Personally Identifiable Information. For this evaluation, such information will be limited to names, titles, and geographic location, which will be removed during data processing. 57 Preparation of Datasets for Public Use As indicated in ADS Chapter 579 (USAID Development Data), the data collected will be prepared for submission to USAID’s Development Data Library (DDL). More specifically, notes of responses for individual key informant interviews and focus group discussions will be typed, removing any personally identifiable information. In addition, indicator target and actual values will be presented in table format. Ethical Considerations According to USAID Scientific Research Policy 9, initiatives such as performance evaluations, assessments for strategy development or training activities for scientific and technical personnel, are not considered research and therefore not required to pursue a review from an Institutional Review Board (IRB) for human subjects’ protections. For its part, USAID Evaluation Policy 10 emphasizes that: “Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of strategies, projects, and activities as a basis for judgments to improve effectiveness and timed to inform decisions about current and future programming.” Given that the subject of inquiry of the final performance evaluation of the GIP-IL is a project, rather than a research on human beings, or animals, the Evaluation Team will not pursue an approval of any IRB. 9 USAID Scientific Research Policy, 2014 10 USAID Evaluation Policy. January 2011, updated October 2016 58 ANNEX 3. NUMBER INTERVIEWED, SURVEYED, OR IN FOCUS GROUPS 59 Location Affiliation Country UCD GIP-IL Director / PI / GIP-IL /Management Research Team U.S. UCD GIP-IL Project Manager GIP-IL Management Team U.S. UCD GIP-IL Gender Advisor U.S. UCD GIP-IL Research Team U.S. UCD GIP-IL former Director U.S. UCD GPI-IL RAC / UCD U.S. UCD USAID GIP-IL AOR U.S. Virtual Collaborator Virtual GIP-IL Management Research Team Virtual GIP-IL Research Team Virtual GPI-IL RAC / Cornell University U.K. Virtual GPI-IL RAC / University of Nairobi Kenya Virtual GPI-IL RAC / INRA France Virtual GPI-IL RAC / Friedrich Loeffler Institute Germany Virtual GPI-IL RAC / University of Sydney Australia Virtual GPI-IL RAC / Hy-Line International U.S. Virtual Collaboration USDA U.S. Virtual GPI-IL RAC U.S. UCD Graduate Student U.S. Virtual Graduate Student U.S. Virtual Graduate Student U.S. Virtual Graduate Student U.S. SUA GIP-IL Research Team Tanzania SUA GIP-IL Research Team Tanzania SUA GIP-IL Research Team Tanzania SUA Graduate Student Tanzania SUA Graduate Student Tanzania SUA Tanzania Poultry Breeder's Association Tanzania SUA International Livestock Research Institute Tanzania SUA GPI-IL RAC SUA Graduate Student Ghana SUA Graduate Student Ghana UoG GIP-IL Research Team Ghana UoG GIP-IL Research Team Ghana UoG GIP-IL Research Team Ghana UoG Graduate Student Ghana UoG Graduate Student Ghana UoG Graduate Student Ghana 60 ANNEX 4. DATA COLLECTION INSTRUMENTS 61 Questionnaire for Management Team KII 1. GIP-IL EVALUATION / QUESTIONNAIRE FOR THE MANAGEMENT TEAM Goals & Objectives 1. How has the GIP-IL program, so far, contributed to improve production of chicken by households and small farmers and, thereby, improve food security, nutrition and livelihoods in Africa? 2. To what extent did the GIP-IL make progress towards developing poultry with enhanced NDV and stress heat resistance? 3. What other research and work remains to fully complete each research objective? 4. To what extent did the GIP -IL program adequately propose studies to define the genetic basis of innate resistance to NDV? 5. Has the GIP -IL program adequately proposed studies using advanced genetic technologies (e.g. high-density SNP chips and RNA sequence)? 6. To what extent did the Poultry Genomics IL produce quality research outputs, using appropriate metrics (e.g., peer reviewed publications)? 7. How has the GIP -IL program provided essential knowledge of value to existing projects developing new vaccine approaches to control the disease? 8. Has UC Davis met its expectation of a substantial improvement in NDV resistance through the GIP-IL program? 9. To what extent has the GIP-IL program contributed to expand the work of African universities’ veterinary and animal science departments? 10. How have UCD and ISU brought advanced genetic and genomic methods to the GIP-IL program. Which technologies? 11. Have African students and scientists participated in all stages of UCD and ISU bird NDV challenge, sample collection, assay and data analysis of large scale SNP studies and selection of superior breeding stock? 12. To what extent has the GIP-IL program worked with existing chicken breeders in the private sector in African countries? Implementation and Monitoring 13. Which have been the major difficulties for the management team to work effectively? 14. What performance indicators did the project adopt? 15. Does the project have a M&E framework? How has such a framework been followed so far? 16. What have been the major change-of-course moments of the project so far? 17. What added value has the Research Advisory Committee brought to the project performance so far? 18. What changes have the Management Team experienced so far, and what have been their effects on project performance? 19. In terms of strengths and weaknesses, how could be the annual research coordination meetings be described? Project Engagement 20. How has the ME performed in establishing productive collaborations with host country governmental and academic institutions, local NGOs, and other relevant USAID programs in Ghana and Tanzania? 21. To what extent has the project built on the research and outreach experience of faculty at SUA and UOG? 22. How has UCD briefed USAID Missions and ministries of agriculture in Ghana and Tanzania on the research progress? 23. What is the status of the compilation of a list of poultry breeders across Sub-Saharan Africa by UCD? 24. What is the status of organization by UCD of a workshop with breeders and small farmer organizations to develop a Breeding Implementation and Distribution Plan for chickens of the new strains with greater NDV and heat resistance? 25. What is the status of the organization by UCD of meetings with representatives of small farmer organizations, poultry suppliers, and women farmers groups to receive input plans to distribute the improved chickens? 26. What is the status of a UCD list of international, national, and local stakeholders that are involved in smallholder poultry production? 27. How have international and local NGOs that are implementing major development programs in nutrition and homestead and small farm production, been briefed and engaged in outreach meetings? 62 KII 1. GIP-IL EVALUATION / QUESTIONNAIRE FOR THE MANAGEMENT TEAM 28. Have outreach meetings included farmer-to-farmer groups, Heifer International, Helen Keller International, Save the Children, the Aga Khan Foundation and others? 29. What is the status of meetings to be organized by UCD (in project’s fifth year) with USAID Mission staff, and ministry extension and other advisory services to fully advise them on the results of the research the Breeding Implementation and Distribution Plan that will provide guidance on the distribution of enhanced chickens? 30. What is the status of collaboration between the project and Schmidt and Lamont USDA-NIFA funded ($4.7 M) climate change project on: "Adapting poultry production to climate change through breeding.” 31. What added value has the project provided to SUA and UOG Faculty’s research on local chicken ecotypes of East and West Africa? Capacity Building at SUA and UOG 32. What have been the program achievements in terms of capacity building at SUA and UOG, with respect to: 1. Infrastructure improvement? 2. Advanced genetics sampling? 3. Laboratory procedures training of faculty, students and staff? 4. Research training of graduate students? 5. Graduate student’s engagement on research and outreach to women farmers, small poultry producers, poultry breeders, and other program stakeholders. 6. SUA and UOG faculty and graduate students training in the U.S.? 7. Long-term collaborative research and outreach relationships developed between UCD, ISU, SUA and UOG Gender 33. How did the project design and implementation include gender-oriented considerations, information, indicators or deliverables? 34. What is the percentage of women involved in: 1. The research programs? 2. UCD training activities? 3. Outreach to breeders, farmers, poultry supply and marketing? Nutrition 35. Have nutrition effects in children, somehow, been considered in the design / implementation of project activities? Environment 36. How has environmental considerations been included in project design and implementation? 37. What were the results of the Initial Environmental Examination (IEE)? Project Improvement and Future 38. In what ways could implementation and management be improved? 39. In what ways could these partnerships be improved, or new partners engaged? 40. What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for FTF countries) moving forward 41. Within the context of the Poultry Genomics IL’s research program, how well has the ME identified and addressed academic and technical capacity needs of host country stakeholders? 42. How could the project better serve and provide for the capacity needs of those host country stakeholders? 43. How could the ME improve its institutional collaborations? 44. In what ways could the Poultry Genomics Innovation Lab’s management, research (i.e., design, implementation, stakeholder involvement), capacity development programs, and/or institutional capacity collaboration be improved to better achieve objectives? 45. What specific technical areas, if any, within the Poultry Genomics IL’s research program merit new or continuing research investment (e.g., continued genetic characterization of different African ecotypes for enhanced NDV resistance, testing poultry breeding programs in country to disseminate/scale improved poultry to smallholder farmers? 46. What topics and areas are promising for further investment in animal genomics for improved livestock and poultry production in developing countries? 63 Questionnaire for the Research Team KII 2. GIP-IL EVALUATION / QUESTIONNAIRE FOR THE RESEARCH TEAM Research Objectives 1. To what extent has GIP-IL identified genes, genetic markers, and genetic pathways associated with NDV resistance and heat stress, in three unique chicken populations in the U.S. using advanced genetic and genomic technologies? (Research Objective 1) 2. To what extent has GIP-IL identified genes and/or genetic markers associated with NDV resistance and heat stress in African indigenous chicken ecotypes using whole genome high density SNP panels? (Research Objective 2) 3. To what extent has GIP-IL developed an economical low-density SNP panel for genetic selection of disease resistance to NDV and to heat stress by integrating analysis of the results from objectives 1 and 2? (Research Objective 3) 4. To what extent has GIP-IL validated genetic enhancement of resistance to NDV infection in African indigenous chicken ecotypes using low-density SNP panel selection? (Research Objective 4) 5. To what extent has UCD developed, validated, improved breeding stock with enhanced NDV resistance in Africa. (Research Objective 5) 6. To what extent has UCD developed a chicken breeding and distribution plan (CBDP)? 7. What IACUC (Institutional Animal Care and Use Committee) protocols has the GIP-IL program obtained prior handling live chickens at UCD, ISU, SUA, and UOG? 8. Has all project staff been trained on IACUC protocols obtained by the GIP-IL program? How? 9. Did the CDBP convene a workshop? When? Where? How many women participated? How many women farmer groups/organizations participated? Which other sub-Saharan countries participated? Has the Plan been shared with USAID Missions, ministries of agriculture of African countries, and potential funders for chicken breeding and distribution activities? (RO6) Research Design and Methods 10.Were research activities conducted as planned? If not why, and what was learned from the revised approach? 11.In retrospect, was the research design and methods chosen proven to be the best option possible in terms of: i.) results, ii) cost, iii) efficiency. 12.What research results, in terms of NDV and heat stress resistance have come out of using the three proposed inbred chicken lines: Leghorn, Fayoumi, HyLine Brown Line? What lessons have been learned? (RO1) 13.In what extent has using LaSota lentogenic NDV strain met research program expectations? (RO1) 14.Was the rationale for the program to conduct animal challenge experiments with NDV only at ISU and with a combination of NDV and heat stress at UCD: i.) correct? ii.) effective? iii.) efficient? (RO1) 15.Where phenotypic measurements collected: i.) as proposed? ii.) On continuous basis? iii) properly recorded? iv.) analyzed as expected.? (RO1) 16.Were chickens’ body fluids, parts and tissue: i) collected? ii) studied? iii) recorded, iv.) analyzed? (RO1) 17.Did studies on inbred chicken lines (Leghorn and Fayoumi) at both UCD and ISU identify gene families and pathways associated with NDV resistance and heat stress? (RO1) 18.Was the RNA-Seq analysis completed as proposed? Were the sequence data properly prepared and recorded? filtered? and mapped? (RO1) 19.Was the chicken 600K SNP genotyping conducted as proposed on the US commercial layer line? Did it use the GenSel software? Did it use the ASREML software? (RO1) 20.Were animal experiments conducted at UOG and SUA? (RO2) 21.Did UCD establish parental stock of the three selected chicken ecotypes? (RO2) 22.Were the Ching’ wekwe, Kuchi, and Morogoro-medium, Myeba ecotypes used in Tanzania? (RO2) 23.Which three ecotypes were selected in Ghana? (RO2) 24.Were the trials in both Tanzania and Ghana conducted as proposed? (RO2) 25.To what extent has using LaSota lentogenic NDV strain met the research objectives expectations? (RO2) 26.Were animal challenge and phenotype collection by both UOG and SUA conducted as proposed? (RO2) 27.Was genotyping and analyses conducted as proposed? Using the methods, collecting the data and doing analysis separately for each ecotype? For each country? Across countries? How was the consistency of results among ecotypes and countries analyzed? (RO2) 64 KII 2. GIP-IL EVALUATION / QUESTIONNAIRE FOR THE RESEARCH TEAM Research Objectives 28.To what extent has UCD been conducting candidate gene analysis as proposed? (RO2) How has these analyses contributed to definitive estimates of effects of the SNPs? (RO2) 29.To what extent has UCD developed a low-density SNP panel for genetic selection of disease resistance to NDV and heat stress by integrating analysis of results from research objectives 1 and 2? (Research Objective 3) 30.To what extent has the following information been collected and used to identify SNPs to be included in a low-density SNP panel: (RO3) 1. Genomic regions and SNPs associated with response to NDV and NDV+stress in the U.S. commercial line? (RO3) 2. Pathways and genes that are associated with response to NDV and NDV+stress from the two inbred lines in the U.S.? (RO3) 3. Genomic regions and SNPs associated with response to NDV in African ecotypes, both within and across ecotypes? (RO3) 31.To what extent were all sources of information used, experiments done as proposed by UCD to develop the low-density panel confirmed by research, and contributed to success of such development? (RO3) 32.To what extent has GIP-IL developed an economical low-density SNP panel for genetic selection of disease resistance to NDV and to heat stress by integrating analysis of the results from objectives 1 and 2?) 33.Was the validation of disease resistance to NDV infection in African indigenous chickens conducted as proposed? How did it go? What lessons were learned? (RO4) 34.How did selection of an improved breed go? What lessons were learned? (RO5) 35.Were the selected ecotypes bred, raised, and distributed to villagers? Did selected primary schools participate as proposed? (RO5) 36.How did development of an improved stock resistant to NDV integrated with the existent school project? Did UCD provide the facilities and opportunities to villagers it committed to? (RO5) 37.How did the 1-year follow up go in terms of performance and survival? (RO5) Questionnaire for the GIP-IL Research Advisory Committee KII 3. GIP-IL EVALUATION / QUESTIONNAIRE FOR RESEARCH ADVISORY COMMITTEE Goals & Objectives 1. How has the GIP-IL program, so far, contributed to improve production of chicken by households and small farmers and, thereby, improve food security, nutrition and livelihoods in Africa? 2. To what extent did the GIP-IL make progress towards developing poultry with enhanced NDV and stress heat resistance? 3. What other research and work remains to fully complete each research objective? 4. To what extent did the GIP -IL program adequately propose studies to define the genetic basis of innate resistance to NDV? 5. Has the GIP -IL program adequately proposed studies using advanced genetic technologies (e.g. high-density SNP chips and RNA sequence)? 6. To what extent did the Poultry Genomics IL produce quality research outputs, using appropriate metrics (e.g., peer reviewed publications)? 7. How has the GIP -IL program provided essential knowledge of value to existing projects developing new vaccine approaches to control the disease? 8. Has UC Davis met its expectation of a substantial improvement in NDV resistance through the GIP-IL program? 9. To what extent has the GIP-IL program contributed to expand the work of African universities’ veterinary and animal science departments? 10. How have UCD and ISU brought advanced genetic and genomic methods to the GIP-IL program. Which technologies? Implementation and Monitoring 11. What have been the major change-of-course moments of the project so far? 65 KII 3. GIP-IL EVALUATION / QUESTIONNAIRE FOR RESEARCH ADVISORY COMMITTEE Goals & Objectives 12. What added value has the Research Advisory Committee brought to the project performance so far? 13. In terms of strengths and weaknesses, how could be the annual research coordination meetings be described? Project Improvement and Future 14. In what ways could implementation and management be improved? 15. In what ways could these partnerships be improved or new partners engaged? 16. What opportunities are there to improve research outputs (in terms of both increasing quality and generating appropriate outputs for FTF countries) moving forward 17. Within the context of the Poultry Genomics IL’s research program, how well has the ME identified and addressed academic and technical capacity needs of host country stakeholders? 18. How could the project better serve and provide for the capacity needs of those host country stakeholders? 19. How could the ME improve its institutional collaborations? 20. In what ways could the Poultry Genomics Innovation Lab’s management, research (i.e., design, implementation, stakeholder involvement), capacity development programs, and/or institutional capacity collaboration be improved to better achieve objectives? 21. What specific technical areas, if any, within the Poultry Genomics IL’s research program merit new or continuing research investment (e.g., continued genetic characterization of different African ecotypes for enhanced NDV resistance, testing poultry breeding programs in country to disseminate/scale improved poultry to smallholder farmers? 22. What topics and areas are promising for further investment in animal genomics for improved livestock and poultry production in developing countries? Questionnaire for the Students KII 4. GIP-IL EVALUATION / QUESTIONNAIRE FOR STUDENTS 1. What is your academic background, prior degrees/institutions? 2. What factors contributed to your participation in the GIP-IL program? 3. What is your specific project under the GIP-IL program? 4. How has your work contributed to the GIP-IL making progress towards developing poultry with enhanced NDV and heat resistance? 5. What other research and work remains to fully complete your research objective? 6. What work in addition to yours remains to be completed in order to get a usable research output (poultry with enhanced resistance to NDV)? 7. To what extent did you contribute to the GIP-IL producing quality research outputs? 8. To what extent has UCD and ISU brought advanced genetic and genomic methods to the GIP-IL program. Which technologies? 9. How have you and other students including those from African participated in all stages of UCD and ISU bird NDV challenge, sample collection, assay and data analysis of large scale SNP studies and selection of superior breeding stock? 10. What opportunities are there to improve your research outputs? 11. What opportunities are there to improve students’ involvement? 12. What improvements could be made regarding training? 13. What were the strengths and weaknesses of the management at your level of work? 14. What management improvements could be made? 66 Questionnaire for GIP-IL Gender Advisor GIP-IL EVALUATION / QUESTIONNAIRE FOR THE GENDER ADVISOR 1. Could you please describe your role in the project? 2. How did you get involved in the project? 3. How would you describe GIP-IL design approach on gender equity? 4. In your opinion, how GIP-IL has integrated gender equity in its implementation? 5. In what degree has GIP-IL been complaint of USAID ADS 205 policies on gender equity? 6. If you were now at the design stage of the GIP-IL what did you do to make sure it fully promotes gender equity? 7. What role have the Universities involved in the project, played to promote gender equity in the project and around it? 8. What opportunities are there to improve gender equity in a project such as GIP-IL? In research? In training? Institutional collaboration? In outreach to households, small farmers, and poultry breeders. 9. Any recommendations to projects’ designers and implementers as to how to promote gender equity? Questionnaire for Representatives of GIP-IL Institutional Collaborations KII 5. GIP-IL EVALUATION / INSTITUTIONAL COLLABORATIONS QUESTIONNAIRE Government, ONGs 1. What do you know about the GIP-IL? 2. Are you aware of GIP-IL research results on NDV and heat stress to improve poultry? 3. What kind of collaboration has your institution had with GIP-IL? 4. How the GIP-IL benefits your institution? 5. Based on your experience with GIP-IL how would have for improvement Breeders / Distributors 1. Are you aware of the GIP-IL project to create an African chicken with resistance to NDV and heat stress? 2. If, so what do you know of the project? 3. Will you accept the improved chicken and multiply it? 4. What financial incentive will you receive? 5. Do you have the facilities to house the improved chickens? 6. If semen is delivered to your farm, is there a trained person who can inseminate the females? 7. Do you have a hatchery where the improved chicks can be produced separately from the regular chickens? 8. Have you been told of the educational program associated with this project? 9. Do you know organizations which can distribute the chickens? 10. If the improved chickens do show reduced mortality due to NDV or heat stress, would be willing to increase the number of chicks that you produce? 11. Do women work at your farm/business? 67 Online Survey Questionnaire 1. Email: 2. Full Name: 3. Your organization SCORING THE PERFORMANCE OF THE GIP-IL Please rate the performance of the Program according to the following criteria 4. How would you rate the relevance of the GIP-IL? 5. How would you rate the EFFECTIVENESS of the GIP-IL? 6. How would you rate the EFFICIENCY of the GIP-IL? 7. How would you rate the FUTURE FINANCIAL SUSTAINABILITY of the GIP-IL? 8. How would you rate the performance of the GIP-IL regarding promotion of gender equity? YOUR OPINION ON GIP-IL ACHIEVEMENTS Please express the extent of your agreement with the following statements related to the Program's achievements: 9. GIP IL has made a great progress towards developing poultry with enhanced NDV resistance and heat stress... 10. GIP-IL has been producing quality research outputs... 68 11. GIP-IL has identified well and addressed the training needs of its partners in Ghana and Tanzania... 12. GIP has established productive collaborations with universities, local NGOs, and the Government in Ghana and Tanzania... 13. By the end of its current funding end in 2018, the GIP-IL will have met its goal of developing ecotypes of African chickens with greater innate resistance to NDV and heat stress... 14. What do you think are the major strengths of the GIP-IL Research program? (Please elaborate as much as you wish). ___________________________________________________________________ __________________________________________________________________________ ________ 15. What do you think are the major weaknesses of the GIP-IL Research program? (Please elaborate as much as you wish) _________________________________________________________________ _______________________________________________________________________ ___________ FOR THE FUTURE... Please share your ideas in regard to possible future improvements on the program. 16. What do you think are the major weaknesses of the GIP-IL Research program? (Please elaborate as much as you wish) ___________________________________________________________________ __________________________________________________________________________ ________ 69 ANNEX 5. GIP-IL PUBLICATIONS 70 GIP-IL PUBLICATIONS AS OF SEPTEMBER 2017 Title-Topic Lead Partner Authors Status Publication Year 1. Does gender impact the immune response of chicks? ISU Herrmann et al. Published Iowa State University Animal Industry Report 2017 2. Different Genetic Resistance Resulted in Distinct Response to Newcastle Disease Virus ISU Zhang et al. Published Iowa State University Animal Industry Report 2017 3. Estimating genetic parameters for growth and response to infection with LaSota lento genetic Newcastle disease virus strain in local chicken breeds in Ghana and Tanzania ISU Amuzu￾Aweh et al. Published Iowa State University Animal Industry Report 2017 4. Commercial layer-type chickens and Newcastle disease virus infection: Toward genetic selection of more resilient chickens ISU Rowland et al. Published ISU Animal Industry report 2017 5. Novel mechanisms revealed in the trachea transcriptome of resistant and susceptible chicken lines following infection with Newcastle disease virus ISU Deist et al. Published Clinical and Vaccine Immunology 2017 6. Responding to environmental stressors using genetics approaches UCD Wang et al. In press Advances in Poultry Welfare 2017 7. Approaches to Develop a Chicken Breeding and Distribution Program with Smallholder Farmers UCD Wilson et al. In press Family Poultry Communicatio ns 2017 8. Physiological responses to heat stress in two genetically distinct chicken inbred lines UCD Wang et al. In review Poultry Science 2017 9. RNA sequence analysis spleen following infection with NDV ISU Zhang et al. In review Scientific Reports 10. Key criteria and models to implement a chicken breeding and distribution program for smallholder farmers in developing countries: A poultry expert survey UCD Wilson et al. In draft Livestock Research for Rural Development 11. RNA sequence analysis harderian gland following infection with NDV ISU Deist et al. In draft Scientific Reports 12. RNA sequence analysis trachea, lung, and harderian gland combined following infection with NDV ISU/UCD Concept TBD 13. RNA sequence analysis liver following treatment with heat stress UCD Planned TBD 14. RNA sequence analysis hypothalamus following treatment with heat stress UCD Planned TBD 15. RNA sequence analysis muscle following treatment with heat stress UCD Planned TBD 71 GIP-IL PUBLICATIONS AS OF SEPTEMBER 2017 Title-Topic Lead Partner Authors Status Publication Year 16. RNA sequence analysis trachea following treatment with NDV and heat stress UCD Planned TBD 17. RNA sequence analysis lung following treatment with NDV and heat stress UCD Planned TBD 18. RNA sequence analysis Harderian gland following treatment with NDV and heat UCD Planned TBD 19. Survival of two genetic lines following infection with velogenic NDV UCD/ISU Planned TBD 20. Genome wide association Hy-Line Brown after infection with NDV ISU Rowland et al. Planned TBD 21. Genome wide association Hy-Line Brown after treatment with heat UCD Planned TBD 22. Genome wide association Hy-Line Brown after treatment with NDV and heat UCD Planned TBD 23. Genome wide association Hy-Line Brown with combined trials from UCD and ISU UCD/ISU Concept TBD 24. Traits of Tanzanian chicken ecotypes quantitative genetic parameters SUA Planned TBD 25. Genome wide association on Tanzanian chicken ecotypes following infection with lentogenic NDV SUA Planned TBD 26. Genome wide association on Tanzanian chicken ecotypes following infection with velogenic NDV SUA Planned TBD 27. Phenotypic characterization of Tanzanian chicken ecotypes SUA Planned TBD 28. Growth and behavioral responses to low dietary levels in selected Tanzanian local chicken ecotypes SUA Manuscript in final stages of Preparation African Journal of Poultry Farming 29. Physiological and biochemical responses to low dietary energy levels in selected Tanzanian local chicken ecotypes SUA Data analysis stage TBD 30. Growth responses to heat stress of selected Tanzania local chicken ecotypes SUA Data analysis stage TBD 31. Physiological and biochemical responses to heat stress in selected Tanzania local chicken ecotypes SUA Laboratory work in progress TBD 32. Traits of Ghanaian chicken ecotypes quantitative genetic parameters UoG Planned TBD 33. Genetic Parameter Estimates for Juvenile Growth and Indicators for Newcastle Disease Virus Resistance in Three Ghanaian Local Chicken Ecotypes UoG Draft MPhil thesis in final stage. TBD 72 GIP-IL PUBLICATIONS AS OF SEPTEMBER 2017 Title-Topic Lead Partner Authors Status Publication Year 34. Genome wide association on Ghanaian chicken ecotypes following infection with lentogenic NDV UoG Planned TBD 35. Genome wide association on Ghanaian chicken ecotypes following infection with velogenic NDV UoG Planned TBD 36. Phenotypic characterization of Ghanaian chicken ecotypes UoG Planned TBD 37. Genetic variations associated with NDV resistance in Hy-Line Brown ISU Rowland et al. Planned TBD 38. Genetic variations associated with NDV and heat stress resistance in Hy-Line Brown UCD Planned TBD 73 ANNEX 6. PROJECT ACTIVITY PROGRESS 74 GIP – IL Project Activities Progress at the Time of the Evaluation Activity Planned for Achieved in % Completed Comments from GIP-IL 1. Personnel hired and trained 3 months 3 months 100 Training activities are on-going as GIP-IL continues to address training needs for each of our partners. 2. Program Director and collaborators begin work in Tanzania and Ghana to coordinate project work, ready the sites, and train personnel 3 months 3 months 100 3. Equipment purchased 3 months 3 months 100 Major equipment needed to complete project activities was purchased in this timeline. However, GIP-IL continues to procure the necessary equipment and supplies to meet project scope of work each year. 4. Begin facilities renovation in Africa and in preparation for challenge trials 3 months 6 months 100 Sub awards to partners at SUA and UoG were delayed by the UCD Sponsored Programs Office causing delays in getting funds in place to begin the renovation according to schedule. 5. Birds acquired for studies in the U.S. 3 months 6 months 100 The birds for challenge studies in the U.S. were acquired from program partner ISU. Sub-award to ISU was delayed by the UCD Sponsored Programs Office causing delays in getting funds to acquire the birds according to schedule. 6. Pathogen and heat-stress trial initiated in the US, samples collected from the first replicates 6 months 6 months 100 7. Meeting of Program Director and research team to review performance of first round of challenge trials, and make adjustments as needed for subsequent trials 6 months 6 months 100 A velogenic NDV trial using highly inbred U.S. lines was added to the scope of work to validate observations in the lentogenic NDV trials. This activity was not in the original scope of work, but was by recommended by the RAC and required additional time and resources. 8. Prepare samples and genotype on high-density SNP chip or sequence transcriptome Year 2 Year 2 100 9. Initiate statistical and bioinformatics analyses of genomic data Year 2 Year 2 100 75 10. Produce written annual progress report Annually Annually 100 11. Meet with Research Advisory Committee to evaluate progress and plans Every 6 months Every 6 months 100 Conducted in-person and remote meetings bi-annually and as needed to receive input from the RAC. 12. Complete statistical and bioinformatics analyses of transcriptome data from inbred lines Year 3 Year 3 100 13. Develop and test technical performance of low-density SNP panel for validation phase Year 3 This activity is delayed due to challenges faced by the GIP-IL, including initial delays in sub-award execution to program partners. Additional delays in completing the animal trials due to power outages resulting in equipment failures and disease outbreaks in the animal challenge facilities. Further, SUA and UoG conducted 5 additional natural velogenic NDV exposure field trials as recommended by the RAC. This activity was added to the scope of work to validate observations in the animal trials with lentogenic NDV challenge with a circulating velogenic NDV strain. These trials added six months for each trial to project timeline. The laboratory analyses have also been met with challenges as human capacity for conducting advanced molecular assays in the SUA and UoG laboratories had to be enhanced through initial and several follow￾ups training sessions, including daily trouble-shooting of the assays conducted through remote meetings. 14. Produce written annual progress report Annually Annually 100 15. Prepare manuscripts for journal publication on completed fundamental and applied aspects of project Year 3 Year 3 100 Two manuscripts have been published and two submitted on completed work. A book chapter, several commercial poultry industry reports, and dozens of conference proceeding abstracts have also been published. Several additional manuscripts are in preparation. 16. Graduate M.S. students in Africa who worked on challenge trial studies Year 4 Year 4 85% One M.S. student in who started in 2014, will graduate in 2017 and another M.S. student only started in 2015 plans to graduate in 2018. Other Ph.D. students’ programs are currently on-going and on￾track. 17. Meet with Research Advisory Committee to assess progress and plans Annually Annually 100 Conducted in-person and remote meetings bi-annually and as needed to receive input from the RAC on progress and plans. 18. For validation phase: facilities readied, animals acquired Year 4 Year 4 90 This activity has been completed at SUA will be completed by the end of September 2017 at UoG. 19. Pathogen trial conducted, samples collected, phenotypes measured. Year 4 Year 4 100 20. Isolate DNA, genotype all animals on low-density SNP panel. Year 4 See comment for #13. 76 21. Statistical analysis of effectiveness of low-density panel to improve performance. Year 4 See comment for #13. 22. Produce written annual progress report. Annually Annually 100 23. Prepare manuscripts for journal publication on fundamental and applied aspects of project, validation phase Year 5 See comment for #15. Manuscripts for validation phase are in currently planned. 24. Meet with Research Advisory Committee to assess progress and plans. Annually Annually 100 25. Conduct outreach in Africa to explain technology and improved performance of chickens genetically selected for NDV-resistance and heat tolerance. Year 5 25 Outreach to stakeholders in Africa on aspects related to the technology and benefits of improved innate NDV resistance in indigenous chickens in Africa has been initiated and is on-going. Further in-depth engagement is planned in future years. 26. Work with local individuals and organizations to plan sustained distribution of technology and improved genetic stocks Year 5 15 Work with local individuals and organizations to plan sustained distribution of technology and improved genetic stocks has been initiated and is planned in future years. 27. Graduate M.S. students in Africa who worked on challenge trial studies Year 5 See comment # 18. 28. Prepare manuscripts for journal publication on fundamental and applied aspects of project, validation phase and outreach activities Year 5 One manuscript has been published and one is in preparation. Additional manuscripts are planned. 29. Prepare Report on first five years Year 5 Planned 30. Equipping facilities to conduct pathogen challenges, and laboratory analyses Year 1 Year 1 100 31. Training individuals to conduct pathogen-challenge trials in the US, Tanzania and Ghana Year 1 Year 1 100 This activity has occurred throughout the program and is on￾going as needed to complete the scope of activities. 32. Production or purchase of chicken stocks for challenge trials, in the US and local ecotypes of chickens of West and East Africa Year 1 Year 1 100 33. Collection of data and samples on live birds and at necropsy from NDV (ISU, Tanzania and Ghana) and NDV and heat￾stress (UCD) trials. Year 3 Year 4 See comment #13. 77 34. Histopathologic analysis of NDV-induced tracheal lesions Year 2 Year 2 100 35. Laboratory analyses of samples to quantify NDV viral load and anti-NDV antibody level to classify severity of infection at all sites. Year 3 Year 4 95 Completed at UCD and ISU and is expected to be completed at SUA and UoG by the end September 2017. See comment #15 regarding delays. 36. High-density SNP genotyping or transcriptome sequencing from chickens in NDV and heat stress trials. Year 3 Year 3 100 37. Development of shared database for large data sets. Year 3 Year 3 100 38. Statistical analyses of associations of genes, gene families and genomic regions with resistance to NDV and heat stress, and maintained high productivity. Year 3 Year 3 100 39. Development of small, economical genotyping test platform (low-density chip) for use in typing African ecotypes for analysis and selection Year 3 See comment #13. 40. Train personnel in Africa to analyze low-density genotyping data to select superior breeding animals Year 5 Planned 41. Production or purchase of chicken stocks for validation phase of research in Africa Year 4 Year 4 90 See comment #13. 42. Low-density genotyping of all chickens to be NDV-challenged in validation studies. Year 4 See comment #13. 43. Directly challenge individuals characterized by low-density panel as either highly or lowly resistant. Year 4 See comment #13. 44. Collect data and samples on live birds and at necropsy from NDV trials. Year 3 Year 4 100 45. Laboratory analyses of samples to quantify viral load to classify severity of infection. Year 3 90 See comment #13. 46. Educate local poultry breeders and smallholder farmers of increased value of chickens with innate resistance. Year 5 10 These activities have been initiated. 47. Develop plan for sustainable development, distribution and use of improved genotypes in African ecotypes of chickens Year 5 20 This activity has been initiated. 78 ANNEX 7. EVALUATION TIMELINE 79 GIP – IL Performance Evaluation Schedule (2017) Planning, Design & Approvals Phase Kick-off meeting June 8 Documents for document review available June 5 Phone call/meeting with the Implementing Partner/Management Entity Submit to PEEL five-page draft Concept Paper June 21 Submit to USAID five-page draft Concept Paper June 22 Receive comments from USAID June 28 Integrate comments and resubmit to USAID July 6 USAID approves five-page draft Concept Paper July 9 Submit to PEEL 20-page draft Evaluation Protocol (including all data collection tools) July 24 Submit to USAID 20-page draft Evaluation Protocol (including all data collection tools) July 24 Receive comments from USAID August 3 Integrate comments from USAID August 3-7 PEEL formats 20-page final Evaluation Protocol (including all data collection tools) August 8 Submit to USAID final 20-page Evaluation Protocol (including all data collection tools) August 9 USAID approves 20-page Evaluation Protocol (including all data collection tools) August 11 USAID travel approval August 29 – Sept 8 Obtain visas, plane tickets, other logistics Aug 29 – Sept 16 Data Collection Phase Data collection in Davis, CA August 29 – Sept 8 Data collection in Tanzania Sept 18 – Sept 23 Data collection in Ghana Sept 25 - Sept 30 Online survey administration Sept 8 – Sept 30 Analysis & Reporting Phase Data organization and analysis Oct 2 – Oct 16 Evaluation Report drafting Oct 2 – Oct 16 Submit first draft 50-page report to PEEL Oct 16 PEEL reviews first draft 50-page report Oct 16-20 ET integrates PEEL review of first draft 50-page report Oct 23 – Oct 26 Submit first draft 50-page report to USAID Oct 27 Receive comments from all reviewers Oct 27 – Nov 3 ET integrates comments from all reviewers Nov 6 – Nov 8 Presentation to USAID / BFS Nov 3 – Nov 8 Submit second draft 50-page report to PEEL Nov 8 PEEL reviews second draft 50-page report Nov 8 – Nov 15 ET integrates comments to second draft from PEEL Nov 15 – Nov 17 Submit second draft 50-page report to USAID Nov 20 ET receives comments from all reviewers Nov 20 – Dec 1 ET integrates comments from all reviewers Dec 1 – Dec 5 ET Submits final 50-page report to PEEL Dec 5 PEEL reviews, formats final 50-page report Dec 5 – Dec 8 PEEL – ET submit final 50-page report to USAID Dec 11 USAID approves Final Report To Be Determined 80 ANNEX 8. EVALUATION TEAM COMPOSITION, ROLES, AND CONFLICTS OF INTEREST 81 Evaluation Team Roles Team Member Qualification Contribution to the Evaluation Roles & Responsibilities Luis Bernal, MPP Economic development, project management, and monitoring and evaluation Specialist Evaluation Team Leader Evaluation design, schedule and logistics management, final production of evaluation deliverables. Chris Ashwell, PhD Poultry science specialist, focused on animal genomics, working on research and teaching at University level Research Program Evaluation Assessing objectives 1,2,3, and 4 (research components of the GIP￾IL), Paul Miller, PhD senior poultry production expert, specialized in bio￾security, value chain, market development, management, and poultry nutrition Poultry Distribution Evaluation Focused in assessing objectives 5 and 6 (improved chicken breeding and distribution). The three members Work collaboratively in evaluation design, field work, data collection, data analysis, and evaluation reporting. Evaluation Team Curricula Summaries Luis Bernal, MPP, Evaluation Team Leader is a senior international development and evaluation consultant with more than 30 years of experience designing, implementing, and evaluating private and public initiatives for: economic growth; local development; private sector competitiveness; entrepreneurship; micro, small, and medium-sized business promotion; poverty alleviation, and social inclusiveness. His extensive Monitoring & Evaluation background is comprised of meta-evaluations, performance and ex-post evaluations of multi-site, multi-million and multi-country projects intended to measure a program’s efficiency, efficacy, impact, outreach, and sustainability. Mr. Bernal has led and collaborated on regional projects in Latin America and the Caribbean, as well as projects in: Barbados, Bolivia, Chile, Colombia, Costa Rica, Ecuador, Guatemala, Ghana, Honduras, Panama, Spain, Liberia, Tanzania, and the United States, working for IDB, USAID, UN-PBSO and UNDP. Mr. Bernal earned his master’s degree in Public Policy from the University of Massachusetts and a certificate in Evaluation and Applied Research Methods from Claremont University. He also holds a bachelor’s degree in Business Administration from the Universidad Externado de Colombia. Chris Ashwell, PhD. Poultry Science Specialist. Associate Professor of Animal Genomics, at North Carolina State University, with a dual appointment consisting of 70 percent Research and 30 percent Teaching. B.S. Biochemistry and Nutrition from Virginia Tech (Virginia Polytechnic Institute and State University) in 1992. His education continued at the Bowman Gray School of Medicine at Wake Forest University where he received his Ph.D. in Biochemistry in 1997. Later, he got a USDA headquarters funded postdoctoral position in the Growth Biology Laboratory of the USDA’s Agriculture Research service at Beltsville, MD. After 3 years with ARS, he was hired as a Research Molecular Biologist at NCSU and ever since has worked on its Poultry Genomics Laboratory focused on his research interests to identify the genetic basis underlying traits in poultry. Dr. Paul Miller, PhD, Poultry Production Specialist, with more than 40 years of experience in bio-security, value chain, farm diagnostics, market development, management, and poultry nutrition. These skills have been provided to a wide range of programs and organizations including multilateral and bilateral donors, INGOs, and private sector entities in: Chile, Ghana, Indonesia, Iraq, Kenya, Kyrgyzstan, Macedonia, Russia, Sri Lanka, Taiwan, and Uzbekistan. Dr. Miller has worked in building up a private 82 sector poultry industry which has resulted in improved economic performance for smallholder poultry producers. Dr. Miller earned his doctoral degree in Poultry Science and Agriculture & Extension Education from the University of Wisconsin. He also holds a bachelor’s degree in Poultry Industry from California Polytechnic State University. GIP-IL Program Management Structure 83 Conflict of Interest Statements 84 85 ANNEX 9. STATEMENT OF DIFFERENCE 86 87