Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change (Leader with Associates Cooperative Agreement No. AID-OAA-L-10-00001) Report of the External Evaluation Team submitted to the Bureau for Food Security, US Agency for International Development March 24, 2014 Photo credits: Karen Brown and Joyce Turk Dr. Karen Brown (team leader) Assistant Vice President for International Scholarship Director, Interdisciplinary Center for the Study of Global Change University of Minnesota Dr. Claire Heffernan Director Low Carbon Livestock Trust Senior Visiting Fellow, SSEE, University of Oxford Dr. Wyn Richards Independent Consultant in Livestock Research for Development Agricultural Development Practice (Associate of the NRGroup), UK Table of Contents Acknowledgements.......................................................................................................... 2 List of Acronyms............................................................................................................... 3 List of Tables .................................................................................................................... 4 List of Figures................................................................................................................... 5 Executive Summary .......................................................................................................... 6 Introduction................................................................................................................... 11 Background and Context............................................................................................................................................ 11 ALSCC Innovation Lab Aims and Priorities......................................................................................................... 12 External Evaluation Aims and Objectives............................................................................................................ 14 EET Approach and Methods...................................................................................................................................... 14 Program Management.................................................................................................... 16 Technical leadership.................................................................................................................................................... 16 Administration................................................................................................................................................................ 22 Financial management................................................................................................................................................ 30 Monitoring and evaluation ........................................................................................................................................ 35 Research Program .......................................................................................................... 37 Program Future .............................................................................................................. 58 APPENDICES ................................................................................................................... 65 Appendix A: EET Scope of Work ..........................................................................................................................666 Appendix B: External Evaluation Team Biographies...................................................................................811 Appendix C: Knowledge Gap Table……………………………………………………………………………………….83 Appendix D: Evaluation Plan ….……………………………………………………………………………………………93 Appendix E: Questionnaires...................................................................................................................................106 Appendix F: EET Travel Itinerary and Site Visits……………………………………………………………….. 118 Appendix G: List of Persons Contacted..............................................................................................................127 Appendix H: Documents Reviewed.....................................................................................................................132 Appendix I: ME Comments on Report...............................................................................................................133 1 Acknowledgements This evaluation process and production of this report depended upon the cooperation, knowledge and experience of many people. More than 100 individuals participated in the evaluation process through in-person, telephone and Skype interviews and meetings and/or completion of evaluation questionnaires. The Management Entity staff at Colorado State University made particularly critical contributions through many days of meetings, interviews, document provision and assistance with arranging some of the Team’s international site visit activities. Project Principal Investigators (PIs) on funded long term research projects under the ALSCC Innovation Lab devoted substantial time to completing questionnaires, participating in interviews, facilitating visits to their project field sites and collaborators, and providing information. The host country collaborators (co-investigators, collaborating organizations, and community stakeholders) played crucial roles in enabling the Team to understand national and local level project activities and in many cases volunteered their time and effort to make site visits possible and productive. In addition, the USAID team – Carole Levin, Joyce Turk, Saharah Moon Chapotin, the staff of the Strategic Planning & Performance Management (SPPM) Office - gave the Team indispensable information, feedback and guidance on the evaluation plan and process. The Foreign Agricultural Service staff of the Office of Capacity Building and Development, United States Department of Agriculture, assisted Team members with travel arrangements and logistical information. The University of Missouri staff provided support on contract issues. Interview and questionnaire respondents included Management Entity (ME) staff and several other Colorado State University faculty and senior administrators, contractors engaged by the ME (consultants, internal evaluators), research project PIs and staff, host country collaborators (co-investigators, government agencies and labs, universities, non￾governmental organization partners, and community members), TIRI Scholars and graduate fellows, USAID staff (mission staff in several countries and the AOR in Washington), and program Advisory Board current and former members. We appreciate the time and effort required to assist in the evaluation process and offer sincere thanks to all participants. The statements in the draft report are not representative of the positions of any participants in the evaluation process or of USAID. This report offers the Team’s evaluation of the program based on our interpretation of the data we collected in the evaluation process. We remain responsible for any omissions or errors. 2 List of Acronyms ALSCC Adapting Livestock Systems to Climate Change AOR/AOTR Agreement Officer’s (Technical) Representative AVSF Agronomes et Vétérinaires Sans Frontières BFS Bureau for Food Security BIFAD Board for International Food and Development CA Cooperative Agreement CGIAR Consultative Group on International Agricultural Research CRSP Collaborative Research Support Program CSU Colorado State University EET External Evaluation Team FAO Food and Agriculture Organization of the United Nations GL-CRSP Global Livestock Collaborative Research Support Program GIZ German Agency for Technical Cooperation (Deutsche Gesellschaft für InternationaleZusammenarbeit) HICD Human and Institutional Capacity Development IFPRI International Food Policy Research Institute ILRI International Livestock Research Institute ISRA Institut Senegalais de Recherches Agricoles LTRP Long Term Research Projects LWA Leader with Associates Award MDG Millennium Development Goals ME Management Entity M&E Monitoring and Evaluation NIRS Near Infrared Spectrometer NSF National Science Foundation NGO Non-Governmental Organization PI Principal Investigator RFA Request for Applications RFP Request for Proposals TIRI Targeted Investment in Research Infrastructure US United States USAID United States Agency for International Development USDA United States Department of Agriculture 3 List of Tables Table 1. ALSCC Innovation Lab ME Staff Percent Effort Table 2. Budget and expenditures 2010-2015 Table 3. Expenditure on Capacity Strengthening and Dissemination (to November 2013) Table 4. Feed the Future Research priorities and evaluation scores Table 5. Research vs. development-focused activities and outputs Table 6. LTRP: Capacity Development Table 7. Impact Scores Table 8. In-country Collaborating Institutions by type Table 9. Program Future Decision Scenarios 4 List of Figures Figure 1: ALSCC Innovation Lab Themes (from www.lcccrsp.org) Figure 2. ME Progress Reports 5 Executive Summary The following report is the result of the External Evaluation commissioned by USAID from October, 2013 to March, 2014 to provide an evidence-based assessment of the Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change (ALSCC Innovation Lab). In 2010, Colorado State University was awarded a five-year, $15 million Leader with Associate Award from the U.S. Agency for International Development (USAID) to manage the Collaborative Research Support Program, Adapting Livestock Systems to Climate Change (Livestock-Climate Change CRSP). By 2013, in order to better consolidate the wide range of activities under the umbrella of the Feed the Future Initiative, those research programs collectively known as the Collaborative Research Support Programs (CRSPs) were rebranded as Innovation labs. Therefore, the Adapting Livestock Systems to Climate Change (Livestock-Climate Change) CRSP transitioned to the Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change. The remit of the ALSCC Innovation Lab is to examine the role of climate change as an important determinant of animal, human, and environmental health and aid the resiliency and adaptation of vulnerable livestock-keeping communities in the Global South. The research area, goals and priorities of the program are detailed as follows: 1 The Livestock-Climate Change Innovation Lab seeks to increase resilience and augment the income of livestock producers in regions where agricultural systems are changing, available resources are shrinking, and climate is having an impact. It supports research that aids individuals and communities to make choices and take actions that lead to sustainable livelihoods in the face of climate change. USAID commissioned the external performance evaluation in order to provide direct recommendations to both the ME and USAID on any program implementation issues. A three-person external evaluation team (EET) was designated to conduct this work consisting of a senior international programs administrator and two global livestock experts (see Appendix B for EET biographies). The EET conducted a multi-method evaluation of the program management and research dimensions (see Appendix C for the Evaluation Plan), including a systematic review of secondary data (i.e. program documents including reports, proposals, contracts, and plans; program website; program financial data) followed by a series of meetings, interviews, phone conferences, and site visits at the CSU location of the ME as well as in each of the countries where ALSCC Innovation Lab-funded projects currently operate (Nepal, Tanzania, Kenya, Ethiopia, and Senegal). Approximately 100 individuals participated in the review process by completing questionnaires or interviews (in person, phone or Skype) or participating in meetings with the EET. Appendices D, E and F contain details of travel to site visits, persons contacted, and documents reviewed. 1 http://lcccrsp.org/approach/ 6 A critical outcome of the review is to help inform USAID’s decision on whether to extend the Livestock-Climate Change Innovation Lab funding for a second five year phase or to terminate the project at the conclusion of its current phase in 2015 (see Appendix A for Scope of Work for EET). Other outcomes, if deemed appropriate, are to create a series of forward-looking recommendations for program changes or improvements. The ALSCC Innovation Lab achievements to date are as follows: • Seed Grant Projects: Twelve were funded in East and West Africa, Mongolia and Nepal in years one and two. • Long Term Research Projects: Ten were funded and are currently operating in East Africa (4), West Africa (2) and Nepal (4). • TIRI Scholars: Ten have been funded in East Africa with an additional 18 Scholars funded in Nepal. Six to eight additional TIRI Scholars are likely to be funded in Senegal based upon review of revised research proposals on a team model. • Graduate Fellows: Fifteen from Africa and Nepal are being supported for post￾graduate qualifications either in the US or in host country research/academic institutions. • TIRI Scholar Workshops: Six organized in Nepal, Ethiopia and Senegal. • PI Group/coordination meetings (including some Co-PIs): Two held (May 2011 and October 2013). • Communication and dissemination: The ME created an attractive, informative and current website; also Facebook and LinkedIn profiles. In addition, Research Communiques and other publications serve to disseminate program accomplishments and research findings. ALSCC Innovation Lab accomplishments in launching a set of research projects and capacity development initiatives made important progress toward program goals. Individual long￾term research projects offer a range of approaches to understanding the climate change￾related challenges facing livestock keepers and data gathering that could be used to shape future interventions to enhance adaptive capacity. Key areas of focus include pastoralists and how their livelihood practices are changing in the face of climate change; identifying varieties of and cultivation methods for improved fodder or income-generating crops; enhancing household- and community-level capacity with respect to livestock husbandry and management; and collecting weather data to build models of climate variability and ultimately climate change. However, the range of projects is broad, resulting in an overall research portfolio that is not sharply focused or coordinated. Cross-cutting themes of gender, nutrition and climate change were not sufficiently integrated in LTRP RFPs and therefore, in initial project designs. To promote their inclusion among the LTRP a range of thematic consultants were hired. The uptake of the current cross-cutting advisory services appears low. A clear research strategy driving the selection and guidance of individual research projects – integrating climate change, gender and nutrition from the outset - would serve to make the portfolio as a whole greater than the sum of its parts. 7 Collaboration and outreach activities by the ME include PI meetings, linkages with some USAID missions, and dissemination of research findings via a comprehensive and attractive website, recently developed research briefs, and scholarly publications by PIs. Incentive grants to encourage cross-project collaboration by PIs were introduced in 2013. These activities must be more extensive and regular to offer significant impact, including ME￾driven connections to host country USAID mission staff and related donors to link ALSCC￾funded activities to the broader livestock and development context. Monitoring and evaluation (M&E) data remains critical to understanding and documenting program impact. At the program level, M&E appears ad-hoc and in its present form does not produce sufficient and high quality impact data. At the individual project level, M&E occurs but is not consistent across the portfolio, with many projects having strong M&E components but others lacking in focus. While it is difficult to assess development impact of 3- to 5-year projects, a cohesive M&E strategy and guidance at the ME level would assist in better data collection and ultimately produce research more responsive to Feed the Future priorities and community demand. The ALSCC Innovation Lab has invested heavily in capacity development initiatives including TIRI Scholars, graduate fellows, and various training courses and workshops in host countries for research staff, government and NGO staff, and farmers. These HICD investments promise to create impact in terms of individual scholar capacity and community-level capacity (e.g., drip irrigation and fodder crop demonstrations; artificial insemination training). The HICD dimension of the ALSCC Innovation Lab activities must be more closely and strategically connected to the research portfolio and designed for sustainability to maximize impact. The large proportion of the ME’s time presently spent on the TIRI scholar program should be rationalized and better focused on developing a strong M&E framework as detailed above. The ME team at Colorado State University (CSU) has faced several significant challenges in program implementation. This has had an enduring effect on program leadership structures and expenditures. Despite changes in program staff and activities, the current design and management of the program portfolio has not addressed program goals in a consistent and coherent manner. In addition, stronger oversight of program activities and progress is needed to ensure that funded projects adequately incorporate components of climate change, nutrition and gender as core components and that M&E activities and indicators tie closely to program objectives. An overstretched leadership structure and organizational challenges persist. Presently, there is the significant under-spend of budgeted funds and an uneven allocation of program funds in research and HICD areas. The ALSCC Innovation Lab Advisory Board has included a strong pool of expertise in livestock and climate change as they relate to development. The Advisory Board could play a much more substantial part in program governance and direction to include identification of new and emerging opportunities in the field. Its role would be enhanced with reduced turnover among Board members and a revised governance structure. CSU also offers a range of resources in centers or clusters of relevant expertise for the ALSCC that could be better utilized to leverage ALSCC Innovation Lab investments. 8 Summary of Recommendations The EET sees the overarching goal for the program future as ensuring the establishment of an ME with a priority on formulating and supporting a cohesive research portfolio with associated HICD activities and tied to long term and sustainable improvements in adaptive capacity of poor livestock keepers in the context of climate change. Research focused on livestock and climate change is essential to a pro-poor international development agenda. With strong leadership the portfolio of projects has the potential to have a greater collective impact (the whole will be more than the sum of its parts). Over the next 13 months, considerable investment in and implementation of gap filling projects has the potential to have a significant collective impact. The focus must be on filling core researchable constraints important to better understanding the effects of climate change on poor livestock keepers in South Asia and Africa. While the EET believes that a restructured ME with deep expertise in international livestock development is necessary to maximize the potential of the ALSCC Innovation Lab, we support the strong continuation of the program research area into a reconfigured Phase 2. The potential of a strong research agenda in this area, with attendant impacts was recognized as beneficial by a wide range of stakeholders interviewed by the EET. Clearly, the investment represents a crucial element of USAID’s international development portfolio. USAID and a few bilateral partners are among a small group of donors still prepared to fund research aimed at the supporting the livelihoods of increasingly vulnerable populations of poor livestock keepers. Positive impacts resulting from the ALSCC Innovation Lab are likely to shape future investments by other donors. The EET, while recognizing accomplishments made by the ALSCC Innovation Lab to date, recommends a restructuring of the ME and related research and capacity development activities. During year five of the current Phase I program, the EET recommends formulating a plan to reconfigure the ME. One strategy to accomplish this aim is to issue an open call for proposals to compete for Phase II. Key elements of the EET’s recommendations for the program’s future in a reconfigured Phase II include: 1. A focused research strategy emphasizing adaptive capacity for poor livestock keepers and corresponding to Feed the Future priorities should frame and structure all program activities and expenditures. 2. Research projects should integrate gender and nutrition dimensions and expertise from the start, climate change must be inherent to funded projects, and community demands and interests must shape program focus. 3. Capacity development activities should be connected to the research dimensions of the projects and focused on durability. 4. The ME leadership team should proactively develop and guide outreach and communications to internal university and external (USAID Missions, NGOs, universities, policy makers, public and private sector partners) stakeholders as well as to the global donor and academic community in this field. 9 5. The ME team’s work should be guided by clear position descriptions tied to program activities and be supported by adequate time allocations for the Director and Associate Director. 6. In order to assess program impact, a dedicated staff member should implement an evidence-based M&E strategy. In the remaining months of Phase I, the following recommendations demand urgent attention: 1. The ME identifies and supports a leadership team including a Director devoting a minimum of 50% time to the program and an Associate Director to manage day-to-day program operations at 100%. 2. The Director should engage in outreach to external stakeholders, raise awareness of program accomplishments to date and connect funded research with relevant policy makers. 3. The role of cross-cutting thematic advisors should be re-evaluated. 4. The program database access issue must be resolved and the appropriate collection and analysis of recognized impact indicators undertaken. 5. The ME should provide clear guidance to PIs and co-PIs of research projects to structure reporting on projects and collection of Feed the Future indicator data. 6. Program expenditures must be reviewed and reassessed for balance among research, capacity development and outreach as well as to consider budget revisions for the remaining months of the program. 7. The EET recommends that USAID affords a measure of flexibility in budget revision and the consideration of shorter term project no-cost extensions to allow the completion and success of currently funded projects as well as to capture the benefit of investments made to date and planned program-level activities over the coming year. These recommendations appear in greater detail in the body of this report. 10 Introduction Currently it is estimated that two-thirds of the global poor depend partially or fully on livestock for their food security. Notwithstanding the predicted increase in the demand for livestock products by southern consumers, livestock in the coming decades are likely to remain a crucial livelihood activity for the poor. As the global population increases, it is likely that livestock keeping will remain one of the few accessible livelihoods. Further, livestock development projects and programs have been shown to be one of the most effective means of alleviating poverty. Nevertheless, livestock have a range of both positive and negative impacts on the environment. Indisputably, livestock contribute to global greenhouse gas emissions. Conversely, livestock provide a range of ecosystem services from animal traction to Carbon recycling via the use of manure in crop production. Disaggregating the extent of both of these contributions to global climate change is a crucial knowledge gap. Equally, on the farmer-level there is a recognized need to invest in technologies, which decrease adverse environmental impacts while enhancing sustainable milk and meat production. However, investment in the global livestock sector, particularly within the research sub-sector has declined in recent decades. Within this context of rising knowledge gaps and declining support, The Colorado State University (CSU)-led Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change (hereafter called the ALSCC Innovation Lab or the Livestock-Climate Change Innovation Lab) is one of the few operational global research programs on livestock and climate change. The aim of the ALSCC Innovation Lab is to support integrated research to help small-scale livestock holders adapt to environmental and health impacts of climate change in Sub-Saharan Africa and South Asia. As such, the ALSCC Innovation Lab is both timely and offers a sentinel opportunity to forge new knowledge in this field. Background and Context In 2010, Colorado State University received a five-year, $15 million Leader with Associate Award from the U.S. Agency for International Development (USAID) to manage the Collaborative Research Support Program, Adapting Livestock Systems to Climate Change (Livestock-Climate Change CRSP). The CRSPs were created under Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage U.S. land grant and other eligible universities in addressing the needs of developing nations while also contributing to U.S. food security and agricultural development. Originally established in 1978 as the Small Ruminant CRSP, the Global Livestock CRSP was one of nine CRSP programs developed under Title XII. In 2000, Title XII was reauthorized, enabling the continuation of the CRSPs as one of several types of U.S. university-led research efforts. In 2013, USAID announced that “[a]s part of a larger strategy designed to integrate the many diverse aspects of its Feed the Future Initiative, USAID’s agricultural research 11 programs that were collectively known as the Collaborative Research Support Programs (CRSPs) are being rebranded. Each CRSP will henceforth be known as the “Feed the Future Innovation Lab for Collaborative Research on [a subject area].”2 This transition of the CRSP to Innovation Labs responded to two key recommendations from a Board for International Food and Agricultural Development (BIFAD) commissioned CRSP review: • To develop an overarching and coordinated strategy for engaging U.S. universities in agriculture and food security research and human and institutional capacity development that includes the CRSPs as a central component; and • To leverage the impact of CRSP investments by strengthening links across universities, U.S. government, global programs, foundations, and other donors. The Innovation Labs are viewed as integral to the new Feed the Future Food Security Innovation Center, established to implement the Feed the Future Global Hunger and Food Security Research Strategy. 3 Following suit, the Adapting Livestock Systems to Climate Change (Livestock-Climate Change CRSP) transitioned to the Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change. The ALSCC Innovation Lab remains funded under the authorization of Title XII. ALSCC Innovation Lab Aims and Priorities The ALSCC Innovation Lab focuses on climate change as an important determinant of animal, human, and environmental health. The research area, goals and priorities as detailed on the program website and in related documents are as follows: 4 The Livestock-Climate Change Innovation Lab seeks to increase resilience and augment the income of livestock producers in regions where agricultural systems are changing, available resources are shrinking, and climate is having an impact. It supports research that aids individuals and communities to make choices and take actions that lead to sustainable livelihoods in the face of climate change. The ALSCC Innovation Lab focuses on the following key principles to achieve its goals: • Improving the health and productivity of livestock of the rural poor, working with small-scale farmers to support their efforts to nourish their families and increase village resiliency. • Supporting research that is informed by the local realities of small-scale farmers, with an interest in evidence-based solutions that are relevant and affordable. • Increasing productivity that enhances animal, human, and environmental health, 2 http://crsps.net/key-facts-2/#Timeline%20and%20History 3http://transition.usaid.gov/our_work/agriculture/bifad/BIFADREVIEW_CRSP_August2012.pdf 4 http://lcccrsp.org/approach/ 12 with a focus on innovative approaches that increase productivity, enhance health, and conserve resources. • Funding research where women play a central role, recognizing that women are fundamental to the success of farm-based initiatives and expecting research solutions to address gender gaps and address inequalities. Core Program research themes are described as follows (also expressed in Figure 1): (1) Climate extremes and long term change (2) Pro-poor value chains, market access and reliability (3) Animal health: disease, distribution and resiliency (4) Ecosystem health: resiliency of socio-ecological systems Figure 1: ALSCC Innovation Lab Themes (from www.lcccrsp.org) The ALSCC Innovation Lab achievements to date are as follows: • Seed Grant Projects: Twelve were funded in East and West Africa, Mongolia and Nepal in years one and two. • Long Term Research Projects: Ten were funded and are currently operating in East Africa (4), West Africa (2) and Nepal (4). • TIRI Scholars: Ten have been funded in East Africa with an additional 18 Scholars funded in Nepal. Six to eight additional TIRI Scholars are likely to be funded in Senegal based upon review of revised research proposals on a team model. • Graduate Fellows: Fifteen from Africa and Nepal are being supported for post￾graduate qualifications either in the US or in host country research/academic institutions. 13 • TIRI Scholar Workshops: Six organized in Nepal, Ethiopia and Senegal. • PI Group/coordination meetings (including some Co-PIs): Two held in May 2011 and October 2013. • Communication and dissemination: The ME created an attractive, informative and current website; also Facebook and LinkedIn profiles. In addition, Research Communiques and other publications serve to disseminate program accomplishments and research findings. The ME leadership at Colorado State University (CSU) recently commissioned a systems￾oriented internal evaluation to look across the research projects funded. In October 2013, USAID commissioned an external review to assess program management, research depth, breadth and impacts. This report provides the results of the external evaluation. External Evaluation Aims and Objectives The overall aim of the external performance evaluation is to provide an evidence-based assessment of ALSCC Innovation Lab progress to date and direct recommendations to both the ME and USAID on any program implementation issues. A critical outcome of the review is to help inform USAID’s decision on whether to extend the Livestock-Climate Change Innovation Lab funding for a second five year phase or to terminate the project at the conclusion of its current phase in 2015 (see Appendix A for Scope of Work for EET). Other outcomes, if deemed appropriate, are to create a series of forward-looking recommendations for program changes or improvements. USAID commissioned a three-person external evaluation team (EET) to conduct this work consisting of a senior international programs administrator and two livestock experts (see Appendix B for EET biographies). During the period October 2013 to March 2014, the EET undertook a series of meetings, conference calls, key informant interviews, site visits and a systematic document review in order to produce an evaluation report containing findings, conclusions and recommendations related to the Livestock-Climate Change Innovation Lab. EET Approach and Methods The EET conducted a multi-method evaluation of the program management and research dimensions of the ALSCC Innovation Lab (see Appendix C for the Evaluation Plan). The first stage of data collection involved a review of secondary data (program documents including reports, proposals, contracts, and plans; program website; program financial data). Following this desk review, the EET undertook a series of meetings, interviews, phone conferences, and site visits at the CSU location of the ME as well as in each of the countries where ALSCC Innovation Lab-funded projects currently operate (Nepal, Tanzania, Kenya, Ethiopia, and Senegal). Appendices D, E and F contain details of travel to site visits, persons contacted, and documents reviewed. 14 In reviewing the secondary data, the EET systematically reviewed and analyzed documents to identify strengths and challenges of the ALSCC Innovation Lab for further investigation in primary data collection. Particular attention was paid to analysis of issues emerging across multiple contexts or aspects of the project, and to the identification of indicator data based on our evaluation plan outcome measures. Indicators – both those compiled by EET￾collected data and those provided by the ME and projects - were analyzed for correspondence with program objectives and Feed the Future objectives. Primary data collection took place through site visits to the ME and to each of the countries currently involved in the Innovation Lab (Nepal, Tanzania, Ethiopia, Kenya and Senegal). The EET interviewed or collected questionnaire data from more than 90 individuals and also met with dozens of community stakeholders and farmers during site visits. During the EET visit to the ME at Colorado State University, we held meetings and conducted interviews with ME leadership and program staff and consultants as well as other senior academic and administrative leaders. The ME site visit also enabled the EET to collect budget data and additional documentation. International site visits varied in length and scope. Across the sites, the EET visited components of, or collaborators with, the Long Term Research Projects (LTRPs) in these countries, interviewed TIRI scholars and graduate trainees, interviewed PIs and co￾investigators, field staff and leadership from partner organizations, government agencies/ministries, USAID Mission staff, and community stakeholders. The EET also observed two training workshops (one in Nepal and one in Senegal) for TIRI Scholars and graduate fellows, visited project field sites where research and development activities were underway in several countries, and visited a lab where nutrition research is supported by a project in the case of Senegal. Appendix D contains international travel debriefs detailing activities in host countries. Questionnaires were devised for specific stakeholders to assist with ensuring consistency across evaluation sites (see Appendix C for questionnaires). Questionnaires were administered in person in a semi-structured interview format in some cases, and were returned with written responses in cases where meetings were not possible. In general, these focus on: observing and evaluating the quality of research design and implementation; the extent and quality of capacity building activities; integration of research and capacity building activities in overarching project logic; evidence of addressing cross-cutting themes (gender, nutrition and climate change); monitoring and evaluation activities; impact evaluation; management practices; communications strategy; and community participation in project design. Quantitative indicators were developed and analyzed for correspondence with project objectives and Feed the Future objectives. Additional quantitative data collected on training opportunities and outcomes as well as project-related data are provided here as descriptive statistics linked to project objectives. Qualitative data (interview data and field notes of project observation) were analyzed to address evaluation questions as outlined in EET Scope of Work concerning themes of Program Management (technical leadership; administration; financial management, and monitoring and evaluation) and Research Program (research depth, breadth and rigor; 15 collaboration, outreach and technology dissemination; human and institutional capacity building; gender inclusion). Program Management Technical leadership Assess the ME’s technical leadership of the program, including how it has built on past investments while having a vision for new opportunities and constraints; engaged partners in the U.S. and overseas, including USAID Missions, CGIAR centers and NGOs; balanced research, technology dissemination, training and capacity building demands; and promoted scientific collaboration and exchange among all its partners. 1. Overall program leadership Findings: Program leadership by the ME faced a very challenging start with personnel conflicts preventing most activity in the first year. PIs noted that shifting reporting structures early in their interactions with the ME, and continued lack of clarity around key points of contact at the ME, created some difficulties. A new leadership team after year one made some improvements, but an overstretched leadership structure persists. Recent personnel changes at the ME create additional uncertainty about leadership capacity. The Director has a time commitment of approximately 33% (this might increase to 50%) and the Deputy Director (recently resigned) had a time commitment of 90%-95%. CSU’s institutional buy-in to the project could significantly enhance program impact. However, the EET saw few sustained efforts to integrate this program in the broader university context and with related initiatives or to draw upon the university’s existing expertise in related scientific fields. For example, CSU has developed an internationalization strategy that emphasizes strategic geographically defined partnerships. In seeking to advance partnerships in Africa where they had little activity, Ethiopia and Kenya were identified for partnership development. While the ALSCC Innovation Lab works in both of these countries, CSU senior administrators reported that this work was not leveraged or linked to CSU’s institutional strategy and activities. A notable exception to a relatively low level of broader institutional impact was the ME staff’s involvement in CSU’s One Health initiative. In addition, a number of centers or clusters of relevant expertise at CSU are not linked with the program. CSU mentioned these resources in their original RFA submission, but the ALSCC Innovation Lab appears not to be integrated with these resources. Conclusions: Disputes among the original management team resulted in delays in program implementation. A new management team showed leadership and implemented a first round of funded projects as one-year seed grants. However, subsequent design and management of the program portfolio has not addressed program goals in a consistent and coherent manner In addition, stronger oversight of program activities and progress is needed to ensure that funded projects adequately incorporate components of climate change, nutrition and gender as core components. 16 The multiple talents of CSU academic staff and CSU’s institutional strengths as an extensive research university are not effectively linked to this program. In short, CSU’s institutional resources could be much better leveraged with respect to this program for the benefit of the program and the institution as a whole. Technical Leadership Recommendation 1: The ME must establish and support a strong leadership team with linkages to the university’s broader context and strategy. 2. Clearly articulated research vision a. Indicator: Research plan or strategy demonstrates overarching vision b. Indicator: Project reports and communication, RFPs and awards demonstrate linkage between individual project and overarching strategy Findings: In terms of leadership of the research function, it is important to note that some PIs appreciated that the ME did not attempt to micromanage their projects and allowed them research independence. Others believed that the program could benefit from greater guidance from the ME on the key contribution of project outputs to the achievement of the program goal. A general theme among program stakeholders was that the program lacked a cohesive strategy (program logic) – a key leadership function. Individual LTRPs did not always connect clearly to an overall strategy. PIs generally reported being unfamiliar with the program logframe that was developed by ME to align ALSCC Innovation Lab activities with Feed the Future objectives. Efforts by PIs to report Feed the Future indicator data helped to link the program’s activities to the broader USAID framework, although the selected indicators did not align well with the program goals. Conclusions: Several project PIs are world leaders in their particular research fields yet it is hard to see how the program will generate a coherent set of outputs and thereby impacts. The current mix of projects is unlikely to achieve program goals. PIs appropriately bring a focus on achieving project goals and research publications. ME leadership and expertise is needed to focus on overall program goals. While acknowledging the supportive responses and enthusiasm of the ME team, the EET believes that the program would benefit from better alignment of the experience and time allocation of the leadership team with the demands of leading an international development research program of this complexity. Insufficient person-days were allocated for program leadership especially for the Director role, leaving a Deputy Director position overburdened by the extensive time demands of portfolio management. Technical Leadership Recommendation 2: The ME must establish an overall strategic vision for the ALSCC Innovation Lab that addresses pressing issues in livestock and climate change adaptation research while also defining a clear niche for this program. In particular, the ME needs to significantly improve the coherence of project selection with program goals. Technical Leadership Recommendation 3: ME needs to coordinate and integrate program initiatives and outputs which address common challenges, especially those working across similar production systems, e.g. pastoralism, or in a common country or region. 17 3. Increased investment in livestock-climate change related research and capacity development through engaging partners Findings: a. Amount of co-funding from CSU, other donors, and partners There is little evidence of increased investment in related research and capacity development as a result of program activities. Several PIs report that the ALSCC Innovation Lab funded research project is nested within or connected to other funded work (e.g., NSF funded research). However, it appears that co-funding does not depend upon the ALSCC Innovation Lab component of the project and existed prior to or was independent of the ALSCC Innovation Lab award (in short, the EET did not find evidence that ALSCC Innovation Lab funding was used to leverage significant additional funding). b. Number and value of associate awards One Associate Award from a USAID mission was reported for this program. This award, from the Mali mission in July 2010, is not currently in effect due to the security situation in Mali. The focus of this Associates Award was on the improvement of productivity and income of livestock producers in northern regions of Mali. Technical Leadership Recommendation 4: As part of a coordinated strategy to engage USAID missions, the ME leadership team should actively identify opportunities to pursue associate awards. 4. Increased scientific collaboration and exchange a. Number and type of opportunities created for scientific collaboration (e.g., Regional Innovation Consortium (RIC) workshops, TIRI Scholar workshops, PI meetings) The ME took a number of steps in the most recent year of the program to provide opportunities for collaboration. Initiatives are outlined in the introduction to this report (under ME accomplishments): TIRI Scholar workshops and all-PI meetings are key examples of this kind of activity. i) TIRI Scholars initiative Findings: With regard to the design and implementation of new opportunities and innovations, ME and in-country mentors have devoted considerable support to the TIRI initiatives in East Africa and Nepal. However, management time spent on this initiative occurred at the expense of closer management oversight of the LTRPs and focus on creating and implementing a robust impact and M&E strategy. Notwithstanding, the TIRI initiative has generated a cadre of well-trained and dynamic young scientists in East Africa and Nepal who can now better address development challenges. The TIRI initiative in West Africa is about to be implemented although we foresee that their research accomplishments will not be as significant as those from the other regions due to the comparative lack of research experience among the group, on the whole. 18 Conclusions: The EET believes that to fully justify ME’s time and financial investment in the TIRI initiative, significant benefits should be accrued by the program as well as by the individual TIRI Scholars. However, very few linkages currently exist between TIRI scholars and LTRPs despite considerable overlap and relevance of the former’s research outputs. This is a potential missed opportunity for all parties. Technical Leadership Recommendation 5: ME provides continuing support to TIRI Scholars for a further 13 months to enable them to disseminate their work, establish formal linkages with and contribute to the activities of pertinent in-country LTRPs – at least until the end of Phase 1. This may require the ME to proactively facilitate collaboration with the PIs and in￾country teams. ii) Regional Innovation Centers (RIC) Findings: The RIC held promise as an innovative feature in the ME’s proposal document. The EET found that the RIC concept has not been adequately developed nor promoted, and many program stakeholders were unfamiliar with the concept. Conclusions: The RIC remains a missed opportunity for increasing regional collaboration, sustaining scientific development, promoting research communication, and providing a forum for information exchange and mentoring in regional contexts. Technical Leadership Recommendation 6: ME needs to urgently address the status of the RIC initiative and realize the concept in the target regions though persuasive promotion, the identity of local champions to run the initiative and the provision of appropriate low level funding for facilitation and meeting expenses. iii) PI meetings Findings: PIs reported the two PI meetings in 2011 and 2013 offered notable value by providing an opportunity for sharing ideas and forming new alliances and research partnerships among the LTRPs. The meetings also enabled the PIs and Co-PIs to become more aware of the requirements of the program (project deliverables, M&E indicators, etc.). In particular, the 2013 PI meeting offered some resources designed to integrate the projects. However, PIs felt that collaborative program-level impetus had been lost in the intervening two-year period between the meetings and that some interventions offered in 2013 came at a point when projects could not fully benefit from them. Conclusions: Collaboration among PIs has not been consistently and intentionally promoted, but offers substantial potential benefit. Technical Leadership Recommendation 7: Annual face-to-face PI meetings are required to enable and sustain program and project coordination, effective communication and increased partnership. These meetings could be complemented with more frequent ME￾organized online meetings around specific topics and involve in-country Co-PIs. b. New or enhanced collaborations developed Findings: In responding to a question about linkages across projects and with other initiatives/donors, PIs reported that they typically formed these connections themselves (often based on pre-existing collaboration) and not at the behest of the ME team. Some 19 projects are continuations of previously funded projects from the GL-CRSP and bring pre￾existing ties among researchers. The ME has promoted connections between researchers and host country collaborators in a few cases, but cross-project connections seem to result largely from researcher efforts rather than ME guidance. However, the ME is currently reviewing one or more proposals for incentive grants to encourage PIs to collaborate with each other. In addition to the two PI meetings, other efforts by the ME to promote scientific collaboration among program stakeholders included the appointments of consultants with specialized skills related to gender analysis, nutrition, climate change, and biostatistics; these were contracted in 2013 to provide advisory services to funded projects. Uptake of this service by projects has been uneven. Some PIs indicated frustration with the consultants appearing late in the process and did not see a role for them in already￾developed projects. Several PIs met the consultants at the 2013 PIs meeting but have not interacted with them since that meeting. Some others have interacted with the consultants several times and find their insights helpful. In particular three projects have benefited from the gender and/or nutrition consultants, and one project received detailed assistance from the climate change consultant. The biostatistics consultant has been closely involved with the graduate fellows/TIRI Scholars in Nepal in strengthening research proposals. The EET found that the ME has played a small role in fostering linkages with other donors and has unevenly forged connections with USAID Missions. Some Mission staff remain largely unfamiliar with the ME personnel and ALSCC Innovation Lab projects. The EET also discovered little evidence of proactive ME efforts to identify emerging opportunities with other donors and public and private sector collaborators in the field. In particular, systematic linkages with policy makers are not evident. Conclusions: There is currently significant overlap in project theme, activity and location within countries and in project theme and activity between countries. In this context, a collaborative initiative is likely to harmonize activities and outputs and yield additional benefits to the program. Technical Leadership Recommendation 8: The ME should proactively collaborate with LTRPs in establishing new partnerships with policy making institutions and with those which can facilitate the transfer of adaptation strategies to target livestock keepers and increase their resilience to climate change (e.g., with private sector, government agencies, NGOs and financial institutions). c. Number and type of collaborative publications or funding awards received or in process, and d. Scope and content of research dissemination Findings: See research section of report for evidence concerning publications and research dissemination at the project level. In addition, the ME has recently undertaken additional efforts to promote research findings through the publication of reports on research projects on the ALSCC Innovation Lab website and in newsletter-type publications. Some PIs collaborate informally and occasionally co-publish. These relationships appear to pre-date the ALSCC Innovation Lab. 20 Conclusions: Research dissemination efforts remain key to research-for-development efforts. A clear strategy for disseminating findings to policy makers, community level stakeholders, and academic communities would enhance the impact of project on pro-poor development initiatives. Technical Leadership Recommendation: See Research Section for recommendations related to research dissemination. 5. Clear organizational leadership structure with well defined roles Findings: a. PI/Director and other program leadership work in clearly defined areas of responsibility The ME team roles are not always clearly defined, particularly in the leadership team. The role of the Director, Co-Director/Deputy Director, and regional team leaders need further clarification. In addition, the thematic consultants (gender, nutrition, climate change) need a clear mandate with respect to a (revised) role in view of the mixed reaction to their contribution by the PIs and the near-end status of the program. Balance and responsibility across functional areas (M&E, outreach and dissemination, research, HICD) is not well defined and in need of urgent attention. b. Position descriptions for PI/Director and other program leadership positions establish roles and responsibilities Staffing protocol for altered ME team offers sparse position descriptions. Position descriptions do not appear to be closely tied to specific program goals and objectives. Conclusions: The ME’s efforts to address issues in the research portfolio, such as insufficient attention to gender analysis or climate change frameworks, have been compromised by the lack of a clear mandate for the thematic consultants as well as their late introduction to projects. The ME leadership team’s lack of sharply defined roles and responsibilities leaves some aspects of the project under-resourced (e.g., outreach to USAID missions is uneven in the fourth year of the program). Well defined roles for program leadership would assist the ME in balancing the various project components (research, HICD, outreach and dissemination). Technical Leadership Recommendation 9: The ME immediately develops clear and detailed position descriptions for the leadership team members with the Director overseeing the fulfillment of these responsibilities and uses these to employ an Associate Director for the remaining 13 months of the Phase I award. 21 Lessons Learned • The ALSCC Innovation Lab was hampered in launching program activities and expenditures in the first year or more due to personnel challenges. This has had an enduring effect on program leadership structures and program expenditures. • A clear research strategy must drive program activities and expenditures, including shaping RFPs and the selection of research projects as well as the nature of HICD and outreach activities, in order to construct a cohesive portfolio with significant impact in the field. • ME staffing structures can affect the implementation of program activities in a variety of ways and must be intentionally linked to program goals and objectives. The ME team must also allocate sufficient time and resources to specific positions tied to program goals. • In order to assess program fit with Feed the Future priorities and impact at local, national and global levels, strong data collection (especially M&E data) and management guidelines and practices are critical. • Budget management practices must include not only adherence to USAID financial regulations and processes (which the ME team has accomplished), but also explicit attention to balance across areas of activity (e.g., research, HICD, communications) and spending projections or budget revisions to make effective use of the funds provided. • Collaboration across funded activities as well as linkages to potential external partners (USAID missions, other donor agencies, policymakers) remain crucial pathways to impact on global and national policy as well as in the research arena. Strong ME-level leadership can drive these connections and raise the profile and impact of the ALSCC Innovation Lab initiatives. Administration Assess the ME’s administration and management of the Livestock-Climate Change Innovation Lab taking into consideration what systems are in place to ensure research activities are on track in accordance with program goals; roles and functions of advisory committees; and appropriate staffing levels, functions and level of effort. 1. Research activities are effectively managed and administered and on track a. Tracking research activities and linking to program goals Findings: The program research and capacity development portfolio lacks a clear strategy. While funded projects are sometimes loosely connected by content area (e.g., pastoralist community livestock management and health) and region, these connections do not appear to be strategic and the projects do not form a coherent whole. The absence of a strong and coherent program focus hinders efforts to link individual projects to overall goals. The 22 program research goals appear to have changed more than once, exacerbating the lack of clear focus. The ME’s activities to promote program research – calls for proposals, program logframe, program research strategies, selection of funded projects – do not work together to create a cohesive portfolio. Individual projects have strong mechanisms to track research activities and progress. As a Management Entity, CSU has played a relatively small part in identifying and fostering cross￾project linkages. Linkages that do exist across projects appear to result more from individual PI efforts or pre-existing connections (some funded projects are continuations or extensions of projects that have been in process for years and often have other funding for core components). To enhance impact, the ME should focus the overall research portfolio to create a strategically focused and linked portfolio rather than a loosely defined collection of related efforts. A program logframe was established later in the project, but it does not seem to be used to drive project development as indicated by the lack of knowledge of the logframe by project PIs and the lack of fit with logframe goals of individual projects. Most PIs expressed confidence that their individual projects would ultimately accomplish their goals on or close to their original time lines. Several of the currently funded long-term research projects lag in implementation due to administrative and funding reasons. Two projects have fallen behind schedule for in-country security reasons (Mali and Kenya). Some PIs viewed the breadth of the research portfolio as a strength, while several see a lack of focus. The ME is currently involved in efforts to identify areas of focus and future directions, in part through an internal evaluation process. The concurrently running processes of internal and external evaluation were poorly timed and created confusion and some resistance among project stakeholders. A large number of individuals noted that the processes and time lines for the two evaluations, and the distinction between them, were not clearly communicated. Some stakeholders to whom the EET spoke indicated that they were not really certain about the purpose of the evaluation or even the fact that an external evaluation was in progress. Others were informed of the evaluation but had difficulty sorting out the two evaluation processes and found the multiple interactions to be demanding. In order to maintain a high standard in collecting data for the external evaluation, the EET requested separation from internal evaluation activities on more than one occasion. The ME faced the challenge of being required to integrate Feed the Future metrics and goals after the program was already underway. For instance, projects originally envisioned for Mongolia could no longer be supported as long-term research projects because Mongolia was not a Feed the Future priority country. This challenge was outside the control of the ME and they have made efforts to align their project with Feed the Future objectives. These efforts, including selection of indicators, are addressed in greater detail in the research program section of this report. Administration Recommendation 1: ME should establish an overarching research strategy and focus in order to provide framework for tracking project progress in meeting overall program goals. Administration Recommendation: See research section of this report for further recommendations concerning tracking research outputs and impact. 23 b. Project staff, consultants and collaborators report effective and regular communication with project management Findings: Communications and interactions with ME staff are uneven across project PIs. While some report that the ME can be responsive to specific inquiries and that they communicate regularly (several times per month or sometimes weekly in two cases), others report that communications are ad hoc. A minority of PIs reported some interaction around scientific issues, with most reporting that interactions with the ME focus on logistical concerns. PIs report that email is the primary form of communication with ME staff and primarily with the Deputy Director. Occasional phone calls and two PI meetings have supplemented more routine email correspondence. PI concerns noted include communications clarity, timeliness and value. From a project-level perspective, the EET found during this evaluation that in those cases where a full-time facilitator (coordinator, junior scientist or mentor) was associated with in￾country project activities, projects progressed with much greater ease than when such a person was not appointed. These appointments complemented the work of part-time in￾country scientific co-PIs but also dealt with the plethora of management, financial, training and facilitating activities associated with projects managed and financed from overseas. While employing dedicated coordinators for host country activities incurs additional personnel costs, the benefit to project activities and impact justifies the investment. c. Level and quality of engagement with PIs and host country partners (based on interview data) Findings: The ME is not viewed in general as having a strong leadership presence (again, some PIs reported that the ME staff are ‘approachable,’ helpful and responsive in a timely way – but generally did not identify technical or scientific leadership). Most PIs describe communication to be ad hoc and focused primarily on logistical issues or routine reporting. Improvements here were noted in the third year of the program with more effort to share information on scientific collaboration and publication opportunities and in one case more frequent communications from the ME staff. Several PIs expressed the need for improved communications. d. Quality and availability of narrative and trip reporting mechanisms Findings: PIs perceive ME guidance on reporting requirements to be vague. Poor reporting guidance to PIs has resulted in some uncertainty concerning ME expectations as well as less than optimal collection of indicator and impact data on project accomplishments. e. Scope and quality of program database and overall information management strategy Findings: The EET was unable to access the program database during the evaluation period. The ME reported that a contracting issue with the database vendor prevented even their staff from accessing the database. Thus, the EET is unable to evaluate the quality and scope of the database. The EET also encountered challenges in accessing data from the ME and sometimes received conflicting information, including financial/budget data and personnel effort figures. 24 f. Sub-award financial reports timely and complete Findings: The EET did not have complete documentation on sub-awards but from available data, PI institutions appear to have submitted adequate and timely financial reports. ME staff indicated that once funds are disbursed in host countries, ME accountability is no longer in effect. PIs monitor host country projects, and for individual researchers such as TIRI Scholars progress reports rather than financial reports are required. PIs offered suggestions to improve communications including identifying a point person for their communications, having clear and transparent communications processes, and providing clear reporting guidelines early in the projects. For other ALSCC Innovation Lab stakeholders (including host country collaborators or co-PIs), interactions with the ME are generally very limited or nonexistent. Some co-PIs had communications or interactions with the Deputy Director during field visits. Conclusions for items b-f: On the whole, the ME is perceived as offering primarily logistical rather than technical guidance. While appropriate financial procedures are generally followed by finance staff, program management and financial disbursement structures have sometimes contributed to delays in project implementation (see financial management section for recommendations on this topic). Administration Recommendation 2: ME should identify a key point of contact for PIs and other project stakeholders. This individual can direct issues to relevant staff if unable to address them. Administration Recommendation 3: ME should develop and disseminate clear reporting guidelines to PIs to address both Feed the Future Indicators and general progress and impact of each project with reporting at 6-month and one-year intervals on project progress. Administration Recommendation 4: ME should urgently resolve database access issues and begin to evaluate and analyze available program data. Administration Recommendation 5: The ME should consider regularizing host country facilitator positions in LTRPs. 2. Information available on all projects and activities commissioned since the inception of the program Findings: Basic information is available on projects and activities, including RFPs and proposals, and research profiles for all LTRPs. Additional documentation is available for many projects, including some trip reports and progress reports, project contracts, and indicator data, as well as partial documentation of Advisory Board meetings, training courses, etc. A complete set of information is not available across all projects. The program database was not available to the EET during the evaluation period (the ME reported that a database had been developed by a vendor but was unavailable due to contract and funding issues with the vendor). Conclusions: The EET is uncertain about the scope of data available in the database, but believe it might have provided more complete information. 25 Administration Recommendation: See Administration Recommendation 4, above. 3. Governance: Advisory Board exercises appropriate and effective scientific oversight Findings: The Advisory Board for the ALSCC Innovation Lab has terms of reference (2012) outlining a three-part mission: provide guidance on program policy and budget; evaluate the global approach and content of the program; and review the yearly progress of the program. The Board is to consist of a chair and up to eight members serving two- to three￾year terms. The terms of reference identify annual board meetings with more frequent meetings (every three months) during the “start-up period.” The composition and meeting attendance of the Advisory Board has varied over the course of the program, with some meetings having as few as two or three members present. Of the original five Advisory Board members noted in the ALSCC CRSP 2010 Annual Report, one remains on the Board. The current Board consists primarily of relatively new board members, with three having joined in 2013 and attended only one meeting to date. a. Advisory board members level of engagement Findings: Five of six current board members responded to the EET request for input via a questionnaire developed specifically for them. Those board members who have attended only one meeting emphasized that they are still learning about the program and have had limited exposure to its opportunities and challenges. It is notable that the board composition has been unstable over its three-year lifespan with turnover of board members. The ME leadership team itself identifies turnover in board members and challenges in getting them to attend meetings as a concern, as well as noting that their input has been largely limited to “tweaking” program management. As a whole, board members view the program mission – i.e., international collaborative research focused on issues of livestock and climate change as well as capacity building – as very important. Their views vary as to whether the program should continue as currently constituted. b. Evidence that issues identified in board meeting documentation are pursued by ME Findings: The structure of the board and its activities meets with mixed reviews from board members. The challenges identified include the meeting format and leadership structure (no board member as chair, no executive session with ME representatives absent, leadership by ME inconsistent with little role for Director) and the quality of board-ME interactions (limited or no follow through on board recommendations, no follow through on requests to board to serve as project evaluators). The relationship of the ME and the board is not clearly defined, and this results in a constrained role for the board in program governance and a less than optimal environment to become engaged with the program. c. Other advisory board concerns Findings: Board members offered a range of opinions as to the progress made by the ALSCC Innovation Lab as a whole. Some board members expressed the opinion that progress was being made and the ALSCC Innovation Lab was on track to produce results (although delayed), while others suggested that the ME needs to better prioritize ideas, develop a 26 coherent program/portfolio design, and focus its efforts in a context of multiple stakeholders and finite resources. Board members believed that the board should have more information and a deeper knowledge of funded projects and reporting back by the ME on board recommendations. Conclusions: The Advisory Board for this program has not been effectively used either for technical guidance or program governance. Turnover of board members as well as a gap between board recommendations and concerns and ME action limit the board’s impact. The board’s structure and practices also serve to constrain its leadership role. Administration Recommendation 6: ME and advisory board clarify respective roles in program governance and establish new board structures and processes to better utilize board expertise. These might include the following practices: advisory board member serves as chair and collaborates in developing board meeting agendas; board meetings have an executive session during which ME staff are not present; board members consistently receive minutes of meetings prior to subsequent meeting; ME reports back to the board at each meeting on implementation of board recommendations; a feedback mechanism is established to ensure critical issues are identified and managed (e.g., regular communications or check-ins with board members). 4. Program is adequately and appropriately staffed a. Staff positions and effort levels are appropriate to meeting project goals and objectives Findings: ME staffing levels appear to be inadequate to manage an international program of this size and complexity. Some advisory board members identified this as a concern, and several project PIs noted that ME staff appeared overstretched. The EET’s experiences in the evaluation process bear out these observations. We had difficulty in obtaining documents and data due to time and organizational issues, we encountered inconsistent and confusing communications in a number of settings, and various project stakeholders reported that they experienced similar issues. For example, to date we have not been given access to the overall program database that was originally promised. Specifically, the ALSCC Innovation Lab Director’s time commitment to the project appears inadequate leading to an unmanageable program management burden for the Deputy Director. The Deputy Director was identified by a range of stakeholders (Advisory Board members, project PIs, host country partners) as the primary presence and contact for the ALSCC Innovation Lab. However, several stakeholders also noted that as a result, the Deputy Director role involved over-commitment and too much travel to project sites. 27 Table 1. ALSCC Innovation Lab ME Staff Percent Effort* Year 1 Year 2 Year 3 Year 4 (qtrs 1, 2, 3) Year 4 Year 5 Bowen, Richard 22.92% 32.33% 33.00% 33.00% 50.00% Bradley, Michele 40.00% 40.00% 40.00%* Braley, Karlie 40.00%* 40.00% Butler, Leslie 70.00% 70.00% Davis, Jessica 80.00% 80.00%* Fahrenbruck, Diana 70.00% 70.00% 100.00% 100.00% Gillette, Shana 60.00% 92.50% 96.95% 100.00% Gutierrez, Nicole 70.00% 70.00% 70.00% Hoag, Dana 50.00% 50.00% 33.00% 33.00% 50.00% 50.00% Lupis, Sarah 69.69% 77.02% Rao, Sangeeta 25.00% 20.00% 28.76% 25.00% 25.00% Ryan, Elizabeth 31.40% 21.35% 30.00% 30.00% Salman, M D 21.69% 6.94% *Year 4, 4th Quarter and Year 5 are estimates. Financial staff members seem well acquainted with the program and have worked to develop and implement solutions to challenges such as the difficulty host country collaborators encountered with cost reimbursements. The financial and budget staff have been responsive to requests for budget data. Budget data provided has sometimes been inconsistent. The ME’s apparent division of staff responsibility by geographic area adds to a lack of overall cohesiveness in the program portfolio and does not necessarily link substantive expertise to project oversight. The staff member overseeing projects in East Africa appears not to have a scope of work or terms of reference, and interviews with project stakeholders suggest a lack of clarity about his role vis-à-vis the research projects. Conclusions: The ME staffing structure does not adequately provide for the demands of managing an international research portfolio of this type. Redefining roles and responsibilities in relation to program goals and tasks is needed. Administration Recommendation 7: The program Director role must be provided with adequate time to devote to managing, promoting and enhancing the portfolio and the individual in this position must exercise a clear leadership role both at the ME/staffing level, with the Advisory Board, and with the funded projects. EET recommends that the Director position has a minimum of 50% time and a position focus on promoting the ALSCC Innovation Lab globally and within the ME’s institution, forging and leveraging connections with related donors and projects, USAID missions, and USAID Washington, and providing scientific guidance and focus to the portfolio Administration Recommendation 8: Program staffing levels must be evaluated by function with adequate oversight for M&E, communications and technical/financial support. EET 28 recommends that an Associate Director position at 100% effort should serve to manage the ALSCC Innovation Lab on a daily basis, oversee staff, serve as primary point of contact for program stakeholders, and oversee program communications and outreach. EET recommends that the ME engages a dedicated M&E specialist at 50% who reports to the Director and takes responsibility for portfolio M&E activities, PI and collaborator M&E, information and training, and collection of data. 5. USAID missions are engaged with local projects a. Evidence of communication between USAID missions and project ME, PIs, co-PIs, and host country collaborators Findings: Interviews with USAID mission staff, ME staff, PIs and collaborators reveal uneven engagement and communication with USAID missions. ME staff report little or no engagement with Kenya Mission, while Nepal Mission staff were familiar with ME staff and some funded projects. USAID mission staff expressed both a desire for more communication concerning funded projects as well as an expectation that they would be kept informed of program developments in their host country. Individual PIs sometimes play a key role in keeping mission staff engaged by making regular visits when they go to host country research sites as well as by sharing updates and documentation related to their research projects. Again, expectations and practices for USAID mission engagement are not clearly delineated or regularized across LTRPs. USAID mission staff interviewed for this evaluation report in some cases that they were unfamiliar with ALSCC Innovation Lab funded projects. Some mission staff have stronger connections to the program, with Nepal staff expressing the highest degree of familiarity with the ALSCC Innovation Lab projects. In this case, the ALSCC Innovation Lab Deputy Director as well as the USAID AOR and others had formed connections with mission staff to raise their awareness of regional activities. Conclusions: More and consistently communicated information for mission staff would help to forge connections with research projects. For instance, a recent training course undertaken by mission staff to provide capacity development for local researchers would have been complementary to TIRI Scholar training. This is an example of a missed opportunity to partner. b. Role of USAID mission in project development (proposal review, identification of research priorities, assistance with host country linkages) Findings: USAID mission staff in Nepal and Senegal reported having a role in reviewing project proposals and offering feedback on the proposed projects. However, while recognizing the significance of livestock-focused research and HICD, they also reported that the focus areas of the ALSCC Innovation Lab projects do not often fit their highest development priorities. USAID mission staff expressed willingness in some cases to assist in linking projects to local counterparts. 29 Conclusions: Mission staff generally expressed a willingness to form linkages with project personnel, and expect to have project PIs or other staff keep them informed about the research projects. Frequent turnover in mission staff makes forming sustained relationships difficult. ME efforts to engage mission staff have not been sufficiently regular to ensure a close fit between ALSCC Innovation Lab and mission priorities or to capitalize on the mission’s ability to create connections with potential partners at local, national and international levels. While some PIs proactively connect with mission staff, better knowledge of the ALSCC Innovation Lab program on the part of mission staff would benefit efforts to leverage funding and linkages with related programs. Increased awareness would also serve to raise the profile of the ALSCC Innovation Lab program and foster appreciation for the significance of this area of development research and capacity building across USAID. Administration Recommendation 9: The ME leadership team, and particularly the Director, should be tasked with establishing and maintaining program-level connections with USAID mission staff in countries with funded projects, with the global donor community in this field, and with the academic and international development communities in the livestock and climate change field. This could include more frequent travel to host countries and international organizations or conferences as well as increased program publicity efforts. ME staff position descriptions should include designated responsibility for overseeing linkages with USAID in Washington and USAID missions in host countries to ensure consistency across projects and increased visibility for the program as a whole. The ME, in consultation with USAID, should provide specific guidance to PIs concerning expectations for engagement with USAID mission staff. Financial management Assess how well the ME has managed the financial aspects of the Livestock-Climate Change Innovation Lab taking into consideration project resource allocations; checks and balances regarding grantee disbursements, expenditures, and reimbursement; and if cost matching requirements are being met. The following evaluation is based on budget and expenditure information provided by the ME’s Financial Manager. It reflects actual expenditures up to November 2013 and does not include subsequent estimates of expenditures submitted by the ME to April 2015. Two additional installments of funding - totaling approximately US$5.5 million - are expected from USAID prior to closure of this phase of the program in April 2015. The total funding amount anticipated from USAID is US$14,853,291. In assessing how well the ME has managed the financial aspects of the Livestock-Climate Change Innovation Lab, the EET employed two ‘outcome measures’ and associated indicators identified in its Evaluation Plan – Appendix B. These measures capture performance concerning allocation and disbursement of resources as well as financial management and monitoring procedures. In undertaking this evaluation of financial management during Year 4 of this five-year program, the EET accepts the dynamic nature of expenditures associated with various budget categories. However, an attempt was made to ascertain whether any significant variance had occurred between foreseen and real 30 expenditures, whether expenditure issues of concern could be detected and whether the ME had followed USAID financial rules in managing, reporting, dispensing and monitoring program funds. 1. Program resources allocated in a balanced manner and disbursed efficiently a. Budget data shows sufficient level of investment in research, capacity building and dissemination. Findings: Budgetary investments in research (mainly the LTRPs) were predicted to be of the order of 45% of total program allocation at program inception – see Table 2. However, with the delay in implementation during Year 1 (and part of Year 2) of the program, the ME’s commissioning of research and associated activities has been on a catch-up trajectory since that time. Currently, expenditure on sub-contracts represents 31% of allocation to date – although the predicted total spend on research is foreseen to reach 37% of total allocation by the end of the Phase I grant period. Whereas this level of investment is less than foreseen, the current ME team has significantly improved the unsustainable situation in program expenditure at the end of year one. Actual expenditures related to capacity building and dissemination are difficult to differentiate from the financial data supplied by the ME. General guidelines of 80%, 15% and 5% are employed for expenditure on research, capacity strengthening and dissemination by ME staff and PIs. However, analysis of actual time and related expenditure of these three activities to date by the ME as a proportion of total program expenditure is approximately 47%, 21% and 10% - Table 3. Whereas the relatively low spend on research is not unexpected because of the delayed implementation of the program, the relatively high spend on capacity building was more surprising. The ME team provided occasional specific training courses in-country on gender, climate change, nutrition, M&E, RIC etc. A considerable amount of capacity development has also been undertaken by the individual LTRPs; interviews with PIs and Co-PIs estimated that 15-50% of their research budgets have been allocated to a wide array of training activities such as PRA tools, interview techniques, modeling, mapping, data collection, climate change indicators, and nutritional estimates/analyses. Effort related to the TIRI Scholar initiative consumed considerable senior staff time in capacity building activities. Expenditure on TIRI Scholars will increase shortly as the West African TIRI initiative is implemented. The expenditure on graduate fellows is not likely to increase further. Conclusions: On balance, expenditure on capacity building has been generally far in excess of the initial guidelines. While capacity building constitutes a laudable activity and may result in higher quality research outputs and sustainable research capacity, it has also undoubtedly diluted the budget available for research per se and compromised the ability of some of the world class research PIs involved in the program to address the research challenges. Expenditure on dissemination activities (interpreted as transforming research knowledge into different formats for discrete end-user audiences) has been largely carried out by the ME communications team in producing web-based products such as a bi-monthly newsletter, managing the website and social media, research summaries, synthesis reports, fact sheets, and slide share. Expenditure on dissemination is therefore related to the 31 salaries of the members of the ME Dissemination team (see Tables 3 and 4). The LTRPs have been more actively involved in conventional dissemination activities – the generation of scientific papers and presentation of conference proceedings – the associated costs of which are provided through the sub-contracts. Discussions with PIs indicate that their major investment in active in-country dissemination will occur in Year 5 of the program. EET is therefore of the opinion that the ME should ensure that such activities are undertaken by the projects and provide supplementary funding for this activity if necessary. Financial Management Recommendation 1: ME should monitor more closely the actual expenditure by the program (ME staff and LTRPs) on capacity building, dissemination and research. b. Allocation and disbursement of funding for other program activities. Findings: In general, the expenditure of budgetary resources (particularly salaries and sub￾contracts) was quite uneven over the first four years (see Table 2). Allocation for Materials and Supplies were grossly over-budgeted (by approximately US$300K representing a 91% under-spend); no significant further expenditure from this budget line is expected for Year 5. For travel too, the existing budget of US$611K has only been half spent. Expenditures on sub-contracts show a considerable under-spend of about one-third (approximately US$ 1.398 million) although ME are currently actively commissioning new work valued at US$1.24 million. Finally, the budget for salaries (initially foreseen to be about 25% of total budget) predicts a considerable under-spend of US$864K by the end of Year 5 (see Table 2) although more recent financial reports indicate a smaller balance. Actual expenditure on salaries is expected to be 16.2% of total allocation. Considering all of these budgetary and expenditure factors, the program has a considerable foreseen underspend of more than US$3million – although more recent information on planned expenditures suggest that this is more likely to be approximately US$1.873 million (or 12.6% of total budget). The ME and the Advisory Board need to urgently consider the implications of this scenario, including the opportunity it presents for new programming and the potential negative response of USAID administration to under-spend. Whereas the foregoing budget reports indicate that all program expenditures did not match budget projections, no reallocation of funds between budget lines appears to have taken place as yet despite the fact that USAID allows virement between budget lines. Conclusions: For the remaining 13 months of the currently funded program, and depending upon current discussions with USAID on reallocation of budget categories, there appears to be adequate funding available to enable the ME to address the gaps in the research portfolio and enable the program to achieve more of its objectives. This might include increasing the allocations of current LTRPs, commissioning new activities and/or recruiting experienced field staff to assist the ME (and PIs) to ensure delivery of program outputs. From a program structure perspective, the expectation of research teams with relatively limited research budgets generating development outcomes within a short time line (3-4 year project) is flawed. The involvement of development partners (NGOs and government agencies) in the research projects must be fostered to facilitate field research activities. However, the research teams do not currently have the appropriate skill sets (nor is it their project objective) to up- and out-scale research findings to benefit poor communities on a 32 wide scale. With the available balance of funding, the ME in collaboration with the research teams and in-country public and private sector institutions may be able to recruit appropriate skills to promote greater adoption of research products and technologies for target audiences. Financial Management Recommendation 2: The ME should focus efforts to increase the scale of commissioning of research to target program needs. We would suggest that much of this should be in gap-filling research to address program objectives. ME should urgently review their budget forecasts and expenditures to facilitate virement options in an effort to target spending at gap research and reduce foreseen underspend. Commissioned research, however, must meet the aims and objectives of both the ALSCC and FTF. In country follow up: The ME does not track funds once disbursed to the institutions involved nor is reconciliation with actual expenditure demanded from overseas institutions. This could put funds at risk of misappropriation. For example, the ME indicated that it has no way to track expenditures of funds once allocated to TIRI Scholars’ institutions. Financial Management Recommendation 3: ME should review its expenditure reconciliation practices to ensure that its grantees receive and utilize their funds as planned. 1. Project budget is well managed and monitored a. Processes in place for disbursing funds, reporting and monitoring expenditures Findings: Based on the information provided by the ME and interview of the Financial Manager appointed to manage the program budget, the ME team in general appears to satisfy financial rules and follows the monitoring and reporting requirements of the program, CSU and USAID. The ME team satisfactorily manages time-sensitive mandated reporting to USAID on budgetary matters such as quarterly accruals, pipeline analyses and annual expense reports in partnership with CSU’s Office of Sponsored Programs. The EET’s findings suggest that the ME has complied with the financial rules related to USAID’s disbursements of program funds. The ME team also dispenses multiple grants to project partners, prepares transfers and pays program invoices and bills. However, the program has encountered several challenges in the disbursement of funds. The cost reimbursement mechanism initially used for host country collaborators has proved difficult; several PIs noted significant delays in receiving fund transfers from CSU and attribute slower-than-anticipated project implementation to an irregular pattern of funding disbursement. While acknowledging that funding disbursement has been delayed at times, the ME’s Financial Manager indicated that the ME has been proactive in getting funds to PIs but due to delayed receipt of funding from USAID, along with international issues (SAM registrations, institutions and banks holding funds, and currency exchange issues) funding disbursement has unfortunately been delayed at times. In some cases, project activities have had to be temporarily supported by other funds (e.g., PI personal research funds). An alternative system of fixed-price contracts with deliverables tied to payments has since been established for some collaborating researchers. Cost matching levels appear to have been achieved as evidenced by budget documents. As far as we can interpret from the budget reports available to us, these reflect appropriate budget expenditures. 33 Conclusions: The EET is satisfied that the ME has complied with the financial rules related to USAID’s domestic and international disbursement of program funds. Improvements can be made in monitoring expenditure of funds for individual researchers. Financial Management Recommendation 4: ME should review its disbursement practices and devise a better mechanism to provide more speedy transfer of funds to sub-awardees and TIRI grantees – particularly at start-up. Table 2. Budget and expenditure, 2010-2015 - status as per 15 November 2013 All figures below in US$’000 Category Budget to date* % of total budget forecast Year 1 spend Year 2 spend Year 3 spend Year 4 through to11/13/ 13 Total spend to date Spend as % of total budget Balance Salary 2,376 25% 499 473 334 206* 1,512 16.2% +864 Travel 611 6.5% 50 85 88 69* 293 3.1% +317 Materials and Supplies 345 3.7% 24 10 8 2* 44 0.5% +301 Other **Direct 450 4.8% 28 63 112 73* 276 2.9% +174 Sub￾contracts 4,254 45.5% 16 478 1,422 940* 2,856 31% +1,398 Total Direct 8,036 618 1,108 1,964 1,290 4,981 53.2% 3,054 I/C@26% 1,086 159 234 230 136 761 - - TOTAL 9,122 777 1,343 2,194 1,426 5,742 61.3% +3,379 Total +Transfers 9,358 *** .* part year; ** non-CSU staff costs,***Total Budget available Table 3. Expenditure on Capacity Strengthening and Dissemination (November 2013) Expenditure to date in US$’000 Research support by ME + LTRPs +Seed grants +TIRIs and Grad Fellows Expenditure on TIRIS and Grad Fellows only Expenditure on Capacity Building Expenditure on Dissemination Total 3,168 722 1,468 663 As proportion of total expenditure 46.7% 10.6% 21.6% 9.8% 34 Monitoring and evaluation Assess the effectiveness of monitoring and evaluation efforts of the Livestock-Climate Change Innovation Lab to include whether there are systems in place to capture research impacts and how effective they are, whether baselines and targets have been established and met, the appropriateness of indicators, and quality of data. 1. Monitoring and evaluation of the program and funded projects is effective and timely Findings: a. A coherent and well-documented M&E plan is in place and functioning effectively An M&E strategy for the program’s LTRPs exists but not for the program per se. Most proposal documents include a flow chart that includes project objectives, processes, outcomes, and impacts and include tables that list the USAID indicators to be measured, how the indicators connect with the program research themes, and how many outputs might result. In addition, the PIs were asked to include a description of how each of the listed indicators would be measured. However, in practice most LTRPs have not focused on M&E issues. In part, this is due to the fact that only a modicum of M&E training was given by the ME to support in-country project level monitoring and evaluation activities; responsibility for M&E issues was often not specifically designated to individuals at the project level; and as many PIs considered their projects to be in their infancy, M&E was generally not prioritized as a key activity. Consequently, some projects use custom indicators as per their original project proposals, others have modified them in light of experience and others use the USAID Target Indicator Counts. b. Proportion of projects submitting narrative reports, indicators and trip reports in timely manner and indicator data is comprehensive and linked to Feed the Future strategy In 2013, each PI was given annual Target Indicator Counts as goals for the period 2013-2015. These were based on a selection of quantitative USAID Feed the Future indices that most closely conformed to the activities of each individual project and/or addressed one or more of the four research goals of the program. Accordingly, each project reported actual versus targeted counts for a range of sub-categories. These were reported for 2013 and were aggregated at the program level for website presentation. c. Quality of key M&E data In a few instances the indicator data reported appear to be impressive quantitative achievements - but these data were impossible to verify at the project level due to the time available to the EET and the absence of a program level database. A few PIs or those deemed to have responsibility for M&E issues within projects indicated they were prepared to report their progress in achieving USAID indicators once the data base is ready for such reporting. 35 d. Baseline data and targets are available While all projects have undertaken baseline surveys, the reporting of indicators should be regularized and the data analyzed for overall program impact. The ME was unable to provide extensive data and data analysis to the EET due to lack of access to program database during the external evaluation period (due to a contracting problem with the database vendor related to inadequate funding). Existing indicators should be better linked to program impacts (understanding that Feed the Future parameters drive many indicators). During much of the life of the program, PIs were not asked for annual progress reports and reporting requirements and expectations remained unclear. Instead, they were required to submit trip reports, which did not reflect the progress or impact of their activities (rather they reflected their activities in the field). Some PIs submitted progress or annual reports independently as they believe it enhances their work. The ME is now making a transition from trip reports to progress reports. e. Indicator data is comprehensive and linked to Feed the Future strategy Indicator data was available for some projects. The recent start of the long term research projects means that it is too early to demonstrate significant impact on many dimensions. Feed the Future indicators were selected for each research area. See research section of this report for consideration of appropriateness of indicator selection. Conclusions for items a-e: The EET considers that increased guidance and concerted action is required from the ME in implementing a meaningful M&E strategy for each project – and to ensure these contribute to the delivery of foreseen program output. Accomplishing this will require a designated and specialized M&E person in each project responsible to the PI and with strong oversight from an M&E responsible officer in the ME. M&E Recommendation 1: The ME should have an appropriately experienced staff person whose position description is dedicated to the management of program level M&E, In addition, he/she should have the authority to provide oversight for the implementation, collection and analysis of project level indicators/impact data. M&E Recommendation 2: The ME needs to devise a strategy to monitor and evaluate the funding devoted to capacity strengthening in its various guises – Graduate Fellows, TIRI scholars, specialist training provided by the ME and LTRPs, with a particular focus on durability of impact M&E Recommendation 3: The ME should also communicate and enforce clear expectations to PIs concerning reporting requirements. EET recommends that annual progress reporting (at least) should be obligatory. 36 Research Program To explore the depth, breadth and rigor of the ALSCC Innovation Lab research portfolio from a management perspective, the questions underpinning this portion of the report are as follows: 1. Does the ALSCC Innovation Lab meet the identified Feed the Future research priorities, and if not, why not? 2. What are the goals of the ALSCC Innovation Lab and do projects funded under the Program contribute to these goals? 3. Is there evidence that the ALSCC Innovation Lab research portfolio, as a whole, will contribute to solving critical researchable constraints impacting poor livestock keepers at the global level? 4. In the absence of standard M&E data, what approach can be applied to support an accurate assessment of the impact of the ALSCC Innovation Lab? Critical to answering these questions is an understanding of the Feed the Future research ethos. Feed the Future supports the concept of ‘purpose driven research’, which is outlined as follows (Feed the Future, 2011): ‘…a new paradigm of sustainable intensification to catalyze agriculture-led economic growth by focusing on environmentally-sustainable productivity gains.’ To achieve such gains, other pillars of the Feed the Future research strategy include climate smart-development5 and a focus on technology development. Research outputs should have clear delivery pathways via ‘extension, education, evaluation and feedback at the individual country level’ (Feed the Future, 2011). Key research themes identified within the Feed the Future Research strategy for the livestock sector include (Feed the Future, 2011): 1. ‘control of infectious diseases of livestock, some of which are zoonoses; 2. developing management approaches for improved agricultural productivity; 3. improved quality of animal feeds… 4. livestock genetics and breeding for improved productivity.’ The document lists the priority animal health constraints for research in sub-Saharan Africa as: Contagious bovine pleural pneumonia (CBPP), East Coast Fever (ECF), Trypanosomiasis, Rift Valley Fever and Foot and Mouth Disease (Feed the Future, 2011). The Feed the Future research framework also identified three cross-cutting issues: gender, nutrition and climate change. Given the orientation of this particular Innovation Lab, it is clear that climate change should not be perceived or deployed as a cross-cutting theme but rather as a core component implicit to all research funded by the Program. 5 The defining features of climate smart development generally relate to enhancing resilience to climate change at the community level and lowering GHG emissions (Thomson-Rueters, 2013). 37 Alignment with the Feed the Future framework requires that, from the outset, the ALSCC Innovation Lab should support activities relating to the sustainable intensification of the livestock sector in target countries from a climate-smart approach. The ALSCC Innovation Lab should address the priority research areas above and equally, the program should work closely with national bodies to disseminate outputs. The following evaluation of the research dimensions of the ALSCC Innovation Lab, however, aims to explore the portfolio as a whole and should not be read as an explicit review of the impact, quality or value of individual research projects per se. Therefore, the EET anonymized project details, as far as possible, within the scope of the assessment. Further, the lack of formal monitoring and evaluation data has meant that the EET has had to glean impact data from a wide range of sometimes contradictory project documents including project proposals, PI trip reports, consultant reports, EET travel logs etc. While we anticipate that some omissions or errors will have inevitably occurred, by creating a broad￾brush framework the overall findings can inform decision-making at the donor level. 1. Findings 1.1 Feed the Future Research Priorities and the ALSCC Innovation Lab The evaluation of the ALSCC Innovation Lab revealed a number of divergent areas with the Feed the Future research strategy. First, projects funded under the ALSCC Innovation Lab program included a number of interventions with an exclusive development orientation with little or no ability to meet the criteria for ‘purpose-driven’ research, as detailed above. In relation to animal health priorities and themes, few of the projects had significant animal health components, while most projects in sub-Saharan Africa did not have a particular focus on the identified animal disease threats. Equally, across the portfolio, new technology creation or explorations of the adoption of existing technologies were rare. Unusually while most projects at least ostensibly mentioned climate elements, few projects incorporated a climate-smart approach. Thus, as a whole the ALSCC Innovation Lab investments were not firmly embedded within Feed the Future research priorities. This lack of alignment can be attributed in part to the timing of the Feed the Future Research Priorities document, which was published in May 2011 after the ASLCC Innovation Lab start-up. Nevertheless, the first RFP for long-term research projects was released in July 2011. In this document, while the Feed the Future cross-cutting themes were mentioned, the Feed the Future priority topics were not. This early oversight meant that the Feed the Future priorities did not adequately inform ALSCC Innovation Lab research investments. Further, the ALSCC Innovation Lab has operationalized the cross-cutting themes as areas for added expertise rather than embedding these themes within funded projects. As such, expert advisors have been employed to inform PIs on how to better incorporate these elements into their projects. Nevertheless, the approach suffers from critical shortcomings. As project activities have already been determined, applying such a focus ex-post means that cross-cutting themes will only be addressed in so far as PIs have time and resources to incorporate extra data collection or analysis. Equally, the approach appeared not to have large buy-in from the PIs themselves with some questioning the need for such advisors. 38 Therefore, uptake of the advisory element of the program was inconsistent and generally low. To explore the overall fit of ALSCC Innovation Lab long-term research projects and Feed the Future research priorities a scoring system for the core elements of the Feed the Future strategy was devised (Table 4). As the table demonstrates, most projects scored 50% or below in matching Feed the Future research priorities. Table 4: Feed the Future Research priorities and evaluation scores Region & Project Area Addresses Priority Diseases/ Zoonoses (5) Addresses Management for Increased Production (5) Addresses Climate￾smart development (5) Improve d Feed or Animal Genetics (5) Deploys outputs via Extension /Education (5) Technology Development or adoption component (5) Gender, Climate, Nutritional Component (5) Total Score (35) Africa 1 Partial: base-line survey on zoonoses No No No Yes (training of Livestock Extension Agents) No Nutritional survey: women only. No climate component. 15 Africa 2 No but focus on poultry diseases Yes: trains children in management techniques Partial: household food security impacts No Yes (works with schools) No Gender equality: yes; climate component: no 15 Nepal 1 Partially MSc student projects on FMD/Live r Fluke Yes: developing training material for farmers Yes Yes: fodder producti on, micro￾irrigation Partial: workshop for policy makers at project conclusion Partial: exploring adoption of micro-irrigation Collecting weather data. Kitchen garden component. Training in gender awareness 25 Nepal 2 No Yes: Farmer training to increase productivity Unclear: increased feed requirements of exotic buffalo Yes: AI for Buffalo with improve d breeds Yes: Training of AI inseminator s, works w/ NARs and Gov. Services No No specific gender component. Weather data collection. 17.5 Nepal 3 No Yes: fodder development and water tanks No No Partial: Reference Groups includes Gov actors No Gender inclusivity unclear. No nutritional component 7.5 39 Africa 3 No Partial: identifying impact of changes in management practices Yes: exploring resilience Yes: exploring cross￾breds with NGO Partners Yes (works w/National Lab) No Gender and nutritional component 22.5 Africa 4 No Partial: Creation of community action plans Partial: Cross￾breds use increased resources. Yes: improve d grazing strategie s Partial: Gov Research Institute No Promotes gender participation 14.5 Nepal 4 Partial: base-line data on animal health Partial: Creation of community action plans Yes but household level integration unclear No Yes No Promotes gender participation, nutritional component 17.5 Africa 5 No Partial: Community￾level awareness raising Yes: But unclear how model will impact resilience No No No Promotes gender awareness/ nutritional comp unclear 12 Africa 6 No Yes Partial: indirectly via value chains No Yes, working with policy makers No Yes women’s groups nutritional comp 15 From the table, it is clear that the LTRP crossed a wide range of activities and stakeholders therefore, the breadth of the portfolio was high. Such a breadth however, comes with attendant costs. Indeed, it is likely that the focus on ancillary development activities will drain time and resources away from the researchable constraint itself. However, while such an approach tells us about the scope of the investment, it tells us little of the actual and potential impact of the portfolio of projects. Nor does it offer an indication of how closely the research matches the actual vs. perceived needs of the communities themselves. To meet the criteria of sustainable intensification, any such research must, at minimum, acknowledge community demands and interests. Next, the assessment explored if the funded research was in-line with the stated program goals. 1.2 Organization of research within the ALSCC Innovation Lab Unusually, the program goals were altered over time. In the ALSCC Innovation Lab foundation document, the five goals of the program were described as: a. ‘Assess risks, constraints and opportunities of climate disruption on livelihoods. 40 b. Document and monitor dimensions of environmental and social change. c. Strengthen infrastructure and repair disjointed policies and co-ordination. d. Address conflict over limited resources. e. Minimize impact of production systems on environment and health.’ Clearly, the above goals were focused on delineating the impact and effect of climate change on livestock keeping communities in the Global South with strengthening infrastructure and policies the key response. Such goals fit the remit of climate-smart development and indeed four out of five goals address the environment. However, by 2012, ALSCC Innovation Lab goals were listed as follows (ALSCC Innovation Lab, 2012): a. ‘Apply science and technology and build human and institutional capacity to address hunger and poverty. b. Build local human and institutional capacity for livestock productivity, veterinary and human health and environmental sciences. c. Extend and apply research findings and technical knowledge to livestock producer households, associated industries and public and private extension and agricultural services to strengthen capacity to adapt to and cope with unpredictable climate variability. d. Develop policies that support national and regional programs to support livestock producers, traders, veterinary and human health officials and government agencies in planning for and responding to climate-change induced resource variability.’ Thus, the emphasis of the ALSCC Innovation Lab was re-focused to include a broader mandate on human and institution capacity development and the explicit creation of policies to support actors working within the livestock sector. Three out of the four objectives have a capacity building aim. The focus on climate change was altered to a more limited purview of climate variability, which crosses two of the goals. Indeed, Goal c above appears to be a focus on Extreme Weather Events (EWE) while Goal d, focuses on the policy implications of climate-induced resource variability (although to date such variability has not yet been adequately characterized in the Global South). While the driving force behind this change of objectives was not elaborated on, it is presumed that this was an attempt by the ME to enhance Program alignment with the current Feed the Future outcome indicators (rather than Feed the Future research priorities). The implications of the change in goals over the course of the project are twofold. First, clearly there is less time for projects to meet the new objectives. Second, it is unlikely that existing projects, funded under the old objectives will be on a trajectory to deliver relevant outputs. Replacing environmental and climate change research with that which supports capacity building and assessments of the weather (climate variability) is also likely to decrease the impact of the ALSCC Innovation Lab. 41 The Program ascribes to four research themes. However, the themes are not directly related to the new goals. And equally, some themes e.g. the animal health and pro-poor value chains, as described, require a more explicit link to climate change adaptation. a. Climate extremes and long-term climate change b. Animal health, disease distribution and resiliency c. Ecosystem health: resiliency of socio-ecological systems d. Pro-poor value chains: market access and reliability However, the themes, which begin to emerge in Program documents in 2012, appear to be an attempt by the ME to capture the focus of currently funded projects rather than as a means of focusing or organizing Program investments. The risk of such a gulf between the objectives and research themes is that Program outputs will less easily meet the identified criteria for impact. 1.3 Project Orientation and Activities As noted above, within the context of ALSCC Innovation Lab projects, both research and development activities were being undertaken by project teams. Nevertheless, operationalizing the ‘conjoined’ approach was problematic on a number of levels. In some cases NGO partners were responsible for both the collection and analysis of household-level data. Conversely in other projects, researchers were responsible for implementing the development activities. Therefore, it may be argued that the expertise of both groups was stretched. Equally, the cost-effectiveness of such an approach is questionable. Development projects tend to impact greater numbers of beneficiaries, at a lower cost, than those with research aims. Further, it may be argued that the development elements tend to dilute staff time and resources away from the particular researchable constraint involved. The following table disaggregates the research vs. development activities and outputs across the project portfolio (Table 5). Where the orientation of the research was categorized as ‘strategic’6 , ‘applied’, ‘domestic’7 or ‘international’8 and the overall approach classified as qualitative, quantitative or based on mixed methods. Training taking place before project findings were also classified as ‘development’. 6 Within this context, ‘strategic research’ is defined as that which ‘is mission-oriented and involves the application of established scientific Knowledge and methods to broad social or economic objectives, often extending over a considerable period’. http://www.quantum3.co.za/CI Glossary.htm 7 Where domestic research refers to that which produces outputs directly relevant to the nation involved e.g. both forging and directly aiding national development plans, priorities and policies. 8 Where international research refers to that which produces outputs relevant to global or transnational issues or agendas. 42 Table 5: Research vs. development-focused activities and outputs Region & Project Area Research￾focused Activities Approach Orientation Development￾focused activities Research Participants Development Beneficiaries Africa 1 2 base-line surveys: animal health, human nutrition Mixed Domestic/ Strategic Training of Livestock Extension Agents (10). Training of CHWs (42) Input curriculum on climate change. 196 52 Africa 2 Base-line Survey by MA student Mixed methods Domestic 10 Chicken Coops stocked in 10 participating Schools 150 600 school children Nepal 1 Weather station data, Village level surveys Mixed methods Domestic Kitchen gardens, micro-irrigation plots, fodder plots, rain water harvesting 181 550 25households / area fodder dev 9/per area kitchen garden Nepal 2 Weather station data Domestic Fodder plots, AI kits for inseminators Nepal 3 Base-line survey and Action research component Applied Domestic/ Applied Community water tank, fodder plots 2500 individual base-line surveys, 7 focus groups 1 Community Water Tank (partial funding); Fodder plots Africa 3 Participatory Corridor Mapping Mixed methods Domestic/ Applied Technical advice on zero grazing with NGO 360 (base￾line surveys) (total planned 2400) CAPs on corridor mapping Africa 4 PRA Assessment Qualitative Domestic/ Applied Funding for CAPs: interventions community determined CAPs Nepal 4 PRA assessments, Climate Data Mixed methods Int. component (Climate) Funding for CAPs: interventions community 12 PRAs first year of project. Village Model Farms CAPs 43 analysis, Research on PRA approach determined, Village Model Farms (VMFs) implemented Africa 5 Environment al Modelling, GPS herd migration Mixed methods Domestic 148 (base￾line surveys) 48 workshop participants Africa 6 Household surveys, ethnography, market data collection Mixed methods Domestic 140 household￾level surveys From the table, it is clear that across the ALSCC Innovation Lab investment, development activities constituted a large part of overall activities. Further in line with the Feed the Future research priorities most of the research was orientated at the domestic level. 1.4 Human and Institutional Capacity Building (Project-level) As noted above, capacity building crosses three core objectives of the program. Overall, both target groups and level of training varied widely across projects. Further, the aims and focus of the training were equally divergent. As capacity building tends to increase over time, it is likely that more HICD activities will be recorded as project cycles conclude. In some instances, however, it was clear that PIs were sometimes conflating capacity building at the community level with awareness-raising regarding project aims and objectives. More crucially, however, capacity building is not a guaranteed outcome of training activities. Yet, across the projects the impact and uptake of training at both the community and partner level was frequently not subject to M&E protocols. This is a significant weakness at both the project and programmatic levels. Across the portfolio of projects the ratio of ‘durable’ HICD as opposed to shorter-term training is relatively low. ‘Durability’ in this case relates to the ability of training to have a longer-term and deeper impact on an individual’s behavior or outlook. Utilizing this definition, capacity building of a durable nature includes tertiary education and training and at the community level, that which responds to specific and articulated demands. Conversely, stakeholder meetings or awareness raising workshops specifically related to project outputs are not particularly durable as the longer-term impacts are likely to be low. To assess durability a five point score was utilized which accounted for the number of individuals and target group (i.e. farmers, teachers etc.) trained, the duration/topic of training and the likely long-term impact (Table 6). 44 Table 6: LTRP: Capacity Building9 Region & Project Area Farmer training Partner Training Degree Training Other training Durability Score (1-5) Africa 1 Via a partner NGO school children training in climate data collection 7 (ecosystem services modelling) 38 (GIS) 25 (modelling) 4-6 (training of field team in data collection and analysis) 1 BSc, 1 MA 61 (teacher training climate data collection) 132 (teacher training) (NGO partner training on related project) 10 Extension agents 42 Community Health Workers 3 Africa 2 Development and implementation of educational curriculum on poultry for school children (600 children). 49 (debriefing and planning & impact) 1MA 33 (workshop for Peace Corp Volunteers) 18 (primary school teachers) 22 (primary school teachers) 12 (extension staff) 2 Nepal 1 81 (workshop participants in gender awareness) 20 (animal health) 46 (animal feed) 16 (in climate hazard mapping) 12 (in hydrological models/scenarios) 21 (in quantitative analysis) 21 (in story telling) 1 Nepal 2 AI training (Numbers unavailable) 20 (forage cultivation) 34 (forage cultivation) 30 (buffalo rearing) 51 (climate change and buffalo production) 2 Nepal 3 Planned training for ‘vulnerable communities and veterinary workers’ 10 (in survey methods) 128 (in project awareness raising) 1 9 The training of enumerators or other project staff in the implementation of their duties (while collated by some PIs) was not included in the above figures. 45 Africa 3 72 Community Focus Groups 1 PhD, 3 students field training only Technical training in GIS 1 Africa 4 Community action plans implemented 191 (PRA tools) 2 MA students 1 Nepal 4 260 (in PRA tools) 145 (in micro￾climate data collection) 1 MA student 2 Africa 5 48 (workshop participants) 13 2MA, 1 PhD Support for training community animal health workers, planned trainings with partners in modelling 2 Africa 6 18 (workshop in GIS) 3 PhDs, 3 MAs 2 post￾doctoral fellows 3 As the table illustrates projects were engaged in a wide variety of HICD activities from specific training for staff/students at collaborating Universities to farmer-level workshops to the involvement of school children in data collection activities. While the breadth of training is laudable, the overall level of training, particularly at the community level is low. Equally problematic, the quality of much of the training is uncertain. Further, it was clear that much of the training did not have a specific climate change focus. Hence, the actual and potential impact of the HICD component to support the ‘sustainable intensification’ of livestock keeping under conditions of climate change at the farmer or household level is likely to be limited. Across the portfolio, the durability of training in Africa-based projects was higher than that in Nepal, however, more of the training in Nepal was situated at the farmer-level. A more balanced portfolio in HICD at the project level would include a better mix between tertiary and farmer-led training. Overall, policy-maker engagement in training activities was low. The strength of institutional relationships also varied across the projects. Some project teams had long-standing relationships with in-country collaborators/institutions from the earlier CRSP program while other relationships had been developed over the course of the present project. 1.5 Program-Level Capacity Building As noted in the section above on financial management, the ALSCC Innovation Lab allocates a substantial proportion of program funds to capacity building. Several activities contribute 46 to the human and institutional capacity development (HICD) mission of the ALSCC Innovation Lab: • ME-organized occasional specific training courses in host countries (gender, climate change, nutrition, M&E) • Individual LTRP-sponsored training activities (PIs estimated that 15-50% of their research budgets have been allocated to HICD activities including PRA tools, interview techniques and other enumerator training, modeling, mapping, data collection, nutritional analyses, climate change indicators) • TIRI Scholar initiative including funding of individual TIRI Scholars, TIRI Scholar workshops and trainings, consultant work with TIRI Scholars, ME leadership team work with TIRI Scholars • Graduate fellows’ funding for individual research projects While the graduate fellows’ initiative appears to be stable, the development of the West Africa TIRI Scholars group will require increased expenditures to support research project development. An important distinction must be made between individual versus institutional capacity development. While it can be difficult to draw a clear line between the two – e.g., training for individual scientists can enhance not only their individual capacity but also the capacity of the institution with which they are affiliated – there are some types of HICD that more directly support institutional strengthening. For instance, at the individual LTRP level, project PIs work with national labs and research institutes to provide staff training and occasionally equipment. However, the investment in training for students and TIRI Scholars emphasizes the individual human capacity dimension – with possible spillover for institutions. In addition, the expectations for individual LTRPs to provide community-level training as HICD or other forms of HICD are not always clearly delineated, nor is their impact assessed within the framework of HICD potential. 1.6 Gender Inclusion Gender inclusion strategies for stakeholders (collaborators/co-PIs, researchers and enumerators, graduate students and TIRI Scholars, and community participants): Across the portfolio of projects, discerning the level of gender inclusion was often difficult. Little data at the program level has been collated/collected regarding the role and input of women in individual projects. The gender advisor on the whole could be better utilized. Some projects have explicitly targeted women for inclusion in a portion of the research and some have been successful. Overall, the number of women participating in the TIRI Scholar program and as host country collaborating co-PIs and research staff is low. Reasons offered by the ME team included the difficulty of recruiting qualified women candidates. Nevertheless, no outreach strategies were subsequently put in place to attract women who may have benefitted from the program. There appeared to be little push from the ME to expand this element of the program either across projects or within the TIRI/Graduate Scholar component. Reported efforts to include female scientists and students should be coordinated and strengthened, perhaps through ME assistance in facilitating institutional linkages. The gender advisor 47 consulted on a call for proposals from women in the first round. After advice that the ME could not have a gender-specific call for proposals, the ME did not further engage the gender consultant to strategize alternative recruitment mechanisms to increase the representation of women. Gender in project design and research questions: Individual projects varied dramatically in their incorporation of gender considerations and gender analysis in project and research design. Researchers were cognizant of the issue of employing female enumerators where possible, but did not always succeed in doing so. In addition, while researchers expressed awareness of constraints posed for female participants at the community level, effective strategies were not always in place to facilitate participation. Some projects, though, designed project activities in ways that enable female community members to participate (e.g., separate focus groups for women, or targeting female-headed households). Additional attention to incorporating a gender analysis in project design could benefit several of the projects. The ME’s effort to encourage this through employment of a gender consultant was poorly timed (projects already underway) and not always well received by PIs whose projects were in progress. 1.7. Feed the Future Performance Management Indicators and Project-level Impacts As part of the Feed the Future impact strategy, each Innovation lab is expected to report on three to five outcome indicators. The ALSCC Innovation Lab chose to report on the following indicators (as detailed in the Feed the Future Indicator Handbook, 2013): • 4.5.2-5 Number of farmers and others who have applied new technologies or management practices as a result of USG assistance. • 4.5.2-32 Number of stakeholders using climate information in their decision making as a result of USG assistance. • 4.5.2-39 Number of firms (excluding farms) or Civil Society Organizations (CSOs) engaged in agricultural and food security-related manufacturing and services now operating more profitably (at or above cost) because of USG assistance • 4.5.1-9-13 Numbers of Policies/Regulations/Administrative Procedures in each of the following stages of development as a result of USG assistance in each case: o Stage 1: Analyzed; o Stage 2: Drafted and presented for public/stakeholder consultation; o Stage 3: Presented for legislation/decree; o Stage 4: Passed/approved; o Stage 5: Passed for which implementation has begun. While limited data on the progress to the indicators was available to the ET, it is clear that these indicators are not the most reflective of Program activities to date. First, as noted above, the ALSCC Innovation Lab has not funded technology development per se. While 48 improving livestock management practices is a goal of many projects, M&E data on the adoption or uptake of these practices is lacking. Second, climate change and climate-smart development has not been prioritized across the project portfolio. Further, as detailed below (Table 8) private sector collaboration/involvement was uncommon. Finally, policy development is also not an explicit output of the majority of the projects, although some projects interface with national agencies with input to policy processes. Rather, meeting the following indicators from the Feed the Future Handbook (2013) may form a more relevant contribution of the ALSCC Innovation Lab to Feed the Future performance. • 4.5.2-6 Number of individuals who have received USG supported long-term agricultural productivity or food security training (S). • 4.5.2-7 Number of individuals who have received USG supported long-term agricultural sector productivity or food security training (RIA) (WOG). • 4.5.2-14 Number of vulnerable households benefiting directly from USG interventions. • 4.5.2-39 Number of new technologies or management practices in one of the following phases of development: o Phase I: under research as a result of USG assistance. o Phase II: under field testing as a result of USG assistance. o Phase III: made available for transfer as a result of USG assistance. First, given the capacity goals of the Program and the relative focus on this element from the TIRI scholars to the project level funding for tertiary education, indicators 4.5.2-6 and 4.5.2-7 should be easily measurable for the ME. Equally, the programmatic focus in both Nepal and East/West Africa has been on vulnerable subsistence farming and pastoralist households. Finally, as noted above, as a number of projects are focusing on improving livestock management strategies, these figures should be readily available to the ME. In December 2011, a consultancy was commissioned by USAID’s Bureau for Food Security to support ALSCC Innovation Lab alignment with Feed the Future Performance Indicators. The consultancy outputs generated both an ‘Alignment Plan’ and a knowledge management plan to aid ALSCC Innovation Lab collation of performance indicators and organize data inputs. It is unclear how closely the alignment and knowledge management plan was followed in the ensuing years of the Program. Conversely at the individual project level, research impact is generally based on two broad criteria. First, the ability of a piece of research to generate new knowledge or thinking on a topic. Second, impact relates to the ability of such new knowledge to influence thinking or behavior among a peer group (generally other academics) or participant group (ranging from individuals to industries). As such, the quality or impact of a piece of work is often only truly known long after the project cycle as ended. Development impacts, however, have different criteria. The impact of a development project largely relates to uptake and adoption of the intervention involved and its sustainability over the longer-term. Such 49 impacts largely relate to the demand for the intervention and/or product itself at the community level. Given the dearth of traditional M&E data at the Program level, to assess impacts within the scope of this review, an amalgamation of the following indicators was undertaken: 1. the overall fit of the ALSCC Innovation Lab investment with Feed the Future priorities; 2. an estimate of the level of innovation in methods or approach (derived from the project proposals) and therefore, the likelihood of specific project outputs in forging new thinking; 3. the durability of HICD; and 4. if an assessment of demand, undertaken at the community-level, informed outputs. Therefore, to collate project-level impacts the following metric was devised: Overall Impact Score = Feed the Future Priority Score + w [New Knowledge Score + HICD Score + Demand-led Score] Where w = the weighting factor (in this case wf=3.33 for a maximum total score of 50 for the combined indicators of New Knowledge, Capacity building and Demand-led). Table 7: Impact Scores Region & Project Area Feed the Future Priority New Knowledge HICD Durability Demand Overall Impact Score (*wf) Africa 1 15 2 3 1 31.5 Africa 2 15 1 2 1 28.2 Nepal 1 25 2 1 2 34.9 Nepal 2 17.5 2 1 1 30.7 Nepal 3 7.5 2 1 1 20.7 Africa 3 22.5 3 1 3 45.6 Africa 4 14.5 3 3 3 44.2 Nepal 4 17.5 3.5 2 2 42.3 Africa 5 12 3 2 1 35.1 Africa 6 15 2 3 2 38.1 * multiply II. Collaboration, outreach and technology dissemination 1. Findings Overall, the depth and reach of collaboration across the LTRPs was low. In-country NGOs comprised the majority of partner institutions for the LTRP followed by Universities (Table 8). 50 Table 8: In-country Collaborating Institutions by type While University collaborators dominated in Africa, NGO collaborators were often responsible for all levels of project administration including the actual research work in Asia. Indeed, in some cases, the role of the US institution appeared limited to management oversight. Additionally, in Nepal a single NGO (HKI International) worked across multiple projects. Overall, the ability of the NGOs to undertake research of this nature was uneven. In some cases, reports from NGOs regarding survey activities appeared exaggerated but could not be verified given the time constraints of the external review. As a stated aim of the Program is to create and influence policy, the current mix of institutions involved is unlikely to fully achieve this aim. Indeed, across the projects in many cases clear communication pathways at the national level are lacking to support the translation of research outputs into policy guidelines. While many of the PIs noted that they had informal relationships with policy makers, formalizing these relationships at the collaborator or partner-level may better aid Program goals. The implications of the above partnership choices at the project level are twofold. First, while NGOs tend to have good community relationships, historically Universities located in the Global South, tend not to work at the community level. Therefore, while University level partnerships are good for building durable community and institutional capacity, such partnerships are likely to be less adept at forging ‘sustainable intensification’ at the community level. Second, clear tramlines for policy development and influence are lacking. Outreach strategies are poorly documented across the Program. While activities between partners were reported on, informal linkages between researchers and wider stakeholders Region & Project Area NGO University/Institutions of higher learning NARs Government Bodies CBOs/CBIs Private sector/ Consulting Firms CGIAR/UN Bodies Africa 1 1 1 Africa 2 1 1 Nepal 1 1 1 Nepal 2 1 1 1 1 Nepal 3 1 Africa 3 1 1 1 Africa 4 1 1 Nepal 4 1 1 1 Africa 5 3 2 Africa 6 2 1 1 51 including USAID missions were often not documented. Indeed, relationships between the PIs and in-country missions were often patchy. In some cases interactions were limited by the time constraints of Mission staff, while in others stated interest by the Mission was apparently not followed up or leveraged by the ME. As such, the ME does not appear to play the expected bridging role between Missions and PIs and therefore research outcomes. Across the projects, dissemination of research outputs was low (accounting for the time￾period involved). From 2011-2013, 25 outputs were disseminated ranging from conference presentations (16), books/booklets (2), refereed journal articles (2), videos (3) and research reports (2). As noted above, few of the projects are developing or disseminating livestock-centric technologies. While some of the projects in Nepal had development components, which dispersed agricultural technologies (such as drip irrigation), overall focus of the research was not to develop the technologies per se. This may be construed as a major weakness of the research portfolio vis-à-vis Feed the Future objectives. Finally, related to the above, knowledge and information management strategies at the ME level were often inconsistent. Indeed, some critical communication protocols and information-based systems were lacking (such as the incomplete web-based data-base) with a likely impact on timely and effective decision-making. Other efforts at dissemination were more successful in reaching a global audience such as the monthly Research Communiqués. While the Communiqués were useful and informative, the focus was on wider opportunities and news relevant to livestock and climate change. Given internal divisions at the ME level some program-level material has been disseminated via a Google drive: https://drive.google.com/folderview?id=0B4bfIMjiCjSqMU1Ec2YtcEdRaXM&usp=sharing But this material only relates to the projects funded in Ethiopia. Hence, a consistent and formal approach to disseminating project-led material is lacking. While the recently redesigned website is an improvement and makes a clearer effort at engaging users in research activities, the ME has not exploited web-based or social media related activities in terms of wider marketing and communication strategies. Indeed, according to the ME, social media based-communication strategies such as Facebook, Twitter and most recently, LinkedIn have not been a priority. The ME has recently published a series of four-page Research Briefs on-line. This is an important step in public engagement. It is unclear, however, if the research briefs have been further targeted at particular audiences of interest such as policy makers and other actors working in the global livestock sector. Other on-line reports include two from 2012-2013 detailing the impact pathways and alignment with Feed the Future goals. 52 Figure 2. ME Progress Reports The ME has also published a series of ‘fact sheets’ and specific reports on Gender and Nutrition. Overall, the communication strategy appears ad-hoc and largely responsive rather than strategic. Little attention has been accorded to the differing needs of particular target audiences ranging from specialist researchers to community-level workers in the Global South to the general public. 2. Conclusions Climate change is likely to be the biggest challenge facing livestock keepers globally living on less than $1/day. Yet the overall impact of the ALSCC Innovation Lab portfolio, as currently constituted, is likely to be low. First, the overall lack of clarity regarding core program objectives and the chasm between the objectives and research themes is an impediment to the success of the program. Second, the conjoined research and development approach appears to drain resources away from key researchable issues. Equally, adding development elements to a research project is a costly way to pursue development objectives. Both of the above issues hinder the creation of a robust and responsive research program, which directly addresses the needs of poor livestock keepers. The demand-led orientation of projects varied dramatically. Those projects with more consultative and demand-led elements tended to have the highest number of development, as opposed to, research-led activities. However those LTRPs, which were more research focused, were often the most extractive at the community level. To achieve the wider Feed the Future goal of ‘sustainable intensification’ via ‘purpose-driven research’, a balanced approach in which research projects respond to specific and climate-focused issues of direct relevance to poor livestock keepers is required. As such, the ALSCC Innovation Lab should fund applied research responding to constraints with both global and domestic implications for the livestock sector under conditions of climate change. Hence, omissions at the RFP stage had repercussions on both the quality and collective impact of the ALSCC Innovation 53 Lab portfolio of projects. The shift from environmentally-focused to capacity building goals reflected a lack of strategic intervention in the Livestock and Climate Change research landscape. The overall breadth of LTRPs contributed to constraining research impact. Rather than an exclusive focus on a single, researchable constraint, the inclusion of a variety of activities to address a wide range of issues, often meant that both researcher and collaborating partner skills were stretched. In some cases, researchers were implementing development activities and NGO partners were carrying out research. Such a mismatch of skills equally has attendant costs on both the quality and sustainability of the outputs produced. Capacity building at the project-level was equally inconsistent and often lacked a significant monitoring and evaluation component (particularly with respect to durability). Hence, across many projects, training was being undertaken with little or no understanding of how such training was influencing the target group, particularly at the farmer-level. The inclusion of HICD activity is a necessary and integral element of Innovation Labs. However, various forms of HICD are often closely interconnected, and it is critical to have an intentional balance between activities that develop individual researchers and staff members (e.g., of NGOs) and those that contribute to long term and sustainable institutional capacity. The ME has made efforts to encourage a gender-inclusive perspective in LTRPs, but these efforts came after projects were underway and only some PIs have made use of the gender consultant. Uneven efforts across individual LTRPs suggest the need for a consistent policy requiring gender expertise and gender analysis in project design as well as specific gender recruitment strategies for project researchers, field staff and collaborators. Sex￾disaggregated data is a minimum starting point for gender-inclusive projects. The dearth of impact data at the Program level is a crucial oversight. By initially collating trip reports rather than performance indicators, the ME lost the ability, early in the project cycle, to assess impacts. As a result, guidance from the ME to PIs on impact assessment was lacking. Equally problematic, the Feed the Future indicators chosen by the ME for performance monitoring were likely not the most reflective of Program activities. Changing the indicators will help the ME better align with Feed the Future objectives. Finally, outreach to policy makers is another critical weakness across the Program. Public engagement activities were low across both the individual projects and the wider program. In this manner, engagement is ad hoc and largely left to the PIs involved. The ME needs to urgently support PIs in collating policy findings and situate these findings within a wider livestock development context for uptake by both national and global decision-makers. The development of clear communication pathways at the Program level are required to aid such research findings in influencing the wider global livestock development community. 54 3. Recommendations on Research Dimension Summary Research Recommendation 1: Leadership with deep expertise in the field of livestock and climate change and extensive experience in the international livestock development sector is required. Research Recommendation 2: Program objectives should be refocused to reflect current knowledge gaps within the livestock and climate change landscape. Research Recommendation 3: In order to enhance impact, during the time-period remaining, a stable of short-term projects, with an applied and technology-based focus, should be funded which clearly align with both Feed the Future research priorities and new program objectives. Research Recommendation 4: The advisory service on the cross-cutting themes, as currently constituted, should be reconsidered with the possibility of applying available resources to the short-term research projects detailed above. Research Recommendation 5: Robust monitoring and evaluation protocols should be put in place with the creation of an attendant information management system to inform decision￾making at the ME level. Research Recommendation 6: Experts in livestock and climate change should inform the RFPs with additional expert inputs on gender and nutrition to enhance the innovation and quality of research solicited by the program from the outset of the investment rather than ex-post. Research Recommendation 7: Program management needs to explicitly identify new project investments to address gaps in the current portfolio; offer robust and persistent leadership in implementing revised strategies in M&E, reporting, and communications; take a stronger role in ensuring LTRP projects address program goals; provide more encouragement to promote stronger collaboration between projects with common aims/target beneficiaries; offer more leadership in forming alliances with in-country institutions, private and public; and lead with more awareness of the challenges of research and development programs vs research for development initiatives. Research Recommendation 8: ME should adopt a consistent policy requiring gender expertise and gender analysis in project design as well as specific gender recruitment strategies for project researchers, field staff and collaborators. Research Recommendation 9: The ME should include in its overall vision for the program with a clear strategy for components of HICD at the individual and institutional levels, distinguishing training activities from durable capacity building and providing clear guidance to PIs concerning expectations for HICD in individual projects. 3.1 Recommendations re: Feed the Future Priorities 1. Any projects funded during the final phase of the Program should explicitly align with Feed the Future research priorities. 55 2. A scoring system/guide for external reviewers at the project proposal stage would better ensure that Feed the Future criteria are met. 3. Critical gaps for increased funding include animal health and technology development/adoption studies within a context of climate-smart development. 4. Climate change should be not be perceived as a cross-cutting theme but rather a core focus of future investigations. 5. Projects should equally directly incorporate gender and nutrition obviating the need for additional advisory inputs during the project cycle. 6. Advisory inputs on the cross-cutting themes should occur at the RFP and proposal evaluation stage to ensure the calls adequately reflect contemporary thinking and are in line with Feed the Future research priorities in these areas. 3.2 Recommendations re: Research organization 1. A recognized expert in the field should rewrite the program goals to support innovation and excellence and better reflect current thinking in the climate change and livestock development interface. 2. The linkages between the research goals and themes need to be clearly articulated. Projects should directly respond to core goals and related themes. The aim, as far as possible is to balance the portfolio across themes. 3. Again, given the overall goal of the ALSCC Innovation Lab and the explicit focus on climate and climate impacts, climate should be a topic inherent to all elements of funded projects. 3.3 Recommendations re: Project Orientation 1. Funded projects should have a clearly identified and exclusive researchable (as opposed to development) constraint at the community, national, regional or global level. 2. The program should focus on applied, responsive research with clearly identified impacts at the community level. 3. To enhance impact, project investments should support outputs at the international level. 3.4 Recommendations re: Feed the Future Performance Indicators and Impact 1. The selected Feed the Future performance indicators do not suitably reflect the likely ALSCC Innovation Lab outcomes and should be altered. 2. Changing the performance indicator should ease the ability of the ME to demonstrate relevance to wider Feed the Future programmatic goals. 3. Recommendations detailed in the 2012 consultants report on the Alignment Plan and Knowledge Management Plan should be implemented within the context of the new indicators detailed above. 56 3.5 Recommendations re: Collaboration and Outreach 1. Enhancing outreach strategies at the Program level should be a major activity of the ME across the remaining time-period. A range of activities to strengthen relationships with USAID staff both in Nepal and East/ West Africa should be prioritized and actioned. 2. To enhance policy impact in the remaining time-period of the Program, funds should be released to individual projects for the specific purpose of engaging policy makers directly with project outputs as opposed to project aims and goals. 3. Projects funded during the remainder of the Program should have explicitly detailed policy outcomes and appropriate communications pathways described at project start-up to achieve such outcomes. 4. Appropriate information management protocols should be urgently developed and implemented, including immediate attention to gaining access to database and undertaking analysis of data to better assess program impacts. 3.6 Recommendations re: HICD 1. To increase the overall impact of this aspect of the program, criteria for training vs. capacity development should be more clearly outlined at the RFP stage. 2. Ensuring impact from training requires significant project-level investment in M&E protocols specific to learning and knowledge transfer. 3. If the ALSCC Innovation Lab continues to prioritize capacity development, expertise in farmer training and assessment requires further development across the Program. 4. Training activities were often reported that reflected the knowledge needs of the project staff such as training for enumerators in data collection methodologies etc. rather than durable knowledge transfer. 5. The durability of HICD inputs needs to be assessed and balanced across the project portfolio. Measurement of durability will aid the Program team in determining the impact of investments in this area. 6. The ME should include in its overall vision for the program a clear strategy for components of HICD at the individual and institutional levels, including a balance between individual PI efforts and ME efforts as well as clear guidance to PIs concerning expectations for HICD in individual projects. 3.7 Recommendations re: Gender Inclusion 1. The ME should adopt a consistent policy requiring gender expertise and gender analysis in project design as well as specific gender recruitment strategies for project researchers, field staff and collaborators. 57 4. Lessons Learned • From the outset, the lack of clarity between ALSCC Innovation Lab goals and themes has had a ‘cascade’ effect on the overall cohesion and quality of the funded research, the attendant ability to assess impacts and the capacity of the Program to meet Feed the Future Performance indicators. • The operationalization of the cross-cutting themes as an ‘extra layer’ as opposed to a core component of projects is costly in terms of external advisors and does not promote high quality research across these areas. • Funding projects with both research and development components is inefficient and has a negative impact on the quality of the outputs. • While HICD crossed three out of four ALSCC Innovation Lab goals, without a mechanism for impact assessment at the project and subsequently program level, the ability of such training to foster capacity development is questionable. • In order for ALSCC Innovation Lab to have policy impacts the ME requires a clear understanding of the form and content of policy-led research and the relevant communication pathways required for engaging policy makers at the national vs. the global level. Program Future The ALSCC Innovation Lab has enjoyed a number of accomplishments in funding seed grant and long-term research projects, supporting TIRI Scholars and graduate fellows, and organizing a variety of capacity building activities. However, the ME as currently constituted was often perceived as being disorganized and less than fully effective across the range of stakeholders interviewed during the course of the review. There was a further notion, detailed by some stakeholders, that a broader scope of experience in the international development arena would strengthen the ME team’s implementation efforts and the overall impact of the program. Some recent improvements in ME communication and efforts to problem-solve were identified, as well as increased attention to fostering collaboration and integrating cross￾cutting themes across projects. These efforts are laudable and should be noted. However, from advisory board members to project PIs concerns were expressed as to the ability of the ME to implement strategies to ensure the success of the program as a whole. Some PIs did not feel the need for extensive ME guidance and preferred independence in the conduct of their research projects; however, the lack of guidance did impact effectiveness of the program as a whole. The effectiveness of individual research projects relies upon the initiative of project PIs, with ME leadership playing little role. It is the opinion of the EET that the ALSCC Innovation Lab has the potential to be a flagship program for USAID under appropriately resourced management within a constructive and committed institutional environment. As described above, attention to research and capacity development in livestock and climate change is essential to a pro-poor international development agenda. 58 With strong leadership the portfolio of projects has the potential to have a greater collective impact (the whole will be more than the sum of its parts). Over the next 13 months, considerable investment in and implementation of gap filling projects has the potential to have a significant collective impact. The focus must be on filling core researchable constraints important to better understanding the role of climate change on poor livestock keepers in South Asia and Africa. The present position of climate change as a cross-cutting theme rather than a core constituent of all program activities is a key factor in preventing this aim. While we have reservations over the capacity of the existing ME to maximize the potential of the ALSCC Innovation Lab, we do support the strong continuation of the program research area into a reconfigured Phase 2. The potential of a strong research agenda in this area, with attendant impacts was recognized as beneficial by a wide range of stakeholders interviewed by the EET. Clearly, the investment represents a crucial element of USAID’s international development portfolio and with appropriate management a program of this nature can rapidly advance thinking. USAID and a few bilateral partners are among a small group of donors still prepared to fund research aimed at the supporting the livelihoods of increasingly vulnerable population of poor livestock keepers. Positive impacts resulting from the ALSCC Innovation Lab are likely to shape future investments by other donors. As the program moves forward into its last 13 months of the first five-year award, the EET remains concerned about the expectation of the program to address the multiple objectives that demand different capacities. The expectation of research findings generating development outcomes within the short time-line of a 3-5 year project is flawed. Whereas the involvement of in-country development partners in the research projects is essential to engage with local actors and communities, this program has illustrated that significant capacity strengthening is needed to ensure the necessary research skills. Similarly, researchers may lack the necessary skill sets to up- and out-scale research findings. Having invested time and effort in capacity strengthening, it is therefore crucial for the benefits of such an investment (in training of LTRP staff, TIRI Scholars and Graduate Fellows) to be captured by the program during the remainder of Phase 1 of the program and into Phase 2. At present, the monitoring and evaluation and indeed impact assessment protocols are ineffective and need total revision in-line with current development thinking. In seeking to build strong M&E protocols, strategic field visits by the USAID AOR to assess progress and impacts at the meta-level could help to tie the portfolio more closely to USAID’s research and development strategies. With respect to the remaining 13 months of the program, and attending to maximizing the impact of program activities to date, it is important for the ME to increase the time afforded by a Director to at least 50% and urgently identify an appropriately experienced Associate Director. The ME should also consider whether the substantial investment in the in-program specialist advisers for the last year of the program is a sensible use of funds. The advisers could play a role in defining lessons learned from currently funded projects in their area of expertise. The EET suggests an urgent review of both staff skills and time devoted to this program and other components of the ME’s work to facilitate an effective completion to the existing Phase 1 (and to inform a possible Phase 2). 59 During Phase 2 we propose that the ME should consider a return to the use of nested logframes (or a similar program logic that connects the individual research projects explicitly to the overall framework) to help ensure that project outputs contribute meaningfully to program level indicators. The EET sees that substantial alterations are required to the current configuration of the program. We recommend that USAID take three steps with respect to the ALSCC Innovation Lab: (1) In the final 13 months of Phase 1, work closely with the ME to oversee the implementation of core recommendations concerning staff time/effort allocation, plans for expenditure of remaining funds in a focused manner and one that advances the previous investments in capacity building for TIRI Scholars and graduate fellows, and immediate attention to the program database, M&E and the collection and analysis of impact data. (2) During year five of the current Phase I program, plan to reconfigure the ME. One strategy to accomplish this is issuing an open call for proposals to compete for Phase II. The current ME at CSU could compete with a revised plan and structure, but the call would be open to other potential MEs as well. Key elements for Phase II is an ME team with strong experience in the fields of livestock and climate change/livestock development. Further, evidence that the program will be deeply embedded in University activities should be a core requirement of a successful bid. See decision scenarios table, below, for additional analysis. (3) Ensure the establishment of an ME with a priority on formulating and supporting a cohesive research portfolio with associated HICD activities and tied to long term and sustainable improvements in adaptive capacity of poor livestock keepers in the context of climate change. Decision scenarios on the future of the ALSCC Innovation Lab program Finally, based on the accumulated evidence gathered by the EET, together with its interpretation and the conclusions and recommendations presented in this report, the series of scenarios outlined in the following table might assist USAID in identifying the appropriate actions regarding the continuation of the ALSCC Innovation Lab program, whether with the current ME or an opportunity for a new entity. Based on an ‘if..’, ‘then..’ and ‘and..’ approach, we present decision scenarios related to Phase I (remaining 13 months) and Phase II (subsequent 5 year period) for USAID’s consideration. 60 Table 9. Program Future Decision Scenarios Phase I/Short Term Decision Scenarios through 2015 If… Then… And… CSU’s ME creates an appropriate and viable work plan (which addresses the gap filling research mentioned in this report) and puts a clear monitoring plan in place Funding continues to April 2015 (end of Phase 1) with some flexibility in budget allocation to maximize benefit of existing activities. Minor disruption to research and field activities and delivery of some program outputs by capitalizing on investments to date. CSU’s ME establishes and supports a strong leadership team with adequate time allocation and linkages to CSU’s sectoral expertise, broader context and strategy. Funding continues to April 2015 (end of Phase 1) with some flexibility in budget allocation to maximize benefit of existing activities. The rigor of research will improve and CSU’s considerable institutional and research resources will be leveraged to benefit the ALSCC Innovation Lab and CSU as a whole. CSU’s ME immediately develops clear and detailed position descriptions for the leadership team, reconsiders the role of the specialist advisers, designates and M&E-responsible staff person and rationalizes the current regional allocation of responsibilities. Funding continues to April 2015 (end of Phase 1) with some flexibility in budget allocation to maximize benefit of existing activities. Improved balance among various project components (research, HICD, outreach and dissemination), improved M&E, decreased ambiguity of individual roles and responsibilities and resulting improvements in program functioning. CSU ME does not implement changes in the above (management, staffing roles and/or no improvement in M&E) over next 3 months Funding continues to April 2015 (end of Phase 1) with closeout of existing projects and USAID oversight of proposed new expenditures. Disruption to existing projects and activities. 61 Phase II/Program Continuation Decision Scenarios If… Then… And… CSU ME restructures program management and embeds it in University processes, supports an ME leadership team with expertise in the field and ensures that the Director and Associate Director allocate 50% (Director) and 100% (Associate Director) of their time to the positions. CSU can renegotiate with USAID and continue to fund Phase 2 under newly restructured CSU ME. or CSU can join a competitive rebid process. but USAID should implement Phase 2 of livestock and climate change initiative in either case. Continuation of many existing projects and relatively minor disruption to management support staffing; build on existing activities and connections with a renewed focus; potential challenges of ME restructuring. Potential disruption of existing projects and initiatives by rebidding process and/if new ME selected; refocused program could achieve greater impact from research and HICD activities and better advance Feed the Future priorities. Continued focus on critical livelihood, poverty reduction, food security, and environmental issues at core of livestock and climate change theme. CSU ME does not meet above criteria. CSU not invited to join rebid process. Disruption of program activities until new ME is in place and active. Based on open competition, an experienced ME team with a program Director who devotes adequate time (minimum 50%) to directing, promoting and enhancing the portfolio of research and supportive capacity development initiatives. Possible that another institution will become management entity; USAID implements Phase 2. Director and team have the opportunity to orchestrate the design of a focused portfolio of new and selected Phase I projects to address the program goal. Uncertain period for existing projects and loss of momentum of current program activities. New ME will need to identify staff and consequent delay in program activities, new calls for proposals, etc. (Irrespective of which institution serves as Phase II ME), the Program Director exercises a clear technical and strong management leadership role both at the ME/staffing level, with the Advisory Board, and with the funded projects. USAID implements Phase 2. Director’s central role with ME staff, research teams and their in-country collaborators will advance work to achieve program goals. 62 The ME devises an overall strategic vision for the ALSCC Innovation Lab based on appropriate, realizable and coherent research outputs contributing to identification and development of adaptive strategies for livestock exposed to climate change. USAID implements Phase 2. Research on climate change adaptation, nutrition and gender become core components of research portfolio; A clear niche is identified for the program in global livestock and climate change research for development. Research outputs provide basis for strengthening resilience of communities facing climate change￾based livelihood challenges. Program staffing levels and reporting lines are revised to ensure clearer and more balanced responsibility for M&E, capacity development, communications and technical/financial support. USAID implements Phase 2. Clearer guidance and easier access to information for stakeholders involved in implementation of research and capacity development activities. The ME coordinates program initiatives and outputs that address common challenges or regions and links capacity development activities to core program goals. USAID implements Phase 2. Synergies and collaboration among projects adds value to the program activities and outputs with particular benefit for HICD linked to program focus. The ME (and Director in particular) establishes and maintains program-level connections with USAID mission staff in countries with funded projects, with the global donor community in this field, and with the academic and international development communities in the livestock and climate change field. USAID implements Phase 2. Promote new linkages and partnerships with interested public and private sector institutions, internationalize the program, advance USAID priorities and leverage complementary funding. ME actively promotes the RIC initiative and realizes the concept in the target regions. USAID implements Phase 2. Collaboration enhanced by RIC mechanism will sustain scientific development, improve research communication, and provide a forum for information exchange and mentoring in-region. 63 ME proactively collaborates with LTRPs to establish new partnerships with policy making institutions and with those able to facilitate the transfer of adaptation strategies and technologies to target livestock keepers. USAID implements Phase 2. Greater harmonization of activities and outputs, yielding additional benefits of research to community￾level and national stakeholders, and increased resilience to climate change of livestock-dependent communities. 64 APPENDICES 65 Appendix A: EET Scope of Work Scope of Work: External Performance Evaluation of the Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change10 Award Number: EEM-A-00-10-00001 Purpose The purpose of this external performance evaluation of the Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change (hereafter referred to as the Livestock-Climate Change Innovation Lab) is to evaluate both the program management and the research program, provide recommendations to inform the decision on program extension and, if appropriate, provide recommendations as to any suggested program changes or improvements. The evaluation will help inform USAID on whether to extend the Livestock-Climate Change Innovation Lab for a second five year phase or end funding at the conclusion of its current five year phase. Background The Feed the Future Innovation Labs with U.S. universities (formerly called CRSPs) were created under Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage U.S. land grant and other eligible universities to address the needs of developing nations while also contributing to U.S. food security and agricultural development. In 2000, Title XII was reauthorized, enabling the continuation of the CRSPs as one of several types of U.S. university research efforts helping “to achieve the mutual goals among nations of ensuring food security, human health, agricultural growth, trade expansion, and the wise and sustainable use of natural resources”. The Innovation Labs are an integral part of the new Feed the Future Food Security Innovation Center, established to implement the Feed the Future Global Hunger and Food Security Research Strategy and to respond to two key recommendations from a Board for International Food and Agricultural Development (BIFAD) commissioned CRSP review11 : • To develop an overarching and coordinated strategy for engaging U.S. universities in agriculture and food security research and human and institutional capacity development that includes the CRSPs as a central component; and • To leverage the impact of CRSP investments by strengthening links across universities, U.S. government, global programs, foundations, and other donors. The Food Security Innovation Center will enable USAID to manage its research, policy and capacity-strengthening portfolio by thematic area rather than by institutional home. To this point, CRSP programs have been renamed Feed the Future Innovation Labs. This name change does not alter USAID’s commitment to funding the integrated research and training exemplified by CRSPs and other types of research and capacity strengthening programs with 10 Formerly called: Adapting Livestock Systems to Climate Change Collaborative Research Support Program (CRSP) 11http://transition.usaid.gov/our_work/agriculture/bifad/BIFADREVIEW_CRSP_August2012.pdf 66 U.S. universities. On the contrary, USAID is significantly expanding opportunities for Title XII universities and their partners to participate in competitive awards in a number of the Food Security Innovation Center program areas. Each of the former CRSP programs are now included in one of the following seven Center programs: 1. Program for Research on Climate Resilient Cereals – helps smallholder farmers adapt to climate change and build resilience by developing new cereal varieties with enhanced yield and tolerance to drought, heat, salinity and low soil fertility and delivering these varieties in diversified, sustainable farming systems. 2. Program for Research on Legume Productivity – increases the production and consumption of critical, protein-rich legumes, by developing disease and stress tolerant, high-yielding varieties, improving market linkages and post-harvest processing and integrating legumes into major farming systems to improve household nutrition and incomes, especially for women. 3. Program for Advanced Approaches to Combat Pests and Diseases -- harnesses US scientific expertise and emerging molecular tools to develop new animal vaccines and crops and animals resistant to pests and diseases that cause significant production losses in tropical systems. 4. Program for Research on Nutritious and Safe Foods -- addresses under nutrition, especially in women and children, by increasing the availability and access to nutrient dense foods through research on horticulture crops, livestock, fish and dairy, food safety threats such as mycotoxins and other contaminants and on household nutrition and food utilization. 5. Program for Markets and Policy Research and Support -- works to achieve inclusive agricultural growth and improved nutrition through research on enabling policies, socioeconomics and technology targeting and by building the capacity of partner governments to effect sustainable change in areas such as land tenure, financial instruments, input policies and regulatory regimes. 6. Program for Sustainable Intensification -- works with smallholder farmers to incorporate sustainable, productivity enhancing technologies and farming practices into major production systems where the poor and undernourished are concentrated, and through intensification and diversification of these systems, to enhance resilience, nutrition and agricultural growth. 7. Program for Human and Institutional Capacity Development -- strengthens individuals, scientists, entrepreneurs, educators and institutions, ensuring that food and agriculture systems in developing countries are capable of meeting the food security challenge and that women especially are poised to take advantage of new opportunities and provide critical leadership in agricultural research, private sector growth, policy development, higher education and extension services. 67 Description of the Livestock-Climate Change Innovation Lab Purpose The goal of the Livestock-Climate Change Innovation Lab is to increase resilience and augment the income of livestock producers in regions where agricultural systems are changing, available resources are shrinking, and climate is having an impact. The scope for the research, training and outreach activities under the Livestock-Climate Change Innovation Lab include the impacts of climate change on livestock-human- environmental health. Health includes nutrition. Areas of research are: livestock production and health, human nutrition and health, food safety and marketing, and environmental health. Program activities also may investigate the indirect impacts of climate change on value chains for both live animals as well as livestock products for human consumption that have high market value and are important for household economic and food security. The Livestock-Climate Change Innovation Lab employs a systems approach with cross￾cutting themes to address its research priorities under the following goals: • Improve the health and productivity of livestock while benefiting the health and welfare of farm families and conserving natural resources. • Integrate market research with the needs of small-scale farmers. • Collect and analyze data from farms, partners, and governments that informs evidence￾based solutions. • Increase research capacity through training and educational support. • Extend the reach of funded research by leveraging resources and partnering with other organizations with similar research and development goals. The Livestock-Climate Change Innovation Lab is included within the Program for Research on Safe and Nutritious Foods of the Food Security Innovation Center upon Program creation in early 2013. Management Constraints The Livestock-Climate Change Innovation Lab experienced a setback of approximately six months at its start up due to the composition of the team that makes up the Management Entity (ME) at Colorado State University. Disparate personalities conflicted and the previous Director was not pro-active in resolving the differences of opinions related to program structure and content. Seed grants (pilot projects) were awarded for a period of one year then re-competed in an open competition for long-term awards. However, by the time those grants were awarded the program had already entered the second year of funding. The USAID Agreement Officer’s Representative (AOR) intervened at a higher level within the University and, with the support of the USAID contract office, was successful in the restructuring of the ME. Due to internal politics, the University appeared reluctant to make the changes necessary for a smoother flow of operations, but did cooperate to resolve the differences. 68 Activities to Date The research activities are participatory and collaborative at both the design and implementation stages. This has resulted in research activities that are driven by practical considerations and the needs of the host country(s) and are not just a reflection of the interests of U.S. scientists. All activities take place in Feed the Future priority countries (see below). The Livestock–Climate Change Innovation Lab activities are focused on the following research themes: Climate Extremes and Long-term Change – One project is examining monsoon patterns in Nepal and how shifts in precipitation are affecting livestock production. Another project is down scaling regional climate data and providing guidance on areas that are at high risk of extreme climate-related events in Nepal. Key accomplishments include: climate assessment in far western Nepal and another initiated in the Gandaki River Basin indicated the likelihood of impending drought; initial results form a socio-ecological assessment in the same region identified current adaptation strategies for herders and has helped build a framework for action by policymakers; and early career researchers competed and are being supported by mini-grants for one year. Animal Health: Disease Distribution & Resiliency – The HALI project (Health for Animals and Livelihood Improvement) in Tanzania is assessing the effects of zoonotic disease and water management on human and animal health and livelihoods. Climatic changes are linked to changes in hydrological processes and threats to public health. Water scarcity during the dry season brings animals and people together more frequently to share water for agricultural and domestic use, thereby affecting health and food safety. Led by the University of California in Davis, the project is increasing the training and diagnostic capacity of Livestock Extension Officers, creating educational opportunities for women and children, and collecting real-time data on animal health, human nutrition, and livelihoods. It also funds research on poultry education in schools. Funding under this research theme is also supporting several graduate student fellowships and early-career research grants that are looking at animal disease issues in South Asia and Africa. Ecosystem Health: Resiliency of Socio-Ecological Systems – The Transhumance, Natural Resources, and Conflict in the Sahel (TRANS) in Senegal predicts the effect of climate change and land-use patterns on key resources such as water, fodder, and movement corridors by using a pastoral ecosystem model. It is collaborating with another Livestock-Climate Change Innovation Lab project that is mapping livestock movement corridors and identifying potential areas of conflict. This research explores how changes in climatic conditions are affecting the mobility patterns of pastoralists. In collaboration with Senegalese partners, South Dakota State University (SDSU) researchers are collecting survey data that will be used in a coupled pastoral systems model that will help policymakers and pastoralists explore the impacts that climate and land-use changes will have on pastoral resources in the future. Syracuse University is leading a complementary project that is examining the services and support that riverine resources in the Senegal River Valley provide for pastoralists along transhumance corridors. The researchers are using geo-referenced data to identify key corridor points that are supported by riverine resources. Riverine systems are 69 important because they provide dry season grazing reserves for herders and sources of water. An initial map of a “transhumance shed” has already been generated. SDSU is strengthening regional capacity for GIS and remote sensing while Syracuse is building capacity in the analysis of feed and forage quality. In Nepal, another Livestock-Climate Change Innovation Lab researcher is identifying the impact of ecosystem changes on livestock systems in the high hills. Pro-Poor Value Chains: Market Access and Reliability – Livestock holders in East Africa are facing increasing temperatures, longer periods of drought, and heavy but erratic rains. The increasing variability in precipitation is affecting river runoff, water availability and subsequently the natural recharge of groundwater and surface water. The Livestock-Climate Change Innovation Lab is funding research on livestock market behavior under changing climatic conditions, disease distribution and animal health. Current research is actively dedicated to mitigating livestock loss by identifying “best-bet” strategies for herders. Emory University researchers are collaborating with Ethiopian colleagues to provide a qualitative and quantitative assessment of market opportunities for small scale livestock buyers and traders in an uncertain physical and commercial climate. Kenyan and Ethiopian partners are determining how herders access market chains in remote areas, how weather-related risks affect market access, and which producer groups (including women) benefit from different markets. Another market-focused research project in Ethiopia serves as a complement to this work. A graduate student fellow is working on market-related issues in this same area and seven early career scholars (three Ethiopian and four Kenyan) have been funded with mini-grants to investigate climate impacts on livestock disease distribution and animal health. Additional information on the Livestock–Climate Change Innovation Lab can be found at: http://lcccrsp.org/. Geographic Focus The Livestock-Climate Change Innovation Lab is working in the following Feed the Future countries: Ethiopia, Kenya, Senegal, Tanzania and Nepal. Mali activities were suspended in 2011 due to the political unrest. Key U.S. Partners Arizona State University, Emory University, Michigan State University, Syracuse University, Texas A&M (only in Mali), South Dakota State University, City University of New York-City College (CUNY), University of California-Davis, Utah State University, Heifer International, Helen Keller International Funding Mechanism A Cooperative Agreement Leader with Associates Award was awarded to Colorado State University as the Management Entity (ME) for the Livestock-Climate Change Innovation Lab. The Livestock-Climate Change Innovation Lab is in its fourth year of its first five-year award which ends on April 31, 2015. Total funding received, as of September, 2012, is $9,358,096. In July 2011, the Livestock-Climate Change Innovation Lab also received a three-year Associate Award from USAID/Mali for $5,250,000 but was unable to complete the project due to political unrest in 2012. There are no other Associate Awards at this time. The ceiling 70 for the five-year Leader with Associates award is $14,853,291.This evaluation scope of work will only focus on the Cooperative Agreement Leader funded activities. Scope of Work This performance evaluation will provide USAID and the ME with constructive feedback on the program management and research performance of the Livestock-Climate Change Innovation Lab. Furthermore, since the Livestock-Climate Change Innovation Lab will be completing its first five year phase in April 2015, the External Evaluation Team (EET) should consider whether a program extension for a second phase is warranted, and if so, make recommendations to USAID on any necessary management adjustments and potential research focus changes during a second phase. Specifically, the EET will objectively evaluate the following using an evidence-based and data-driven approach: Program Management 1. Technical leadership – Assess the ME’s technical leadership of the program, including how it has built on past investments while having a vision for new opportunities and constraints; engaged partners in the U.S. and overseas, including USAID Missions, CGIAR centers and NGOs; balanced research, technology dissemination, training and capacity building demands; and promoted scientific collaboration and exchange among all its partners. 2. Administration – Assess the ME’s administration and management of the Livestock￾Climate Change Innovation Lab taking into consideration what systems are in place to ensure research activities are on track in accordance with program goals; roles and functions of advisory committees; and appropriate staffing levels, functions and level of effort. 3. Financial management - Assess how well the ME has managed the financial aspects of the Livestock-Climate Change Innovation Lab taking into consideration project resource allocations; checks and balances regarding grantee disbursements, expenditures, and reimbursement; and if cost matching requirements are being met. 4. Monitoring and evaluation – Assess the effectiveness of monitoring and evaluation efforts of the Livestock-Climate Change Innovation Lab to include whether there are systems in place to capture research impacts and how effective they are, whether baselines and targets have been established and met, the appropriateness of indicators, and quality of data. Research Program 1. Research depth, breadth and rigor - Assess how well the Livestock-Climate Change Innovation Lab research activities have contributed to stated program goals and objectives including the sub-award process and the balance between domestic and international, strategic and applied research. 71 2. Collaboration, outreach and technology dissemination – Assess the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in these areas to include whether they have partnered with appropriate collaborators, whether outreach strategies have been integrated into project design, how research outputs are disseminated, and the extent to which progress has been made in technology dissemination and scalable technologies. 3. Human and institutional capacity building – Assess the effectiveness of the Livestock￾Climate Change Innovation Lab’s human and institutional capacity building in terms of level of effort, investment, and selection of candidates and institutions. 4. Gender inclusion – Access the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in gender inclusion including how gender is incorporated into project design, training and output activities. Program Future Based on the EET’s evaluation of the program’s management, research activities and results to date, provide recommendations to inform USAID’s decision as to whether to seek a five￾year extension for the Livestock-Climate Change Innovation Lab. Note, a recommendation to not pursue a second phase of the Livestock-Climate Change Innovation Lab does not mean that USAID would necessarily discontinue research in this area. A likely outcome would be a new competition for continued research in this area for which the current ME institution could compete. If a program extension is warranted, please provide recommendations as to how the program could be strengthened or alignment with Feed the Future improved during a second phase, including potential changes in program management, sub-award selection, research focus, and/or collaborating partners. Evaluation Methodology 12 The evaluation will be based on the following tasks, conducted in the following order, and completed by the dates given. 1) Conference call with USAID - between October 28 – November 1, 2013 A conference call will be scheduled with the USAID Evaluation Manager, the USAID Livestock-Climate Change Innovation Lab AOR, and other USAID officials in the Research and Monitoring & Evaluation Divisions to review the scope of work and answer questions concerning the implementation and delivery of the evaluation. 2) Desk review - completed by November 8, 2013 The EET will conduct a desk review of Livestock-Climate Change Innovation Lab publications and materials. The purpose of the desk review is to obtain needed background and context about the Livestock-Climate Change Innovation Lab and USAID in order to complete the Knowledge Gap Table and the Evaluation Plan (see below). 12 The start of the Evaluation was delayed due to the three week U.S. Government shut down in October 2013. This pushed back the due dates for all the deliverables and required an extension of the evaluation team’s scope of work to March 31, 2014. 72 Documents to be reviewed will include, but are not limited to, the RFP (request for proposal), approved program proposal, the Leader Cooperative Agreement, annual reports, work plans, program operation documentation, monitoring and evaluation documentation, and funded research proposals. The material to be examined will be related to the current phase of the Livestock-Climate Change Innovation Lab and will be made available by the AOR and the ME13. Previous livestock related Collaborative Research Support Programs (CRSPs) are not to be included in this evaluation. Team members will also familiarize themselves with the Feed the Future Global Food Security Research Strategy14 and the USAID Evaluation Policy15. 3) Knowledge Gap Table – due November 11, 2013 Based on the desk review, the EET will provide the USAID Evaluation Management the completed Knowledge Gap Table (see Appendix A). 4) Evaluation Plan - due November 18, 2013 Using the Knowledge Gap Table as a guide, the EET will submit to the USAID Evaluation Manager the Evaluation Plan (see Appendix B). The purpose of the Evaluation Plan is, in part, for the EET to present their evaluation design which includes, in part, research questions, methodology for quantitative and qualitative data collection and data analysis, work plan, timeline and proposed domestic and international travel. The Evaluation Plan must be approved by the USAID Evaluation Manager before the EET can travel and begin their field work. USAID will provide approval or request changes by November 20. If required, the EET will submit a revised Evaluation Plan by November 22. 5) Domestic and international travel – to be completed by January 25, 2014 The EET will need to travel domestically and internationally to gather the needed information to implement the evaluation plan and complete this scope of work. Domestic travel is limited to one trip to visit the Livestock-Climate Change Innovation Lab ME at Colorado State University. International travel is limited to two separate trips to visit international collaborators and stakeholders with the Livestock-Climate Change Innovation Lab. The USAID Evaluation Manager will pre-approve all travel. All travel must be in accordance with U.S. Government travel regulations and follow the travel protocol supplied by the USAID Evaluation Manager. Proposed international travel plans will take into account national holidays of the countries to be visited. 6) International travel debriefs – prior to country departure A short summary of data collected and preliminary findings will be sent to the USAID Evaluation Manager for each country visited before departure from the country. This is not to be a trip report, nor should time be billed to write a trip report. Instead, it is meant to provide the USAID Evaluation Manager with progress made against the Evaluation Plan. 7) Preliminary findings – due February 7, 2014 13 Many of these documents can also be found at: http://crsps.net/resources/by-crsp/livestock-climate￾change/?details=1 and http://lcccrsp.org/. 14 http://pdf.usaid.gov/pdf_docs/PDACR702.pdf 15 http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf 73 The EET will provide in writing to the USAID Evaluation Manager the preliminary findings that will be used to develop the draft evaluation report. 8) Draft evaluation report – due February 14, 2014 A draft of the evaluation report will be submitted electronically in MS Word format to the USAID Evaluation Manager. USAID will review the draft for content. The ME will review the draft for accuracy. All comments, corrections and suggestions for consideration will be sent to the EET by February 21. 9) Final evaluation report – due February 28, 2014 The final evaluation report should sufficiently address all comments and corrections provided to the draft report. Evaluation Report Format The evaluation report will present findings, evidence-based recommendations and conclusions of the topics outlined in this Scope of Work. The EET may include other topics that are deemed relevant and are evidence-based. The report should follow the format and page limits as outlined in Appendix C. The USAID Evaluation Manager will be made available to the EET as a resource person but will not contribute directly to the preparation of the report. Level of Effort The level of effort for the entirety of this Scope of Work will consist of no more than 45 billable days for the Team Leader and 40 billable days for each of the other two team members. All billable work is to be performed between October 1, 2013 and February 28, 2014. The following is the authorized number of billable days for each team member and leader for each task/ deliverable of this scope of work. Changes of more than two days for a task/deliverable must be authorized by the USAID Evaluation Manger in advance, before the days are worked. Significant changes will require the submission and approval of a new Evaluation Plan work plan (see Appendix B) before additional days are approved. LEVEL OF EFFORT (by billable days) Task/Deliverable Each Team Member Team Leader Conference Call/Desk Review 3 3 Knowledge Gap Table 2 2 Evaluation Plan 2 2 Travel16 17& Travel Debriefs 21 21 Preliminary Findings 5 5 Draft Report 5 8 Final Report 2 4 Total 40 45 16 The EET is expected to work a six day work week while traveling. 17 Four and six travel days were added to the LOE for the Team Leader and Wyn Richards, respectively. 74 Payment of Services The EET will be paid by the University of Missouri. The University will negotiate contracts with each EET member and determine the daily rate of compensation in accordance with U.S. Government regulations and based on verifiable past work experience. Payment will be made on a monthly basis in accordance with the billable day limits per task/deliverable outlined in the Level of Effort table above. Team Composition and Qualifications The technical qualifications of EET members must be matched with the technical areas of focus of the Livestock-Climate Change Innovation Lab. Team members must have the expertise necessary to evaluate the Livestock-climate Change Innovation Lab and to address the Scope of Work topics. USAID will designate one team member as the Team Leader. Administrative/management member (1): A senior administrator with a minimum of ten years experience managing and/or evaluating multifaceted international development research and/or university-based programs. The preferred candidate will be familiar with both university-based programs and USAID (or other donor) funded programs. A background in agricultural development, with technical expertise in a field relevant to animal science, veterinary medicine and climate change research is preferred. The candidate will also have: a) a demonstrated capacity to conduct independent program evaluation; b) an understanding of USAID’s foreign assistance goals, and its particular objectives related to collaborative research, agricultural development and food security; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Technical team members (2): Must be recognized experts in international development related to agriculture with specific expertise in livestock and climate change research. Team members will be chosen from those who have experience in such areas as animal science, sustainable agriculture production, climatology, agricultural economics, and/or natural resource management particularly rangelands. Technical team members will also have demonstrated the following: a) the capacity to conduct independent program evaluation; b) a thorough understanding of research methodology; c) experience in effectively conducting outreach and dissemination to policymakers, development practitioners and/or the private sector; and d) the ability to analyze issues and formulate concrete recommendations orally and in writing. 75 Appendix A: Knowledge Gap Table Key Knowledge Knowledge Gaps Program Management Technical leadership Administration Financial management Monitoring & evaluation Research Program Depth, breath, rigor Collaboration, outreach, technology dissemination HICD Gender inclusion Future of Program 76 Appendix B: Evaluation Plan FTF Activity/Mechanism Name: FTF Activity Country/Countries: Evaluation Lead Investigator: USAID Evaluation Manager: Approximate start date: Preface This document describes the components needed to complete an Evaluation Plan for Feed the Future (FTF) Activities. A. FTF Project Evaluation Design 1. FTF Activity/Mechanism Description Describe the FTF activity/mechanism being evaluated. Provide enough detail to make clear the justification for the proposed methodology. Include the following items: activity/mechanism goals and objectives, main program components/interventions and delivery mechanisms, key activity/mechanism outcomes and indicators, target areas and target population groups, criteria for selecting target areas, criteria for selecting program participants, program implementation plan (start date, duration, deployment plan and timeline).(Note: much of this material can come from project documents.) 2. Program Logic Please include either a diagram and/or a narrative that describes the program logic and articulates the causal pathways from activity implementation to the desired impacts. The description should include intermediate outcomes that would change along the way to final impacts or objectives of the project. (Note: this should also be available in project documents.) 3. Evaluation Research Questions Succinctly state the primary questions that the evaluation will seek to answer.(Note: this should be available in the evaluation SOW.) 4. Methodology for Quantitative and Qualitative Data Collection Please indicate briefly the methods and plans for data collection. This section should include all methods for primary collection (interviews, surveys, direct observation, etc.) and secondary data collection (project documents, performance reports, etc.). Provide the timing of any qualitative and quantitative data collection and explain how the two 77 will be integrated. Include the number of planned survey rounds as well as the expected local data collection partner if applicable. 5. Methodology for Quantitative and Qualitative Data Analysis Describe the methods you will use to analyze the quantitative and qualitative information collected. Analysis methods should be described in detail for both quantitative (descriptive statistics, regression analyses, etc.) and qualitative (domain analysis, network analysis, etc.). Also, specific software that will be used should be mentioned (SPSS, STATA, ATLAS, etc.). 6. Outcome Measures Briefly discuss the outcome measures that will be used for this study (quantitative and qualitative) and relate them to the evaluation research questions. Explain which evaluation questions the quantitative and qualitative data will help address and how. Define the variables or indicators that will be used to measure these outcomes. (A quantitative example would be an outcome measure of “Greater access to new technologies among partner developing countries” and corresponding indicator “Number of new technologies under research, field testing or made available for transfer”. A qualitative example would be an outcome measure of “Effective management” and corresponding indicator of “Communication processes are well-established”.) 7. Additional Pertinent Information Use this section to describe any further information that is pertinent to this particular evaluation and should be considered as part of the evaluation design. For example, this section could be used to discuss collaboration agreements for analysis with other institutions or overlaps with other evaluations and coordination with those evaluations. Evaluation Work Plan (adapt timeline as required) Activities Dates of Activity 1st Month 2nd Month 1 2 3 4 5 6 7 8 TASK 1. Develop evaluation design and implementation plan Activity 1: Activity 2: etc. TASK 2: Data Collection Activity 1: Activity 2: etc. TASK 3: Data Analysis Activity 1: 78 Activities Dates of Activity 1st Month 2nd Month 1 2 3 4 5 6 7 8 Activity 2: etc. TASK 4: Report Writing Activity 1: Activity 2: etc. B. Data Collection and Management Plan 1. Interviewer/Enumerator Training (if any) Describe the plans for training for all data collection (if any), including length of training, location, expected number of participants, topics covered, and the approach to piloting or field testing during training. 2. Data Management and Security Describe how all data collected will be gathered, entered, managed, and stored. Please specify how data will be kept secure. 3. Data Collection Approvals Describe the process and results of all data collection approvals. C. Data Collection Instruments Submit a draft of any data collection instruments that will be used for the evaluation. 79 Appendix C: Report Format Title Page Table of Contents List of Acronyms List of Tables List of Figures Executive Summary (3 pages) Program Management (15 pages) - Findings - Conclusions - Lessons Learned - Recommendations Research Program (15 pages) - Findings - Conclusions - Lessons Learned - Recommendations Program Future (5 pages) - Program Management - Research Program Appendices A. Scope of work B. Evaluation Plan C. Travel itinerary, locations and dates of field visits D. List of persons contacted E. List of materials reviewed F. Digital photographs: high resolution with caption and photo credit (5 photographs) G. Management Entity Comments on Report (to be added by USAID after final report received) 80 Appendix B: External Evaluation Team Biographies Karen Brown (team leader) Dr. Karen Brown is Assistant Vice President for International Scholarship and Director of the Interdisciplinary Center for the Study of Global Change (ICGC) at the University of Minnesota. Dr. Brown has responsibility for a system-wide program of international research funding and co-directs the Master of International Development Practice Program with the Humphrey School of Public Affairs and in collaboration with several other partner colleges. She develops and directs international and interdisciplinary initiatives including a graduate education and research partnership with the University of the Western Cape (South Africa) and the Development Studies and Social Change Ph.D. Minor. Dr. Brown also serves as affiliated faculty in Feminist Studies and as a member of the University’s Human Rights Program Advisory Committee. She holds a Ph.D. in Political Science (international relations and comparative politics) and M.A. in East Asian Studies from the University of Minnesota, and a B.S. in Chinese from Georgetown University. Her research and teaching interests focus on gender and public policy, international feminist theory, international women’s and children’s human rights, international research ethics, and girls in development. She was a member of the BIFAD-appointed external review team that evaluated the USAID Collaborative Research Support Program (CRSP) in 2012. Claire Heffernan Dr. Claire Heffernan is the Executive Director of the Low Carbon Livestock Trust, a public￾private partnership aimed at the creation of new technologies and approaches to enhance the sustainability of the global livestock sector. She is also the Director of the Livestock Development Group at the University of Reading. A veterinarian by training, she founded the group in 2000 based on the need for a meta-disciplinary approach to pro-poor development. Group disciplines include psychology, linguistics, philosophy, law, veterinary medicine, informatics, and computer science. She is currently a Senior Visiting Fellow at the Smith School of Enterprise and the Environment, University of Oxford where her work explores climate change and emerging infectious disease impacting both humans and animals. In 2011, she was chair of the Livestock Theme of the G20/Gates/World Bank US $100 million AgResults initiative. Her work in impact assessment and the creation of research programs has informed a wide variety of institutions including the World Bank and the Wellcome Trust. She holds a BA from Smith College, a DVM from Tufts University, and a PhD from the University of Reading. John Irwyn (Wyn) Richards Dr. Wyn Richards has more than 40 years development experience in Africa, Latin America and South Asia: as an academic and researcher for DFID in the development of pro-poor livestock systems; as a research manager/designer of large competitive livestock R+D programs for the UN and DFID; as a Communications Director of a large global ARD program for DFID; as an agricultural science journal reviewer; as a Convenor/Organizer of global donor and implementation group meetings in livestock research for development; as an UN 81 diplomat; and as an M+E Consultant. He is a Senior Fellow, Wolfson College, Cambridge where he mentors young development professionals. He earned his PhD (Cantab) from Cambridge University focusing on physiological responses of cattle to vitamin A+D deficiencies, and his B.Sc (Hons) in Agricultural Chemistry – Animal Nutrition from Reading University. 82 Appendix C: Knowledge Gap Table Key Knowledge Knowledge Gaps Program Management Technical leadership (1) Has the ME : -built on past investments while having a vision for new opportunities and constraints? -engaged partners in the US and overseas including USAID Missions, CGIAR centres, NGOs etc? -balanced research, technology dissemination, training and capacity building demands? -promoted scientific collaboration and exchange among all its partners? -resolved early concerns by USAID re -what is the distinct difference and organizational relationship between the Principal Investigator (PI) and Director? Establishing Regional Innovation Consortia to promote collaboration and increase dissemination. -Engaged with US Universities and in￾country academic and Govt counterparts on projects. -A list of potential partners has been provided at start-up, its presently unclear how these relationships have evolved over time. Appears to be ok from documentation, need additional information on budget allocation . Appears to be ok from documentation, will learn more from interviews with stakeholders (PIs, TIRIs, etc.) Roles and Responsibilities of the 2 Directors and •Is there evidence of a strategy to develop a cohesive programme from the diverse set of research topics/ team leadership? How is such a strategy developed, and does advisory board play a role? •Is there a program logframe to enable the management to collate and integrate findings from the different projects into a cumulative whole – or from to enable manage to construct outputs to outcome statements? •Does ME effectively leverage program investments with other funding streams or projects? •Is engagement with partners effective in project design and implementation? •Have a specific set of metrics to measure impacts been put in place? How have the Feb 2013 recommendations been implemented re: ME and what is the expected relationship with FtF expected outcomes? Proportional budget allocation between research, capacity building and management requires further explication. Need more on the wider efforts at public engagement in research outputs and impact of efforts to date. •Justification for establishing RICs and their status •Justification for Seed Grant Projects •Need more on overall communications strategy •Relationship between TIRI and LRTPs is unclear at the moment 83 support team provided. Original PI resigned early on but the position not filled. •Difference between Tiri scholars and Graduate Research Scholars? •Justification of regional differences in funding levels. •Need more information on hierarchy/governance, especially relationship between Directors and Board. Is the Board solely advisory? What is Board role in scientific oversight and program oversight? •Were the responsibilities of the PI assumed by the Director and Deputy? •Why did the PI resign and who took over responsibilities? Did M&E responsibilities all remain with Deputy Director? •Clarify roles of Director/Deputy Director and who provides interface with USAID, and distinction from PI role (PI as spokesperson/liaison between USAID and University as well as lead expert? Director addresses daily management issues of team and research portfolio? Administration -Are systems in place to ensure research activities are on track in accordance with program goals? -Is information available on all projects and activities commissioned since the inception of the programme? There is a Policy and Operating Procedures Manual (2012) and a monitoring and evaluation system. Information provided on all TIRI and LRTP projects currently being funded. •Need full proposal doc submitted by 11 August 2009 •Is there a logframe or an outcome mapping doc? •Further information is required - the POP manual offers little detail and references to the M&E system have been made and indicators requiring further explication (e.g. gender) but across the documents there is often a lack context and detail. •A number of short projects were initiated by CRSP in 2011 and finished in 2012. They’re mentioned on the website but need more information. E.g., a pastoral transformation project run by Kathleen Galvin and Robin Reed (CSU) and a UCD project on Livestock and Livelihoods. •Need more detail on the justification of the projects funded, how they fit into the new themes (or indeed why the 5 84 -Are the roles and functions of advisory committees compatible with achieving these goals? -Are staffing levels, their function and the level of effort appropriate to achieving program goals?* -Has the management office routinely considered environmental issues, as appropriate, in both solicitations for sub awards and as a selection criterion? -Is the administrative and management relationship between the Management Office and sub-award institutions satisfactory? -Is the relationship and communication with USAID and Missions satisfactory? Current membership of the Advisory Board and External Review Panel on website ** see below. Also see TORs of Board and external review panel. Several new members on the Board. LCC-IL personnel list provided – including their roles and responsibilities. themes changed to 4) or how these themes meet the aims of the wider FtF programme. •Clarification is required over the number of pilots and communities/households impacted, the number of LTR projects and the number of communities/households impacted. Basic monitoring data is not yet available. •A timeline of events i.e. when specific elements were commissioned/came on board has been requested. •Need to interview Director (and co￾Dir) of Program with respect to their authority vis a vis the Board and External Review Panel, and the overall governance structure of the Program. •Current Co-Dir has assumed her previous responsibilities (in M+E) and the previous Deputy Director responsibilities. Is this too much? •Unclear whether the staff positions are full-time or part-time and whether they are fully or partly funded by the Program or whether staff have been seconded from CSU •A program of the size, complexity and nature of this one needs full-time staff members with dedicated responsibility (e.g., for communications and M+E). Four people identified with communications roles but need communications strategy doc composed in 2012. •Has the lead university ensured that environmental issues are considered and incorporated into its research, training and outreach activities? Evidence? •The primacy of climate change adaptation at the community level is often conflated with other issues across the documents, as presented. 85 •Have any IEESs (Initial Environmental Examinations) been completed? •Interviews with PIs and their institutions via Skype or phone to assess (both US universities and host country co-PIs). •Interviews with in-country Missions in the countries we visit – and by Skype in those target countries we don’t visit. Only one Associate Award made? Why? Particular attention to relationship with Missions and role of Missions and other host country stakeholders in developing projects. Financial management -How well has the ME managed the financial aspects of the LCCIL taking into consideration project resource allocations, grantee disbursements, expenditures, reimbursements? - Are cost-matching requirements being met by CSU? Document Review indicates that budgets are tracked in near‐ real time and spreadsheet structure designed to help ME stay up to date with the situation Cost sharing budget of US$ 2.3 million by CSU in contract. •Is this working? •How can it be improved? •Need to confirm during interviews with Finance Dept in CSU and field projects. •As noted above the proportional allocation of funding across activities requires further explication. •Confirm with Finance Dept realized cost sharing. Monitoring & evaluation -External evaluation – -Internal monitoring an review How effective are the M+E efforts of the LCCIL? -Are systems in place to capture research impacts and how 2 external evaluations described in the Head Contract Co Director has designed and operationalized a •Not clear at the moment which of the 2 external evaluations described we are undertaking. There seems to be a great deal of overlap between the two. When is the 2nd evaluation going to be conducted? What is the relationship between the two? • Review performance monitoring plan – PMP or an M+E strategy for the program and the short and long term projects. 86 effective are they? -Have baseline targets been established and met? -Are the M+E indicators used appropriate? - Will the program deliver three to five measurable development outcomes by the conclusion of their five year award – as stipulated in the call doc? - Are the data generated of high quality, robust and relevant? Output indicators Evaluation issues Performance Monitoring Plan (PMP) and Monitoring and Evaluation Plan (M&E) to support field based sub-grantees with their monitoring and evaluation strategies. Latest Performance Narrative reports (Oct 2013) indicate completion of collection of baseline values by most LTRPs and TIRIs Use of Agriculture Standard USAID indicators suggested in Annex 6 of contract. Also mention of Regional Vulnerability Analysis. To be confirmed. Some projects much more advanced than others. Performance narratives indicate so Evidence to be gathered of 2 things: 1. Strengthening the Agriculture Enabling and Policy Environment and 2. •Review these document •Evidence of MEs selection of at least six expected results from the three (or four?) research areas presented and evaluation against in the fourth year of the program. •Monitoring of performance information seen on website indicate some of the Agriculture Standard USAID quantitative indicators are being collected. •The Performance Narratives on their own are unlikely to provide scalable M&E data. A collation of these narratives has not presently been viewed by the review team. Therefore key gaps in the approach are not yet clear. •Need to confirm rigor of such data during field visits – and relevance of baseline of indicators employed •Need to review M+E strategy, guidance given to PIs and teams on this topic and interviews with staff in￾country with responsibility for M&E issues. •No data is available on the agreed development outcomes as detailed in the Feb 2013 document. •This topic to be evaluated during field visits •Need review of raw data and their interpretation during our field visits. 87 Improving Agricultural Sector Productivity A framework has been developed to assess individual project accomplishments across LCC CRSP strategic objectives and cross-cutting themes (i.e., gender equity, nutrition), within a regional context. •Need to confirm whether these indicators likely to be met during our field visits. •With all these evaluation issues, the presence of an overall strategic framework (like a logframe) is probably needed. We need to confirm whether such a structure exists. Research Program Depth, breath, rigor --How would you rate the quality and progress of the research conducted and products generated and disseminated to date with respect to performance and relevance to USAID development priorities? Rigor of research processes -How well have the LCCIL research activities contributed to the stated program goals and objectives? - Has the research achieved outreach and development impact benchmarks? - What is the degree to which the research activities have achieved Published a typology of adaptations that specifically addresses livestock production and its subsectors (e.g., water availability, vegetation and land use, fodder, disease, market access, value chains). Oversight by the RVA Coordinator Reports indicate design and implementation of LRTPs are very strong. The East/West Africa PIs are very senior researchers likely to deliver on their objectives and outputs. Goals and Objectives stated at base of this doc*. These benchmarks identified in Revised Implementation plan •Need to review typology Who is the RVA Coordinator? •Some TIRIs have somewhat dubious methodologies and over ambitious aspirations of outputs. These are really scoping projects with minimum funding. Can’t work out what happens to successful TIRIs – whether they then become candidates for LRTPs – or are they stand alone? What is the strategy? •Evaluate this in relation to human capacity development and goal of institutional capacity building and scientific network creation. •Reports indicate that all LRTPs are aware of the goals and objectives of the programme. •Need to check whether these benchmarks likely to be achieved. 88 integration and relevance to development policy and programming in￾country and more broadly? - Has the sub-award process contributed positively? -Has the balance between domestic, international strategic and applied research been effective in addressing the programme goal? -What are the pros and cons of research conducted to date? Themes, processes, staffing, funding, timelines etc Have funded projects integrated nutrition within their planned scopes-of-work as required by FtF? July 2011, pages 6-9. Most PIs have made an effort to include Govt bodies in team activities. The RICs too will include policy￾making and implementing bodies so there is strong possibility that emerging policies will be implemented. Appears effective Appears effective Nutrition expert in place to ensure this is implemented. A single cross-cutting project incorporating nutrition has recently been funded. •During field visits we need to interview senior Govt staff to ascertain whether they are really active in addressing CC - or is the interest one of capitalizing on a global concern? •Need to distinguish applied research and actual development activities. This is an area that needs much better exploration within the context of the in-country field visits. •As noted above, the thematic shifts require further explanation. •Are there data available on complementary issues which affect environment such as population, farming practices, livestock numbers, demand for livestock products, exploitative issues such as forestry, tourism etc that impact on the environment and local climate? -Call required at least 6 researchable topics within 4 given areas. Where are they and why were these selected? •Rigor issues – in assessing CC, what is the long-term trend in climate in the target countries? Is there evidence to indicate that any of this is man made at the local level so that mitigation measures might also be considered as well as adaptive measures? •Speak to Nutrition expert in CSU to find out how she accomplishes this. Collaboration, outreach, technology dissemination •Evidence of linkages with USAID, international and national livestock initiatives? 89 -How effective has the LCCIL been in collaboration, outreach and technology dissemination? In particular: -Has the program partnered with appropriate collaborators? -Have outreach strategies been integrated into project design? - How are research outputs disseminated? -What is the extent of progress in technology dissemination and scalable technologies? •Evidence from other USAID programmes of policies that have been applied in this programme? Eg credit, food safety etc •Evidence of improved academic curricula and technical training programmes? •Evidence of collaboration/linkages with all the players involved in pro￾poor livestock development? •Is there an electronic system that will store information and make it widely accessible? •Establishment of RICs and their influence to date? •Why no PIs from CSU? What happened to the Dr. Galvin-led Colorado State University proposal in the Horn of Africa? •The programme dissemination strategy is presently unclear. HICD - Human and Institutional Capacity Development -How effective has the research program’s human and institutional capacity building process been in terms of effort, investment and selection of candidates and institutions •Need more on how host country institutional partners are identified, and how longer term connections with host country researchers are maintained. •Data on gender composition of host country scientific partners needed. Capacity Strengthening -How effective has the LCCIL’s efforts been in i)strengthening the capacity of targeted in￾country colleagues to address challenging livestock-human￾environmental health issues? ii) increasing the Reports indicate considerable training been conducted. Field visits and stakeholder meetings have been conducted. Across the meetings a range of issues have been raised. Information on the activities agreed and subsequent •Need to compile evidence •Is there evidence of training policy makers on more beneficial policies to assist resource poor livestock keepers adapt to CC? 90 resilience of livestock keepers in adapting to climate change? outcome and impacts are unknown. Reports indicate training undertaken •Need to confirm the value of this training during field visits. Community/household level monitoring data is presently unavailable. Gender inclusion -What level of effort has been expended by the LCCIL in gender inclusion? -How is gender incorporated into project design, training and output activities? -How effective have these efforts been? All funded projects address gender inclusiveness within their planned scope-of￾work. Assured by Gender expert. •Need to check this with Gender expert. Does she use the Gender Parity Index or WEAI or other indicators to ensure appropriate gender inclusion in project design and research uptake? •To be confirmed. Need to interview Gender expert. Does gender also include age related issues of both sexes? •Need data on gender composition of various categories of stakeholders, as well as understanding of how gender analysis is incorporated beyond counting women. E.g. are questions of property ownership and household decision making integrated in analytical frameworks for projects? •Need to assess this during field visits Future of Program -Recommendations to inform decisions on program extension, modification, closure. -Suggested changes/improvements • Programme Goal – from the contract doc. The goal of the Livestock-Climate Change Innovation Lab is to increase resilience and augment the income of livestock producers in regions where agricultural systems are changing, available resources are shrinking and climate is having an impact. From the RFP 2010 - The goal of this program is to increase the incomes of livestock producers and reduce risk associated with climate change Four research themes are: i) Climate extremes and long-term change; ii) Animal Health: Disease Distribution and Resilience; 91 iii) Ecosystem health: Resilience of socio-ecological systems; iii) Pro-poor Value Chains: Market access and Reliability. ** Advisory Board TORs • Promote the mission and activities of the program • Review program steps and accomplishments • Identify knowledge gaps and priority research areas that should be considered • Participate in the discussion for the future direction of the program • Be an advocate for the program Members (Current) • Adegbola Adesogan • Alice Pell • John Johnston (Scientific Liaison, USDA, Food Safety and Inspection Service) • Linda Logan • Tag Demment • Iain Wright External Review Panel Ozzie Abaye, Virginia Tech DeeVon Bailey, Utah State University Larry Granger, USDA-APHIS:VS Jesse Njoka, University of Nairobi Mark Powell, USDA-ARS Gene Takle, Iowa State University Tom Wirth, EPA TORs of ERP Provides objective evaluation of research proposals on an as-needed basis, reporting their findings and recommendations to the ME. Assist in the evaluation of on-going research activities on an as-needed basis. Members represent the disciplines covered in the program goals and objectives. Members mid 2012: Ozzie Abaye (Virginia Polytechnic Institute and State University), DeeVon Bailey (Utah State University), Milton Boyd (University of Manitoba), Larry Granger (USDA Animal Plant Health Inspection Service), Mark Powell (USDA Agricultural Research Service), Gene Takle* (Iowa State University), Tom Wirth* (Environmental Protection Agency). *Rotating off the ERP in May 2011. Programme Objectives • Build local human and institutional capacity for livestock productivity, veterinary and human health and environmental sciences. • Extend and apply research findings and technical knowledge to livestock producers, associated industries and public and private extension and agricultural services and strengthen the capacity of livestock producing households and related businesses to adapt to or cope with the impacts that unpredictable climate variability is imposing on them. • Develop policies that support national and regional programs that bring livestock producers, traders, veterinary and human health officials as well as government policy makers around the agenda of responding appropriately to the anticipated changes in resource availability. 92 Appendix D: Evaluation Plan Evaluation Plan Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change FEED THE FUTURE Activity/Mechanism Name: Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change FEED THE FUTURE Activity Country/Countries: Nepal, Ethiopia, Kenya, Tanzania, Senegal Evaluation Lead Investigator: Karen Brown USAID Evaluation Manager: Carole Levin Approximate start date: October 1, 2013 A. FEED THE FUTURE PROJECT EVALUATION DESIGN 1. FEED THE FUTURE ACTIVITY/MECHANISM DESCRIPTION Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change The Colorado State University (CSU)- led Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change (hereafter called the Livestock￾Climate Change Innovation Lab) supports integrated research that helps small-scale livestock holders adapt to environmental and health impacts of climate change in Sub￾Saharan Africa and South Asia. Colorado State University received a five-year, $15 million Leader-with-Associate Award from the U.S. Agency for International Development (USAID) in 2010 to manage the Collaborative Research Support Program, Adapting Livestock Systems to Climate Change (Livestock-Climate Change CRSP). This transitioned to the Feed the Future Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change. The Livestock-Climate Change Innovation Lab receives funding under the authorization of Title XII of the International Development and Food Assistance Act of 1975 in order to achieve the mutual goals among nations of ensuring food security, human health, agricultural growth, trade expansion, and the wise and sustainable use of natural resources. The Livestock-Climate Change Innovation Lab has its origins in the previous Global Livestock CRSP. Originally established in 1978 as the Small Ruminant CRSP, the Global Livestock CRSP was one of nine CRSP programs developed under Title XII. The current Livestock-Climate 93 Change Innovation Lab builds on the successes of the previous Global Livestock CRSP while addressing new and emerging challenges related to global climate change. Climate variability is an important determinant of animal, human, and environmental health. Changes in precipitation and temperature can affect the quality and quantity of forage available to animals at times when it is needed most. In addition to facing challenges from a changing climate, populations in South Asia and Sub-Saharan Africa are also undergoing rapid social transformation, facing reductions in communal resources and the reorganization of social obligations, networks, and governance. Recognizing that issues in environmental, animal, and human health are inextricably linked Livestock-Climate Change Innovation Lab focuses its research on eco-regions and/or river systems that extend across national boundaries and meet the following criteria: 1. Climate change is predicted to significantly increase vulnerability of livestock systems in the area because of predicted impacts on already stressed social and environmental systems; 2. Opportunities for collaborative research activities exist with other institutions and organizations; 3. Opportunities exist for trans-boundary research that has applicability at the community level. The Livestock-Climate Change Innovation Lab seeks to increase resilience and augment the income of livestock producers in regions where agricultural systems are changing, available resources are shrinking, and climate is having an impact. It supports research that aids individuals and communities to make choices and take actions that lead to sustainable livelihoods in the face of climate change. The Livestock-Climate Change Innovation Lab focuses on the following research priorities to achieve its goals: • Improve the health and productivity of livestock while benefiting the health and welfare of farm families and conserving natural resources. • Integrate market research with the needs of small-scale farmers and facilitate partnerships among farmers, buyers, processors, and farmer organizations. • Collect and analyze data from farms, partners, and governments that informs evidence-based solutions. • Increase research capacity through training and educational support to individuals in Sub-Saharan Africa and South Asia. • Extend the reach of funded research by leveraging resources and partnering with other organizations with similar research and development goals. The Livestock-Climate Change Innovation Lab focuses on the following key principles to achieve its goals: • Improving the health and productivity of livestock of the rural poor, working with small-scale farmers to support their efforts to nourish their families and increase village resiliency. • Supporting research that is informed by the local realities of small-scale farmers, with an. interest in evidence-based solutions that are relevant and affordable. • Increasing productivity that enhances animal, human, and environmental health, 94 with a focus on innovative approaches that increase productivity, enhance health, and conserve resources. • Funding research where women play a central role, recognizing that women are fundamental to the success of farm-based initiatives and expecting research solutions to address gender gaps and address inequalities. Key research themes: (5) Climate extremes and long term change (6) Pro-poor value chains, market access and reliability (7) Animal health: disease, distribution and resiliency (8) Ecosystem health: resiliency of socio-ecological systems 2. PROGRAM LOGIC Please see attached Logframe for program logic. 3. EVALUATION RESEARCH QUESTIONS I. Program Management A. Technical leadership 1. Has the ME built on past investments while having a vision for new opportunities and constraints? 2. Has the ME engaged partners in the US and overseas including USAID Missions, CGIAR centres, NGOs etc? 3. Has the ME balanced research, technology dissemination, training and capacity building demands? 4. Has the ME promoted scientific collaboration and exchange among all its partners? 5. Has the ME resolved early concerns by USAID concerning the– roles and organizational relationship of the Principal Investigator (PI) and Director? B. Administration 1. Are systems in place to ensure research activities are on track in accordance with program goals? 2. Is information available on all projects and activities commissioned since the inception of the program? 3. Are the roles and functions of advisory committees compatible with achieving these goals? 4. Are staffing levels, their function and the level of effort appropriate to achieving program goals? 95 5. Has the management office routinely considered environmental issues, as appropriate, in both solicitations for sub awards and as a selection criterion? 6. Is the administrative and management relationship between the Management Entity and sub-award institutions satisfactory? 7. Is the relationship and communication with USAID and Missions satisfactory? C. Financial management 1. How well has the ME managed the financial aspects of the Livestock-Climate Change Innovation Lab taking into consideration project resource allocations, grantee disbursements, expenditures, reimbursements? 2. Are cost-matching requirements being met by CSU? D. Monitoring & evaluation 1. How effective are the M+E efforts of the Livestock-Climate Change Innovation Lab? 2. Are systems in place to capture research impacts and how effective are they? 3. Have baselines and targets been established and met? 4. Are the M+E indicators used appropriate and compatible with USAID indicators? 5. Will the program deliver three to five measurable development outcomes by the conclusion of their five year award? 6. Are the data generated of high quality, robust and relevant? II. Research Program A. Depth, breadth, rigor 1. How would you rate the quality and progress of the research conducted and products generated and disseminated to date with respect to performance and relevance to USAID development priorities? 2. How well has the Livestock-Climate Change Innovation Lab research activities contributed to the stated program goals and objectives? 3. Has the research achieved outreach and development impact benchmarks? 4. What is the degree to which the research activities have achieved integration and relevance to development policy and programming in-country and more broadly? 5. Has the sub-award process contributed positively? 96 6. Has the balance between domestic, international strategic and applied research been effective in addressing the program goals? 7. What are the strengths and challenges of research conducted to date- themes, processes, staffing, funding, timelines, etc? 8. Have funded projects integrated nutrition within their planned scopes-of-work as required by Feed the Future? B. Collaboration, outreach, technology dissemination 1. How effective has the Livestock-Climate Change Innovation Lab been in collaboration, outreach and technology dissemination? In particular: a. Has the program partnered with appropriate collaborators? b. Have outreach strategies been integrated into project design? c. How are research outputs disseminated? d. What is the extent of progress in technology dissemination and scalable technologies? C. HICD - Human and Institutional Capacity Development 1. How effective has the research program’s human and institutional capacity building process been in terms of effort, investment and selection of candidates and institutions? 2. How effective has the Livestock-Climate Change Innovation Lab efforts been in: a. strengthening the capacity of targeted in-country colleagues to address challenging livestock-human- environmental health issues? b. increasing the resilience of livestock keepers in adapting to climate change? D. Gender inclusion 1. How is gender incorporated into project design, training and output activities? 2. How effective have these efforts been? 97 4. METHODOLOGY FOR QUANTITATIVE AND QUALITATIVE DATA COLLECTION Secondary Data Collection Sources: EET will review secondary data from the following sources: Program Management Documents: • Livestock-Climate Change Innovation Lab Leader Award, RFP and related correspondence • Policy and Operating Procedures including personnel and marking plans • Livestock-Climate Change Innovation Lab Advisory Board Documentation • Communications plan and products • Feed the Future Indicator Data for Livestock-Climate Change Innovation Lab projects • Monitoring and Evaluation plan • Internal Evaluation of Livestock-Climate Change Innovation Lab • Work Plans and Performance Narratives • Project Reports • Project Publications (e.g., brochures, newsletters, progress reports, research briefs and fact sheets) • Reports on PI meetings and other meetings • Terms of Reference (Scope of Work) for associated management staff Scholars and Fellows • TIRI Scholars and Graduate Research Fellows: RFP, Proposals, Contracts, Work Plans and Reports where available Long Term Research Projects (LTRPs) • LTRP Proposals and Budgets • LTRP Performance Narratives • Additional documentation from LTRP collaborators (e.g., annual reports, training materials, data collection records) Seed Grant Research Projects • Seed Grant Proposals • Seed Grant reports, products, indicators and trip reports Budget Information • EET requested and received current budget overview data Analysis: EET will systematically review and analyze documents to identify strengths and challenges of Livestock – Climate Change Innovation Lab for further investigation in primary data collection. Particular attention will be paid to analysis of issues emerging across multiple contexts or aspects of the project. Indicators will be analyzed for correspondence with project objectives and Feed the Future objectives. 98 Primary Data Collection We propose to visit all of the countries currently involved in the Innovation Lab (Nepal, Tanzania, Ethiopia, Kenya and Senegal). We base this recommendation upon the desirability of visiting as many projects as possible as well as the travel availability of EET members. We would attempt to visit components of most of the Long Term Research Projects (LTRPs) in these countries, interview as many TIRI scholars (and graduate trainees) as possible, observe scholars workshops intended for training purposes, visit USAID missions as well as meet with and interview other stakeholders (host country co-PIs and other collaborators, ministries, NGOs as relevant). Questionnaires have been devised for specific stakeholders to assist with ensuring consistency across evaluation sites (see attached documents). In general, these focus on: observing and evaluating the quality of research design and implementation; the extent and quality of capacity building activities; integration of research and capacity building activities in overarching project logic; evidence of addressing cross-cutting themes (gender, nutrition and climate change); monitoring and evaluation activities; impact evaluation; management practices; communications strategy; and community participation in project design. A proposed EET travel schedule follows: South Asia (Nepal) • Dates: November 24-December 2, 2013 • EET members participating: Karen Brown, Wyn Richards, Claire Heffernan • Evaluation data collection activities: • Development and field-testing interview protocols for PIs and project partners and stakeholders in order to ensure comparability in the data generated across the multi-country study set. Evaluate the robustness and reliability of the data generated in order to ensure the consistency and quality of our outputs. Refine interview instrument to guide semi-structured interviews across projects. • Interview project PIs and co-PIs, host country collaborators, USAID mission staff, NGOs, government agencies, project climate consultant, graduate fellows and TIRI Scholars in Kathmandu and at project sites in Kathmandu and Pokhara regions. • Observe and evaluate project activities at project sites in Kathmandu and Pokhara regions (for quality of research design and implementation; extent and quality of capacity building activities; integration of research and capacity building activities in overarching project logic; evidence of addressing cross￾cutting themes (gender, nutrition and climate change); monitoring and evaluation activities; impact evaluation; management practices; communications strategy; community participation in project design) • Observe training and research collaboration and innovation activities at TIRI Scholars workshop in Kathmandu (for scope and quality of collaboration opportunities; quality of research projects; effectiveness of professional training activities; resources provided to Scholars). 99 East Africa (Ethiopia, Tanzania, Kenya) • Dates: December 9-13, 2013 (exact travel dates TBD) • EET members participating: Wyn Richards (Ethiopia), Claire Heffernan (Kenya and Tanzania) • Evaluation data collection activities: • Ethiopia (4 days including travel): Interview project co-PIs, host country collaborators, USAID mission staff, NGOs, government agencies, graduate fellows and TIRI Scholars as possible using interview guide developed in Nepal. Trip will focus on Addis Ababa (TIRI Scholars, project staff, host country collaborators including ILRI staff, USAID staff, Program Advisory Board Member). Possible visit to Hawassa University if time permits to interview staff members involved in LTRPs and TIRI Scholars. • Tanzania (3-4 days including travel): trip will include one project site visit. Observe and evaluate project activities at project site in Tanzania. • Kenya (2-3 days including travel): trip will focus on Nairobi and interviews with TIRI Scholars, host country collaborators, USAID Mission staff. West Africa (Senegal) • Dates: Mid-January 2014 (4-5 days including travel) to correspond with TIRI Scholars workshop • EET members participating: Wyn Richards, Karen Brown • Evaluation data collection activities: • Interview project PIs, host country collaborators, USAID mission staff, NGOs, government agencies and TIRI Scholars using interview guide developed in Nepal. • Observe training and research collaboration and innovation activities at TIRI Scholars workshop (for scope and quality of collaboration opportunities; quality of research projects; effectiveness of professional training activities; resources provided to Scholars). • Observe and evaluate project activities at 1-2 project sites for quality of research design and implementation; extent and quality of capacity building activities; integration of research and capacity building activities in overarching project logic; evidence of addressing cross-cutting themes (gender, nutrition and climate change); monitoring and evaluation activities; impact evaluation; management practices; communications strategy; community participation in project design. 5. METHODOLOGY FOR QUANTITATIVE AND QUALITATIVE DATA ANALYSIS • Quantitative indicators will be analyzed for correspondence with project objectives and Feed the Future objectives. • Additional quantitative data collected on training opportunities and outcomes as well as project-related data will be provided as descriptive statistics linked to project objectives. • Interview data and field notes of project observation will be analyzed to address evaluation questions as identified above concerning themes of Program Management (technical leadership; administration; financial management, and monitoring and evaluation) and Research Program (research depth, breadth and 100 rigor; collaboration, outreach and technology dissemination; human and institutional capacity building; gender inclusion). 6. OUTCOME MEASURES Program Management Technical leadership Outcome measure: Clearly articulated research vision Indicator: Research plan or strategy demonstrates overarching vision Indicator: Project reports and communication, RFPs and awards demonstrate linkage between individual project and overarching strategy Outcome measure: Increased investment in Livestock-Climate Change-related research and capacity development through engaging partners Indicator: Number and value of associate awards Indicator: Amount of co-funding from CSU, other donors, and partners Outcome measure: Increased scientific collaboration and exchange Indicator: Number and type of opportunities created for scientific collaboration (e.g., RIC workshops, TIRI Scholar workshops, PI meetings) Indicator: New or enhanced collaborations developed (based on ME, PI and host country partner interview data) Indicator: Number and type of collaborative publications or funding awards received or in process (based on ME, PI and host country partner interview data) Indicator: Scope and content of research dissemination Outcome measure: Clear organizational leadership structure with well defined roles Indicator: Position descriptions for PI/Director and other program leadership positions establish roles and responsibilities Indicator: PI/Director and other program leadership work in clearly defined areas of responsibility Administration Outcome measure: Research activities are effectively managed and administered Indicator: Project staff, consultants and collaborators report effective and regular communication with project management Indicator: Level and quality of engagement with PIs and host country partners (based on interview data) Indicator: Quality and availability of narrative and trip reporting mechanisms Indicator: Scope and quality of project database and overall information management strategy Indicator: Sub-award financial reports timely and complete Outcome measure: Advisory Committee exercises appropriate and effective scientific oversight Indicator: Evidence that issues identified in board meeting documentation are pursued by ME 101 Indicator: Advisory board members level of engagement (based on questionnaire to advisory board members followed up by selected in-depth interviews to explore issues raised in questionnaire responses) Outcome measure: Program is adequately and appropriately staffed Indicator: Staff positions and effort levels are appropriate to meeting project goals and objectives. Outcome measure: USAID missions are engaged with local projects Indicator: Evidence of communication between USAID missions and project ME, PIs, co￾PIs, and host country collaborators Indicator: Role of USAID mission in project development (proposal review, identification of research priorities, assistance with host country linkages) Financial management Outcome measure: Program resources are allocated in a balanced manner Indicator: Budget data showing sufficient level of investment in research, capacity building and dissemination Outcome measure: Project budget is well managed and monitored Indicator: Processes in place for disbursing funds and monitoring expenditures Indicator: Cost matching levels (as evidenced in budget documents) Indicator: Budget reports reflect appropriate project expenditures Monitoring and evaluation Outcome measure: Monitoring and evaluation of the program and funded projects is effective and timely. Indicator: A coherent and well-documented M&E plan is in place and functioning effectively (based on interviews with project staff and project documents). Indicator: Proportion of projects submitting narrative reports, indicators and trip reports in timely manner Indicator: Baseline data and targets are available Indicator: Indicator data is comprehensive and linked to Feed the Future strategy Indicator: Quality of key M&E data (as compared to existing, well established protocols including USAID’s data quality criteria). Research Program Research depth, breadth and rigor Outcome measure: Investment portfolio is balanced Indicator: Level and proportion of project investments in domestic and international expenditures and in strategic and applied research Indicator: Sub-awards made to projects addressing four research objectives of project Indicator: Number of and dissemination of research publications Indicator: Evidence of support for and inclusion of cross-cutting themes in each LTRP (gender, nutrition, climate change) Collaboration, outreach and technology dissemination Outcome measure: Collaboration and outreach enhanced in project focus areas Indicator: Evidence of linkages with USAID, international and national livestock initiatives 102 Indicator: Evidence of improved academic curricula and technical training programs (# of training opportunities and # of participants where available) Indicator: Existence of electronic/web-based system that will store information and make it widely accessible Indicator: Evidence of establishment and sustainability of RICs and their influence (as described in stakeholder interviews) Indicator: Number of, quality and dissemination of research publications Indicator: Number and quality of technologies created Indicator: Number of households reached with technology dissemination efforts Human and institutional capacity building Outcome measure: Livestock-Climate Change Innovation Lab HICD fosters effective and sustainable HICD in host countries Indicator: Evidence of process in place to identify host country institutional partners Indicator: Evidence of longer-term connections with host country researchers Indicator: Level of investment in training and HICD dimensions of project Indicator: Evidence of training policy makers on more beneficial policies to assist resource poor livestock keepers in adapting to climate change Indicator: Number of farmers trained in climate change adaptation tools/approaches and/or technology adoption Indicator: Number of individuals who have received short-term agricultural sector or food security training (focus on climate change adaptation tools/approaches and/or technology adoption) Indicator: Number of individuals who have received long-term agricultural sector productivity or food security training (focus on climate change adaptation tools/approaches and/or technology adoption) Indicator: Number of farmers and others who have applied new technologies or management practices as a result of USG assistance (focus on climate change adaptation tools/approaches and/or technology adoption) Gender inclusion Outcome measure: Significant and increasing number of women scientists involved in projects and training opportunities Indicator: Equitable gender composition of host country scientific partners with significant representation of female scientists at all levels Indicator: Evidence of effective gender recruitment strategy in place Outcome measure: Gender analysis is factored into project design and implementation (e.g., use of the Gender Parity Index or WEAI or other indicators to ensure appropriate gender inclusion in project design and research uptake; e.g., gender analysis is incorporated beyond counting women as in integrating questions of property ownership and household decision making in analytical frameworks for projects) Indicator: Processes and resources in place to support gender analysis Indicator: Data on gender composition of various categories of stakeholders Indicator: LTRPs demonstrate integration of gender analysis in research 103 7. ADDITIONAL PERTINENT INFORMATION EET expects that some indicators might be revised or added in the course of field site visits. Evaluation Work Plan Activities Dates of Activity18 TASK 1. Develop evaluation design and implementation plan Nov 2013 Activity 1: Devise analytical framework for impact assessment Nov 2013 Activity 2: Create and perform stakeholder interviews Nov 2013 – Jan 2014 TASK 2: Data Collection Activity 1: Interviews with stakeholders Nov 2013 – Jan 2014 Activity 2: Interviews with project PIs Nov 2013 – Jan 2014 Activity 3: Interviews at the community-level Nov 2013 – Jan 2014 TASK 3: Data Analysis Activity 1: Synthesis of core issues and impacts from stakeholder interviews Dec 2013 - Jan 2014 TASK 4: Report Writing Activity 1: Write up of stakeholder meetings/ME/PI interviews Nov 2013 – Jan 2014 Activity 2: Write up of findings from field visits/community-level data analysis Dec 2013 – Jan 2014 Activity 3: Final Report Delivered Feb 28, 2014 D. DATA COLLECTION AND MANAGEMENT PLAN 1. Interviewer/Enumerator Training (if any) None at this time. 18 As indicated previously, the due dates were pushed back due to the U.S. Government shutdown. 104 2. Data Management and Security Data will be collected via semi-structured interviews and will be stored both in hard copy and electronic formats. Interviews with stakeholders will be recorded where possible and subsequently transcribed. Documents will be stored in a Dropbox folder for additional security. The qualitative interviews at the community level will be entered into a database and coded for analysis as necessary. All consultants will share the data electronically. Spreadsheets and databases, where required, will be password protected. 3. Data Collection Approvals None needed at this time. E. DATA COLLECTION INSTRUMENTS Interview guides developed during team visit to Nepal in relation to outcome measures and indicators noted above. Questionnaires for semi-structured interviews are attached for the following groups: • Projects PIs • Project co-PIs and host country collaborators • Project community level stakeholders • Livestock-Climate Change Innovation Lab Advisory Board members 105 Appendix E: Questionnaires The Feed the Future Adapting Livestock Systems to Climate Change (ALSCC) Innovation Lab External Evaluation Team (EET) Advisory Board Interview Guide Name: Date: Tenure on board (dates of appointment): How many meetings have you attended (dates if available): Role on Board/Background Expertise: Evaluation Questions 1. What is the primary role of the Advisory Board with respect to the work of the ME and the ALSCC program as a whole? 2. What are the key strengths and challenges faced by the program ME in program technical leadership and management? 3. Have these challenges been discussed in Advisory Board Meetings with the ME? 4. How did the ME respond? Do you think these issues have been resolved satisfactorily? 5. What other issues/concerns have arisen in the advisory board meetings? 6. Do you think the ALCCS is on track to meeting their outputs? 7. Has the board been made aware of the activities of the TIRI Scholars, graduate fellows, or the proposed Regional Innovation Centers? 8. In your opinion, how might the management structure and program direction be strengthened or improved? 9. Do you feel the board offers adequate oversight? Or are other structures required? 10. Do you feel that the program should continue after April 2014? Why or why not? 106 Feed the Future Innovation Lab on Adapting Livestock Systems to Climate Change External Evaluation 2013 Questionnaire: Project PIs (US university-based) Key Aims: • To evaluate progress towards outputs/tasks and milestones. • To identify project-level communication and monitoring protocols and any issues with implementation • To detail any changes in project design/activities • To detail M& E activities and identify potential impacts • To document the approach to community engagement and the research-led activities to date. • To explore if the sampling strategy chosen by the project reaches diverse socio￾economic/cultural elements of participating communities (thereby assessing the spread of project benefits across social groups). • To identify how gender awareness/sensitivity has informed project design and related activities. • To identify the effectiveness of wider linkages between partners and policy makers. Project PIs Interview Guide Name: Date: Organization: Element of expertise/Overall role in project: Responsibilities for specific project milestones and output indicators: Responsibilities for particular activities: When did the project start: Month, Year Program Management Technical leadership – Assess the ME’s technical leadership of the program, including how it has built on past investments while having a vision for new opportunities and constraints; engaged partners in the U.S. and overseas, including USAID Missions, CGIAR centers and NGOs; balanced research, technology dissemination, training and capacity building demands; and promoted scientific collaboration and exchange among all its partners. 1) How often and in what ways do you interact with the ME? How might this interaction be improved? 2) Does the ME promote scientific collaboration and exchange with other partners (e.g., other project PIs, collaborators, graduate fellows, early career researchers)? How and how often does this occur? Do you see sustainable connections resulting from these activities? 107 3) Has your project made linkages with related projects or donors (e.g., USAID mission, CGIAR centers, NGOs, local or national government)? Administration –Assess the ME’s administration and management of the Livestock-Climate Change Innovation Lab taking into consideration what systems are in place to ensure research activities are on track in accordance with program goals; roles and functions of advisory committees; and appropriate staffing levels, functions and level of effort. 1) What interactions have you had with the ME consultants for gender, climate change, nutrition, and statistics? 2) Which consultants? How often do you interact with the team? Financial management - Assess how well the ME has managed the financial aspects of the Livestock-Climate Change Innovation Lab taking into consideration project resource allocations; checks and balances regarding grantee disbursements, expenditures, and reimbursement; and if cost matching requirements are being met. 1) How have the financial dimensions of your project worked? Specifically, have funding disbursements or reimbursements been effective and timely? What are your financial reporting requirements? Monitoring and evaluation – Assess the effectiveness of monitoring and evaluation efforts of the Livestock-Climate Change Innovation Lab to include whether there are systems in place to capture research impacts and how effective they are, whether baselines and targets have been established and met, the appropriateness of indicators, and quality of data. 1) Have you established baselines and targets for your project? How? 2) What M&E protocols have you devised and who is responsible for M&E? 3) What impact criteria does the project ascribe to and who assesses impact? 4) What issues/challenges have you faced with M&E? 5) Are measures taken to ensure that both women and men access and benefit from project outcomes? 6) How do you assess and ensure data quality? Research Program Research depth, breadth and rigor - Assess how well the Livestock-Climate Change Innovation Lab research activities have contributed to stated program goals and objectives including the sub-award process and the balance between domestic and international, strategic and applied research. 1) Is the project on track to meet the milestones? If not, what changes have occurred? 2) The ALSCC Innovation Lab program addresses 4 ‘research goals’ – infrastructural strengthening; environmental and social change; constraints of climate change; and policy strengthening. Which of these does your project address? 108 3) The ALSCC Innovation Lab addresses three cross-cutting themes: gender, nutrition and climate change. How does your project integrate each of these themes? 4) How might the implementation components of the project be improved (e.g., scientific rigor, communications/outreach, technical/financial management, effectiveness/impact, value for money)? 5) Have field activities been changed or altered? Why? By whom? 6) How was the sample frame derived? At the national, local and household level? Collaboration, outreach and technology dissemination – Assess the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in these areas to include whether they have partnered with appropriate collaborators, whether outreach strategies have been integrated into project design, how research outputs are disseminated, and the extent to which progress has been made in technology dissemination and scalable technologies. 1) What is your management strategy for the collaborators? 2) Do you issue work plans from project staff (in the field)? 3) What is your communication strategy for field staff? 4) Have you engaged local or national policy makers in project activities/outputs? Who? 5) What engagement strategy did you use? How is it implemented? 6) How do you disseminate research outputs? 7) Has your project resulted in the dissemination of technologies? Can you provide examples? Human and institutional capacity building – Assess the effectiveness of the Livestock￾Climate Change Innovation Lab’s human and institutional capacity building in terms of level of effort, investment, and selection of candidates and institutions. 1) Describe the capacity building aspects of your project. 2) How do you select participants in your capacity building activities – e.g., host country collaborators, graduate fellows, student researchers, community participants in training? 3) What is the proportion of your funds and effort devoted to capacity building (as distinct from research)? Gender inclusion – Access the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in gender inclusion including how gender is incorporated into project design, training and output activities. 1) What recruitment strategies have you utilized to ensure the participation of women across all aspects of the project from research to fieldwork? 2) Does the project ascribe to a gender-inclusivity framework? 3) Has gender been included in project design – when conducting stakeholder research, are women included and is attention paid to gender dimensions of 109 stakeholder input (e.g., who speaks for household, how are work roles gendered within household and community)? 4) Are research questions posed in gender-inclusive manner (e.g., do questions concerning the work of farmers encompass men’s and women’s work? Is household structure and decision-making taken into account?) 5) Are female researchers administering surveys and interviews to women in the community? Feed the Future Innovation Lab on Adapting Livestock Systems to Climate Change External Evaluation 2013 Questionnaire: Project PIs and Co-PIs Key Aims: • To explore partner perceptions regarding the overall aim of the project, understanding of outputs/tasks, milestones and indicators of achievement. • To assess perceptions of the effectiveness of the Management Entity (with respect to oversight of the ME’s scientific, technical, financial, administrative and general support role) • To identify project-level communication and monitoring protocols and any issues with implementation • To detail how any changes to community-level perceptions/demands/issues are identified and how changes are responded to over time. • To discuss partner perceptions of key challenges with community engagement and the research-led activities to date. • To gain an understanding of the level of community engagement per partner. • To explore if the sampling strategy chosen by the project reaches diverse socio￾economic/cultural elements of participating communities (thereby assessing the spread of project benefits across social groups). • To assess gender inclusivity of partners and related activities. • To identify the effectiveness of wider linkages between partners and policy makers. Project Co-PI/Collaborator Interview Guide Name: Date: Organization: Element of expertise/Overall role in project: Responsibilities for specific project milestones and delivery of output indicators: Responsibilities for particular activities: When did the project start: Month, Year 110 Program Management Technical leadership – Assess the ME’s technical leadership of the program, including how it has built on past investments while having a vision for new opportunities and constraints; engaged partners in the U.S. and overseas, including USAID Missions, CGIAR centers and NGOs; balanced research, technology dissemination, training and capacity building demands; and promoted scientific collaboration and exchange among all its partners. 4) How often and in what ways do you interact with the project PI? How might this interaction be improved? 5) How often and in what ways do you interact with the Colorado State ME? 6) Do you have opportunities for scientific collaboration and exchange with other project partners (e.g., other project PIs, collaborators, graduate fellows, early career researchers)? How and how often does this occur? Do you see sustainable connections resulting from these activities? 7) Has your project made linkages with related projects or donors (e.g., USAID mission, CGIAR centers, NGOs, local or national government)? Administration –Assess the ME’s administration and management of the Livestock-Climate Change Innovation Lab taking into consideration what systems are in place to ensure research activities are on track in accordance with program goals; roles and functions of advisory committees; and appropriate staffing levels, functions and level of effort. 3) What interactions have you had with the ME consultants for gender, climate change, nutrition, and statistics? 4) Which consultants? How often do you interact with the team? Financial management - Assess how well the ME has managed the financial aspects of the Livestock-Climate Change Innovation Lab taking into consideration project resource allocations; checks and balances regarding grantee disbursements, expenditures, and reimbursement; and if cost matching requirements are being met. 2) How have the financial dimensions of your project worked? Specifically, have funding disbursements or reimbursements been effective and timely? What are your financial reporting requirements? Monitoring and evaluation – Assess the effectiveness of monitoring and evaluation efforts of the Livestock-Climate Change Innovation Lab to include whether there are systems in place to capture research impacts and how effective they are, whether baselines and targets have been established and met, the appropriateness of indicators, and quality of data. 7) Have you established baselines and targets for your project? How? 8) What M&E protocols have you devised and who is responsible for M&E? 9) What impact criteria does the project ascribe to and who assesses impact? 10) What issues/challenges have you faced with M&E? 11) Are measures taken to ensure that both women and men access and benefit from project outcomes? 12) How do you assess and ensure data quality? 111 Research Program Research depth, breadth and rigor - Assess how well the Livestock-Climate Change Innovation Lab research activities have contributed to stated program goals and objectives including the sub-award process and the balance between domestic and international, strategic and applied research. 7) Is the project on track to meet the milestones? If not, what changes have occurred? 8) The ALSCC Innovation Lab program addresses 4 ‘research goals’ – infrastructural strengthening; environmental and social change; constraints of climate change; and policy strengthening. Which of these does your project address? 9) The ALSCC Innovation Lab addresses three cross-cutting themes: gender, nutrition and climate change. How does your project integrate each of these themes? 10) How might the implementation components of the project be improved (e.g., scientific rigor, communications/outreach, technical/financial management, effectiveness/impact, value for money)? 11) Have field activities been changed or altered? Why? By whom? 12) How was the sample frame derived? At the national, local and household level? Collaboration, outreach and technology dissemination – Assess the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in these areas to include whether they have partnered with appropriate collaborators, whether outreach strategies have been integrated into project design, how research outputs are disseminated, and the extent to which progress has been made in technology dissemination and scalable technologies. 8) What is your management strategy for project staff and collaborators? 9) What is your communication strategy for field staff? 10) Have you engaged local or national policy makers in project activities/outputs? Who? 11) What engagement strategy did you use? How is it implemented? 12) How do you disseminate research outputs? 13) Has your project resulted in the dissemination of technologies? Can you provide examples? Human and institutional capacity building – Assess the effectiveness of the Livestock￾Climate Change Innovation Lab’s human and institutional capacity building in terms of level of effort, investment, and selection of candidates and institutions. 4) Describe the capacity building aspects of your project. 5) How do you select participants in your capacity building activities – e.g., student researchers, government staff, community participants in training? 6) What is the proportion of your funds and effort devoted to capacity building (as distinct from research)? 112 Gender inclusion – Access the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in gender inclusion including how gender is incorporated into project design, training and output activities. 6) What recruitment strategies have you utilized to ensure the participation of women across all aspects of the project from research to fieldwork? 7) Does the project ascribe to a gender-inclusivity framework? 8) Has gender been included in project design – when conducting stakeholder research, are women included and is attention paid to gender dimensions of stakeholder input (e.g., who speaks for household, how are work roles gendered within household and community)? 9) Are research questions posed in gender-inclusive manner (e.g., do questions concerning the work of farmers encompass men’s and women’s work? Is household structure and decision-making taken into account?) 10) Are female researchers administering surveys and interviews to women in the community? Feed the Future Innovation Lab on Adapting Livestock Systems to Climate Change External Evaluation 2013 Questionnaire: Project PIs and Co-PIs Key Aims: • To explore partner perceptions regarding the overall aim of the project, understanding of outputs/tasks, milestones and indicators of achievement. • To assess perceptions of the effectiveness of the Management Entity (with respect to oversight of the ME’s scientific, technical, financial, administrative and general support role) • To identify project-level communication and monitoring protocols and any issues with implementation • To detail how any changes to community-level perceptions/demands/issues are identified and how changes are responded to over time. • To discuss partner perceptions of key challenges with community engagement and the research-led activities to date. • To gain an understanding of the level of community engagement per partner. • To explore if the sampling strategy chosen by the project reaches diverse socio￾economic/cultural elements of participating communities (thereby assessing the spread of project benefits across social groups). • To assess gender inclusivity of partners and related activities. • To identify the effectiveness of wider linkages between partners and policy makers. Project Co-PI/Collaborator Interview Guide Name: Date: 113 Organization: Element of expertise/Overall role in project: Responsibilities for specific project milestones and delivery of output indicators: Responsibilities for particular activities: When did the project start: Month, Year Program Management Technical leadership – Assess the ME’s technical leadership of the program, including how it has built on past investments while having a vision for new opportunities and constraints; engaged partners in the U.S. and overseas, including USAID Missions, CGIAR centers and NGOs; balanced research, technology dissemination, training and capacity building demands; and promoted scientific collaboration and exchange among all its partners. 8) How often and in what ways do you interact with the project PI? How might this interaction be improved? 9) How often and in what ways do you interact with the Colorado State ME? 10) Do you have opportunities for scientific collaboration and exchange with other project partners (e.g., other project PIs, collaborators, graduate fellows, early career researchers)? How and how often does this occur? Do you see sustainable connections resulting from these activities? 11) Has your project made linkages with related projects or donors (e.g., USAID mission, CGIAR centers, NGOs, local or national government)? Administration –Assess the ME’s administration and management of the Livestock-Climate Change Innovation Lab taking into consideration what systems are in place to ensure research activities are on track in accordance with program goals; roles and functions of advisory committees; and appropriate staffing levels, functions and level of effort. 5) What interactions have you had with the ME consultants for gender, climate change, nutrition, and statistics? 6) Which consultants? How often do you interact with the team? Financial management - Assess how well the ME has managed the financial aspects of the Livestock-Climate Change Innovation Lab taking into consideration project resource allocations; checks and balances regarding grantee disbursements, expenditures, and reimbursement; and if cost matching requirements are being met. 3) How have the financial dimensions of your project worked? Specifically, have funding disbursements or reimbursements been effective and timely? What are your financial reporting requirements? Monitoring and evaluation – Assess the effectiveness of monitoring and evaluation efforts of the Livestock-Climate Change Innovation Lab to include whether there are systems in place to capture research impacts and how effective they are, whether baselines and targets have been established and met, the appropriateness of indicators, and quality of data. 114 13) Have you established baselines and targets for your project? How? 14) What M&E protocols have you devised and who is responsible for M&E? 15) What impact criteria does the project ascribe to and who assesses impact? 16) What issues/challenges have you faced with M&E? 17) Are measures taken to ensure that both women and men access and benefit from project outcomes? 18) How do you assess and ensure data quality? Research Program Research depth, breadth and rigor - Assess how well the Livestock-Climate Change Innovation Lab research activities have contributed to stated program goals and objectives including the sub-award process and the balance between domestic and international, strategic and applied research. 13) Is the project on track to meet the milestones? If not, what changes have occurred? 14) The ALSCC Innovation Lab program addresses 4 ‘research goals’ – infrastructural strengthening; environmental and social change; constraints of climate change; and policy strengthening. Which of these does your project address? 15) The ALSCC Innovation Lab addresses three cross-cutting themes: gender, nutrition and climate change. How does your project integrate each of these themes? 16) How might the implementation components of the project be improved (e.g., scientific rigor, communications/outreach, technical/financial management, effectiveness/impact, value for money)? 17) Have field activities been changed or altered? Why? By whom? 18) How was the sample frame derived? At the national, local and household level? Collaboration, outreach and technology dissemination – Assess the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in these areas to include whether they have partnered with appropriate collaborators, whether outreach strategies have been integrated into project design, how research outputs are disseminated, and the extent to which progress has been made in technology dissemination and scalable technologies. 14) What is your management strategy for project staff and collaborators? 15) What is your communication strategy for field staff? 16) Have you engaged local or national policy makers in project activities/outputs? Who? 17) What engagement strategy did you use? How is it implemented? 18) How do you disseminate research outputs? 19) Has your project resulted in the dissemination of technologies? Can you provide examples? 115 Human and institutional capacity building – Assess the effectiveness of the Livestock￾Climate Change Innovation Lab’s human and institutional capacity building in terms of level of effort, investment, and selection of candidates and institutions. 7) Describe the capacity building aspects of your project. 8) How do you select participants in your capacity building activities – e.g., student researchers, government staff, community participants in training? 9) What is the proportion of your funds and effort devoted to capacity building (as distinct from research)? Gender inclusion – Access the level of effort and effectiveness of the Livestock-Climate Change Innovation Lab in gender inclusion including how gender is incorporated into project design, training and output activities. 11) What recruitment strategies have you utilized to ensure the participation of women across all aspects of the project from research to fieldwork? 12) Does the project ascribe to a gender-inclusivity framework? 13) Has gender been included in project design – when conducting stakeholder research, are women included and is attention paid to gender dimensions of stakeholder input (e.g., who speaks for household, how are work roles gendered within household and community)? 14) Are research questions posed in gender-inclusive manner (e.g., do questions concerning the work of farmers encompass men’s and women’s work? Is household structure and decision-making taken into account?) 15) Are female researchers administering surveys and interviews to women in the community? The Feed the Future Adapting Livestock Systems to Climate Change (ALSCC) Innovation Lab External Evaluation Team (EET) USAID Mission Interview Guide Name: Date of Interview: Title of position in Mission: Responsibilities: Evaluation Questions 1. Are you familiar with the activities of the two Feed the Future ALSCC projects in (country)? What linkages have you formed with them? If none, why ? 2. Are Associate Awards being considered for the present projects? Could you elaborate why or why not? 3. Do you have any concerns about the current ALSCC project? 116 4. How important/relevant are projects like this to the work of the mission? 5. What mechanism is employed to ensure coordination and potential collaboration between USAID Washington initiatives (such as the ALSCC program) and the initiatives of the Mission? 117 Appendix F: EET Travel Itinerary and Site Visits External Evaluation Team (EET) visit to Colorado State University (Management Entity) AGENDA: USAID Feed the Future Innovation Lab External Evaluation Visit19 Shana Gillette: 970-581-4853; Dick Bowen: (970) 231-2555; Diana Fahrenbruck, Admin Assistant: 970-491-1648 Thursday, November 14 7:45 Meet Shana and Dick in Hilton Lobby for walk over to the Dean’s Office 8:00-9:00 Meeting with Key Administrators, Dean’s Office Dean Mark Stetter, College of Veterinary Medicine and Biomedical Sciences Associate Dean Sue VandeWoude, CVMBS Dept. Head, Chris Orton—Clinical Sciences, CVMBS Dept. Head, Colin Clay—Biomedical Sciences, CVMBS Associate Dean Jeff Steiner, College of Agricultural Sciences Dept. Head, Greg Perry—Agricultural and Resource Economics, CAS Introduction to External Evaluation (15 minutes) • External evaluation process and measurements Key Administrator Discussion Topics (45 minutes) • Dept/College resources that provide an unique setting for the LCC Lab • Dept/College research areas that complement the LCC Lab • Dept/College perceptions of the Innovation Lab’s overall visibility, institutionalization, and impact on campus 9:00-10:00 Campus Tour 10:00-11:00 Office of Sponsored Programs 11:00-noon VP for International Affairs Noon-1:00pm Lunch: Catered, Regional Innovation Center, Foothills Campus, RIC D234 1:00-2:30pm Management Entity Overview, Diagnostic Medicine Conference Room • Scientific and Technical Assessment 19 This is agenda provided by ME – actual agenda varied slightly from this plan. 118 o Organizational Structure o Leadership o Innovation and Contribution to Feed the Future o Key Technical Messages • Scientific Portfolio and Contribution to Development o “Basic” Science o Applied and Adaptive Research o Policy Research o Gender Analysis and Integration o Strategic Vision and Positioning o Longterm Training o Shortterm Training and Capacity Building o Global Network o Research Investment o Research Productivity and Impact 3:00-4:00pm VP of Research 4:00-5:00pm • Management Assessment o Challenges o Advisory Board o Award Processing and Administration o Web Presence and Monitoring o Communication 6:30-8:30pm Dinner: Sonny Lubick’s Steakhouse Friday, November 15 7:30-8:30 Continental Breakfast at the Hilton with Shana Gillette and Dick Bowen 8:30-9:30 Foothills Campus Tour 10:00-10:30 South Campus Tour 10:30-noon Internal Evaluation, Diagnostic Medicine Conference Room • Preliminary Report • In-country evaluators • Technical Review Noon-1pm Lunch: El Monte Grill, meeting with Kathy Galvin 1:15-3:15pm X-cutting Themes: Nutrition, Gender, & Climate Advisory Support, DMC Conference Room • Developing strategy • Implementing inclusive activities • Accountability—gender/nutrition budgeting approaches 3:15-8:30pm Poudre River Canyon—Mishawaka Restaurant for dinner? 119 External Evaluation Team (EET) visit to Nepal Dates: November 23rd-December 2nd Participants: EET members Drs Karen Brown (KB), Wyn Richards (WR), Claire Heffernan (CH) Note: USAID AOR for Livestock-Climate Change Innovation Lab Joyce Turk (JT) was present for many EET activities in Nepal. Aims: The visit had two key objectives: • To evaluate Livestock-Climate Change Innovation Lab management functions in Nepal with regard to overall coordination, communication, monitoring and evaluation of Long-term Research Projects (LTRP),TIRI Scholars, and graduate fellows. • To evaluate the quality, impact and uptake of related research and project-conjoined development activities on participating communities. Expected Outcomes: Inform and elaborate on key management and research quality issues across the ME and the South East Asia LTRP and TIRI Scholar investments. Description of In-country Evaluation: Each of the LTRPs in Nepal contains both a research component and a suite of related development interventions hitherto referred to as ‘project-conjoined development activities’. Therefore, the evaluation will examine the quality, impact and uptake of both the research and development components of the specific LTRPs involved: PIs Krakauer, Chettri, Joshi and Gillies. Time Line of EET Activities: November 24th: Arrival in Nepal (KB and CH) November 25th: Field Visit Krakauer LTRP (KB and CH) • EET Tasks (KB and CH): Meeting with Co-PIs, community-level focus groups, demonstration plots, kitchen garden plot, weather station. Demonstration of drip irrigation. • Arrival in Nepal (WR) November 26th: Attended TIRI Scholar Pre-Conference Workshop (KB, CH, WR) • Observed scholar presentations/interactions with ME, attended Climate Data Seminar. Informal discussions with several TIRI Scholars. • Interviews with USAID Mission Staff (3) and Graduate Research Fellow • Interviews with PI (R. Gillies) and Co-PIs (Helen Keller International) November 27th: In-country travel/Stakeholder Meetings • Meetings with Graduate Fellows (2) • Flight to Pokhara (KB, WR, CH, JT) 120 • Meeting with Co-PIs (Chhetri Project): LI-BIRD Director and Program Director/Co-PI Chaudary (KB, WR, CH) November 28th : Site Visit (Chhetri LTRP): KB, WR, CH, JT • Interviewed key stakeholders at the community level (Committee Chairman at Village Level and other Village Leaders); attended wider focus group meeting with pre-identified community • Viewed water tank investment by LI-BIRD/Project November 29th: Site Visit (Joshi LTRP): KB, WR, CH, JT • Interviewed Co-PI and visited training site • Interviewed Project staff • Visited six of 30 participating farm households • Viewed weather data station and student research demonstration plots November 30th: Site Visit (Krakauer LTRP)/EET Evaluation Planning (WR, CH, KB) • EET Evaluation Planning (KB, CH): travel and activity plan for remainder of field trips; finalize advisory board questionnaire; development of draft and final report analytical frameworks • EET Tasks (WR): Krakauer LRTP in field site, Syangja District: Interviews of Co-PIs; meeting with community-level focus group (ca 50 members of dairy cooperative). Brief interview with DDG of National Extension Service. Visits to: project demonstration plots; kitchen garden plot; weather station; demonstration of drip irrigation; on-farm visit to witness forage crop initiatives/benefits; milk collection/processing plant. • Interview ME climate change consultant Tom Hobson (KB, CH, WR) December 1: In-country travel/EET meeting • Flight to Kathmandu (KB, WR, CH) • EET debrief meeting December 2: Departure from Nepal (KB, CH, WR) Critical Evaluation Functions/Performance Indicators: Over the course of the EET trip to Nepal, three interview protocols were created for the PIs, Community Focus Groups and Co-PIs and Host Country Collaborators. The semi-structured interviews were piloted in the field and amended accordingly. To ensure comparability, the interview protocol was designed to collect data on four core factors across the three stakeholder groups: a. The overall level of interaction with the Livestock-Climate Change Innovation Lab ME including the number of meetings and type of interactions. 121 b. The level of coordination across the Livestock-Climate Change Innovation Lab, i.e. the integration of the different stakeholder groups both with the wider program (ME, ME consultants, researchers) and within and between LTRPs. c. The level of integration of funded TIRI Scholars and graduate fellows with both the LTRP and the Livestock-Climate Change Innovation Lab ME. d. Issues of quality regarding research design, implementation and outputs; integration of research and project-conjoined development activities; and the potential impacts of the development and training activities. Via this analysis expected outcomes include: a. An assessment of the suitability and effectiveness of the Livestock-Climate Change Innovation Lab ME communication protocols. b. An assessment of the suitability and effectiveness of the ME devised M & E protocols. c. An assessment of Livestock-Climate Change Innovation Lab host country investments, future benefits and cost-effectiveness. d. An assessment of the level of integration of Livestock-Climate Change Innovation Lab activities with both the host country USAID Mission, national and local level policy makers, and related donors/projects. e. An assessment of Livestock-Climate Change Innovation Lab activities and the ability of these activities to meet Feed the Future expected indicators and outcomes. Analytical Framework A framework for the programmatic analysis was devised to co-ordinate both data collection and stakeholder analysis from the additional national-level site visits in Ethiopia, Kenya, Tanzania and Senegal. EET Visit to Kenya and Tanzania Dates: December 10th-December 16th Participants: EET member Dr Claire Heffernan (CH) Aims: The visit had following objectives: • To meet with former advisory board member Dr Jimmy Smith (Kenya). • To evaluate the East African TIRI (EATIRI) scholar investment (Kenya) • To evaluate Livestock-Climate Change Innovation Lab management functions in Kenya and Tanzania with regard to overall coordination, communication, monitoring and evaluation and impact of the EATIRIs and Long-term Research Projects (LTRP). • To evaluate the quality, impact and uptake of related research and the impact and sustainability of project-conjoined development activities at the community level. • To assess the level of coordination and collaboration between the LRTPs (and TIRI) activities and key in-country institutions and development partners. 122 Expected Outcomes: • To inform and elaborate on key program, project management and research quality issues across the ME, the US University LRTP PIs and associated Tanzanian Co-PIs and field teams. • To evaluate the selection and management of the EATIRI scholar investments • To initiate overall recommendations to inform the future of the program. Description of In-country Evaluation: Both LTRPs in Tanzania contain a research component and a suite of related development interventions. The evaluation examined the quality, impact and uptake of both the research and development components of the 2 LTRPs: Bunn (UC Davis) and Mazet (the HALI project, UC Davis). Time-line of EET activities and description of key elements: a. Arrival in Nairobi 11pm December10th. b. December 11th: Meeting with Jimmy Smith, Director of ILIR. Dr Smith was on the advisory board from project start-up until 2011. Discussed key elements of previous project management, the impact on the project of CSU internal dynamics, perceptions of progress over Year 1 and 2 of the programme and his perceptions regarding administration and management structures. c. Organised meetings with Tiri Scholars. Afternoon flight to Dar es Salaam d. December 13th: Drive to Morogoro (5.5 hours, 238 km), met with Peter Msoffe (co-PI Bunn Project); Innocent Kimweri (MSc student Bunn Project). School visit, discussion with Teacher: Mrs Aleha Philipo; Visit to local poultry farmer (Mrs Jennifer Clemmons). Key elements of project co-ordination, management, issues with survey design and implementation were discussed. Sustainability issues, curriculum development and student engagement featured in the discussion. The results of the initial pilot and relationship with both the wider HALI project and USAID mission were also explored. Evening meeting with Professor Kazwala, Sokoine University (co-PI Mazet Project). Professor Kazwala detailed the wider components of the HALI II project and the integration between the other USAID funded projects being implemented by Mazet e.g. PREDICT and the TB study. Professor Kazwala detailed the ‘One Health’ nature of HALI and recommended that specific elements of the ALSCC-funded project in relation to the training, monitoring and evaluation and communication strategies were discussed with the field team in Iringa. e. December 14th: Drive to Iringa (5.5 hours, 339 km), meet with HALI field staff Reviewed the component portions of HALI II Programme and the ALSCC Innovation Lab component work plan and individual project roles. Discussed communication strategies, training and curriculum development and logistics, any knowledge of, or involvement in, monitoring and evaluation and/or impact assessment and other 123 elements of the development and research activities as suggested by Professor Kazwala. f. December 15th: Drive to communities (3.5 hours, 179km), met with base-line data collection team: discussed the Animal Health questionnaire, key challenges working with pastoralist communities, implementation protocols. Meeting with livestock extension worker: discussed training and information needs, changing dynamics of animal health and production among the pastoralist production systems (including climate change and the impact of recent government policies regarding Animal Genetic Resources (AnGR); potential equipment/diagnostic needs (as discussed with the field team regarding the distribution of Animal Health kits as part of the proposed Livestock Extension training). Meeting with teacher and pupil: to discuss Friends of Ruaha (FoR) environmental curriculum development element of ALSCC Innovation Lab. Site visits by the FoR team were reviewed in addition to the the data records kept by the teachers. Flight to Dar es Salaam. Flight to Nairobi, Flight to UK. g. December 16th: Arrive UK 5:40 AM EET Visit to Ethiopia Dates: December 9th-December 14th (includes 2 travel days) Participants: EET member Dr Wyn Richards (WR) Aims: The visit had three key objectives: • To evaluate ALSCC Innovation Lab management functions in Ethiopia with regard to overall coordination, communication, monitoring and evaluation of Long-term Research Projects (LTRP) and TIRI Scholars – completed. • To evaluate the quality, impact and uptake of related research and project-conjoined development activities on participating communities – completed. • To assess the level of coordination and collaboration between the LRTPs (and TIRI) activities and key in-country institutions and development partners – completed. Expected Outcomes: Inform and elaborate on key program and project management and research quality issues across the ME, the US University LTRP PIs and associated Ethiopian Co-PIs and the progress of TIRI scholar investments and begin to compose recommendations to inform USAID’s decision as to the future of the program. Completed collection of information and began to formulate report. Description of In-country Evaluation: Both LTRPs in Ethiopia contain a research component and a suite of related development interventions. The evaluation examined the quality, impact and uptake of both the research and development components of the 2 LTRP PIs : 124 Coppock (Utah State University) – KALO project; and Little (Emory University) - CHAINS project. Time Line of EET Activities: December 10th am: Arrival in Ethiopia December 10th pm: EET Task 1- Interviewed senior member of the Board, Dr Iain Wright, Director ILRI, Addis Ababa using standard Advisory Board Questionnaire devised in Nepal. EET Task 2- Finalised meeting agendas and appointments with target individuals and institutions involved with the LRTPs and TIRIs. December 11th am: EET Task 3 – Meeting with and interview of senior in-country team members of the KALO project held at the Oromia Agricultural Research Institute, Addis Ababa. Employed standard Co-PI Questionnaire. December 11th pm: EET Task 4- Meeting with and interview of senior in-country team members of the CHAINS project held at Addis Ababa University. Used co-|PI Questionnaire. December 11th pm: EET Task 5- Attempted to arrange interviews with Mohamed Abdinoor and other livestock representatives in the USAID Mission. Despite several brief telephone exchanges with MA to arrange convenient time for interview, a meeting never materialized December 12th am: EET Task 6. Presentations by and discussions with 6 Ethiopian TIRI scholars held at the Hotel Siyonat, Addis Ababa – in presence of their principal mentors. Devised and used questionnaires for TIRI scholars to tease out successes, concerns, issues. December 12th pm: EET Task 7. Meetings with Ministry of Agriculture Directors of Pastoral Development and Animal Production and Feeds to inform them of the project activities and discuss potential future alliances. December 12th pm : EET Task 8. Skype call with Waktole Uma, principal in-country field researcher of CHAINS project on visit to the PI at Emory University. December 12th eve: Write up of travel debrief. December 13th am : Departure for UK. December 23rd pm– Skype meeting with MA from the UK. using draft questionnaire for USAID missions. EET Visit to Senegal Dates: January 5-10, 2014 Participants: EET members Dr. Karen Brown (KB) and Dr. Wyn Richards (WR) Aims: The visit had the following objectives: • To evaluate the West African TIRI Scholar investment 125 • To evaluate ALSCC Innovation Lab management functions in Senegal with regard to overall coordination, communication, monitoring and evaluation and impact of the TIRI Scholars initiative and the Long-term Research Projects (LTRP). • To evaluate the quality, impact and uptake of related research and the impact and sustainability of project-conjoined development activities at the community level. • To assess the level of coordination and collaboration between the LTRPs (and TIRI) activities and key in-country institutions and development partners. Expected Outcomes: • To inform and elaborate on key program, project management and research quality issues across the ME, the US University LRTP PIs and associated Senegalese Co-PIs and field teams. • To evaluate the selection and training of the TIRI Scholars. • To initiate overall recommendations to inform the future of the program. Description of In-country Evaluation: Both LTRPs in Senegal contain research components and a suite of related development interventions. The evaluation examined the quality, impact and uptake of both the research and development components of the 2 LTRPs: McPeak (Syracuse) and Hanan (South Dakota State University). Time-line of EET activities and description of key elements: Note: all EET activities conducted by both KB and WR unless otherwise noted January 5: Arrive Dakar (KB and WR); initial meeting with LTRP PI John McPeak (KB) January 6: Interview LTRP PI John McPeak; travel from Dakar to Thies (1.5 hours); EET meeting. January 7: Travel from Thies to Linguere (4 hours) to interview AVSF collaborators with Niall Hanan’s LTRP and visit AVSF (Agronomes et Vétérinaires Sans Frontières) pastoralist information center; interview Niall Hanan LTRP in-country project coordinator Peter Shapland; return from Linguere to Thies (4 hours). January 8: Observe TIRI Scholars workshop held in Thies; read proposals of TIRI Scholars and listen to research presentations; observe part one of AWhere training on weather data tool available to TIRI Scholars through CSU. January 9: Observe part two of AWhere training on weather data tool at TIRI Scholars workshop; travel from Thies to Dakar (1.5 hours); visit USAID Dakar Mission to interview relevant staff (key contact: Ronit Gerard, Agriculture Officer). January 10: Meet with and interview local research collaborators on John McPeak’s LTRP at Institut Senegalais de Recherches Agricoles (ISRA) National Laboratory and Bureau of Social and Economic Research; tour nutrition lab; view demonstration of NIRS (Near Infrared Scanner) used in project and meet with staff responsible for scanning and lab analysis of vegetation samples; meet with director of National Lab; meet with director and key research staff of Bureau of Social and Economic Research January 10 (evening): Depart Senegal. 126 Appendix G: List of Persons Contacted Management Entity, Colorado State University Dr. Richard Bowen, Director, ALSCC Innovation Lab, and Professor, Department of Biomedical Sciences, College of Veterinary Medicine and Biomedical Sciences (CVMBS) Dr. Shana Gillete, Co-Director, ALSCC Innovation Lab; and Assistant Professor of Risk Communication, Department of Clinical Sciences, CVMBS Dean Mark Stetter, CVMBS Associate Dean Sue VandeWoude, CVMBS Dr. Chris Orton, Department Head, Clinical Sciences, CVMBS Dr. Colin Clay, Department Head, Biomedical Sciences, CVMBS Associate Dean Jeff Steiner, College of Agricultural Sciences (CAS) Dr. Greg Perry, Department Head, Agricultural and Resource Economics, CAS Doug Leavall, Director, Sponsored Programs, Office of the Vice President for Research Karlie Braley, Accounting and Fiscal Management, ALSCC Innovation Lab Dr. James Cooney, Vice Provost for International Affairs, Office of the Provost and Executive Vice President Beverly Parsons, InSites (internal evaluator on contract with ME) Dr. Sarah McKune, Director of Public Health Programs in the College of Public Health and Health Professions, and Clinical Assistant Professor in Epidemiology and Health Liaison, University of Florida (ALSCC Innovation Lab ME gender consultant with Sandra Russo) Dr. Sangeeta Rao, Animal Population Health Institute (biostatistics consultant to ALSCC Innovation Lab ME) Dr. Elizabeth Ryan, Assistant Professor, Toxicology, Environmental and Radiologic Health Sciences. CVMBS (nutrition consultant for ME) Dr. Dana Hoag, ALSCC Innovation Lab East Africa Coordinator, and Professor, Department of Agricultural and Resource Economics, CAS Dr. Kathleen Galvin, Department of Anthropology; Senior Research Scientist at the Natural Resource Ecology Laboratory. Dan Broman, graduate student (University of Colorado, Boulder) working with Dr. Tom Hopson (climate change consultant to ALSCC Innovation Lab ME) USAID Joyce Turk, ALSCC Innovation Lab AOR Long Term Research Project (LTRP) PIs Dr. John McPeak, Associate Professor and Vice Chair, Public Administration and International Affairs, Syracuse University Dr. David Bunn, Associate Director of Africa Programs, African Studies, UC Davis 127 Dr. Layne Coppock, Associate Professor, Environment and Society, Utah State University Dr. Joanna Mazet, Professor, Veterinary Medicine, UC Davis Dr. Gillies, Utah State (Nepal) – conducted brief interview, do full questionnaire Dr. Peter Little, Professor, Development Studies, Emory University Dr. Nanda Joshi . MSU (Nepal) – sent questionnaire by email Dr. Nir Krakauer. Assistant Professor, Earth System Science, City College of New York Dr. Netra Chhetri, Assistant Professor, Science Policy, Arizona State University Dr. Niall Hanan, Senior Research Scientist & Professor, Geographic Information Science, South Dakota State University Advisory Board Members Dr. Iain Wright, Program Leader, Animal Science for Sustainable Productivity and Director General's Representative in Ethiopia, ILRI Dr. Jimmy Smith, Director General, ILRI Dr. Adegbola Adesogan, Professor of Ruminant Nutrition, Department of Animal sciences, University of Florida Dr. John Johnston, Scientific Liaison, U.S. Department of Agriculture, Food Safety and Inspection Service, Office of Public Health Science Dr. Montague W. Demment, Vice President for International Programs, Association of Public and Land-grant Universities (APLU); Professor, Emeritus, Department of Plant Sciences, University of California, Davis Nepal USAID, Kathmandu: Bronwyn Llewellyn, Environment Officer, USAID Netra Sharma, NRM & GCC Programs Specialist, USAID Collaborators and Co-Investigators: Dale Davis, Country Director, Nepal, Helen Keller International Mahesh Sherestha, Monitoring and Evaluation Manager, Helen Keller International Nirmala Pandey, Agricultural and Food Security Coordinator, Nepal, Helen Keller International Dr. Ajay Jha, Assistant Professor, Agriculture and Resource Economics, Colorado State University Dr. Tarendra Lakhankar, CUNY Research Scientist, Civil Engineering, City College of New York Jeeban Panthi, Development and Research Officer, Environmental Science, Small Earth Nepal Program Co-Investigator Dr. Chet Raj Upreti, Nepal Agricultural Research Council (NARC) (Joshi project) 128 Co-Investigator Dr. Pashupati Chaudary, Program Director, Animal Science, Local Initiatives for Biodiversity, Research, and Development (Chettri project) Dr. Balaram Thapa, Executive Director, LI-BIRD, Pokhara Dr. Thomas Hopson, Research Scientist, National Center for Atmospheric Research, Boulder, CO (ALSCC Innovation Lab ME climate consultant) TIRI Scholars: Prazila Shresha, Nepal Agric Univ – Zoonotic importation of Spp Salmonella infections in backyard poultry Meera Prajapati, NARC- Screeing for Nipah virus infection in Nepal Prakesh Karn, Heifer International Nepal, Discovering resilient strategies for goat production systems Naryan Paudyal, NARC, Bovine Campylobacteriosis and Trichomonasiosis in Western Nepal Suraj Subedi, Agric and Forestry Univ, AFU, Addis Ababa Improved pasture management practices within community forests Renu Shakya, Environmental Camps for Conservation Awareness (ECCA-Nepal) Livestock health in Humla and relationship to climate change Muna Sharma AFU Impact of Climate Change on livelihood of transhumance herding in the High Hills of Nepal Tapendra Bohara, Animal Health Training and Consultancy (AHTCS) Tick ecology and incidence of tick borne diseases in exotic, crossbred and native breeds of cattle in Midwest Nepal Shatrughan Shah, AFU, Response of tomato on performance of heat- stressed Broilers in Chitwan Prakesh Karn, Heifer International, Resilient strategies for goat production system against climatic severity Sulochana Shresha, AFU Climate change on emerging and re-emerging goat diseases across altitude gradient in Nepal. Graduate Fellow: Chandra Dhakal, Nepal Development Research Institute Met with multiple village household members, farmers and community groups, local extension agents, to discuss their experience of climate change affecting livestock and agricultural production and their local livestock and agricultural practices Kenya TIRI Scholar: Peter Lamuka, Lecturer, University of Nairobi 129 Tanzania Bunn Project: Co-Investigator Peter Msoffe Innocent Kimweri, MSc student funded on Bunn Project Mrs Aleha Philipo, Teacher at project school Mrs Jennifer Clemmons, Poultry Farmer Mazet Project: Co-Investigator Professor Kazwala, Sokoine University Research Team: Good Luck, Zikankuba Sijali, Zena Baby, Mwokozi Mwakzalila Base-line data collection team: Asha Makweta, Elizabeth Komba, Alphonse Msigwa Valentine Mvungi, Livestock Extension Worker Agustino Ngailo, Teacher, Friends of Ruaha (FoR) Emanuel Mswani, Student, Friends of Ruaha (FoR) Ethiopia CHAINS project: Co-Investigator Dr. Workneh Negatu, Director Centre for Rural Development, Addis Ababa University Co-Investigator Dr. Dejene Negasse Debsu, Post Doc Research Associate, College of Development Studies, Addis Ababa University Dr. Waktole Tiki, Post Doc Research Associate Dr Ali Hassen, Chair, Centre for Rural Development (CRD), Addis Ababa University Tessema Bekele, University of Addis Ababa Ms. Getenesh Aliyou, MSc student funded by project, University of Addis Ababa KALO project: Dr. Aliye Hussein, Director, Oromia Agricultural Research Institute, Addis Ababa, Co-PI Dr. Solomon Desta, Co-Director Managing Risk for Improved Livelihoods (MARIL), Addis Ababa, Co-PI Dr. Getachew Gebru, Co-Director MARIL, Co-PI TIRI Scholars: Habtamu Tassew Tarakegn, Mekelle University, College of Veterinary Medicine Aklilu Nigussie Megos, Ethiopian Institute of Agricultural Research, Werer Agricultural Center Beyene Teklu Mellisse, Hawassa University, Wondo Genet College of Forestry and Natural Resources 130 Melaku Berhe Redda, Mekelle University, College of Dryland Agriculture and Natural Resources Yibeltal Tebikew Wassie, Hawassa University Debre Tsegahun, Debre Birhan Research Centre, Ethiopia Ministry of Agriculture: Gifawosen Tessema, Pastoral Areas Development Directorate, Livestock Resources Development Sector, Ministry of Agriculture, Addis Ababa Mr Tadder Sori, Livestock Feeds Directorate, Ministry of Agriculture, Addis Ababa USAID: Mohamed Abdinoor, Livestock/Pastoralist specialist Senegal Government: Djiby Dia, Director, Institut Senegalais de Recherches Agricoles (ISRA) National Laboratory and Bureau of Social and Economic Research of (BAME) (Bureau d’analysis macro￾economiques), Dakar Mame Nahe Diouf, ISRA-BAME, Dakar Mumar Talla Seck, ISRA-LNERV, Dakar Civil Society: Paul Biagi, Agriculture et Veterinaires sans Frontiere (AVSF), Coordinator of Pastoralist Information Center, Ligueres Modou Niang, AVSF, Livestock Technician Peter Shapland, Research for Development Specialist with Niall Hanan’s project. USAID, Dakar: Ronit Kirshner Gerard, Agriculture Officer, Economic Growth Office Dr. PapaNouhine Dieye, Agriculture Specialist 131 Appendix H: Documents Reviewed Program Management Documents: • USAID Adapting Livestock Systems to Climate Change (ALSCC) Collaborative Research Support Program (CRSP) Request for Applications (RFA) Number M-OAA-EGAT-AG￾09-1155 • Livestock-Climate Change Innovation Lab Leader Award and related correspondence • Policy and Operating Procedures including personnel and marking plans • Livestock-Climate Change Innovation Lab Advisory Board Documentation • Communications plan and products • Feed the Future Indicator Data for Livestock-Climate Change Innovation Lab projects • Monitoring and Evaluation plan • Internal Evaluation of Livestock-Climate Change Innovation Lab • Work Plans and Performance Narratives • Project Reports • Project Publications (e.g., brochures, newsletters, progress reports, research briefs and fact sheets) • Program Website • Reports on PI meetings and other meetings • Terms of Reference (Scope of Work) for associated management staff where available Scholars and Fellows • TIRI Scholars and Graduate Research Fellows: RFP, Proposals, Contracts, Work Plans and Reports where available Long Term Research Projects (LTRPs) • LTRP Proposals and Budgets • LTRP Performance Narratives • Additional documentation from LTRP collaborators (e.g., annual reports, training materials, data collection records, websites) Seed Grant Research Projects • Seed Grant Proposals • Seed Grant reports, products, indicators and trip reports Budget Information • Budget data requested from ME 132 Appendix I: ME Comments on Report Date: April 8, 2014 To: Carole Levin From: Richard Bowen Subject: Response to the Report of the External Evaluation of the Innovation Lab for Adapting Livestock Systems to Climate Change The first thing to state is how much I appreciate the efforts of Drs. Brown, Heffernan and Richards in evaluating our program. Overall, they expended an amazing amount of effort, in the face of several obstacles (e.g. government shutdown, travel difficulties), in garnering information and synthesizing a critique and recommendations. There is no denying that our program suffered significant startup difficulties related to the original personalities involved, but I firmly believe we are past that stage and have learned many lessons over the past three years on how to effectively manage such a program. The evaluation document has, in my opinion, a number of inaccuracies, some of which I address below. Many of these are due to our deficient job of documenting and explaining our activities, which made it difficult for the evaluators to gather requisite information. Nevertheless, this evaluation provides numerous excellent recommendations for improving our program, and I can assure you that I will seek to address them all over the coming years and, if granted, a renewal plan. Page 16: Clearly, there were serious personnel issues during the first year of this project that essentially required a reboot, but it is an overstatement that these difficulties “[prevented] most work during the first year”. During the first year, we conducted competitive reviews and funded 12 seed grants and 14 fellowships. Page 16 and later: I agree that we are stretched for management time, particularly with the resignation of Dr. Gillette. Our commitment is to increase Dr. Bowen’s effort to 50% and to hire a full time Deputy Director, both by May 1, 2014. Two impressive candidates for the Deputy Director position are being interviewed. Page 16: The statement that the ME does not interact with other CSU entities involved in development efforts in Africa was likely true under the original Director, but has changed substantially. Additionally, senior leadership at CSU has shown strong support for these efforts. Members of our team (Hoag and Bowen) have worked with others to develop MOUs and exploit relationships with Hawassa University in Ethiopia and the University of Nairobi in Kenya. We interact substantially with Drs. Robin Reed and Kathy Galvin in their collaborations with the Sustainable Drylands Institute at University of Nairobi. Dr. Hoag spent many days working on a plan for teaching and extension at Hawassa and Dr. Bowen has been working the multiple entities here and in Nairobi to solidify these relationships. Finally, we have recruited multiple scientists from the 133 Natural Resource Ecology Lab at CSU to implement the recent call from USAID’s Africa Bureau to develop two scholarly works on carbon mitigation and grasslands in Africa. Certainly, there is room to further enhance these intra-­‐CSU collaborations for the benefit of all international programs, but considerable interactions are already in place. Page 18: We indeed did have one associate award in Mali, which was curtailed due to the coup. We are enthusiastic about establishing additional such awards, but our understanding is that we are not able to propose these, so we are somewhat at the mercy of the missions in this regard. Page 18: Coordinating inter-project collaborations: Indeed, we have set aside roughly $500,000 for supplemental grants that involve substantive collaborations between existing projects that we fund in West Africa, East Africa and Nepal. Page 21: Gender and climate experts: Addressing climate and gender issues has been in requirement in all proposals that we have funded, but I agree that the effectiveness of this has varied considerably. For that reason, we have retained experts in both of those areas who have not only been interacting with PIs but reviewing all of the submissions to our recent RFPs. I believe these interactions have substantively improved the work we have ongoing with respect to climate and gender issues. Page 24: Communications with PIs: There is always a delicate balance between micromanaging and not providing enough guidance to assure program coherence. I believe that we have erred too much of the side of being hands off and are attempting to rectify that deficiency. Page 25: Database for program activities -- The database we commissioned just prior to the evaluation was not adequate for our needs, nor amenable to use by the evaluators. It became obvious that a large expenditure would be required to deal with these deficiencies and we therefore abandoned that effort. In its place we have enhanced the database Dr. Bowen wrote when he became Director to allow access by PIs and USAID. Unfortunately this was in place too late for use by the evaluators, but it is virtually finished and should eliminate this justified criticism. Page 26: Interactions with Advisory Board: Admittedly, we have not exploited our board considering the quality of their expertise. The recommendations of the evaluators are valuable in this sense. Page 28: It is stated the “The staff member [Dr. Hoag] overseeing projects in East Africa appears not to have a scope of work or terms of reference”. The work in East Africa, particularly with the scholars but also the LTRP, PIs is exceptionally robust. Page 29: Engagement with Missions – It is true that we have had uneven engagement with Missions in the different regions, but I believe it is not accurate to infer that this has been for lack of effort on our part, as our success largely reflects the willingness of the Missions to engage with us. Mission officers are exceptionally busy and I fully understand the effort they must expend to interact with us. We have established an excellent relationship with the Nepal Mission. The Ethiopian mission has been less 134 responsive, but has sent a representative to each of our scholar/PI workshops. Last November, Dr. Bowen had an excellent meeting with Issac Thendiu from the Kenya Mission which made clear that the Mission wants us to interact with NGOs in Kenya. Finally, we have repeatedly encouraged our PIs to meet with Mission personnel when in country. Clearly, we will continue to try and establish and enhance these interactions. Page 33: Disbursement of funds – The implication is that that we have had difficulty in speedy transfer of funds to subawardees and TIRI scholars. There certainly have been delays, at times substantial, but in essentially all cases is because of errors or illegitimacies in investigator budgets or, particularly in the case of the TIRI scholars, massive difficulties in getting money to some foreign institutions. I honestly feel that our administrators have gone above and beyond to expedite these transfers within the limit of federal and state regulations. Another factor that is currently impeding funds transfers is the requirement for USAID contracting officers to approve all subawards. Throughout: It is painfully clear to me that we have done a suboptimal job of documenting and explaining our activities, which is, in part, responsible for a number of misunderstandings presented in the evaluation. We are working hard to reorganize and extend our web site as a source of information and, as mentioned above, have a new database on the cusp of deployment, both of which should greatly alleviate this deficiency. I applaud the insights of the evaluation team and believe that the most important response I can make is to describe my own thoughts about how best to proceed over the next year, hopefully for Phase 2 of this project. I plan to utilize the recommendations of the evaluators as an explicit road map to improving our program. Indeed, I consider every single recommendation that was made to be valid and worthy of our attention; a number of the recommendations are self-­‐ evident and are already being implemented. Three fundamental changes to our program are paramount to these efforts: Restructuring the Management Entity: There is no doubt that, as Director, I have not been able to commit adequate time to this program due to other duties. That will change within the next month, and my intention is to request 50% support from this program, which means considerably more than a half-­‐ time effort. I will be able to do this because one of my major time commitments over the past 10 years has been Director of the Animal Models Core for an NIH funding Regional Center of Excellence for Biodefense and Emerging Infectious Disease; that program ends in May, which will free up substantial amount of my time. A large fraction of my career has been devoted to livestock, including considerable efforts in the international arena, and I believe I have the background and expertise to continue as Director of the ALSCC Innovation Lab; I am also firmly committed to this role and to the ALSCC Innovation Lab at CSU. My plan to provide the required additional personnel support for our program includes the following changes: • Replacement of Shana Gillette with a full time Deputy Director: I have identified two individuals at CSU that have extensive experience in international agriculture. 135 I have interviewed one and will interview the second next week. This position will be filled, at 100% commitment and a start date of May 1, 2014. • Administrative Officer: Diana Fahrenbruck has been serving this role for approximately one year and is doing an excellent job. She has been working virtually full time, but paid at 75% effort. My intention is to increase her compensation to 100% effort, in large part to maintain our new database, which will greatly facilitate monitoring and evaluation. • Communications Manager: We have what I consider an outstanding communications leader who manages our web site, not only for esthetics, but content development. Nicole was hired at 75% effort and the communications team was filled out with several student interns. All but one of these interns will be graduating this spring and my intention is not replace them and to increase this position to a full time effort. It is obvious from the evaluator’s report that one of our weaknesses is in communications and public engagement. The Manager will be instrumental in alleviating this deficit. • Financial Officer: Our current financial officer has done a great job but is also significantly overcommitted with other departmental duties and has asked to be replaced. She has received 40% salary support, but has expended 60% of her effort to the ALSCC Innovation Lab. I will replace her with a very experienced accountant in our department at 70% salary support. The increased level of support for this position reflects increased demands for financial reporting and approvals from USAID. I believe that the restructuring of personnel efforts outlined above will address many of the concerns voiced by the evaluators. Developing a work plan for year 5 of Phase 1 and preparation for possible extension to Phase 2. In all candor, the approach of the ALSCC Innovation Lab to supporting research for development in Adapting Livestock Systems to Climate Change Innovation Lab was initiated prior to my appointment as Director. We have expanded our efforts with gap filling projects and the scholars program (both of which I consider quite successful), but I am anxious to reorganize this program. I believe that we are supporting a truly outstanding group of investigators, but I agree totally with the evaluators that our program does suffer from a lack of coherence and synergy among projects. I am currently preparing a work plan for the coming year and, in addition to enhancing our existing program, I intend to devote considerable effort toward establishing a strategy to make our efforts much more than simply of sum of the parts. My objective is to garner opinions and advice from USAID personnel (DC and Missions), our current PIs, NGOs working in the countries we engage, producer groups, and a variety of other knowledgeable individuals and groups (for example, just this week, I spoke to representatives of Peace Corps about how we could work together to our mutual benefit). The insights I gain from these discussions will then be used to synthesize a list of critical research objectives for coming years. My vision is to then devise a strategy to address each of these objectives by enlisting a multidisciplinary coalition of investigators that work closely together to “attack” the problem and devise solutions for implementation. The problems we seek to address demand a diversity of expertise and as the managers of this program, our primary job will be to assure coordination among 136 individual projects working on the same developmental challenges. Plainly, these efforts must incorporate the objectives of Feed the Future and be overlaid with substantive attention to improvements in human nutrition and gender equity. Improving engagement with our investigators and USAID missions. A mistake I made in directing the ALSCC Innovation Lab was to take too much of a hands off approach in managing our long-­‐term research projects, probably because I generally believe that micromanagement can be counterproductive. Our evaluators conclude that a middle ground approach should be taken, and upon reflection, I believe they are correct. My intent over the next year is to promote substantially more engagement and discussion with our PIs, particularly with respect to item 2 above. This will be facilitated by the funds we are about to provide for inter-­‐project collaborations, briefly mentioned in the evaluation report. I believe that the program has strongly mentored and supported our early career researchers (scholars), but more frequent face-­‐ to-­‐ face engagement with those individuals and enhancing their interactions with our U.S. PIs are also important goals for this last year. We have had some success in aligning our efforts with USAID Missions, but it is often easier said than done to engage those personnel and related organizations (e.g. CGIAR centers and the new Innovation Labs). Nonetheless, this is a critical part of our efforts and we must do a better job of being proactive in enlisting such collaborations. Livestock production is an essential component of global efforts to enhance food security and improve the livelihoods of subsistence farmers throughout the developing world. I believe that Drs. Brown, Heffernan and Richards have helped us identify deficiencies in our program and have assisted us greatly in articulating areas for improvement. I intend to follow up on their efforts by vigorously implementing their recommendations. 137