PERFORMANCE EVALUATION OF THE INNOVATION FOR AGRICULTURAL TRAINING AND EDUCATION (INNOVATE) PROJECT November 09, 2018 PERFORMANCE EVALUATION OF THE INNOVATION FOR AGRICULTURAL TRAINING AND EDUCATION (INNOVATE) PROJECT Prepared for the United States Agency for International Development under USAID Contract Number AID-OAA-1-15-00024/AID-OAA-TO-1600008 Prepared by: Carl Erik Schou Larsen, Team Leader Christopher Collins, Education Development Specialist Florence Nyangara, PEEL Senior Evaluation Specialist Sandra Scham, PEEL Senior Social Science and Gender Advisor Johanna Beduhn, PEEL Research Analyst Contractor: ME&A, Inc. (PEEL) 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 www.MEandAHQ.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government ABSTRACT The Innovation for Agricultural Training and Education (InnovATE) project was established by the United States Agency for International Development (USAID) Bureau for Food Security (BFS) to support agricultural education and training (AET) in three ways: strengthen the AET knowledge base and make it more accessible (LEARN); provide technical assistance for AET assessments and Human and Institutional Capacity Development (HICD) programming, and develop tools for project design (DESIGN); and enable development of AET capacity, and develop training modules and instructional materials (TRAIN). Virginia Polytechnic Institute and State University implemented this project as the Management Entity and was joined by three sub-awardees. This final evaluation focused on questions related to institutional sustainability, capacity development, institutional and stakeholder use of InnovATE materials, and project response to AET practitioners and USAID Mission needs. The evaluation used document reviews and remote key informant interviews to collect data. The Evaluation Team (ET) found that InnovATE achieved planned objectives in developing and disseminating valuable educational and training materials and contributed to AET’s knowledge base in several cross-cutting fields, including gender, vocational training, youth entrepreneurship, quality assurance, and accountability. Several other objectives were not achieved, however, partly due to lack of Mission responsiveness, and limited in-country outreach activities. The ET recommends that similar projects in the future develop viable plans to work with Missions to create awareness, get buy-in and support from the outset, and create closer linkages between other Feed the Future projects, Missions, and institutions in host￾country. CONTENTS EXECUTIVE SUMMARY i Evaluation Purpose i Project Background i Evaluation Questions, Methods, and Limitations i Main Findings, Conclusions and Recommendations iii 1.0 EVALUATION PURPOSE AND QUESTIONS 1 1.1 Evaluation Purpose 1 1.2 Evaluation Audience 1 1.3 Evaluation Questions 1 2.0 PROJECT BACKGROUND 2 2.1 Context 2 2.2 InnoVATE’s Theory of Change 3 2.3 Project Design 3 3.0 EVALUATION METHODS AND LIMITATIONS 4 3.1 Evaluation Methods for Data Collection and Analysis 4 3.2 Data and Methodological Limitations 7 4.0 FINDINGS AND CONCLUSIONS 9 4.1 EQ 1: To What Extent Has innovate Achieved Its Objective of Defining and/or Disseminating Good Practice Strategies, Approaches, and Investments for Establishing Efficient, Effective, and Financially Sustainable AET Institutions And Systems? 9 4.2 EQ 2: In What Ways Have USAID Missions, Donors, Policy Makers, and AET Professionals Used Information Generated From innovate’s Good Practice Papers, Analyses, and Thematic Studies, As Well As The Training Modules and Training Courses (From Both the Train and Learn Components)? What Opportunities Are There for These Papers, Analyses, and Studies to Be Used By the Different Stakeholder Groups to Enhance AET? 12 4.3 EQ3: How Effectively Has Innovate Communicated Its Objectives and Potential Value to Missions? 15 4.4 EQ 4: How Effectively Did Innovate Address Mission Demands Relating to AET Capacity Building? 19 4.5 EQ5: What Were Some of the Implementation Challenges and How Did the Project Address Them? 21 ANNEXES 23 Annex 1: Expression of Interest 24 Annex 2: Phase I (2017) Evaluation Protocol 31 Annex 3: Addendum to Phase I Protocol for Phase Ii (2018) 63 Annex 4: Phase II (2018) Interview Guides 67 Annex 5: Data and Data Analysis Tables 70 ACRONYMS Acronym Description AA Associate Award AET Agricultural Education and Training Agrilinks Achieving Agriculture-led Food Security Through Knowledge Sharing ARP Agricultural Research and Policy ATC Agricultural Training Center BFS Bureau for Food Security CDCS Country Development Cooperation Strategy COP Chief of Party D.C. District of Columbia DLEC Developing Local Extension Capacity EOI Expression of Interest EQ Evaluation Question ET Evaluation Team EU European Union FTF Feed the Future GFSS Global Food Security Strategy HICD Human and Institutional Capacity Development HQ Headquarters ICD Institutional Capacity Development InnovATE Innovation for Agricultural Training and Education Project IP Implementing Partner IRB Institutional Review Board KI Key Informant KII Key Informant Interview LWA Leader with Associates M&E Monitoring and Evaluation ME Management Entity MEAS Modernizing Extension and Advisory Services MSC Most Significant Change NAIP National Agricultural Investment Plan NGO Non-Governmental Organization OIRED Office of International Research, Education, and Development PEEL Program Evaluation for Effectiveness and Learning Penn State Pennsylvania State University PE Performance Evaluation PI Partner Institution PoC Point of Contact PPP Public Private Partnership RFA Request for Applications Acronym Description ToC Theory of Change TU Tuskegee University U.S. United States UF University of Florida USAID United States Agency for International Development USG United States Government Virginia Tech Virginia Polytechnic Institute and State University i EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of this midpoint performance evaluation (PE) of the Innovation for Agricultural Training and Education (InnovATE) project was to assess: 1) the overall rationale and strategy for the project and what has or has not worked well in its implementation; and 2) how these factors contributed to the project’s progress toward achieving targeted outcomes. The evaluation provides recommendations that the United States Agency for International Development (USAID) Bureau for Food Security (BFS) can use to inform future Agricultural Education and Training (AET) programming, improve project effectiveness, better achieve intended outcomes, and contribute to its collaboration, learning, and adapting agenda. In addition, it should be noted that the Feed the Future Global Performance Evaluation Report1 states that Feed the Future's monitoring and evaluation should focus more on capacity development. Therefore, capacity building is a significant emphasis for this evaluation. PROJECT BACKGROUND InnovATE was designed to help develop the necessary human and institutional capacity for a strong and growing agricultural sector in Feed the Future focus countries. The project's Theory of Change (ToC) incorporates and builds upon a three-part model—LEARN, DESIGN, and TRAIN—conceived by USAID/BFS. Although InnovATE activities were influenced by specific needs and constraints within each country, the project’s overall approach is an effort to improve food security through the following activities: • Conducting country assessments to identify supply and demand skills gaps and advise USAID on target areas for future AET capacity development investments (equity and access considerations for youth and gender are central to the approach). • Developing workshops, materials, and training to build capacity in agricultural education institutions and create linkages among stakeholders in AET networks. • Providing recommendations on good practices and creating tools for practitioners to share knowledge about AET systems. EVALUATION QUESTIONS, METHODS, AND LIMITATIONS Evaluation Questions The Expression of Interest (EOI) from Feed the Future (Annex I) identified the following core evaluation questions (EQs) and sub-questions (see below) to be addressed by the evaluation team (ET): • EQ1: To what extent has InnovATE achieved its objective of defining and/or disseminating good practice strategies, approaches, and investments for establishing efficient, effective and financially sustainable AET institutions and systems? • EQ 2: How effective has InnovATE been in encouraging stakeholders, including AET institutions, to utilize good AET practices information? • EQ 3: How effectively has InnovATE communicated its objectives and potential value to Missions, and what did they find most valuable? • EQ 4: How effectively did InnovATE address Mission demands relating to AET capacity building? 1 Feed the Future Global Performance Evaluation Report, https://pdf.usaid.gov/pdf_docs/PBAAF131.pdf, p. 82. ii • EQ 5: What were some of the implementation challenges and how did the project address them? Evaluation Methods To address the above EQs, the ET utilized a mixed methods approach, including: • Document review. The ET reviewed InnovATE reports for performance indicators and compared the achievements against its set targets. Other document review provided information on case study and other information that helped inform the interview guides and recommendations. • An online survey sent to 1,000 subscribers via a large listserv and Twitter (only 3.7 percent of those contacts, however, responded to the survey and most of those responses were incomplete.) • Forty-seven (47) key informant interviews (KIIs) drawn from two cohorts. The first was composed of 27 respondents—mostly representatives of the lead and partner institutions and USAID/BFS staff based in the United States (U.S.)—who were interviewed in 2017. The second was composed of 20 respondents, mostly from Missions, who were interviewed in 2018. Of the second group, 18 were based in countries purposively selected by the ET in collaboration with USAID/BFS. • Three case studies developed through document review and other relevant publications, to demonstrate InnovATE’s experiences in three countries (Armenia, Honduras, and Senegal). • Triangulation of findings from 2017 and 2018 interviews, document review, monitoring data, and limited survey data. Evaluation Limitations In interpreting the findings of this evaluation, the following limitations should be taken into consideration: • Data collection for the evaluation of InnovATE was conducted in two different periods (May 2017 and May 2018) by two different ETs. The datasets from 2017 were re-analyzed by the 2018 ET, which can cause data quality issues in terms of measurement errors. • Three 2017 interviewees (two from Virginia Tech and one USAID Mission staff) were interviewed again in 2018. • There were no field visits as part of the approved evaluation design. InnovATE is implemented in several Feed the Future countries supporting USAID Missions. Neither ET visited any of these countries, which affected the amount and quality of data collected, as well as their interpretation. Instead, remote interviews via telephone and Skype calls were conducted, which proved to be logistically difficult to arrange, especially with Mission staff, and not the most effective method to collect sufficient and detailed data to address the “how” and “why” questions. This limited the amount and quality of data (in terms of depth) that are used in this evaluation. • Low response rate to the online survey conducted in 2017 did not yield statistically valid sample size (only 37 responses out of 1,000 requests sent, for a response rate of 3.7 percent). Only three questions (on the effectiveness of the project) received a sufficient number of responses (14) to be included in the analysis and report. The number of responses for the other questions was insufficient and, therefore, they were not taken into consideration. • Response bias may have been introduced because the key informants interviewed in 2017 were primarily from InnovATE’s partner institutions and only a couple were from other stakeholder groups, including USAID Missions, which provided information based upon that group’s perspective. To mitigate this problem, the 2018 interviews were conducted remotely with a iii more diverse set of stakeholders, with an emphasis on Mission staff, Chiefs of Party (COPs) (to a more limited extent), non-governmental organizations (NGOs), and AET practitioners. • The identification of interviewees either in Missions or implementing partners (IPs) across different countries was dependent on the introductions from InnovATE’s leaders at USAID/BFS or Virginia Tech, which may have introduced selection bias. MAIN FINDINGS, CONCLUSIONS AND RECOMMENDATIONS EQ1: To what extent has InnovATE achieved its objective of defining and/or disseminating good practice strategies, approaches, and investments for establishing efficient, effective and financially sustainable agricultural education and training institutions and systems? In what ways has the project generated, documented and disseminated information on key constraints to AET systemic development, innovations to address those constraints, and engaged stakeholders and collaborators to contribute? In what ways did/did not the information generated address the specific needs of the missions requesting and provide usable, actionable recommendations? How effective were project outputs in achieving outcomes in terms of mission AET program investment, design, and operations? FINDINGS • Representatives from the Armenia (n=2) and the Honduras (n=5) Missions praised both the products’ usefulness and the project implementer's communication with their Missions. These representatives went on to say that Missions used project products to 1) improve existing programs for training in nutrition-sensitive agriculture; 2) design programs to support accountability and gender inclusion in agricultural training; and, 3) support new education-based interventions to improve citizen security in rural areas. • Other Mission respondents (n=8) thought project training modules and materials were useful for continuing education programs that enabled AET professionals to encourage girls to consider a range of careers in agriculture. These respondents also noted that products helped students select courses and curricula commensurate with their skill levels, support new curriculum development, and meet training needs for agricultural value chains, • USAID/BFS (n=5) interviewees, U.S. partner institution key informants (KIs) (n=10), and survey respondents were positive about the project’s training achievements. • Some Mission respondents (n=8) found value in pilot studies that focused on gender, AET in conflict areas, vocational training, and youth in agricultural education; while other Mission respondents did not indicate using InnovATE products for any of their AET strengthening programs. • Only 11 interviewees (out of 47 in both cohorts), found InnovATE products to be useful or effective for establishing efficient, effective, and financially sustainable AET institutions and systems. • Eight Mission respondents who worked directly with InnovATE found products helpful for designing programs to address key constraints in their host countries and, in particular, to assure that AET institutions worked on alleviating problems of engaging women in agricultural training. • Interviewees in both cohorts (n=34) and 14 of the 2017 survey respondents found that a major element missing from InnovATE products was a discussion of broad-based interdisciplinary long￾term solutions and activities designed to build human capital. A preponderance of interviewees from both cohorts (n=32) thought that Missions did not engage enough in the project to generate usable, actionable recommendations or outcomes in terms of Mission AET program investment, design, and operations. iv • InnovATE’s monitoring data demonstrate that the project exceeded its targets with respect to completing studies (including a thematic study on rural youth violence in El Salvador, Guatemala, and Honduras) and technical notes, and developing tools, workshops, and AET assessment tools. • InnovATE signed associate awards (AAs) in Afghanistan, Armenia, Senegal, and prospectively in Honduras. • The project exceeded targets for how USAID Missions, donors, policy-makers, and AET professionals used the information generated from InnovATE’s products and training materials. • InnovATE was not able to meet its targets for some Feed the Future indicators, including public￾private partnerships (PPPs) formed through Feed the Future’s assistance and institutions/organizations undertaking capacity/competency strengthening as a result of U.S. Government (USG) assistance. • Thirteen representatives from five Missions were interviewed. Of these, only one responded positively to the question of whether or not the information generated by the project addressed the country’s (Honduras) key constraints to AET systemic development. CONCLUSIONS • In the last year of the project some positive effects of InnovATE activities were observed in several USAID Missions in terms of the development of AAs, the completion of well-received studies, and the development and dissemination online of a significant number of products that have contributed to the AET knowledge pool. • As of 2018, InnovATE has generated for three Missions useful and actionable research and recommendations on AET systemic development, and on engaging stakeholders and collaborators; however, even those Mission representatives were doubtful about the project's long-term contribution to sustainable AET institutions. • InnovATE’s products have not been very successful in addressing key in-country AET constraints in terms of Mission AET program investment, design, and operations. While the project did generate innovative products, especially in the form of training modules and thematic studies, it does not appear from the project reports reviewed and interviews with Mission representatives that the project addressed the key constraints to overcoming the most significant obstacles to AET institution development, i.e., limited financial resources and a dearth of well qualified professional staff. RECOMMENDATIONS • To mitigate the difficulties in achieving Mission buy-ins, which stem from lack of time and resources on the part of Missions (and, with respect to some Missions, such as Nepal, a lack of interest), BFS and/or Regional Bureaus could sponsor AA's for Missions with the greatest need for improvement in AET training. Need could be determined by scoping assessments • Since a number of interviewees found that InnovATE's online resources were useful, dissemination of good practices materials for AET institutions could be expanded and improved. For example, a specific web portal for AET institutions could be set up for use in selected countries; 2) webinars and interactive video conferences with AET institutions could be conducted on specific issues identified as concerns; or 3) a smart-phone-based network of AET professionals could be established; as could regional online communities of AET institutions. EQ 2: In what ways have USAID Missions, donors, policy makers, and AET professionals used information generated from InnovATE’s good practice papers, analyses, and thematic studies, as well as the training modules and training courses (from both the TRAIN and v LEARN components)? What opportunities are there for these papers, analyses and studies to be used by the different stakeholder groups to enhance AET?FINDINGS • In Honduras, all representatives rated the use of the products in training as very good overall and one informant from Senegal thought that the tools InnovATE developed have the potential to support better in-country AET training. • A few well-received studies in Honduras and the sexual harassment policy (gender empowerment) symposium in Senegal with university administrators around Africa were products of InnovATE that a few Missions (n=2) report using. Only a limited number of Missions (n=2) report using InnovATE products in program design. • Specific studies, conducted in collaboration with Missions, are rated by interviewees familiar with them (n=5) as having impact on the design of programs to disseminate good AET practices. • Only nine of 18 in-country respondents felt that they could talk about InnovATE’s effectiveness in encouraging stakeholders to utilize its good practice papers, thematic papers, training modules, and other products. Of the nine, three said InnovATE was highly effective, three said InnovATE was somewhat effective, and three considered InnovATE ineffective in achieving this objective. • Mission respondents in Honduras and Nicaragua were particularly positive about the extent to which the studies, papers, and training modules InnovATE produced enabled them to plan AET support programs like reducing barriers to women’s participation and disadvantaged youth participation in agricultural education. • According to 15 of 18 in-country respondents, lack of awareness of the project made it difficult to brand and disseminate InnovATE's products among stakeholders effectively. • Five interviewees suggested that translating InnovATE materials into local languages would increase their use. • Agrilinks users (n=4) were not always aware that information on the site was attributable to InnovATE. • Some Mission staff (n=7) interviewed stated that they would use InnovATE's online resources more if they receive notification when InnovATE posts new content on Agrilinks. • Ten (10) interviewees (out of 14) thought that the project was generally not successful in encouraging stakeholders to utilize good AET practices. USAID interviewees (n=4) agreed that the materials were not utilized. Two of the 4 interviewees also questioned the quality of materials produced. • A few knowledgeable stakeholders (n=3) in Missions (technical advisors) believed that InnovATE’s training materials were at a more basic level, therefore useful only to non￾professional audiences. One key informant suggested that InnovATE's online resources are also suitable for an audience with a more basic level of knowledge on AET topics and that, while this may be useful to some users, Mission staff would benefit from resources targeting an audience of technical experts. Another key informant emphasized that resources disseminated online are less likely to be used if their applicability to a country or region is not made clear. • Representatives of partners and the lead institution (n=11) agreed that the project was not successful in reaching the private sector, donors, policymakers, and AET professionals to any great extent. CONCLUSIONS • InnovATE's greatest success in contributing toward sustainable institutions has been in Honduras, Senegal, and Armenia, thus far, where AAs were established or are being considered. • Informants from these three countries in three distinct regions described InnovATE's research as “very useful” and sometimes even "game-changing" for Missions in designing new programs. vi • InnovATE’s activities have contributed to the AET knowledge pool and produced some good papers and studies but have not succeeded in making their products known broadly in Feed the Future countries in general. • Translating InnovATE materials into local languages would increase their use. • Training materials would be more useful if they are tailored to the knowledge, experience and needs of the audience. RECOMMENDATIONS • Project implementers should develop ways to make more direct contacts with AET institutions to promote good AET practices. This could be done through networks partner institutions already established or by conducting workshops on a regular basis. • Implementers should prioritize collaboration with the private sector to leverage the positive impact of their projects. Interviewees had several suggestions for doing this, including 1) conducting campaigns and workshops highlighting the skills of graduates from InnovATE￾supported institutions and demonstrating benefits of their work to businesses and business associations; 2) investing in developing more internship and/or apprenticeship programs for AET students and graduates; 3) supporting public/private partnerships at the secondary education level that support equity by targeting low income and marginalized students and women; and, 4) helping expand the range of AET courses at local institutions to include other types of relevant training (i.e., involving business schools or social science courses related to leadership and policy) to assure that AET graduates develop skills important to the private sector. • In-country stakeholders who were aware of InnovATE studies suggested that thematic studies focusing on immediate issues confronting agricultural education in the country should be required in future projects of this kind. Further, they suggested the ME make a concerted effort to ensure recruitment of experts from institutional partners or outside consultants who have experience working with these issues. • Many interviewees agreed that applicable studies must be translated into local languages to be useful to a wider range of AET practitioners. EQ3: How effectively has InnovATE communicated its objectives and potential value to missions? What did missions find most valuable about InnovATE project outputs arising from scoping assessments and associate awards (as applicable)? FINDINGS • The majority of representatives of partner institutions and the lead institution (n=11) stated that USAID/BFS handled communications with Missions and that they only had direct contact with Senegal. • The majority of partner institution representatives (n=7) interviewed stated that they were not very clear about the process for obtaining buy-ins and AAs and stated that they found the protocols established for communicating with Missions bureaucratic and cumbersome. • Of the eight (out of 18) respondents across four countries who agreed to provide their assessments on how effectively InnovATE communicated its objectives and potential value to Missions, three rated the project as highly effective, three as somewhat effective, one as having low effectiveness, and one as not at all effective. • A few KIs (n=2) in the Missions were unaware of any AET activities in their country, even though activities were recorded as having taken place and, except in Missions where the project worked directly, other Mission staff was largely unaware of the project. • Representatives of other Missions (n=3) stated that they became more interested in InnovATE products as the project advanced, but this was only after the project had been ongoing for vii several years and after all of their funds had been obligated, which made them unable to follow through with InnovATE. • One USAID/Nicaragua informant stated that they were still conducting training on how to expose at-risk youth to agricultural education, and that this resulted from InnovATE’s work in their country. • Two USAID/Armenia respondents stated that InnovATE contributed to curriculum reform under their AA, which was critically needed, and that the project also added a food safety certificate program to the AET institution's course offerings. • Two key informants in Senegal very positively described InnovATE's pilot exercise on community participatory curriculum development (CPCD) and the plans for the resulting AA. • Findings from the analysis of 2017 interviews indicate that partner institution representatives tended to be more critical of the products (produced by other institutions) than others interviewed. They rated papers produced by InnovATE as having had little effect but rated the thematic studies as more successful (seven positives versus five negative comments). • USAID (BFS) interviewees considered papers and studies produced by InnovATE as similarly successful (10 positive versus eight negative statements). • In-country Feed the Future Flagship project COPs who were interviewed (n=3) were not familiar with the InnovATE project and no Mission respondents were able to provide any instances where the project attempted to connect with Feed the Future projects. • Representatives of partner institutions and the lead institution (n=8) stated their belief that the project needed to have a greater in-country presence in Feed the Future Missions. CONCLUSIONS • In general, Missions’ views on the value of the project were mixed depending upon whether they were at a point in their programming where they could take advantage of what InnovATE had to offer and follow through on it. • Both Mission and partner institution respondents perceived that InnovATE did not organize its research and design activities to make efficient use of other Feed the Future projects or of Mission feedback. • Communication between InnovATE and USAID Missions was hampered in some ways by timing and procedural issues and in other ways by differing perceptions of what the project brought to the table in terms of value. • In spite of scoping assessments and other outreach measures that took place in more than 10 Missions, InnovATE‘s attempts to communicate its objectives and potential value to Missions were largely unsuccessful, except in the case of a few Missions. • The reasons for this lack of success, as seen by interviewees, were divided. Partner institution respondents suggested that the program had a late start and did not reach Missions at the right time in their program cycle (which is refuted by the ME and BFS). On the other hand, Mission interviewees suggested that they were either confused about or completely unaware of the services InnovATE had to offer. RECOMMENDATIONS • Project implementers engaged in activities aimed at supporting Missions should take stock of where Feed the Future projects are in the implementation cycle to prioritize resources towards Missions during the project design phase. • If BFS is interested in developing similar technical services concepts such as InnovATE, it should provide support for pilot studies in selected countries while ensuring that Missions and AET institutions of that country are actively engaged in AET institution building. viii • As local Mission staff (FSNs) is the institutional memory for most USAID Missions, they should be given priority by project implementers for inclusion in scoping assessments, workshops and other activities. • BFS should work to develop champions of AET at USAID as another strategy to help alleviate the concern that AET often is not an active part of USAID programming, as pointed out by several key informants. • Project implementers should assure that scoping assessments do not require the expenditure of Mission resources and should focus on Mission priorities as well as assessing the needs of AET institutions. EQ4: How effectively did InnovATE address mission demands relating to AET capacity building? In what ways could InnovATE better align with mission strategies relating to AET and educational capacity building? How effectively did InnovATE align with Feed the Future (as well as other USAID strategies) (e.g., desire to support country AET capacity development needs and generate a body of knowledge on good practice in AET strengthening)? FINDINGS • In describing the progress of the Armenia AA, which was designed to meet AET needs for capacity building in financial sustainability, two key informants indicated that project implementers’ efforts to support the financial sustainability of the local AET institutions were not successful. They suggested that this was in part because the model used was based upon the view, (one not supported by independent research), that funds might be forthcoming from the Armenian diaspora community in the United States. • Representatives of Missions that worked with InnovATE (n=7) stated that the project implementers had responded well to their demands. • Interviewees from partner institutions (n=10) stated that they did not believe the project impacted AET capacity building because the work performed for Missions did not address the key constraints of lack of resources and qualified professionals to train and teach students. • Of seven respondents that provided feedback on how effectively InnovATE aligned with Feed the Future strategy, five rated InnovATE as somewhat effective and two as highly effective. These latter two individuals were in Missions that had worked with InnovATE on specific studies tailored to their needs. The other five respondents were not aware of specific InnovATE activities. • Some Mission informants (n=3) felt that the project schedule was not in alignment with the program cycles of Feed the Future Missions initially but believed that this changed later. CONCLUSIONS • InnovATE has, according to Mission informants that dealt with the project, met their demands for AET training materials, studies, and assessments. Most of them are very satisfied with those materials. • Respondents in Missions that worked with InnovATE believed that the project aligned well with their strategies and with USAID strategies. • Respondents who were less familiar with InnovATE activities in the Missions were not persuaded that the program could contribute to their existing programs or align with their existing agricultural education strategy. • To the extent that AET institutions were surveyed and/or interviewed (n=5), respondents felt the program’s effectiveness depended upon whether InnovATE worked with the institutions in country or simply disseminated materials. ix RECOMMENDATIONS • Future implementers of similar projects should make sure that all assessments address the major gaps and shortfalls of AET institutions, and that they also address Mission priorities and strategies for building AET capacity. • Longer-term AET Institutional Capacity Development (ICD) programs should be developed to make a sustained impact on the quality of AET education and the number of qualified professionals trained. Programs should be specifically targeted to create sustainable institutions rather than contributing to time-limited programs that require Mission buy-ins. These should be based upon capacity building models and benchmarks used for other institutions with similar challenges. • Prior to instituting demand-driven projects for building capacity of host country institutions, implementers should conduct a thorough study of the policy-enabling environment in each country and the existing in-country projects with similar goals. This would allow implementers to determine the kind of support that would focus on key constraints not being addressed. EQ5: What were some of the implementation challenges and how did the project address them? In what ways could implementation have been improved to more efficiently and effectively carry out scoping assessments? Disseminate results? Encourage project requests? FINDINGS • According to BFS interviewees and partner institution and Mission representatives, Mission demand for InnovATE’s products was not as great as was originally expected. • Findings from 2018 interviews revealed that some Mission respondents (n=3) believed that the InnovATE team’s ability to meet objectives was limited by Missions’ low awareness of InnovATE’s activities in their countries. This is despite InnovATE providing information prior to their visits in country. • InnovATE organized all its activities through its home office in Virginia, with occasional temporary duty employees/consultants to selected countries. • USAID Missions found scoping assessments of AET institutions in 10 countries to be useful; however, some Mission informants (n=5) felt that the assessment teams were not able to spend enough time in country to work with and establish linkages with local Mission staff. • As a result of not forming contacts with Missions as successfully as anticipated, some interviewees (n=7) believe that InnovATE tended to focus on producing generic papers rather than country-specific ones that could be used in the design and on-the-ground implementation of AET activities. • Another factor inhibiting more in-depth collaboration with local stakeholders, as noted by three informants, was that Mission staff are extremely busy and hence they could not realistically be expected to open their schedules for the task of helping design and implement Mission buy-ins. • In the project’s final year, research and design activities began to include more in-depth engagement of local stakeholders to address the issue of limited in-country collaboration (examples cited in four interviews included design work in Honduras and Tanzania). • Partner institution interviewees from 2017 and two interviewees from BFS stated that, from their perspective, the project's lack of impact (with respect to host country education and lack of Mission interaction) was, in part, due to the fact that the project focused on meta-level analysis rather than on country-level design and on-the-ground implementation. CONCLUSIONS • InnovATE's most prominent challenge was a lack of Mission demand for project services and engagement that remained throughout life of the project. x • In the last year of the project, the relevance and frequency of contacts with some Missions were increased. The project implementers have had moderate success in addressing identified AET priorities in those countries. • The design of the project did not provide for an on-the-ground presence. • Because of lack of connections to local AET institutions (or to institutions that Missions were familiar with) and lack of institutional-strengthening PPP's to work with, they focused on meta￾level analysis rather than on country-level design and on-the-ground implementation. RECOMMENDATIONS • Project implementers need to establish public outreach programs that focus on raising the awareness of Mission staff, other Feed the Future projects, and the AET institution community. • USAID/BFS should prioritize the development of local networks at country level. Networks could be composed of public officials, educators, private sector representatives, and other stakeholders to create needed basic platforms for cross-country collaborations. This would remove dependence on the efforts of U.S. based universities to establish lasting connections from short-term visits and remote activities. 1 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE Innovation for Agriculture Education and Training (InnovATE) is a 5-year project funded by the United States Agency for International Development (USAID) Bureau for Food Security (BFS), in October of 2012. The project was originally conceived as Modernizing Agricultural Education and Training Systems (MAETS) project and had a goal of developing the human and institutional capacity in developing countries to promote the innovation necessary to achieve sustainable food security, reduce poverty, conserve natural resources, and address other rural problems. The project’s objective was to define and disseminate good practice strategies, approaches, and investments. The project was also expected to support country programs for agricultural education and training (AET) reform and investment, and establish efficient, effective, and financially sustainable AET institutions and systems. The purpose of evaluating InnovATE was to assess1) the overall rationale and strategy for the project; 2) what worked (or did not work) in implementation; and 3) progress toward outcomes. It should be noted that the Feed the Future Global Performance Evaluation Report2 states that Feed the Future's monitoring and evaluation should focus more on capacity development. Therefore, capacity building is a significant emphasis for this evaluation. 1.2 EVALUATION AUDIENCE The audience for this evaluation includes BFS, the Office of Agricultural Research and Policy (ARP), USAID Missions, and InnovATE project implementers and staff. These entities will use the evaluation results to make decisions about extending, terminating, or re-competing activities. BFS may use the results to design future multi-country projects that focus on AET, rural workforce development, youth, and to offer support for USAID Missions in designing projects. InnovATE may use the evaluation results to reflect upon achievements and shortfalls and the reasons for both, which are elucidated by the evaluation findings. 1.3 EVALUATION QUESTIONS Except evaluation question (EQ) 5 (as listed below), the EQs are specifically focused on the results of InnovATE's activities. The EQs and sub-questions detailed in the Expression of Interest (EOI) in Annex 1 are as follows: EQ1: To what extent has InnovATE achieved its objective of defining and/or disseminating good practice strategies, approaches, and investments for establishing efficient, effective and financially sustainable AET institutions and systems? In what ways has the project generated, documented, and disseminated information on key constraints to AET systemic development, innovations to address those constraints, and engaged stakeholders and collaborators to contribute? In what ways did/did not the information generated address the specific needs of the Missions requesting and provide usable, actionable recommendations? How effective were project outputs in achieving outcomes in terms of Mission AET program investment, design, and operations? 2 Feed the Future Global Performance Evaluation Report, https://pdf.usaid.gov/pdf_docs/PBAAF131.pdf, p. 82. 2 EQ2: In what ways have USAID Missions, donors, policy makers, and AET professionals used information generated from InnovATE’s good practice papers, analyses, and thematic studies, as well as the training modules and training courses (from both the TRAIN and LEARN components)? What opportunities are there for these papers, analyses, and studies to be used by the different stakeholder groups to enhance AET? EQ3: How effectively has InnovATE communicated its objectives and potential value to Missions? What did Missions find most valuable about InnovATE project outputs arising from scoping assessments and associate awards (as applicable)? EQ4: How effectively did InnovATE address Mission demands relating to AET capacity building? In what ways could InnovATE better align with Mission strategies relating to AET and educational capacity building? How effectively did InnovATE align with Feed the Future (as well as other USAID strategies) (e.g., desire to support country AET capacity development needs and generate a body of knowledge on good practice in AET strengthening)? EQ5: What were some of the implementation challenges and how did the project address them? In what ways could implementation have been improved to more efficiently and effectively carry out scoping assessments? Disseminate results? Encourage project requests? 2.0 PROJECT BACKGROUND 2.1 CONTEXT Global food security and climate change challenges require focus on capacity building to promote agricultural sector innovation and modification. Capacity is needed throughout the sector—on farms, in producer organizations, in private firms, in research institutions, in government agencies, in rural financial institutions, and throughout civil society. Many investments contribute to capacity but using training and education to develop human capacity is critical. However, many AET institutions and programs have stagnated in recent years and failed to adapt to the changing needs in agriculture. AET institutions face key problems that include the inability of tertiary and workforce development programs to produce workers with skills relevant to support country development goals and compete in global and national markets. Other problems are poor admissions and recruitment systems, lack of university involvement in national agricultural innovation systems, and lack of aggressive outreach activities that would tie AET institutions to both sector institutions and to practical development needs. InnovATE was designed to provide broad support to agricultural development and food security initiatives and catalyze the program and institutional development to meet country needs. The project targeted the most neglected—yet critical—human and institutional building blocks required to ensure modern and sustainable AET systems for food security at a country-level. The project supported investments in: AET institutional development; the short-term human capacity development to implement agriculture and rural development programs; training and leadership development for women in agriculture; training for agricultural technicians, managers, and scientists; and coordination of investments in tertiary agricultural education. The project was designed to provide thought leadership in AET strengthening, define and disseminate good practices, and stimulate innovation in (and awareness of) the importance of AET systems. The InnovATE cooperative agreement was awarded to a consortium of four United States (U.S.) universities. Virginia Polytechnic Institute and State University (Virginia Tech) was both the project leader and the Management Entity (ME). The three other universities were the University of Florida 3 (UF), Pennsylvania State University (Penn State), and Tuskegee University (TU). InnovATE was housed within the Office of International Research, Education, and Development (OIRED) at Virginia Tech. The project’s key objective was defining and disseminating good practice strategies, approaches, and investments for establishing efficient, effective, and financially sustainable AET institutions and systems. The project also aimed to support country programs geared toward reforming and investing in AET. The project was designed to work with existing public and private education and training institutions. The project objectives focused on human and institutional development, accommodating country￾specific needs, and were designed to be demand-driven and respond to requests for assistance from USAID Missions. The approach to achieving these objectives was to focus on human and institutional development by accommodating country-specific needs “to avoid repeating the mistakes of past investments in agricultural institutional development.” 2.2 INNOVATE’S THEORY OF CHANGE The ToC was based upon the concept that InnovATE and Missions collaborating and disseminating good practice principles and approaches would contribute substantially to both the capacity building and sustainability of AET institutions in Feed the Future countries. InnovATE’s partners were to include higher education, secondary schools (e.g., vocational education), primary schools (e.g., 4H clubs), and other stakeholders (e.g., policy makers, private sector entities). The parallel Modernizing Extension and Advisory Services (MEAS) activity would support work with extension entities and be followed by the Developing Local Extension Capacity activity. Through training activities and development of good practice materials, InnovATE would contribute to improving AET systems and institutions. InnovATE was expected to supplement and reinforce Feed the Future’s flagship projects and other Mission efforts designed to strengthen food security and resilience. Although InnovATE’s activities were influenced by specific needs and constraints within each country, the project’s approach can be broadly understood as seeking to improve food security through the following activities: 1) assessing human capital demands of the private sector; 2) analyzing the needs and constraints within Feed the Future countries for innovative AET research and tools; 3) developing and supporting both an in-person and an online Community of Practice, and addressing needs and constraints involved in this community; and 4) designing training modules, curricula, and other learning products tailored to Missions’ and host country needs. InnovATE’s Theory of Change 2.3 PROJECT DESIGN InnovATE was designed to assess human capital needs and AET reform priorities by collaborating with Mission staff, implementers of other Feed the Future projects and activities, and AET points of contact. InnovATE’s technical expertise in AET issues was to derive from host country and global specialists. 4 Missions were to guide its analytical work. For customized products, InnovATE planned to rely on requests for assistance from these stakeholders and, ultimately, on Mission buy-ins or associate awards (AAs). Support for AET systems or institutions could be administrative, policy-related, or pedagogical. InnovATE’s AAs were designed to depend on Mission funding; therefore, although the project itself was only $5 million, Mission and other office buy-ins were anticipated to total up to $1.25 million more. The buy-in ceiling was later raised to $3 million, with a total ceiling of $72 million, which is the leader plus the hypothetical ceiling of AAs. The project design incorporated an implicit assumption that much of the AA design work would come from Mission staff input and other key stakeholders. Occasional short-term technical support was to come from the InnovATE team. Therefore, the Request for Application (RFA) specifically stated that, “The MAETS [later InnovATE] Project must accommodate country-specific needs to avoid repeating the mistakes of past investments in agricultural institutional development.” Sections of the RFA that relate to InnovATE activities include: 1) The project will be demand-driven, responding to requests for assistance and providing [a Leader with Associates (LWA)—hereafter AA or Associate Award] mechanism for Missions to fund AET activities. 2) Since country capacities and needs vary widely, the project will tailor strategies and activities to the specific country and sector context. 3) Project activities will likely require sourcing technical expertise in AET issues from host country and global specialists. 4) Design work will be responsive to Missions’ or other offices’ requests for assistance, generally requiring funding or co-funding of such activities. 5) Analytical work will also be guided by Mission and other practitioner needs and interests, as reflected in funding for AET programs and in consultations on AET issues. This will often require proactive project selection of institutions and case studies, evaluations, and analytical work to provide the best evidence relating to good practice and AET issues. 6) The project will seek collaboration with regional fora and institutions in carrying out activities. 3.0 EVALUATION METHODS AND LIMITATIONS This evaluation used a mixed methods approach that included desk review of relevant documents, remote key informant interviews (KIIs), online surveys, and case studies. Data collection was conducted in both May of 2017 and in May of 2018 by two different evaluation teams (ETs). 3.1 EVALUATION METHODS FOR DATA COLLECTION AND ANALYSIS The first round of data collection for the evaluation was conducted in May of 2017 using a) document review; b) remote key informant interviews with ME and Partner Institution representatives, USAID/BFS staff and other stakeholders s; c) a web-survey with U.S. university partners, USAID/BFS Washington and Missions, project implementers, and AET institutions; and d) a case study design. The initial interviews were limited because the ET was unable to collect data from other key stakeholder groups during this period. Data was collected through interviews primarily from project implementers, partner institutions, and USAID/BFS representatives. Although attempts were made to contact them, there were no interviews conducted with host country AET institutions, non-governmental organizations (NGO), or USAID Missions except one interview held with a USAID Mission respondent. 5 USAID/BFS deemed the initial draft report inadequate. As a result, the Feed the Future Global Program Evaluation for Effectiveness and Learning project (PEEL), which is a USAID/BFS mechanism designed to manage evaluations of Feed the Future activities, and USAID/BFS discussed and agreed to collect additional information to fully answer all key EQs. This second round was conducted between April and May of 2018, and used semi-structured interviews conducted remotely with key informants (KIs), document reviews such as the monitoring data reports, and a review of other reports for experiential stories to more fully answer each EQ. The purpose of the second round was to collect supplemental data that would help clarify and adequately answer all EQs7. Document Review: The research began by reviewing project documents, work plans, monitoring and evaluation (M&E) plans, monitoring reports, previous audits and evaluation reports, sector reports from host-country governments and donors, and relevant policy documents. The ET also reviewed the final monitoring report on project results. Although the EQs do not directly focus on the extent to which InnovATE met its performance targets, the final monitoring report on project results is relevant to assessing the project’s overall success because it: 1) provides a snapshot of the evidence project implementers presented for having met significant benchmarks; 2) provides a baseline against which to measure post-project sustainability of results; 3) can be analyzed to determine whether the articulated results could be achieved by meeting the benchmarks established by the project; 4) provides a framework for a discussion about significant changes reported by stakeholders in interviews and surveys, and illustrates the extent to which those changes were intended or unintended; and 5) provides additional documentation to add to project implementer interviews. The document review was initially conducted in 2017 as a basis for developing the questionnaire survey and in-depth interview guides. However, document analysis continued through data collection because KIs either sent documents or indicated documents related to their perception of the project. In 2018, the second ET obtained updated project reports to review for further project information and monitoring data. The reports were obtained and reviewed and analyzed in alignment with relevant EQs determined. The monitoring data results were categorized in accordance with outputs and outcomes and compared with Life of Project (LoP) targets. The ET ranked each interviewee able to provide information about the project. The ranking system was developed to capture interviewee statements within as complete a context as possible during a telephone discussion. Key informant Interviews: Data was collected from interviews using semi-structured questionnaires that specifically referenced all the EQs. As interviews began, the ET used each participant as an entry-level referent to identify other KIs who could provide pertinent information. This method of sampling is often called “chain referral” or “snowball sampling.” The ET conducted 27 interviews in 2017. In 2018, 18 additional interviews were conducted with Mission points of contact (PoCs) and other stakeholders. In each case, semi-structured interview questions were asked and accompanied by follow-on probes. Interviewees were invited to speak about various aspects of the project as much as they wanted— including activities, achievements, and challenges not addressed in the original interview guide. KIs were identified based on review of project documents and on recommendations from InnovATE and BFS. Interviewees in 2017 included representatives from institutions that received the cooperative agreement (n=10), USAID/BFS (n=5), a USAID Mission PoC (n=1), and other stakeholders (n=11), including the ME, the Chief of Party (COP) of another similar project, and a representative from a regional agricultural education forum. Apart from four instances where participants declined to be recorded, interviews were digitally recorded and transcribed verbatim. For the four participants who did not consent to being recorded, copious notes were taken during the interviews and subsequently expanded into a written summary. 6 Interview guides were designed to allow interviewers freedom to ask spontaneous questions in an effort to uncover important and unanticipated information. The depth of information in this evaluation was primarily generated during the interview phase. Post-interview data review produced new information, after which additional stakeholders were identified. The design was arranged to be flexible and responsive. Online survey: The online survey was only conducted in 2017 through SurveyMonkey platform and a link sent to identified stakeholders. The sampling approach was targeted—not intended to be random or made general. Data from surveys was intended to be triangulated with secondary data and KII findings. The U.S. university partner questionnaire was distributed to 47 people and was completed by eight. The survey for USAID/Washington and Missions was distributed to 77 people and completed by two. The survey of project implementers was distributed to 32 people and completed by only three. The survey of AET institutions and private sector was distributed to 23 people, but only one completed. To mitigate effects of the poor response rates, a short version of the surveys—with basic questions about perceptions of the purpose, clarity, efficiency, and relevant nature of InnovATE—was distributed to over 1,000 subscribers through a large listserv and via Twitter. This shorter survey only generated 17 complete responses. Given that each survey was distributed multiple times in 2017, and the response rate was still low, the 2018 ET elected not to distribute another survey. Instead, the ET analyzed the 37 responses stakeholders submitted in 2017. Only three questions from the survey garnered enough responses (n=14) to make further analysis worthwhile. Those questions focused on overall project effectiveness in terms of each of its three primary objectives (trained agricultural professionals, dissemination of materials, and human and institutional capacity). Table 1. Sample of Stakeholder Types by Data Collection Method Survey Respondents 2017 KI's 2018 KIs USAID Representative USAID/BFS PoC USAID/BFS PoC Agrilinks Representative USAID Mission PoC USAID/Senegal PoC Partner Institution3 Project Implementer USAID/Honduras PoC Workshop Participant U.S. Partner Institutions USAID/Armenia PoC Project Staff Feed the Future Project Staff USAID/Nepal PoC - Other U.S. Institutions USAID/Nicaragua PoC - - Project Implementer - - Project Staff - - Feed the Future COP - - Stakeholder (Honduras) - - Stakeholder (Armenia) Through document review and other relevant publications, the ET identified and compiled illustrative examples in three countries (Armenia, Honduras, and Senegal) that detail their experiences with 3 Both U.S. and abroad. 7 InnovATE activities. These examples provide detailed information on how the program has been implemented and the effects and lessons learned. Data Analysis In 2017, both academic and non-academic respondents answered the three survey questions described above. The non-academic respondents included two USAID staff members, one individual associated with Agrilinks, and three project implementers. The academic survey respondents were from both U.S. and host country universities and included workshops and symposia (n=1) participants and people working with InnovATE (n=7) in non-project implementation roles. Rankings for these responses are 0=no answer (excluded from the charts), 1=less than 20 percent effective; 2=20 percent to 49 percent effective; 3=50 percent to 79 percent effective; and 4=80 percent or more effective. In addition to deeper examination of 2017 survey data, the ET further analyzed transcripts from KIIs conducted in 2017. This analysis used opinion mining, content analysis, and other qualitative data analysis techniques. Seeing these transcripts in the light of the new data obtained from 2018 interviews and document reviews, the ET was able to categorize them according to specific rankings for concepts and themes. To compare 2017 data with 2018 data, the ET utilized the Most Significant Change (MSC) approach to gather change stories and help construct case studies, based on interviews and review of updated project reports review, for each country. The MSC method is widely used for these types of analyses to help understand key stakeholder perspectives regarding effects of their project over time. There are three basic steps to using MSC: 1) determine what types of stories should be collected (in this case, stories about InnovATE’s effectiveness in the last years of the project were collected); 2) collecting the change stories from interview data and documents to determine which are the most significant; and 3) discussing those change stories (during data collection) with other KIs who could corroborate. MSC is not just about collecting and reporting stories but is also about elucidating processes and results and learning reasons behind different experiences. This approach takes maximum advantage of the potential interviews have to bring out individual experiences with the project. MSC stories are highlighted in text boxes throughout the Findings and Conclusions sections. The MSC stories also refer to a hierarchy of changes (and note at which of three levels the change occurred—from increasing knowledge to changing value to actions at a systemic level) and the sustainability of changes (see Annex 5, Table 4). The above analyses included collecting keyword frequencies, mining opinion, and using sentiment analysis tools to assess the perceptions each group of respondents held about aspects of the project mentioned frequently in interview statements. The analysis also included responses from the open-ended survey questions and involved a comprehensive examination of word contexts that denoted positive, neutral, and negative sentiments. For purposes of discourse analysis, the analysis began with an inductive process of identifying the specific relationships between the texts of the interviews/focus groups and survey data, and institutional and social meanings that carried over from the project goals to the interview discourses. Each sentence containing these words was read in context to determine whether the speaker was using them in a positive, negative, or neutral sense. Annex 5 includes tables that list the positive, negative, and neutral instances of keywords relating to the EQs. First, the keywords were reduced to word stems so that all tenses and forms of the words appearing in interviews and responses were recorded. 3.2 DATA AND METHODOLOGICAL LIMITATIONS This evaluation design had several limitations during the two data collection processes that should be taken into consideration while interpreting the findings and recommendations. However, the ET worked to mitigate these limitations as much as possible. The major limitations directly resulted from the evaluation plan, which included an online survey, remote interviews and a document review. 8 • Different ETs conducted interviews at two separate periods (2017 and 2018). Even though the data collectors in 2018 made every effort to use similar methodologies, data collected at two different periods by different evaluators, targeting the same pool of stakeholders introduces respondent fatigue and raises concerns about the extent to which it can be combined into a coherent whole. • Lack of field visits. InnovATE is implemented in several Feed the Future countries. Due to limited resources, the ET was unable to visit any of these countries and collected data remotely through phone interviews. This limitation may have affected the amount, quality, and interpretation of data collected. Professional development program evaluators generally see field visits as mandatory for reaching definitive conclusions in evaluations (see Michael Quinn Patton, “Evaluation in the Field: The Need for Site Visit Standards” American Journal of Evaluation, Vol 36, No 3, 2015). Studies show that telephone interviews are typically, and on average, shorter than interviews conducted face-to-face. This shorter duration results from the participant speaking for less time, holding the floor for shorter stretches at a time, and the researcher talking more than in face-to-face interactions. Together, these factors result in a reduction in the coverage of important themes and in the collection of important information (see Annie Irvine, “Duration, Dominance and Depth in Telephone and Face-to-Face Interviews: A Comparative Exploration,” International Journal of Qualitative Methods Vol 10, No 3, 2011). • Low response rate to an online survey. An online survey conducted in 2017 had a low response rate (3.7 percent) that did not yield a large enough sample size (n=37) to be statistically valid. This affected the quality of the data collected and the applicability of findings. Although the 2017 ET worked to alleviate the issue of low response rates by using multiple reminders, phone calls, and reducing the survey length, none of these strategies worked. To account for this limitation, this portion of the data collection was included in the findings with respect to only three questions on effectiveness. This resulted in an analysis that depended almost completely upon document reviews and phone interviews. • Response bias: In 2017, the KIs interviewed were primarily from InnovATE’s partner institutions and, subsequently, provided limited perspectives. In 2018 the interviews were conducted with a more diverse set of stakeholders—especially from Missions. This second collection of data was still not able to reach sufficient numbers of certain types of stakeholders—including Missions and representatives of AET institutions. The number of KIIs in the selected countries was limited by how much USAID, implementer staff, and other potential key stakeholders were willing to be interviewed and how in-depth their knowledge of InnovATE was. • Selection bias: Interviewee identification in Missions and implementing partners (IPs) across different countries was facilitated by USAID/BFS or Virginia Tech. This introduced more selection bias into the process than a typical on-site evaluation with focus groups and KIIs would. 9 4.0 FINDINGS AND CONCLUSIONS 4.1 EQ 1: TO WHAT EXTENT HAS INNOVATE ACHIEVED ITS OBJECTIVE OF DEFINING AND/OR DISSEMINATING GOOD PRACTICE STRATEGIES, APPROACHES, AND INVESTMENTS FOR ESTABLISHING EFFICIENT, EFFECTIVE, AND FINANCIALLY SUSTAINABLE AET INSTITUTIONS AND SYSTEMS? In what ways has the project generated, documented, and disseminated information on key constraints to AET systemic development, innovations to address those constraints, and engaged stakeholders and collaborators to contribute? In what ways did/did not the information generated address the specific needs of the Missions requesting and provide usable, actionable recommendations? How effective were project outputs in achieving outcomes in terms of Mission AET program investment, design, and operations? FINDINGS Among the Mission respondents who answered the questions on effectiveness across all focus areas— efficiency, relevance, effectiveness, and sustainability—in 2018 (n=14), three rated the project as “moderately or highly effective” in each category. Representatives from the Armenia (n=2) (see Text Box I below) and Honduras (n=5) Missions praised both the use of InnovATE products and the project implementer’s effective communication with the Mission. Text Box 1 – Armenia Most Significant Change: InnovATE’s AA to the Armenia’s International Center for Agribusiness Research and Education (ICARE) enabled them to support the well-regarded Agricultural Training Center (ATC), an AET that, in InnovATE’s first year, achieved high placement rates for its graduates. The AA focused on curriculum reform and establishing a U.S.-based endowment fund. Hierarchy of Changes: InnovATE achieved results at the knowledge level in Armenia by broadening research, policy, implementation, and enterprise horizons. To some extent, InnovATE also saw growth in investments through supporting expansion of research, policy, implementation, and enterprise capacities. InnovATE helped develop a Food Safety Systems Management (FSSM) Professional Certificate Program, which raised human capacity needs in food safety. The intensive program focused on improving the safety of Armenia’s food supply chain from production to consumption. InnovATE also enabled ATC personnel to travel to multiple U.S. cities to solicit endowment contributions from the Armenian diaspora. Sustainability of Changes: Eventually the endowment was established but, to date, has only received approximately $50,000—well below the $300,000 target. This may have been due to perceptions of corruption in Armenia, which scored high on Transparency International’s corruption index in 2017. This attitude may be changing given recent developments in the country. The AA still has a year to run, during which the implementer will continue trying to meet its target endowment level for the ATC. These Missions report using products to 1) improve existing programs for training in nutrition-sensitive agriculture; 2) design programs to support accountability and gender inclusion in agricultural training; and, 3) support new education-based interventions to improve citizen security in rural areas. One member said that: “There was flexibility not only to disseminate preliminary findings to the Missions but to also to external stakeholders. The quality of the reports was excellent, and there was also ongoing 10 follow-up. It incorporated Mission comments and responded to questions and produced a final report that was useful to the Mission.” – USAID/Honduras Other Mission respondents (n=8) considered the project’s training modules and materials to be useful— particularly for continuing education programs to enable AET professionals to encourage girls to consider a range of careers in agriculture, assist students in choosing courses and curricula commensurate with their skill levels, support new curriculum development, and meet training needs for agricultural value chains, Eight Mission respondents who worked directly with InnovATE, found their products helpful in designing programs to address some key constraints in their host countries, particularly in assuring that AET institutions worked on alleviating problems of engaging women in agricultural training. Interviewees in both cohorts (n=34) and 14 respondents to the 2017 survey found that a major element missing from InnovATE products was a discussion of broad-based interdisciplinary, long-term solutions and activities designed to build human capital. A preponderance of interviewees from both cohorts (n=32) thought that Missions did not engage enough in the project to generate usable, actionable recommendations or outcomes in terms of Mission AET program investment, design, and operations. Some interviewees, however, who had limited or non-existent awareness of the project, including a representative from one Mission, said that although they had heard of the project, they were mostly not acquainted with its activities. The interviewee cited limited Mission funds that were mostly committed, which disallowed participation in the project. One Mission contact, interviewed in 2018, stated that the multi-country scope of the project was a challenge from the beginning: “It just struck me in the early days several years ago when I was talking to them that in terms of this part of their project [capacity building for AET institutions] they didn’t seem to understand the lay of the land fast enough. They were spending a lot of time trying to figure it out.” – USAID/Armenia Respondent Comments from the interviews conducted in 2017 had varied but similar perspectives about InnovATE’s effectiveness. Opinions about the project ranged from middling to high. The USAID/BFS (n=5) interviewees and U.S. partner institution (n=10) KIs were moderately positive about the project’s training achievements. Some interviewees (n=3) considered the scoping reports and papers valuable and high quality and noted that substantial effort went into refining an analytic agenda for AET. The project completed pilot studies focusing on gender, AET in conflict areas, vocational training, and youth in agricultural education. Missions found these valuable and the studies resulted in the development of progressive education modules on these topics. Representatives from the BFS and U.S. partner institutions who were interviewed in 2017 generally did not rate the project as useful in terms of its contributions to sustainable AET institutions. BFS interviewees (n=5)—even those close to the project—had no positive comments about InnovATE’s sustainability or its effect on institutions. Out of 12 partner institution interviewees, only one had positive comments about sustainability. Several respondents agreed with the individuals who said that: “It didn’t take a brilliant person to take a look at the budget and feel like this is not sustainable.” – Partner institution representative “I ask myself if some of the stuff that they’re doing that the Mission is very happy with--I have little hope that anything they’ve done is very sustainable once InnovATE pulls out.” – BFS representative In 2017, some interview respondents (n=5) spoke positively about the project’s efficiency and investments; however, most comments related to the general concept of an activity like InnovATE rather than the actual InnovATE activity. Partner institution respondents (n=9) rated the project 11 efficiency and return on investment positively. Interviewee comments from 2017 were moderately negative (ratio of 6/0) about sustainability and largely negative (22/6) about institutional development. In 2016 and 2017, a thematic study on rural youth violence in El Salvador, Guatemala, and Honduras concluded, and AAs were signed in Afghanistan, Armenia, and Senegal (in 2017), and, prospectively, in Honduras. After this increased activity, some interviewees were positive about the project’s utility and effectiveness. Representatives from the Armenia (n=2) and Honduras (n=5) Missions praised both the products’ usefulness and the project implementer’s coordination. However, among the 2018 interviewees, there were also individuals among the Mission respondents (n=5) who had limited awareness of the project and its products. Eleven (11) individuals who were interviewed (out of 47 interviewed in both cohorts) did not find InnovATE products useful or effective for establishing efficient, effective, and financially sustainable AET institutions and systems in countries where these were the project’s focus. The majority of interviewees in both cohorts (n=31), and 14 of the 32 people who responded to the 2017 survey, did not rate the project as successful in creating products addressing key in-country AET constraints. Some Mission respondents who worked directly with InnovATE (n=8) found the products helpful in designing programs to address AET constraints. However, 32 interviewees from both cohorts believed Missions did not engage in the project enough to generate usable, actionable recommendations or outcomes in terms of Mission AET program investment, design, and operations. Representatives of partner institutions mentioned the term “effective” only nine times in connection with InnovATE activities, and eight of them mentioned the term in a negative context (keyword/ sentiment analysis—Annex 5). For the most part, however, remarks were not highly critical of the project. In fact, most interviewees agreed that the concept was good but there were many challenges (this was mentioned twice as many times as “effective” was) in communication among the partners, with the Missions, and with USAID. Multiple PI and BFS interviewees agreed that, while the project did not achieve its full potential, there were indications it would have if given more time. One individual noted: “InnovATE was a little bit ahead of its time. I think that there’s now attention being paid to human and institutional capacity development at a level that it hasn’t been for a while. Were InnovATE to have another couple of years, it would be really effective because we’ve been the only ones playing in the field for the last five years.” – Representative of an InnovATE PI. CONCLUSIONS In the last year of the project, some positive effects of InnovATE activities were observed in several USAID Missions. Effects related to development of AAs, the completion of well-received studies, and the development and dissemination (online) of a significant number of products that contributed to the AET knowledge pool. As of 2018, InnovATE has generated useful and actionable research and recommendations on AET systemic development and on engaging stakeholders and collaborators for three Missions. Even those Mission representatives, however, were doubtful about the project's long￾term contribution to sustainable AET institutions. InnovATE’s products have not been very successful in addressing key in-country AET constraints in terms of Mission AET program investment, design, and operations. While the project did generate innovative products, especially in the form of training modules and thematic studies, the project reports reviewed and interviews with Mission representatives do not indicate that the project addressed key constraints to overcoming the most significant obstacles to AET institution development—limited financial resources and a dearth of well-qualified professional staff—except in Armenia. InnovATE’s most prominent challenge was in educating Missions about the assistance the project could provide. As of 2018, however, InnovATE generated useful and actionable research on key constraints to AET systemic development, on addressing those constraints, and on engaging stakeholders and 12 collaborators for three Missions. While the few Missions who utilized InnovATE rate most of its aspects as highly effective—including dissemination of useful materials—the project has not had much success contributing to sustainable institutions and the target audience still lacks awareness of the project. RECOMMENDATIONS • To mitigate difficulties achieving Mission buy-ins, which stem from lack of time and resources on the part of Missions (and, with respect to some Missions—such as Nepal—a lack of interest), BFS and/or Regional Bureaus could sponsor AA's for Missions with the greatest need for improvement in AET training. Need could be determined by scoping assessments. • Since a number of interviewees found that InnovATE's online resources were useful, some steps could be taken to expand and improve dissemination of good practices materials for AET institutions. Steps might include setting up a specific web portal for AET institutions in selected countries; conducting webinars and interactive video conferences with AET institutions on specific issues of concern; establishing a smart-phone-based network of AET professionals, and/or facilitating the establishment of regional online communities of AET institutions. 4.2 EQ 2: IN WHAT WAYS HAVE USAID MISSIONS, DONORS, POLICY MAKERS, AND AET PROFESSIONALS USED INFORMATION GENERATED FROM INNOVATE’S GOOD PRACTICE PAPERS, ANALYSES, AND THEMATIC STUDIES, AS WELL AS THE TRAINING MODULES AND TRAINING COURSES (FROM BOTH THE TRAIN AND LEARN COMPONENTS)? WHAT OPPORTUNITIES ARE THERE FOR THESE PAPERS, ANALYSES, AND STUDIES TO BE USED BY THE DIFFERENT STAKEHOLDER GROUPS TO ENHANCE AET? FINDINGS All representatives in Honduras rated highly the products used in training (see Text Box 2 below.) Text Box 2 – Honduras Most Significant Change: The highlands of western Honduras have limited resources and educational opportunities and are being inundated by families escaping drug violence. Families move from urban centers experiencing rampant gang activity into rural towns like Sta. Rosa de Copan. Most have no means of supporting themselves. Gangs actively recruit youth as early as primary school. InnovATE funded an AA that supported a widely known and well-regarded study focusing on ways to stem youth violence in Central American drug corridors. The study defined key constraints facing youth in Sta. Rosa de Copan and recommended bolstering youth education in agriculture to provide viable career alternatives to young people in the region. The Mission used the study to design new programs geared at improving livelihoods of vulnerable youth. As of this report, the Honduras Mission had funded a buy-in and then later an associate award that is just coming online now. Hierarchy of Changes: The AET system in Honduras is not robust, agriculture is not viewed as a prestigious career, and the extension system was effectively abolished in the 1990s. Vocational education, donor activities, and the private sector remain. In this context, InnovATE’s contributions to change at the knowledge level (by introducing new concepts to frame debates and put ideas on the agenda) and at the values level (by increasing Mission knowledge, access to data, and developing their capacity to understand and respond to important issues) are substantial. There is potential in Honduras for InnovATE to effect change at the action level by modifying existing programs or policies and fundamentally re-designing programs, policies, strategies etc. Sustainability of Changes: Given that the mayor of Sta. Rosa de Copan used InnovATE’s research findings to request funding from the central government, negotiations are underway for a follow-on AA from the Mission. 13 One informant from Senegal feels that the tools InnovATE developed have the potential to support better in-country AET training. A few Missions (n=2) report using InnovATE projects, such as the well￾received studies in Honduras and the sexual harassment policy (gender empowerment) symposium in Senegal, which involved university administrators from around Africa. Only a limited number of Missions (n=2) report using InnovATE products in program design. Specific studies, conducted in collaboration with Missions, were rated by interviewees familiar with them (n=5) as impacting the design of programs to disseminate good AET practices. Only nine of 18 in￾country respondents felt that they could talk about InnovATE’s effectiveness in encouraging stakeholders to utilize its good practice papers, thematic papers, training modules, and other products. Of the nine, three said InnovATE was highly effective, three said InnovATE was somewhat effective, and three considered InnovATE ineffective in achieving this objective. Mission respondents in Honduras and Nicaragua were particularly positive about the extent to which the studies, papers, and training modules produced by InnovATE enabled them to plan AET support programs to reduce barriers to participation of women and disadvantaged youth in agricultural education. According to 15 of 18 in-country respondents, lack of awareness of the project made it difficult to effectively brand and disseminate InnovATE's products among stakeholders. Five interviewees suggested that translating InnovATE materials into local languages would increase their use. Agrilinks users (n=4) were not always aware that information on the site was attributable to InnovATE. Some Mission staff (n=7) interviewed stated that they would use InnovATE's online resources more if they received notification when new InnovATE content posts on Agrilinks. Ten (10) interviewees (out of 14) felt the project was generally not successful in encouraging stakeholders to utilize good AET practices. USAID interviewees (n=4) agreed that the materials were not utilized. Two of the 4 interviewees questioned the quality of materials produced. A few knowledgeable stakeholders (n=3) in Missions (technical advisors) believed that InnovATE’s training materials were at a basic level and useful only to non-professional audiences. One key informant suggested that InnovATE's online resources, too, are suitable for audiences with more basic levels of knowledge on AET topics. They felt that while this may be useful to some users, Mission staff would benefit from resources geared to an audience of technical experts. Another key informant emphasized that resources disseminated online are less likely to be used if their applicability to a country or region is not made clear. Representatives of partners and the lead institution (n=11) agreed that the project was not successful in reaching the private sector, donors, policymakers, and AET professionals to any great extent. Only nine of 18 in-country respondents felt that they could speak to how effectively InnovATE encouraged stakeholders to utilize its good practice papers, thematic papers, training modules, and other products. Of the nine, three said InnovATE was somewhat effective in achieving this objective, three said InnovATE was highly effective, and three considered InnovATE ineffective. According to 15 of those 18 in-country respondents, lack of awareness of the project made it difficult to effectively brand and disseminate InnovATE's products among stakeholders. Agrilinks users who were interviewed had thoughts on this issue: four noted that they were not always aware that information on the site was attributable to InnovATE (a situation acknowledged by the representative of the lead institution in an interview); five suggested that translating InnovATE materials into local languages would increase their use; and some Mission staff (n=7) said they would use InnovATE’s online resources more if they received notification when new content was posted on Agrilinks. Informants who recommended that translating InnovATE materials into local languages would increase accessibility among AET officials gave several reasons, which included increased likelihood that AET officials would find InnovATE publications using Internet search engines, and that oral presentations in 14 local languages were helpful, but presentation slides and materials written in local languages would help participants retain more information. Rankings by KIs appear to vary by country and stakeholder category. Stakeholders working in Missions that had worked with InnovATE (n=7) gave the most positive responses overall. These are illustrated by the following comment: “The research study [on youth and rural violence] was fantastic and fed into a broader assessment being done for purposes of Mission programming.” – Mission representative. One KI suggested that InnovATE’s online resources are suitable for an audience with basic knowledge of AET topics and that, while this may be useful to some users, Mission staff would benefit from resources targeted to an audience of technical experts. Representatives from partners and from the lead institution (n=11) said they believed the project did not successfully reach the private sector, donors, policymakers, or AET professionals. Ten (10) interviewees expressed negative opinions on the project’s effectiveness in encouraging stakeholders to utilize good AET practices. According to three informants, InnovATE had attempted to set up its own web platform to establish a Community of Practice but was not successful. In 2017, two BFS interviewees referred to this attempt as expensive and impractical. Midway through implementation, InnovATE began using Agrilinks for its Community of Practice and, according to project staff, partner institutions, and Mission staff, the change to Agrilinks has been a useful and informative way of disseminating materials. One informant at a USAID Mission expressed concern that thematic papers disseminated online to a specific country would not have clear, immediate applicability to AET officials and that these officials would not have the time or resources to determine whether findings were relevant to their country, region, or AET system constraints. Most interviewees from the first cohort (2017) noted that Mission use of products, outputs, and outcomes was a significant problem. One individual summed up a particular roadblock that several others (14 interviewees in all) alluded to: “A lot of times, the problem is that the topic of AET falls between Mission [priorities]. The education people are used to primary education and the workforce people don’t think about tertiary education. They certainly don’t think about agriculture. And then agriculture people don’t think about education training and so, there’s--in a rare Mission will you find somebody there who can do that. I think the original plan was that there was going to be a lot of buy￾ins…[but] the way they’re structured, with the exception of a few Missions, I think it was a hard sell.” – Representative of InnovATE partner institutions Only nine of 18 KIs felt they could speak to how effectively InnovATE encouraged stakeholders to utilize its good practice papers, thematic papers, training modules, and other products. Three KIs said InnovATE was somewhat effective and three said InnovATE was highly effective in achieving this objective. Most of the 18 respondents (n=15) mentioned that lack of awareness made it difficult to effectively brand and disseminate InnovATE’s products among stakeholders. One Mission respondent who was very satisfied with the project said: “The challenges were in the awareness of InnovATE. Because of that, they had lower effectiveness in terms of documenting and dissemination than they should have.” CONCLUSIONS Thus far, InnovATE's greatest success contributing toward sustainable institutions has been in Honduras, Senegal, and Armenia. These are places where AAs were established or are being considered. Informants from these three countries in three distinct regions described InnovATE's research as “very 15 useful” and sometimes even "game-changing" for Missions designing new programs. InnovATE’s activities contributed to the AET knowledge pool and produced some good papers and studies. However, they have not succeeded in making their products broadly known in Feed the Future countries in general. By developing and disseminating products online, InnovATE activities contributed to the AET knowledge pool. In 2017, respondents noted that the project produced good papers and studies. Mission representatives interviewed in 2018 said that InnovATE generated useful and actionable research on key constraints to AET systemic development, developed information for addressing those constraints, and provided tips on engaging stakeholders and collaborators for Missions in Central America and at least one Mission in Africa. Programmatic decisions at Missions limit opportunities for Missions to use information generated from InnovATE products. Further, when AET officials in Feed the Future focus countries experience language and time constraints, it limits the extent to which they can locate, and consult InnovATE resources. RECOMMENDATIONS • To promote good AET practices, project implementers should develop ways to make more direct contacts with AET institutions. This could be done through networks already established by partner institutions or through conducting regular workshops. • Implementers should prioritize private sector collaboration to leverage the positive impact of their projects. Interviewees had several suggestions for doing this, including 1) conducting campaigns and workshops highlighting skills of InnovATE-supported institution graduates, and demonstrating benefits of their work to businesses and business associations; 2) investing in developing more internship and/or apprenticeship programs for AET students and graduates; 3) supporting public/private partnerships at the secondary education level that support equity by targeting women and low-income and marginalized students; and, 4) helping expand the range of AET courses at local institutions to include other types of relevant training (i.e., involving business schools or social science courses related to leadership and policy) to assure that AET graduates develop skills important to the private sector. • In-country stakeholders aware of InnovATE studies suggested that thematic studies focusing on the most immediate issues confronting agricultural education in the country should be required in future projects of this kind. They went on to say that the ME should make a concerted effort to ensure recruitment of experts from institutional partners or outside consultants with experience working with these issues. • Many interviewees agreed that in order to be useful to a wider range of AET practitioners, applicable studies must be translated into local languages. 4.3 EQ3: HOW EFFECTIVELY HAS INNOVATE COMMUNICATED ITS OBJECTIVES AND POTENTIAL VALUE TO MISSIONS? What did Missions find most valuable about InnovATE project outputs arising from scoping assessments and AAs (as applicable)? Most representatives from partner institutions and the lead institutions (n=11) stated that communication with Missions was handled by USAID/BFS. These individuals stated their belief that this was due to procurement sensitivities. However, it appears from other evidence that this strategy was adopted to reduce the burden on Missions and to ensure that BFS was aware of contacts with missions. The partner institution representatives did say that they had direct contact with Senegal, which, they believed, enabled them to respond to the Mission's needs more efficiently. Some of these representatives (n=5) also mentioned that, initially, they did not understand the difference between buy-ins and AAs. For AAs, USAID/BFS advises that there is a procurement sensitivity 16 dimension, which means that the prime applicant should only be in contact with the Contracts Office during that process. The majority of consortium institution representatives (n=7) who were interviewed said that they were not clear about the process for obtaining buy-ins and awards. They stated that they found the established protocols for communicating with Missions bureaucratic and cumbersome. Of the eight (out of 18) respondents across four countries who agreed to provide assessments on how effectively InnovATE communicated its objectives and potential value to Missions, three rated the project as highly effective, three as somewhat effective, one rated it as having low effectiveness, and another one stated that InnovATE was not effective at all. A few KIs (n=2) in the Missions were not aware of any AET activities in their country, even though activities had taken place. Representatives of other Missions (n=3) stated that they were interested in InnovATE projects, but that they only heard of the project after it had been going on for several years. Because they became aware of InnovATE only after their funds had been obligated, they were unable to participate in InnovATE projects. One USAID/Nicaragua KI said that they were conducting additional training that resulted from work InnovATE performed in their country. Two USAID/Armenia respondents stated that InnovATE contributed to curriculum reform under their AA, which was critically needed, and that the project added a food safety certificate program to the AET institution's course offerings. Two KIs in Senegal had positive descriptions of InnovATE's pilot exercise on community participatory curriculum development (CPCD) and plans for the resulting AA. Findings from the analysis of the 2017 interviews indicate that partner institutions rating papers, (most of which addressed general issues in AET) that InnovATE produced were neutral about the utility of good practices papers (three positive and three negative statements). However, they rated thematic studies designed to address specific problems in AET, such as gender inclusion and youth violence, as being moderately successful (seven positives versus five negative statements). USAID/BFS interviewees considered papers and studies produced by InnovATE as slightly less than successful (10 positive versus eight negative statements). COPs from in-country Feed the Future flagship projects who were interviewed (n=3) were not familiar with the InnovATE project. No Mission respondents were able to provide any instances where the project attempted to connect with Feed the Future projects. Representatives from partner institutions and the lead institution (n=8) believed that the project needed to have a greater in-country presence in Feed the Future focus country missions. In interviews conducted in 2018, eight of 18 KIs across four countries provided assessments of how effectively InnovATE communicated its objectives and potential value to Missions. Three KIs rated InnovATE as highly effective and another three KIs rated InnovATE as somewhat effective in this area. One Mission informant stated that InnovATE was not effective in this respect and another reported low effectiveness. According to two KIs, the InnovATE implementer’s previous interactions in Senegal contributed significantly to success in connecting with Mission staff and sharing information about InnovATE’s objectives. (see Text Box 3 on the next page). One USAID/Senegal respondent suggested that InnovATE had certain positive “cascade effects” not accounted for in direct programming with USAID Missions. In Guinea-Bissau, an InnovATE presentation inspired the Mission to support more gender inclusive programs to strengthen AET. “A colleague of ours at a conference in Guinea-Bissau was very engaged with InnovATE— particularly with suggestions for future programming that he learned about at the conference. He saw their model of research as a good stepping stone to on-the-ground activities.” – USAID/Senegal. 17 Text Box 3 – Senegal Most Significant Change: Senegalese AET Institutions have historically lacked structure and pedagogical tools. Usually they conduct courses without syllabi and are not responsive to private sector demand for human capacity. Through eight years of work in Senegal, under a previous award that was also led by Virginia Tech, InnovATE's implementers built close relationships with USAID/Senegal and gathered information about AET needs. Hierarchy of Changes: The new five-year award will have two components designed to connect the AET system to the private sector 1) curriculum reform to enhance youth development and employment; and 2) a private sector extension to promote entrepreneurship. This extension will focus on food science and food safety to expand business opportunities. This award has potential to contribute to all three levels in the hierarchy of changes applicable to research programs. Sustainability of Changes: The Mission prioritized finalizing the previous award before committing funds to InnovATE, so the AA occurred only in the final year of the activity. It is too soon to assess the sustainability of the changes; however, it should be noted that the AA will be set up to include small grants to local universities. Grants will help implement a specific part of the work plan. Other grants to local research entities will help run a research program. Several interviewees from 2017 (n=6) and 2018 (n=5) suggested that this would be a good way to make InnovATE's activities more sustainable. In Nepal, the KIs said they had virtually no knowledge of the project and were not aware of any AET activities in the country. According to project documents, InnovATE conducted a teaching workshop at USAID/Nepal in 2015 and a scoping mission/AET assessment in 2013. At least one KI in Nepal was in country while these activities were conducted. A USAID/Nicaragua respondent said they were still conducting follow-on activities that resulted from studies InnovATE performed there, particularly studies that related to vocational education, which the Mission has used in conducting training and in designing programs. In Honduras, two Mission staff members had the impression that there was a logistical burden placed on the Mission to support the research team’s efforts in the field even though a local coordinator was hired by the project and, it was their belief, that this had a negative effect on InnovATE’s value to the Mission. InnovATE had no field presence and worked through local NGOs. At least one KI believed that the activity was extremely high maintenance because it required the Mission to support local researchers with local transport, hotels, communications, and other logistics. Nevertheless, the Mission was satisfied with InnovATE's overall results. One respondent noted: “InnovATE has provided the base to do some different programs here especially the study of how violence is connected to agriculture. The last study that was done here has been really good. We found a lot of information.” USAID/Honduras commissioned InnovATE to do a report to benefit the Garifuna community, "focusing on the current and future demand for agricultural education and training (AET) for rural youth . . . with particular attention to the Garífuna communities" of coastal Honduras. The study recommended, among other things, that the Mission should put an emphasis on primary and secondary education. The Mission understood from this (according to the Mission staff interviewee) that the community was not ready for a tertiary educational institution at that time (https://innovate.cired.vt.edu/wp-content/uploads/2015/08/InnovATE￾Honduras-Scoping-Assessment-Final-Report-June-2015.pdf ). This study was conducted by an InnovATE Scoping Team led by the University of Florida along with Pennsylvania State University, Virginia Tech, TU, and North Carolina State University in collaboration with BFS. 18 USAID/Armenia respondents stated that InnovATE contributed to necessary curriculum reform under the AA there, and had also added a food safety certificate program to the AET institution’s course offerings. These addressed the gap in meeting private sector demand and, by attracting students, contributed to the institution’s financial sustainability. Two respondents, however, suggested that the InnovATE team had a steep learning curve regarding the Armenian legal and institutional systems, and that they arrived in the country without first researching these aspects of the Armenian environment. Two KIs in Senegal described InnovATE’s scoping study and plans for the resulting AA very positively. They mentioned that the implementer’s previous work with the Senegal Mission contributed to its ability to conduct a study that produced valuable findings and recommendations. U.S. partner institutions made more positive comments about the project’s outputs (three positives versus two negative statements). However, their opinions on the educational aspects of the project (15 positive versus six negative statements) in addition to those of USAID/BFS representatives interviewed, were more negative (seven positives versus three negative). A number of informants interviewed in 2017 (n=7) pointed out that the project had not been as focused on student learning as it should have been and that its effects on curriculum design and other improvements in AET educational institutions had been negligible. Some Mission respondents in 2018, however, (n=3) noted that progressive education modules had been developed as a result of thematic studies. Scoping reports and papers were considered valuable and of high quality by some interviewees (n=3). Some partner institution respondents (n=5) said that substantial effort went into refining an analytic agenda for AET, and that the project completed pilot studies focusing on gender, AET in conflict areas, vocational training, and youth in agricultural education, all of which were valuable to Missions. Despite the fact that BFS helped facilitate phone calls with 15-20 missions over the course of the project and helped facilitate briefings about InnovATE by partners in the countries on other business, some respondents from those institutions (n=5) said that there was buy-in and some follow-up activities from Missions, but that other Missions expressed interest in the project and then moved on without following through. Several partner institution respondents (n=7) cited the distance between USAID and the project as an impediment—a distance partly due to geographical issues (i.e., their lack of on-the-ground presence) and partly due to InnovATE’s lack of knowledge about Mission priorities and Mission lack of awareness about what the project actually did. One respondent from a partner institution commented: “If I may add just one more thing, there is also a little bit of a firewall between the projects in the USAID Missions. We're not allowed to use these funds in this kind of projects to go and cultivate relationships to the mission, and yet somehow, we're supposed to magically help them understand the value proposition of a project like this in terms of what they are trying to achieve in that country.” CONCLUSIONS In general, Mission views on the value of the project were mixed and depended upon whether or not they were at a point in their programming where they could take advantage of, and follow through on, what InnovATE had to offer. In that respect, InnovATE did not seem to organize its research and design activities in a way that made efficient use of other Feed the Future projects or of Mission feedback. Communication between InnovATE and USAID Missions was hampered by timing, procedural issues, and by the different perceptions of what the project brought to the table. InnovATE’s attempts to communicate its objectives and potential value to Missions was largely unsuccessful—in spite of scoping assessments and other outreach measures that took place in more than 10 Missions. InnovATE attempts to communicate its objectives and potential value to Missions were unsuccessful, except in the case of a few Missions. Interviewees gave various reasons for this lack of success. Partner institution respondents 19 suggested that the program had a late start and did not reach Missions at the right time in their program cycle (which is refuted by the ME and BFS). Mission interviewees suggested that they were either confused about or completely unaware of the services that InnovATE had to offer. BFS interviewees (n=3) emphasize that, as long as they kept relevant BFS staff informed, universities were encouraged to communicate with Missions directly. To attest to this statement, these interviewees pointed out that a number of emails from InnovATE personnel were sent directly to Mission staff, with BFS cc'd. Both Mission and partner institution respondents perceived that InnovATE did not organize its research and design activities to make efficient use either of other Feed the Future projects or Mission feedback. Further, it appears that that some InnovATE partners didn't communicate well and were not effective spokespeople for the activity. RECOMMENDATIONS • Project implementers engaged in activities aimed at supporting Missions should take stock of where Feed the Future projects are in the implementation cycle so resources can be prioritized for Missions during the project design phase. • If BFS is interested in developing similar technical services concepts such as InnovATE, it should provide support for pilot studies in selected countries while ensuring that Missions and AET institutions of that country are actively engaged in AET institution building. • Local Mission staff (FSNs) are the institutional memory for most USAID Missions, so project implementors should give them priority for inclusion in scoping assessments, workshops, and other activities. • BFS should work to develop AET champions at USAID as another strategy to help alleviate the concern that AET often is not an active part of USAID programming, as pointed out by several key informants. • Project implementers should ensure that scoping assessments do not require Mission resources and should focus both on Mission priorities and assessing the needs of AET institutions. 4.4 EQ 4: HOW EFFECTIVELY DID INNOVATE ADDRESS MISSION DEMANDS RELATING TO AET CAPACITY BUILDING? How effectively did InnovATE align with Feed the Future (as well as other USAID strategies) (e.g., desire to support country AET capacity development needs and generate a body of knowledge on good practice in AET strengthening)? FINDINGS Of seven respondents who provided feedback on InnovATE’s effectiveness in aligning with Feed the Future strategy, five rated InnovATE as somewhat effective and two rated it as highly effective. These individuals were in Missions that had worked with InnovATE on specific studies that were tailored to their needs. Some Mission informants (n=3) felt that the project schedule did not initially align with Feed the Future Mission program cycles but believed that InnovATE later corrected this misalignment. In describing Armenia’s AA progress, two KIs indicated that, even though they made an effort, project implementers could not support the local AET institutions’ financial sustainability. Otherwise, representatives of Missions working with InnovATE (n=7) stated that the project implementers responded well to their demands. However, interviewees from partner institutions (n=10) expressed negative opinions about the project’s effects on AET capacity building. According to BFS interviewees and partner institution and Mission representatives, Mission demand for InnovATE’s products was not as great as originally expected. Based on project documents and interviews with project personnel, it appears that, overall, a critical assumption of the ToC was that Missions were ready to invest in building AET institutional capacity. While this proved to be correct for 20 some Missions, it was not true for all Feed the Future focus country Missions. The plan was that the initial research and design work would be supported by scoping missions and funded jointly from core funds and Mission buy-ins. These would be followed by Mission (or other office) investments in AAs to build sustainable AET institutions. Because the demand was less than anticipated, InnovATE’s effectiveness in supporting Feed the Future strategies of the Missions was limited. Seven interviews with KIs who agreed to provide feedback on whether InnovATE met Mission demands revealed that InnovATE was effective to a certain degree. Three said InnovATE was somewhat effective and three said it was highly effective. Representatives from USAID/Honduras and USAID/Nicaragua were in the latter group. As one Mission respondent said: “In terms of accessibility, InnovATE was very responsive to the Mission when it reached out to it for support to conduct the assessment on the subject of vocational education. It provided us with technical expertise at the level that was required.” In Nepal, no KIs responded to this question. Limited funding precluded requests to InnovATE, so they had little or no knowledge of the project. In Senegal and Honduras, AAs are in process. KIs at the IPs and Missions (n=8) are optimistic that InnovATE projects will be effective. In describing the Armenia AA’s progress, two KIs indicated that the Mission’s demand for assistance to increase the AET’s financial sustainability is not being met. A main objective under the AA was establishing an endowment, but there were difficulties in soliciting donations and, at the time of this evaluation, the endowment was at approximately $50,000. According to one KI, because InnovATE was unable to meet Mission demand and the target amount for the endowment under the AA, they tried other business models. This alternative included building fundraising capacity within the AET institution. However, it soon became clear that this skill is not easily transferable. According to the informant, InnovATE tried to encourage the institution to charge for research and other products, but the Armenian private sector showed little interest in paying. This informant believed that InnovATE was not on track to achieve the goal of AET’s financial sustainability. In response to the question of how well InnovATE aligned with Feed the Future, only seven KIs provided feedback. Five of them rated InnovATE as somewhat effective and two rated InnovATE as highly effective. According to feedback provided in KIIs with Mission staff in Senegal, Honduras, Armenia, and Tanzania, InnovATE aligned well with Mission strategies—both in relation to AET and to educational capacity building. In Nepal, where no AA was made or even considered, two KIs suggested that InnovATE support agricultural vocational training through the Council for Technical Education and Vocational Training (CCEVT). They explained that the primary need in Nepal is integrating agriculture and business training. Both AET faculty and graduates have limited business knowledge, which means small and medium-sized enterprises have weak financial marketing arms. This is a problem that could be solved through MBA programs associated with AET Institutions. KIs acknowledged that activities like InnovATE exemplify Feed the Future projects that address AET system gaps by placing capable graduates in private sector internships. Gap analysis of the AET system was last conducted in 2013. As noted, several KIs felt that the project schedule did not align with the Feed the Future Mission program cycles. They noted that the need for AET technical assistance during the first round of Feed the Future project formulation diminished by the time the project began. At that point, most Missions had finalized the design phase and moved on to obtaining approval and starting implementation. Whether or not this was the case, the InnovATE project later aligned its activities with some Mission program cycles. 21 CONCLUSIONS Since 2017, USAID Missions have made more demands for InnovATE research, but demand still falls short of the expected target. InnovATE reports indicate that the main factor for the shortfall was the absence of InnovATE staff on the ground in the project’s early years. Other evidence suggests that it was simply not a top priority of Missions, so staff on the ground might not have made a difference. Also, the program didn't communicate effectively in early years when it was under poor leadership. According to Mission informants that dealt with InnovATE (n=8), the project met their specific demands for AET training materials. Most are very satisfied with the outcomes. Respondents in Missions that worked with InnovATE (n=5) believed that the project aligned well both with their strategies and with USAID strategies. However, these respondents were not able to assess the project's impact on long-term capacity building. RECOMMENDATIONS • Future implementers of similar projects should ensure all assessments address major gaps and shortfalls of AET institutions and also address Mission priorities and strategies for building AET capacity. • Longer-term AET Institutional Capacity Development (ICD) programs should be developed to make a sustained impact on the quality of AET education and the number of qualified professionals trained. Programs should be specifically targeted to create sustainable institutions rather than contributing to time-limited programs that require Mission buy-ins. These should be based upon capacity building models and benchmarks used for other institutions with similar challenges. • Prior to instituting demand-driven projects for building capacity of host country institutions, implementers should conduct a thorough study of the policy-enabling environment in each country and the existing in-country projects with similar goals. This would allow implementers to determine the kind of support that would focus on key constraints not being addressed. 4.5 EQ5: WHAT WERE SOME OF THE IMPLEMENTATION CHALLENGES AND HOW DID THE PROJECT ADDRESS THEM? In what ways could implementation have been improved to more efficiently and effectively carry out scoping assessments? Disseminate results? Encourage project requests? FINDINGS One partner institution suggested that projects planned by USAID/Washington were more likely to create tensions between implementers and the Missions than projects the Missions themselves planned: “We kind of—as an implementing organization—are kind of caught in the middle of this weird USAID crossfire between Washington and the Missions. Which is not necessarily your—it’s not your fault, it’s not easy to navigate either. I think that's a huge issue—the lack of in-country consistent presence to build those relationships that really has—that’s a challenge.” There is no way of determining whether this was an issue that affected project success, but it was brought up by respondents from other partner institutions (n=4) and by the lead institution when they were interviewed in 2017. However, the design of the AAs, which are planned and procured by Missions, was intended to make the program country-focused. Findings from 2018 interviews revealed that some Mission respondents (n=3) believed the InnovATE team was not specifically aware of USAID Feed the Future activities taking place in countries they were proposing to work—even though this information was available to them before their visits. Further, those same Mission respondents (n=3) suggested that the InnovATE team was hampered by their lack of 22 specific country knowledge. Implementers noted that part of the problem was the broad geographical scope of InnovATE activities. Scoping missions and AET institutional assessments in 10 countries were useful, but the assessment teams were not able to spend adequate time in country. According to respondents, teams arrived in country without having researched Mission activities or the operational context of AET institutions. One respondent from a Mission that did use InnovATE shared the following: “It was slow in the beginning. Possibly having to gain an understanding of the work environment, management style, and overall systems was the culprit. Good, but slow.” The same KI later mentioned that InnovATE brought more people on board to facilitate and speed things along. Project implementers cited a key difference between InnovATE and Innovation Lab projects—InnovATE had no core funds for work at the host country level so their work (mostly on AAs) depended on Mission interest. In turn, Mission interest depended on factors like ongoing projects, the Country Development Cooperation Strategy (CDCS) cycle, and funding priorities. InnovATE was more focused on general AET strengthening solutions that were not country specific. The result was that InnovATE did not develop connections to local education institutions that would have allowed them to work on AET curricula. Research and design activities began to include more in-depth engagement of local stakeholders in the project’s final year. The increased depth addressed the issue of limited in-country collaboration (examples cited in four interviews included design work in Honduras and Tanzania). In 2017, both partner institution interviewees and two BFS interviewees stated that, from their perspective, the project's lack of impact on host country education and lack of Mission interaction was due to project design. CONCLUSIONS InnovATE's most prominent challenge was the lack of Mission demand and engagement. To some extent, this remained an issue through the final year of the project. As evidenced by 2018 findings, the relevance and frequency of contact with some Missions was increased that year. In those countries, project implementers had moderate success addressing identified AET priorities. The design of the project did not allow for an on-the-ground presence. InnovATE organized its activities through its home office in Virginia, and by occasionally sending temporary duty employees or consultants to select countries. InnovATE did not have broad connections to local educational institutions and stakeholders that would, for example, allow them to work on AET institutional strengthening or public-private partnerships (PPP). Because of lack of connections to local AET institutions (or to institutions that Missions were familiar with) and no institutional-strengthening PPP's to work with, they focused on meta-level analysis rather than on country-level design and on-the-ground implementation. RECOMMENDATIONS • Project implementers need to establish public outreach programs that focus on raising the awareness of Mission staff, other Feed the Future projects, and the community of AET institutions. • USAID/BFS should prioritize the development of local networks at country level. Networks could be composed of public officials, educators, private sector representatives, and other stakeholders to create needed basic platforms for cross-country collaborations. This would remove dependence on the efforts of U.S. based universities to establish lasting connections from short-term visits and remote activities. 23 ANNEXES 24 ANNEX 1: EXPRESSION OF INTEREST EXPRESSION OF INTEREST PEEL TASK ORDER I. BACKGROUND INFORMATION A) Identifying Information 1. Project/Activity Title: Innovation for Agricultural Training and Education (Innovate) 2. Award Number: AID-OAA-L-12-0004 3. Award Dates: 9/30/12 - 9/30/17 4. Project/Activity Funding: BFS and missions 5. Implementing Organization(s): Virginia Tech, Penn State, University of Florida, and Tuskegee University 6. Project/Activity COR/AOR: Clara Cohen B) Development Context 1. Problem or Opportunity Addressed by the Project/Activity Being Evaluated Outline the specific problem or opportunity the project was designed to address. Describe why the project was initiated. Identify any unique circumstances that prevailed in the context at the time the project was designed. Global challenges related to food security and climate change require new focus on building capacities to promote agricultural sector innovation and modification. Capacity is needed throughout the sector—on farms, in producer organizations, private firms, research institutions, government agencies, rural financial institutions, civil society, and other organizations. While many investments contribute to capacity, development of human capacity through training and education is central to this process and many Agricultural and Education Training (AET) institutions and programs have stagnated in recent years and failed to adapt to the changing needs in agriculture. Key problems in the AET space include: the inability of tertiary and workforce development programs to produce a workforce with the relevant skills to support country development goals and compete in global and national markets; poor admissions and recruitment systems; lack of university involvement in national agricultural innovation systems; and lack of aggressive outreach activities tying AET institutions to sector institutions and practical development needs. InnovATE is designed to provide broad support to agricultural development and food security initiatives, catalyzing development of programs and institutions to meet country needs. The Program targets the most neglected yet critical human and institutional building blocks required for modern and sustainable agricultural education, and training systems for food security at a country-level: a) investments in AET institutional development; b) short-term human capacity development to implement agriculture and rural development programs; c) training and leadership development for women in agriculture; d) training for agricultural technicians, managers, and scientists; and e) support for coordination of investments in tertiary agricultural education. The Program was designed to provide thought leadership and stimulate innovation in and awareness of the importance of AET systems. The key objective of the Project is to define and disseminate good practice strategies, approaches, and investments for establishing efficient, effective, and financially sustainable agricultural education and training institutions and systems, and to support country programs for AET reform and investment. The general Project approach builds on existing institutions, drawing on experience from current and past AET investments to: assess the quality and productivity of AET institutions and approaches; document 25 and draw lessons from such experiences; disseminate an understanding of the state-of-the-art with regard to effective AET systems; and integrate good practice recommendations into developing country AET systems. Developing public-private partnerships is likely to be an important aspect of some activities under the Project. 2. Target Areas and Groups Identify the target areas where the project was implemented, or the area or group it was attempting to affect. Specify each of the target populations. Include a map as applicable. Key partners include AET institutions in developing countries and their related stakeholder groups. These institutions span the full range of university degree and non-degree programs, diploma-level college/polytechnic schools, certificate-level training institutions, vocational/technical schools, secondary and primary schools with agricultural course content, adult education, and participant training programs. AET refers to the full range of such formal training programs but does not include extension and advisory services. Among the most important related stakeholder groups are the in-country employers of those trained by AET institutions and the degree to which their needs for skilled employees are being met. C) Intended Results of the Project/Activity Being Evaluated Describe the problem the intervention is aiming to solve. Is the problem well-conceived? Explain the development hypothesis, i.e., how specific intervention(s) are expected to lead to intended result(s). This can be in the form of a narrative “if-then” statement and include the results framework and logical framework. As applicable, describe the initial development hypothesis and any significant modifications. Are the development hypotheses valid? The InnovATE Project is designed to support agricultural education and training capacity development globally through three integrated components: LEARN, TRAIN, and DESIGN that identify good practice in AET investments, disseminate information on good practice and issues in AET, and influence program and project design relating to AET. These components are intended to contribute to the AET knowledge base and disseminate good practices, providing a foundation for capacity development for AET at all levels and linking the supply of trained agricultural professionals to and market demand for skilled employees and opportunities for youth and women. The Goal for the Project (originally conceived as Modernizing Agricultural Education and Training Systems (MAETS) Project) was to develop the human and institutional capacity necessary for developing countries to promote rural innovation needed to achieve sustainable food security, reduce poverty, conserve natural resources, and address other rural problems. This capacity must rely largely on in￾country AET programs and institutions to be cost-effective, sustainable, and relevant. The Project Objective was to define and disseminate good practice strategies, approaches, and investments for establishing efficient, effective and financially sustainable agricultural education and training institutions and systems, with the Project expected to support country programs for AET reform and investment. D) Approach and Implementation Describe the approach USAID adopted to transform project resources into results. This should be done both through a brief narrative, as well as a graphic depiction. Describe any modifications that have been made to a project’s design and budget, since inception. 26 LEARN: InnovATE provides intellectual leadership for AET by generating practical and policy-oriented scholarship for USAID and AET reformers in host countries. The project works to foster an active AET Community of Practice among USAID, reformers in host-countries, educational institutions at all levels, and professionals in international and regional organizations. A primary goal for the project is facilitating communication and engagement and providing a forum for agriculture educators. Public and private stakeholders include those working with youth or workforce development, training centers, private sector trainers, and primary, secondary, vocational, and higher education institutions. TRAIN: The TRAIN component is to widely disseminate good practice in AET investments and improve understanding of issues and challenges relating to AET programs and institutions. Tasks for the TRAIN component are driven by stakeholder demand and the results of the LEARN and DESIGN components. Good practices, analyses, and thematic studies are disseminated to USAID Missions and other donors, developing country policy makers, national/international AET professionals, and other interested parties through online training modules, a database of training activities, pedagogy workshops, and short courses. DESIGN: The end goal of the design phase is to help stakeholders make investments for education and training that support sustainable agricultural development. In this phase opportunities are identified, recommendations drafted, and implementation plans developed for strengthening specific AET institutions and systems by applying assessment tools that adapt good practices to provide project design guidance. The result is a set of analyses and recommendations that can lead to systemic capacity improvements tailored to fit the needs of the particular institution and agricultural sector. Cumulatively, these activities result in measurable impacts for improved AET systems and institutions that effectively serve value chains. Realization of those impacts feeds back on our learning and we adapt in a new cycle. E) Documents Clearly specify all the documents that you plan to utilize [e.g., project’s work plan, performance management plan (PMP), or M&E plan, etc.]. Identify any baseline data, quarterly or annual reports, or previous audits or evaluations you may use. Attach or provide links to documents or clarify when and how the ET will gain access to these documents. Also highlight any additional reference documents you plan to utilize (e.g., sector analyses or reports from host-country governments or other donors). The ET will review a wide variety of documents provided by BFS and the partner. Other documents may be requested during the fieldwork preparation phase, or while the ET is in the field. The primary documents and types of documents that will be provided are listed below. ∉ Technical Application proposal ∉ Annual work plans and budgets ∉ Semi-annual and annual progress reports ∉ Statements of Work (SOWs) and budgets for country-level activities ∉ PMP or M&E Plan ∉ Project Evaluation & Review Report(s) containing Scope, Schedule & Specification Deviations and Lessons Learned ∉ Program descriptions ∉ Other website materials and information ∉ Information on other major USAID AET investments 27 II. EVALUATION RATIONALE A) Evaluation Purpose The purpose of this external performance evaluation is to provide empirical evidence to respond to evaluation questions designed to support lessons learned and continuous improvement for BFS’ work in agricultural education and development. The evaluation will assess the overall rationale and strategy for the project, what is working well and what is not working well in implementation, assess progress toward outcomes, and provide information and recommendations that BFS can use to inform future programming, improve project effectiveness, and better achieve intended outcomes. B) Audience and Intended Uses BFS/ARP will use the results of this evaluation to make decisions about extending, terminating, or re￾competing activities. Missions may use the evaluation results to inform decision making regarding rural workforce development, youth, and agricultural education and training, and programming, including future program buy-ins. The results will also be used by the project to assess why some Missions chose not to follow up on scoping results with a request for implementation. C) Evaluation Questions 1. To what extent has InnovATE achieved its objective of defining and/or disseminating good practice strategies, approaches, and investments for establishing efficient, effective, and financially sustainable agricultural education and training institutions and systems? In what ways has the project generated, documented, and disseminated information on key constraints to AET systemic development, innovations to address those constraints, and engaged stakeholders and collaborators to contribute? In what ways did/did not the information generated address the specific needs of the Missions requesting and provide usable, actionable recommendations? How effective were project outputs in achieving outcomes in terms of Mission AET program investment, design, and operations? 2. In what ways have USAID Missions, donors, policy makers, and AET professionals used information generated from InnovATE’s good practice papers, analyses, and thematic studies, as well as the training modules and training courses (from both the TRAIN and LEARN components)? What opportunities are there for these papers, analyses, and studies to be used by the different stakeholder groups to enhance AET? 3. How effectively has InnovATE communicated its objectives and potential value to Missions? What did Missions find most valuable about InnovATE project outputs arising from scoping assessments and Associate Awards (as applicable)? 4. How effectively did InnovATE address Mission demands relating to AET capacity building? In what ways could InnovATE better align with Mission strategies relating to AET and educational capacity building? How effectively did InnovATE align with Feed the Future (as well as other USAID strategies) (e.g., desire to support country AET capacity development needs and generate a body of knowledge on good practice in AET strengthening)? 5. What were some of the implementation challenges and how did the project address them? In what ways could implementation have been improved to more efficiently and effectively carry out scoping assessments? Disseminate results? Encourage project requests? 28 III. EVALUATION DESIGN AND METHODOLOGY A) Evaluation Design The ET will finalize and elaborate the evaluation approach in the evaluation plan. However, a mixed methods or process evaluation approach to collecting and analyzing qualitative and quantitative data is suggested to arrive at robust findings and conclusions and fully address all the evaluation questions. Qualitative and quantitative data should be used to provide objective evidence to answer the evaluation questions. Qualitative questions will address the experiences, perceptions, opinions, motivations, and knowledge of the stakeholders interviewed, the context within which the activity is operating; and the factors (causal mechanisms) leading to observed results (or non-results). In-depth conversations guided by structured and semi-structured key informant interview instruments provide the opportunity for a flow of information that is not constrained by pre-determined response categories. This allows for exploration of subjects that surface during the interview and elicits information that provides nuance and insight for addressing the evaluation questions. Qualitative data will be synthesized and analyzed to provide robust evidence of findings. Sampling Parameters: The performance evaluation will cover the entire scope of the activity at the mission level; FTF focus country missions with educational strategies but that did not request a scoping assessment or a buy-in will also be sampled for interviews. At this time, site visits will likely not be necessary, however depending on the evaluation design, it could be considered. Key informant interviews and/or surveys should be carried out with stakeholders (via phone, Skype, Internet, etc.). B) Data Collection Methods Briefly suggest ideas on the design and methodology for data collection. Link the data collection methods to the specific evaluation questions the data will address. Qualitative data could be collected at a minimum through key informant interviews, focus group discussions, and/or site visits/observation. Quantitative data can include project performance monitoring data, survey data, and other secondary data. The ET will more fully elaborate data collection methods in the evaluation plan, which will be reviewed and approved by the Activity Manager. Key informant interviews: Key informant interviews (KIIs) could be used to collect information from stakeholders. KIIs will be employed in answering all evaluation questions. The ETs could use structured or semi-structured interview guides for KIIs to ensure key questions are systematically answered. These guides will also allow interviewers the freedom to ask spontaneous questions to uncover important and unanticipated information. The ET could design a separate key informant interview guide for each stakeholder group. Focus group discussions: FGDs can be carried out with key stakeholders, and direct and indirect beneficiaries. FGDs allow researchers to collect information about more people in a shorter amount of time and provide valid ways of identifying trends or conclusions with respect to counterfactuals or external factors. Focus group discussion sampling plan: Stakeholders should be identified in consultation with the Partnership’s COP and staff, taking into consideration the limitations on LOE and travel. As mentioned above, the sampling plan for selection of focus group participants will be included in the evaluation plan and in consultation with USAID and InnovATE to minimize potential bias. Surveys: Online surveys allow collection of information from a stakeholder group in a systematic way in a low-cost manner. Attention must be paid to sampling and response bias in analyzing survey result. 29 C) Data Analysis Methods Analysis will entail triangulating data from different sources to reduce bias and provide robust results. The methods for analyzing both quantitative and qualitative data will be elaborated more fully in the Evaluation plan, that will be reviewed, and approved by the Activity Manager. D) Methodological strengths and limitations The ET will elaborate methodological strengths and limitations in the evaluation plan, that will be reviewed and approved by the Activity Manager. IV. Timeline & Deliverables Timeframe for evaluation: July 2016 – November 2016 Deliverables: Evaluation Plan (at least 2 revisions, pending USAID approval) Mission Out Briefs – Short presentation of country-wide findings for USAID Mission for each country where site visits were conducted following both phases of data collection Presentation of findings: following the data collection phase, short presentation of initial findings/results for USAID and IP (utilizing web-based interface) Evaluation report (at least 2 revisions, pending USAID approval) following data collection phase, including evidence-based findings, and short-term and longer-term actionable recommendations, relating to the evaluation questions and activity implementation. All deliverables as specified in the PEEL-TO contract V. TEAM COMPOSITION Describe the intended size of the ET, the roles and responsibilities of team members, specific qualifications that team members are expected to possess, and the manner that their qualifications will contribute towards the evaluation effort. The evaluation envisions a team of two including an Evaluation Specialist and an Education Development Specialist. The team members need familiarity/experience in three areas: a) agricultural education and training principles and issues; b) agricultural development, workforce needs, and youth development; and c) quantitative and qualitative evaluation, organizational and capacity development, and gender analysis. A mix of resources is proposed below but the team composition and individual skill/experience mix may vary. Evaluation Team Lead: A senior-level evaluator with a minimum of 10 years of experience designing, managing and/or evaluating multifaceted international development teams, involving agriculture, education, and sector development. The candidate will also have: a) a demonstrated capacity to conduct independent program evaluation; b) an understanding of USAID’s foreign assistance goals, and its particular objectives related to agricultural development and food security; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Experience in sub-Saharan Africa, Latin America, and/or South Asia is necessary. Education Development Specialist: Must be experienced expert (5-10 years or more) in educational development related to agricultural sector institutional development and policy. Experience working in sub-Saharan African, South Asian, and/or Latin American agricultural education system contexts is recommended. Technical team member will also have demonstrated the following: 30 experience in effectively conducting outreach and dissemination to policy makers, development practitioners, and/or the private sector; and the ability to analyze issues and formulate concrete recommendations orally and in writing. VI. Suggested Level of Effort – LOE (per hour) Task/Deliverable Team Member Team Leader Conference Call/Desk Review 3 5 Evaluation Plan & Revisions 3 5 Data Collection & Travel4 15 15 Presentation of Findings 1 1 Draft Report 5 5 Revisions of Final Report 2 3 Total 29 34 4 The Evaluation Team is expected to work a six-day work week while traveling. 31 ANNEX 2: PHASE I (2017) EVALUATION PROTOCOL Feed the Future Evaluation Protocol for the Innovation for Agricultural Training and Education (InnovATE) Program Submitted: April 17, 2017 32 Number AID-OAA-L-12-0004 Submitted to: USAID/Feed the Future Contact: Clara Cohen Email: ccohen@usaid.gov Submitted by: Carl Erik Schou Larsen (Team Leader) Christopher Collins (Education Development Specialist) Contractor: David Hughes Mendez England & Associates Program Evaluation for Effectiveness & Learning (PEEL) 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 33 CONTENTS I. Program Information 1 a. Description of Activities Being Evaluated 1 b. Theories of Change (ToC) 2 II. Evaluation Approach 4 a. Evaluation Purpose 4 b. Evaluation Research Questions and Indicators 4 III. Methodology for Formative and Process Evaluation 12 a. Document Analysis 12 b. Quantitative Data Collection and Analysis 13 c. Qualitative Data Collection and Analysis 13 d. Methodological Strengths and Limitations 14 IV. Management Plan 16 a. Team Members and Roles and Responsibilities 16 b. Logistics and Supplies 16 d. Quality Control 17 e. Data Management & Security 18 f. Preparation of Datasets for Public Use 19 V. Ethical Considerations 19 a. Approvals 19 34 ACRONYMS Acronym Description AET Agriculture Education and Training ARP Agriculture Research and Policy BFS Bureau for Food Security CDCS Country Development Cooperation Strategy DDL USAID Data Development Library FTF Feed the Future GFSS Global Food Security Strategy HICD Human Institutional Capacity Development InnovATE Evaluation Protocol for the Innovation for Agricultural Training and Education IRB Institutional Review Board M&E Monitoring and Evaluation NAIP National Agricultural Investment Plan PEEL Program Evaluation for Effectiveness and Learning – USAID task order contract TAE Tertiary Agriculture Education ToC Theories of Change USAID United States Agency for International Development 35 1. Program Information a. Description of Activities Being Evaluated This is a performance evaluation (formative and summative) of the USAID/BFS/HICD Portfolio which consists of one long-term activity that contributes to Human Institutional Capacity Development (HICD). This portfolio is one of several of the overarching Feed the Future Initiative, sponsored by USAID/BFS. InnovATE is designed to provide broad support to agricultural development and food security initiatives, catalyzing the development of programs and institutions to meet country needs. The Program targets the most neglected yet critical human and institutional building blocks required for modern and sustainable agricultural education, and training systems for food security at a country-level: a) investments in Agricultural Education and Training (AET) institutional development; b) short-term human capacity development to implement agriculture and rural development programs; c) training and leadership development for women in agriculture; d) training for agricultural technicians, managers, and scientists; and e) support for coordination of investments in tertiary agricultural education. The key objective of the Project is to define and disseminate good practice strategies, approaches, and investments for establishing efficient, effective, and financially sustainable agricultural education and training institutions and systems, and to support country programs for AET reform and investment. The InnovATE Project supports agricultural education and training capacity development globally through three integrated components: LEARN, TRAIN, and DESIGN that identify good practice in AET investments, disseminate information on good practice and issues in AET, and influence program and project design relating to AET. These components are intended to contribute to the AET knowledge base and disseminate good practices, providing a foundation for capacity development for AET at all levels and linking the supply of trained agricultural professionals to, and market demand for, skilled employees and opportunities for youth and women. The Project objective is to define and disseminate good practice strategies, approaches, and investments for establishing efficient, effective, and financially sustainable agricultural education and training institutions and systems. The Project is also expected to support country programs for AET reform and investment. LEARN: InnovATE provides intellectual leadership for AET by generating practical and policy-oriented scholarship for USAID and AET reformers in host countries. The Project works to foster an active AET Community of Practice among USAID, reformers in host-countries, and educational institutions at all levels as well as professionals in international and regional organizations. A primary goal for the Project is facilitating communication and engagement and providing a forum for agriculture educators. TRAIN: The TRAIN component is to widely disseminate good practice in AET investments and improve understanding of issues and challenges relating to AET programs and institutions. Tasks for the TRAIN component are driven by stakeholder demand and the results of the LEARN and DESIGN components. Good practices, analyses, and thematic studies are disseminated to USAID Missions and other donors, developing country policy makers, national/international AET professionals, and other interested parties through online training modules, a database of training activities, pedagogy workshops, and short courses. DESIGN: The end goal of the design phase is to help stakeholders make investments for education and training that support sustainable agricultural development. In this component, opportunities are identified, recommendations drafted, and implementation plans developed for strengthening specific AET Institutions and systems, by applying assessment tools that adapt good practices to provide project design guidance. The result is a set of analyses and recommendations that can lead to systemic capacity improvements tailored to fit the needs of the particular institution and agricultural sector. Cumulatively, these activities result in measurable impacts for improved AET systems and institutions that effectively serve value chains. Realization of those impacts feeds back on learning that the Project can adapt in a new cycle. 36 The InnovATE setup being evaluated includes the Program Overview in Figure 1 on the next page. b. Theories of Change (ToC) The InnovATE program under review is designed to contribute to the achievement of the overarching Feed the Future goal, which is to “sustainably reduce global poverty and hunger.” InnovATE works to stimulate agricultural sector innovation and entrepreneurship, contributing to sustainable food security and poverty reduction. InnovATE works to support capacity development in AET throughout the developing world. Its model is guided by its ToC with integrated learn, design, and train components aimed to strengthen the full range of AET institutions that engage agricultural professionals— from primary and secondary institutions to vocational schools, technical colleges, and universities—in areas such as: curriculum review; pedagogy reform; faculty development; gender balance and equity; infrastructure; administration and management; outreach; student services; and educational policy. Figure I: Program Overview (designed by VTU InnovATE) II. Evaluation Approach a. Evaluation Purpose The purpose of this external mid-term performance evaluation is to provide empirical evidence to respond to evaluation questions designed to support lessons learned and continuous improvement for BFS’ work in agricultural education and development. The evaluation will: assess the overall rationale and strategy for the Project; determine what is working well and what is not working well in implementation; assess progress toward outcomes; and provide information and recommendations that BFS can use to inform future programming, improve project effectiveness, and better achieve intended outcomes. Thus, the evaluation 37 broken down into four aspects: (i) Efficiency or the ability of InnovATE to design and deliver products and implement communication strategies and implementation processes in a resources efficient manner. By efficiency we mean response time, use of workhours per assignment, and ability to perform or over perform within the given resource endowment. (ii) Relevance of InnovATE products and services to the specific needs and demands of beneficiaries and of USAID Missions. (iii) Effectiveness measured as the ability of InnovATE to support USAID Missions and beneficiaries in reaching their targets. (iv) The application of InnovATE products and approaches by U.S. Missions in a broader and longer￾term policy context, and the suite of InnovATE products and approaches to the AET institutions’ financial, human, and institutional resource endowment leading to enhanced sustainability of the program. Missions may use the evaluation results to conduct better-informed decision-making in programs with components of rural workforce development, youth, and agricultural education and training. The results can also be used by the InnovATE team to assess future direction and design through a better understanding why some Missions chose not to follow up on scoping results with a request for implementation. The audience and intended beneficiaries comprise the following: ∉ USAID and BFS as donors, particularly on cost-effectiveness and merits and demerits of integrated learn, design, and train model. ∉ U.S. Agriculture Research and Policy (ARP) on decisions about whether to extend, terminate, or re-compete activities and insight for future activity design. ∉ USAID Missions about whether or not the activities achieved their objectives and the extent to which they contributed to the overall goal. ∉ AET Institutions in partner countries, whether or not the activities, services, and products were appropriate and added value to their Mission and mandate. ∉ Developing partners’ use of the activities, services, and products in their institutional strengthening projects targeting AET institutions. b. Evaluation Research Questions and Indicators The aim of the InnovATE program is to develop the human and institutional capacity that will contribute to agricultural development through training and education of high-quality professionals at all levels. In this context and pursuant to USAID Feed the Future policy, this evaluation will investigate evaluation questions that are descriptive and normative in nature and focused on efficiency, relevance, effectiveness, and sustainability at the levels of Mission support and AET systemic impact, as well as recommendations on how these aspects would be improved in future implementation. Table 1 presents (i) evaluation questions, sub￾questions, and corresponding indicators, and (ii) evaluation questions, data sources, and methods and analysis, respectively. 38 Table 1: Evaluation Questions and Indicators Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews Efficiency of InnovATE’s communicate strategy - # and types of ways/activities used to communicate information to U.S. Missions/USAID HQ/project implementers - # and % of target group reached through different means of communication - Resource used to produce content and maintain communication platforms Semi-annual and annual progress reports Project evaluation and review report(s) Annual work plan and budget for 2017 Partner universities question 14 Interviews with key informants; specific feedback about communication efforts Efficiency in project coordination - # and types of coordination activities and resources allocated - # and types of joint activities and resources allocated - # and types of activities jointly with U.S. Missions/USAID HQ/project implementers/ third parties and resources allocated Semi-annual and annual progress reports Minutes of Consortium and board meetings Project evaluation and review report(s) Annual work plan and budget for 2017 Partner universities questions 17, 18 Interviews with partner university participants and other constituents Efficiency in addressing Mission needs - # and types of processes to address Mission needs and resources allocated to develop and implement these compared to the impact on Missions - Efficiency of the Learn-Design￾Training model as a tool to address Mission needs measures Semi-annual and annual progress reports Minutes of Consortium and board meetings Project evaluation and review report(s) Annual work plan and budget for 2017 Partner universities questions 11, 12, 15 Interviews with partner university and Mission representatives Are matching funds an efficient way to engage Missions in AET systems strengthening? - # and types of activities to utilize matching funds - U.S. Missions’ perception of matching funds as a way of facilitating their objective - Level that matching funds needed to trigger Missions’ interest in AET systems strengthening Financial records and indications of matching funds Partner universities question 20 Interviews with Mission representatives who can comment on the degree to which matching funds were compelling 39 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews What were some of the challenges in reaching or responding to the Mission need and how did InnovATE address these challenges? - # and types of challenges reported - # and types of ways that is used to address these challenges and resources allocated InnovATE Country Development Cooperation Strategy (CDCS) and Feed the Future Analysis [prepared by Virginia Tech] Mission/HQ survey questions 11, 12 Interviews with key informants who reported to work on the challenges In what ways could the process for conducting outreach to the Missions, carrying out the scoping assessments, and/or disseminating the results be improved? - # and types of recommendations for improving outreach and disseminating approaches N/A U.S. Mission/HQ: question 11 Partner universities question 16 Interviews with U.S. Mission/HQ and Consortium Partners Mission Support 2. Relevance: To what degree were outputs used as a guiding principle in design and implementation of activities and M&E? Indicator Literature Survey Interviews To what degree were design and implementation of new activities guided by experience from previous activities and outputs? - # and types guiding principles developed by InnovATE - # and types of M&E/feedback mechanism used in design and implementation of activities - Which outputs/experience had the biggest impact on future design? Guiding and core documents including reports and brochures Partner universities questions 7, 13 Interviews with university partners How important was the AET agenda within the Missions’ portfolio? - How high on the Mission agenda was AET? - Level of AET related activity within the Missions’ portfolio (% of project portfolio, % of annual budget envelope) - # and type of Mission staff involved in AET related activities (part time or full time) Project outcomes summaries in reports U.S. Mission/HQ: questions 7, 8 Interviews with Mission representatives To what degree did the relative importance of the AET agenda influence the interest in InnovATE information and activities? - # and types of anchor persons for AET within the Mission with whom InnovATE and BFS could liaise - # and types of interaction between InnovATE and the Mission N/A U.S. Mission/HQ: questions 7, 8 Interviews with Mission representatives and other relevant constituents 40 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews How relevant to Mission needs were the documents and information that InnovATE generated? - # and types of InnovATE products, services, and interventions used by the Mission - # and types of other AET products, services, and interventions used by the Mission - # and types of challenges faced by Mission in using InnovATE material to respond to Mission needs - Recommendations for addressing these challenges InnovATE documents and communication to Mission U.S. Mission/HQ: questions 16, 17 Interviews with university partners and Mission representatives How well did the information generated address the specific needs of, and provide actionable recommendations to, the requesting Missions? - Linked to the above indicators - # and ways that InnovATE has affected the way that Missions approach the AET agenda - # and ways that InnovATE has affected the attitude of Missions towards the AET agenda Possibly completion reports and meeting notes U.S. Mission/HQ: variety of questions and cross tabs Interviews with Mission representatives What did Missions find most valuable about InnovATE Project outputs (generated materials) and how did they use the materials? - Which InnovATE products/services were perceived as the most relevant and why? - Which InnovATE products/services were perceived as the least relevant and why? N/A U.S. Mission/HQ: variety of questions and cross tabs Interviews with Mission representatives Mission Support 3. Effectiveness: What was the mechanism put in place for measuring impact effectiveness? Was the effectiveness measurement mechanism used uniformly by all the implementing institutions? Indicator Literature Survey Interviews To what extent has InnovATE achieved its objective? - Degree to which InnovATE achieved its objectives - # and types of reasons explaining how InnovATE achieved this level of objectives Completion reports All surveys: matrix of questions on the achievement of objectives Interviews with all representatives and constituents 41 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews InnovATE’s institutional setup impact on effectiveness - Total # and types of products and services produced - % of products and services produced jointly by two or more Consortium members - % of products and services produced in collaboration with USAID Report components focused on design and partnership - Interviews with all representatives and constituents To what degree did the four institutions utilize each other’s comparative advantages to increase effectiveness? Linked to the indicators above - # and types of ways that assignments were delegated within the Consortium - # and types of ways that inter-Consortium collaboration was facilitated - Guiding principles for assignment allocation N/A Partner universities questions 17, 18 Interviews with university partnership and USAID Missions/HQ How effective were the InnovATE products and services in contributing to the U.S. Missions’ efforts to strengthen AET institutions and systems? - # and types of US AET activities/projects strengthened due to InnovATE products and services - Challenges in using InnovATE products and services to effectively assist AET institutions and systems - Suggestions for improvement of InnovATE products, services, and mode of operation Project evaluation and review report(s) Local governments, AET institutions, Private Sector and Development Partners survey (specifically question 10) Interviews with university partnership and USAID Missions/HQ In what ways have USAID Missions, donors, policy makers, and AET professionals used the information generated by InnovATE? - # and types of ways that InnovATE products are used by: o USAID Missions o Donors o Policymakers o AET professionals used Project evaluation and review report(s) U.S. Mission/HQ and Local governments, AET institutions, Private Sector and Development Partners survey Interviews with USAID Mission, partners, and professionals in the field Mission Support 4. Sustainability: How applicable and robust is information generated for U.S. Missions in a broader and longer-term policy context? Indicator Literature Survey Interviews 42 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews What opportunities exist for using the information generated by InnovATE for enhanced investment in AET? - # and types of opportunities to use InnovATE products and services for enhanced investment in AET by: o USAID/BFS/Missions o Donors o Policymakers o AET professionals - Recommendations for making use of these opportunities Recommendations given in internal and external reports - Interviews with university partners, USAID Missions, and auxiliary partners With a view toward sustainability, how well have the design recommendations for AET strengthening been tailored to context-specific differences and demands? - Perceived ease of using InnovATE products and services among: o USAID/BFS/Missions o Donors o Policymakers o AET professionals - # of type of activities/project where the InnovATE products and services were directly usable - # of type of activities/project where the InnovATE products and services were somewhat usable - # of type of activities/project where the InnovATE products and services were not usable - Recommendations on how to increase the usability of and effectiveness of InnovATE products and services Recommendations given in internal and external reports Local governments, AET institutions survey question 23, Project implementers survey question 24, Partner universities question 25, Mission/HQ question 16 Interviews with all key informants 43 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews How useful are the InnovATE findings and approaches for BFS’s forward-looking strategy in context of the Global Food Security Strategy (GFSS) and for achieving CDCS objectives? - Especially related to: Increased Youth Empowerment and Livelihoods - Improved Human, Organizational, and System Performance - Level of genericity/specificity measured in % of InnovATE products - Level of relevance measured in % of InnovATE finding and approaches to: o U.S. Missions o BFS o AET institutions - # and types of ways that InnovATE findings support linkages to GFSS - # and types of ways that InnovATE findings support different CDCS - # and types of ways that InnovATE products and investments are expected to be utilized in the future - Recommendations on how to promote AET in the future work portfolio of the Bureau Project evaluation and review report(s) Survey questions on specificity. Local governments, AET institutions survey question 21. Project implementers question 22. USAID Mission/HQ question 21 Interviews with all key informants AET Systemic Support 1. Efficiency: To what degree was the issue of resource efficiency addressed and included in the recommendations of best practice and new approaches? Indicator Literature Survey Interviews Efficiency in project coordination platforms - # and types of coordination of activities and resources allocated - # and types of joint activities and resources allocated - # and types of activities jointly with U.S. Missions/USAID HQ/project implementers/ third parties and resources allocated Reports documenting joint activities Local governments, AET institutions survey questions 10, 11, 12 Interviews with constituents who experienced the products related to InnovATE To what degree did the four institutions utilize each other’s strength to increase effectiveness? - Percentages of joint assignment (two or more universities) out of total number of assignments N/A Partner universities questions 17, 18 Interviews with university partners 44 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews Efficiency in addressing AET institutions and systems needs - # and types of processes to directly support AET institutions and systems as well as resources allocated to develop and implement these compared to the impact measured - Efficiency of the Learn-Design￾Train model as a tool to address AET institutions and systems needs Project evaluation and review report(s) Local governments, AET institutions survey questions 10, 11, 12 Interviews with Constituents who experienced the products related to InnovATE AET Systemic Support 2. Relevance: To what degree were activities designed to support recipients’ (beneficiaries’) specific needs and circumstances? Indicator Literature Survey Interviews How relevant to AET institutions were the documents and information that InnovATE generated? - # and types of InnovATE products, services, and interventions used by AET institutions - # and types of other AET products, services, and interventions used by AET institutions - # and types of challenges faced by AET institutions in using InnovATE material - Recommendations for addressing these challenges All materials distributed to AET institutions Local governments, AET institutions survey questions 8, 20, 21 Interviews with constituents who experienced the products related to InnovATE How able was InnovATE's work in brokering partnerships between the AET institutions and the private sector? - # of AET institution interested in liaising with private sector - # and types of ways that InnovATE’s facilitated brokering partnerships between AET institutions and the private sector N/A - Interviews with InnovATE constituents AET Systemic Support 3. Effectiveness: What was the mechanism put in place for measuring impact effectiveness? Was the effectiveness measurement mechanism used uniformly by all the implementing institutions? Indicator Literature Questionnaire Interviews 45 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews To what extent has InnovATE achieved its objective towards AET institutions and systems? - Degree to which InnovATE achieved its objectives - # and types of reasons explaining why InnovATE achieved this level of objectives Review stated objectives and results in reports Project implementers survey questions 25, 26, Local governments, AET institutions survey questions 24, 25 Interviews with InnovATE constituents How effective were the InnovATE products and services in strengthening AET institutions and systems? - # and types of AET institutions and systems that were strengthened due to InnovATE products and services - Challenges is using InnovATE products and services to effectively assist AET institutions and systems - Suggestions for improvement of InnovATE products, services, and mode of operation Project evaluation and review report(s) Local governments, AET institutions survey questions 14, 15, 17, 22; Project implementers survey questions 10, 11, 12, 15 Interviews with InnovATE constituents To what degree did InnovATE thought leadership support or complement AET transformation efforts at the regional level in sub-Saharan Africa? - Degree that InnovATE achieved its objectives - # and types of ways by which InnovATE thought leadership support or complement AET transformation efforts at the regional level in sub-Saharan Africa - Recommendations for improving InnovATE’s support and complementarity Project evaluation and review report(s) Local governments; AET institutions survey questions 14, 15, 17, 22; Project implementers survey questions 10, 11, 12, 15 Interviews with InnovATE constituents; Focus on probing the congruency between the program and the region AET Systemic Support 4. Sustainability: To what degree were the AET institutions’ financial, human, and institutional resource endowment (ability to implement) taken into account when developing recommendations and best practices? Indicator Literature Survey Interviews To what extent did InnovATE catalyze sustainable business models for AET strengthening (i.e., through PPPs)? - Extent that InnovATE catalyzed sustainable business models for AET strengthening - # and ways that InnovATE catalyzed sustainable business models for AET strengthening - Recommendation for improvement of the catalysis of sustainable business models Reports describing InnovATE model Mission/HQ survey question 25 All key informants 46 Mission Support 1. Efficiency: To what degree were communication strategies and implementation processes designed and refined from a resources efficiency point of view? - Indicator Literature Survey Interviews To what degree are InnovATE products and services useful for target countries’ AET strategies and National Agricultural Investment Plan (NAIP)? - # and ways that that target countries’ AET strategies and National Agricultural Investment Plan (NAIP) can use InnovATE products and services InnovATE design components related to NAIP Local governments, AET institutions survey question 15; Project implementers question 16; USAID Mission/HQ question 18 All key informants with knowledge of NAIP III. Methodology for Outcome and Process Evaluation In an effort to explore a large, multi-million-dollar grant project with sites all over the world, we will use a broad, multiple methods, case study design and approach the complex program with specific contextual parameters (Flyvbjerg, 2013; Yin, 2003; Creswell, 2003). The boundaries of the case will be the core institutions that were awarded a grant, USAID Missions and central office, and agricultural and educational training organizations that became connected to the gran through various means. The value of a case study lies in the detail, richness, completeness, and variance of data collected (Flyvbjerg, 2013). These benefits equate to depth of context and relation to the larger environment. Although this evaluation lacks prolonged engagement, the variance of data collection in the case will provide additional insights that align with the complexity-aware discussion note produced by USAID (2016). Case study design typically includes the use of many sources of data that fit within specified boundaries (i.e., the InnovATE program). Using multiple methods to understand the case will include basic survey research, document analysis, and qualitative inquiry through interviews with key informants. The research will begin with project documents, work plans, monitoring and evaluation plans, monitoring reports, previous audits and evaluation reports, sector reports from host-country governments and donors, and relevant policy documents. The document review will be conducted first as a basis for developing the questionnaire survey, and in-depth interview guides. Primary data will be collected using the survey and in-depth semi-structured interviews. The survey is based on the evaluation questions and the reflections on secondary information. For the purposes of data entry and analysis, the questionnaire will be distributed in an online survey format. The sampling approach will be targeted and therefore not intended to be random or generalizable. Data from surveys will be triangulated with secondary data and interview findings. Based on initial findings from the survey and from document reviews, interview guidelines will be verified by the results of the survey and adapted for different issues and different groups of key informants to ensure key questions are systematically answered. These guides will also allow interviewers the freedom to ask spontaneous questions to uncover important and unanticipated information. Key informants will be identified based on secondary literature, recommendations from InnovATE and BFS as well as issues emerging from analyzing the quantitative data that need to be further investigated/triangulated/double-checked. The depth of information in this evaluation will primarily be generated during the interview phase. Furthermore, the case study design explained here is preliminary and exploratory. While evaluating a case, new information and participants emerge and the design is arranged to be flexible and responsive. As a result, a high level of detail explaining the case and the methods typically emerges after data collection as opposed to before. What follows is a preliminary design. a. Document Analysis 47 Discourse and document analysis will be used to help provide an in-depth perspective on organizational processes, implementation, effectiveness, and efficiency. The documents to be reviewed for the purpose of this evaluation include, but are not limited to: ∉ Technical Application proposal ∉ Annual work plans and budgets ∉ Semi-annual and annual progress reports ∉ SOWs and budgets for country-level activities ∉ PMP or M&E Plan ∉ Project Evaluation & Review Report(s) containing Scope, Schedule & Specification Deviations, and Lessons Learned ∉ Program descriptions ∉ Other website materials and information ∉ Information on other major USAID AET investments. b. Quantitative Data Collection and Analysis The quantitative data will be collected through the questionnaire survey on the web-based platform, Survey Monkey. Surveys for targeted groups have been developed (see Annexes II, III, IV, and V) and will only be distributed to people presumed to have had contact with the InnovATE program. Lists are being generated by gatekeepers in the InnovATE program and from USAID to assist in distribution and recruitment. We estimate that each survey will be distributed to 30-40 people. There is no intent obtain a random sample of constituents having involvement with the program. Instead, the purpose of the quantitative phase is to generate some broad notions about the perception of InnovATE among key groups. An Institutional Review Board will review the materials and methods for human subjects’ protection. The data will be analyzed using simple descriptive statistical procedures in Excel or SPSS to determine general perceptions of the Project at the U.S. Missions and at country level, and to highlight areas requiring more in-depth examination. The analytic plan is to cross-tabulate responses and summarize descriptive frequencies to generate a broad understanding of the program and its impact among various constituent groups. Within the multiple methods case study design, the purpose of this quantitative phase is to provide a very general overview and basic data trends of respondents and the qualitative phase will further examine those trends through in￾depth interviews. In this way, the purpose of the survey is to provide very basic points of reference with the intent of having in depth interviews (Creswell, 2003). c. Qualitative Data Collection and Analysis Building upon the survey data, the primary source of qualitative data collection will be through semi￾structured interviews. The details of an interview protocol will come to fruition following a review of the quantitative data. Annexes VI, VII, VIII, and IX show a sample of what the protocol will look like. The review protocol will be piloted, and probes will be developed to enhance the potential lines of questioning based on the expertise of the participant. Participants will be recruited from each constituency group and interviews will be conducted either by telephone or in person. We expect that out of each constituency group we will interview 6-10 key informants to equal a total of around 30 interviews. As interviews begin, we will use each participant as an entry-level referent to identify other key informants who can provide pertinent information. This method of sampling is often called chain referral or snowball sampling (Creswell, 2003). With permission, interviews will be digitally recorded and then transcribed verbatim using CabbageTree Solutions, which charges around $30/recorded hour. If participants do not consent to recording, then copious notes will be taken during the interviews and then expanded into a written summary of the interview. Participants will be given a chance to review their transcript and will be contacted through email. This method of respondent validation will help to ensure the trustworthiness of the data. Data collected through various qualitative methods will be analyzed through content analysis, data reduction, coding, and categorizing into themes as well as interpretation and presentation based on evaluation questions and objectives. Data analysis will include disaggregating by demographics, constituency groups, and other variables that might emerge. Coding will occur in through multiple rounds of sorting and 48 analyzing in search of trends and themes that are either frequent or salient within the body of data. These analytic processes will address the triangulation and valuation of data interpretation and finding presentation through cross-checking between different data sources and types. d. Methodological Strengths and Limitations The methodological strengths of this evaluation lie in the multiple methods approach. The document analysis and survey data will provide background information and a general overview of what may be trends in perceptions of the Project. The qualitative phase will provide more in-depth information. Drawing from Lincoln and Guba (1985), we will use multiple approaches to triangulation. Triangulation involves the use of multiple sources to evaluate a case. Within the parameters of the case, we will draw from different constituencies, using a variety of methods, and examine how the Project evolved over various points in time. Several methodological limitations might affect this evaluation. First, there are high risks and uncertainties when collecting quantitative data with online format. The questionnaire return rate is impossible to control, which might influence the richness of the collected data and generalization of findings. At present, to cover all the issues that need to be addressed, the questionnaires are quite comprehensive. There is a risk that the length of the questionnaires, in their present form, will have a negative influence on the response rate. It might, therefore, be necessary to cut down on the number of questions and, thereby, the depth of the information to secure enough responses. We will work with each of the four constituencies to develop reminders to take the survey to enhance the response rate. InnovATE operates in Anglophone, Francophone, Lusophone, and Spanish-speaking countries, while Missions are all English-speaking. We anticipate that everyone taking the surveys will be English-speaking, but, if they are not, we will have to examine the volume of persons that might benefit from a survey or an interview in a different language and develop those materials and opportunities accordingly. In general, the case study design and the multiple methods being used are reflective of several layers of bias. The survey is not a psychometrically validated instrument and will not be given to a random sample of participants. These generate biases, which prevent the ability to generalize the results of the survey to the larger InnovATE constituencies. As a result, this aspect of the design is categorized as general frequencies and trends among those who are willing to take the survey. The lack of in-depth information given that there will be no inferential statistical analysis is supplemented through the multiple methods during the qualitative phase. Interviews also include many layers of bias given the role of the researcher conducting the interview and the willingness of the participant to disclose. In order to enhance the trustworthiness of the study, both researchers will be conducting and analyzing interviews, although there will not be any formal inter-rater reliability measured. The interviews are semi-structured and will follow probes based on the perception and expertise of the participant. To further enhance the trustworthiness, participants will be given transcripts of the interviews to check for accuracy and to be given a chance to review what they said. Another limitation is that the interviews will occur over the phone and not in person, due to budget limitations. In terms of sampling, the case design limits the types of participants. Furthermore, response rates and availability of participants make the sampling approach a convenience sample and a chain referral sample. The findings from the study will be a reflection of those that were able and willing to participate. Creswell, J.W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: Sage. Flyvbjerg, B. (2011). Case study. In N.K. Denzin & Y.S. Lincoln (Eds.), The Sage handbook of qualitative research. Thousand Oaks, CA: Sage. Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage. USAID (2016). Discussion note: Complexity-aware monitoring. Washington DC: USAID. Retrieved 49 April 14, 2017 from: https://usaidlearninglab.org/library/complexity-­‐aware-­‐monitoring-­‐ discussion-­‐ note-brief Yin, R. (2003). Case study research: Design and methods. 3rd ed. Thousand Oaks, CA: Sage. IV. Management Plan a. Team Members and Roles and Responsibilities The ET is comprised of two senior members: an evaluation specialist (who serves as the team lead) and a senior educational development specialist. The team, their roles, and their qualifications are articulated below. Dr. Carl Erik Schou Larsen is a senior-level evaluation expert he has more than 20 years of experience working with research-based education and outreach within life sciences in Africa and Southeast Asia. For the past 10 years, Dr. Larsen’s work has focused on reforming Tertiary Agriculture Education (TAE) in Africa through institutional capacity building projects aimed at enhancing performance within science-based education. He has firsthand experience as a project leader in a university setting where he implemented EU￾and DANIDA-funded institutional capacity building projects. He also has experience as a donor representative responsible for oversight and control of TAE institutional capacity building projects with the World Bank. Dr. Carl Larsen has a doctoral degree in Tropical Animal Husbandry, The Royal Veterinary and Agricultural University (now University of Copenhagen), Denmark. He also has a Master of Public Administration from Copenhagen Business School, Denmark. Dr. Christopher Collins is a Higher Education Specialist and Researcher who has taught courses on qualitative research methods, higher education law, global policy, curriculum development, globalization, and program evaluation. He has written extensively in the field of education particularly in areas including: higher education; access; education strategy in the developing world; the impact of higher education on poverty; development of higher education in Africa; education policy; and extension. This included a study in Rwanda of the impact of higher education on coffee extension in Rwanda and the positive impact of this expertise on the coffee sector. Dr. Chris Collins has a doctoral degree in Higher Education and Organizational Change, University of California, Los Angeles. Table 2: Team Members and Roles Team Member Primary Activities Additional Activities Carl Larsen Administration of the evaluation; design of the evaluation; interact with key stakeholders; ensure adherence to timelines and deliverables Point of contact for USAID and U.S. in-country contacts Chris Collins Serve as expert on education research methods and assist with the design of the evaluation; interviews of key stakeholders; data analysis Point of contact for U.S. in￾country contacts b. Logistics and Supplies This is, in principle, a desk study. However, the ET request that a field trip to the InnovATE Consortium members, at Virginia Tech in Blacksburg, Virginia be included in the budget. Such a visit has been requested by the InnovATE team to enhance the quality of the final report. The ET consider it essential that preliminary findings are discussed and validated through a face-to-face dialogue with the implementation team. Such a meeting is also believed to enhance the implementation teams’ ability to use the evaluation findings in future design of InnovATE and other USAID/Feed the Future projects/activities, partly due to a deeper and richer discussion and understanding of the finding and partly due to a stronger acceptance of the findings as the InnovATE team have been personally consulted in the evaluation process before publication. 50 Such a field trip, if approved, will require the LOE to be modified. Table 3 below is the ET’s estimate of minimum time required to travel from LA and Denmark to Virginia Tech, including a two-day workshop with the team there. It will also require two economy fare tickets (LA/Washington/Virginia and Copenhagen/Washington/Virginia), as well as accommodation and per diem for a minimum of three days. Table 3: Suggested LOE Revision Tasks Team Lead (LOE days) Team Member (LOE days) Desk Review/Concept Note 5 3 Evaluation Plan & Revisions (3+2)= 5 (3+2)= 5 Survey and Analysis Work 10 9 Field Visit to Virginia Tech 4 4 Draft Report 6 5 Final Report 3 2 Total LOE (27+6)= 33 (22+6)= 28 Included in the suggested LOE revision is also an upgrade of the days for protocol preparation and revision. The amount of work requested before being ready to launch the actual evaluation work was clearly underestimated in the original LOE. The ET is also requesting a revision to the schedule of delivery. Table 4 below is a suggestion for an update of the approved table from the Concept Paper. Table 4: Suggested Revised Schedule of Delivery Deliverables Dates Concept Note Approved March 22, 2017 Evaluation Protocol and Data Collection Tools Approved April 20, 2017 IRB Approval April 24, 2017 Draft Report and Mission Out briefs May 26, 2017 Field Visit to Virginia Tech May 30 – 31, 2017 (Tentative) Final Report June 20, 2017 c. Quality Control As discussed in section III above, the evaluation will primarily apply the following methodological components: ∉ Review and summary of secondary data through reports and data provided by the implementers. ∉ An online survey. ∉ Interviews (direct and by phone). These approaches will be applied by the ET members. The team will put the following data collection and analysis control measures in place for each of the above instruments. See Table 5 below for details. Table 5: Data Collection and Analysis Control Measures 51 Methodology/Instrument Data Collection and Analysis Control Measures Secondary Data Review and Summary (includes cost data) – Team members will take the lead in identifying sources of secondary data from documentation made available to the Team. – Team members themselves will review and summarize secondary data. Data summaries will be done in English and typed up. – Team members will apply the data contained in the summaries for analysis. Interviews – Team members will take the lead in development of interview. – Interview protocols will be developed by two researchers and then piloted. – Interviews will be done in English by team members, recorded, and transcribed by a professional service or summarized in notes. If a participant does not consent to the recording of the interview, Team members will take notes verbatim as the interview is conducted. Team members will apply the data contained in the notes for analysis. – Transcriptions will be sent to participants to review for accuracy and intended meaning. Online Survey – Distribution of the surveys will take place through gatekeepers in each constituency. – Reminders will be sent to potential participants at multiple time points. d. Data Management and Security Data security is primarily a concern for personal information acquired from fellows who are interviewed by the ET or provide feedback in the form of questionnaires and online, self-managed surveys. For online surveys, the data security features of the selected online survey service provider will be followed. Online survey providers ensure that information is encrypted, thwarting any cyber-attacks that may occur on data that may expose confidential information. For direct interviews and emailed questionnaires, person-specific data will be managed by the responsible team member following generally adopted standards for data security, who will also take the overall responsibility for any data handling by the RA. The team leader will maintain the team depository for person-specific data following data analysis by the responsible team member. Likewise, data sourced from secondary data sources and interviews and questionnaires administered to institutional representatives will be managed by the responsible team member following generally adopted standards for data security. This will be accomplished through Dropbox whose security is encrypted and only limited to those who have been allowed to share in the information. Again, the team leader will maintain the team depository for collected data following data analysis. This depository will be handed over to USAID at the end of the evaluation. Prior to dissemination to programs, institutions, and USAID, all data will only be shared between the ET. Post-dissemination, the data will be shared with interested parties, sans identifying information (e.g., names, serials numbers, etc.). e. Preparation of Datasets for Public Use Both quantitative and qualitative datasets will be developed and released in a manner that ensures the anonymity of the participants and stakeholders. Any identifying information will be replaced either by 52 random numeric and/or reported in aggregate. The team will collect and ready data for public consumption as it is collected from the in-country sites. Each institution will receive a dataset specific to their context and each program will receive a dataset of their specific data at the end of the Project as well as the final report. USAID will retain access to all real-time databases (data available immediately after collection), both qualitative and quantitative. These databases will be submitted to the USAID Data Development Library for access by the general public. Below is the timeline of submission to sites and to USAID’s DDL. V. Ethical Considerations a. Approvals Prior to field data collection, the ET will submit the evaluation study protocol and instruments for ethical review by the Chesapeake Institutional Review Board (IRB). The application to the IRB will be for exempt status given that the nature of the data collection is for an evaluation as opposed to generalizable research. The purpose of a review and designation of exempt status should provide a layer of ethical evaluation to ensure the protection of participants in the evaluation study. The response from the IRB will be shared with USAID via email upon receipt. 53 Time Schedule 54 Interview Guideline 1 – U.S. Partner Universities [Interviews are intended to be semi-structured and open-ended. The questions below serve as a general guideline with the intention of leaving room for probing questions and focusing an interview on a participant’s interest or area of expertise.] Thank you for agreeing to participate. The results from this evaluation will be written to keep your identity anonymous, meaning that none of your responses will be attached to personally identifying information. 1. General personal information Name: Position: Organization: Interview date: 2. AET Systemic Impact 1. Efficiency 1. How did you/InnovATE communicate InnovATE objectives and potential value with: (i) Mission; (ii) USAID HQ/BFS; and (iii) InnovATE project implementers? a. What could be your recommendations to improve the current communication? 2. How did you/InnovATE collaborate with: (i) Mission; (ii) USAID HQ/BFS; and (iii) InnovATE project implementers? a. What could be your recommendations to improve the current collaboration? 3. How did U.S. partner universities collaborate with each other in implementing InnovATE activities? a. What could be your recommendations to improve the current collaboration? 4. What implementing processes/approaches were in place for InnovATE to address the U.S. Mission needs and demands relating to AET systems strengthening? a. How efficient were these implementing processes/approaches b. What could be your recommendations to improve these processes? 5. How did BFS perform the role as a broker for relationships between InnovATE and the Missions? a. What could be your recommendations to improve BFS broker role? 6. In what ways have InnovATE’s matching funds efficiently engaged: (i) the U.S. Missions, and (ii) AET project implementer in AET systems strengthening? a. What could be your recommendations for improvement? 7. What were some of the challenges in reaching or responding to the Mission’s need? a. What could be your recommendations for addressing these challenges? 8. How did InnovATE conduct outreach to the Missions, carry out the scoping assessments, and/or disseminate the results a. What could be your recommendations for improvement? 3. AET Systemic Impact 2. Relevance: 1. What are relevance of InnovATE products to: a. BFS’s forward-looking strategy in context of the GFSS? b. Local AET institutions? c. Target countries’ forward-looking AET strategy? 2. What could be your recommendations to improve the relevance of these products? 3. Can you give specific examples that show that the information generated by InnovATE address 55 very well the specific needs of, and provide actionable recommendations to, the requesting Missions? 4. What opportunities exist for U.S. partner universities to use the InnovATE results/products for enhanced investment in AET? 4. AET Systemic Impact 3. Effectiveness: 1. What is your assessment about effectiveness of InnovATE processes of defining and/or disseminating products and approaches? a. How can the effectiveness be improved for the better outreach and dissemination? 2. What is your assessment about effectiveness of InnovATE products, approaches, and investments to AET systems strengthening? a. How the effectiveness can be improved? 3. To what degree was the issue of resource effectiveness addressed and included in the InnovATE products, approaches, and investments? 4. To what degree did the four institutions utilize each other’s strengths to increase the effectiveness? 5. In which ways did InnovATE thought leadership support or complement AET transformation efforts at the regional level in sub-Saharan Africa? a. What are your recommendations to improve the leadership support and AET transformation efforts? 6. What were the mechanisms put in place for measuring InnovATE impact? a. What is the effectiveness of the mechanisms? b. How can the effectiveness of the mechanisms be improved? 5. AET Systemic Impact 4. Sustainability: 1. What opportunities exist for AET professionals to use the InnovATE results/products for enhanced investment in AET? a. How can U.S. Missions/BFS use these opportunities? 2. To what extent did InnovATE catalyze sustainable business models for AET strengthening (i.e., through PPPs)? How? 3. How can all InnovATE products and investments best be utilized in the future by U.S. partner universities? a. Are there ways to enhance the usability of the InnovATE and investment? 4. How easy is it to adapt the InnovATE products to country-specific context/the CDCS objectives? 5. How easy it to convert the InnovATE products into specific project proposals and work plans? 56 Interview Guideline 2 – Missions, USAID HQ (Interviews are intended to be semi-structured and open-ended. The questions below serve as a general guideline with the intention of leaving room for probing questions and to focus an interview on a participant’s interest or area of expertise.) Thank you for agreeing to participate. The results from this evaluation will be written to keep your identity anonymous, meaning that none of your responses will be attached to personally identifying information. 1. General personal information Name: Position: Organization: Interview date: 2. Mission Support 1. Efficiency 1. How did you find InnovATE’s ability to communicate its objectives and potential value? 2. How did you find communication with InnovATE to be? 3. How did you find collaborating with InnovATE? 4. (Mission) What role did the USAID HQ/BFS play, if any, in your collaboration with InnovATE? 5. (USAID HQ/BFS) In what way, if any, did you facilitate InnovATE’s service provision to the Mission(s)? 3. Mission Support 2. Relevance 1. How important was the AET agenda within your Mission’s portfolio? 2. How did the importance of the AET agenda influence your collaboration with InnovATE? 3. How was the fit between your needs in that regard and the services InnovATE offered? a. Did it develop over time? How? b. Do you have suggestions for improvements? 4. Are you aware of the concept of matching funds and did you use this mechanism? a. If so, what is your experience with this mechanism? 5. Compared to other initiatives aimed at AET systems strengthening, how would you assess InnovATE? 6. Which AET systems strengthening initiative do you think has most impact? Why? 7. In what ways were innovate products relevant to U.S. Mission/USAID HQ/BFS? a. Which of InnovATE’s services and products did you find most relevant? Why? b. Did the webinars, blogs, and other online services improve your work performance? How? 4. Mission Support 3. Effectiveness: 1. Describe the relevance of InnovATE to your Mission/Agency? 2. Could you elaborate on how you have used the outputs and services from InnovATE? 3. Do you know of other partners that have used outputs and services from InnovATE? a. If so, could you elaborate on this? 4. To what degree would you say that InnovATE enhanced your work performance or your Mission’s/Bureau’s performance? 57 5. Was it always the same person from InnovATE you liaised with? a. If not, how did you find the working relationship? 5. Mission Support 4. Sustainability: 1. How can all InnovATE products and investments best be utilized in the future? a. By the Bureau, U.S. Missions, donors, policymakers, and AET professionals, etc.? b. Are there ways to enhance the usability of the InnovATE products and investment? 2. Would you say that InnovATE products are mainly generic or specific? Why? 3. How easy is it to adapt the InnovATE products to country specific context/the CDCS objectives? 4. How easy it to convert the InnovATE products into specific project proposals and work plans? 5. How InnovATE products support linkages to GFSS? a. Intermediate result 4 Increased Youth Empowerment and Livelihoods? b. Intermediate result 6 Improved Human, Organizational, and System Performance? 6. If you were asked to give one recommendation on how to promote AET in the future work portfolio of the Bureau, what would it be? 58 Interview Guideline 3 - InnovATE Project Implementers [Interviews are intended to be semi-structured and open-ended. The questions below serve as a general guideline with the intention of leaving room for probing questions and focusing an interview on a participant’s interest or area of expertise.] Thank you for agreeing to participate. The results from this evaluation will be written to keep your identity anonymous, meaning that none of your responses will be attached to personally identifying information. 1. General personal information Name: Position: Organization: Interview date: 2. InnovATE Project: Efficiency 1. How did you find InnovATE’s ability to communicate its objectives and potential value? 2. How did you find communication with InnovATE to be? 3. How did you find collaborating with InnovATE U.S. partner universities? 4. How efficient was InnovATE in addressing needs and demands relating to AET systems strengthening? a. What were some of the challenges in reaching or responding to the needs and demands? b. What could be your recommendations to address these challenges? 3. InnovATE Project: Relevance 1. What is importance of the InnovATE interventions to your organization? 2. What is relevance of the InnovATE approach as a tool for AET systems strengthening? 3. Did the matching fund efficiently engage your organization in AET systems strengthening? Why? a. What is your experience with this matching fund mechanism? 4. In what ways were InnovATE products relevant to your organization? a. Which of InnovATE’s services and products did you find most relevant? Why? b. What are your recommendations to improve the relevance of InnovATE products? 5. Did the webinars, blogs, and other online services improve your work performance? How? 4. InnovATE Project: Effectiveness 1. What is your assessment about effectiveness of the InnovATE processes of defining and/or disseminating products and approaches? a. How could the effectiveness be improved for the better outreach and dissemination? 2. What is your assessment about effectiveness of InnovATE products, approaches, and investments to AET systems strengthening? a. How could the effectiveness be improved? 3. To what degree would you say that InnovATE enhanced your work performance or your organization? 4. Do you know of other partners that have used outputs and services from InnovATE? a. If so could you elaborate on this? 5. What were the mechanisms put in place for measuring InnovATE impact? 59 a. What is effectiveness of the mechanisms? b. How could the effectiveness of the mechanisms be improved? 6. What would be the future potential impact of InnovATE to the AET systems? 5. InnovATE Project: Sustainability 1. What opportunities exist for AET professionals to use the information generated by InnovATE for enhanced investment in AET? a. How can U.S. Missions/BFS use these opportunities? 2. How relevant are the InnovATE findings and approaches for target countries’ AET strategies and institutions? a. How can the relevance be improved? 3. How can all InnovATE products and investments best be utilized in the future by your organization? a. Are there ways to enhance the usability of the InnovATE products and investment? 4. Would you say that InnovATE products are mainly generic or specific? Why? 5. How easy is it to adapt the InnovATE products and investments to country specific context/the CDCS objectives? 6. How easy it to convert the InnovATE products into specific project proposals and work plans? 60 Interview Guideline 4—Local Governments, AET Institutions, Private Sector, and Development Partners [Interviews are intended to be semi-structured and open-ended. The questions below serve as a general guideline with the intention of leaving room for probing questions and focusing the interview on participant’s interest or area of expertise.] Thank you for agreeing to participate. The results from this evaluation will be written to keep your identity anonymous, meaning that none of your responses will be attached to personally identifying information. 1. General personal information Name: Position: Organization: Interview date: 2. InnovATE Project: Efficiency 1. Do you know about InnovATE interventions and products? a. Have you used any of these? b. Do you find their products and information an efficient way of communicating and responding to AET issues? Why? 2. What could be done to improve impact efficiency? 3. Have you interacted directly with InnovATE people? a. How did you find collaborating with InnovATE Project implementers? 4. How efficient was InnovATE in addressing your organization’s needs and demands relating to AET systems strengthening? a. What were the challenges in reaching or responding to your needs and demands? b. Do you have any recommendations on how to address these challenges? 3. InnovATE Project: Relevance 1. How relevant are the InnovATE interventions and products to your organization? 2. To what degree do they assist with the broader AET systems strengthening? 3. Compared to other initiatives aiming at AET systems strengthening, how would you assess InnovATE? 4. Which AET systems strengthening initiative do you think has most impact? Why? 5. In what ways were the InnovATE products relevant to your organization? a. Which of InnovATE’s services and products did you find most relevant? Why? b. Do you have any recommendations on how to improve the relevance of InnovATE products? 6. Would you say that InnovATE products are mainly generic or specific? Why? 7. How easy is it to adapt the InnovATE products and investments to your organization’s work on: a. AET system strengthening? b. Relevant value chains? c. Agricultural higher education? d. Other: 61 4. InnovATE Project: Effectiveness 1. What is your assessment on effectiveness of InnovATE products, approaches, and investments in broader AET systems strengthening? a. How can the effectiveness be improved? 2. Do you know of other partners that have used outputs and services from InnovATE? a. If so could you elaborate on this? 3. What were the mechanisms put in place for measuring InnovATE impact? a. How effective are these mechanisms? b. How could the effectiveness of the mechanisms be improved? 5. InnovATE Project: Sustainability 1. Do you or your organization still use outputs and services from InnovATE? a. If so could you elaborate on this? b. To what degree would you say that the utilization of InnovATE products enhanced work performance of you and your organization? 2. How can InnovATE products and investments best be utilized in the future by your organization? a. Are there ways to enhance the usability of the InnovATE products and investment? 3. What are your recommendations for other organization in regard to using InnovATE’s products and approaches? 4. What would be the future potential impact of InnovATE to the AET systems? 5. What are your recommendations to improve sustainability of InnovATE products and impacts? 62 ANNEX 3: ADDENDUM TO PHASE I PROTOCOL FOR PHASE II (2018) ADDENDUM TO WORK ASSIGNMENT DOCUMENT In response to: PEEL Expression of Interest to Evaluate the Innovation for Agricultural Training and Education (InnovATE) Project Award AID-OAA-L-12-0004 March 29, 2018 Submitted by: 4350 East-West Highway, Suite 210 Bethesda, MD 20814 Contact Person: Thomas C. England, President Tel: (301) 652-4334 ◆ Fax: (301) 652-3733 ◆ Email: tengland@engl.com 63 ADDENDUM Table 1: Approach to Addressing Major Issues Identified in InnovATE Draft Report Major issues identified by COR (and AOR) Recommended measures for PEEL to address 1. There is a concern of a skewed sample. The KIIs didn’t cover the range of stakeholders involved in this project, which led to a representation of the project that is biased and less nuanced. This could be addressed by collecting more data from under￾represented stakeholders, displaying a table of KIIs and their organization/affiliation and further elaborate it in the limitations. I’m not certain how much detail on the online survey respondents are available. This also likely led to the evaluation not generating all that much information on the mission-focused questions. ∉ Obtain additional information from under￾represented stakeholders through purposive sample—especially of AET institution and private sector representatives (re. Q1 and Q2), and also policy makers, other donors (re. Q2); and display in table within report ∉ Re-establish links with local contacts utilized for scoping missions in up to five countries— Armenia, Honduras, Nepal, Senegal, Tanzania—to help set up telephone interviews with these additional stakeholders ∉ Re-work questionnaire for the telephone interviews as needed to address evaluation questions 2. There is a concern that the ET doesn’t fully understand the greater project purpose of the leader award. The AOR noted that the project purpose is much greater than a mission support mechanism. This language is suggested for a project description: The InnovATE Project was established by USAID Bureau for Food Security (BFS) to provide development practitioners with cutting￾edge knowledge and training on how to design and implement agricultural education and training (AET) projects needed to build local capacity to support the global Feed the Future program. ∉ Request copies of TOC/LogFrame and M&E Plan for both award and sub-awards (if these documents exist; if not, interpret from available project documentation) ∉ Improve understanding through open-ended discussion of AET applications, both potential and actual, with stakeholders, as part of telephone interviews of stakeholders 3. Conclusion about lack of systems level results. While we acknowledge that results at the systems level weren’t what we had wanted, but the evidence describing the reasons why are more speculative. ∉ As part of telephone interviews of stakeholders, ask them to identify barriers to achieving systems-level results and how they might be overcome ∉ Include within draft recommendations concrete measures to facilitate achievement of systems-level results 4. Mission demand. Define criteria for what is “low” since other leader awards that were funded at much higher level had comparable buy-ins. FSP IL had 6 buy-ins and 6 AAs and was much better funded; MEAS was 4 AAs. ∉ Contextualize findings with respect to other leader awards o See specific comments in draft report 64 Table 1: Approach to Addressing Major Issues Identified in InnovATE Draft Report Major issues identified by COR (and AOR) Recommended measures for PEEL to address 5. Tone. I appreciate the work you already put in to modify the tone, but the tone is still not coming across as neutral as it could be. ∉ Careful copy editing to balance tone o See specific comments in draft report 6. I am also attaching the AOR’s responses to the recommendations. As you can see, some of them were already done by the project, or at least an approximation of them have been carried out—so it’s not particularly useful as a recommendation unless there was something different that should have been done in the process. Some other recommendations she completely agrees with but would like more information and detail as to how to carry this out. Again, specific and actionable recommendations are useful with who clearly laid out (BFS? Mission staff? IP?). ∉ Field test draft recommendations during o Telephone interviews with AET institution and private sector representatives o Teleconference (or webinar) to present preliminary findings, conclusions & recommendations to Mission (and BFS) staff o In both cases: ▪ Open-ended discussion, focusing on specificity & actionability ▪ Identification of to whom the action should fall (BFS, Mission staff, IP?) Table 2: Tasks and Timeline for Completion of InnovATE Performance Evaluation 65 Table 3: Revised Budget for Completion of Innovate Performance Evaluation 66 ANNEX 4: PHASE II (2018) INTERVIEW GUIDES Key Informant Interview (KII) Guides A. Feed the Future POCs/Flagship FtF Activity AORs in Missions Introduction We’re conducting a performance evaluation of InnovATE, a USAID Feed the Future activity. You have been recommended to us as someone to talk to about the project’s key strengths and accomplishments, as well as some of the areas where it could improve. We will not record the interview, nor cite your name in the evaluation or any of the documentation that we provide to USAID. But with your permission, we will note your responses and may use them in our report. Note: Before starting the questionnaire, ask the respondents to describe how they have interacted with InnovATE, and also to describe what their flagship Feed the Future activities are in (country). EQ 1: Defining and disseminating good AET practices A primary InnovATE objective is to define and/or disseminate good practice strategies, approaches, and investments for establishing efficient, effective and financially sustainable agricultural education and training (AET) institutions and systems in developing countries. 1. To what extent would you say that InnovATE has been effective in achieving this objective in (country)? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) a. Why? 2. Has InnovATE generated, documented and disseminated information on key constraints to AET system development that you have found useful? a. If so, please give specific examples. i. Were innovations to address those constraints articulated (Yes/No)? ii. Were stakeholders and collaborators engaged in the process (Yes/No)? 3. Has InnovATE generated information that addresses the specific AET needs of the Mission (Yes/No)? a. If so, please give specific examples i. Were actionable recommendations generated (Yes/No)? 4. Was InnovATE successful in generating Mission investments in AET program design and operations (Yes/No)? a. If so, please provide specific examples. 67 EQ 2: Utilizing good AET practices information InnovATE also aims to encourage key stakeholders—USAID Missions, other donors, policy makers and AET professionals—to utilize information from InnovATE’s good practice papers, analyses, and thematic studies, as well as its training modules and courses. 5. To what extent would you say that InnovATE has been effective in achieving this objective in (country)? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) a. Why? 6. What opportunities are there for these papers, analyses and studies to be used by the different stakeholder groups listed above to enhance AET? Please provide any suggestions. 7. Are there stakeholders among AET institutions, other donors and policy makers in (country) who you would recommend that we speak with about InnovATE? b. If so, what are their names and contact points? c. Would you send an email introducing us? EQ 3: Communicating with Missions 8. How effectively has InnovATE communicated its objectives and potential value to Missions? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) Why? EQ 4: Responsiveness 9. How effectively did InnovATE address Mission demands relating to AET capacity building? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) 10. In what ways could InnovATE better align with Mission strategies relating to AET and educational capacity building? Please provide suggestions. 11. How effectively did InnovATE align with Feed the Future strategies (e.g., desire to support country AET capacity development needs and generate a body of knowledge on good practice in AET strengthening)? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) 12. What did Missions find most valuable about InnovATE project outputs arising from scoping assessments and associate awards (as applicable)? Please provide examples. 68 B. Representatives of AET Institutions, private sector, policy makers Introduction We’re conducting a performance evaluation of InnovATE, a USAID Feed the Future activity. You have been recommended to us as someone to talk to about the project’s key strengths and accomplishments, as well as some of the areas where it could improve. We will not record the interview, nor cite your name in the evaluation or any of the documentation that we provide to USAID. But with your permission, we will note your responses and may use them in our report. Note: Before starting the questionnaire, ask the respondents to describe how they have interacted with InnovATE. EQ 1: Defining and disseminating good AET practices A primary InnovATE objective is to define and/or disseminate good practice strategies, approaches, and investments for establishing efficient, effective and financially sustainable agricultural education and training (AET) institutions and systems in developing countries. 1. To what extent would you say that InnovATE has been effective in achieving this objective in (country)? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) a. Why? 2. Has InnovATE generated, documented and disseminated information on key constraints to AET system development that you have found useful? a. If so, please give specific examples. i. Were innovations to address those constraints articulated (Yes/No)? ii. Were stakeholders and collaborators engaged in the process (Yes/No)? EQ 2: Utilizing good AET practices information InnovATE also aims to encourage key stakeholders—USAID Missions, other donors, policy makers and AET professionals—to utilize information from InnovATE’s good practice papers, analyses, and thematic studies, as well as its training modules and courses. 5. To what extent would you say that InnovATE has been effective in achieving this objective in (country)? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) a. Why? 6. What opportunities are there for these papers, analyses and studies to be used by the different stakeholder groups listed above to enhance AET? Please provide any suggestions. EQ 4: Responsiveness 11. How effectively did InnovATE align with Feed the Future strategies (e.g., desire to support country AET capacity development needs and generate a body of knowledge on good practice in AET strengthening)? (1=Not effective, 2=Low effectiveness, 3=Some effectiveness, 4= Highly effective) 0 ANNEX 5: DATA AND DATA ANALYSIS TABLES Table 1: Monitoring Data and Evaluation Questions Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments 1. To what extent has InnovATE achieved its objective of defining and/or disseminating good practice strategies, approaches, and investments for establishing efficient, effective and financially sustainable agricultural education and training institutions and systems? DoS 1. Number of higher education partnerships between U.S. and host country higher education institutions that address regional, national, and/or local development needs Effective higher education partnerships are key to ensuring that good￾practice strategies, approaches and investments are defined and disseminated to enable efficient, effective, and financially sustainable AET institutions and systems No, the LOP target was adjusted downward from 24 to 11 after FY14. No, the adjusted LOP target was 11, and the number achieved was 8. The annual reports stated that the target was adjusted downward because the projected number of AAs was lower than anticipated. T9. Regional/ international symposia hosted International symposia are useful in fostering partnerships, engaging professionals and disseminating best practices Yes, the target was one per year, reasonable considering the difficult logistics of these symposia Yes - 1.a. In what ways has the project generated, documented, and disseminated information on key constraints to AET systemic development, innovations to address those constraints, and engaged stakeholders and collaborators to contribute? L3. Studies completed (background/ thematic studies) Studies document and disseminate information than can be applied to solutions to overcome significant constraints Targets may have been low for this type of program, and they were not adjusted after the annual target was exceeded in FY15 by more than 50 percent. Yes. The FY 15 result—57 percent over target— would have generated a deviancy report and might have been the occasion to increase the targets in the final years. L4. Technical notes and good practice papers disseminated Technical notes and other informal communications on good practices are an efficient way to inform stakeholders about available solutions LOP target was exceeded by 60 percent—so the targets appear to have been too low; they were not adjusted after FY14, when the annual target was exceeded by 100 percent Yes. - 1 Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments 1.b. In what ways did/did not the information generated address the specific needs of the Missions requesting and provide usable, actionable recommendations? D7. AET assessment tools developed Usable and actionable assessments are the basis for ongoing improvements in AET programming LOP target of 5 seems low but 14 tools were developed which is high; most of these were developed in the first year of the project. Yes. Although it is listed under “Design” indicators, most (60 percent) of these tools were developed in the first year of the project, so these tools were not developed in response to specific Mission needs. Instead, according to the COP, these tools were utilized as guides for conducting scoping visits in order to assess Mission needs. 1.c. How effective were project outputs in achieving outcomes in terms of Mission AET program investment, design, and operations? D5. AET reform and investment plans implemented Mission implementation of AET reform and investment plans are a measure of the effectiveness of InnovATE outputs in terms of producing desired outcomes. No, the LOP target of 8 plans was not adjusted, although it was clear by FY15 that the targets were not going to be met. No, the LOP target of 8 and the number achieved was only 2. The explanation of the implementer was that the target was missed because the number of AAs was lower than anticipated. D6. AET institutions strengthened The number of AET institutions strengthened as an outcome of InnovATE support is a measure of the effectiveness of the activity. No, the LOP target of 20 AET institutions strengthened was not adjusted, although it was clear by FY14 that it was not going to be met. No, the LOP target was 20 and the number achieved was only 7, of which 6 occurred in the final project year - 2 Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments 2. In what ways have USAID Missions, donors, policy makers, and AET professionals used information generated from InnovATE’s good practice papers, analyses, and thematic studies, as well as the training modules and training courses (from both the TRAIN and LEARN components)? FF2. Number of public-private partnerships formed as a result of FTF assistance Formation of PPPs around AET good practices and training would be one way in which information generated by InnovATE might be used. No, the LOP target was adjusted downward from 11 to 7 in FY15, but despite this, no PPPs were developed. No PPPs were formed. The implementer noted in its FY17 reporting that “When the public/private partnership indicator targets were originally set, InnovATE anticipated fostering public-private partnerships through associate awards. InnovATE did not have an associate award in Year 3 or 4 and the core project had no on-the-ground activities.” L2. Total number of times training database accessed (originally “Number of users accessing project databases”) The InnovATE website is a good source for high￾quality papers, analyses and studies, but it is not possible based on this indicator to determine in what ways the information is utilized by stakeholders. No, they were too low. The LOP target of 5,000 hits on the project database was exceeded in the project’s second year and the hits more than doubled in the third year, following which the indicator was dropped. Yes. - T7. Development practitioners trained in AET Training development practitioners is a way of extending the reach of AET good practice papers, analyses, and thematic studies, but it is not possible based on this indicator to determine in what ways the information is utilized by stakeholders. Yes, the LOP target of 320 seems reasonable. Yes, LOP targets were exceeded by more than 14 percent. - 3 Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments 2.a. What opportunities are there for these papers, analyses and studies to be used by the different stakeholder groups to enhance AET? T1. Training workshops hosted A program of regular training workshops especially those aimed at particular stakeholder groups is an opportunity to demonstrate the usefulness of program products. Yes. Yes. - T2. Short term training supported for curriculum development Strengthening AET capacity in curriculum development through short courses will help ensure that local institutions are utilizing program products and technical assistance. No. The LOP target was 40 short courses, which was adjusted downward to 20 and still not met. No targets were met—only one workshop was held. Apparently, it was believed that this training support would occur only in FY14 and FY15, but when this did not occur, the task was dropped; we need to find out why. T5. Policy makers trained in AET Training of policy makers is of paramount importance in ensuring that there is an enabling environment for AET improvement. Yes, but the FY15 achievement was only 33 percent of target, and the project never caught up. No, the LOP target was missed by more than 13 percent. Why there was such a dramatic drop in FY15 needs to be explained by looking in the annual reports. L5. Academic papers published Papers in peer-reviewed journals reach stakeholders beyond the countries where the program is being implemented and ensure that InnovATE’s results reach a wider academic and scientific community. Yes, the LOP target of 10 academic papers published over a period of 5 years was actually quite low considering the number of academic institutions engaged in the project. No, only 2 were published. We need to find out why this was missed. 4 Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments 3. How effectively has InnovATE communicated its objectives and potential value to missions? T8. Development professionals trained in AET Training USAID (and other donor) development professionals and practitioners is a way of communicating the value of the program and providing a foundation for future program designs. Yes. Yes, LOP targets were exceeded by more than 14 percent. - 3.a. What did missions find most valuable about InnovATE project outputs arising from scoping assessments and associate awards (as applicable)? D1. Consultancies for AET development linkages and AET support services To the extent that consultancies for AET development linkages and support services occurred, they are an indication that the Missions found InnovATE project outputs valuable; but the indicator does not provide information on the types of support they found most valuable. Yes, the LOP target of 16 consultancies seems reasonable. No, although 13 consultancies were commissioned, the number fell short of the LOP target. - - D2. AET system program evaluations As with AET consultancies, AET program evaluations are an indication that Missions found InnovATE products valuable, but the indicator does not provide information on the types of support they found most valuable. Yes, although how a target of 11 AET program evaluations was set, it appears reasonable for this scale of activity. Yes, the LOP achievement of 10 AET program evaluations was less than 10 percent below the LOP target. - 5 Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments 4. How effectively did InnovATE address Mission demands relating to AET capacity building? D6. AET institutions strengthened (Note: overlaps with EQ1.c, effectiveness of project outputs in achieving desired Mission outcomes) The number of AET institutions that are strengthened is an indication that Mission demands are being addressed in relation to AET capacity building. No, the LOP target of 20 AET institutions strengthened was not adjusted, although it was clear by FY14 that it was not going to be met. No, the LOP target was 20 and the number achieved was only 7, of which 6 occurred in the final project year. The explanation of the implementer was that the target was missed because the number of AAs was lower than anticipated. 4.a. In what ways could InnovATE better align with Mission strategies relating to AET and educational capacity building Not amenable to being addressed through review of indicators and results. - 4.b. How effectively did InnovATE align with Feed the Future (as well as other USAID strategies) (e.g., desire to support country AET capacity development needs and generate a body of knowledge on good practice in AET strengthening)? FF3. Number of institutions/ organizations undergoing capacity/competency assessments as a result of USG assistance The number of institutions/organizations undergoing capacity/competency assessments is a direct indicator of alignment with Feed the Future’s intention to support AET capacity development. No, the LOP target of 25 institutions was exceeded by 100 percent, an indication that the targets were set too low. Yes. Again, this indicator should have generated deviancy reports beginning in FY14, the question being why were the targets never adjusted. FF4. Number of institutions/ organizations undertaking capacity/competency strengthening as a result of USG assistance The number of institutions/organizations undertaking capacity/competency strengthening is a direct indicator of alignment with FTF’s intention to support AET capacity development. No, the LOP target of 7 institutions undertaking capacity/competency strengthening was fallen short by nearly threefold. No. The LOP number achieved was seven (7) as compared with a target of 20. - 6 Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments T3. Training modules developed The continuing development of training modules to meet changing needs of AET institutions is necessary to support capacity development and dissemination of up-to￾date knowledge, in accordance with FTF support to AET capacity development. No, the targets were missed in the first three project years, and yet not adjusted. No, the LOP achievement fell short of the LOP target by more than 50 percent. - T4. Number of training module users The number of unique views of AET training modules is an indication that Feed the Future support to AET capacity development is succeeding. No, because once the modules were online (FY15), the number of users greatly exceeded targeted numbers. Yes, LOP targets were exceeded by 70 percent. This is another case in which the targets were exceeded by more than enough starting in FY15 to generate a deviancy report, resulting in an upward adjustment of the targets. T5. Total number of training module views. The total number of page views is also an indication that Feed the Future support to AET capacity development is succeeding. No, because the total number of page views greatly exceeded targeted numbers throughout the calculation of this indicator. Yes, LOP targets were exceeded by nearly 90 percent. This is another case in which the targets were exceeded by more than enough in FY15 to generate a deviancy report, resulting in an upward adjustment of the targets. 5. What were some of the implementation challenges and how did the project address them? Not amenable to being addressed through review of indicators and results. 7 Evaluation Questions InnovATE Indicators Relationship of Indicator to Evalution Question Were LOP Targets Reasonable? Were LOP Targets Met? Comments 5.a. In what ways could implementation have been improved to more efficiently and effectively carry out scoping assessments, disseminate results and encourage project requests? Not amenable to being addressed through review of indicators and results. 8 Table 2. Interviewee Affiliations 2017 2018 USAID/Armenia USAID/Senegal USAID/Armenia USAID/Senegal USAID/Armenia USAID/Nepal USAID/BFS USAID/Nepal USAID/BFS USAID/Honduras USAID/BFS USAID/Honduras USAID/BFS USAID/Honduras USAID/BFS USAID/Armenia Lead Institution/InnovATE USAID/Armenia Lead Institution/InnovATE USAID Nicaragua U.S. Partner/InnovATE Key Stakeholder Honduras U.S. Partner /InnovATE Key Stakeholder Honduras U.S. Partner/InnovATE Key Stakeholder Honduras U.S. Partner/InnovATE Key Stakeholder Armenia U.S. Partner/InnovATE FtF flagship project COP/Nepal U.S. Partner/InnovATE FtF flagship project COP/Honduras U.S. Partner/InnovATE InnovATE/Honduras U.S. Partner/InnovATE InnovATE/Honduras U.S. Partner/InnovATE Director/InnovATE U.S. Partner/InnovATE BFS/AOR U.S. Partner/InnovATE N/A U.S. Partner/InnovATE N/A Other U.S. Institutions N/A Other U.S. Institutions N/A Key Stakeholder/Uganda N/A Key Stakeholder/Uganda N/A FtF Project Staff N/A 9 Table 3: Phase II (2018) Interviewees and Rankings Affiliation Aware EQ1 EQ2 EQ3 EQ4 EQ5 USAID/Senegal 1 3 0 3 4 3 USAID/Senegal 1 1 0 0 0 0 USAID/Nepal 1 2 1 1 0 0 USAID/Nepal 1 1 0 0 0 0 USAID/Honduras 0 2 1 4 3 0 USAID/Honduras 1 4 4 4 4 3 USAID/Honduras 1 3 3 2 2 4 USAID/Armenia 1 2 0 0 0 0 USAID/Armenia 0 4 3 4 3 3 USAID Nicaragua 1 4 4 4 4 4 Key Stakeholder Honduras 1 4 0 0 0 0 Key Stakeholder Honduras 1 3 0 0 0 0 Key Stakeholder Honduras 1 4 0 0 0 0 Key Stakeholder Armenia 1 4 0 0 0 3 FtF flagship project COP/Nepal 1 1 1 0 0 0 FtF flagship project COP/Honduras 0 1 0 0 0 0 InnovATE/Honduras 0 3 4 3 3 3 InnovATE/Honduras 1 4 0 0 0 0 Director/InnovATE 1 0 0 0 0 0 BFS/AOR 1 0 0 0 0 0 0=no answer [discounted for total] 1=not effective, 2=low effectiveness, 3=moderate effectiveness, 4=highly effective 10 Table 4: Phase II (2018) Keyword Frequencies and Contexts PHASE 1(2017) INTERVIEWS Negative Positive Neutral Article/s 1 0 0 Capacit/y/ies 13 4 9 Challeng/e/es/ing 8 14 16 Collaborat/e/es/ing/ation 1 5 9 Communicat/e/es/ing/ion 7 4 34 Demand 12 7 8 Disseminat/e/es/ing/ion 0 4 9 Educat/e/ing/ed/ion 18 10 138 Engag/e/es/ing/ment 15 12 25 Focus/es/ed/ing 15 15 24 Gender/s 1 5 6 Impact/s/ed/ing/ful 2 6 19 Interact/s/ed/ing/ion 2 3 3 Knowledge 0 8 4 Manag/e/es/ing/ed/ement 10 12 52 Mission/s 63 35 124 Organiz/e/es/ed/ing/ation/ational 5 2 14 Outcome/s 1 2 3 Output/s 2 3 5 Paper/s 7 3 7 Partner/s/ed/ing/ship 8 15 29 Polic/y/ies/maker/s 1 0 9 Research/es/ed 5 5 17 Respon/d/ds/ed/ing/se/ses/ive 6 4 3 Strateg/y/ic/ies/ize/ically 6 12 11 Student/s 7 3 26 Stud/y/ies 5 9 18 Train/s/ed/ing 5 3 62 Valu/e/able 8 9 12 Website/s 3 2 5 11 Table 5: Hierarchy and Sustainability of Changes5 Knowledge-- Broadening research, policy, implementation and enterprise horizons Values-- Expanding research, policy, implementation and enterprise capacities Action—Affecting research, policy, implementation and enterprise regimes ∉ Providing opportunities for networking/learning within the domain or with colleagues elsewhere. ∉ Introducing new concepts to frame debates, putting ideas on the agenda, or stimulating debate. ∉ Educating researchers and others who take up new positions with broader understanding of issues. ∉ Stimulating dialogue among decision- makers6 ∉ Changing perceptions and public opinion. ∉ Setting an agenda by reframing the way an issue is debated and creating pressure for change.7 ∉ Promote and support access to public information. ∉ Framing debates and getting issues on to the political agenda: attitudinal change, drawing attention to new issues and affecting the awareness, attitudes or perceptions of key stakeholders. ∉ Encouraging discursive commitments from states and other policy actors: affecting language and rhetoric is important to, for example, promote recognition of specific groups or endorsements of international declarations.8 ∉ Improving the knowledge/ data of certain actors. ∉ Supporting research users to develop innovative ideas. ∉ Improving capabilities to communicate ideas. ∉ Developing new talent and institutional arrangements for research and analysis ∉ Building networks that support delivery of change. ∉ Developing capacity within organizations to allow them to understand and respond to an issue. ∉ Citizen engagement--enlarging or strengthening democratic spaces. ∉ Supporting people-centered policy making. ∉ Empowering traditionally excluded groups. ∉ Influencing behavior change in key actors: policy change requires changes in behavior and implementation at various levels in order to be meaningful and sustainable. ∉ Enhancing collaboration and alignment within networks. ∉ Developing CSOs’ capacity to monitor and evaluate policies, program management and implementation. ∉ Increasing civil society participation in relevant policy and implementation processes. ∉ Increasing civil society participation in relevant policy and implementation processes. ∉ Strengthening state agencies’ capacity to interact with citizens, communities and their representatives.9 ∉ Modification of existing programs or policies. ∉ Fundamental re-design of programs, policies, strategies and initiatives. ∉ Changing institutions, such as influencing strategy and resource allocations within organizations, institutions. ∉ Securing procedural change at domestic or international level: changes in the process whereby policy decisions are made, such as opening new spaces for policy dialogue. ∉ Affecting policy content: while legislative change is not the sum total of policy change, it is an important element. 5 Adapted from ESPA guide to working with Theory of Change for research projects Isabel Vogel. 6 Lindquist, E. (2001) Discerning Policy Influence: Framework for a Strategic Evaluation of IDRC-Supported Research, Working Paper, School of Public Administration, University of Victoria 7 Steven D. (2007), Evaluation and the New Public Diplomacy, presentation to the Future of Public Diplomacy. 842nd Wilton Park Conference, River Path Associates. 8 Weyrauch, V. and Díaz Langou, G.L. (2011) “Sound expectations: from impact evaluations to policy change”, Working Paper No. 12, Delhi: 3ie and the Centre for the Implementation of Public Policies Promoting Equity and Growth. 9 Jones, H. (2011), ‘A guide to monitoring and evaluating policy influence’, Background Paper, Overseas Development Institute. 12 Table 6: Frequencies and Contexts for Keywords PHASE 1(2017) INTERVIEWS Negative Positive Neutral Article/s 1 0 0 Capacit/y/ies 13 4 9 Challeng/e/es/ing 8 14 16 Collaborat/e/es/ing/ation 1 5 9 Communicat/e/es/ing/ion 7 4 34 Demand 12 7 8 Disseminat/e/es/ing/ion 0 4 9 Educat/e/ing/ed/ion 18 10 138 Engag/e/es/ing/ment 15 12 25 Focus/es/ed/ing 15 15 24 Gender/s 1 5 6 Impact/s/ed/ing/ful 2 6 19 Interact/s/ed/ing/ion 2 3 3 Knowledge 0 8 4 Manag/e/es/ing/ed/ement 10 12 52 Mission/s 63 35 124 Organiz/e/es/ed/ing/ation/ational 5 2 14 Outcome/s 1 2 3 Output/s 2 3 5 Paper/s 7 3 7 Partner/s/ed/ing/ship 8 15 29 Polic/y/ies/maker/s 1 0 9 Research/es/ed 5 5 17 Respon/d/ds/ed/ing/se/ses/ive 6 4 3 Strateg/y/ic/ies/ize/ically 6 12 11 Student/s 7 3 26 Stud/y/ies 5 9 18 Train/s/ed/ing 5 3 62 Valu/e/able 8 9 12 Website/s 3 2 5